certified-kubernetes-administrator-course Documentation

Certified Kubernetes Administrator - CKA Course

10,915 stars Shell Shell 100 files · ~49,729 tokens
RAW Doc

Technical Documentation: kodekloudhub/certified-kubernetes-administrator-course

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (README.md)

Certified Kubernetes Administrator (CKA) Course

These are notes from the Certified Kubernetes Administrator Course hosted on KodeKloud.

Sections

16 Ultimate Mocks README (`docs/16-Ultimate-Mocks/README.md`)

CKA Ultimate Mocks

NOTE: CKA Ultimate Mocks is a separate course from the main CKA course, and as such requires a separate payment or is included in Pro subscription.

In this section, we will go through some of the most troublesome questions - these being the ones that get the most requests for help on our various forums.

17 Tips And Tricks README (`docs/17-tips-and-tricks/README.md`)

Tips and Tricks

In this bonus section we will discuss some useful tips that can be used preparation for the exam

)

16 Ultimate Mocks 02 Troubleshooting README (`docs/16-Ultimate-Mocks/02-Troubleshooting/README.md`)

SECTION: Troubleshooting

16 Ultimate Mocks 04 Storage README (`docs/16-Ultimate-Mocks/04-Storage/README.md`)

SECTION: Storage

16 Ultimate Mocks 05 Services Networking README (`docs/16-Ultimate-Mocks/05-Services-Networking/README.md`)

SECTION: Services/Networking

01 Introduction 01 Course Introduction (`docs/01-Introduction/01-Course-Introduction.md`)

Course Introduction

This course focuses on Administration part of the kubernetes.

Course Structure

  • Lectures
  • Demos
  • Quizzes
  • Practice Questions
  • Q & A Section

Pre-Requisites

  • Docker
  • Basics of kubernetes
    • PODS, Deployments, Services ...
  • YAML
  • Setting up a basic lab environment with VirtualBox

If you are a beginner we highly recommend Kubernetes for the Absolute Beginners Hands-On course

Course Objectives

The objectives of this course are aligned to match the Certified Kubernetes Administration Exam (CKA)

  • Core Concepts
    • Cluster Architecture
    • API Primitives
    • Services & Other Network Primitives
  • Scheduling
    • Labels & Selectors
    • Daemon Sets
    • Resource Limits
    • Multiple Schedulers
    • Manual Scheduling
    • Scheduler Events
    • Configure Kubernetes Scheduler
  • Logging & Monitoring
    • Monitor Cluster Components
    • Monitor Cluster Components Logs
    • Monitor Applications
    • Application Logs
  • Application Lifecycle Management
    • Rolling Updates and Rollbacks in Deployments
    • Configuring Applications
    • Scale Applications
    • Self-Healing Applications
  • Cluster Maintenance
    • Cluster Upgrade Process
    • Operating System Upgrades
    • Backup and Restore Methodologies
  • Security
    • Authentication & Authorization
    • Kubernetes Security
    • Network Policies
    • TLS Certificates for Cluster Components
    • Image Security
    • Network Policies
    • Security Contexts
    • Secure Persistent Key Value Store
  • Storage
    • Persistent Volumes
    • Access Modes for Volumes
    • Persistent Volume Claims
    • Kubernetes Storage Object
    • Configure Applications with Persistent Storage
  • Networking
    • Pre-Requisites - Network, Switching, Routing, Tools
    • Pre-Requisites - Network Namespaces
    • Pre-Requisites - Networking in Docker
    • Networking Configuration on Cluster Nodes
    • Service Networking
    • POD Networking Concepts
    • Network Loadbalancer
    • Ingress
    • Cluster DNS
    • CNI
  • Installation, Configuration & Validation
    • Design a Kubernetes Cluster
    • Install Kubernetes Master and Nodes
    • Secure Cluster Communication
    • HA Kubernetes Cluster
    • Kubernetes Release Binaries
    • Provision Infrastructure
    • Choose a Network Solution
    • Kubernetes Infrastructure Config
    • Run & Analyze end-to-end test
    • Node end-to-end tests
  • Troubleshooting
    • Application Failure
    • Control Plane Failure
    • Worker Node Failure
    • Networking

Practice Tests

CKA exam is a practical hands-on exam it is very important to practice what you learn. Which is why we build a custom solution that will give you access to a Real Kubernetes Environment right in your browser along with Quiz Portal

Kubernetes the hard way

01 Introduction 02 Certification (`docs/01-Introduction/02-Certification.md`)

Certification

Certification Details

Use the code - KUBERNETES15 - while registering for the CKA or CKAD exams at Linux Foundation to get a 15% discount.

02 Core Concepts 01 Core Concepts Section Introduction (`docs/02-Core-Concepts/01-Core-Concepts-Section-Introduction.md`)

Core Concepts Section Introduction

In this section, we will take a look at the below

  • Cluster Architecture
  • API Primitives
  • Services & Other Network Primitives

k8s reference docs:

02 Core Concepts 02 Cluster Architecture (`docs/02-Core-Concepts/02-Cluster-Architecture.md`)

Cluster Architecture

In this section , we will take a look at the kubernetes Architecture at high level.

  • 10,000 Feet Look at the Kubernetes Architecture

K8s Reference Docs:

02 Core Concepts 03 Docker Vs ContainerD (`docs/02-Core-Concepts/03-Docker-vs-ContainerD.md`)

Docker vs. ContainerD

In this section we will look at the differences between Docker and ContainerD

So you’re going to come across Docker and containerd many times. Going forward, when you read older blogs or documentation pages , you’ll see Docker mentioned along with Kubernetes and when you read newer blogs you’ll see containerd and you’ll wonder what the difference is between the two. And there are a few CLI tools like ctr, crictl or nerdctl and you’ll wonder what are these CLI tools and which one should you be using, so that’s what I’m going to explain.

Let’s go back in time to the beginning of the container era. In the beginning, there was just Docker. There were a few other tools like Rocket (rkt), but Docker’s user experience made working with containers super simple and hence Docker became the most dominant container tool. And then came Kubernetes to orchestrate Docker so Kubernetes was built to orchestrate Docker specifically in the beginning so Docker and Kubernetes were tightly coupled and back then Kubernetes only worked with Docker and didn't support any other container solutions.

And then Kubernetes grew in popularity as a container orchestrator and now other container runtimes like rkt wanted in so Kubernetes users needed it to work with container runtimes that are other than just Docker, and so Kubernetes introduced an interface called container runtime interface or CRI, so CRI allowed any vendor to work as a container runtime for Kubernetes as long as they adhere to the OCI standards.
So OCI stands for Open Container Initiative and it consists of an image spec and a runtime spec. Image spec means the specifications on how an image should be built. It defines the specifications on how an image should be built and the runtime spec defines the standards on how any container runtime should be developed so keeping these standards in mind, anyone can build a container runtime that can be used by anybody to work with Kubernetes, so that was the idea.

So rkt and other container runtimes that adhere to the OCI standards were now supported as container runtimes for Kubernetes via the CRI, however Docker wasn’t built to support the CRI standards, remember Docker was built way before CRI was introduced and Docker still was the dominant container tool used by most, so Kubernetes had to continue to support Docker as well and so Kubernetes introduced what is known as dockershim which was a hacky but temporary way to support Docker outside of the CRI.

So while most other container runtimes worked to the CRI, Docker continued to work without it, so now you see Docker isn’t just a container runtime alone. Docker consists of multiple tools that are put together, for example the Docker CLI, the Docker API, the build tools that help in building images. There was support for volumes, security, and finally the container runtime called runc, and the daemon that managed runc, that was called containerd. So containerd is CRI compatible and can work directly with Kubernetes as all other runtimes, so containerd can be used as a runtime on its own separate from Docker.

So now you have containerd as a separate runtime and Docker separately, so Kubernetes continued to maintain support for Docker engine directly however having to maintain the dockershim was an unnecessary effort and added complications so it was decided in v1.24 release of Kubernetes to remove dockershim completely and so support for Docker was removed. But you see all the images that were built before Docker was removed so all the Docker images continued to work because Docker followed the image spec from the OCI standards so all the images built by Docker follow the standard so they continued to work with containerd but Docker itself was removed as a supported runtime from Kubernetes. So that’s kind of the whole story, now let’s look into containerd more specifically.

So containerd although is part of Docker, is a separate project on its own now and is a member of CNCF with the graduated status, so you can now install containerd on its own without having to install Docker itself so if you don’t really need Docker’s other features you could ideally just install containerd alone. So typically we ran containers using the docker run command when we had Docker and if Docker isn’t installed then how do you run containers with just containerd? Now, once you install containerd it comes with a command line tool called ctr, and this tool is solely made for debugging containerd and is not very user friendly as it only supports a limited set of features and this is all you can see in the documentation pages for this particular tool. So for the other than limited set of features that require any other way you want to interact with containerd you have to rely on making API calls directly which is not the most user friendly way for us to operate.

So just to give you an idea, the ctr command can be used to perform basic container-related activities such as pull images, for example to pull redis image you would run

text
ctr images pull docker.io/library/redis:alpine

To run a container we use the ctr run command

text
ctr run docker.io/library/redis:alpine redis

But as I mentioned, this tool is solely for debugging containerd and is not very user friendly and is not to be used for managing containers on a production environment.
So a better alternative recommended is the nerdctl tool. So the nerdctl tool is a command line tool that’s very similar to Docker, so it’s a Docker-like CLI for containerd. It supports most of the CLI options that Docker supports and apart from that it has the added benefit that it can give us access to the newest features implemented in containerd, so for example we can work with encrypted container images or other new features that will eventually be implemented into the regular Docker command in the future. It also supports lazy pulling of images, P2P image distribution, image signing and verifying and namespaces in Kubernetes which are not available in Docker. So the nerdctl tool works very similar to Docker cli, so instead of Docker you would simply have to replace it with nerdctl so it can run almost all Docker commands that interact with containers like this

So that’s pretty easy and straightforward so now that we have talked about ctr and the nerdctl tool, it’s important to talk about another command like tool known as crictl. So earlier we talked about the CRI which is a single interface used to connect CRI compatible container runtimes, containerd, rkt and others. So the crictl is a command line utility that is used to interact with the CRI compatible container runtime, so this is kind of interaction from the Kubernetes perspective. So this tool is developed and maintained by the Kubernetes community and this tool works across all the different container runtimes and because earlier you had the ctr and nerdctl utility that was built by the containerd community specifically for containerd, but this particular tool is from the Kubernetes perspective that works across different container runtimes.

So it must be installed separately and is used to inspect and debug container runtimes so this again is not ideally used to create containers unlike the Docker or the nerdctl utility but is again a debugging tool. You can technically create containers with the crictl utility but it’s not easy. It’s only to be used for some special debugging purposes. And remember that it kind of works along with the kubelet so we know that the kubelet is responsible for ensuring that a specific number of containers or pods are available on a node at time, so if you kind of go through the crictl utility and try and create containers with it, then eventually kubelet is going to delete them because kubelet is unaware of some of those containers or pods that are created outside of its knowledge so anything that it sees it’s going to go and delete it, so because of those things remember that the crictl utility is only used for debugging purposes and getting into containers and all of that.

So let’s look at some of the command line examples so you simply run the crictl command for this and this can be used to perform basic container-related activities such as pull images, or list existing images, list containers, very similar to the Docker command where you use the PS commands, so in Docker you run the ps command, and here you run the crictl ps command and to run a command in since a container docker remember we use the exec command and it’s the same here and along with the same options such as -i and -t and you specify the container id. The view the logs, you use the crictl logs command, again very similar to the docker command.

One major difference is that the crictl command is also aware of pods so you can list pods by running the crictl pods command so this wasn’t something that Docker was aware of. So while working with Kubernetes in the past, we used Docker commands a lot to troubleshoot containers and view logs especially on the worker nodes and now you’re going to use the crictl command to do so. So the syntax is a lot similar and so it shouldn’t be really hard.
So here’s a chart that lists the comparison between the Docker and crictl command line tools. So as you can see, a lot of command such as attach exec, images, info, inspect, logs, ps, stats, version etc., work exactly the same way, and some of the commands to create, remove and start and stop images work similarly too. So a full list of differences can be found in this link.

docker cli crictl Description Unsupported Features
attach attach Attach to a running container --detach-keys, --sig-proxy
exec exec Run a command in a running container --privileged, --user, --detach-keys
images images List images
info info Display system-wide information
inspect inspect, inspecti Return low-level information on a container, image or task
logs logs Fetch the logs of a container --details
ps ps List containers
stats stats Display a live stream of container(s) resource usage statistics Column: NET/BLOCK I/O, PIDs
version version Show the runtime (Docker, ContainerD, or others) version information

So, since as I mentioned, crictl can be used to connect to any CRI compatible runtime, remember to set the right endpoint if you have multiple container runtimes configured, or if you want crictl to interact with a specific runtime, for example if you haven’t configured anything by default it’s going to connect to these sockets in this particular order, so it’s going to try and connect to dockershim first, then containerd, then CRI-O, then the CRI-dockerd – that’s kind of the order that it falls. But if you want to override that and set a specific endpoint, you use the --runtime-endpoint option with the crictl command line, or you could use the CONTAINER_RUNTIME_ENDPOINT environment variable. Set the environment variable to the right endpoint.

So to summarize we have the ctr command line utility that comes with containerd and works with containerd which is used for debugging purposes only and has a very limited set of features, so ideally you wouldn’t be using this at all so you can kind of ignore this. Then we have the nerdctl CLI which is again from the containerd community but this is a Docker-like CLI for containerd used for general purpose to create containers and supports the same or more features than Docker CLI, so it’s something that I think we’ll be using a lot more going forward. Then we have the crictl utility which is from the Kubernetes community and mainly used to interact with CRI compatible runtimes, so it’s not just for containerd – this can be used for all CRI supported runtimes – again this is mainly to be used for debugging purposes.

So if we look at the comparisons here, you can see that ctr and crictl are used mainly for debugging purposes, whereas the nerdctl is used for general purpose. The ctr and nerdctl are from the containerd community and work with containerd, whereas crictl is from the Kubernetes community and works across all CRI compatible runtimes.
So our labs originally had Docker installed on all the nodes so we used the Docker commands to troubleshoot, but now it’s all containerd so remember to use the crictl command instead to troubleshoot.

02 Core Concepts 04 ETCD For Beginners (`docs/02-Core-Concepts/04-ETCD-For-Beginners.md`)

ETCD for Beginners

In this section, we will take a quick look at introduction to ETCD for beginners.

  • What is ETCD?
  • What is a Key-Value Store?
  • How to get started quickly with ETCD?
  • How to operate ETCD?

What is a ETCD?

text
- ETCD is a distributed reliable key-value store that is simple, secure & Fast.

What is a Key-Value Store

  • Traditionally, databases have been in tabular format, you must have heard about SQL or Relational databases. They store data in rows and columns

  • A Key-Value Store stores information in a Key and Value format.

Install ETCD

  • It's easy to install and get started with ETCD.

    • Download the relevant binary for your operating system from github releases page (https://github.com/etcd-io/etcd/releases)

      For Example: To download ETCD v3.5.6, run the below curl command

      text
      $ curl -LO https://github.com/etcd-io/etcd/releases/download/v3.5.6/etcd-v3.5.6-linux-amd64.tar.gz

    • Extract it.

      text
      $ tar xvzf etcd-v3.5.6-linux-amd64.tar.gz

    • Run the ETCD Service

      text
      $ ./etcd

    • When you start ETCD it will by default listen on port 2379

    • The default client that comes with ETCD is the etcdctl client. You can use it to store and retrieve key-value pairs.

      text
      Syntax: To Store a Key-Value pair
$ ./etcdctl put key1 value1
      text
      Syntax: To retrieve the stored data
$ ./etcdctl get key1
      text
      Syntax: To view more commands. Run etcdctl without any arguments
$ ./etcdctl

    K8s Reference Docs:

02 Core Concepts 05 ETCD In Kubernetes (`docs/02-Core-Concepts/05-ETCD-in-Kubernetes.md`)

ETCD in Kubernetes

In this section, we will take a look at ETCD role in kubernetes

ETCD Datastore

  • The ETCD Datastore stores information regarding the cluster such as Nodes, PODS, Configs, Secrets, Accounts, Roles, Bindings and Others.
  • Every information you see when you run the kubectl get command is from the ETCD Server.

Setup - Manual

  • If you setup your cluster from scratch then you deploy ETCD by downloading ETCD Binaries yourself
  • Installing Binaries and Configuring ETCD as a service in your master node yourself.
    text
    $ wget -q --https-only "https://github.com/etcd-io/etcd/releases/download/v3.3.11/etcd-v3.3.11-linux-amd64.tar.gz"

Setup - Kubeadm

  • If you setup your cluster using kubeadm then kubeadm will deploy etcd server for you as a pod in kube-system namespace.
    text
    $ kubectl get pods -n kube-system

Explore ETCD

  • To list all keys stored by kubernetes, run the below command
    text
    $ kubectl exec etcd-master -n kube-system -- sh -c "ETCDCTL_API=3 etcdctl --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt get / --prefix --keys-only"
  • Kubernetes Stores data in a specific directory structure, the root directory is the registry and under that you have various kubernetes constructs such as minions, nodes, pods, replicasets, deployments, roles, secrets and Others.

ETCD in HA Environment

  • In a high availability environment, you will have multiple master nodes in your cluster that will have multiple ETCD Instances spread across the master nodes.
  • Make sure etcd instances know each other by setting the right parameter in the etcd.service configuration. The --initial-cluster option where you need to specify the different instances of the etcd service.

K8s Reference Docs:

02 Core Concepts 06 Kube API Server (`docs/02-Core-Concepts/06-Kube-API-Server.md`)

Kube API Server

In this section, we will talk about kube-apiserver in kubernetes

Kube-apiserver is the primary component in kubernetes.

  • Kube-apiserver is responsible for authenticating, validating requests, retrieving and Updating data in ETCD key-value store. In fact kube-apiserver is the only component that interacts directly to the etcd datastore. The other components such as kube-scheduler, kube-controller-manager and kubelet uses the API-Server to update in the cluster in their respective areas.

Installing kube-apiserver

  • If you are bootstrapping kube-apiserver using kubeadm tool, then you don't need to know this, but if you are setting up using the hardway then kube-apiserver is available as a binary in the kubernetes release page.
    • For example: You can downlaod the kube-apiserver v1.13.0 binary here kube-apiserver
      text
      $ wget https://storage.googleapis.com/kubernetes-release/release/v1.13.0/bin/linux/amd64/kube-apiserver

View kube-apiserver - Kubeadm

  • kubeadm deploys the kube-apiserver as a pod in kube-system namespace on the master node.
    text
    $ kubectl get pods -n kube-system

View kube-apiserver options - Kubeadm

  • You can see the options with in the pod definition file located at /etc/kubernetes/manifests/kube-apiserver.yaml
    text
    $ cat /etc/kubernetes/manifests/kube-apiserver.yaml

View kube-apiserver options - Manual

  • In a Non-kubeadm setup, you can inspect the options by viewing the kube-apiserver.service

    text
    $ cat /etc/systemd/system/kube-apiserver.service

  • You can also see the running process and effective options by listing the process on master node and searching for kube-apiserver.

    text
    $ ps -aux | grep kube-apiserver

K8s Reference Docs:

02 Core Concepts 07 Kube Controller Manager (`docs/02-Core-Concepts/07-Kube-Controller-Manager.md`)

Kube Controller Manager

In this section, we will take a look at kube-controller-manager.

Kube Controller Manager manages various controllers in kubernetes.

  • In kubernetes terms, a controller is a process that continuously monitors the state of the components within the system and works towards bringing the whole system to the desired functioning state.

Node Controller

  • Responsible for monitoring the state of the Nodes and taking necessary actions to keep the application running.

Replication Controller

  • It is responsible for monitoring the status of replicasets and ensuring that the desired number of pods are available at all time within the set.

Other Controllers

  • There are many more such controllers available within kubernetes

Installing Kube-Controller-Manager

  • When you install kube-controller-manager the different controllers will get installed as well.
  • Download the kube-controller-manager binary from the kubernetes release page. For example: You can download kube-controller-manager v1.13.0 here kube-controller-manager
    text
    $ wget https://storage.googleapis.com/kubernetes-release/release/v1.13.0/bin/linux/amd64/kube-controller-manager
  • By default all controllers are enabled, but you can choose to enable specific one from kube-controller-manager.service
    text
    $ cat /etc/systemd/system/kube-controller-manager.service

View kube-controller-manager - kubeadm

  • kubeadm deploys the kube-controller-manager as a pod in kube-system namespace
    text
    $ kubectl get pods -n kube-system

View kube-controller-manager options - kubeadm

  • You can see the options within the pod located at /etc/kubernetes/manifests/kube-controller-manager.yaml
    text
    $ cat /etc/kubernetes/manifests/kube-controller-manager.yaml

View kube-controller-manager options - Manual

  • In a non-kubeadm setup, you can inspect the options by viewing the kube-controller-manager.service

    text
    $ cat /etc/systemd/system/kube-controller-manager.service

  • You can also see the running process and effective options by listing the process on master node and searching for kube-controller-manager.

    text
    $ ps -aux | grep kube-controller-manager

K8s Reference Docs:

02 Core Concepts 08 Kube Scheduler (`docs/02-Core-Concepts/08-Kube-Scheduler.md`)

Kube Scheduler

In this section, we will take a look at kube-scheduler.

kube-scheduler is responsible for scheduling pods on nodes.

  • The kube-scheduler is only responsible for deciding which pod goes on which node. It doesn't actually place the pod on the nodes, that's the job of the kubelet.

Why do you need a Scheduler?

Install kube-scheduler - Manual

  • Download the kubescheduler binary from the kubernetes release pages kube-scheduler. For example: To download kube-scheduler v1.13.0, Run the below command.
    text
    $ wget https://storage.googleapis.com/kubernetes-release/release/v1.13.0/bin/linux/amd64/kube-scheduler
  • Extract it
  • Run it as a service

View kube-scheduler options - kubeadm

02 Core Concepts 09 Kubelet (`docs/02-Core-Concepts/09-Kubelet.md`)

Kubelet

In this section we will take a look at kubelet.

Kubelet is the sole point of contact for the kubernetes cluster

  • The kubelet will create the pods on the nodes, the scheduler only decides which pods goes where.

Install kubelet

  • Kubeadm does not deploy kubelet by default. You must manually download and install it.
  • Download the kubelet binary from the kubernetes release pages kubelet. For example: To download kubelet v1.13.0, Run the below command.
    text
    $ wget https://storage.googleapis.com/kubernetes-release/release/v1.13.0/bin/linux/amd64/kubelet
  • Extract it
  • Run it as a service

View kubelet options

  • You can also see the running process and affective options by listing the process on worker node and searching for kubelet.
    text
    $ ps -aux |grep kubelet

K8s Reference Docs:

02 Core Concepts 10 Kube Proxy (`docs/02-Core-Concepts/10-Kube-Proxy.md`)

Kube Proxy

In this section, we will take a look at kube-proxy.

Within Kubernetes Cluster, every pod can reach every other pod. This is accomplished by deploying a pod networking cluster to the cluster.

  • Kube-Proxy is a process that runs on each node in the kubernetes cluster.

Install kube-proxy - Manual

  • Download the kube-proxy binary from the kubernetes release pages kube-proxy. For example: To download kube-proxy v1.13.0, Run the below command.
    text
    $ wget https://storage.googleapis.com/kubernetes-release/release/v1.13.0/bin/linux/amd64/kube-proxy
  • Extract it
  • Run it as a service

View kube-proxy options - kubeadm

  • If you set it up with kubeadm tool, kubeadm tool will deploy the kube-proxy as pod in kube-system namespace. In fact it is deployed as a daemonset on master node.
    text
    $ kubectl get pods -n kube-system

K8s Reference Docs:

02 Core Concepts 11 Pods (`docs/02-Core-Concepts/11-Pods.md`)

Pods

In this section, we will take a look at PODS.

  • POD introduction
  • How to deploy pod?

Kubernetes doesn't deploy containers directly on the worker node.

Here is a single node kubernetes cluster with single instance of your application running in a single docker container encapsulated in the pod.

Pod will have a one-to-one relationship with containers running your application.

Multi-Container PODs

  • A single pod can have multiple containers except for the fact that they are usually not multiple containers of the same kind.

How to deploy pods?

Lets now take a look to create a nginx pod using kubectl.

  • To deploy a docker container by creating a POD.

    text
    $ kubectl run nginx --image nginx

  • To get the list of pods

    text
    $ kubectl get pods

K8s Reference Docs:

02 Core Concepts 12 Practice Test Introduction (`docs/02-Core-Concepts/12-Practice-Test-Introduction.md`)

Practice Test Introduction

In this section, we will take a look at practice test demo.

02 Core Concepts 13 Practice Test PODs (`docs/02-Core-Concepts/13-Practice-Test-PODs.md`)

Practice Test - PODs

Here are the solutions to the practice test

  1. How many pods exist on the system?

    bash
    kubectl get pods

    Count the number of pods (if any)

  2. Create a new pod with the nginx image.

    bash
    kubectl run nginx --image=nginx

  3. How many pods are created now?

    bash
    kubectl get pods

    Count the number of pods (if any)

    To get the system to tell you you can also do this

    bash
    kubectl get pods --no-headers | wc -l
    • --no-headers should be obvious - output only the details.
    • wc is the word count program. -l tells it to count lines instead, and it will count the lines emitted by kubectl

  4. What is the image used to create the new pods?

    kubectl describe outputs lots of information. The following will describe all pods whose name starts with newpods, and then we filter with grep to get what we are looking for.

    bash
    kubectl describe pod newpods | grep image

    We see that all three are pulling the same image.

  5. Which nodes are these pods placed on?

    bash
    kubectl get pods -o wide

    Note the node column for each of the 3 newpods pods

  6. How many containers are part of the pod webapp?

    bash
    kubectl describe pod webapp

    Look under the Containers section. Note there is nginx and agentx

  7. What images are used in the new webapp pod?

    Examine the output from Q6. For each of the identified containers, look at Image:

  8. What is the state of the container agentx in the pod webapp?

    bash
    kubectl describe pod webapp

    Examine the State: field for the agentx container.

  9. Why do you think the container agentx in pod webapp is in error?

    Examine the output from Q8 and look in the Events: section at the end. Look at the event that says failed to pull and unpack image ...

  10. What does the READY column in the output of the kubectl get pods command indicate?

    bash
    kubectl get pods

    Look at the webapp pod which we know has two containers and one is in error. You can deduce which of the answers is correct from this.

  11. Delete the webapp Pod.

    bash
    kubectl delete pod webapp

    To delete the pod without any delay and confirmation, we can add --force flag. Note that in a real production system, forcing is a last resort as it can leave behind containers that haven't been cleaned up properly. Some may have important cleanup jobs to run when they are requested to terminate, which wouldn't get run.

    You can however use --force in the exam as it will gain you time. No exam pods rely on any of the above.

  12. Create a new pod with the name redis and with the image redis123.Use a pod-definition YAML file.

    To create the pod definition YAML file:

    --dry-run=client tells kubectl to test without actually doing anything. -o yaml says "Output what you would send to API server to the console", which we then redirect into the named file.

    bash
    kubectl run redis --image=redis123 --dry-run=client -o yaml > redis.yaml

    And now use the YAML you created to deploy the pod.

    bash
    kubectl create -f redis.yaml

  13. Now change the image on this pod to redis.Once done, the pod should be in a running state.

    There are three ways this can be done!

    1. Method 1
      Edit your manifest file created in last question

      bash
      vi redis.yaml

      Fix the image name in the redis.yaml to redis, save and exit.

      Apply the edited yaml

      bash
      kubectl apply -f redis.yaml

    2. Method 2
      Edit the running pod directly (note not all fields can be edited this way)

      text
      kubectl edit pod redis

      This will bring the pod YAML up in vi. Edit it as per method 1. When you eixt vi the change will be immediately applied. If you make a mistake, you will be dropped back into vi

    3. Method 3
      Patch the image directly. For this you need to know the name of the container in the pod, as we assign the new image to that name, as in container_image_name=new_image

      bash
      kubectl set image pod/redis redis=redis

02 Core Concepts 14 ReplicaSets (`docs/02-Core-Concepts/14-ReplicaSets.md`)

ReplicaSets

In this section, we will take a look at the below

  • Replication Controller
  • ReplicaSet

Controllers are brain behind kubernetes

What is a Replica and Why do we need a replication controller?

Difference between ReplicaSet and Replication Controller

  • Replication Controller is the older technology that is being replaced by a ReplicaSet.
  • ReplicaSet is the new way to setup replication.

Creating a Replication Controller

Replication Controller Definition File

text
apiVersion: v1
    kind: ReplicationController
    metadata:
      name: myapp-rc
      labels:
        app: myapp
        type: front-end
    spec:
     template:
        metadata:
          name: myapp-pod
          labels:
            app: myapp
            type: front-end
        spec:
         containers:
         - name: nginx-container
           image: nginx
     replicas: 3
  • To Create the replication controller
    text
    $ kubectl create -f rc-definition.yaml
  • To list all the replication controllers
    text
    $ kubectl get replicationcontroller
  • To list pods that are launch by the replication controller
    text
    $ kubectl get pods

Creating a ReplicaSet

ReplicaSet Definition File

text
apiVersion: apps/v1
    kind: ReplicaSet
    metadata:
      name: myapp-replicaset
      labels:
        app: myapp
        type: front-end
    spec:
     template:
        metadata:
          name: myapp-pod
          labels:
            app: myapp
            type: front-end
        spec:
         containers:
         - name: nginx-container
           image: nginx
     replicas: 3
     selector:
       matchLabels:
        type: front-end

ReplicaSet requires a selector definition when compare to Replication Controller.

  • To Create the replicaset
    text
    $ kubectl create -f replicaset-definition.yaml
  • To list all the replicaset
    text
    $ kubectl get replicaset
  • To list pods that are launch by the replicaset
    text
    $ kubectl get pods

Labels and Selectors

What is the deal with Labels and Selectors? Why do we label pods and objects in kubernetes?

How to scale replicaset

  • There are multiple ways to scale replicaset
    • First way is to update the number of replicas in the replicaset-definition.yaml definition file. E.g replicas: 6 and then run
text
apiVersion: apps/v1
   kind: ReplicaSet
   metadata:
     name: myapp-replicaset
     labels:
       app: myapp
       type: front-end
   spec:
    template:
       metadata:
         name: myapp-pod
         labels:
           app: myapp
           type: front-end
       spec:
        containers:
        - name: nginx-container
          image: nginx
    replicas: 6
    selector:
      matchLabels:
       type: front-end
text
$ kubectl apply -f replicaset-definition.yaml
  • Second way is to use kubectl scale command.
text
$ kubectl scale --replicas=6 -f replicaset-definition.yaml
  • Third way is to use kubectl scale command with type and name
text
$ kubectl scale --replicas=6 replicaset myapp-replicaset

K8s Reference Docs:

02 Core Concepts 15 Practice Tests ReplicaSet (`docs/02-Core-Concepts/15-Practice-Tests-ReplicaSet.md`)

Practice Test - ReplicaSets

Solutions for the replicaset practice tests

  1. How many pods exist on the system?

    bash
    kubectl get pods

    Count the number of pods (if any)

  2. How many ReplicaSets exist on the system?

    bash
    kubectl get replicasets

    Count the number of ReplicaSets (if any)

  3. How about now? How many ReplicaSets do you see?

    bash
    kubectl get replicasets

    Count the number of ReplicaSets (if any)

  4. How many PODs are DESIRED in the new-replica-set?

    From the output of Q3, look in DESIRED column

  5. What is the image used to create the pods in the new-replica-set?

    text
    kubectl describe replicaset

    ...and look under the containers section --- or --

    text
    kubectl get rs -o wide

    ...and look in the IMAGES column. Kubernetes accepts rs as shorthand for replicaset.

  6. How many PODs are READY in the new-replica-set?

    text
    kubectl get rs

    Look in the READY column.

  7. Why do you think the PODs are not ready?

    text
    kubectl describe pods

    Look in the Events section at the end.

  8. Delete any one of the 4 PODs.

    text
    kubectl get pods

    Choose any of the four.

    text
    kubectl delete pod new-replica-set-XXXX

  9. How many PODs exist now?

    text
    kubectl get pods

  10. Why are there still 4 PODs, even after you deleted one?

    ReplicaSets ensures that desired number of PODs always run

  11. Create a ReplicaSet using the replicaset-definition-1.yaml file located at /root/.There is an issue with the file, so try to fix it.

    text
    kubectl create -f replicaset-definition-1.yaml

    Note the error message.

    Get the apiVersion for replicaset

    text
    $ kubectl explain replicaset | grep VERSION

    Update the replicaset definition file in vi with correct version and then retry creation.

    text
    $ kubectl create -f replicaset-definition-1.yaml

  12. Fix the issue in the replicaset-definition-2.yaml file and create a ReplicaSet using it.

    text
    kubectl create -f replicaset-definition-1.yaml

    Note the error message.

    Selector matchLabels should match with POD labels - Update replicaset-definition-2.yaml

    The values for labels on lines 9 and 13 should match.

    text
    $ kubectl create -f replicaset-definition-2.yaml

  13. Delete the two newly created ReplicaSets - replicaset-1 and replicaset-2

    text
    kubectl delete replicaset replicaset-1
kubectl delete rs replicaset-2

    --- OR ---

    text
    kubectl delete replicaset replicaset-1 replicaset-2

  14. Fix the original replica set new-replica-set to use the correct busybox image.

    text
    kubectl edit replicaset new-replica-set

    Edit the image to be busybox, save and exit.

  15. Fix the original replica set new-replica-set to use the correct busybox image.

    text
    kubectl edit replicaset new-replica-set

    Fix the image, save and exit.

    You will note if you do kubectl get pods, that they are still broken. ReplicaSets are not very smart and do not redeploy pods when the container specification has been edited.

    We must either delete and recreate the replicaset by exporting its YAML...

    text
    kubectl get rs new-replica-set -o yaml > rs.yaml
kubectl delete rs new-replica-set
kunectl create -f rs.yaml

    -- OR --

    Delete each broken pod. The ReplicaSet will deploy a new one in its place which should be working.

    -- OR --

    Scale it to zero, then back to 4

    text
    kubectl scale rs new-replica-set --replicas 0
kubectl scale rs new-replica-set --replicas 4

  16. Scale the ReplicaSet to 5 PODs.

    text
    kubectl scale rs new-replica-set --replicas 5

02 Core Concepts 16 Deployments (`docs/02-Core-Concepts/16-Deployments.md`)

Deployments

In this section, we will take a look at kubernetes deployments

Deployment is a kubernetes object.

How do we create deployment?

text
apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: myapp-deployment
      labels:
        app: myapp
        type: front-end
    spec:
     template:
        metadata:
          name: myapp-pod
          labels:
            app: myapp
            type: front-end
        spec:
         containers:
         - name: nginx-container
           image: nginx
     replicas: 3
     selector:
       matchLabels:
        type: front-end

  • Once the file is ready, create the deployment using deployment definition file

    text
    $ kubectl create -f deployment-definition.yaml

  • To see the created deployment

    text
    $ kubectl get deployment

  • The deployment automatically creates a ReplicaSet. To see the replicasets

    text
    $ kubectl get replicaset

  • The replicasets ultimately creates PODs. To see the PODs

    text
    $ kubectl get pods

  • To see the all objects at once

    text
    $ kubectl get all

K8s Reference Docs:

02 Core Concepts 17 Practice Tests Deployments (`docs/02-Core-Concepts/17-Practice-Tests-Deployments.md`)

Practice Test - Deployments

Solutions to the deployments practice test
1.
How many pods exist on the system?

bash
kubectl get pods

Count the number of pods (if any)

  1. How many ReplicaSets exist on the system?

    bash
    kubectl get replicasets

    Count the number of ReplicaSets (if any)

  2. How many Deployments exist on the system?

    bash
    kubectl get deployments

    Count the number of Deployments (if any)

  3. How many Deployments exist on the system now?

    bash
    kubectl get deployments

    Count the number of Deployments (if any)

  4. How many ReplicaSets exist on the system now?

    bash
    kubectl get replicasets

    Count the number of ReplicaSets (if any)

  5. How many Pods exist on the system?

    bash
    kubectl get pods

    Count the number of pods (if any)

  6. Out of all the existing PODs, how many are ready?

    From the output of the previous command, check the READY column

  7. What is the image used to create the pods in the new deployment?

    text
    kubectl describe deployment

    Look under the containers section.

    Another way - run the following and check the IMAGES column

    text
    kubectl get deployment -o wide

  8. Why do you think the deployment is not ready?

    text
    kubectl describe pods

    Look under the events section.

  9. Create a new Deployment using the deployment-definition-1.yaml file located at /root/.There is an issue with the file, so try to fix it.

    text
    kubectl create -f deployment-definition-1.yaml

    Note the error

    Edit the file with vi...

    The value for kind is incorrect. It should be Deployment with a capital D. Update the deployment definition and create the deployment with the above command.

  10. Create a new Deployment with the below attributes using your own deployment definition file.

    Create a deployment definition file in vi, e.g. my-deployment.yaml with the following

    yaml
    apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: httpd-frontend
  name: httpd-frontend
spec:
  replicas: 3
  selector:
    matchLabels:
      app: httpd-frontend
  template:
    metadata:
      labels:
        app: httpd-frontend
    spec:
      containers:
      - image: httpd:2.4-alpine
        name: httpd
    text
    kubectl create -f my-deployment.yaml

    Or we could create it imperatively...

    text
    kubectl create deployment httpd-frontend --image=httpd:2.4-alpine --replicas=3

02 Core Concepts 18 Namespaces (`docs/02-Core-Concepts/18-Namespaces.md`)

Namespaces

In this section, we will take a look at Namespaces

So far in this course we have created Objects such as PODs, Deployments and Services in our cluster. Whatever we have been doing we have been doing in a NAMESPACE.

  • This namespace is the default namespace in kubernetes. It is automatically created when kubernetes is setup initially.

  • You can create your own namespaces as well.

  • To list the pods in default namespace

    text
    $ kubectl get pods

  • To list the pods in another namespace. Use kubectl get pods command along with the --namespace flag or argument.

    text
    $ kubectl get pods --namespace=kube-system

  • Here we have a pod definition file, when we create a pod with pod-definition file, the pod is created in the default namespace.

text
apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
     app: myapp
     type: front-end
spec:
  containers:
  - name: nginx-container
    image: nginx
text
$ kubectl create -f pod-definition.yaml

  • To create the pod with the pod-definition file in another namespace, use the --namespace option.

    text
    $ kubectl create -f pod-definition.yaml --namespace=dev

  • If you want to make sure that this pod gets you created in the dev env all the time, even if you don't specify in the command line, you can move the --namespace definition into the pod-definition file.

text
apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  namespace: dev
  labels:
     app: myapp
     type: front-end
spec:
  containers:
  - name: nginx-container
    image: nginx
  • To create a new namespace, create a namespace definition as shown below and then run kubectl create
text
apiVersion: v1
kind: Namespace
metadata:
  name: dev
text
$ kubectl create -f namespace-dev.yaml

Another way to create a namespace

text
$ kubectl create namespace dev

  • By default, we will be in a default namespace. To switch to a particular namespace permenently run the below command.

    text
    $ kubectl config set-context $(kubectl config current-context) --namespace=dev

  • To view pods in all namespaces

    text
    $ kubectl get pods --all-namespaces

  • To limit resources in a namespace, create a resource quota. To create one start with ResourceQuota definition file.

text
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: dev
spec:
  hard:
    pods: "10"
    requests.cpu: "4"
    requests.memory: 5Gi
    limits.cpu: "10"
    limits.memory: 10Gi
text
$ kubectl create -f compute-quota.yaml

K8s Reference Docs:

02 Core Concepts 19 Practice Test Namespaces (`docs/02-Core-Concepts/19-Practice-Test-Namespaces.md`)

Practice Test - Namespaces

Solutions to practice test for namespaces

  1. How many Namespaces exist on the system?

    text
    kubectl get namespace

    Count the number of namespaces (if any)

  2. How many pods exist in the research namespace?

    text
    kubectl get pods --namespace=research

    Count the number of pods (if any)

  3. Create a POD in the finance namespace.

    text
    kubectl run redis --image=redis --namespace=finance

  4. Which namespace has the blue pod in it?

    text
    kubectl get pods --all-namespaces

    Examine the output.

    Or use grep to filter the results, knowing that NAMESPACE is the first result column

    text
    kubectl get pods --all-namespaces | grep blue

  5. Access the Blue web application using the link above your terminal!!

    Press the blue-application-ui button at the top of the terminal. Try the following:

    text
    Host Name: db-service
Host Port: 6379

  6. What DNS name should the Blue application use to access the database db-service in its own namespace - marketing?

    db-service

    To access services in the same namespace, only the host name part of the fully qualified domain name (FQDN) is required.

  7. What DNS name should the Blue application use to access the database db-service in the dev namespace?

    Either FQDN

    db-service.dev.svc.cluster.local

    Or, it is sufficient just to append the namespace name

    db-service.dev

02 Core Concepts 20 Services (`docs/02-Core-Concepts/20-Services.md`)

Kubernetes Services

In this section we will take a look at services in kubernetes

Services

  • Kubernetes Services enables communication between various components within and outside of the application.

Let's look at some other aspects of networking

External Communication

  • How do we as an external user access the web page?

    • From the node (Able to reach the application as expected)

    • From outside world (This should be our expectation, without something in the middle it will not reach the application)

Service Types

There are 3 types of service types in kubernetes

  1. NodePort

    • Where the service makes an internal port accessible on a port on the NODE.

      text
      apiVersion: v1
kind: Service
metadata:
 name: myapp-service
spec:
 types: NodePort
 ports:
 - targetPort: 80
   port: 80
   nodePort: 30008

      To connect the service to the pod

      text
      apiVersion: v1
kind: Service
metadata:
 name: myapp-service
spec:
 type: NodePort
 ports:
 - targetPort: 80
   port: 80
   nodePort: 30008
 selector:
   app: myapp
   type: front-end

      To create the service

      text
      $ kubectl create -f service-definition.yaml

      To list the services

      text
      $ kubectl get services

      To access the application from CLI instead of web browser

      text
      $ curl http://192.168.1.2:30008

      A service with multiple pods

      When Pods are distributed across multiple nodes

  2. ClusterIP

    • In this case the service creates a Virtual IP inside the cluster to enable communication between different services such as a set of frontend servers to a set of backend servers.

  3. LoadBalancer

    • Where the service provisions a loadbalancer for our application in supported cloud providers.

K8s Reference Docs:

02 Core Concepts 21 Services ClusterIP (`docs/02-Core-Concepts/21-Services-ClusterIP.md`)

Kubernetes Services - ClusterIP

In this section we will take a look at services - ClusterIP in kubernetes

ClusterIP

  • In this case the service creates a Virtual IP inside the cluster to enable communication between different services such as a set of frontend servers to a set of backend servers.

What is a right way to establish connectivity between these services or tiers

  • A kubernetes service can help us group the pods together and provide a single interface to access the pod in a group.

To create a service of type ClusterIP

text
apiVersion: v1
kind: Service
metadata:
 name: back-end
spec:
 types: ClusterIP
 ports:
 - targetPort: 80
   port: 80
 selector:
   app: myapp
   type: back-end
text
$ kubectl create -f service-definition.yaml

To list the services

text
$ kubectl get services

K8s Reference Docs:

02 Core Concepts 22 Practice Test Services (`docs/02-Core-Concepts/22-Practice-Test-Services.md`)

Practice Test - Services

Solutions to Practice test - Services

  1. How many Services exist on the system?

    text
    kubectl get services

    Count the number of services (if any)

  2. Information only

  3. What is the type of the default kubernetes service?

    From the output of Q1, examine the TYPE column.

  4. What is the targetPort configured on the kubernetes service?

    text
    $ kubectl describe service | grep TargetPort

  5. How many labels are configured on the kubernetes service?

    text
    kubectl describe service

    ...and look for labels.

    --- OR ---

    text
    kubectl get service --show-labels

  6. How many Endpoints are attached on the kubernetes service?

    text
    kubectl describe service

    ...and look for endpoints

  7. How many Deployments exist on the system now?

    text
    kubectl get deployment

    Count the deployments (if any)

  8. What is the image used to create the pods in the deployment?

    text
    kubectl describe deployment

    Look in the containers section.

    --- OR ---

    text
    kubectl get deployment -o wide

    Look in the IMAGES column

  9. Are you able to access the Web App UI?

    Try to access the Web Application UI using the tab simple-webapp-ui above the terminal.

  10. Create a new service to access the web application using the service-definition-1.yaml file.

    text
    vi service-definition-1.yaml

    Fill in the values as directed, save and exit.

    text
    kubectl create -f service-definition-1.yaml

  11. Test newly deployed service.

02 Core Concepts 23 Imperative Commands With Kubectl (`docs/02-Core-Concepts/23-Imperative-Commands-with-kubectl.md`)

Certification Tips - Imperative Commands with kubectl

02 Core Concepts 24 Practice Test Imperative Commands (`docs/02-Core-Concepts/24-Practice-Test-Imperative-Commands.md`)

Practice Test - Imperative Commands

Solutions for Practice Test - Imperative Commands

  1. Information

  2. Deploy a pod named nginx-pod using the nginx:alpine image.

    text
    kubectl run nginx-pod --image=nginx:alpine

  3. Deploy a redis pod using the redis:alpine image with the labels set to tier=db.

    text
    kubectl run redis --image=redis:alpine -l tier=db

  4. Create a service redis-service to expose the redis application within the cluster on port 6379.

    text
    kubectl expose pod redis --port=6379 --name redis-service

  5. Create a deployment named webapp using the image kodekloud/webapp-color with 3 replicas.

    text
    kubectl create deployment webapp --image=kodekloud/webapp-color --replicas=3

  6. Create a new pod called custom-nginx using the nginx image and expose it on container port 8080.

    text
    kubectl run custom-nginx --image=nginx --port=8080

  7. Create a new namespace called dev-ns.

    text
    kubectl create ns dev-ns

  8. Create a new deployment called redis-deploy in the dev-ns namespace with the redis image. It should have 2 replicas.

    text
    kubectl create deployment redis-deploy -n dev-ns --image redis --replicas 2

  9. Create a pod called httpd using the image httpd:alpine in the default namespace.Next, create a service of type ClusterIP by the same name (httpd).The target port for the service should be 80.

    text
    kubectl run httpd --image httpd:alpine --expose --port 80

02 Core Concepts 25 Attachments (`docs/02-Core-Concepts/25-Attachments.md`)

KodeKloud - CKA Course Documents

03 Scheduling 01 Scheduling Section Introduction (`docs/03-Scheduling/01-Scheduling-Section-Introduction.md`)

Scheduling Section Introduction

03 Scheduling 02 Manual Scheduling (`docs/03-Scheduling/02-Manual-Scheduling.md`)

Manual Scheduling

In this section, we will take a look at Manually Scheduling a POD on a node.

How Scheduling Works

  • What do you do when you do not have a scheduler in your cluster?
    • Every POD has a field called NodeName that by default is not set. You don't typically specify this field when you create the manifest file, kubernetes adds it automatically.
    • Once identified it schedules the POD on the node by setting the nodeName property to the name of the node by creating a binding object.
      text
      apiVersion: v1
kind: Pod
metadata:
 name: nginx
 labels:
  name: nginx
spec:
 containers:
 - name: nginx
   image: nginx
   ports:
   - containerPort: 8080
 nodeName: node02

No Scheduler

  • You can manually assign pods to node itself. Well without a scheduler, to schedule pod is to set nodeName property in your pod definition file while creating a pod.

  • Another way

    text
    apiVersion: v1
kind: Binding
metadata:
  name: nginx
target:
  apiVersion: v1
  kind: Node
  name: node02
    text
    apiVersion: v1
kind: Pod
metadata:
 name: nginx
 labels:
  name: nginx
spec:
 containers:
 - name: nginx
   image: nginx
   ports:
   - containerPort: 8080

K8s Reference Docs:

03 Scheduling 03 Practice Test Manual Scheduling (`docs/03-Scheduling/03-Practice-Test-Manual-Scheduling.md`)

Practice Test - Manual Scheduling

Solutions to Practice Test - Manual Scheduling

  1. A pod definition file nginx.yaml is given. Create a pod using the file.

    text
    kubectl create -f nginx.yaml

  2. What is the status of the created POD?

    text
    kubectl get pods

    Examine the STATUS column

  3. Why is the POD in a pending state?Inspect the environment for various kubernetes control plane components.

    text
    kubectl get pods --namespace kube-system

    There is a key pod missing here!

  4. Manually schedule the pod on node01.

    We will have to delete and recereate the pod, as the only property that may be edited on a running container is image

    text
    vi nginx.yaml

    Make the following edit

    yaml
    ---
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  nodeName: node01    # add this line
  containers:
  -  image: nginx
     name: nginx
    text
    kubectl delete -f nginx.yaml
kubectl create -f nginx.yaml

  5. Now schedule the same pod on the controlplane node.

    Repeat the steps as per the previous question. Edit nodeName to be controlplane

03 Scheduling 04 Labels And Selectors (`docs/03-Scheduling/04-Labels-and-Selectors.md`)

Labels and Selectors

In this section, we will take a look at Labels and Selectors

Labels and Selectors are standard methods to group things together.

Labels are properties attached to each item.

Selectors help you to filter these items

How are labels and selectors are used in kubernetes?

  • We have created different types of objects in kubernetes such as PODs, ReplicaSets, Deployments etc.

How do you specify labels?

text
apiVersion: v1
 kind: Pod
 metadata:
  name: simple-webapp
  labels:
    app: App1
    function: Front-end
 spec:
  containers:
  - name: simple-webapp
    image: simple-webapp
    ports:
    - containerPort: 8080

Once the pod is created, to select the pod with labels run the below command

text
$ kubectl get pods --selector app=App1

Kubernetes uses labels to connect different objects together

text
apiVersion: apps/v1
 kind: ReplicaSet
 metadata:
   name: simple-webapp
   labels:
     app: App1
     function: Front-end
 spec:
  replicas: 3
  selector:
    matchLabels:
     app: App1
  template:
    metadata:
      labels:
        app: App1
        function: Front-end
    spec:
      containers:
      - name: simple-webapp
        image: simple-webapp

For services

text
```
  apiVersion: v1
  kind: Service
  metadata:
   name: my-service
  spec:
   selector:
     app: App1
   ports:
   - protocol: TCP
     port: 80
     targetPort: 9376 
   ```

Annotations

  • While labels and selectors are used to group objects, annotations are used to record other details for informative purpose.
    text
    apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: simple-webapp
  labels:
    app: App1
    function: Front-end
  annotations:
     buildversion: 1.34
spec:
 replicas: 3
 selector:
   matchLabels:
    app: App1
template:
  metadata:
    labels:
      app: App1
      function: Front-end
  spec:
    containers:
    - name: simple-webapp
      image: simple-webapp

K8s Reference Docs:

03 Scheduling 05 Practice Test Labels And Selectors (`docs/03-Scheduling/05-Practice-Test-Labels-and-Selectors.md`)

Practice Test - Labels and Selectors

Solutions to Practice Test - Labels and Selectors
1.
We have deployed a number of PODs. They are labelled with tier, env and bu. How many PODs exist in the dev environment (env)?

text
Here we are filtering pods by the value olf a label

```
kubectl get pods --selector env=dev
```

Count the pods (if any)

  1. How many PODs are in the finance business unit (bu)?

    Similarly ...

    text
    kubectl get pods --selector bu=finance

    Count the pods (if any)

  2. How many objects are in the prod environment including PODs, ReplicaSets and any other objects?

    text
    kubectl get all --selector env=prod

    Count everything (if anything)

  3. Identify the POD which is part of the prod environment, the finance BU and of frontend tier?

    We can combine label expressions with comma. Only items with all the given label/value pairs will be returned, i.e. it is an and condition.

    text
    kubectl get all --selector env=prod,bu=finance,tier=frontend

  4. A ReplicaSet definition file is given replicaset-definition-1.yaml. Try to create the replicaset. There is an issue with the file. Try to fix it.

    text
    kubectl create -f replicaset-definition-1.yaml

    Note the error message.

    Selector matchLabels should match with POD labels - Update replicaset-definition-2.yaml

    The values for labels on lines 9 and 13 should match.

    text
    $ kubectl create -f replicaset-definition-2.yaml

03 Scheduling 06 Taints And Tolerations (`docs/03-Scheduling/06-Taints-and-Tolerations.md`)

Taints and Tolerations

In this section, we will take a look at taints and tolerations.

  • Pod to node relationship and how you can restrict what pods are placed on what nodes.

Taints and Tolerations are used to set restrictions on what pods can be scheduled on a node.

  • Only pods which are tolerant to the particular taint on a node will get scheduled on that node.

Taints

  • Use kubectl taint nodes command to taint a node.

    Syntax

    text
    $ kubectl taint nodes <node-name> key=value:taint-effect

    Example

    text
    $ kubectl taint nodes node1 app=blue:NoSchedule

  • The taint effect defines what would happen to the pods if they do not tolerate the taint.

  • There are 3 taint effects

    • NoSchedule
    • PreferNoSchedule
    • NoExecute

Tolerations

  • Tolerations are added to pods by adding a tolerations section in pod definition.
    text
    apiVersion: v1
kind: Pod
metadata:
 name: myapp-pod
spec:
 containers:
 - name: nginx-container
   image: nginx
 tolerations:
 - key: "app"
   operator: "Equal"
   value: "blue"
   effect: "NoSchedule"

Taints and Tolerations do not tell the pod to go to a particular node. Instead, they tell the node to only accept pods with certain tolerations.

  • To see this taint, run the below command
    text
    $ kubectl describe node kubemaster |grep Taint

K8s Reference Docs

03 Scheduling 07 Practice Test Taints And Tolerations (`docs/03-Scheduling/07-Practice-Test-Taints-and-Tolerations.md`)

Practice Test - Taints and Tolerations

Solutions to the Practice Test - Taints and Tolerations

  1. How many nodes exist on the system?

    text
    $ kubectl get nodes

    Count the nodes

  2. Do any taints exist on node01 node?

    text
    $ kubectl describe node node01

    Find the Taints property in the output.

  3. Create a taint on node01 with key of spray, value of mortein and effect of NoSchedule

    text
    kubectl taint nodes node01 spray=mortein:NoSchedule

  4. Create a new pod with the nginx image and pod name as mosquito.

    text
    kubectl run mosquito --image nginx

  5. What is the state of the POD?

    text
    kubectl get pods

    Check the STATUS column

  6. Why do you think the pod is in a pending state?

    Mosqitoes don't like mortein!

    So the answer is that the pod cannot tolerate the taint on the node.

  7. Create another pod named bee with the nginx image, which has a toleration set to the taint mortein.

    Allegedly bees are immune to mortein!

    1. Create a YAML skeleton for the pod imperatively

      text
      kubectl run bee --image nginx --dry-run=client -o yaml > bee.yaml

    2. Edit the file to add the toleration

      text
      vi bee.yaml

    3. Add the toleration. This goes at the same indentation level as containers as it is a POD setting.

      yaml
      tolerations:
  - key: spray
    value: mortein
    effect: NoSchedule
    operator: Equal

    4. Save and exit, then create pod

      text
      kubectl create -f bee.yaml

  8. Information only.

  9. Do you see any taints on controlplane node?

    text
    kubectl describe node controlplane

    Examine the Taints property.

  10. Remove the taint on controlplane, which currently has the taint effect of NoSchedule.

    text
    kubectl taint nodes controlplane node-role.kubernetes.io/control-plane:NoSchedule-

  11. What is the state of the pod mosquito now?

    text
    $ kubectl get pods

  12. Which node is the POD mosquito on now?

    text
    $ kubectl get pods -o wide

    This also explains why the mosquito pod colud schedule anywhere. It also could not tolerate controlplane taints, which we have now removed.

03 Scheduling 08 Node Selectors (`docs/03-Scheduling/08-Node-Selectors.md`)

Node Selectors

In this section, we will take a look at Node Selectors in Kubernetes

We add new property called Node Selector to the spec section and specify the label.

  • The scheduler uses these labels to match and identify the right node to place the pods on.

    text
    apiVersion: v1
kind: Pod
metadata:
 name: myapp-pod
spec:
 containers:
 - name: data-processor
   image: data-processor
 nodeSelector:
  size: Large

  • To label nodes

    Syntax

    text
    $ kubectl label nodes <node-name> <label-key>=<label-value>

    Example

    text
    $ kubectl label nodes node-1 size=Large

  • To create a pod definition

    text
    apiVersion: v1
kind: Pod
metadata:
 name: myapp-pod
spec:
 containers:
 - name: data-processor
   image: data-processor
 nodeSelector:
  size: Large
    text
    $ kubectl create -f pod-definition.yml

Node Selector - Limitations

  • We used a single label and selector to achieve our goal here. But what if our requirement is much more complex.

  • For this we have Node Affinity and Anti Affinity

K8s Reference Docs

03 Scheduling 09 Node Affinity (`docs/03-Scheduling/09-Node-Affinity.md`)

Node Affinity

In this section, we will talk about "Node Affinity" feature in kubernetes.

The primary feature of Node Affinity is to ensure that the pods are hosted on particular nodes.

  • With Node Selectors we cannot provide the advance expressions.

    text
    apiVersion: v1
kind: Pod
metadata:
 name: myapp-pod
spec:
 containers:
 - name: data-processor
   image: data-processor
 nodeSelector:
  size: Large
    text
    apiVersion: v1
kind: Pod
metadata:
 name: myapp-pod
spec:
 containers:
 - name: data-processor
   image: data-processor
 affinity:
   nodeAffinity:
     requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: size
            operator: In
            values: 
            - Large
            - Medium
    text
    apiVersion: v1
kind: Pod
metadata:
 name: myapp-pod
spec:
 containers:
 - name: data-processor
   image: data-processor
 affinity:
   nodeAffinity:
     requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: size
            operator: NotIn
            values: 
            - Small
    text
    apiVersion: v1
kind: Pod
metadata:
 name: myapp-pod
spec:
 containers:
 - name: data-processor
   image: data-processor
 affinity:
   nodeAffinity:
     requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: size
            operator: Exists

Node Affinity Types

  • Available
    • requiredDuringSchedulingIgnoredDuringExecution
    • preferredDuringSchedulingIgnoredDuringExecution
  • Planned
    • requiredDuringSchedulingRequiredDuringExecution
    • preferredDuringSchedulingRequiredDuringExecution

Node Affinity Types States

K8s Reference Docs

03 Scheduling 10 Practice Test Node Affinity (`docs/03-Scheduling/10-Practice-Test-Node-Affinity.md`)

Practice Test - Node Affinity

Solutions to practice test - node affinity

  1. How many Labels exist on node node01?

    text
    kubectl describe node node01

    Look under Labels section

    --- OR ---

    text
    kubectl get node node01 --show-labels

  2. What is the value set to the label key beta.kubernetes.io/arch on node01?

    From the output of Q1, find the answer there.

  3. Apply a label color=blue to node node01

    text
    kubectl label node node01 color=blue

  4. Create a new deployment named blue with the nginx image and 3 replicas.

    text
    kubectl create deployment blue --image=nginx --replicas=3

  5. Which nodes can the pods for the blue deployment be placed on?

    Check if master and node01 have any taints on them that will prevent the pods to be scheduled on them. If there are no taints, the pods can be scheduled on either node.

    text
    kubectl describe nodes controlplane | grep -i taints
kubectl describe nodes node01 | grep -i taints

  6. Set Node Affinity to the deployment to place the pods on node01 only.
    Now we edit in place the deployment we created earlier. Remember that we labelled node01 with color=blue? Now we are going to create an affinity to that label, which will "attract" the pods of the deployment to it.

    1. text
      $ kubectl edit deployment blue
    2. Add the YAML below under the template.spec section, i.e. at the same level as containers as it is a POD setting. The affinity will be considered only during scheduling stage, however this edit will cause the deployment to roll out again.
    yaml
    affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: color
            operator: In
            values:
            - blue

  7. Which nodes are the pods placed on now?

    text
    $ kubectl get pods -o wide

  8. Create a new deployment named red with the nginx image and 2 replicas, and ensure it gets placed on the controlplane node only.

    1. Create a YAML template for the deploymemt

      text
      kubectl create deployment red --image nginx --replicas 2 --dry-run=client -o yaml > red.yaml

    2. Edit the file

      text
      vi red.yaml

    3. Add the toleration using the label stated in the question, and placing it as before for the blue deployment

    yaml
    affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node-role.kubernetes.io/control-plane
            operator: Exists
    1. Save, exit and create the deployment
    text
    kubectl create -f red.yaml
    1. Check the result
    text
    $ kubectl get pods -o wide

03 Scheduling 11.Taints And Tolerations Vs Node Affinity (`docs/03-Scheduling/11.Taints-and-Tolerations-vs-Node-Affinity.md`)

Taints and Tolerations vs Node Affinity

In this section, we will take a look at Taints and Tolerations vs Node Affinity

  • Taints and Tolerations do not guarantee that the pods will only prefer these nodes; in this case, the red pods may end up on one of the other nodes that do not have a taint or toleration set.

  • As such, a combination of taints and tolerations and node affinity rules can be used together to completely dedicate nodes for specific parts.

K8s Reference Docs:

03 Scheduling 12 Resource Limits (`docs/03-Scheduling/12-Resource-Limits.md`)

Resource Limits

In this section we will take a look at Resource Limits

Let us take a look at 3 node kubernetes cluster.

  • Each node has a set of CPU, Memory and Disk resources available.
  • If there is no sufficient resources available on any of the nodes, kubernetes holds the scheduling the pod. You will see the pod in pending state. If you look at the events, you will see the reason as insufficient CPU.

Resource Requirements

  • By default, K8s assume that a pod or container within a pod requires 0.5 CPU and 256Mi of memory. This is known as the Resource Request for a container.

  • If your application within the pod requires more than the default resources, you need to set them in the pod definition file.

    text
    apiVersion: v1
kind: Pod
metadata:
  name: simple-webapp-color
  labels:
    name: simple-webapp-color
spec:
 containers:
 - name: simple-webapp-color
   image: simple-webapp-color
   ports:
    - containerPort:  8080
   resources:
     requests:
      memory: "1Gi"
      cpu: "1"

Resources - Limits

  • By default, k8s sets resource limits to 1 CPU and 512Mi of memory

  • You can set the resource limits in the pod definition file.

    text
    apiVersion: v1
kind: Pod
metadata:
  name: simple-webapp-color
  labels:
    name: simple-webapp-color
spec:
 containers:
 - name: simple-webapp-color
   image: simple-webapp-color
   ports:
    - containerPort:  8080
   resources:
     requests:
      memory: "1Gi"
      cpu: "1"
     limits:
       memory: "2Gi"
       cpu: "2"

Note: Remember Requests and Limits for resources are set per container in the pod.

Exceed Limits

  • what happens when a pod tries to exceed resources beyond its limits?

K8s Reference Docs:

03 Scheduling 13 Practice Test Resource Limits (`docs/03-Scheduling/13-Practice-Test-Resource-Limits.md`)

Practice Test - Resource Limits

Solutions to practice test - resource limtis

  • Run the command 'kubectl describe pod rabbit' and inspect requests.

    text
    $ kubectl describe pod rabbit

  • Run the command 'kubectl delete pod rabbit'.

    text
    $ kubectl delete pod rabbit

  • Run the command 'kubectl get pods' and inspect the status of the pod elephant

    text
    $ kubectl get pods

  • The status 'OOMKilled' indicates that the pod ran out of memory. Identify the memory limit set on the POD.

  • Generate a template of the existing pod.

    text
    $ kubectl get pods elephant -o yaml > elephant.yaml

    Update the elephant.yaml pod definition with the resource memory limits to 20Mi

    text
    resources:
    limits:
      memory: 20Mi
---

    Delete the pod and recreate it.

    text
    $ kubectl delete pod elephant
$ kubectl create -f elephant.yaml

  • Inspect the status of POD. Make sure it's running

    text
    $ kubectl get pods

  • Run the command 'kubectl delete pod elephant'.

    text
    $ kubectl delete pod elephant

03 Scheduling 14 DaemonSets (`docs/03-Scheduling/14-DaemonSets.md`)

DaemonSets

In this section, we will take a look at DaemonSets.

DaemonSets are like replicasets, as it helps in to deploy multiple instances of pod. But it runs one copy of your pod on each node in your cluster.

DaemonSets - UseCases

DaemonSets - Definition

  • Creating a DaemonSet is similar to the ReplicaSet creation process.

  • For DaemonSets, we start with apiVersion, kind as DaemonSets instead of ReplicaSet, metadata and spec.

    text
    apiVersion: apps/v1
kind: Replicaset
metadata:
  name: monitoring-daemon
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: monitoring-agent
  template:
    metadata:
     labels:
       app: monitoring-agent
    spec:
      containers:
      - name: monitoring-agent
        image: monitoring-agent
    text
    apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: monitoring-daemon
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: monitoring-agent
  template:
    metadata:
     labels:
       app: monitoring-agent
    spec:
      containers:
      - name: monitoring-agent
        image: monitoring-agent

  • To create a daemonset from a definition file

    text
    $ kubectl create -f daemon-set-definition.yaml

View DaemonSets

  • To list daemonsets
    text
    $ kubectl get daemonsets
  • For more details of the daemonsets
    text
    $ kubectl describe daemonsets monitoring-daemon

How DaemonSets Works

K8s Reference Docs

03 Scheduling 15 Practice Test DaemonSets (`docs/03-Scheduling/15-Practice-Test-DaemonSets.md`)

Practice Test - DaemonSets

Solutions to practice test daemonsets

  • Run the command kubectl get daemonsets --all-namespaces

    text
    $ kubectl get daemonsets --all-namespaces

  • Run the command kubectl get daemonsets --all-namespaces

    text
    $ kubectl get daemonsets --all-namespaces

  • Run the command kubectl get all --all-namespaces and identify the types

    text
    $ kubectl get all --all-namespaces

  • Run the command kubectl describe daemonset kube-proxy --namespace=kube-system

    text
    $ kubectl describe daemonset kube-proxy --namespace=kube-system

  • Run the command kubectl describe daemonset kube-flannel-ds-amd64 --namespace=kube-system

    text
    $ kubectl describe daemonset kube-flannel-ds-amd64 --namespace=kube-system

  • Create a daemonset

    text
    $ vi ds.yaml
    text
    apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: elasticsearch
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
spec:
  selector:
    matchLabels:
      name: elasticsearch
  template:
    metadata:
      labels:
        name: elasticsearch
    spec:
      tolerations:
      # this toleration is to have the daemonset runnable on master nodes
      # remove it if your masters can't run pods
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: elasticsearch
        image: k8s.gcr.io/fluentd-elasticsearch:1.20

  • To create the daemonset and list the daemonsets and pods

    text
    $ kubectl create -f ds.yaml
$ kubectl get ds -n kube-system
$ kubectl get pod -n kube-system|grep elasticsearch

03 Scheduling 16 Static Pods (`docs/03-Scheduling/16-Static-Pods.md`)

Static Pods

In this section, we will take a look at Static Pods

How do you provide a pod definition file to the kubelet without a kube-apiserver?

  • You can configure the kubelet to read the pod definition files from a directory on the server designated to store information about pods.

Configure Static Pod

  • The designated directory can be any directory on the host and the location of that directory is passed in to the kubelet as an option while running the service.
    • The option is named as --pod-manifest-path.

Another way to configure static pod

  • Instead of specifying the option directly in the kubelet.service file, you could provide a path to another config file using the config option, and define the directory path as staticPodPath in the file.

View the static pods

  • To view the static pods
    text
    $ docker ps

The kubelet can create both kinds of pods - the static pods and the ones from the api server at the same time.

Static Pods - Use Case

Static Pods vs DaemonSets

K8s Reference Docs

03 Scheduling 17 Practice Test StaticPods (`docs/03-Scheduling/17-Practice-Test-StaticPods.md`)

Practice Test - Static Pods

Solutions to the practice test - static pods

  • Run the command kubectl get pods --all-namespaces and look for those with -controlplane appended in the name

    text
    $ kubectl get pods --all-namespaces

  • Which of the below components is NOT deployed as a static pod?

    text
    $ kubectl get pods --all-namespaces

  • Which of the below components is NOT deployed as a static POD?

    text
    $ kubectl get pods --all-namespaces

  • Run the kubectl get pods --all-namespaces -o wide

    text
    $ kubectl get pods --all-namespaces -o wide

  • Run the command ps -aux | grep kubelet and identify the config file - --config=/var/lib/kubelet/config.yaml. Then checkin the config file for staticPdPath.

    text
    $ ps -aux | grep kubelet

  • Count the number of files under /etc/kubernetes/manifests

  • Check the image defined in the /etc/kubernetes/manifests/kube-apiserver.yaml manifest file.

  • Create a pod definition file in the manifests folder. Use command kubectl run --restart=Never --image=busybox static-busybox --dry-run -o yaml --command -- sleep 1000 > /etc/kubernetes/manifests/static-busybox.yaml

    text
    $ kubectl run --restart=Never --image=busybox static-busybox --dry-run=client -o yaml --command -- sleep 1000 > /etc/kubernetes/manifests/static-busybox.yaml

  • Simply edit the static pod definition file and save it

    text
    /etc/kubernetes/manifests/static-busybox.yaml

    OR

    text
    Run the command with updated image tag:
kubectl run --restart=Never --image=busybox:1.28.4 static-busybox--dry-run=client -o yaml --command -- sleep 1000 > /etc/kubernetes/manifests/static-busybox.yaml

  • Identify which node the static pod is created on, ssh to the node and delete the pod definition file. If you don't know theIP of the node, run the kubectl get nodes -o wide command and identify the IP. Then SSH to the node using that IP. For static pod manifest path look at the file /var/lib/kubelet/config.yaml on node01

    text
    $ kubectl get pods -o wide
$ kubectl get nodes -o wide
$ ssh <ip address of pod>
$ grep staticPodPath /var/lib/kubelet/config.yaml
$ node01 $ rm -rf /etc/just-to-mess-with-you/greenbox.yaml

03 Scheduling 18 Multiple Schedulers (`docs/03-Scheduling/18-Multiple-Schedulers.md`)

Multiple Schedulers

In this section, we will take a look at multiple schedulers

Custom Schedulers

  • Your kubernetes cluster can schedule multiple schedulers at the same time.

Deploy additional scheduler

  • Download the binary
    text
    $ wget https://storage.googleapis.com/kubernetes-release/release/v1.12.0/bin/linux/amd64/kube-scheduler

Deploy additional scheduler - kubeadm

  • To create a scheduler pod
    text
    $ kubectl create -f my-custom-scheduler.yaml

View Schedulers

  • To list the scheduler pods
    text
    $ kubectl get pods -n kube-system

Use the Custom Scheduler

  • Create a pod definition file and add new section called schedulerName and specify the name of the new scheduler

    text
    apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - image: nginx
    name: nginx
  schedulerName: my-custom-scheduler

  • To create a pod definition

    text
    $ kubectl create -f pod-definition.yaml

  • To list pods

    text
    $ kubectl get pods

View Events

  • To view events
    text
    $ kubectl get events

View Scheduler Logs

  • To view scheduler logs
    text
    $ kubectl logs my-custom-scheduler -n kube-system

K8s Reference Docs

03 Scheduling 19 Practice Test Multiple Schedulers (`docs/03-Scheduling/19-Practice-Test-Multiple-Schedulers.md`)

Practice Test - Multiple Schedulers

Solutions to practice test - multiple schedulers

  • Run the command 'kubectl get pods --namespace=kube-system'

    text
    $ kubectl get pods --namespace=kube-system

  • Run the command 'kubectl describe pod kube-scheduler-controlplane --namespace=kube-system'

    text
    $ kubectl describe pod kube-scheduler-controlplane --namespace=kube-system

  • Use the imperative command to create the configmap with option --from-file

    text
    $ kubectl create -n kube-system configmap my-scheduler-config --from-file=/root/my-scheduler-config.yaml

  • Use the file at /root/my-scheduler.yaml to create your own scheduler with correct image.

    text
    $ kubectl create -f /root/my-scheduler.yaml

  • Set schedulerName property on pod specification to the name of the new scheduler. File is located at /root/nginx-pod.yaml

    text
    master $ grep schedulerName /root/nginx-pod.yaml
schedulerName: my-scheduler

$ kubectl create -f /root/nginx-pod.yaml

03 Scheduling 20 Configuring Kubernetes Schedulers (`docs/03-Scheduling/20-Configuring-Kubernetes-Schedulers.md`)

Configuring Kubernetes Schedulers

In this section, we will take a look at configuring kubernetes schedulers.

References

03 Scheduling 21 Download Presentation Deck (`docs/03-Scheduling/21-Download-Presentation-Deck.md`)

Download Presentation Deck

04 Logging And Monitoring 01 Logging And Monitoring Section Introduction (`docs/04-Logging-and-Monitoring/01-Logging-and-Monitoring-Section-Introduction.md`)

Logging and Monitoring Section Introduction

In this section, we will take a look at the below

  • Monitor Cluster Components
  • Monitor Applications
  • Monitor Cluster Components Logs
  • Application Logs

04 Logging And Monitoring 02 Monitor Cluster Components (`docs/04-Logging-and-Monitoring/02-Monitor-Cluster-Components.md`)

Monitor Cluster Components

In this section, we will take a look at monitoring kubernetes cluster

How do you monitor resource consumption in kubernetes? or more importantly, what would you like to monitor?

Heapster vs Metrics Server

  • Heapster is now deprecated and a slimmed down version was formed known as the metrics server.

Metrics Server

How are the metrics generated for the PODs on these nodes?

Metrics Server - Getting Started

  • Clone the metric server from github repo

    text
    $ git clone https://github.com/kubernetes-incubator/metrics-server.git

  • Deploy the metric server

    text
    $ kubectl create -f metric-server/deploy/1.8+/

  • View the cluster performance

    text
    $ kubectl top node

  • View performance metrics of pod

    text
    $ kubectl top pod

04 Logging And Monitoring 03 Practice Test Monitor Cluster Components (`docs/04-Logging-and-Monitoring/03-Practice-Test-Monitor-Cluster-Components.md`)

Practice Test - Monitor Cluster Components

Solutions to practice test - monitor cluster components
1.
We have deployed a few PODs running workloads. Inspect it.

text
```
kubectl get pods
```

  1. Let us deploy metrics-server to monitor the PODs and Nodes. Pull the git repository for the deployment files.

    text
    git clone https://github.com/kodekloudhub/kubernetes-metrics-server.git

  2. Deploy the metrics-server by creating all the components downloaded.

    Run the 'kubectl create -f .' command from within the downloaded repository.

    text
    cd kubernetes-metrics-server
kubectl create -f .

  3. It takes a few minutes for the metrics server to start gathering data.

    Run the kubectl top node command and wait for a valid output.

    text
    kubectl top node

  4. Identify the node that consumes the most CPU(cores).

    Run the kubectl top node command

    text
    kubectl top node

    Examine the CPU(cores) column of the output to get the answer.

  5. Identify the node that consumes the most Memory(bytes).
    Run the kubectl top node command

    text
    $ kubectl top node

    Examine the MEMORY(bytes) column of the output to get the answer.

  6. Identify the POD that consumes the most Memory(bytes).

    Run the kubectl top pod command

    text
    kubectl top pod

    Examine the MEMORY(bytes) column of the output to get the answer.

text
Identify the POD that consumes the least CPU(cores).

Run the `kubectl top pod` command

```
kubectl top pod
```

Examine the `CPU(cores)` column of the output to get the answer.

04 Logging And Monitoring 04 Managing Application Logs (`docs/04-Logging-and-Monitoring/04-Managing-Application-Logs.md`)

Managing Application Logs

In this section, we will take a look at managing application logs

Let us start with logging in docker

Logs - Kubernetes

text
apiVersion: v1
kind: Pod
metadata:
  name: event-simulator-pod
spec:
  containers:
  - name: event-simulator
    image: kodekloud/event-simulator
  • To view the logs
    text
    $ kubectl logs -f event-simulator-pod
  • If there are multiple containers in a pod then you must specify the name of the container explicitly in the command.
    text
    $ kubectl logs -f <pod-name> <container-name>
$ kubectl logs -f even-simulator-pod event-simulator

K8s Reference Docs

04 Logging And Monitoring 05 Download Presentation Deck (`docs/04-Logging-and-Monitoring/05-Download-Presentation-Deck.md`)

Download Presentation Deck

04 Logging And Monitoring 06 Practice Test Managing Application Logs (`docs/04-Logging-and-Monitoring/06-Practice-Test-Managing-Application-Logs.md`)

Practice Test - Managing Application Logs

Solutions to practice test - managing application logs

  • We have deployed a POD hosting an application. Inspect it. Wait for it to start.

    text
    $ kubectl get pods

  • Inspect the logs of the POD

    text
    $ kubectl logs webapp-1

  • We have deployed a new POD - 'webapp-2' - hosting an application. Inspect it. Wait for it to start.

    text
    $ kubectl get pods

  • Inspect the logs of the webapp in the POD

    text
    $ kubectl logs webapp-2

05 Application Lifecycle Management 01 Application Lifecycle Management Section Introduction (`docs/05-Application-Lifecycle-Management/01-Application-Lifecycle-Management--Section-Introduction.md`)

Application Lifecycle Management Section Introduction

In this section, we will take a look at application lifecycle management

  • Rolling Updates and Rollbacks in Deployments
  • Configure Applications
  • Scale Applications
  • Self-Healing Application

05 Application Lifecycle Management 02 RollingUpdates And Rollback (`docs/05-Application-Lifecycle-Management/02-RollingUpdates-and-Rollback.md`)

Rolling Updates and Rollback

In this section, we will take a look at rolling updates and rollback in a deployment

Rollout and Versioning in a Deployment

Rollout commands

  • You can see the status of the rollout by the below command
    text
    $ kubectl rollout status deployment/myapp-deployment
  • To see the history and revisions
    text
    $ kubectl rollout history deployment/myapp-deployment

Deployment Strategies

  • There are 2 types of deployment strategies
    1. Recreate
    2. RollingUpdate (Default Strategy)

kubectl apply

  • To update a deployment, edit the deployment and make necessary changes and save it. Then run the below command.
    text
    apiVersion: apps/v1
kind: Deployment
metadata:
 name: myapp-deployment
 labels:
  app: nginx
spec:
 template:
   metadata:
     name: myap-pod
     labels:
       app: myapp
       type: front-end
   spec:
    containers:
    - name: nginx-container
      image: nginx:1.7.1
 replicas: 3
 selector:
  matchLabels:
    type: front-end
    text
    $ kubectl apply -f deployment-definition.yaml
  • Alternate way to update a deployment say for example for updating an image.
    text
    $ kubectl set image deployment/myapp-deployment nginx=nginx:1.9.1

Recreate vs RollingUpdate

Upgrades

Rollback

  • To undo a change
    text
    $ kubectl rollout undo deployment/myapp-deployment

kubectl create

  • To create a deployment
    text
    $ kubectl create deployment nginx --image=nginx

Summarize kubectl commands

text
$ kubectl create -f deployment-definition.yaml
$ kubectl get deployments
$ kubectl apply -f deployment-definition.yaml
$ kubectl set image deployment/myapp-deployment nginx=nginx:1.9.1
$ kubectl rollout status deployment/myapp-deployment
$ kubectl rollout history deployment/myapp-deployment
$ kubectl rollout undo deployment/myapp-deployment

K8s Reference Docs

05 Application Lifecycle Management 03 Practice Test RollingUpdates Rollback (`docs/05-Application-Lifecycle-Management/03-Practice-Test-RollingUpdates-Rollback.md`)

Practice Test - Rolling Updates and Rollback

Solutions to practice test - rolling updates and rollback

  • We have deployed a simple web application. Inspect the PODs and the Services

    text
    $ kubectl get pods
$ kubectl get services

  • What is the current color of the web application?

    text
    Access the web portal

  • Execute the script at /root/curl-test.sh.

  • Run the command 'kubectl describe deployment' and look at 'Desired Replicas'

    text
    $ kubectl describe deployment

  • Run the command 'kubectl describe deployment' and look for 'Images'

    text
    $ kubectl describe deployment

  • Run the command 'kubectl describe deployment' and look at 'StrategyType'

    text
    $ kubectl describe deployment

  • If you were to upgrade the application now what would happen?

    text
    PODs are upgraded few at a time

  • Run the command 'kubectl edit deployment frontend' and modify the required feild

    text
    $ kubectl edit deployment frontend

  • Execute the script at /root/curl-test.sh.

  • Look at the Max Unavailable value under RollingUpdateStrategy in deployment details

    text
    $ kubectl describe deployment

  • Run the command 'kubectl edit deployment frontend' and modify the required field. Make sure to delete the properties of rollingUpdate as well, set at 'strategy.rollingUpdate'.

    text
    $ kubectl edit deployment frontend

  • Run the command 'kubectl edit deployment frontend' and modify the required feild

    text
    $ kubectl edit deployment frontend

  • Execute the script at /root/curl-test.sh.

05 Application Lifecycle Management 04 Commands And Arguments In Docker (`docs/05-Application-Lifecycle-Management/04-Commands-and-Arguments-in-Docker.md`)

Commands and Arguments in Docker

In this section, we will take a look at commands and arguments in docker

  • To run a docker container
    text
    $ docker run ubuntu
  • To list running containers
    text
    $ docker ps
  • To list all containers including that are stopped
    text
    $ docker ps -a

Unlike virtual machines, containers are not meant to host operating system.

  • Containers are meant to run a specific task or process such as to host an instance of a webserver or application server or a database server etc.

How do you specify a different command to start the container?

  • One Option is to append a command to the docker run command and that way it overrides the default command specified within the image.

    text
    $ docker run ubuntu sleep 5

  • This way when the container starts it runs the sleep program, waits for 5 seconds and then exists. How do you make that change permanent?

  • There are different ways of specifying the command either the command simply as is in a shell form or in a JSON array format.

  • Now, build the docker image

    text
    $ docker build -t ubuntu-sleeper .

  • Run docker container

    text
    $ docker run ubuntu-sleeper

Entrypoint Instruction

  • The entrypoint instruction is like the command instruction as in you can specify the program that will be run when the container starts and whatever you specify on the command line.

K8s Reference Docs

05 Application Lifecycle Management 05 Commands And Arguments In Kubernetes (`docs/05-Application-Lifecycle-Management/05-Commands-and-Arguments-in-Kubernetes.md`)

Commands and Arguments in Kubernetes

In this section, we will take a look at commands and arguments in kubernetes

  • Anything that is appended to the docker run command will go into the args property of the pod definition file in the form of an array.
  • The command field corresponds to the entrypoint instruction in the Dockerfile so to summarize there are 2 fields that correspond to 2 instructions in the Dockerfile.
    text
    apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-sleeper-pod
spec:
 containers:
 - name: ubuntu-sleeper
   image: ubuntu-sleeper
   command: ["sleep2.0"]
   args: ["10"]

K8s Reference Docs

05 Application Lifecycle Management 06 Practice Test Commands And Arguments (`docs/05-Application-Lifecycle-Management/06-Practice-Test-Commands-and-Arguments.md`)

Practice Test - Commands and Arguments

Solutions to practice test - commands and arguments

  • Run the command 'kubectl get pods' and count the number of pods.

    text
    $ kubectl get pods

  • Run the command 'kubectl describe pod' and look for command option

    text
    $ kubectl describe pod

  • Set the command option to ['sleep', '5000']. Answer file at: /var/answers/answer-ubuntu-sleeper-2.yaml

  • Both sleep and 1200 should be defined as a string. Answer file at: /var/answers/answer-ubuntu-sleeper-3.yaml

  • Answer file at: /var/answers/answer-ubuntu-sleeper-3-2.yaml

  • Inspect the file 'Dockerfile' given at /root/webapp-color. What command is run at container startup?

    text
    python app.py

  • Inspect the file 'Dockerfile2' given at /root/webapp-color. What command is run at container startup?

    text
    python app.py --color red

  • The 'command' (entrypoint) is overridden in the pod definition. So the answer is --color green

  • Inspect the two files under directory 'webapp-color-3'. What command is run at container startup?

    text
    python app.py --color pink

  • Answer file located at /var/answers/answer-webapp-color-green.yaml

05 Application Lifecycle Management 07.Configure Environment Variables In Applications (`docs/05-Application-Lifecycle-Management/07.Configure-Environment-Variables-in-Applications.md`)

Configure Environment Variables In Applications

ENV variables in Docker

text
$ docker run -e APP_COLOR=pink simple-webapp-color

ENV variables in kubernetes

  • To set an environment variable set an env property in pod definition file.

    text
    apiVersion: v1
kind: Pod
metadata:
  name: simple-webapp-color
spec:
 containers:
 - name: simple-webapp-color
   image: simple-webapp-color
   ports:
   - containerPort: 8080
   env:
   - name: APP_COLOR
     value: pink

  • There are other ways of setting the environment variables such as ConfigMaps and Secrets

K8s Reference Docs

05 Application Lifecycle Management 08 Configure ConfigMaps In Applications (`docs/05-Application-Lifecycle-Management/08-Configure-ConfigMaps-in-Applications.md`)

Configure ConfigMaps in Applications

In this section, we will take a look at configuring configmaps in applications

ConfigMaps

  • There are 2 phases involved in configuring ConfigMaps.
    • First, create the configMaps
    • Second, Inject then into the pod.
  • There are 2 ways of creating a configmap.

    • The Imperative way

      text
      $ kubectl create configmap app-config --from-literal=APP_COLOR=blue --from-literal=APP_MODE=prod
$ kubectl create configmap app-config --from-file=app_config.properties (Another way)

    • The Declarative way

      text
      apiVersion: v1
kind: ConfigMap
metadata:
 name: app-config
data:
 APP_COLOR: blue
 APP_MODE: prod
      text
      Create a config map definition file and run the 'kubectl create` command to deploy it.
$ kubectl create -f config-map.yaml

View ConfigMaps

  • To view configMaps
    text
    $ kubectl get configmaps (or)
$ kubectl get cm
  • To describe configmaps
    text
    $ kubectl describe configmaps

ConfigMap in Pods

  • Inject configmap in pod
    text
    apiVersion: v1
kind: Pod
metadata:
  name: simple-webapp-color
spec:
 containers:
 - name: simple-webapp-color
   image: simple-webapp-color
   ports:
   - containerPort: 8080
   envFrom:
   - configMapRef:
       name: app-config
    text
    apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  APP_COLOR: blue
  APP_MODE: prod
    text
    $ kubectl create -f pod-definition.yaml

There are other ways to inject configuration variables into pod

  • You can inject it as a Single Environment Variable
  • You can inject it as a file in a Volume

K8s Reference Docs

05 Application Lifecycle Management 09 Practice Test Env Variables (`docs/05-Application-Lifecycle-Management/09-Practice-Test-Env-Variables.md`)

Practice Test Env Variables

Solutions to practice test env variables

  • Run the command 'kubectl get pods' and count the number of pods.

    text
    $ kubectl get pods

  • Run the command 'kubectl describe pod' and look for ENV option

    text
    $ kubectl describe pod

  • Run the command 'kubectl describe pod' and look for ENV option

    text
    $ kubectl describe pod

  • View the web application UI by clicking on the 'Webapp Color' Tab above your terminal.

  • Set the environment option to APP_COLOR - green.

    text
    $ kubectl get pods webapp-color -o yaml > green.yaml
$ kubectl delete pods webapp-color
Update APP_COLOR to green
$ kubectl create -f green.yaml

  • View the changes to the web application UI by clicking on the 'Webapp Color' Tab above your terminal.

  • Run kubectl get configmaps

    text
    $ kubectl get configmaps

  • Run the command 'kubectl describe configmaps' and look for DB_HOST option

    text
    $ kubectl describe configmaps

  • Create a new ConfigMap for the 'webapp-color' POD. Use the spec given on the right.

    text
    $ kubectl create configmap webapp-config-map --from-literal=APP_COLOR=darkblue

  • Set the environment option to envFrom and use configMapRef webapp-config-map.

    text
    $ kubectl get pods webapp-color -o yaml > new-webapp.yaml
$ kubectl delete pods webapp-color
 Update pod definition file, under spec.containers section update the below.
- envFrom:
  - configMapRef:
     name: webapp-config-map
    text
    $ kubectl create -f new-webapp.yaml

  • View the changes to the web application UI by clicking on the 'Webapp Color' Tab above your terminal.

05 Application Lifecycle Management 10.Secrets (`docs/05-Application-Lifecycle-Management/10.Secrets.md`)

Secrets

In this section, we will take a look at secrets in kubernetes

Web-Mysql Application

  • One way is to move the app properties/envs into a configmap. But the configmap stores data into a plain text format. It is definitely not a right place to store a password.

    text
    apiVersion: v1
kind: ConfigMap
metadata:
 name: app-config
data:
  DB_Host: mysql
  DB_User: root
  DB_Password: paswrd

  • Secrets are used to store sensitive information. They are similar to configmaps but they are stored in an encrypted format or a hashed format.

There are 2 steps involved with secrets

  • First, Create a secret
  • Second, Inject the secret into a pod.

There are 2 ways of creating a secret

  • The Imperative way

    text
    $ kubectl create secret generic app-secret --from-literal=DB_Host=mysql --from-literal=DB_User=root --from-literal=DB_Password=paswrd
$ kubectl create secret generic app-secret --from-file=app_secret.properties

  • The Declarative way

    text
    Generate a hash value of the password and pass it to secret-data.yaml definition value as a value to DB_Password variable.
$ echo -n "mysql" | base64
$ echo -n "root" | base64
$ echo -n "paswrd"| base64

    Create a secret definition file and run kubectl create to deploy it

    text
    apiVersion: v1
kind: Secret
metadata:
 name: app-secret
data:
  DB_Host: bX1zcWw=
  DB_User: cm9vdA==
  DB_Password: cGFzd3Jk
    text
    $ kubectl create -f secret-data.yaml

Encode Secrets

View Secrets

  • To view secrets
    text
    $ kubectl get secrets
  • To describe secret
    text
    $ kubectl describe secret
  • To view the values of the secret
    text
    $ kubectl get secret app-secret -o yaml

Decode Secrets

  • To decode secrets
    text
    $ echo -n "bX1zcWw=" | base64 --decode
$ echo -n "cm9vdA==" | base64 --decode
$ echo -n "cGFzd3Jk" | base64 --decode

Configuring secret with a pod

  • To inject a secret to a pod add a new property envFrom followed by secretRef name and then create the pod-definition

    text
    apiVersion: v1
kind: Secret
metadata:
 name: app-secret
data:
  DB_Host: bX1zcWw=
  DB_User: cm9vdA==
  DB_Password: cGFzd3Jk
    text
    apiVersion: v1
 kind: Pod
 metadata:
   name: simple-webapp-color
 spec:
  containers:
  - name: simple-webapp-color
    image: simple-webapp-color
    ports:
    - containerPort: 8080
    envFrom:
    - secretRef:
        name: app-secret
    text
    $ kubectl create -f pod-definition.yaml

There are other ways to inject secrets into pods.

  • You can inject as Single ENV variable
  • You can inject as whole secret as files in a Volume

Secrets in pods as volume

  • Each attribute in the secret is created as a file with the value of the secret as its content.

Additional Notes: [A Note on Secrets](https://kodekloud.com/topic/a-note-on-secrets/)

Remember that secrets encode data in base64 format. Anyone with the base64 encoded secret can easily decode it. As such the secrets can be considered not very safe.

The concept of safety of the Secrets is a bit confusing in Kubernetes. The kubernetes documentation page and a lot of blogs out there refer to secrets as a “safer option” to store sensitive data. They are safer than storing in plain text as they reduce the risk of accidentally exposing passwords and other sensitive data. In my opinion it’s not the secret itself that is safe, it is the practices around it.

Secrets are not encrypted, so it is not safer in that sense. However, some best practices around using secrets make it safer. As in best practices like:

  • Not checking-in secret object definition files to source code repositories.
  • Enabling Encryption at Rest for Secrets so they are stored encrypted in ETCD.

Also the way kubernetes handles secrets. Such as:

  • A secret is only sent to a node if a pod on that node requires it.
  • Kubelet stores the secret into a tmpfs so that the secret is not written to disk storage.
  • Once the Pod that depends on the secret is deleted, kubelet will delete its local copy of the secret data as well.

Read about the protections and risks of using secrets here

Having said that, there are other better ways of handling sensitive data like passwords in Kubernetes, such as using tools like Helm Secrets, HashiCorp Vault. I hope to make a lecture on these in the future.

K8s Reference Docs

05 Application Lifecycle Management 11.Practice Test Secrets (`docs/05-Application-Lifecycle-Management/11.Practice-Test-Secrets.md`)

Practice Test - Secrets

Solutions for pracice test - Secrets

  • Run the command 'kubectl get secrets' and count the number of pods.

    text
    $ kubectl get secrets

  • Run the command 'kubectl get secrets' and look at the DATA field

    text
    $ kubectl get secrets

  • Run the command 'kubectl describe secret'

    text
    $ kubectl describe secret

  • Run the command 'kubectl describe secret'

    text
    $ kubectl describe secret

  • We have already deployed the required pods and services. Check out the pods and services created. Check out the web application using the 'Webapp MySQL' link above your terminal, next to the Quiz Portal Link.

  • Run command kubectl create secret generic db-secret --from-literal=DB_Host=sql01 --from-literal=DBUser=root --from-literal=DB_Password=password123

    text
    $ kubectl create secret generic db-secret --from-literal=DB_Host=sql01 --from-literal=DB_User=root --from-literal=DB_Password=password123

  • Check Answer at /var/answers/answer-webapp.yaml

    text
    $ kubectl get pod webapp-pod -o yaml > web.yaml
$ kubectl delete pod webapp-pod

    Update web.yaml with secret section

    envFrom:

    • secretRef:
      name: db-secret
    text
    $ kubectl create -f web.yaml

  • View the web application to verify it can successfully connect to the database

05 Application Lifecycle Management 12.Multi Containers PODs (`docs/05-Application-Lifecycle-Management/12.Multi-Containers-PODs.md`)

Multi-Container Pods

In this section, we will take a look at multi-container pods

Monolith and Microservices

Multi-Container Pods

  • To create a new multi-container pod, add the new container information to the pod definition file.
    text
    apiVersion: v1
kind: Pod
metadata:
  name: simple-webapp
  labels:
    name: simple-webapp
spec:
  containers:
  - name: simple-webapp
    image: simple-webapp
    ports:
    - ContainerPort: 8080
  - name: log-agent
    image: log-agent

K8s Reference Docs

05 Application Lifecycle Management 13 Practice Test Multi Container Pods (`docs/05-Application-Lifecycle-Management/13-Practice-Test-Multi-Container-Pods.md`)

Practice Test - Multi-Container Pods

Solutions to practice test - multi-container pods

  • Identify the number of containers running in the 'red' pod.

    text
    $ kubectl get pod red

  • Identify the name of the containers running in the 'blue' pod.

    text
    $ kubectl describe pod blue

  • Answer file is located at /var/answers/answer-yellow.yaml

    text
    $ kubectl create -f /var/answers/answer-yellow.yaml

  • We have deployed an application logging stack in the elastic-stack namespace. Inspect it.

    text
    $ kubectl get pods -n elastic-stack

  • Inspect the Kibana UI using the link above your terminal. There shouldn't be any logs for now.

  • Run kubectl describe pod -n elastic-stack

    text
    $ kubectl describe pod -n elastic-stack

  • Run the command 'kubectl -n elastic-stack exec -it app cat /log/app.log'

    text
    $ kubectl -n elastic-stack exec -it app cat /log/app.log

  • Answer file is located at /var/answers/answer-app.yaml

  • Inspect the Kibana UI. You should now see logs appearing in the 'Discover' section. You might have to wait for a couple of minutes for the logs to populate. You might have to create an index pattern to list the logs. If not sure check this video: https://bit.ly/2EXYdHf

05 Application Lifecycle Management 14 Multi Container Pods Design Patterns (`docs/05-Application-Lifecycle-Management/14-Multi-Container-Pods-Design-Patterns.md`)

Multi-Container Pods Design Patterns

K8s Reference Docs

05 Application Lifecycle Management 15.Init Containers (`docs/05-Application-Lifecycle-Management/15.Init-Containers.md`)

Init Containers

In a multi-container pod, each container is expected to run a process that stays alive as long as the POD’s lifecycle. For example in the multi-container pod that we talked about earlier that has a web application and logging agent, both the containers are expected to stay alive at all times. The process running in the log agent container is expected to stay alive as long as the web application is running. If any of them fails, the POD restarts.

But at times you may want to run a process that runs to completion in a container. For example a process that pulls a code or binary from a repository that will be used by the main web application. That is a task that will be run only one time when the pod is first created. Or a process that waits for an external service or database to be up before the actual application starts. That’s where initContainers comes in.

An initContainer is configured in a pod like all other containers, except that it is specified inside a initContainers section, like this:

yaml
apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app: myapp
spec:
  containers:
  - name: myapp-container
    image: busybox:1.28
    command: ['sh', '-c', 'echo The app is running! && sleep 3600']
  initContainers:
  - name: init-myservice
    image: busybox
    command: ['sh', '-c', 'git clone <some-repository-that-will-be-used-by-application> ;']

When a POD is first created the initContainer is run, and the process in the initContainer must run to a completion before the real container hosting the application starts.

You can configure multiple such initContainers as well, like how we did for multi-containers pod. In that case, each init container is run one at a time in sequential order.

If any of the initContainers fail to complete, Kubernetes restarts the Pod repeatedly until the Init Container succeeds.

yaml
apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app: myapp
spec:
  containers:
  - name: myapp-container
    image: busybox:1.28
    command: ['sh', '-c', 'echo The app is running! && sleep 3600']
  initContainers:
  - name: init-myservice
    image: busybox:1.28
    command: ['sh', '-c', 'until nslookup myservice; do echo waiting for myservice; sleep 2; done;']
  - name: init-mydb
    image: busybox:1.28
    command: ['sh', '-c', 'until nslookup mydb; do echo waiting for mydb; sleep 2; done;']

K8s Reference Docs

05 Application Lifecycle Management 16 Practice Test Init Containers (`docs/05-Application-Lifecycle-Management/16-Practice-Test-Init-Containers.md`)

Practice Test - Init-Containers

Solutions to practice test - init-containers

  • Identify the pod that has an initContainer configured.

    text
    $ kubectl get pods
$ kubectl describe pods

  • What is the image used by the initContainer on the blue pod?

    text
    $ kubectl describe pods blue

  • Run the command kubectl describe pod blue and check the state field of the initContainer.

    text
    $ kubectl describe pod blue

  • Check the reason field of the initContainer

    text
    $ kubectl describe pod blue

  • Run the command kubectl describe pod purple

    text
    $ kubectl describe pod purple

  • Run the kubectl describe pod purple command and look at the container state

    text
    $ kubectl describe pod purple

  • Check the commands used in the initContainers. The first one sleeps for 600 seconds (10 minutes) and the second one sleeps for 1200 seconds (20 minutes)

    text
    $ kubectl describe pod purple

  • Update the pod red to use an initContainer that uses the busybox image and sleeps for 20 seconds

    text
    $ kubectl get pod red -o yaml > red.yaml
$ kubectl delete pod red

    Update the red.yaml with sleep 20 seconds

    text
    $ kubectl create -f red.yaml

  • Check the command used by the initContainer. Looks like there is a type in sleep command. Fix it, it should be sleep 2 not sleeeep 2

    text
    $ kubectl describe pod orange
$ kubectl get pod orange -o yaml > orange.yaml
$ kubectl delete pod orange

Update the orange.yaml with correct sleep command and recreate the pod
$ kubectl create -f orange.yaml

05 Application Lifecycle Management 17.Self Healing Applications (`docs/05-Application-Lifecycle-Management/17.Self-Healing-Applications.md`)

Self Healing Applications

Self Healing Applications

Kubernetes supports self-healing applications through ReplicaSets and Replication Controllers. The replication controller helps in ensuring that a POD is re-created automatically when the application within the POD crashes. It helps in ensuring enough replicas of the application are running at all times.

Kubernetes provides additional support to check the health of applications running within PODs and take necessary actions through Liveness and Readiness Probes. However these are not required for the CKA exam and as such they are not covered here. These are topics for the Certified Kubernetes Application Developers (CKAD) exam and are covered in the CKAD course.

05 Application Lifecycle Management 18.Download Presentation Deck (`docs/05-Application-Lifecycle-Management/18.Download-Presentation-Deck.md`)

Download Presentation Deck

06 Cluster Maintenance 01 Cluster Maintenance Section Introduction (`docs/06-Cluster-Maintenance/01-Cluster-Maintenance-Section-Introduction.md`)

Cluster Maintenance Section Introduction

In this section, we will take a look at cluster maintenance

  • Cluster Upgrade Process
  • Operating System Upgrades
  • Backup and Restore Methodologies

06 Cluster Maintenance 02 OS Upgrades (`docs/06-Cluster-Maintenance/02-OS-Upgrades.md`)

OS Upgrades

In this section, we will take a look at OS upgrades.

If the node was down for more than 5 minutes, then the pods are terminated from that node

  • You can purposefully drain the node of all the workloads so that the workloads are moved to other nodes.
    text
    $ kubectl drain node-1
  • The node is also cordoned or marked as unschedulable.
  • When the node is back online after a maintenance, it is still unschedulable. You then need to uncordon it.
    text
    $ kubectl uncordon node-1
  • There is also another command called cordon. Cordon simply marks a node unschedulable. Unlike drain it does not terminate or move the pods on an existing node.

K8s Reference Docs

06 Cluster Maintenance 03 Practice Test OS Upgrades (`docs/06-Cluster-Maintenance/03-Practice-Test-OS-Upgrades.md`)

Practice Test - OS Upgrades

Solutions to practice test - OS Upgrades

  • Let us explore the environment first. How many nodes do you see in the cluster?

    text
    $ kubectl get nodes

  • How many applications do you see hosted on the cluster?

    text
    $ kubectl get deploy

  • Run the command 'kubectl get pods -o wide' and get the list of nodes the pods are placed on

    text
    $ kubectl get pods -o wide

  • Run the command kubectl drain node01 --ignore-daemonsets

    text
    $ kubectl drain node01 --ignore-daemonsets

  • Run the command 'kubectl get pods -o wide' and get the list of nodes the pods are placed on

    text
    $ kubectl get pods -o wide

  • Run the command kubectl uncordon node01

    text
    $ kubectl uncordon node01

  • Run the command kubectl get pods -o wide

    text
    $ kubectl get pods -o wide

  • Why are there no pods on node01?

    text
    Only when new pods are created they will be scheduled

  • Use the command kubectl describe node master and look under taint section to check if it has any taints.

    text
    $ kubectl describe node master

  • Run the command kubectl drain node02 --ignore-daemonsets

    text
    $ kubectl drain node02 --ignore-daemonsets

  • Check the applications hosted on the node02.

    text
    node02 has a pod not part of a replicaset
$ kubectl get pods -o wide

  • Check the list of pods

    text
    $ kubectl get pods -o wide

  • What would happen to hr-app if node02 is drained forcefully?

    text
    $ kubectl drain node02 --ignore-daemonsets --force
hr-app will be lost forever

  • Run the command kubectl drain node02 --ignore-daemonsets --force

    text
    $ kubectl drain node02 --ignore-daemonsets --force

  • Run the command kubectl cordon node03

    text
    $ kubectl cordon node03

06 Cluster Maintenance 04 Kubernetes Software Versions (`docs/06-Cluster-Maintenance/04-Kubernetes-Software-Versions.md`)

Kubernetes Software Versions

In this section, we will take a look at various kubernetes releases and versions

We can see the kubernetess version that we installed

text
$ kubectl get nodes

Let's take a closer look at the version number

  • It consists of 3 parts
    • First is the major version
    • Second is the minor version
    • Finally, the patch version

Kubernetes follows a standard software release versioning procedure

Downloaded package has all the kubernetes components in it except **`ETCD Cluster`** and **`CoreDNS`** as they are seperate projects.

References

06 Cluster Maintenance 05 Cluster Upgrade Introduction (`docs/06-Cluster-Maintenance/05-Cluster-Upgrade-Introduction.md`)

Cluster Upgrade Introduction

Is it mandatory for all of the kubernetes components to have the same versions?

  • No, The components can be at different release versions.

At any time, kubernetes supports only up to the recent 3 minor versions

  • The recommended approach is to upgrade one minor version at a time.

Options to upgrade k8s cluster

Upgrading a Cluster

  • Upgrading a cluster involves 2 major steps

There are different strategies that are available to upgrade the worker nodes

  • One is to upgrade all at once. But then your pods will be down and users will not be able to access the applications.

  • Second one is to upgrade one node at a time.

  • Third one would be to add new nodes to the cluster

kubeadm - Upgrade master node

  • kubeadm has an upgrade command that helps in upgrading clusters.

    text
    $ kubeadm upgrade plan

  • Upgrade kubeadm from v1.11 to v1.12

    text
    $ apt-get upgrade -y kubeadm=1.12.0-00

  • Upgrade the cluster

    text
    $ kubeadm upgrade apply v1.12.0

  • If you run the 'kubectl get nodes' command, you will see the older version. This is because in the output of the command it is showing the versions of kubelets on each of these nodes registered with the API Server and not the version of API Server itself

    text
    $ kubectl get nodes

  • Upgrade 'kubelet' on the master node

    text
    $ apt-get upgrade kubelet=1.12.0-00

  • Restart the kubelet

    text
    $ systemctl restart kubelet

  • Run 'kubectl get nodes' to verify

    text
    $ kubectl get nodes

kubeadm - Upgrade worker nodes

  • From master node, run 'kubectl drain' command to move the workloads to other nodes

    text
    $ kubectl drain node-1

  • Upgrade kubeadm and kubelet packages

    text
    $ apt-get upgrade -y kubeadm=1.12.0-00
$ apt-get upgrade -y kubelet=1.12.0-00

  • Update the node configuration for the new kubelet version

    text
    $ kubeadm upgrade node config --kubelet-version v1.12.0

  • Restart the kubelet service

    text
    $ systemctl restart kubelet

  • Mark the node back to schedulable

    text
    $ kubectl uncordon node-1

  • Upgrade all worker nodes in the same way

Demo Video on [Cluster Upgrade](https://kodekloud.com/topic/demo-cluster-upgrade/)

K8s Reference Docs

06 Cluster Maintenance 06 Practice Test Cluster Upgrade Process (`docs/06-Cluster-Maintenance/06-Practice-Test-Cluster-Upgrade-Process.md`)

Practice Test - Cluster Upgrade Process

Solutions to practice test cluster upgrade process

  • What is the current version of the cluster?

    text
    $ kubectl get nodes

  • How many nodes are part of this cluster?

    text
    $ kubctl get nodes

  • Check what nodes the pods are hosted on.

    text
    $ kubectl get pods -o wide

  • Count the number of deployments

    text
    $ kubectl get deploy

  • Run the command kubectl get pods -o wide

    text
    $ kubectl get pods -o wide

  • You are tasked to upgrade the cluster. User's accessing the applications must not be impacted. And you cannot provision new VMs. What strategy would you use to upgrade the cluster?

    text
    Upgrade one node at a time while moving the workloads to the other

  • Run the kubeadm upgrade plan command

    text
    $ kubeadm upgrade plan

  • Run the kubectl drain master --ignore-daemonsets

    text
    $ kubectl drain master --ignore-daemonsets

  • Run the command apt install kubeadm=1.18.0-00 and then kubeadm upgrade apply v1.18.0 and then apt install kubelet=1.18.0-00 to upgrade the kubelet on the master node

    text
    $ apt install kubeadm=1.18.0-00
$ kubeadm upgrade apply v1.18.0 
$ apt install kubelet=1.18.0-00

  • Run the command kubectl uncordon master

    text
    $ kubectl uncordon master

  • Run the command kubectl drain node01 --ignore-daemonsets

    text
    $ kubectl drain node01 --ignore-daemonsets

  • Run the commands: apt install kubeadm=1.18.0-00 and then kubeadm upgrade node. Finally, run apt install kubelet=1.18.0-00.

    text
    $ apt install kubeadm=1.18.0-00
$ kubeadm upgrade node
$ apt install kubelet=1.18.0-00

  • Run the command kubectl uncordon node01

    text
    $ kubectl uncordon node01

06 Cluster Maintenance 07 Backup And Restore Methods (`docs/06-Cluster-Maintenance/07-Backup-and-Restore-Methods.md`)

Backup and Restore Methods

In this section, we will take a look at backup and restore methods

Backup Candidates

Resource Configuration

  • Imperative way

  • Declarative Way (Preferred approach)

    text
    apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app: myapp
    type: front-end
spec:
  containers:
  - name: nginx-container
    image: nginx

  • A good practice is to store resource configurations on source code repositories like github.

Backup - Resource Configs

text
$ kubectl get all --all-namespaces -o yaml > all-deploy-services.yaml (only for few resource groups)
  • There are many other resource groups that must be considered. There are tools like ARK or now called Velero by Heptio that can do this for you.

Backup - ETCD

  • So, instead of backing up resources as before, you may choose to backup the ETCD cluster itself.

  • You can take a snapshot of the etcd database by using etcdctl utility snapshot save command.

    text
    $ ETCDCTL_API=3 etcdctl snapshot save snapshot.db
    text
    $  ETCDCTL_API=3 etcdctl snapshot status snapshot.db

Restore - ETCD

  • To restore etcd from the backup at later in time. First stop kube-apiserver service

    text
    $ service kube-apiserver stop

  • Run the etcdctl snapshot restore command

  • Update the etcd service

  • Reload system configs

    text
    $ systemctl daemon-reload

  • Restart etcd

    text
    $ service etcd restart

  • Start the kube-apiserver

    text
    $ service kube-apiserver start

With all etcdctl commands specify the cert,key,cacert and endpoint for authentication.

text
$ ETCDCTL_API=3 etcdctl \
  snapshot save /tmp/snapshot.db \
  --endpoints=https://[127.0.0.1]:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/etcd-server.crt \
  --key=/etc/kubernetes/pki/etcd/etcd-server.key

K8s Reference Docs

06 Cluster Maintenance 08 Working With ETCDCTL (`docs/06-Cluster-Maintenance/08-Working-With-ETCDCTL.md`)

Working with ETCDCTL

06 Cluster Maintenance 09 Practice Test Backup And Restore Methods (`docs/06-Cluster-Maintenance/09-Practice-Test-Backup-and-Restore-Methods.md`)

Practice Test - Backup and Restore Methods

Take me to Practice Test

Solutions to practice test - Backup and Restore Methods

  1. How many deployments exist in the cluster?

    text
    kubectl get deployments

  2. What is the version of ETCD running on the cluster?

    text
    kubectl describe pod -n kube-system etcd-controlplane

    Find the entry for Image

  3. At what address can you reach the ETCD cluster from the controlplane node?

    text
    kubectl describe pod -n kube-system etcd-controlplane

    Under Command find --listen-client-urls

  4. Where is the ETCD server certificate file located?

    On kubeadm clusters like this one, the default location for certificate files is /etc/kubernetes/pki/etcd

    Choose the correct certificate

  5. Where is the ETCD CA Certificate file located?

    On kubeadm clusters like this one, the default location for certificate files is /etc/kubernetes/pki/etcd

    Choose the correct certificate

  6. Take a snapshot of the ETCD database using the built-in snapshot functionality.Store the backup file at location /opt/snapshot-pre-boot.db

    text
    ETCDCTL_API=3 etcdctl snapshot save \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  /opt/snapshot-pre-boot.db

  7. Information only.

  8. Wake up! We have a conference call! After the reboot the master nodes came back online, but none of our applications are accessible. Check the status of the applications on the cluster. What's wrong?

    All of the above

  9. Luckily we took a backup. Restore the original state of the cluster using the backup file.

    1. Restore the backup to a new directory

      text
      ETCDCTL_API=3 etcdctl snapshot restore \
  --data-dir /var/lib/etcd-from-backup \
  /opt/snapshot-pre-boot.db

    2. Modify the etcd pod to use the new directory.

      To do this, we need to edit the volumes section and change the hostPath to be the directory we restored to.

      text
      vi /etc/kubernetes/manifests/etcd.yaml
      yaml
      volumes:
  - hostPath:
      path: /etc/kubernetes/pki/etcd
      type: DirectoryOrCreate
    name: etcd-certs
  - hostPath:
      path: /var/lib/etcd      # <- change this
      type: DirectoryOrCreate
    name: etcd-data

      New value: /var/lib/etcd-from-backup

      Save this and wait for up to a minute for the etcd pod to reload.

    3. Verify

      text
      kubectl get deployments
kubectl get services

See also: https://github.com/kodekloudhub/community-faq/blob/main/docs/etcd-faq.md

06 Cluster Maintenance 10 Practice Test Backup And Restore Methods 2 (`docs/06-Cluster-Maintenance/10-Practice-Test-Backup-and-Restore-Methods-2.md`)

Practice Test - Backup and Restore Methods 2

Solutions to practice test - Backup and Restore Methods 2

In this test, we practice both with stacked and external etcd clusters.

  1. Information only

  2. Explore the student-node and the clusters it has access to.

    bash
    kubectl config get-contexts

  3. How many clusters are defined in the kubeconfig on the student-node?

    bash
    kubectl config get-contexts

    2

  4. How many nodes (both controlplane and worker) are part of cluster1?

    bash
    kubectl config use-context cluster1
kubectl get nodes

    2

  5. What is the name of the controlplane node in cluster2?

    bash
    kubectl config use-context cluster2
kubectl get nodes

    cluster2-controlplane

  6. Information only

  7. How is ETCD configured for cluster1?

    bash
    kubectl config use-context cluster1
kubectl get pods -n kube-system

    From the output we can see a pod for etcd, therefore the answer is Stacked ETCD

  8. How is ETCD configured for cluster2?

    bash
    kubectl config use-context cluster2
kubectl get pods -n kube-system

    From the output we can see no pod for etcd. Since no etcd is not an option for a functioning cluster the answer must therefore be External ETCD

  9. What is the IP address of the External ETCD datastore used in cluster2?

    For this, we need to exampine the API server configuration

    bash
    kubectl config use-context cluster2
kubectl get pods -n kube-system kube-apiserver-cluster2-controlplane -o yaml | grep etcd

    From the output, locate --etcd-servers. The IP address in this line is the answer.

  10. What is the default data directory used the for ETCD datastore used in cluster1?

    For this, we need to examine the etcd manifest on the control plane node, and we need to find out the hostpath of its etcd-data volume.

    bash
    kubectl config use-context cluster1
kubectl get pods -n kube-system etcd-cluster1-controlplane -o yaml

    In the output, find the volumes section. The host path of the volume named etcd-data is the answer.

    /var/lib/etcd

  11. Information only

  12. What is the default data directory used the for ETCD datastore used in cluster2?

    For this, we need to examine the system unit file for the etcd service. Remember that for external etcd, it is running as an operating system service.

    bash
    ssh etcd-server
    bash
    # Verify the name of the service
systemctl list-unit-files | grep etcd

# Using the output from above command
systemctl cat etcd.service

    Note the comment line in the output. This tells you where the service unit file is. We are going to need to edit this file in a later question!

    From the output, locate --data-dir

    /var/lib/etcd-data

    Return to the student node:

    bash
    exit

  13. How many other nodes are part of the ETCD cluster that etcd-server is a part of?

    This question is somewhat contentious. It ought not to contain the word other. The required answer is

    1

  14. Take a backup of etcd on cluster1 and save it on the student-node at the path /opt/cluster1.db

    For this, we need to do the backup on the control node, then pull it back to the student node.

    bash
    ssh cluster1-controlplane
    bash
    ETCDCTL_API=3 etcdctl snapshot save \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/server.crt \
  --key /etc/kubernetes/pki/etcd/server.key \
  cluster1.db

# Return to student node
exit
    bash
    scp cluster1-controlplane:~/cluster1.db /opt/

  15. An ETCD backup for cluster2 is stored at /opt/cluster2.db. Use this snapshot file to carry out a restore on cluster2 to a new path /var/lib/etcd-data-new.

    As you recall, cluster2 is using external etcd. This means

    • etcd does not have to be on the control plane node of the cluster. In this case, it is not.
    • etcd runs as an operating system service not a pod, therefore there is no manifest file to edit. Changes are instead made to a service unit file.

    There are several parts to this question. Let's go through them one at a time.

    1. Move the backup to the etcd-server node

      bash
      scp /opt/cluster2.db etcd-server:~/

    2. Log into etcd-server node

      bash
      ssh etcd-server

    3. Check the ownership of the current etcd-data directory

      We will need to ensure correct ownership of our restored data. We determined the location of the data directory in Q12

      bash
      ls -ld /var/lib/etcd-data/

      Note that owner and group are both etcd

    4. Do the restore

      bash
      ETCDCTL_API=3 etcdctl snapshot restore \
    --data-dir /var/lib/etcd-data-new \
    cluster2.db

    5. Set ownership on the restored directory

      bash
      chown -R etcd:etcd /var/lib/etcd-data-new

    6. Reconfigure and restart etcd

      We will need the location of the service unit file which we also determined in Q12

      bash
      vi /etc/systemd/system/etcd.service

      Edit the --data-dir argument to the newly restored directory, and save.

      Finally, reload and restart the etcd service. Whenever you have edited a service unit file, a daemon-reload is required to reload the in-memory configuration of the systemd service.

      bash
      systemctl daemon-reload
systemctl restart etcd.service

      Return to the student node:

      bash
      exit

    7. Verify the restore

      bash
      kubectl config use-context cluster2
kubectl get all -n critical

06 Cluster Maintenance 11 Download Presentation Deck (`docs/06-Cluster-Maintenance/11-Download-Presentation-Deck.md`)

Download Presentation Deck

07 Security 01 Security Section Introduction (`docs/07-Security/01-Security-Section-Introduction.md`)

Security Section Introduction

In this section, we will take a look at security section introduction

  • Kubernetes Security Primitives
  • Authentication
  • TLS certificates for cluster components
  • Secure Persistent key-value store
  • Authorization
  • Images Security
  • Security Contexts
  • Network Policies

07 Security 02 Kubernetes Security Primitives (`docs/07-Security/02-Kubernetes-Security-Primitives.md`)

Kubernetes Security Primitives

In this section, we will take a look at kubernetes security primitives

Secure Hosts

Secure Kubernetes

  • We need to make two types of decisions.
    • Who can access?
    • What can they do?

Authentication

  • Who can access the API Server is defined by the Authentication mechanisms.

Authorization

  • Once they gain access to the cluster, what they can do is defined by authorization mechanisms.

TLS Certificates

  • All communication with the cluster, between the various components such as the ETCD Cluster, kube-controller-manager, scheduler, api server, as well as those running on the working nodes such as the kubelet and kubeproxy is secured using TLS encryption.

Network Policies

What about communication between applications within the cluster?

07 Security 03 Authentication (`docs/07-Security/03-Authentication.md`)

Authentication

In this section, we will take a look at authentication in a kubernetes cluster

Accounts

Different users that may be accessing the cluster security of end users who access the applications deployed on the cluster is managed by the applications themselves internally.

  • So, we left with 2 types of users

    • Humans, such as the Administrators and Developers
    • Robots such as other processes/services or applications that require access to the cluster.

  • All user access is managed by apiserver and all of the requests goes through apiserver.

Authentication Mechanisms

  • There are different authentication mechanisms that can be configured.

Authentication Mechanisms - Basic

kube-apiserver configuration

  • If you set up via kubeadm then update kube-apiserver.yaml manifest file with the option.

Authenticate User

  • To authenticate using the basic credentials while accessing the API server specify the username and password in a curl command.

    text
    $ curl -v -k http://master-node-ip:6443/api/v1/pods -u "user1:password123"

  • We can have additional column in the user-details.csv file to assign users to specific groups.

Note

K8s Reference Docs

07 Security 04 TLS Certificates (`docs/07-Security/04-TLS-Certificates.md`)

TLS Certificates Introduction

In this section, we will take a look at TLS certificates

  • What are TLS certificates?
  • How does kubernetes use certificates?
  • How to generate them?
  • How to configure them?
  • How to view them?
  • How to troubleshoot issues related to certificates

07 Security 05 TLS Basics (`docs/07-Security/05-TLS-Basics.md`)

TLS Basics

In this section, we will take a look at TLS Basics

Certificate

  • A certificate is used to guarantee trust between 2 parties during a transaction.
  • Example: when a user tries to access web server, tls certificates ensure that the communication between them is encrypted.

Symmetric Encryption

  • It is a secure way of encryption, but it uses the same key to encrypt and decrypt the data and the key has to be exchanged between the sender and the receiver, there is a risk of a hacker gaining access to the key and decrypting the data.

Asymmetric Encryption

  • Instead of using single key to encrypt and decrypt data, asymmetric encryption uses a pair of keys, a private key and a public key.

How do you look at a certificate and verify if it is legit?

  • who signed and issued the certificate.
  • If you generate the certificate then you will have it sign it by yourself; that is known as self-signed certificate.

How do you generate legitimate certificate? How do you get your certificates singed by someone with authority?

  • That's where Certificate Authority (CA) comes in for you. Some of the popular ones are Symantec, DigiCert, Comodo, GlobalSign etc.

Public Key Infrastructure

Certificates naming convention

07 Security 06 TLS In Kubernetes (`docs/07-Security/06-TLS-in-Kubernetes.md`)

TLS in Kubernetes

In this section, we will take a look at TLS in kubernetes

The two primary requirements are to have all the various services within the cluster to use server certificates and all clients to use client certificates to verify they are who they say they are.

  • Server Certificates for Servers
  • Client Certificates for Clients

Let's look at the different components within the k8s cluster and identify the various servers and clients and who talks to whom.

07 Security 07 TLS In Kubernetes Certificate Creation (`docs/07-Security/07-TLS-in-Kubernetes-Certificate-Creation.md`)

TLS in kubernetes - Certificate Creation

In this section, we will take a look at TLS certificate creation in kubernetes

Generate Certificates

  • There are different tools available such as easyrsa, openssl or cfssl etc. or many others for generating certificates.

Certificate Authority (CA)

  • Generate Keys
    text
    $ openssl genrsa -out ca.key 2048
  • Generate CSR
    text
    $ openssl req -new -key ca.key -subj "/CN=KUBERNETES-CA" -out ca.csr
  • Sign certificates
    text
    $ openssl x509 -req -in ca.csr -signkey ca.key -out ca.crt

Generating Client Certificates

Admin User Certificates

  • Generate Keys

    text
    $ openssl genrsa -out admin.key 2048

  • Generate CSR

    text
    $ openssl req -new -key admin.key -subj "/CN=kube-admin" -out admin.csr

  • Sign certificates

    text
    $ openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key -out admin.crt

  • Certificate with admin privilages

    text
    $ openssl req -new -key admin.key -subj "/CN=kube-admin/O=system:masters" -out admin.csr

We follow the same procedure to generate client certificate for all other components that access the kube-apiserver.

Generating Server Certificates

ETCD Server certificate

Kube-apiserver certificate

Kubectl Nodes (Server Cert)

Kubectl Nodes (Client Cert)

07 Security 08 View Certificate Details (`docs/07-Security/08-View-Certificate-Details.md`)

View Certificate Details

In this section, we will take a look how to view certificates in a kubernetes cluster.

View Certs

  • To view the details of the certificate
    text
    $ openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout

Follow the same procedure to identify information about of all the other certificates

Inspect Server Logs - Hardware setup

  • Inspect server logs using journalctl
    text
    $ journalctl -u etcd.service -l

Inspect Server Logs - kubeadm setup

  • View logs using kubectl

    text
    $ kubectl logs etcd-master

  • View logs using docker ps and docker logs

    text
    $ docker ps -a
$ docker logs <container-id>

K8s Reference Docs

07 Security 09 Certificate Health Check Spreadsheet (`docs/07-Security/09-Certificate-Health-Check-Spreadsheet.md`)

Certificate Health-Check Spreadsheet

07 Security 10 Practice Test View Certificate Details (`docs/07-Security/10-Practice-Test-View-Certificate-Details.md`)

Practice Test - View Certificates

Solutions to practice test - view certificates

  • Identify the certificate file used for the kube-api server

    text
    $ cat /etc/kubernetes/manifest/kube-apiserver.yaml

Answer: /etc/kubernetes/pki/apiserver.crt

  • Identify the Certificate file used to authenticate kube-apiserver as a client to ETCD Server

    text
    $ cat /etc/kubernetes/manifest/kube-apiserver.yaml
Answer: /etc/kubernetes/pki/apiserver-etcd-client.crt

  • Look for kubelet-client-key option in the file /etc/kubernetes/manifests/kube-apiserver.yaml

    text
    Answer: /etc/kubernetes/pki/apiserver-kubelet-client.key

  • Look for cert file option in the file /etc/kubernetes/manifests/etcd.yaml

    text
    Answer: /etc/kubernetes/pki/etcd/server.crt

  • Look for CA Certificate in file /etc/kubernetes/manifests/etcd.yaml

    text
    Answer: /etc/kubernetes/pki/etcd/ca.crt

  • Run the command openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text

    text
    $ openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text

  • Run the command openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text and look for issuer

    text
    $ openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text

  • Run the command openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text and look at Alternative Names

    text
    $ openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text

  • Run the command openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -text and look for Subject CN.

    text
    $ openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -text

  • Run the command openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text and check on the Expiry date.

    text
    $ openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text

  • Run the command 'openssl x509 -in /etc/kubernetes/pki/ca.crt -text' and look for validity

    text
    $ openssl x509 -in /etc/kubernetes/pki/ca.crt -text

  • Inspect the --cert-file option in the manifests file.

    text
    $ vi /etc/kubernetes/manifests/etcd.yaml

  • ETCD has its own CA. The right CA must be used for the ETCD-CA file in /etc/kubernetes/manifests/kube-apiserver.yaml.

    text
    View answer at /var/answers/kube-apiserver.yaml

07 Security 11 Certificate API (`docs/07-Security/11-Certificate-API.md`)

Certificate API

In this section, we will take a look at how to manage certificates and certificate API's in kubernetes

CA (Certificate Authority)

  • The CA is really just the pair of key and certificate files that we have generated, whoever gains access to these pair of files can sign any certificate for the kubernetes environment.

Kubernetes has a built-in certificates API that can do this for you.

  • With the certificate API, we now send a certificate signing request (CSR) directly to kubernetes through an API call.

This certificate can then be extracted and shared with the user.

  • A user first creates a key

    text
    $ openssl genrsa -out jane.key 2048

  • Generates a CSR

    text
    $ openssl req -new -key jane.key -subj "/CN=jane" -out jane.csr

  • Sends the request to the administrator and the adminsitrator takes the key and creates a CSR object, with kind as "CertificateSigningRequest" and a encoded "jane.csr"

    text
    apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: jane
spec:
  groups:
  - system:authenticated
  usages:
  - digital signature
  - key encipherment
  - server auth
  request:
    <certificate-goes-here>

    $ cat jane.csr |base64
    $ kubectl create -f jane.yaml

    text

  • To list the csr's

    text
    $ kubectl get csr

  • Approve the request

    text
    $ kubectl certificate approve jane

  • To view the certificate

    text
    $ kubectl get csr jane -o yaml

  • To decode it

    text
    $ echo "<certificate>" |base64 --decode

All the certificate releated operations are carried out by the controller manager.

  • If anyone has to sign the certificates they need the CA Servers, root certificate and private key. The controller manager configuration has two options where you can specify this.

K8s Reference Docs

--- METRICS ---

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →