:hammer::wrench:
A Collection of tools, hands-on walkthroughs with source code. 
| In Video descriptions |
| [Learn Kubernetes (my old series)](./kubernetes/README.md) | | [source](./kubernetes/readme.md) |
| Learn about CI/CD tools :whale: | 
| | | |
| Deploy Kubernetes to the cloud :partly_sunny: | 
| [source](./kubernetes/cloud/readme.md) |
| Monitoring Kubernetes :mag: | 
| [source](./monitoring/prometheus/kubernetes/readme.md) |
| Guide to Logging :page_with_curl: | 
| [source](./monitoring/logging/readme.md) |
| Guide to ServiceMesh :globe_with_meridians: | 
| [source](./kubernetes/servicemesh/readme.md) |
## Docker Development Basics
| Step :heavy_check_mark: | Video :movie_camera: | Source Code :octocat: |
|---|---|---|
| Working with `Dockerfiles` 
| [source](https://github.com/marcel-dempers/docker-development-youtube-series/tree/part1) |
| Working with code 
| [source](https://github.com/marcel-dempers/docker-development-youtube-series/tree/part2) |
| Docker Multistage explained | 
| [source](https://github.com/marcel-dempers/docker-development-youtube-series/tree/part3) |
| Debugging Go in Docker | 
| [source](https://github.com/marcel-dempers/docker-development-youtube-series/tree/master/golang) |
| Debugging .NET in Docker | 
| [source](https://github.com/marcel-dempers/docker-development-youtube-series/tree/part5) |
| Debugging Python in Docker | 
| [source](https://github.com/marcel-dempers/docker-development-youtube-series/tree/debugging-python) |
| Debugging NodeJS in Docker | 
| [source](https://github.com/marcel-dempers/docker-development-youtube-series/tree/master/nodejs) |
## Engineering Toolbox :hammer::wrench:
Checkout the toolbox [website](https://marceldempers.dev/toolbox)
---
### Overview (`course/README.md`)
# π The Ultimate DevOps Course (early access π§)
This course is my ultimate guide to DevOps,SRE & platform engineering for beginners
π§ This course is currently under early access and undergoing constant review and updates
In my many years working in the field, I have noticed a range of challenges and problems that people face when trying to start a DevOps, SRE, platform or cloud engineering career.
DevOps has its challenges:
Qualifications have its challenges:
Cloud certifications have its challenges:
This course provides a practical hands-on roadmap into the world of DevOps, SRE and platform engineering.
It will set you up with all the fundamentals required for successful future learning. During this course you will apply a wide range of DevOps principles and use modern technologies in a real world hands-on manner to create yourself a personal website. While building your website, we will cover topics such as:
The personal website created throughout this course will become your marketing tool for your career to demonstrate all your newly obtained above-mentioned skills and abilities. You will build the ultimate foundation and be equipped to make more informed decisions on what you want to focus on next in your career.
---
### Overview (`drone-ci/README.md`)
# Introduction to Drone CI
---
### Overview (`jenkins/readme.md`)
# Jenkins on Amazon Kubernetes
For running Jenkins on AMAZON, start [here](./amazon-eks/readme.md)
# Jenkins on Local (Docker Windows \ Minikube \ etc)
For running Jenkins on Local Docker for Windows or Minikube :hammer::wrench:
## Full playlist
## Getting Started with Kubernetes on Windows for beginners
## Kubectl
Checkout [kubectl](./kubectl.md) for detailed steps
## Deployments
Checkout [deployments](./deployments/readme.md) for detailed steps
## Configuration Management
## Secret Management
## Services
## Ingress
## StatefulSets
## Persistent Volumes
## Manage YAML
---
### Overview (`tracing/README.md`)
# Introduction to Distributed Tracing
In this episode we take a look at distributed tracing.
We'll take a look at the concept, what distributed tracing is, what problems it solves, how to emit traces and the platform architecture to collect traces.
## Example microservice architecture
First of all, we need an example application.
In this demo, I have a few microservices that work together to form a video catalog.
## A simple Web UI: videos-web
In this video guide we discuss what an AI agent is from a DevOps, SRE & Platform Engineering perspective.
Engineers will play a key role in:
* Hosting Agents in cloud, Clusters, Local, etc
* Understand traffic routing to agents
* Securing Agentic systems
In this guide we learn about:
* What an agent is
* Different forms of AI Agents
* Context engineering
* Agents.md & System Instructions
* Subagents
* Agent Skills
## Agents.md
AI Agents usually follow their own System Instructions depending on the Agent type. For example Gemini looks for `GEMINI.md` at the root of the repo. Claude looks for `CLAUDE.md`
There is a open standard for [AGENTS.md](https://agents.md/).
Our example [AGENTS.md](./AGENTS.md) can be copied to the root of this repo to demonstrate a system instruction for an agent using a tool like OpenCode.
For a deeper dive in setting up agents for each AI (Gemini,Claude,Copilot,etc), checkout the playlist [here](https://www.youtube.com/playlist?list=PLHq1uqvAteVtCsuixHwiNPQGZ1x5UVLo9)
## Subagents
Agent usually have one contenxt window.
Subagents provide capability to have dedicated agents for each specilized task with their own context windows. For example, you can have a dedicated agent for troubleshooting or debugging, vs Code Review. This means you can keep the context window of the main agent free.
Our example [subagent](./k8s_expert.md) can be copied into the subagent directory for OpenCode which is `.opencode/agents/k8s_expert/k8s_expert.md` and our OpenCode terminal will pick it up on restart.
For a deeper dive in setting up subagents for each AI (Gemini,Claude,Copilot,etc), checkout the playlist [here](https://www.youtube.com/playlist?list=PLHq1uqvAteVtCsuixHwiNPQGZ1x5UVLo9)
## Agent Skills
If you look at Agents, you will notice the markdown file may become quite large.
We have a lot of commands and scripts inside that file.
Agent Skills allow us to take this to the next level. Instead of having all the "brains" in the Agents.md file, we can break up functionality and place them in skills.
In this video, we'll discuss why skills are so important and how to use them
There is an open standard written for [Agent Skills](https://agentskills.io/home) as well.
Our example skills folder can be copied to the dedicated `.agents/skills/
## Running an MCP
Start our simple portable sandbox in a linux container.
We'll expose a port so we can showcase `HTTP` calls as well.
```shell
docker run -it --rm \
-p 8080:8080 \
ubuntu:latest bash
# install some tools we'll use:
apt-get update && apt-get install -y curl tree git jq tar
#install nodejs and python as a requirement for many MCP servers
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.4/install.sh | bash
# in lieu of restarting the shell
\. "$HOME/.nvm/nvm.sh"
nvm install 24
node -v
apt-get install -y python3 python3-pip python3-venv# Create the environment
python3 -m venv /opt/venv
source /opt/venv/bin/activate
```
### Example 1: STDIO (File & GIT MCP Demo)
Firstly, let's look at a `git` MCP server.
To do this we need a basic `git` repo, lets create one:
```shell
mkdir -p /workspace && cd /workspace
echo "# Hello MCP" > README.md
echo "console.log('hello')" > app.js
mkdir src && echo "export default {}" > src/index.js
tree
git init
git config user.email "demo@mcp.dev"
git config user.name "MCP Demo"
git add .
git commit -m "initial commit"
# leave an uncommitted change for the diff demo
echo "## Updated" >> README.md
```
#### File MCP Example:
Use the MCP in three steps:
```shell
(
echo '{"jsonrpc":"2.0","method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"1.0"}},"id":1}'
echo '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"list_directory","arguments":{"path":"/workspace"}},"id":2}'
echo '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"read_text_file","arguments":{"path":"/workspace/README.md"}},"id":3}'
sleep 2
) | npx -y @modelcontextprotocol/server-filesystem /workspace 2>/dev/null | jq -c .
```
#### GIT MCP Example:
Setup `git` MCP:
```shell
pip3 install mcp-server-git
```
Use the MCP in three steps:
```shell
(
echo '{"jsonrpc":"2.0","method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"1.0"}},"id":1}'
echo '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"git_log","arguments":{"repo_path":"/workspace","max_count":3}},"id":2}'
echo '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"git_diff_unstaged","arguments":{"repo_path":"/workspace"}},"id":3}'
sleep 2
) | python3 -u -m mcp_server_git --repository /workspace 2>/dev/null | jq -c .
```
### Example 2: HTTP (GitHub & K8s MCP Demo)
In this example, we move from stdio to HTTP which is basically just MCP hosted over a Web Endpoint.
#### Github MCP Example:
There are official MCP servers available online. The [Official GitHub MCP server](https://github.com/github/github-mcp-server) is what we will be using
Let's install it:
```shell
curl -fsSL https://github.com/github/github-mcp-server/releases/download/v1.0.5/github-mcp-server_Linux_x86_64.tar.gz \
| tar -xz -C /usr/local/bin github-mcp-server
github-mcp-server --help
```
Start it:
```shell
github-mcp-server http --port 8080
```
We need to provide authentication with Github:
```shell
export GITHUB_PERSONAL_ACCESS_TOKEN=''
```
Call it in three steps:
```shell
# step-1: initialise
curl -s -i http://localhost:8080/mcp \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-H "Authorization: Bearer ${GITHUB_PERSONAL_ACCESS_TOKEN}" \
-d '{"jsonrpc":"2.0","method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"1.0"}},"id":1}'
MCP_SESSION_ID=''
# step-2: list tools
curl -s http://localhost:8080/mcp \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-H "Authorization: Bearer ${GITHUB_PERSONAL_ACCESS_TOKEN}" \
-H "Mcp-Session-Id: ${MCP_SESSION_ID}" \
-d '{"jsonrpc":"2.0","method":"tools/list","id":2}'
# step-3: tool call
curl -s http://localhost:8080/mcp \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-H "Authorization: Bearer ${GITHUB_PERSONAL_ACCESS_TOKEN}" \
-H "Mcp-Session-Id: ${MCP_SESSION_ID}" \
-d '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"search_repositories","arguments":{"query":"kubernetes gateway-api","sort":"stars","perPage":5}},"id":3}'
```
#### Kubernetes MCP Example:
Lets create a Kubernetes cluster to play with using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/)
```shell
kind create cluster --name mcp --image kindest/node:v1.34.0
```
In this example we will use a Kubernetes MCP and setup a read-only `ServiceAccount` so it can read our cluster.
```shell
kubectl apply -f kubernetes/gateway-api/agentgateway/mcp/kubernetes/02-mcp.yaml
# check mcp server
kubectl get pods
kubectl get sa
kubectl get service
# forward ports
kubectl port-forward svc/mcp-k8s 8080
```
Call it in three steps:
```shell
# step-1: initialise
curl -s -i http://localhost:8080/mcp \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-d '{"jsonrpc":"2.0","method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"1.0"}},"id":1}'
MCP_SESSION_ID=''
# step-2: list tools
curl -s http://localhost:8080/mcp \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-H "Mcp-Session-Id: ${MCP_SESSION_ID}" \
-d '{"jsonrpc":"2.0","method":"tools/list","id":2}'
# step-3: tool call
curl -s http://localhost:8080/mcp \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-H "Mcp-Session-Id: ${MCP_SESSION_ID}" \
-d '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"namespaces_list","arguments":{}},"id":3}'
```
---
### Overview (`apache/kafka/README.md`)
# Introduction to Kafka
This guide is under the messaging section alongside other message brokers like `RabbitMQ` etc.
Checkout the guide under the [messaging/kafka](../../messaging/kafka/README.md) folder
---
### Overview (`argo/argo-cd/README.md`)
# Introduction to Argo CD
---
### Overview (`golang/introduction/readme.md`)
# Introduction to Learning Go
Go can be downloaded from [golang.org](https://golang.org/doc/install) 
Requirements:
* Kubernetes 1.21
* Kind or Minikube
For this tutorial, I will be using Kubernetes 1.21.
If you are watching the old guide for Kubernetes 1.17, go [here](..\vault\readme.md)
Lets create a Kubernetes cluster to play with using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/)
```
cd hashicorp/vault-2022
kind create cluster --name vault --image kindest/node:v1.21.1 --config kind.yaml
```
Next up, I will be running a small container where I will be doing all the work from:
You can skip this part if you already have `kubectl` and `helm` on your machine.
```
docker run -it --rm --net host -v ${HOME}/.kube/:/root/.kube/ -v ${PWD}:/work -w /work alpine sh
```
Install `kubectl`
```
apk add --no-cache curl
curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
chmod +x ./kubectl
mv ./kubectl /usr/local/bin/kubectl
```
Install `helm`
```
curl -LO https://get.helm.sh/helm-v3.7.2-linux-amd64.tar.gz
tar -C /tmp/ -zxvf helm-v3.7.2-linux-amd64.tar.gz
rm helm-v3.7.2-linux-amd64.tar.gz
mv /tmp/linux-amd64/helm /usr/local/bin/helm
chmod +x /usr/local/bin/helm
```
Now we have `helm` and `kubectl` and can access our `kind` cluster:
```
kubectl get nodes
NAME STATUS ROLES AGE VERSION
vault-control-plane Ready control-plane,master 37s v1.21.1
```
Let's add the Helm repositories, so we can access the Kubernetes manifests
```
helm repo add hashicorp https://helm.releases.hashicorp.com
```
## Storage: Consul
We will use a very basic Consul cluster for our Vault backend.
Let's find what versions of Consul are available:
```
helm search repo hashicorp/consul --versions
```
We can use chart `0.39.0` which is the latest at the time of this demo
Let's create a manifests folder and grab the YAML:
```
mkdir manifests
helm template consul hashicorp/consul \
--namespace vault \
--version 0.39.0 \
-f consul-values.yaml \
> ./manifests/consul.yaml
```
Deploy the consul services:
```
kubectl create ns vault
kubectl -n vault apply -f ./manifests/consul.yaml
kubectl -n vault get pods
```
## TLS End to End Encryption
See steps in [./tls/ssl_generate_self_signed.md](./tls/ssl_generate_self_signed.md)
You'll need to generate TLS certs (or bring your own)
Remember not to check-in your TLS to GIT :)
Create the TLS secret
```
kubectl -n vault create secret tls tls-ca \
--cert ./tls/ca.pem \
--key ./tls/ca-key.pem
kubectl -n vault create secret tls tls-server \
--cert ./tls/vault.pem \
--key ./tls/vault-key.pem
```
## Generate Kubernetes Manifests
Let's find what versions of vault are available:
```
helm search repo hashicorp/vault --versions
```
In this demo I will use the `0.19.0` chart
Let's firstly create a `values` file to customize vault.
Let's grab the manifests:
```
helm template vault hashicorp/vault \
--namespace vault \
--version 0.19.0 \
-f vault-values.yaml \
> ./manifests/vault.yaml
```
## Deployment
```
kubectl -n vault apply -f ./manifests/vault.yaml
kubectl -n vault get pods
```
## Initialising Vault
```
kubectl -n vault exec -it vault-0 -- sh
kubectl -n vault exec -it vault-1 -- sh
kubectl -n vault exec -it vault-2 -- sh
vault operator init
vault operator unseal
kubectl -n vault exec -it vault-0 -- vault status
kubectl -n vault exec -it vault-1 -- vault status
kubectl -n vault exec -it vault-2 -- vault status
```
## Web UI
Let's checkout the web UI:
```
kubectl -n vault get svc
kubectl -n vault port-forward svc/vault-ui 443:8200
```
Now we can access the web UI [here](https://localhost/)
## Enable Kubernetes Authentication
For the injector to be authorised to access vault, we need to enable K8s auth
```
kubectl -n vault exec -it vault-0 -- sh
vault login
vault auth enable kubernetes
vault write auth/kubernetes/config \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
issuer="https://kubernetes.default.svc.cluster.local"
exit
```
# Summary
So we have a vault, an injector, TLS end to end, stateful storage.
The injector can now inject secrets for pods from the vault.
Now we are ready to use the platform for different types of secrets:
## Secret Injection Guides
### Basic Secrets
Objective:
----------
* Let's create a basic secret in vault manually
* Application consumes the secret automatically
[Try it](./example-apps/basic-secret/readme.md)
---
### Overview (`hashicorp/vault/readme.md`)
# Hashicorp Vault Guide - Deprecated
# Vault
For this tutorial, I use Kubernetes 1.17
It's critical because we'll need certain [admission controllers](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/) enabled.
To get 1.17 for Linux\Windows, just use `kind` since you can create a 1.17 with admissions all setup.
```
#Windows
kind create cluster --name vault --image kindest/node:v1.17.0@sha256:9512edae126da271b66b990b6fff768fbb7cd786c7d39e86bdf55906352fdf62
#Linux
kind create cluster --name vault --kubeconfig ~/.kube/kind-vault --image kindest/node:v1.17.0@sha256:9512edae126da271b66b990b6fff768fbb7cd786c7d39e86bdf55906352fdf62
```
## TLS End to End Encryption
VIDEO: ```
## Create a cluster
Follow my Introduction to Amazon EKS for beginners guide, to create a cluster 
## Horizontal Pod Autoscaling
HPA allows us to scale pods when their resource utilisation goes over a threshold 
## Requirements
### A Cluster
* For both autoscaling guides, we'll need a cluster. 
## We need a Kubernetes cluster
Lets create a Kubernetes cluster to play with using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/)
```shell
kind create cluster --name certmanager --image kindest/node:v1.34.0
```
## Concepts
It's important to understand the various concepts and new Kubernetes resources that
## Create a cluster with Kind
```
kind create cluster --name demo --image kindest/node:v1.33.1
```
Checkout our cluster
```
kubectl get nodes
NAME STATUS ROLES AGE VERSION
demo-control-plane Ready control-plane 19m v1.33.1
```
Deploy a `ConfigMap`
```
kubectl apply -f kubernetes/configmaps/configmap.yaml
```
Deploy a pod that consumes the `ConfigMap`
```
kubectl apply -f kubernetes/configmaps/deployment.yaml
```
Checking the deployment pods
The `kubectl describe` command is essential for checking out kubernetes resources
```
kubectl describe deploy example-deploy
```
We can checkout the config contents inside the pod's container with the `kubectl exec` command to get shell
```
kubectl exec -it 
## We need a Kubernetes cluster
Lets create a Kubernetes cluster to play with using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/)
Because a Daemonset is all about running pods on every node, lets create a 3 node cluster:
```
cd kubernetes/daemonsets
kind create cluster --name daemonsets --image kindest/node:v1.20.2 --config kind.yaml
```
Test our cluster:
```
kubectl get nodes
NAME STATUS ROLES AGE VERSION
daemonsets-control-plane Ready control-plane,master 65s v1.20.2
daemonsets-worker Ready 
## Installation
Best place to start is the [documentation](https://hub.datree.io/)
I like to start all my work inside a docker container.
Let's run a small Alpine linux container
```
docker run -it -v ${PWD}:/work -v ${HOME}/.kube/:/root/.kube/ -w /work --net host alpine sh
```
### Install some dependancies
Let's install `curl` and `unzip` because the installation script uses those.
Build an example app:
```
# Important!
# make sure you are at root of the repository
# in your terminal
# you can choose which app you want to build!
# aimvector/golang:1.0.0
docker-compose build golang
# aimvector/csharp:1.0.0
docker-compose build csharp
# aimvector/nodejs:1.0.0
docker-compose build nodejs
# aimvector/python:1.0.0
docker-compose build python
```
Take a look at example [deployment yaml](./deployment.yaml)
---
### Overview (`kubernetes/fluxcd/README.md`)
# Introduction to Flux CD v2
## Create a kubernetes cluster
In this guide we we''ll need a Kubernetes cluster for testing. Let's create one using [kind](https://kind.sigs.k8s.io/)
```
kind create cluster --name fluxcd --image kindest/node:v1.26.3
```
## Run a container to work in
### run Alpine Linux:
```
docker run -it --rm -v ${HOME}:/root/ -v ${PWD}:/work -w /work --net host alpine sh
```
### install some tools
```
# install curl
apk add --no-cache curl
# install kubectl
curl -sLO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
chmod +x ./kubectl
mv ./kubectl /usr/local/bin/kubectl
```
### test cluster access:
```
/work # kubectl get nodes
NAME STATUS ROLES AGE VERSION
fluxcd-control-plane Ready control-plane 54s v1.26.3
```
## Get the Flux CLI
Let's download the `flux` command-line utility.
We can get this utility from the GitHub [Releases page](https://github.com/fluxcd/flux2/releases)
It's also worth noting that you want to ensure you get a compatible version of flux which supports your version of Kubernetes. Checkout the [prerequisites](https://fluxcd.io/flux/installation/#prerequisites) page.
```
curl -o /tmp/flux.tar.gz -sLO https://github.com/fluxcd/flux2/releases/download/v2.1.1/flux_2.1.1_linux_amd64.tar.gz
tar -C /tmp/ -zxvf /tmp/flux.tar.gz
mv /tmp/flux /usr/local/bin/flux
chmod +x /usr/local/bin/flux
```
Now we can run `flux --help` to see its installed
## Check our cluster
```
flux check --pre
```
## Documentation
As with every guide, we start with the documentation
The [Core Concepts](https://fluxcd.io/flux/concepts/) is a good place to start.
We begin by following the steps under the [bootstrap](https://fluxcd.io/flux/installation/#bootstrap) section for GitHub
We'll need to generate a [personal access token (PAT)](https://github.com/settings/tokens/new) that can create repositories by checking all permissions under `repo`.
Once we have a token, we can set it:
```
export GITHUB_TOKEN=
# 1 - Kubernetes
Get a Kubernetes Cluster. In this video, I use Docker for Windows.
If you are new to Kubernetes, checkout my videos [here](https://marceldempers.dev/videos/guides/kubernetes-getting-started)
# 2 - Flux CTL
I used Flux 1.18 which I got from the GitHub [Release Page](https://github.com/fluxcd/flux/releases/tag/1.18.0)
Rename it to `fluxctl.exe` & place it in a folder that is on your `$env:Path` environment variable.
Open a new terminal and try
```
fluxctl
```
# 4 - Installing Flux
Make sure you are pointing to the kubernetes cluster you want to use
```
kubectl config current-context
kubectl get nodes
```
```
kubectl create ns flux
$GHUSER = "marcel-dempers"
fluxctl install `
--git-user=${GHUSER} `
--git-email=${GHUSER}@users.noreply.github.com `
--git-url=git@github.com:${GHUSER}/docker-development-youtube-series `
--git-path=kubernetes/configmaps,kubernetes/secrets,kubernetes/deployments `
--git-branch=flux-test `
--namespace=flux | kubectl apply -f -
kubectl -n flux rollout status deployment/flux
$env:FLUX_FORWARD_NAMESPACE = "flux"
fluxctl list-workloads
fluxctl identity
https://github.com/marcel-dempers/docker-development-youtube-series/settings/keys/new
fluxctl sync
annotations:
fluxcd.io/tag.example-app: semver:~1.0
fluxcd.io/automated: 'true'
fluxctl policy -w default:deployment/example-deploy --tag "example-app=1.0.*"
```
---
### Overview (`kubernetes/gateway-api/README.md`)
# Introduction to Gateway API
In this video, we will dive into the features of Gateway API.
Once you finish this video, click [here]() to deep dive each of the Gateway API controllers.
[Documentation](https://gateway-api.sigs.k8s.io/)
## We need a Kubernetes cluster
Lets create a Kubernetes cluster to play with using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/)
```
kind create cluster --name gatewayapi --image kindest/node:v1.34.0
```
Test our cluster and makes sure `kubectl` is configured for it:
```
kubectl get nodes
NAME STATUS ROLES AGE VERSION
gatewayapi-control-plane Ready control-plane 40s v1.34.0
```
## Gateway API CRDs
The Kubernetes Gateway API is not installed on kubernetes by default. My guess is it may at some point.
For now, we can grab it from the [Gateway API SIGS Guide](https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api)
Important Note: At the time of recording this guide, we'd like to look at as many features as possible, hence the `experimental` install is used.
```shell
kubectl apply --server-side -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/standard-install.yaml
```
This will install (Stable Channel):
* Gateway Classes: `kubectl get gatewayclass`
* Gateways: `kubectl get gateway`
* HTTP Routes: `kubectl get httproute`
These APIs are part of the Experimental Channel:
* TLS Routes: `kubectl get tlsroute`
* TCP Routes: `kubectl get tcproute`
* UDP Routes: `kubectl get udproute`
Note: Gateway API is very new, and all of the above is subject to change quite rapidly
## Setup some example applications
The following will deploy a `deployment`, `service` and require `configMap` and `secret` for the applications to work.
```shell
# deploy example apps
kubectl apply -f python/deployment.yaml
kubectl apply -f golang/deployment.yaml
kubectl apply -f kubernetes/gateway-api/web-app.yaml
# check example apps
kubectl get pods
NAME READY STATUS RESTARTS
go-deploy-b9c69978d-529qb 1/1 Running 0
go-deploy-b9c69978d-jfb2b 1/1 Running 0
python-deploy-54cbfc948b-5ch7w 1/1 Running 0
python-deploy-54cbfc948b-qh22f 1/1 Running 0
web-app-67fbb5d844-68wq4 1/1 Running 0
kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
go-svc ClusterIP 10.96.173.22 
|
| Envoy | [Introduction to Envoy Gateway API](./envoy/README.md) | 
|
| Istio | [Introduction to Istio Gateway API](./istio/README.md) | 
|
| NGINX Fabric | [Introduction to NGINX Fabric Gateway API](./nginx-fabric/README.md) | 
|
| Cilium | [Introduction to Cilium Gateway API](./cilium/README.md) | 
|
| kgateway | [Introduction to kgateway (a.k.a Gloo Gateway)](./kgateway/README.md) | 
|
| HAProxy | [Introduction to HAProxy & Gateway API](./haproxy/README.md) | 
|
| Linkerd | [Introduction to Linkerd Gateway API](./linkerd/README.md) | 
---
### Overview (`kubernetes/helm/README.md`)
# Introduction to Helm
## We need a Kubernetes cluster
Lets create a Kubernetes cluster to play with using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/)
```
kind create cluster --name helm --image kindest/node:v1.31.1
```
# Getting Started with Helm
Firstly, I like to do most of my work in containers so everything is reproducible 
To start off this tutorial, we will be using [kind](https://kind.sigs.k8s.io/) to create our test cluster.
You can use `minikube` or any Kubernetes cluster.
Kind is an amazing tool for running test clusters locally as it runs in a container which makes it lightweight and easy to run throw-away clusters for testing purposes.
## Download KUBECTL
We can download `kubectl` from the [Official Docs](https://kubernetes.io/docs/tasks/tools/)
## Create a kubernetes cluster
In this guide we will run two clusters side by side so we can demonstrate cluster access.
Create two clusters:
```
kind create cluster --name dev --image kindest/node:v1.23.5
kind create cluster --name prod --image kindest/node:v1.23.5
```
See cluster up and running:
```
kubectl get nodes
NAME STATUS ROLES AGE VERSION
prod-control-plane Ready control-plane,master 2m12s v1.23.5
```
## Understanding the KUBECONFIG
Default location of the `kubeconfig` file is in `
```
kubectl apply -f kubernetes/kustomize/application/namespace.yaml
kubectl apply -f kubernetes/kustomize/application/configmap.yaml
kubectl apply -f kubernetes/kustomize/application/deployment.yaml
kubectl apply -f kubernetes/kustomize/application/service.yaml
# OR
kubectl apply -f kubernetes/kustomize/application/
kubectl delete ns example
```
# Kustomize
## Build
```
kubectl kustomize .\kubernetes\kustomize\ | kubectl apply -f -
# OR
kubectl apply -k .\kubernetes\kustomize\
kubectl delete ns example
```
## Overlays
```
kubectl kustomize .\kubernetes\kustomize\environments\production | kubectl apply -f -
# OR
kubectl apply -k .\kubernetes\kustomize\environments\production
kubectl delete ns example
```
---
### Overview (`kubernetes/persistentvolume/readme.md`)
# Persistent Volumes Demo
## Container Storage
By default containers store their data on the file system like any other process.
Container file system is temporary and not persistent during container restarts
When container is recreated, so is the file system
```
# run postgres
docker run -d --rm -e POSTGRES_DB=postgresdb -e POSTGRES_USER=admin -e POSTGRES_PASSWORD=admin123 postgres:10.4
# enter the container
docker exec -it 
Start here ππ½[https://www.portainer.io/](https://www.portainer.io/)
Documentation ππ½[https://docs.portainer.io/](https://docs.portainer.io/)
## Portainer installation
In this demo, I will be running Kubernetes 1.22 using `kind`
Which is compatible with portainer 2.11.1
Let's go ahead with a local docker install:
```
cd kubernetes\portainer
mkdir volume-ce
docker run -d -p 9443:9443 -p 8000:8000 --name portainer-ce `
--restart=always `
-v /var/run/docker.sock:/var/run/docker.sock `
-v ${PWD}/volume-ce:/data `
portainer/portainer-ce:2.11.1
```
## SSL & DOMAIN
We can also upload SSL certificates for our portainer.
In this demo, portainer will issue self signed certificates.
We will need a domain for our portainer server so our clusters can contact it.
Let's use [nip.io](https://nip.io/) to create a public endpoint for portainer.
## Create Kubernetes Cluster
Let's start by creating a local `kind` [cluster](https://kind.sigs.k8s.io/)
For local clusters, we can use the public endpoint Agent.
We can get a public endpoint for the portainer agent by:
* Ingress
* LoadBalancer
* NodePort
So we'll deploy portainer agent with `NodePort` for local
For production environments, I would recommend not to expose the portainer agent.
In this case, for Production, we'll use the portainer edge agent.
To get `NodePort` exposed in `kind`, we'll open a host port with a [kind.yaml](./kind.yaml) config
```
kind create cluster --name local --config kind.yaml
```
## Manage Kubernetes Environments
The portainer UI gives us a one line command to deploy the portainer agent.
Note that in the video, we pick the `node port` option.
## Local: Portainer Agent
I download the YAML from [here](https://downloads.portainer.io/portainer-agent-ce211-k8s-nodeport.yaml) to take a closer look at what it is deploying
Deploy the portainer agent in my `kind` cluster:
```
kubectl apply -f portainer-agent-ce211-k8s-nodeport.yaml
```
See the agent:
```
kubectl -n portainer get pods
```
See the service with the endpoint it exposes:
```
kubectl -n portainer get svc
```
Now since we dont have a public load balancer and using nodeport, our service will be exposed on the node IP.
Since the Kubernetes node is our local machine, we should be able to access the portainer agent on `
This guide follows the general instructions of running a [manual rancher install](https://rancher.com/docs/rancher/v2.5/en/quick-start-guide/deployment/quickstart-manual-setup/) and running our own infrastructure on Hyper-v
# Hyper-V : Prepare our infrastructure
In this demo, we will use Hyper-V to create our infrastructure.
For on-premise, many companies use either Hyper-V, VMWare Vsphere and other technologies to create virtual infrastructure on bare metal.
Few points to note here:
* Benefit of Virtual infrastructure is that it's immutable
a) We can add and throw away virtual machines at will.
b) This makes maintenance easier as we can roll updated virtual machines instead of
patching existing machines and turning them to long-living snowflakes.
c) Reduce lifespan of machines
* Bare Metal provides the compute.
a) We don't want Kubernetes directly on bare metal as we want machines to be immutable.
b) This goes back to the previous point on immutability.
* Every virtual machine needs to be able to reach each other on the network
a) This is a kubernetes networking requirements that all nodes can communicate with one another
# Hyper-V : Create our network
In order for us to create virtual machines all on the same network, I am going to create a virtual switch in Hyper-v
Open Powershell in administrator
```
# get our network adapter where all virtual machines will run on
# grab the name we want to use
Get-NetAdapter
Import-Module Hyper-V
$ethernet = Get-NetAdapter -Name "Ethernet 2"
New-VMSwitch -Name "virtual-network" -NetAdapterName $ethernet.Name -AllowManagementOS $true -Notes "shared virtual network interface"
```
# Hyper-V : Create our machines
We firstly need harddrives for every VM.
Let's create three:
```
mkdir c:\temp\vms\linux-0\
mkdir c:\temp\vms\linux-1\
mkdir c:\temp\vms\linux-2\
New-VHD -Path c:\temp\vms\linux-0\linux-0.vhdx -SizeBytes 20GB
New-VHD -Path c:\temp\vms\linux-1\linux-1.vhdx -SizeBytes 20GB
New-VHD -Path c:\temp\vms\linux-2\linux-2.vhdx -SizeBytes 20GB
```
```
New-VM `
-Name "linux-0" `
-Generation 1 `
-MemoryStartupBytes 2048MB `
-SwitchName "virtual-network" `
-VHDPath "c:\temp\vms\linux-0\linux-0.vhdx" `
-Path "c:\temp\vms\linux-0\"
New-VM `
-Name "linux-1" `
-Generation 1 `
-MemoryStartupBytes 2048MB `
-SwitchName "virtual-network" `
-VHDPath "c:\temp\vms\linux-1\linux-1.vhdx" `
-Path "c:\temp\vms\linux-1\"
New-VM `
-Name "linux-2" `
-Generation 1 `
-MemoryStartupBytes 2048MB `
-SwitchName "virtual-network" `
-VHDPath "c:\temp\vms\linux-2\linux-2.vhdx" `
-Path "c:\temp\vms\linux-2\"
```
Setup a DVD drive that holds the `iso` file for Ubuntu Server
```
Set-VMDvdDrive -VMName "linux-0" -ControllerNumber 1 -Path "C:\temp\ubuntu-20.04.3-live-server-amd64.iso"
Set-VMDvdDrive -VMName "linux-1" -ControllerNumber 1 -Path "C:\temp\ubuntu-20.04.3-live-server-amd64.iso"
Set-VMDvdDrive -VMName "linux-2" -ControllerNumber 1 -Path "C:\temp\ubuntu-20.04.3-live-server-amd64.iso"
```
Start our VM's
```
Start-VM -Name "linux-0"
Start-VM -Name "linux-1"
Start-VM -Name "linux-2"
```
Now we can open up Hyper-v Manager and see our infrastructure.
In this video we'll connect to each server, and run through the initial ubuntu setup.
Once finished, select the option to reboot and once it starts, you will notice an `unmount` error on CD-Rom
This is ok, just shut down the server and start it up again.
# Hyper-V : Setup SSH for our machines
Now in this demo, because I need to copy rancher bootstrap commands to each VM, it would be easier to do so
using SSH. So let's connect to each VM in Hyper-V and setup SSH.
This is because `copy+paste` does not work without `Enhanced Session` mode in Ubuntu Server.
Let's temporarily turn on SSH on each server:
```
sudo apt update
sudo apt install -y nano net-tools openssh-server
sudo systemctl enable ssh
sudo ufw allow ssh
sudo systemctl start ssh
```
Record the IP address of each VM so we can SSH to it:
```
sudo ifconfig
# record eth0
linux-0 IP=192.168.0.16
linux-1 IP=192.168.0.17
linux-2 IP=192.168.0.18
```
In new Powershell windows, let's SSH to our VMs
```
ssh linux-0@192.168.0.16
ssh linux-1@192.168.0.17
ssh linux-2@192.168.0.18
```
# Setup Docker
It is required that every machine that needs to join our cluster, has docker running on it.
Firstly, rancher will use docker to run it's agent as well as bootstrap the cluster.
Install docker on each VM:
```
curl -sSL https://get.docker.com/ | sh
sudo usermod -aG docker $(whoami)
sudo service docker start
```
# Running Rancher in Docker
So Rancher can be [deployed](https://rancher.com/docs/rancher/v2.5/en/quick-start-guide/deployment/) almost anywhere.
We can run it in Kubernetes on-prem or the cloud.
Now because we want Rancher to manage kubernetes clusters, we dont want it running in the clusters we are managing.
So I would like to keep my Rancher server outside and separate from my Kubernetes clusters.
So let's setup a single server with [docker](https://rancher.com/docs/rancher/v2.5/en/quick-start-guide/deployment/quickstart-manual-setup/)
## Persist data
We will want to persist ranchers data across reboots.
Rancher stores its data under `/var/lib/rancher`
In this repo, let's create a space to persist data:
```
cd kubernetes/rancher
mkdir volume
```
## Run Rancher
```
docker run -d --name rancher-server -v ${PWD}/volume:/var/lib/rancher --restart=unless-stopped -p 80:80 -p 443:443 --privileged rancher/rancher
```
## Unlock Rancher
Once its up and running we can extract the Rancher initial boostrap password from the logs
```
docker logs rancher-server > rancher.log
```
## Get Rancher IP
It's important that our servers can reach the Rancher server.
As all the VMs and my machine are on the same network, we can use my machine IP as the server IP so the VM's can reach it.
let's grab the IP:
```
ipconfig
```
We can now access Rancher on [localhost](https://localhost)
## Deploy Sample Workloads
To deploy some sample basic workloads, let's get the `kubeconfig` for our cluster
Set kubeconfig:
```
$ENV:KUBECONFIG="
## Create Kubernetes cluster
```
kind create cluster --name rbac --image kindest/node:v1.20.2
```
## Kubernetes CA Certificate
Kubernetes does not have a concept of users, instead it relies on certificates and would only
trust certificates signed by its own CA.
To get the CA certificates for our cluster, easiest way is to access the master node.
Because we run on `kind`, our master node is a docker container.
The CA certificates exists in the `/etc/kubernetes/pki` folder by default.
If you are using `minikube` you may find it under `~/.minikube/.`
Access the master node:
```
docker exec -it rbac-control-plane bash
ls -l /etc/kubernetes/pki
total 60
-rw-r--r-- 1 root root 1135 Sep 10 01:38 apiserver-etcd-client.crt
-rw------- 1 root root 1675 Sep 10 01:38 apiserver-etcd-client.key
-rw-r--r-- 1 root root 1143 Sep 10 01:38 apiserver-kubelet-client.crt
-rw------- 1 root root 1679 Sep 10 01:38 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1306 Sep 10 01:38 apiserver.crt
-rw------- 1 root root 1675 Sep 10 01:38 apiserver.key
-rw-r--r-- 1 root root 1066 Sep 10 01:38 ca.crt
-rw------- 1 root root 1675 Sep 10 01:38 ca.key
drwxr-xr-x 2 root root 4096 Sep 10 01:38 etcd
-rw-r--r-- 1 root root 1078 Sep 10 01:38 front-proxy-ca.crt
-rw------- 1 root root 1679 Sep 10 01:38 front-proxy-ca.key
-rw-r--r-- 1 root root 1103 Sep 10 01:38 front-proxy-client.crt
-rw------- 1 root root 1675 Sep 10 01:38 front-proxy-client.key
-rw------- 1 root root 1679 Sep 10 01:38 sa.key
-rw------- 1 root root 451 Sep 10 01:38 sa.pub
exit the container
```
Copy the certs out of our master node:
```
cd kubernetes/rbac
docker cp rbac-control-plane:/etc/kubernetes/pki/ca.crt ca.crt
docker cp rbac-control-plane:/etc/kubernetes/pki/ca.key ca.key
```
# Kubernetes Users
As mentioned before, Kubernetes has no concept of users, it trusts certificates that is signed by its CA. This allows managed Kubernetes offerings to use their cloud logins to authenticate.
So on Azure, I can use my Microsoft account, GKE my Google account and AWS EKS my Amazon account. You will need to consult your cloud provider to setup authentication. Example [Azure AKS](https://docs.microsoft.com/en-us/azure/aks/azure-ad-integration-cli) ## User Certificates First thing we need to do is create a certificate signed by our Kubernetes CA. We have the CA, let's make a certificate. Easy way to create a cert is use `openssl` and the easiest way to get `openssl` is to simply run a container: ``` docker run -it -v ${PWD}:/work -w /work -v ${HOME}:/root/ --net host alpine sh apk add openssl ``` Let's create a certificate for Bob Smith: ``` #start with a private key openssl genrsa -out bob.key 2048 ``` Now we have a key, we need a certificate signing request (CSR). We also need to specify the groups that Bob belongs to. Let's pretend Bob is part of the `Shopping` team and will be developing applications for the `Shopping` ``` openssl req -new -key bob.key -out bob.csr -subj "/CN=Bob Smith/O=Shopping" ``` Use the CA to generate our certificate by signing our CSR. We may set an expiry on our certificate as well ``` openssl x509 -req -in bob.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out bob.crt -days 1 ``` ## Building a kube config Let's install `kubectl` in our container to make things easier: ``` apk add curl nano curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl chmod +x ./kubectl mv ./kubectl /usr/local/bin/kubectl ``` We'll be trying to avoid messing with our current kubernetes config. So lets tell `kubectl` to look at a new config that does not yet exists ``` export KUBECONFIG=~/.kube/new-config ``` Create a cluster entry which points to the cluster and contains the details of the CA certificate: ``` kubectl config set-cluster dev-cluster --server=https://127.0.0.1:52807 \ --certificate-authority=ca.crt \ --embed-certs=true #see changes nano ~/.kube/new-config ``` kubectl config set-credentials bob --client-certificate=bob.crt --client-key=bob.key --embed-certs=true kubectl config set-context dev --cluster=dev-cluster --namespace=shopping --user=bob kubectl config use-context dev kubectl get pods Error from server (Forbidden): pods is forbidden: User "Bob Smith" cannot list resource "pods" in API group "" in the namespace "shopping" ## Give Bob Smith Access ``` cd kubernetes/rbac kubectl create ns shopping kubectl -n shopping apply -f .\role.yaml kubectl -n shopping apply -f .\rolebinding.yaml ``` ## Test Access as Bob kubectl get pods No resources found in shopping namespace. # Kubernetes Service Accounts So we've covered users, but what about applications or services running in our cluster ? Most business apps will not need to connect to the kubernetes API unless you are building something that integrates with your cluster, like a CI/CD tool, an autoscaler or a custom webhook. Generally applications will use a service account to connect. You can read more about [Kubernetes Service Accounts](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/). Let's deploy a service account ``` kubectl -n shopping apply -f serviceaccount.yaml ``` Now we can deploy a pod that uses the service account ``` kubectl -n shopping apply -f pod.yaml ``` Now we can test the access from within that pod by trying to list pods: ``` kubectl -n shopping exec -it shopping-api -- bash # Point to the internal API server hostname APISERVER=https://kubernetes.default.svc # Path to ServiceAccount token SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount # Read this Pod's namespace NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace) # Read the ServiceAccount bearer token TOKEN=$(cat ${SERVICEACCOUNT}/token) # Reference the internal certificate authority (CA) CACERT=${SERVICEACCOUNT}/ca.crt # List pods through the API curl --cacert ${CACERT} --header "Authorization: Bearer $TOKEN" -s ${APISERVER}/api/v1/namespaces/shopping/pods/ # we should see an error not having access ``` Now we can allow this pod to list pods in the shopping namespace ``` kubectl -n shopping apply -f serviceaccount-role.yaml kubectl -n shopping apply -f serviceaccount-rolebinding.yaml ``` If we try run `curl` command again we can see now we are able to get a json response with pod information --- ### Overview (`kubernetes/reloader/README.md`) # Introduction to K8s Reloader In this demo we will walk through a project called [reloader](https://github.com/stakater/Reloader) ## Create a cluster with Kind ``` kind create cluster --name demo --image kindest/node:v1.33.1 ``` Checkout our cluster ``` kubectl get nodes NAME STATUS ROLES AGE VERSION demo-control-plane Ready control-plane 19m v1.33.1 ``` For this demo, we'll want to deploy a `ConfigMap` and/or a `Secret` and consume these using a `Deployment` Deploy a `ConfigMap` ``` kubectl apply -f kubernetes/configmaps/configmap.yaml ``` Deploy a `Secret` ``` kubectl apply -f kubernetes/secrets/secret.yaml ``` ## Install Reloader ``` helm repo add stakater https://stakater.github.io/stakater-charts helm repo update # see versions helm search repo stakater/reloader -l #pick a version CHART_VERSION="2.1.5" helm install reloader stakater/reloader \ --create-namespace \ --namespace reloader \ --version $CHART_VERSION \ --values kubernetes/reloader/values.yaml ``` ### Usage Deploy an application that consumes the 'ConfigMap' and 'Secret' with auto reload enabled: ``` kubectl apply -f kubernetes/reloader/deployment.yaml ``` We can now update the `ConfigMap` object and observe our `example-app` pods restarting when the `ConfigMap` changes. We can also run `kubectl logs -l app=example-app` in a separate terminal and observe the changing values inside pods. --- ### Overview (`kubernetes/secrets/README.md`) # Introduction to Kubernetes: Secrets
## Create a cluster with Kind
```
kind create cluster --name secrets --image kindest/node:v1.31.1
```
## Our Secret
We have a secret under `kubernetes/secrets/secret.json`
```
cat kubernetes/secrets/secret.json
```
## Using our secret in a container
As a file:
```
docker run -it -v $PWD/kubernetes/secrets/secret.json:/secrets/secret.json ubuntu:latest bash
cat /secrets/secret.json
```
As environment variables:
```
api_key="somesecretgoeshere"
docker run -it -e API_KEY=$api_key ubuntu:latest bash
echo $API_KEY
```
## Kubernetes Secret
Read more about [Kubernetes secrets](https://kubernetes.io/docs/concepts/configuration/secret/)
## Create our secret
There are two main ways we can create a Kubernetes secret.
Either by creating the secret object with `kubectl create secret` or apply\create it declaratively using YAML with `kubectl apply -f`
`kubectl create secret`:
```
kubectl create secret generic mysecret --from-file kubernetes/secrets/secret.json
```
`kubectl apply -f` or `kubectl create -f` allows us to define things declaratively using YAML files:
```
kubectl apply -f kubernetes/secrets/secret.yaml
```
## Use our secret
In order to use our secret we add a `volume` to our pod spec and then mount that using a `volumeMount`
We can also use a secret references as `env` variable
```
kubectl apply -f kubernetes/secrets/pod.yaml
```
---
### Overview (`kubernetes/servicemesh/readme.md`)
# Introduction to Service Mesh
To understand service mesh, we need a good use case.
# Service Mesh Guides
## Introduction to Linkerd
Getting started with Linkerd
Read Me: [readme](./linkerd/README.md)
Video :point_down: 
## Introduction to Istio
Getting started with Istio
Read Me: [readme](./istio/README.md)
Video :point_down: 
In order to understand service monitors, we will need to understand how to monitor
kubernetes environment.
You will need a base understanding of Kubernetes and have a basic understanding of the `kube-prometheus` monitoring stack.
Checkout the video [How to monitor Kubernetes in 2022](https://youtu.be/YDtuwlNTzRc):
## Create a kubernetes cluster
```
# create cluster
kind create cluster --name monitoring --image kindest/node:v1.23.5
# see cluster up and running
kubectl get nodes
NAME STATUS ROLES AGE VERSION
monitoring-control-plane Ready control-plane,master 2m12s v1.23.5
```
## Deploy kube-prometheus
Installation:
```
kubectl create -f ./monitoring/prometheus/kubernetes/1.23/manifests/setup/
kubectl create -f ./monitoring/prometheus/kubernetes/1.23/manifests/
```
Check the install:
```
kubectl -n monitoring get pods
```
After a few minutes, everything should be up and running:
```
kubectl -n monitoring get pods
NAME READY STATUS RESTARTS AGE
alertmanager-main-0 2/2 Running 0 3m10s
alertmanager-main-1 2/2 Running 0 3m10s
alertmanager-main-2 2/2 Running 0 3m10s
blackbox-exporter-6b79c4588b-t4czf 3/3 Running 0 4m7s
grafana-7fd69887fb-zm2d2 1/1 Running 0 4m7s
kube-state-metrics-55f67795cd-f7frb 3/3 Running 0 4m6s
node-exporter-xjdtn 2/2 Running 0 4m6s
prometheus-adapter-85664b6b74-bvmnj 1/1 Running 0 4m6s
prometheus-adapter-85664b6b74-mcgbz 1/1 Running 0 4m6s
prometheus-k8s-0 2/2 Running 0 3m9s
prometheus-k8s-1 2/2 Running 0 3m9s
prometheus-operator-6dc9f66cb7-z98nj 2/2 Running 0 4m6s
```
## View dashboards
```
kubectl -n monitoring port-forward svc/grafana 3000
```
Then access Grafana on [localhost:3000](http://localhost:3000)
## Access Prometheus
```
kubectl -n monitoring port-forward svc/prometheus-operated 9090
```
Then access Prometheus on [localhost:9090](http://localhost:9090).
## Create our own Prometheus
```
kubectl apply -n monitoring -f ./kubernetes/servicemonitors/prometheus.yaml
```
View our prometheus `prometheus-applications-0` instance:
```
kubectl -n monitoring get pods
```
Checkout our prometheus UI
```
kubectl -n monitoring port-forward prometheus-applications-0 9090
```
## Deploy a service monitor for example app
```
kubectl -n default apply -f ./kubernetes/servicemonitors/servicemonitor.yaml
```
After applying the service monitor, if Prometheus is correctly selecting it, we should see the item appear under the [Service Discovery](http://localhost:9090/service-discovery) page in Prometheus.
Double check with with `port-forward` before proceeding.
If it does not appear, that means your Prometheus instance is not selecting the service monitor accordingly. Either a label mismatch on the namespace or the service monitor.
## Deploy our example app
```
kubectl -n default apply -f ./kubernetes/servicemonitors/example-app/
```
Now we should see a target in the Prometheus [Targets](http://localhost:9090/targets) page.
---
### Overview (`kubernetes/services/README.md`)
# Introduction to Kubernetes: Services
## Create a Kubernetes cluster
Firstly, we will need a Kubernetes cluster and will create one using [kind](https://kind.sigs.k8s.io/)
```
kind create cluster --name services --image kindest/node:v1.31.1
```
Test the cluster
```
kubectl get nodes
NAME STATUS ROLES AGE VERSION
services-control-plane Ready control-plane 5m48s v1.31.1
```
## Deploy a few pods
Let's deploy a few pods using a Kubernetes Deployment. To understand services, we need to deploy some pods that become our "upstream" or "endpoint" that we want to access.
```
kubectl apply -f kubernetes/deployments/deployment.yaml
```
## Deploy a service
We can now deploy a Kubernetes service that targets our deployment:
```
kubectl apply -f kubernetes/services/service.yaml
```
---
### Overview (`kubernetes/shipa/README.md`)
# Introduction to Shipa
## We need a Kubernetes cluster
To get the most out of Shipa, I'll be using real Cloud Provider Kubernetes as well as a local 
|
|Amazon | EKS |[Source Code](../cloud/amazon/getting-started.md) | 
|
|Google | GKE |[Source Code](../cloud/google/getting-started.md) | 
|
|Digital Ocean | DO |[Source Code](../cloud/digitalocean/getting-started.md) | 
|
|Linode | LKE |[Source Code](../cloud/linode/getting-started.md) | 
|
## Deploy Shipa to Azure AKS
Let's add Shipa to each one of our clusters running in the Cloud.
In the video I will deploy Shipa to an Azure AKS cluster:
```
# ensure we are pointing to AKS Or the
kubectl get nodes
# deploy Shipa to AKS
cd kubernetes/shipa
# create username + password override
cat > ops/values.override.yaml << EOF
auth:
adminUser: prod-aks-admin@shipa.io
adminPassword: shipa-prod-aks2020
EOF
kubectl create ns shipa-system
kubectl apply -n shipa-system -f limits.yaml
helm install shipa ./installs/shipa-helm-chart-1.1.1 --timeout=1000s --namespace=shipa-system -f ops/values.override.yaml
#wait until the dashboard pod is up!
kubectl --namespace=shipa-system get pods
# get the ingress IP for the Shipa API
kubectl --namespace=shipa-system get svc shipa-ingress-nginx -o jsonpath="{.status.loadBalancer.ingress[0].ip}"
# add the ingress IP as our Shipa target for AKS dev
shipa target-add dev-aks 20.53.128.244 -s
shipa target-set dev-aks
shipa target-list
# lets login, view the node, and dashboard app URL
shipa login prod-aks-admin@shipa.io
shipa node-list
shipa app-list
#add a host file entry for the dashboard, I.E:
20.193.20.106 dashboard.20.193.20.106.shipa.cloud
# create a team for developers
shipa team-create dev-team
#allow dev-team to deploy to AKS pool
shipa pool-constraint-set theonepool team dev-team --append
```
## MultiCloud - Add Shipa to Amazon EKS
```
# add service account to EKS cluster to allow Shipa to connect
kubectl apply -f ops/shipa-admin-service-account.yaml
#add a pool allow people to deploy to Amazon
shipa pool-add amazon --public --kube-namespace dev-team --provisioner kubernetes
# allow dev-team to deploy to Amazon
shipa pool-constraint-set amazon team dev-team --append
# We will need the EKS token of the service account to allow connection
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep shipa-admin | awk '{print $1}')
# We will also need the CA cert of Kubernetes
kubectl get secret $(kubectl get secret | grep default-token | awk '{print $1}') -o jsonpath='{.data.ca\.crt}' | base64 -d
```
On the dashboard, we add the cluster using the above information.
## We need a Kubernetes cluster
Lets create a Kubernetes cluster to play with using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/)
```
kind create cluster --name velero --image kindest/node:v1.19.1
```
## Get a container to work in
Official [Docs](https://kafka.apache.org/)
## Building a Docker file
As always, we start with a `dockerfile`
We can build our `dockerfile`
```
cd .\messaging\kafka\
docker build . -t aimvector/kafka:2.7.0
```
## Exploring the Kafka Install
We can then run it to explore the contents:
```
docker run --rm --name kafka -it aimvector/kafka:2.7.0 bash
ls -l /kafka/bin/
cat /kafka/config/server.properties
```
We can use the `docker cp` command to copy the file out of our container:
```
docker cp kafka:/kafka/config/server.properties ./server.properties
docker cp kafka:/kafka/config/zookeeper.properties ./zookeeper.properties
```
Note: We'll need the Kafka configuration to tune our server and Kafka also requires
at least one Zookeeper instance in order to function. To achieve high availability, we'll run
multiple kafka as well as multiple zookeeper instances in the future
# Zookeeper
Let's build a Zookeeper image. The Apache folks have made it easy to start a Zookeeper instance the same way as the Kafka instance by simply running the `start-zookeeper.sh` script.
```
cd ./zookeeper
docker build . -t aimvector/zookeeper:2.7.0
cd ..
```
Let's create a kafka network and run 1 zookeeper instance
```
docker network create kafka
docker run -d `
--rm `
--name zookeeper-1 `
--net kafka `
-v ${PWD}/config/zookeeper-1/zookeeper.properties:/kafka/config/zookeeper.properties `
aimvector/zookeeper:2.7.0
docker logs zookeeper-1
```
# Kafka - 1
```
docker run -d `
--rm `
--name kafka-1 `
--net kafka `
-v ${PWD}/config/kafka-1/server.properties:/kafka/config/server.properties `
aimvector/kafka:2.7.0
docker logs kafka-1
```
# Kafka - 2
```
docker run -d `
--rm `
--name kafka-2 `
--net kafka `
-v ${PWD}/config/kafka-2/server.properties:/kafka/config/server.properties `
aimvector/kafka:2.7.0
```
# Kafka - 3
```
docker run -d `
--rm `
--name kafka-3 `
--net kafka `
-v ${PWD}/config/kafka-3/server.properties:/kafka/config/server.properties `
aimvector/kafka:2.7.0
```
# Topic
Let's create a Topic that allows us to store `Order` information.
To create a topic, Kafka and Zookeeper have scripts with the installer that allows us to do so.
Access the container:
```
docker exec -it zookeeper-1 bash
```
Create the Topic:
```
/kafka/bin/kafka-topics.sh \
--create \
--zookeeper zookeeper-1:2181 \
--replication-factor 1 \
--partitions 3 \
--topic Orders
```
Describe our Topic:
```
/kafka/bin/kafka-topics.sh \
--describe \
--topic Orders \
--zookeeper zookeeper-1:2181
```
# Simple Producer & Consumer
The Kafka installation also ships with a script that allows us to produce
and consume messages to our Kafka network: 
Docker image over [here](https://hub.docker.com/_/rabbitmq)
```
# run a standalone instance
docker network create rabbits
docker run -d --rm --net rabbits --hostname rabbit-1 --name rabbit-1 rabbitmq:3.8
# how to grab existing erlang cookie
docker exec -it rabbit-1 cat /var/lib/rabbitmq/.erlang.cookie
# clean up
docker rm -f rabbit-1
```
# Management
```
docker run -d --rm --net rabbits -p 8080:15672 -e RABBITMQ_ERLANG_COOKIE=DSHEVCXBBETJJVJWTOWT --hostname rabbit-manager --name rabbit-manager rabbitmq:3.8-management
#join the manager
docker exec -it rabbit-manager rabbitmqctl stop_app
docker exec -it rabbit-manager rabbitmqctl reset
docker exec -it rabbit-manager rabbitmqctl join_cluster rabbit@rabbit-1
docker exec -it rabbit-manager rabbitmqctl start_app
docker exec -it rabbit-manager rabbitmqctl cluster_status
```
# Enable Statistics
```
docker exec -it rabbit-1 rabbitmq-plugins enable rabbitmq_management
docker exec -it rabbit-2 rabbitmq-plugins enable rabbitmq_management
docker exec -it rabbit-3 rabbitmq-plugins enable rabbitmq_management
```
# Message Publisher
```
cd messaging\rabbitmq\applications\publisher
docker build . -t aimvector/rabbitmq-publisher:v1.0.0
docker run -it --rm --net rabbits -e RABBIT_HOST=rabbit-1 -e RABBIT_PORT=5672 -e RABBIT_USERNAME=guest -e RABBIT_PASSWORD=guest -p 80:80 aimvector/rabbitmq-publisher:v1.0.0
```
# Message Consumer
```
docker build . -t aimvector/rabbitmq-consumer:v1.0.0
docker run -it --rm --net rabbits -e RABBIT_HOST=rabbit-1 -e RABBIT_PORT=5672 -e RABBIT_USERNAME=guest -e RABBIT_PASSWORD=guest aimvector/rabbitmq-consumer:v1.0.0
```
# Clustering
https://www.rabbitmq.com/cluster-formation.html
## Note
Remember we will need the Erlang Cookie to allow instances to authenticate with each other.
# Manual Clustering
```
docker exec -it rabbit-1 rabbitmqctl cluster_status
#join node 2
docker exec -it rabbit-2 rabbitmqctl stop_app
docker exec -it rabbit-2 rabbitmqctl reset
docker exec -it rabbit-2 rabbitmqctl join_cluster rabbit@rabbit-1
docker exec -it rabbit-2 rabbitmqctl start_app
docker exec -it rabbit-2 rabbitmqctl cluster_status
#join node 3
docker exec -it rabbit-3 rabbitmqctl stop_app
docker exec -it rabbit-3 rabbitmqctl reset
docker exec -it rabbit-3 rabbitmqctl join_cluster rabbit@rabbit-1
docker exec -it rabbit-3 rabbitmqctl start_app
docker exec -it rabbit-3 rabbitmqctl cluster_status
```
# Automated Clustering
```
docker run -d --rm --net rabbits `
-v ${PWD}/config/rabbit-1/:/config/ `
-e RABBITMQ_CONFIG_FILE=/config/rabbitmq `
-e RABBITMQ_ERLANG_COOKIE=WIWVHCDTCIUAWANLMQAW `
--hostname rabbit-1 `
--name rabbit-1 `
-p 8081:15672 `
rabbitmq:3.8-management
docker run -d --rm --net rabbits `
-v ${PWD}/config/rabbit-2/:/config/ `
-e RABBITMQ_CONFIG_FILE=/config/rabbitmq `
-e RABBITMQ_ERLANG_COOKIE=WIWVHCDTCIUAWANLMQAW `
--hostname rabbit-2 `
--name rabbit-2 `
-p 8082:15672 `
rabbitmq:3.8-management
docker run -d --rm --net rabbits `
-v ${PWD}/config/rabbit-3/:/config/ `
-e RABBITMQ_CONFIG_FILE=/config/rabbitmq `
-e RABBITMQ_ERLANG_COOKIE=WIWVHCDTCIUAWANLMQAW `
--hostname rabbit-3 `
--name rabbit-3 `
-p 8083:15672 `
rabbitmq:3.8-management
#NODE 1 : MANAGEMENT http://localhost:8081
#NODE 2 : MANAGEMENT http://localhost:8082
#NODE 3 : MANAGEMENT http://localhost:8083
# enable federation plugin
docker exec -it rabbit-1 rabbitmq-plugins enable rabbitmq_federation
docker exec -it rabbit-2 rabbitmq-plugins enable rabbitmq_federation
docker exec -it rabbit-3 rabbitmq-plugins enable rabbitmq_federation
```
# Basic Queue Mirroring
```
docker exec -it rabbit-1 bash
# https://www.rabbitmq.com/ha.html#mirroring-arguments
rabbitmqctl set_policy ha-fed \
".*" '{"federation-upstream-set":"all", "ha-mode":"nodes", "ha-params":["rabbit@rabbit-1","rabbit@rabbit-2","rabbit@rabbit-3"]}' \
--priority 1 \
--apply-to queues
```
# Automatic Synchronization
https://www.rabbitmq.com/ha.html#unsynchronised-mirrors
```
rabbitmqctl set_policy ha-fed \
".*" '{"federation-upstream-set":"all", "ha-sync-mode":"automatic", "ha-mode":"nodes", "ha-params":["rabbit@rabbit-1","rabbit@rabbit-2","rabbit@rabbit-3"]}' \
--priority 1 \
--apply-to queues
```
# Further Reading
https://www.rabbitmq.com/ha.html
# Clean up
```
docker rm -f rabbit-1
docker rm -f rabbit-2
docker rm -f rabbit-3
```
# RabbitMQ on Kubernetes
Checkout the Kubernetes walkthrough [here](./kubernetes/readme.md)
---
### Overview (`monitoring/fluentbit/README.md`)
# Introduction to FluentBit
We always start with the [official documentation](https://docs.fluentbit.io/manual)
## Running fluentbit
```
# starts fluentbit with default config
docker run -it cr.fluentbit.io/fluent/fluent-bit:4.0
# see HELP text
docker run -it cr.fluentbit.io/fluent/fluent-bit:4.0 -h
```
## Configure fluentbit
We've created a new [fluent-bit.yaml](./fluent-bit.yaml) with the example content from the documentation.
We can update our `docker` command to mount the config and tell `fluent-bit` to use the new configuration file we created
```shell
docker run -it -p 2020:2020 \
-v ./monitoring/fluentbit/fluent-bit.yaml:/fluent-bit/etc/fluent-bit.yaml \
cr.fluentbit.io/fluent/fluent-bit:4.0 --config=/fluent-bit/etc/fluent-bit.yaml
```
## Inputs
The example config file has a `random` input.
Let's create two inputs, one reading log files and another accepting HTTP inputs
```yaml
pipeline:
inputs:
- name: tail
path: /var/lib/docker/containers/*/*.log
multiline.parser: docker
- name: http
listen: 0.0.0.0
port: 8888
```
We'll need to add a volume mount so our fluent-bit container has access to the container logs, so let's mount `/var/lib/docker/`
We also need port `8888` to match our input plugin port
```shell
docker run -it \
-p 2020:2020 \
-p 8888:8888 \
-v ./monitoring/fluentbit/fluent-bit.yaml:/fluent-bit/etc/fluent-bit.yaml \
-v /var/lib/docker:/var/lib/docker \
cr.fluentbit.io/fluent/fluent-bit:4.0 --config=/fluent-bit/etc/fluent-bit.yaml
```
Test our HTTP input plugin using `curl`
```shell
curl -d '{"namespace":"product","message":"hello from product"}' -X POST -H "content-type: application/json" http://localhost:8888/product.log
```
## Outputs
In the example config file we had a `stdout` output plugin that writes all data from the input plugin to the terminal.
Let's create another output plugin that writes data to file so we can inspect it:
```yaml
- name: file
match: '*'
template: '{log}'
path: /fluent-bit/.data
```
Let's add a volume mount so we can see the output data.
Let's mount a local `.data` folder to `/fluent-bit/.data`
```shell
docker run -it \
-p 2020:2020 \
-p 8888:8888 \
-v ./monitoring/fluentbit/fluent-bit.yaml:/fluent-bit/etc/fluent-bit.yaml \
-v /var/lib/docker:/var/lib/docker \
-v ./monitoring/fluentbit/.data:/fluent-bit/.data \
cr.fluentbit.io/fluent/fluent-bit:4.0 --config=/fluent-bit/etc/fluent-bit.yaml
```
## Concepts
There are key [concepts](https://docs.fluentbit.io/manual/concepts/key-concepts#events-or-records) to learn about fluent-bit, now that we have events being ingested.
### Filter plugin
This plugin only includes requests from "product" namespace
```yaml
filters:
- name: grep
match: '*'
regex: namespace product
```
This plugin excludes all requests from "search" namespace
```yaml
filters:
- name: grep
match: '*'
exclude: namespace search
```
Test our HTTP input plugin using `curl` to send different events to our fluent-bit service:
```shell
#request allowed
curl -d '{"namespace":"product","message":"hello from product"}' -X POST -H "content-type: application/json" http://localhost:8888/product.log
# request dropped\filtered
curl -d '{"namespace":"search","message":"hello from search"}' -X POST -H "content-type: application/json" http://localhost:8888/search.log
```
---
### Overview (`monitoring/logging/readme.md`)
# Logging
## Logging Basics
* Standardised Logging
* Centralised Logging
[Check it out](./fluentd/basic-demo/readme.md)
## Introduction to Fluentd
* What is fluentd
* Configuration
* Plugins
* Demos
[Check if out](./fluentd/introduction/readme.md)
## Introduction to OpenTelemetry
* What is OpenTelemetry
* Configuration
* Demos
---
### Overview (`monitoring/opentelemetry/README.md`)
# OpenTelemetry
## Logs
## Tracing
[Guide here](./tracing/README.md)
## Metrics
[Guide here](./metrics/README.md)
---
### Overview (`monitoring/prometheus/readme.md`)
# Prometheus Application Monitoring
## a Video reference guide
To run any of the commands, please ensure you open a terminal and navigate to the path where this readme is located.
## Start Prometheus, Grafana & Dashboards
```
docker compose up -d prometheus
docker compose up -d grafana
docker compose up -d grafana-dashboards
```
## Start the example app you prefer
```
docker compose up -d --build go-application
docker compose up -d --build python-application
docker compose up -d --build dotnet-application
docker compose up -d --build nodejs-application
```
## Generate some requests by opening the application in the browser
```
http://localhost:80 #Golang
http://localhost:81 #Python
http://localhost:82 #Dotnet
http://localhost:83 #NodeJS
```
## Check Dashboards
```
http://localhost:3000
```
## Prometheus Queries
### Golang Examples
Requests per Second over 2minutes
```
irate(go_request_operations_total[2m])
```
Request duration
```
rate(go_request_duration_seconds_sum[2m]) / rate(go_request_duration_seconds_total[2m])
```
# Prometheus Guide on Kubernetes
Checkout the prometheus guide [here](./kubernetes/readme.md)
---
### Overview (`python/introduction/README.md`)
# Introduction to Learning Python
Best place to start is the documentation [www.python.org](https:#www.python.org/) 
## Docker
Docker image over [here](https://hub.docker.com/_/redis)
## Running redis
```
docker network create redis
docker run -it --rm --name redis --net redis -p 6379:6379 redis:6.0-alpine
```
## Configuration
Redis configuration documentation [here](https://redis.io/topics/config)
Starting Redis with a custom config
```
cd .\storage\redis\
docker run -it --rm --name redis --net redis -v ${PWD}/config:/etc/redis/ redis:6.0-alpine redis-server /etc/redis/redis.conf
```
## Security
Redis should not be exposed to public.
Always use a strong password in `redis.conf`
```
requirepass SuperSecretSecureStrongPass
```
## Persistence
Redis Persistence Documentation [here](https://redis.io/topics/persistence)
```
docker volume create redis
cd .\storage\redis\
docker run -it --rm --name redis --net redis -v ${PWD}/config:/etc/redis/ -v redis:/data/ redis:6.0-alpine redis-server /etc/redis/redis.conf
```
## Redis client application
An example application that reads a key from Redis, increments it and writes it back to Redis.
```
cd .\storage\redis\applications\client\
# start go dev environment
docker run -it -v ${PWD}:/go/src -w /go/src --net redis -p 80:80 golang:1.14-alpine
go build client.go
# start the app
./client
# build the container
docker build . -t aimvector/redis-client:v1.0.0
```
Run our application
```
cd .\storage\redis\applications\client\
docker build . -t aimvector/redis-client:v1.0.0
docker run -it --net redis `
-e REDIS_HOST=redis `
-e REDIS_PORT=6379 `
-e REDIS_PASSWORD="SuperSecretSecureStrongPass" `
-p 80:80 `
aimvector/redis-client:v1.0.0
```
## Redis Replication and High Availability
Lets move on to the [clustering](./clustering/readme.md) secion.
---
### Overview (`ai/agents/openclaw/README.md`)
# Introduction to OpenClaw
In this video, we will dive into the features of OpenClaw.
[Documentation](https://openclaw.ai/)
## Source Code
```shell
git clone --depth 1 https://github.com/openclaw/openclaw.git /tmp/openclaw
```
## Docker
View docker setup scripts:
```shell
tree /tmp/openclaw/scripts/docker/
/tmp/openclaw/scripts/docker/
βββ cleanup-smoke
β βββ Dockerfile
β βββ run.sh
βββ install-sh-common
β βββ cli-verify.sh
β βββ version-parse.sh
βββ install-sh-e2e
β βββ Dockerfile
β βββ run.sh
βββ install-sh-nonroot
β βββ Dockerfile
β βββ run.sh
βββ install-sh-smoke
β βββ Dockerfile
β βββ run.sh
βββ sandbox
β βββ Dockerfile
β βββ Dockerfile.browser
β βββ Dockerfile.common
βββ setup.sh
βββ shared-image-artifact.sh
```
After cloning the code, we can run setup according to the docs.
Use a pre-built image to avoid having to build it ourselves:
```shell
export OPENCLAW_IMAGE="ghcr.io/openclaw/openclaw:latest"
./scripts/docker/setup.sh
```
Build our own custom [dockerfile](./dockerfile):
```shell
docker build . -t aimvector/openclaw
```
### Least privileges
Linux containers run with a default set of capabilities, which are fine-grained privileges that are subsets of what `root` can do. `--cap-drop ALL` removes every single one, leaving the container with zero elevated privileges.
| Capability | Allows |
|---|---|
| `CAP_NET_BIND_SERVICE` | Bind to ports < 1024 |
| `CAP_NET_RAW` | Raw sockets, packet sniffing |
| `CAP_SYS_ADMIN` | Mount filesystems and many other things |
| `CAP_CHOWN` | Change file ownership |
| `CAP_KILL` | Send signals to any process |
In our dockerfile, we also run our container as a non-root unprivileged user.
### Secrets (environment)
We set our secrets in environment variables and inject those into the container :
```shell
export OPENCLAW_GATEWAY_TOKEN="$(openssl rand -hex 32)"
export GEMINI_API_KEY="xxxxxxx"
export DISCORD_BOT_TOKEN="xxxxxxx"
export GITHUB_TOKEN="xxxxxxx"
```
Run our container:
```shell
mkdir ~/.openclaw
cp ai/agents/openclaw/openclaw.json ~/.openclaw/
docker run -it --rm \
--name openclaw \
-e OPENCLAW_GATEWAY_TOKEN=${OPENCLAW_GATEWAY_TOKEN} \
-e GEMINI_API_KEY=${GEMINI_API_KEY} \
-e DISCORD_BOT_TOKEN=${DISCORD_BOT_TOKEN} \
-e GITHUB_TOKEN=${GITHUB_TOKEN} \
--cap-drop ALL \
-v $PWD:/work \
-v ~/.openclaw:/home/openclaw/.openclaw \
-w /work \
--net host \
-v /var/run/docker.sock:/var/run/docker.sock \
aimvector/openclaw
```
### Configration
If we want to generate a new `openclaw.json`, we can run the onboarding process by overriding the entrypoint to bash
We can tweak our `docker run` to add `--entrypoint bash` to get into the container and run the `openclaw onboard` process to get a configuration.
We can run the onboard manually to generate an `openclaw.json` configuration file:
```shell
openclaw onboard --classic
```
## Kubernetes
View docker setup scripts:
```shell
tree /tmp/openclaw/scripts/k8s/
/tmp/openclaw/scripts/k8s/
βββ create-kind.sh
βββ deploy.sh
βββ manifests
βββ configmap.yaml
βββ deployment.yaml
βββ kustomization.yaml
βββ pvc.yaml
βββ service.yaml
1 directories, 7 files
```
Lets create a Kubernetes cluster to play with using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/)
```
kind create cluster --name openclaw --image kindest/node:v1.36.1
```
Test our cluster and makes sure `kubectl` is configured for it:
```
kubectl get nodes
NAME STATUS ROLES AGE VERSION
openclaw-control-plane Ready control-plane 2m30s v1.36.1
```
View kubernetes deployment scripts:
```shell
tree /tmp/openclaw/scripts/k8s/
/tmp/openclaw/scripts/k8s/
βββ create-kind.sh
βββ deploy.sh
βββ manifests
βββ configmap.yaml
βββ deployment.yaml
βββ kustomization.yaml
βββ pvc.yaml
βββ service.yaml
2 directories, 7 files
```
I've taken a copy of these manifests by using `cp`.
```shell
cp /tmp/openclaw/scripts/k8s/manifests/* ai/agents/openclaw/kubernetes/
```
You can now use Helm, Kuztomize, ArgoCD, Flux or any mechanism of your choice to manage this deployment.
## Deploy Secret
We can refer to the `deployment.yaml` and we see the secret references under `secretKeyRef`
We will need a secret for our Gateway UI, a secret for our model provider as well as our channel.
```shell
export OPENCLAW_GATEWAY_TOKEN="$(openssl rand -hex 32)"
export GEMINI_API_KEY="xxxxxxx"
export DISCORD_BOT_TOKEN="xxxxxxx"
```
I'll be modifying the `deployment.yaml` to add my channel token `DISCORD_BOT_TOKEN`.
Note that we can reference this ENV variable in our `openclaw.json` configuration.
```yaml
- name: DISCORD_BOT_TOKEN
valueFrom:
secretKeyRef:
name: openclaw-secrets
key: DISCORD_BOT_TOKEN
optional: false
```
Create the secret:
```shell
kubectl create ns openclaw
kubectl create secret generic openclaw-secrets -n openclaw \
--from-literal OPENCLAW_GATEWAY_TOKEN=${OPENCLAW_GATEWAY_TOKEN} \
--from-literal GEMINI_API_KEY=${GEMINI_API_KEY} \
--from-literal DISCORD_BOT_TOKEN=${DISCORD_BOT_TOKEN}
```
## Configure OpenClaw
We've configured our openclaw instance in our [Configmap.yaml](./kubernetes/configmap.yaml)
## Deploy Openclaw
Apply the manifests
```shell
kubectl -n openclaw apply -f ai/agents/openclaw/kubernetes/
```
## Install Plugins
```shell
export NPM_CONFIG_CACHE=~/.openclaw/.npm
openclaw plugins install @openclaw/discord
kubectl -n openclaw delete pods --all
```
---
### Overview (`ai/kubernetes/llm-kube/README.md`)
# Introduction to LLMKube
Website: [LLMKube](https://llmkube.com)
## What is LLMKube?
LLMKube is a Kubernetes operator for self-hosted LLM inference.
Source Code: [defilantech/LLMKube](https://github.com/defilantech/LLMKube)
Built on [llama.cpp](https://github.com/ggml-org/llama.cpp).
## We need a Kubernetes cluster
Lets create a Kubernetes cluster to play with using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/)
```
kind create cluster --name llmkube --image kindest/node:v1.34.3
```
Test our cluster and makes sure `kubectl` is configured for it:
```
kubectl get nodes
NAME STATUS ROLES AGE VERSION
llmkube-control-plane Ready control-plane 40s v1.34.3
```
## Installing LLMKube with Helm
```shell
helm repo add llmkube https://defilantech.github.io/LLMKube
helm repo update
helm search repo llmkube --versions
CHART_VERSION="0.7.5"
helm install llmkube llmkube/llmkube \
--namespace llmkube-system \
--version ${CHART_VERSION} \
--create-namespace
```
Check installation:
```shell
kubectl get pods -n llmkube-system
```
Two new CRDs `Model` and `InferenceService` are now available in the cluster:
```shell
kubectl get crds | grep llmkube
inferenceservices.inference.llmkube.dev
models.inference.llmkube.dev
```
## The CRDs
### Model
```yaml
# Deploy the Model
kubectl apply -f ai/kubernetes/llm-kube/model.yaml
# Check the Model
kubectl get model
kubectl describe model
```
### InferenceService
```yaml
# Deploy the InferenceService
kubectl apply -f ai/kubernetes/llm-kube/inference.yaml
# Check the InferenceService
kubectl get inferenceservices
kubectl describe inferenceservices
```
## Testing the Inference API
Port-forward the service to reach it from your machine:
```shell
kubectl port-forward svc/gemma-e2b-service 8080:8080
```
Send a test request to the OpenAI-compatible endpoint:
```shell
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gemma-e2b",
"messages": [
{"role": "user", "content": "What is Kubernetes in one sentence?"}
],
"max_tokens": 80
}'
```
## OpenAI-Compatible Integration
The example below connects [Opencode](https://opencode.ai) to the local service
```json
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"llmkube": {
"npm": "@ai-sdk/openai-compatible",
"name": "LLMKube (local)",
"options": {
"baseURL": "http://localhost:8080/v1"
},
"models": {
"gemma-e2b": {
"name": "Gemma 4 E2B (kind)",
"limit": {
"context": 32768,
"output": 4096
}
}
}
}
}
}
```
---
### Overview (`ai/models/llama-cpp/README.md`)
# Introduction to llama.cpp
In this guide, we will learn how to run large language models locally using llama.cpp
We will cover:
* What is llama.cpp
* What is quantization
* How to find and run models
* How to host a local model server
* Interact with models using CLI like OpenCode.
## What is llama.cpp
[llama.cpp](https://llama-cpp.com/) is a lightweight, high-performance implementation designed to run large language models locally on your own machine.
It has no external dependencies and no Python β just pure C++.
Key highlights:
* Runs entirely on your own hardware β no cloud, no API keys required
* Supports CPU, NVIDIA GPU (CUDA), AMD GPU (ROCm), Intel GPU (SYCL), and Vulkan
* Natively supports quantized models (Q2 through Q8) for memory efficiency
* Ships with a CLI (`llama-cli`) and a local HTTP server (`llama-server`)
## Quantization
* AI models are massive
* a 7B parameter model at full precision (F32) needs ~28GB of RAM
* a 31B parameter model needs a really good top of the range GPU like a 4090
* Quantization compresses the model's numbers (e.g. 32-bit down to 4-bit), cutting memory by up to 8x with minimal quality loss
* A 7B model can run in as little as 4GB of RAM on a laptop CPU
* llama.cpp natively supports Q2 through Q8, letting you tune the speed, memory, and quality trade-off to your hardware
Quantization levels:
| Quant | Size | Quality | Use when |
|---|---|---|---|
| `Q2_K` | Smallest | Lowest | Very limited RAM |
| `Q3_K_M` | Very small | Low | Tight on RAM |
| `Q4_K_M` | Small | Good | **Best balance β most popular** |
| `Q5_K_M` | Medium | Better | Have spare RAM |
| `Q6_K` | Large | Very good | Plenty of RAM |
| `Q8_0` | Largest | Near lossless | Maximum quality |
| `F16` | Huge | Lossless | Rarely practical |
## Where to Get Models
[Hugging Face](https://huggingface.co/models) is the primary hub for AI models β think of it as the Docker Hub of models.
We can start with [Gemma 4](https://huggingface.co/collections/ggml-org/gemma-4)
## Source Code
llama.cpp is fully open source and available on [GitHub](https://github.com/ggml-org/llama.cpp).
Pre-built binary releases for all platforms are available at [github.com/ggml-org/llama.cpp/releases](https://github.com/ggml-org/llama.cpp/releases).
## Docker Images
The easiest way to get started is via the official Docker images.
[Official Docker Guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/docker.md)
Three image types are available:
* `light`: includes CLI and completion tools
* `full`: includes light + tools for model conversion (safetensors β GGUF)
* `server`: only the `llama-server` executable
### Choosing the right image
| Image | Backend | Vendor |
|---|---|---|
| `ghcr.io/ggml-org/llama.cpp:full` | CPU | Any |
| `ghcr.io/ggml-org/llama.cpp:full-cuda` | CUDA 12 + cuBLAS | **NVIDIA** |
| `ghcr.io/ggml-org/llama.cpp:full-cuda13` | CUDA 13 + cuBLAS | **NVIDIA** |
| `ghcr.io/ggml-org/llama.cpp:full-rocm` | ROCm + hipBLAS | **AMD** |
| `ghcr.io/ggml-org/llama.cpp:full-intel` | SYCL | **Intel** Arc/Xe |
| `ghcr.io/ggml-org/llama.cpp:full-vulkan` | Vulkan | Cross-vendor |
| `ghcr.io/ggml-org/llama.cpp:full-openvino` | OpenVINO | **Intel** CPU/iGPU/NPU |
## Running llama.cpp
Example 1: Linux (CPU or integrated graphics):
Using `--device /dev/dri` for integrated Graphics
```shell
docker run -it \
-v ~/models:/models \
--device /dev/dri \
-p 8080:8080 \
--entrypoint bash ghcr.io/ggml-org/llama.cpp:full-intel
```
Example 2:
WSL with NVIDIA GPU:
```shell
docker run -it \
-v ~/models:/models \
--gpus all \
-p 8080:8080 \
--entrypoint bash ghcr.io/ggml-org/llama.cpp:full-cuda
```
Once inside the container, verify the CLI works with `./llama-cli --help`
## Running a Model
### Download and run directly from Hugging Face
The `-hf` flag lets you pull a model directly from Hugging Face by repo and quant:
```shell
./llama-cli -hf ggml-org/gemma-4-E4B-it-GGUF:Q4_K_M
```
Note: This downloads and caches the model under `~/.cache/huggingface/` inside the container, not your `/models` mount. To persist models across container restarts, download them manually to `/models` first.
### Run from a local GGUF file
```shell
./llama-cli -m /models/gemma-4-E4B-it-Q4_K_M.gguf -ngl 99
```
Use `-ngl 0` to disable GPU offload and run on CPU only:
```shell
./llama-cli -m /models/gemma-4-E4B-it-Q4_K_M.gguf -ngl 0
```
## Hosting a Model
To expose your model as an HTTP API, use `llama-server` on port `8080`.
Once running, you can access the built-in web UI and API at [http://localhost:8080](http://localhost:8080).
Linux (CPU):
```shell
export LLAMA_ARG_HOST=0.0.0.0
./llama-server -m /models/gemma-4-E4B-it-Q4_K_M.gguf \
--port 8080 \
-ngl 0 \
--jinja \
-c 8192 \
--parallel 1 \
--temperature 1.0 \
--top-p 0.95 \
--top-k 64
```
WSL with NVIDIA GPU:
```shell
export LLAMA_ARG_HOST=0.0.0.0
# gemma-4-31B-it-Q4_K_M.gguf
# gemma-4-26B-A4B-it-Q4_K_M.gguf
# gemma-4-E2B-it-Q8_0.gguf
# gemma-4-E4B-it-Q4_K_M.gguf
./llama-server -m /models/gemma-4-E2B-it-Q8_0.gguf \
--port 8080 \
-ngl 99 \
--jinja \
-c 131072 \
--parallel 1 \
--temperature 1.0 \
--top-p 0.95 \
--top-k 64
```
Key flags:
| Flag | Purpose |
|---|---|
| `-ngl 99` | Offload all layers to GPU (llama.cpp uses as many as fit in VRAM) |
| `-c 65536` | Context window size in tokens |
| `--parallel 1` | 1 concurrent request slot β correct for single-user use |
| `--jinja` | Enables Jinja2 chat templates for proper instruction-tuned model formatting |
| `--temperature` | Default sampling temperature (0.0 = deterministic, 1.0 = creative) |
## Model Interaction via OpenCode
You can interact with your locally hosted model using [OpenCode](https://opencode.ai/docs/providers/#llamacpp).
OpenCode connects to the `llama-server` endpoint over its OpenAI-compatible API.
Configure `~/.opencode/opencode.json`:
```json
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"llama.cpp": {
"npm": "@ai-sdk/openai-compatible",
"name": "llama-server (local)",
"options": {
"baseURL": "http://localhost:8080/v1"
},
"models": {
"gemma-4-26B-A4B-it-Q4_K_M.gguf": {
"name": "gemma-4-26B-A4B-it-Q4_K_M (local)",
"limit": {
"context": 65536,
"output": 16384
}
}
}
}
}
}
```
---
### Overview (`ai/openai/introduction/README.md`)
# Introduction to Open AI
## Overview
What is [Open AI](https://openai.com/) ?
* Research company on AI development
* Builds and provides models
* Builds and provides a standard protocol for using AI
What is a model ?
I see a model as a language super database.
Instead of writing a query, that is slow to query a traditional database like SQL, you can throw a question at a model and it gives you an answer really fast
Model examples:
* GPT 3.5
* GPT 4
## Getting started
The best way to get started and to understand OpenAI, is to learn hands on
* Create an OpenAI account [here](https://openai.com/)
## Chat GPT
Here you can find the link to [ChatGPT](https://chat.openai.com/)
## Open AI Playground
Here you can find the link to the [OpenAI Playground](https://platform.openai.com/playground)
## Build an AI powered app
We can start with a `main.py` that reads a message
```
import sys
message = sys.argv[0]
```
Then we will need the code from the Open AI playground and add it to our `main.py`.
Move the `import` statements to the top
Once you have tidied up everything, you can get the response message from the AI:
```
responseMessage = response.choices[0].message.content
```
Let's build our app
```
cd ai\openai\introduction
docker build . -t ai-app
```
Set my OpenAI API key
```
$ENV:OPENAI_API_KEY=""
```
Run our AI App:
```
docker run -it -e OPENAI_API_KEY=$ENV:OPENAI_API_KEY ai-app
```
When we run the app, notice it has no concept of memory.
The playground works because it keeps track of all the user and AI messages and keeps appending new messages to it
So it can track the conversation.
Let's keep track of messages, by writing it to a local file
We will also take the system message out and keep it as a constant in our code
Full example:
```
import sys
import os
import json
import openai
openai.api_key = os.getenv("OPENAI_API_KEY")
#read the incoming message
message = sys.argv[1]
user_message = {
"role" : "user",
"content" : message
}
systemMessage = {
"role": "system",
"content": "You are a kubernetes exper that can assist developers with troubleshooting deployments\n\nTo help the developer you will need to know the namespaces as well as the pod name. Ask for missing information\n\nGenerate a command to help the developer surface logs or information\n"
}
# read the cached user messages if there are any
userMessages = []
if os.path.isfile("messages.json"):
with open('messages.json', newline='') as messagesFile:
data = messagesFile.read()
userMessages = json.loads(data)
# add the new message to it and update the cached messages
userMessages.append(user_message)
with open('messages.json', 'w', newline='') as messagesFile:
msgJSON = json.dumps(userMessages)
messagesFile.write(msgJSON)
print(msgJSON)
messages = []
messages.append(systemMessage)
messages.extend(userMessages)
response = openai.ChatCompletion.create(
model="gpt-3.5-turbo",
messages=messages,
temperature=1,
max_tokens=256,
top_p=1,
frequency_penalty=0,
presence_penalty=0
)
responseMessage = response.choices[0].message.content
print(responseMessage)
```
Now we can mount our volume so we persist the cache of messages
```
docker run -it -e OPENAI_API_KEY=$ENV:OPENAI_API_KEY -v ${PWD}:/app ai-app "can you help me with my deployment?"
Of course! I'd be happy to help with your deployment. Could you please provide me with the namespace and the name of the pod you're encountering issues with?
docker run -it -e OPENAI_API_KEY=$ENV:OPENAI_API_KEY -v ${PWD}:/app ai-app "my pod is pod-123"
Sure, I can help you with your deployment. Can you please provide me with the namespace in which the pod is running?
docker run -it -e OPENAI_API_KEY=$ENV:OPENAI_API_KEY -v ${PWD}:/app ai-app "its in the products namespace"
Great! To surface the logs for the pod "pod-123" in the "products" namespace, you can use the following command:
```shell
kubectl logs -n products pod-123
```
This command will retrieve the logs for the specified pod in the given namespace. Make sure you have the necessary permissions to access the namespace.
```
---
### Overview (`automation/cicd/octopus-deploy/README.md`)
# Introduction to Octopus Deploy
Start here ππ½[https://octopus.com/](https://octopus.com/)
Documentation ππ½[https://octopus.com/docs](https://octopus.com/docs)
In this video, I sign up with the Free version of Octopus deploy to get my license key.
## Create Kubernetes clusters
In my video I use `kind` to act as my DEV and PROD environment where I run microservices
Let's create that:
```shell
kind create cluster --name dev --image kindest/node:v1.34.0
kind create cluster --name prod --image kindest/node:v1.34.0
```
## Octopus Components & Installation
In this video, I will explore the architecture of Octopus and the required components, so I'll be using the self-hosted option for learning.
I highly recommend you checkout the Cloud version.
My settings:
```
cd automation/cicd/octopus-deploy
export MASTER_KEY=$(openssl rand -base64 32)
export OCTOPUS_SERVER_BASE64_LICENSE=xxxxx
export ADMIN_USERNAME="octo-admin"
export ADMIN_PASSWORD="octo-admin123"
export ADMIN_EMAIL="octo-admin@admin.com"
```
Spin up a database for Octopus to use:
```
docker run -d --name octopus-database \
--net kind \
-u root \
-e ACCEPT_EULA="Y" \
-e MSSQL_SA_PASSWORD="OurStrongPassword123" \
-v ${PWD}/.data/mssql/:/var/opt/mssql/data \
mcr.microsoft.com/mssql/server:2022-latest
```
Create our first Octopus server:
[notes: ports to expose](https://octopus.com/docs/installation/octopus-server-linux-container#exposed-container-ports)
```
docker run -d \
--name octopus \
--net kind \
-p 8080:8080 \
-p 8443:8443 \
-p 10943:10943 \
-e ACCEPT_EULA="Y" \
-e ADMIN_USERNAME=$ADMIN_USERNAME \
-e ADMIN_PASSWORD=$ADMIN_PASSWORD \
-e ADMIN_EMAIL=$ADMIN_EMAIL \
-e OCTOPUS_SERVER_BASE64_LICENSE=$OCTOPUS_SERVER_BASE64_LICENSE \
-e MASTER_KEY=$MASTER_KEY \
-e DB_CONNECTION_STRING="Server=octopus-database,1433;Database=OctopusDeploy;User=sa;Password=OurStrongPassword123" \
-v ${PWD}/.data/octopus/repository/:/repository \
-v ${PWD}/.data/octopus/artifacts/:/artifacts \
-v ${PWD}/.data/octopus/taskLogs/:/taskLogs \
-v ${PWD}/.data/octopus/cache/:/cache \
-v ${PWD}/.data/octopus/import/:/import \
-v ${PWD}/.data/octopus/eventExports/:/eventExports \
octopusdeploy/octopusdeploy:2025.3
```
## Access Octopus Dashboard
Let's get the IP of my machine so we can access Octopus:
```
ifconfig eth0 | grep 'inet' | awk '{print $2}' | cut -d: -f2
```
We can access the dashboard on `http://In this chapter we will cover the fundamental tools you need to know to start a DevOps, SRE or platform engineering career from the beginning.
This starts with knowing how to work with files. Source control allows us to work with files, collaborate with others and track changes and histrory of source code.
We will also take a look at popular tools that software engineers use to work with files.
What your will learn:
GIT is the foundation of DevOps, Programming & Software Engineering This is because it is a source code distributed version control system. In DevOps, you will deal with code, command line, documentation, scripts, automation, infrastructure-as-code and more. Generally, all of this involves writing files and storing them. GIT is the system you use to store these files.
In this module we'll take a read through GIT to understand where it started and why it is so successful and widely used in software engineering.
When working on source code using GIT, you will want to work on files and directories. This includes configuration files, code files, scripts and so on. To get a nice overview of all our files and folders, as well as have access to GIT and command line, we'll use an IDE. An IDE is an integrated development environment. It is software that helps us deal with software development and engineering.
Storing source code in a Git repository on a local machine carries a risk of data loss. If your harddrive crashes, or you lose your laptop, all your work would be lost. Therefore it's always recommended to use a hosted source control provider like Github, BitBucket, Gitlab and others.
It also makes collaboration possible, when you have a team of people working on the same project
In this Chapter, we will cover the fundamentals around servers and what role they play in I.T. Servers make up the entire cloud and it's what's responsible for all compute.
We'll cover the different types of servers, technologies used to create and provision servers, as well as the skills and basic fundamental knowledge required to manage servers.
This includes the Operating system running on the server, the role of the Kernel and the different type of operating system resources that we will need to know about.
What your will learn:
This the previous chapters, we have spent quite a bit of time understanding operating systems and command line and why it's so important. Modern cloud and IT tends to hide or abstract a lot of how servers and operating systems function as well as how they are monitored. This means engineers often lack the knowledge and expertise on how to dismantle a system and monitor it with bare essential command line tools.
In this chapter we'll take a look at monitoring concepts and introduce a few more command line tools that help us monitor our server.
### π¬ [Module 1: Introduction to Linux monitoring](../../content/operating-systems/linux/monitoring/README.md)Web servers are an importand foundation of the IT ecosystem. The internet is fundamentally made up of web servers. In this chapter we'll learn about the basics of web servers and get our first web server up and running.
### π¬ [Module 1: Introduction to Web](../../content/web/README.md)DevOps engineers are often tasked to troubleshoot and assist developers who may have Web applications deployed. Perhaps there are network failures, resources may not be correctly loading or incorrectly deployed.
In this chapter we will learn about the basics of HTML and web assets, how they render and how to use developer tools to troubleshoot networking and other issues with Web applications. It's a high level module designed to give you a glimpse in the shoes of a UI or front-end developer
During this entire course we will touch on a wide spectrum of DevOps tooling, concepts and technology. The outcome of this course is to have real-world hands-on experience with all these topics, all the way from setup, scripting, coding to building, CI/CD and deployment including monitoring and observability.
In previous chapters we've setup our own web server and website.
### π¬ [Module 1: Introduction to CI/CD](../../content/automation/cicd/introduction/README.md)
# Introduction to GitHub Actions: Self hosted runners
## Create a kubernetes cluster
In this guide we we''ll need a Kubernetes cluster for testing. Let's create one using [kind](https://kind.sigs.k8s.io/)
```
kind create cluster --name githubactions --image kindest/node:v1.28.0@sha256:b7a4cad12c197af3ba43202d3efe03246b3f0793f162afb40a33c923952d5b31
```
Let's test our cluster:
```
kubectl get nodes
NAME STATUS ROLES AGE VERSION
githubactions-control-plane Ready control-plane 2m53s v1.28.0
```
## Running the Runner in Docker
We can simply install this directly on to virtual machines , but for this demo, I'd like to run it in Kubernetes inside a container.
### Security notes
* Running in Docker needs high priviledges.
* Would not recommend to use these on public repositories.
* Would recommend to always run your CI systems in seperate Kubernetes clusters.
### Creating a Dockerfile
* Installing Docker CLI
For this to work we need a `dockerfile` and follow instructions to [Install Docker](https://docs.docker.com/engine/install/debian/).
I would grab the content and create statements for my `dockerfile`
Now notice that we only install the `docker` CLI.
This is because we want our running to be able to run docker commands , but the actual docker server runs elsewhere
This gives you flexibility to tighten security by running docker on the host itself and potentially run the container runtime in a non-root environment
* Installing Github Actions Runner
Next up we will need to install the [GitHub actions runner](https://github.com/actions/runner) in our `dockerfile`
Now to give you a "behind the scenes" of how I usually build my `dockerfile`s, I run a container to test my installs:
```
docker build . -t github-runner:latest
docker run -it github-runner bash
```
Next steps:
* Now we can see `docker` is installed
* To see how a runner is installed, lets go to our repo | runner and click "New self-hosted runner"
* Try these steps in the container
* We will needfew dependencies
* We download the runner
Finally lets test the runner in `docker`
```
docker run -it -e GITHUB_PERSONAL_TOKEN="" -e GITHUB_OWNER=marcel-dempers -e GITHUB_REPOSITORY=docker-development-youtube-series github-runner
```
## Deploy to Kubernetes
Load our github runner image so we dont need to push it to a registry:
```
kind load docker-image github-runner:latest --name githubactions
```
Create a kubernetes secret with our github details
```
kubectl create ns github
kubectl -n github create secret generic github-secret `
--from-literal GITHUB_OWNER=marcel-dempers `
--from-literal GITHUB_REPOSITORY=docker-development-youtube-series `
--from-literal GITHUB_PERSONAL_TOKEN=""
kubectl -n github apply -f kubernetes.yaml
```
---
### Overview (`golang/introduction/part-2.json/readme.md`)
# Introduction to Go: JSON
In programming languages, you will very often deal with data structures internally. 
HTTP is a fundamental part of Microservices and Web distributed systems 
Command line apps are a fundamental part of software development 
Up until now, we've learned the fundamentals of Go and built a small web microservice that handles our video data.
Our service has a `/` `GET` endpoint for returning all videos, as well as a simple `/update` endpoint for updating our list of videos.
We've learnt how to read and write from files and learn how to work with `json` data.
This is important for learning Go, however there are a few challenges for using a file as storage.
Code is over [here](../../../storage/redis/clustering/readme.md)
## Go Dev Environment
The same as Part 1+2+3+4, we start with a [dockerfile](./dockerfile) where we declare our version of `go`.
The `dockerfile`:
```
FROM golang:1.15-alpine as dev
WORKDIR /work
```
Let's build and start our container:
```
cd golang\introduction\part-5.database.redis
docker build --target dev . -t go
docker run -it -p 80:80 -v ${PWD}:/work go sh
go version
```
## Our application
Create a new directory that holds defines our `repository` and holds our `module`
```
mkdir videos
```
* Define a module path
```
# change directory to your application source code
cd videos
# create a go module file
go mod init videos
```
## Create our base code
We start out all our applications with a `main.go` defining our `package`, declaring our `import` dependencies 
## We need a Kubernetes cluster
Lets create a Kubernetes cluster to play with using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/)
```
kind create cluster --name vpa --image kindest/node:v1.30.4
```
| 
|
[Official Site](https://agentgateway.dev) | [GitHub](https://github.com/agentgateway/agentgateway) | [Documentation](https://agentgateway.dev/docs/kubernetes/latest/)
## Prerequisites
To get started, you will need to follow the [Introduction to Gateway API](../README.md) first.
You'll need an understanding of the Gateway API.
In the introduction guide, you will:
* Create a local Kubernetes cluster
* Install the Gateway API CRDs
* Example models, either in cluster or external (OpenAI, Anthropic, Gemini)
This will allow us access to the Gateway API so we can go ahead and deploy a Gateway API controller to use.
To get the version you want, you can use the agent gateway [Github Release page](https://github.com/agentgateway/agentgateway/releases).
For this guide, we will use 1.1.0.
You can list helm chart versions too:
```shell
helm show chart oci://cr.agentgateway.dev/charts/agentgateway-crds
helm show chart oci://cr.agentgateway.dev/charts/agentgateway
```
```shell
CHART_VERSION="v1.1.0"
# Install CRDs
helm upgrade -i agentgateway-crds oci://cr.agentgateway.dev/charts/agentgateway-crds \
--create-namespace \
--namespace agentgateway-system \
--version $CHART_VERSION
# Install ControlPlane
helm upgrade -i agentgateway oci://cr.agentgateway.dev/charts/agentgateway \
--namespace agentgateway-system \
--version $CHART_VERSION \
--set controller.image.pullPolicy=Always \
--wait
```
Check our installation
```shell
# check pods are running
kubectl -n agentgateway-system get pods
# check logs
kubectl -n agentgateway-system logs -l app.kubernetes.io/name=agentgateway
```
## Check the AgentGateway Class
Verify the `GatewayClass` was registered:
```shell
kubectl get gatewayclass
```
## Install an AgentGateway Gateway
```shell
kubectl apply -f kubernetes/gateway-api/agentgateway/01-gateway.yaml
# we can see our gateway dataplane pods
kubectl get pods
kubectl get svc
# check gateway
kubectl get gateway
# port forward for access
kubectl -n default port-forward svc/agentgateway 8080:80
```
## Features
* Route to [LLM Models](#feature-llm-routing)
* Route to [MCP Servers](#feature-mcp-routing)
### Feature: LLM Routing
AgentGateway introduces CRD `AgentgatewayBackend`.
`AgentgatewayBackend` is an LLM provider as a routable backend.
#### Create an LLM Backend: Gemini
If your LLM Provider needs an API key, we need a Kubernetes secret for it:
```shell
export GEMINI_API_KEY=""
kubectl apply -f- <
## Prerequisites
This guide will assume you have a basic understanding of the Gateway API.
If it's your first time using Gateway API, please see the [Introduction to Gateway API guide](../README.md) first
This guide has a slight twist from the other Gateway APIs.
It's because Cilium is a network CNI for Kubernetes.
This means we need to disable the default kubernetes CNI networking in our cluster, so we will be creating it slightly differently.
We also need to install the Gateway API CRDs followed by Cilium, before we deploy all our example applications.
## What is Cilium
[Cilium](https://cilium.io/) leverages eBPF technology to provide high-performance networking, security and observability for containerised applications, especially in Kubernetes.
eBPF (Extended Berkeley Packet Filter) is a technology that allows us to run custom programs safely in the Linux Kernel without the need for modules, changing source code.
These programs allow programs like Cilium to add security, performance and observability to the network.
### Create a kubernetes cluster
```shell
kind create cluster --name gatewayapi --image kindest/node:v1.34.0 --config kubernetes/gateway-api/cilium/kind.yaml
```
Test our cluster and makes sure `kubectl` is configured for it:
```shell
kubectl get nodes
NAME STATUS ROLES AGE VERSION
gatewayapi-control-plane NotReady control-plane 93s v1.34.0
gatewayapi-worker NotReady
## Prerequisites
To get started, you will need to follow the [Introduction to Gateway API](../README.md) first.
You'll need an understanding of the Gateway API.
In the introduction guide, you will:
* Create a local Kubernetes cluster
* Install the Gateway API CRDs
* Deploy example apps to our cluster
* Have Domains for our traffic
* Have TLS certificates
This will allow us access to the Gateway API so we can go ahead and deploy a Gateway API controller to use.
Envoy is not only a Gateway API, that's just one of its features.
Let's take a look at the [Official Site](https://www.envoyproxy.io/) and jump to the documentation.
Envoy has a separate web page for the Gateway API feature.
## Envoy: Gateway API controller
Envoy has a `helm` chart specifically for the gateway product.
The `helm` chart relies on the Gateway API CRD's we installed in our cluster already as well as Envoy Proxy.
### Installation
We can install the helm chart with `helm install` or upgrade it with `helm upgrade`:
```shell
CHART_VERSION="v1.6.0"
helm show chart oci://docker.io/envoyproxy/gateway-helm
helm show values oci://docker.io/envoyproxy/gateway-helm > kubernetes/gateway-api/envoy/default-values.yaml
helm install envoy-gateway oci://docker.io/envoyproxy/gateway-helm \
--version $CHART_VERSION \
--values kubernetes/gateway-api/envoy/values.yaml \
--namespace envoy-gateway-system \
--create-namespace
```
Upgrade:
```shell
helm upgrade envoy-gateway oci://docker.io/envoyproxy/gateway-helm \
--values kubernetes/gateway-api/envoy/values.yaml \
--version $CHART_VERSION \
--namespace envoy-gateway-system
```
### Configuration
Most of the Gateway API controllers are installed using `helm`.
Before we install it, let's take a look at the [default-values.yaml](./default-values.yaml)
### Check Installation
Now we should have the Envoy gateway API controller up and running.
This is not the gateway itself, but the controller that will manage the CRDs we get access to and implement some gateway API CRDs.
```shell
# check the controller pods
kubectl -n envoy-gateway-system get pods
# check the controller pod logs
kubectl -n envoy-gateway-system logs -l app.kubernetes.io/instance=envoy-gateway
```
## Install an Envoy Gateway Class
```shell
kubectl apply -f kubernetes/gateway-api/envoy/01-gatewayclass.yaml
```
## Install an Envoy Gateway
```shell
kubectl apply -f kubernetes/gateway-api/envoy/02-gateway.yaml
```
When we apply the gateway, we get a new gateway api pod.
```shell
# check the new gateway-api pod
kubectl -n envoy-gateway-system get pods
# we also have a new service
kubectl -n envoy-gateway-system get svc
```
### Gateway Configuration
`EnvoyProxy` is the CRD that allows us to configure each gateway API instance.
For example, we can change the `deployment` or `service` of the instance like so:
```shell
kubectl apply -f kubernetes/gateway-api/envoy/02.1-gateway-config.yaml
# we should see 2 replicas
kubectl -n envoy-gateway-system get deploy
kubectl -n envoy-gateway-system get pods
# port forward for access (get the correct service)
kubectl -n envoy-gateway-system get svc
kubectl -n envoy-gateway-system port-forward svc/envoy-gateway-default 80
```
## HTTP Traffic management
Feel free to quickly run through the basic [traffic management table](../README.md#traffic-management-features--http-routes) for using `HTTPRoute` routing for traffic.
Note: HTTPRoute features are not specific to this controller and should be available to any other gateway API controller that you choose.
## Gateway API Extensions
Envoy Gateway has its own extensions on top of the native Gateway API features we've seen so far, using custom CRD's.
### Client Traffic Policies
#### Connection Limit
Envoy allows us to limit connections to gateways & listeners.
We can use tools like `hey` to demonstrate connection limits.
```shell
hey -c 10 -q 1 -z 10s -host "example-app.com" http://localhost:80/api/go/status
kubectl apply -f kubernetes/gateway-api/envoy/08-clientpolicy-connectionlimit.yaml
hey -c 10 -q 1 -z 10s -host "example-app.com" http://localhost:80/api/go/status
```
#### Mutual TLS
Currently we have a listener for TLS (HTTPs). For normal TLS, the client does not have to pass a private key.
Envoy supports mutual TLS
```shell
# tell Envoy about the CA cert
kubectl create secret generic secret-tls-ca --from-file=rootCA.pem=kubernetes/gateway-api/tls/rootCA.pem
# generate client certs for curl
mkcert -client client-user.com
kubectl apply -f kubernetes/gateway-api/envoy/09-clientpolicy-mtls.yaml
# browser TLS should now be broken "Access to example-app.com was denied"
# This is because a client needs to pass certificates to trust
curl -v -H "Host: example-app.com" \
--cert kubernetes/gateway-api/tls/client-user.com-client.pem \
--key kubernetes/gateway-api/tls/client-user.com-client-key.pem \
--cacert kubernetes/gateway-api/tls/rootCA.pem \
https://example-app.com/api/go/status
```
### Backend Traffic Policies
#### Local and Global Rate limits
We can rate limit each HTTP route using Local Rate Limit:
```shell
kubectl apply -f kubernetes/gateway-api/envoy/10-backendpolicy-ratelimit.yaml
for i in {1..4}; do
curl -I --header "x-user-id: one" http://example-app.com/api/go/status ; sleep 1;
done
```
### Security Policies
### CORS
```shell
# port-forward to set pre-cors
kubectl port-forward svc/web-app 8000:80
kubectl apply -f kubernetes/gateway-api/envoy/11-securitypolicy-cors.yaml
```
### Basic Auth
```shell
#cleanup client policies to avoid conflict
kubectl delete clienttrafficpolicies mtls-policy
kubectl delete SecurityPolicy go-route-cors
# port-forward to 443
kubectl -n envoy-gateway-system port-forward svc/envoy-gateway-default 443
#install htpasswd (alpine)
apk add apache2-utils
htpasswd -cbs .htpasswd bob password123
kubectl create secret generic basic-auth --from-file=.htpasswd
kubectl apply -f kubernetes/gateway-api/envoy/12-securitypolicy-basicauth.yaml
```
### API Auth
```shell
#cleanup previous basic auth policy
kubectl delete SecurityPolicy go-route-basicauth
kubectl apply -f kubernetes/gateway-api/envoy/13-securitypolicy-apiauth.yaml
# port-forward
kubectl -n envoy-gateway-system port-forward svc/envoy-gateway-default 443
```
You can see `Client authentication failed.` in the browser.
We need to pass the header
```
curl -v -H "Host: example-app.com" \
-H 'x-api-key: secret123' \
--cert kubernetes/gateway-api/tls/client-user.com-client.pem \
--key kubernetes/gateway-api/tls/client-user.com-client-key.pem \
--cacert kubernetes/gateway-api/tls/rootCA.pem \
https://example-app.com/api/go/status
```
Logs:
```shell
kubectl -n envoy-gateway-system logs -l app.kubernetes.io/managed-by=envoy-gateway
```
---
### Overview (`kubernetes/gateway-api/haproxy/README.md`)
# Introduction to HAProxy: Gateway API
## Prerequisites
To get started, you will need to follow the [Introduction to Gateway API](../README.md) first.
You'll need an understanding of the Gateway API.
Important Note: According to testing and [HAProxy documentation](https://www.haproxy.com/documentation/kubernetes-ingress/gateway-api/enable-gateway-api/#deploy-gateway-api-resources), it currently only supports version v0.5.1
Tried `GatewayClass` with `gateway.networking.k8s.io/v1` and does not work
In the introduction guide, you will:
* Create a local Kubernetes cluster
* Install the Gateway API CRDs
```shell
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v0.5.1/experimental-install.yaml
```
* Deploy example apps to our cluster
* Have Domains for our traffic
* Have TLS certificates
This will allow us access to the Gateway API so we can go ahead and deploy a Gateway API controller to use.
At the time of this guide the HAProxy Gateway API solution is built alongside the HAProxy Kubernetes Ingress controller. So its one solution that can achieve both Ingress and Gateway functionality.
The HAProxy Unified Gateway is still under development at the time of this guide.
### Configuration
Most of the Gateway API controllers are installed using `helm`.
Before we install it, let's take a look at the [default-values.yaml](./default-values.yaml)
### Installation
We can install the helm chart with `helm install` or with `kubectl` as stated in the documentation. Important to know that the below Helm chart installs the ingress controller. In order to enable Gateway API functionality, we need to turn it on using our values file.
```shell
CHART_VERSION="1.48.0"
helm repo add haproxytech https://haproxytech.github.io/helm-charts
helm repo update
helm search repo haproxytech --versions
# see the default configuration
helm show values haproxytech/kubernetes-ingress \
> kubernetes/gateway-api/haproxy/default-values.yaml
#apply the cluster roles and bindings (RBAC) to give HAProxy access to Gateway resources
kubectl apply -f https://raw.githubusercontent.com/haproxytech/kubernetes-ingress/master/deploy/tests/config/experimental/gwapi-rbac.yaml
helm install haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
--version $CHART_VERSION \
--create-namespace \
--namespace haproxy-controller \
--values kubernetes/gateway-api/haproxy/values.yaml
```
### Check Installation
Now we should have the HAProxy controller up and running.
This is not the gateway itself, but the controller that will manage the CRDs we get access to and implement some gateway API CRDs.
```shell
# check the controller pods
kubectl -n haproxy-controller get pods
# check the controller pod logs
kubectl -n haproxy-controller logs -l app.kubernetes.io/name=kubernetes-ingress
```
## Install an HAProxy Gateway Class
Please note `apiVersion: gateway.networking.k8s.io/v1alpha2` is used and `v1` does not work. :
```shell
kubectl apply -f kubernetes/gateway-api/haproxy/01-gatewayclass.yaml
```
## Install an HAProxy Gateway
```shell
kubectl apply -f kubernetes/gateway-api/haproxy/02-gateway.yaml
```
## Traffic Routing
At the time of this guide, HAProxy only supports TCPRoute, and not HTTPRoute like the other controllers we covered thus far.
```shell
kubectl apply -f kubernetes/gateway-api/haproxy/03-tcproute.yaml
```
```shell
kubectl -n haproxy-controller port-forward svc/haproxy-kubernetes-ingress 80
```
---
### Overview (`kubernetes/gateway-api/istio/README.md`)
# Introduction to Istio: Gateway API
## Prerequisites
To get started, you will need to follow the the [Introduction to Gateway API](../README.md) first.
You'll need an understanding of the Gateway API.
In the introduction guide, you will:
* Create a local Kubernetes cluster
* Install the Gateway API CRDs
* Deploy example apps to our cluster
* Have Domains for our traffic
* Have TLS certificates
This will allow us access to the Gateway API so we can go ahead and deploy a Gateway API controller to use.
Istio is not only a Gateway API, its a service mesh.
In a distributed system with service-to-service communication, a service mesh places a proxy next to each service.
This proxy intercepts all inbound and outbound communication for it's neighboring service.
Doing so, a service mesh provides many features
* observability: monitor network calls (metrics \ traces)
* traffic management: traffic routing features
* resillience: automatic retry, circuit breaker, rate limiting etc
* security: mutual TLS, authentication, policies etc
Let's take a look at the [Official Site](https://istio.io/latest/docs/overview/what-is-istio/) and jump to the documentation.
## Istio: Gateway API controller
With Istio, we have a few more steps to install the gateway API. We fistly need to install the base `helm` chart which includes Istio specific CRD components required.
### Installation
We can install the helm chart with `helm install` or upgrade it with `helm upgrade`:
```shell
BASE_CHART_VERSION="1.28.1"
ISTIOD_CHART_VERSION="1.28.1"
helm repo add istio https://istio-release.storage.googleapis.com/charts
helm repo update
# get versions for the charts
helm search repo istio/base --versions | head
helm search repo istio/istiod --versions | head
helm install istio-base istio/base \
--version $BASE_CHART_VERSION \
--namespace istio-system \
--create-namespace \
--set defaultRevision=default
helm install istiod istio/istiod \
--version $ISTIOD_CHART_VERSION \
--namespace istio-system \
--wait
```
### Configuration
```
helm show values istio/istiod \
--version $ISTIOD_CHART_VERSION > kubernetes/gateway-api/istio/default-values.yaml
```
### Check Installation
Now we should have the Istio control plane / controller up and running.
This is not the gateway itself, but the controller that will manage the CRDs we get access to and implements some gateway API CRD's.
```shell
# check deployed status
helm ls -n istio-system
# check the controller pods
kubectl get pods -n istio-system --output wide
# check the controller pod logs
kubectl -n istio-system logs -l app.kubernetes.io/instance=istiod
```
## Install an Istio Gateway Class
```shell
kubectl apply -f kubernetes/gateway-api/istio/01-gatewayclass.yaml
```
## Install an Istio Gateway
```shell
kubectl apply -f kubernetes/gateway-api/istio/02-gateway.yaml
```
When we apply the gateway, we get a new gateway api pod.
Unlike Traefik and Envoy, in Istio, the gateway pod is created in the same namespace as the gateway
```shell
# check the new gateway-api pod - this time its in the same namespace as the gateway object!
kubectl -n default get pods
# we also have a new service
kubectl -n default get svc
```
### Gateway Configuration
For global Gateway class defaults, we can use a `ConfigMap` with a special label in the `istio-system` namespace.
This allows us to set global defaults for all our gatways.
```shell
kubectl apply -f kubernetes/gateway-api/istio/01.1-gatewayclass-config.yaml
```
For Gateways, we can use the `parametersRef` field under `infrastructure` and use a `ConfigMap` to configure it further.
For example, we can change the `deployment` or `service` of the instance like so:
```shell
kubectl apply -f kubernetes/gateway-api/istio/02.1-gateway-config.yaml
# we should see 2 replicas
kubectl -n default get pods
# port forward for access (get the correct service)
kubectl -n default port-forward svc/gateway-api-istio 80
```
## HTTP Traffic management
Feel free to quickly run through the basic [traffic management table](../README.md#traffic-management-features--http-routes) for using `HTTPRoute` routing for traffic.
Note: HTTPRoute features are not specific to this controller and should be available to any other gateway API controller that you choose.
## Istio Gateway VS Istio Gateway API
Istio has been around for a lot longer than Gateway API has. Therefore Istio has solved much of the traffic management challenges using it's own CRDs, such as `VirtualService`.
There is a very important difference between Istio and the other Gateway APIs we've covered so far.
Istio has made a strict separation between it's own Gateway (`networking.istio.io/v1`) and the Gateway of Gateway API (`gateway.networking.k8s.io/v1`)
This means, their CRDs such as `VirtualService` does not attach to the new Kubernetes Gateway API like `Gateway` and `HTTPRoute`.
Therefore, to use Gateway API features, one should use Gateway API and not Istio Gateways.
### Installing Classic Istio Gateway
To install the Istio gateway, we also use `helm`
```shell
ISTIOGATEWAY_CHART_VERSION="1.28.1"
helm repo add istio https://istio-release.storage.googleapis.com/charts
helm repo update
# get versions for the charts
helm search repo istio/gateway --versions | head
helm install istio-ingress istio/gateway \
--version $ISTIOGATEWAY_CHART_VERSION \
--namespace istio-system
kubectl -n istio-system port-forward svc/istio-ingress 80
```
### Enable Application Service Mesh
To use virtual services we need to enable the mesh in our cluster.
I will be using the classic side-car mode, so we'll label our `default` namespace with the correct labels so that the Istio control plane will automatically inject sidecar containers into all pods in the namespace
[Virtual Services](https://istio.io/latest/docs/reference/config/networking/virtual-service/) are used throughout Istio to manage traffic.
```shell
kubectl label namespace default istio-injection=enabled
# delete all pods to get sidecars added
kubectl delete pods --all
```
We will also need the Gateway controller that will be managing our Istio Gateways.
### Istio Gateway Ingress
Let's deploy the Istio Gateway
```shell
kubectl apply -f kubernetes/gateway-api/istio/08-istiogateway.yaml
```
### Traffic management using Istio Gateways
Technically, as this is a Gateway API guide, this following bit would be out of scope.
However I wanted to show you the difference by running your thorugh a quick demo of `VirtualService` using Isio Gateway.
As an example, `VirtualService` can be used to deal Traffic in the same way as Gateway API and Ingress
Let's deploy a Virtual service to perform basic traffic routing:
```shell
kubectl apply -f kubernetes/gateway-api/istio/09-virtualservice.yaml
```
---
### Overview (`kubernetes/gateway-api/kgateway/README.md`)
# Introduction to kgateway: Gateway API
## Prerequisites
To get started, you will need to follow the [Introduction to Gateway API](../README.md) first.
You'll need an understanding of the Gateway API.
In the introduction guide, you will:
* Create a local Kubernetes cluster
* Install the Gateway API CRDs
* Deploy example apps to our cluster
* Have Domains for our traffic
* Have TLS certificates
This will allow us access to the Gateway API so we can go ahead and deploy a Gateway API controller to use.
In this guide we will turn on the Gateway API feature in kgateway and use that specifically.
### Installation
To get the version you want, you can use the kgateway [Github Release page](https://github.com/kgateway-dev/kgateway/releases/tag/v2.1.2).
For this guide, we will use 2.1.2.
Install:
```shell
CHART_VERSION="v2.3.3"
helm show chart oci://cr.kgateway.dev/kgateway-dev/charts/kgateway --version $CHART_VERSION
helm show chart oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds --version $CHART_VERSION
helm show values oci://cr.kgateway.dev/kgateway-dev/charts/kgateway --version $CHART_VERSION > kubernetes/gateway-api/kgateway/default-values.yaml
# deploy CRDs
helm upgrade kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds \
--version ${CHART_VERSION} \
--namespace kgateway-system \
--create-namespace \
--install
# Deploy kgateway control plane
helm upgrade kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
--version $CHART_VERSION \
--values kubernetes/gateway-api/kgateway/values.yaml \
--namespace kgateway-system \
--install \
--create-namespace
```
Check our installation
```shell
# check the pods
kubectl -n kgateway-system get pods
# check the logs
kubectl -n kgateway-system logs -l app.kubernetes.io/name=kgateway
```
## Install a kgateway Gateway Class
The `helm` chart will create a `GatewayClass` for us as per [docs](https://kgateway.dev/docs/envoy/latest/setup/default/#gatewayclass).
```shell
kubectl get gc kgateway -o yaml
```
## Install a kgateway Gateway
```shell
kubectl apply -f kubernetes/gateway-api/kgateway/01-gateway.yaml
# we can see our gateway dataplane pods
kubectl get pods
kubectl get svc
# port forward for access
kubectl -n default port-forward svc/gateway-api 80
```
### Configuration
Most of the Gateway API controllers are installed using `helm`.
Before we install it, let's take a look at the [values.yaml](./values.yaml)
kgateway has a section on [Gateway Customization](https://kgateway.dev/docs/envoy/main/setup/) where we can adjust gateway parameters.
Gateway API allows us to inject parameters using the `infrastructure.parametersRef` field
We can customize our gateway:
```shell
kubectl apply -f kubernetes/gateway-api/kgateway/02-gateway-config.yaml
```
## HTTP Traffic management
Feel free to quickly run through the basic [traffic management table](../README.md#traffic-management-features--http-routes) for using `HTTPRoute` routing for traffic.
Note: HTTPRoute features are not specific to this controller and should be available to any other gateway API controller that you choose.
## Policies: TrafficPolicy
kgateway uses `TrafficPolicy` as their custom CRD to apply traffic policies to certain HTTPRoutes or all HTTPRoutes that a Gateway serves.
There are many [TrafficPolicy Features](https://kgateway.dev/docs/envoy/main/reference/api/#trafficpolicy) available.
The cool thing here is that these TrafficPolicies are now pluggable into Gateway API HTTPRoutes.
In `HTTPRoute` objects, under `filters`, we can add extensions using something like:
```yaml
filters:
- type: ExtensionRef
extensionRef:
group: gateway.kgateway.dev
kind: TrafficPolicy
name: cors-feature
```
### TrafficPolicy Example
Example of a Traffic Policy supporting CORS
```shell
kubectl apply -f kubernetes/gateway-api/kgateway/03-trafficpolicy-cors.yaml
```
## Policies: HTTPListenerPolicy
This is a special CRD for attaching policies to listeners on gateways.
We can use this to set custom access log formats and various features of the gateway listener.
The following changes our log format to JSON
```shell
kubectl apply -f kubernetes/gateway-api/kgateway/04-gateway-accesslogs.yaml
# see log format
kubectl logs -l app.kubernetes.io/name=gateway-api
```
## Policies: BackendConfigPolicy
[BackendConfigPolicy](https://kgateway.dev/docs/envoy/latest/about/policies/backendconfigpolicy/) allows us to configure connection settings for a backend like a Kubernetes Service
## AgentGateway
One thing that sets kgateway apart from the other gateway API's is its integration with [Agent Gateway](https://kgateway.dev/docs/agentgateway/latest/)
---
### Overview (`kubernetes/gateway-api/linkerd/README.md`)
# Introduction to Linkerd: Gateway API
## Prerequisites
To get started, you will need to follow the [Introduction to Gateway API](../README.md) first.
You'll need an understanding of the Gateway API.
In the introduction guide, you will:
* Create a local Kubernetes cluster
* Install the Gateway API CRDs
* Deploy example apps to our cluster (below)
In this guide, we will deploy our videos application which has both north\south and east\west traffic:
```shell
kubectl apply -f monitoring/opentelemetry/applications/playlists-api/
kubectl apply -f monitoring/opentelemetry/applications/playlists-db/
kubectl apply -f monitoring/opentelemetry/applications/videos-web/
kubectl apply -f monitoring/opentelemetry/applications/videos-api/
kubectl apply -f monitoring/opentelemetry/applications/videos-db/
```
* Have Domains for our traffic
This will allow us access to the Gateway API so we can go ahead and deploy a Gateway API controller to use.
Linkerd is the first service mesh to achieve CNCF graduation status and is highly matured.
In a distributed system with service-to-service communication, a service mesh uses various methods to intercept all inbound and outbound communication to and from a given service.
So when a service makes an outbound call or has an inbound network call, the mesh is aware of it.
Doing so, a service mesh provides many features:
* observability: monitor network calls (metrics \ traces)
* traffic management: traffic routing features
* resillience: automatic retry, circuit breaker, rate limiting etc
* security: mutual TLS, authentication, policies etc
Each service mesh is different in terms of how it achieves this. Some meshes use side-cars to intercept and proxy traffic that go in and out of pods. Others use node level proxies and even eBPF to proxy at kernel level.
Linkerd uses the sidecar approach with their proxy written in Rust.
There are a few things that make Linkerd very different to other Gateway APIs we looked at.
Linkerd is a big player in the [The GAMMA Initiative](https://gateway-api.sigs.k8s.io/mesh/gamma/) for Gateway API.
Linkerd does not focus on north/south traffic and therefore we will use an alternative gateway API implementation for that.
Linkerd's focus is more about mesh traffic, which is east\west.
## Linkerd: Gateway API
Linkerd provides Gateway API support included within its single installation.
Unlike some other gateways, that provide Gateway API support through a separate `helm` chart, Linkerd is all one single simple install.
The `helm` chart relies on the Gateway API CRD's we installed in our cluster already
We can checkout the official [install via helm](https://linkerd.io/2.19/tasks/install-helm/) for more instructions.
### Installation
We firstly need to create mTLS certificates that Linkerd will use for secure internal communication in its service mesh
```shell
apk add step-cli #alpine linux
#use the linkerd CLI to check the install
export LINKERD2_VERSION=edge-25.12.3
curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/install-edge | sh
export PATH=$PATH:/root/.linkerd2/bin
linkerd check --pre
#generate CA
step certificate create root.linkerd.cluster.local /tmp/ca.crt /tmp/ca.key \
--profile root-ca --no-password --insecure
#generate intermediate cert for csr's
step certificate create identity.linkerd.cluster.local /tmp/issuer.crt /tmp/issuer.key \
--profile intermediate-ca --not-after 8760h --no-password --insecure \
--ca /tmp/ca.crt --ca-key /tmp/ca.key
```
We can install the helm chart with `helm install` or upgrade it with `helm upgrade`
Please note that I will be using the linkerd edge release in this demo to ensure I have the latest functionality. For production, you may want to use the stable release instead.
```shell
CRD_CHART_VERSION="2025.12.3"
CHART_VERSION="2025.12.3"
helm repo add linkerd-edge https://helm.linkerd.io/edge
helm repo update
helm search repo linkerd-edge --versions | grep linkerd-edge/linkerd-crds | head
helm search repo linkerd-edge --versions | grep linkerd-edge/linkerd-control-plane | head
# get values files
helm show values linkerd-edge/linkerd-control-plane > kubernetes/gateway-api/linkerd/default-values.yaml
helm install linkerd-crds linkerd-edge/linkerd-crds \
--version $CRD_CHART_VERSION \
--namespace linkerd \
--create-namespace
helm install linkerd-control-plane \
--version $CHART_VERSION \
--namespace linkerd \
--create-namespace \
--set-file identityTrustAnchorsPEM=/tmp/ca.crt \
--set-file identity.issuer.tls.crtPEM=/tmp/issuer.crt \
--set-file identity.issuer.tls.keyPEM=/tmp/issuer.key \
--values kubernetes/gateway-api/linkerd/values.yaml \
linkerd-edge/linkerd-control-plane
```
### Check Installation
Now we should have the Envoy gateway API controller up and running.
This is not the gateway itself, but the controller that will manage the CRDs we get access to and implement some gateway API CRDs.
```shell
# check the control plane pods
kubectl -n linkerd get pods
linkerd check
```
### Configuration
Most of the Gateway API controllers are installed using `helm`.
Before we install it, let's take a look at the [default-values.yaml](./default-values.yaml)
Linkerd also provide a High Availability YAML file for HA deployments which I found [here on Github](https://github.com/linkerd/linkerd2/blob/main/charts/linkerd-control-plane/values-ha.yaml).
So in this guide I will base my demo off that with the `controllerReplicas` set to 1 for demo purpose as I am running a single node cluster.
## Install a Gateway Class & Gateway
Since Linderd does not have a gateway, we will be using Envoy from our [Envoy Gateway Guide](../envoy/README.md#envoy-gateway-api-controller)
```shell
# deploy gateway class
kubectl apply -f kubernetes/gateway-api/envoy/02.1-gateway-config.yaml
# deploy a gateway
kubectl apply -f kubernetes/gateway-api/envoy/02-gateway.yaml
```
When we apply the gateway, we get a new gateway api pod.
```shell
# check the new gateway-api pod
kubectl -n envoy-gateway-system get pods
# we also have a new service
kubectl -n envoy-gateway-system get svc
# port forward for access to the new gateway
kubectl -n envoy-gateway-system port-forward svc/envoy-gateway-default 80
```
## Mesh our Services
To add our services to the Linkerd mesh, we need to annotate each deployment
```shell
kubectl get deploy playlists-api -o yaml | linkerd inject - | kubectl apply -f -
kubectl get deploy playlists-db -o yaml | linkerd inject - | kubectl apply -f -
kubectl get deploy videos-api -o yaml | linkerd inject - | kubectl apply -f -
kubectl get deploy videos-db -o yaml | linkerd inject - | kubectl apply -f -
kubectl get deploy videos-web -o yaml | linkerd inject - | kubectl apply -f -
```
## Linkerd Dashboard
To access the `linkerd` dashboard, we need to install the viz extension
```
linkerd viz install | kubectl apply -f -
kubectl -n linkerd-viz port-forward svc/web 8084
```
## HTTPRoutes: North \ South Traffic
For north\south traffic, we will use Gateway API and setup a HTTP route
```shell
kubectl apply -f kubernetes/gateway-api/linkerd/03-httproute.yaml
```
## HTTPRoutes - East \ West Traffic
For east\west traffic features:
Let's deploy a V2 of our videos-api that only returns a video about the Gateway API.
```shell
# new V2
kubectl apply -f kubernetes/gateway-api/linkerd/04-east-west-app.yaml
# mesh the new app
kubectl get deploy videos-api-v2 -o yaml | linkerd inject - | kubectl apply -f -
# HTTPRoute that routes traffic east \ west
kubectl apply -f kubernetes/gateway-api/linkerd/04-httproute-east-west.yaml
```
You could use this for a number of reasons:
* Inject latency
* Inject faults to test applications
* Canary deployments
* Dynamic routing
---
### Overview (`kubernetes/gateway-api/nginx-fabric/README.md`)
# Introduction to NGINX Gateway Fabric: Gateway API
## Prerequisites
To get started, you will need to follow the [Introduction to Gateway API](../README.md) first.
You'll need an understanding of the Gateway API.
In the introduction guide, you will:
* Create a local Kubernetes cluster
* Install the Gateway API CRDs
* Deploy example apps to our cluster
* Have Domains for our traffic
* Have TLS certificates
This will allow us access to the Gateway API so we can go ahead and deploy a Gateway API controller to use.
NGINX Fabric is a little different as their `GatewayClass` is managed by helm and there are extra configurations managed by the chart that gets injected into the `GatewayClass` under the `parametersRef` field.
If you've been following this series you will know the `parametersRef` field is used to extend the class. NGINX uses a custom CRD called `NginxProxy` to customize the class, but these settings come from their `helm` chart.
For example:
```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: nginx-gateway
annotations:
meta.helm.sh/release-name: ngf
meta.helm.sh/release-namespace: nginx-gateway
labels:
app.kubernetes.io/managed-by: Helm
spec:
controllerName: gateway.nginx.org/nginx-gateway-controller
parametersRef:
group: gateway.nginx.org
kind: NginxProxy
name: ngf-proxy-config
namespace: nginx-gateway
```
## NGINX Fabric: Gateway API controller
NGINX Fabric covers most of the core Gateway API features.
It also supports custom features using policies
### Configuration
Most of the Gateway API controllers are installed using `helm`.
Before we install it, let's take a look at the [values.yaml](./values.yaml)
The Helm chart allows us to configure many options. Some I find quite important:
* Control plane deployment & Service
* Data plane deployment & Service modification
* Ports to use for incoming traffic
* Control Gateway API
* Default CRDs
* Default GatewayClass
* Default Gateways
There are [helm examples](https://github.com/nginx/nginx-gateway-fabric/tree/v2.2.2/examples/helm) on Github maintained by the NGINX community.
Here is the [helm chart documentation](https://github.com/nginx/nginx-gateway-fabric/blob/v2.2.2/charts/nginx-gateway-fabric/README.md)
It's always good to get a grip on the [default helm values](./default-values.yaml) to see what we can do with the chart.
The values file is used to customise NGINX control & data plane, underlying pods, deployments and services as well as turning features on and off.
We can go ahead and install the controller to start with and use `helm upgrade` to make changes as we need to.
### Installation
Install:
```shell
CHART_VERSION="2.3.0"
helm show values oci://ghcr.io/nginx/charts/nginx-gateway-fabric > kubernetes/gateway-api/nginx-fabric/default-values.yaml
helm show chart oci://ghcr.io/nginx/charts/nginx-gateway-fabric
helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric \
--values kubernetes/gateway-api/nginx-fabric/values.yaml \
--version $CHART_VERSION \
--namespace nginx-gateway \
--create-namespace
```
Upgrade:
```shell
helm upgrade ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric \
--values kubernetes/gateway-api/nginx-fabric/values.yaml \
--version $CHART_VERSION \
--namespace nginx-gateway
```
Check our installation
```shell
# check the pods
kubectl -n nginx-gateway get pods
# check the logs
kubectl -n nginx-gateway logs -l app.kubernetes.io/instance=ngf
```
## Install a NGINX Gateway
So we have the `GatewayClass` managed by the `helm` chart, therefore we skip the `01-gatewayclass.yaml` and go straight to the gateway.
This will deploy NGINX dataplane pods to our target `default` namespace:
```shell
kubectl apply -f kubernetes/gateway-api/nginx-fabric/02-gateway.yaml
# check gateway pods
kubectl get pods
#check logs
kubectl logs -l app.kubernetes.io/instance=ngf
# port forward for access
kubectl port-forward svc/gateway-api-nginx-gateway 80
```
## Check & Configure the NGINX Dataplane
We can configure our NGINX Gateway, using the `spec.infrastructure.parametersRef` and point a Gateway to a `NginxProxy` resource. We can attach this config to a `Gateway` or `GatewayClass`
This allows us to control specific gateways or all gateways globally
Let's create a configuration to set the number of replicas on our Gateway deployment and set our service to `ClusterIP` since we don't have a load balancer.
Apply changes:
```shell
kubectl apply -f kubernetes/gateway-api/nginx-fabric/02.1-gateway-config.yaml
```
In addition -
When we define `HTTPRoute`'s for our traffic, The NGINX control plane detects those and works with the data plane pods to get an `nginx.conf` for the desired traffic rules.
If you are familiar with NGINX you will know all traffic rules are in the `nginx.conf`. So this solution transforms K8s CRDs to NGINX rules under the hood
```shell
# get a dataplane pod name
kubectl get pods
pod=gateway-api-nginx-gateway-5899b6cd6f-dpkbh
# check the generated nginx configuration file
kubectl exec -it -n default $pod -- nginx -T
```
## HTTP Traffic management
Feel free to quickly run through the basic [traffic management table](../README.md#traffic-management-features--http-routes) for using `HTTPRoute` routing for traffic.
Note: HTTPRoute features are not specific to this controller and should be available to any other gateway API controller that you choose.
## Policy APIs
Kubernetes Gateway API provides [policy attachment](https://gateway-api.sigs.k8s.io/reference/policy-attachment/) that allows us to augment or add configuration to existing resources
### Client Settings Policies
`ClientSettingsPolicy` is an NGINX Fabric CRD that allows us to customize how our Gateway behaves with client connections.
We can set things like Keep Alive , Timeout settings and HTTP Body sizes.
This CRD uses the above-mentioned policy attachment API for Gateway API.
Ensure the policy is accepted
```shell
kubectl apply -f kubernetes/gateway-api/nginx-fabric/08-clientsettings-keepalive.yaml
# check
kubectl describe clientsettingspolicies gateway-client-settings
```
Make more than 5 requests and notice every one re-using the connection except for the last:
```shell
# make 6 requests
curl -v http://example-app.com/api/go \
http://example-app.com/api/go \
http://example-app.com/api/go \
http://example-app.com/api/go \
http://example-app.com/api/go \
http://example-app.com/api/go
# note connection re-use
* Re-using existing connection with host example-app.com
```
### SnippetsFilter API
If you have used the NGINX Ingress controller, you may recall "custom snippets" , "server snippets" or "location snippets"
`SnippetsFilter` is an API that allows NGINX Gateway Fabric to expose some of NGINX raw configuration for tweaking.
Similar to NGINX Ingress, the NGINX proxy in Gateway Fabric also generates NGINX configuration files as demonstrated earlier. `SnippetsFilter` allows us to insert our own NGINX configuration into these files.
This feature is enabled in our [values.yaml](./values.yaml) file.
```shell
kubectl apply -f kubernetes/gateway-api/nginx-fabric/09-snippetsfilter.yaml
# check
kubectl describe SnippetsFilter custom-snippet
```
---
### Overview (`kubernetes/gateway-api/nginx-ingress/README.md`)
# Ingress-NGINX Migration to Gateway API
### Ingress Class
```shell
kubectl get ingress -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,CLASS:.spec.ingressClassName"
```
### Ingress to Gateway Tool
```shell
wget -q "https://github.com/kubernetes-sigs/ingress2gateway/releases/download/v0.4.0/ingress2gateway_Linux_x86_64.tar.gz" -O /tmp/igtogw.tar.gz && \
tar -C /tmp -xzf /tmp/igtogw.tar.gz && \
mv /tmp/ingress2gateway /usr/local/bin/ingress2gateway && \
chmod +x /usr/local/bin/ingress2gateway
./ingress2gateway print --providers=ingress-nginx
```
### Annotations Audit
To get a list of all NGINX Ingress Annotations we are using:
```shell
kubectl get ingress -A -o json | \
jq -r '.items[].metadata.annotations | keys[]' | \
grep "nginx.ingress" | \
sort -u
```
Now we can generate our audit list by selecting the above annotated output:
```shell
apk add util-linux
audit_fields=(
"NAMESPACE:.metadata.namespace"
"NAME:.metadata.name"
"HOSTS:.spec.rules[*].host"
"MIGRATED:.metadata.labels.migrated"
"REWRITE:.metadata.annotations.nginx\.ingress\.kubernetes\.io/rewrite-target"
"SNIPPET:.metadata.annotations.nginx\.ingress\.kubernetes\.io/configuration-snippet"
"AUTH:.metadata.annotations.nginx\.ingress\.kubernetes\.io/auth-type"
"PathType:.spec.rules[*].http.paths[*].pathType"
"TLS:.spec.tls[*].secretName"
)
IFS=$','
column_args="${audit_fields[*]}"
kubectl get ingress -A -o custom-columns="$column_args"
```
### Gateway API migration
| Feature | Ingress Nginx Annotation | Native Gateway API Resource / Filter | Example |
| :--- | :--- | :--- | :---|
| Basic Routing | `
## Prerequisites
To get started, you will need to follow the [Introduction to Gateway API](../README.md) first.
You'll need an understanding of the Gateway API.
In the introduction guide, you will:
* Create a local Kubernetes cluster
* Install the Gateway API CRDs
* Deploy example apps to our cluster
* Have Domains for our traffic
* Have TLS certificates
This will allow us access to the Gateway API so we can go ahead and deploy a Gateway API controller to use.
In this guide we will turn on the Gateway API feature in Traefik and use that specifically.
### Configuration
Most of the Gateway API controllers are installed using `helm`.
Before we install it, let's take a look at the [values.yaml](./values.yaml)
Traefik allows us to configure many options. Some I find quite important:
* Access logs - fields and format
* Ports to use for incoming traffic
* Infrastructure annotations \ labels
* Control Gateway API
* Default CRDs
* Default GatewayClass
* Default Gateways
It's always good to get a grip on the [default helm values](https://github.dev/traefik/traefik-helm-chart/blob/master/traefik/values.yaml)
The values file is used to customise Traefik, underlying pods, deployments and services as well as turning features on and off.
We can go ahead and install the controller to start with and use `helm upgrade` to make changes as we need to.
### Installation
Install:
```shell
CHART_VERSION="37.3.0" # traefik version v3.6.0
helm repo add traefik https://helm.traefik.io/traefik
helm repo update
helm search repo traefik --versions
helm install traefik traefik/traefik \
--version $CHART_VERSION \
--values kubernetes/gateway-api/traefik/values.yaml \
--namespace traefik \
--create-namespace
```
Upgrade:
```shell
helm upgrade traefik traefik/traefik \
--version $CHART_VERSION \
--values kubernetes/gateway-api/traefik/values.yaml \
--namespace traefik
```
Check our installation
```shell
# check the pods
kubectl -n traefik get pods
# check the logs
kubectl -n traefik logs -l app.kubernetes.io/instance=traefik-traefik
# port forward for access
kubectl -n traefik port-forward svc/traefik 80
```
## HTTP Traffic management
Feel free to quickly run through the basic [traffic management table](../README.md#traffic-management-features--http-routes) for using `HTTPRoute` routing for traffic.
Note: HTTPRoute features are not specific to this controller and should be available to any other gateway API controller that you choose.
## Middlewares
One of the key strengths of using Traefik is that they have a catalogue of middlewares that are plugins to perform popular common actions on traffic.
There are many [Middlewares](https://doc.traefik.io/traefik/reference/routing-configuration/http/middlewares/overview/) available.
The cool thing here is that these Middlewares are now pluggable into Gateway API HTTPRoutes.
In `HTTPRoute` objects, under `filters`, we can add extensions using something like:
```yaml
filters:
- type: ExtensionRef
extensionRef:
group: traefik.io
kind: Middleware
name: add-prefix
```
### Prefix
Note: To use `ExtensionRef` in Traefik you need to enable `providers.kubernetesCRD.enabled = true`. We have that enabled in our values.yaml file.
Expectation:
http://example-app.com/anything ππ½ http://go-svc:5000/prefix/anything
```shell
kubectl apply -f kubernetes/gateway-api/traefik/08-middleware-addprefix.yaml
```
We can follow the log of our upstream to see the HTTP request:
```shell
kubectl logs -f -l app=go-app
```
### Basic Auth
We can also add HTTP basic authentication to allow users to access secured resources protected by username and passwords.
The Basic Auth middleware points to a secret which supports Kubernetes basic auth secret type.
```shell
kubectl apply -f kubernetes/gateway-api/traefik/09-middleware-basicauth.yaml
```
### Headers
Traefik allows us to manipulate headers too using Middleware component called `headers`
We can use this to handle scenarios like CORS (Cross-Origin Resource Sharing)
Let's access our web app directly over `localhost` which makes a call to `example-app.com` and should be blocked by CORS.
```shell
kubectl port-forward svc/web-app 8000:80
```
Once we apply the Middleware update to our HTTPRoute, we can perform cross origin API calls:
```shell
kubectl apply -f kubernetes/gateway-api/traefik/10-middleware-headers.yaml
```
---
### Overview (`kubernetes/secrets/sealed-secrets/README.md`)
# Introduction to Sealed Secrets
Checkout the [Sealed Secrets GitHub Repo](https://github.com/bitnami-labs/sealed-secrets)
There are a number of use-cases where this is a really great concept.
1) GitOps - Storing your YAML manifests in Git and using GitOps tools to sync the manifests to your clusters (For example Flux and ArgoCD!)
2) Giving a team access to secrets without revealing the secret material.
developer: "I want to confirm my deployed secret value is X in the cluster"
developer can compare `sealedSecret` YAML in Git, with the `sealedSecret` in the cluster and confirm the value is the same.
## Create a kubernetes cluster
In this guide we we'll need a Kubernetes cluster for testing. Let's create one using [kind](https://kind.sigs.k8s.io/)
```
kind create cluster --name sealedsecrets --image kindest/node:v1.23.5
```
See cluster up and running:
```
kubectl get nodes
NAME STATUS ROLES AGE VERSION
sealedsecrets-control-plane Ready control-plane,master 2m12s v1.23.5
```
## Run a container to work in
### run Alpine Linux:
```
docker run -it --rm -v ${HOME}:/root/ -v ${PWD}:/work -w /work --net host alpine sh
```
### install kubectl
```
apk add --no-cache curl
curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
chmod +x ./kubectl
mv ./kubectl /usr/local/bin/kubectl
```
### install helm
```
curl -o /tmp/helm.tar.gz -LO https://get.helm.sh/helm-v3.10.1-linux-amd64.tar.gz
tar -C /tmp/ -zxvf /tmp/helm.tar.gz
mv /tmp/linux-amd64/helm /usr/local/bin/helm
chmod +x /usr/local/bin/helm
```
### test cluster access:
```
/work # kubectl get nodes
NAME STATUS ROLES AGE VERSION
sealedsecrets-control-plane Ready control-plane,master 3m26s v1.23.5
```
## Install Sealed Secret Controller
### download the YAML
In this demo we'll use version [0.19.1](https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.19.1/controller.yaml) of the sealed secrets controller downloaded from the
[Github releases](https://github.com/bitnami-labs/sealed-secrets/releases) page
```
curl -L -o ./kubernetes/secrets/sealed-secrets/controller-v0.19.1.yaml https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.19.1/controller.yaml
```
### install using Helm
You can also install the controller using `helm`
```
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm search repo sealed-secrets --versions
helm template sealed-secrets --version 2.7.0 -n kube-system sealed-secrets/sealed-secrets \
> ./kubernetes/secrets/sealed-secrets/controller-helm-v0.19.1.yaml
```
With `helm template` we can explore the YAML and then replace the `helm template` with `helm install`
to install the chart
### install using YAML manifest
```
kubectl apply -f kubernetes/secrets/sealed-secrets/controller-v0.19.1.yaml
```
### Check the installation
The controller deploys to the `kube-system` namespace by default.
```
kubectl -n kube-system get pods
```
Check the logs of the sealed secret controller
```
kubectl -n kube-system logs -l name=sealed-secrets-controller --tail -1
```
From the logs we can see that it writes the encryption key its going to use as a kubernetes secret
Example log:
```
2022/11/05 21:38:20 New key written to kube-system/sealed-secrets-keymwzn9
```
## Encryption keys
```
kubectl -n kube-system get secrets
kubectl -n kube-system get secret sealed-secrets-keygxlvg -o yaml
```
## Download KubeSeal
The same way we downloaded the sealed secrets controller from the [GitHub releases](https://github.com/bitnami-labs/sealed-secrets/releases) page,
we'll want to download kubeseal from the assets section
```
curl -L -o /tmp/kubeseal.tar.gz \
https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.19.1/kubeseal-0.19.1-linux-amd64.tar.gz
tar -xzf /tmp/kubeseal.tar.gz -C /tmp/
chmod +x /tmp/kubeseal
mv /tmp/kubeseal /usr/local/bin/
```
We can now run `kubeseal --help`
## Sealing a basic Kubernetes Secret
Looks at our existing Kubernetes secret YAML
```
cat kubernetes/secrets/secret.yaml
```
If you run `kubeseal` you will see it pause and expect input from `stdin`.
You can paste your secret YAML and press CTRL+D to terminate `stdin`.
You will notice it writes a `sealedSecret` to `stdout`.
We can then automate this using `|` characters.
Create a sealed secret using `stdin` :
```
cat kubernetes/secrets/secret.yaml | kubeseal -o yaml > kubernetes/secrets/sealed-secrets/sealed-secret.yaml
```
Create a sealed secret using a YAML file:
```
kubeseal -f kubernetes/secrets/secret.yaml -o yaml > kubernetes/secrets/sealed-secrets/sealed-secret.yaml
```
Deploy the sealed secret
```
kubectl apply -f kubernetes/secrets/sealed-secrets/sealed-secret.yaml
```
Now few seconds later, see the secret
```
kubectl -n default get secret
NAME TYPE DATA AGE
mysecret Opaque 1 25s
```
## How the encryption key is managed
By default the controller generates a key as we saw earlier and stores it in a Kubernetes secret.
By default, the controller will generate a new active key every 30 days.
It keeps old keys so it can decrypt previous encrypted sealed secrets and will use the active key with new encryption.
It's important to keep these keys secured.
When the controller starts it consumes all the secrets and will start using them
This means we can backup these keys in a Vault and use them to migrate our clusters if we wanted to.
We can also override the renewal period to increase or decrease the value. `0` turns it off
To showcase this I can set `--key-renew-period=
This guide is aimed to fast-track your Kubernetes learning by focusing on a practical hands-on overview guide.
When learning Kubernetes, you usually have an idea of some existing system you own and manage, or a website that you are building.
The challenge is understanding which Kubernetes building blocks you need in order to run your workloads on Kubernetes
Create a cluster with [kind](https://kind.sigs.k8s.io/docs/user/quick-start/)
```
kind create cluster --name rabbit --image kindest/node:v1.18.4
```
## Namespace
```
kubectl create ns rabbits
```
## Storage Class
```
kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
standard (default) rancher.io/local-path Delete WaitForFirstConsumer false 84s
```
## Deployment
```
kubectl apply -n rabbits -f .\kubernetes\rabbit-rbac.yaml
kubectl apply -n rabbits -f .\kubernetes\rabbit-configmap.yaml
kubectl apply -n rabbits -f .\kubernetes\rabbit-secret.yaml
kubectl apply -n rabbits -f .\kubernetes\rabbit-statefulset.yaml
```
## Access the UI
```
kubectl -n rabbits port-forward rabbitmq-0 8080:15672
```
Go to htttp://localhost:8080
---
### Overview (`monitoring/opentelemetry/kubernetes/README.md`)
# Introduction to OpenTelemetry Operator
OpenTelemetry Operator [documentation](https://opentelemetry.io/docs/platforms/kubernetes/operator/)
## We need a Kubernetes cluster
Lets create a Kubernetes cluster to play with using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/)
```
kind create cluster --name otel --image kindest/node:v1.34.0
```
Test our cluster:
```
kubectl get nodes
NAME STATUS ROLES AGE VERSION
otel-control-plane Ready control-plane 40s v1.34.0
```
## Helm charts
### OpenTelemetry Operator chart
```
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update
helm search repo open-telemetry --versions
```
We'll install version `0.93.1` at the time of this guide.
I would suggest to make sure the version you pick is compatible with the Kubernetes version you are running.
```
OTEL_VERSION=0.93.1
```
### Cert-manager chart
We will need cert-manager deployed which Otel uses for local TLS certificate management
```
helm repo add jetstack https://charts.jetstack.io
helm search repo jetstack --versions
```
We'll install version `v1.18.2` of cert-manager at the time of this guide which is compatible with our Kubernetes version.
```
CERTMANAGER_VERSION=v1.18.2
```
### Install helm charts
Install cert-manager:
```
helm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--version $CERTMANAGER_VERSION \
--set crds.enabled=true \
--set startupapicheck.timeout="5m"
```
Install OpenTelemetry Operator:
```
helm install opentelemetry-operator open-telemetry/opentelemetry-operator \
--namespace opentelemetry-operator-system \
--create-namespace \
--version $OTEL_VERSION \
--values=monitoring/opentelemetry/kubernetes/values.yaml
```
View our install:
```
kubectl get pods -n cert-manager
kubectl get pods -n opentelemetry-operator-system
```
## Create a Collector
In this guide I copied a starter collector from the [official documentation](https://opentelemetry.io/docs/platforms/kubernetes/operator/#getting-started) and changed some of the values.
To split out the guides, we'll have separate collectors for logs, metrics and traces.
Let's deploy our collectors in a central `monitoring` namespace:
```
kubectl create namespace monitoring
kubectl apply -n monitoring -f monitoring/opentelemetry/kubernetes/collector-tracing.yaml
```
View our collector:
```
kubectl -n monitoring get pods
kubectl -n monitoring get svc
```
## Tracing
### Instrumentation
To start receiving traces, we'll be using the OpenTelemetry [auto instrumentation](https://opentelemetry.io/docs/platforms/kubernetes/operator/automatic/) to inject the instrumentation into our Kubernetes pods.
To make use of the auto injection for Opentelemetry, we use the `Instrumentation` resources.
Note that I am creating this in the default namespace next to where my applications are.
```
kubectl apply -f monitoring/opentelemetry/kubernetes/instrumentation.yaml
```
### Deploy Microservices
Build applications:
```
docker compose --file monitoring/opentelemetry/docker-compose.yaml build
```
Load images into `kind`:
```
kind load docker-image aimvector/service-mesh:videos-web-1.0.0 --name otel
kind load docker-image aimvector/service-mesh:playlists-api-1.0.0 --name otel
kind load docker-image aimvector/service-mesh:videos-api-1.0.0 --name otel
```
```
kubectl apply -f monitoring/opentelemetry/applications/playlists-api/
kubectl apply -f monitoring/opentelemetry/applications/playlists-db/
kubectl apply -f monitoring/opentelemetry/applications/videos-web/
kubectl apply -f monitoring/opentelemetry/applications/videos-db/
kubectl apply -f monitoring/opentelemetry/applications/videos-api/
```
Generate some traffic with `port-forward`
```
kubectl port-forward svc/videos-web 80:80
kubectl port-forward svc/playlists-api 81:80
```
### Tracing Data Store
In this guide, we'll use a Tempo database for tracing data
```
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm search repo grafana/tempo
TEMPO_VERSION=1.23.3
helm install tempo grafana/tempo \
--create-namespace \
--namespace grafana \
--version $TEMPO_VERSION \
--values monitoring/opentelemetry/kubernetes/tempo.yaml
```
### Metrics Data Store
In this guide, we'll use a Prometheus for our metrics data
```
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm search repo prometheus-community --versions
PROMETHEUS_STACK_VERSION=77.5.0
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
--version ${PROMETHEUS_STACK_VERSION} \
--namespace prometheus-operator-system \
--create-namespace \
--set prometheusOperator.enabled=true \
--set prometheusOperator.nodeSelector."kubernetes\.io/os"=linux \
--set prometheusOperator.fullnameOverride="prometheus-operator" \
--set prometheusOperator.manageCrds=true \
--set alertmanager.enabled=false \
--set grafana.enabled=false \
--set prometheus-node-exporter.enabled=false \
--set nodeExporter.enabled=false \
--set kubeStateMetrics.enabled=false \
--set prometheus.enabled=false
kubectl -n prometheus-operator-system get pods
# Deploy our dedicated Prometheus for metrics storage
kubectl apply -n monitoring -f monitoring/opentelemetry/kubernetes/prometheus.yaml
```
### Dashboards
In this guide I use a simple Grafana for dashboards
```
helm search repo grafana/grafana
GRAFANA_VERSION=9.4.4
helm install grafana grafana/grafana \
--namespace grafana \
--version $GRAFANA_VERSION \
--values monitoring/opentelemetry/kubernetes/grafana.yaml
```
### Access Grafana
We can `port-forward` to Grafana
```
kubectl -n grafana port-forward svc/grafana 3000:80
```
---
### Overview (`monitoring/opentelemetry/metrics/README.md`)
# OpenTelemetry Metrics
## Build the applications
To build the applications navigate your terminal to the location of the compose file
```
cd monitoring/opentelemetry/
```
Build:
```
docker compose build
```
## Docker
Start up the stack using local docker compose, by running:
```
docker compose up
```
## Kubernetes
For Kubernetes, we will use the OpenTelemetry Operator.
See that guide over [here](../kubernetes/README.md)
## Generate from Traffic
You can generate some traffic by visiting the application on http://localhost
Note: You will need port 80 and 81 open for the applications to function
## Access Grafana
You can access Grafana on http://localhost:3000 to test the Tempo datasource.
---
### Overview (`monitoring/opentelemetry/tracing/README.md`)
# Tracing with OpenTelemetry
## Build the applications
To build the applications navigate your terminal to the location of the compose file
```
cd monitoring/opentelemetry/
```
Build:
```
docker compose build
```
## Docker
Start up the stack using local docker compose, by running:
```
docker compose up
```
## Kubernetes
For Kubernetes, we will use the OpenTelemetry Operator.
See that guide over [here](../kubernetes/README.md)
## Generate from Traffic
You can generate some traffic by visiting the application on http://localhost
Note: You will need port 80 and 81 open for the applications to function
## Access Grafana
You can access Grafana on http://localhost:3000 to test the Tempo datasource.
---
### Overview (`monitoring/prometheus/kubernetes/readme.md`)
# Kubernetes monitoring with Prometheus
Kubernetes [1.14.8](./1.14.8/readme.md) Hello, World!
" ``` To get this to work, we need to look at Pythons Library system, or `pip` ## External Packages If we look at the `from flask import Flask` statement, it assumes we have a `flask` dependency. Our previous import statements worked, because they are built-in libraries in python. Flask is an external package. External packages are managed by [pip](https://pypi.org/project/pip/) We may use `pip` to install `Flask`. If we need to connect to databases, we may install packages to do so. Like a `mysql` or a `redis` package. With `pip`, we define our dependencies in a `requirements.txt` file. We can check for the latest version of `Flask` [here](https://pypi.org/project/Flask/) We can also use `pip install` commands to install dependencies. However, this means we need a new `pip install` command for every dependency which will grow as our application and needs grow. Best is to use a `requirements.txt` file. Let's create a `requirements.txt` file: ``` Flask == 2.0.2 ``` We can install our dependencies using: ``` pip install -r src/requirements.txt ``` Let's implement the minimal example we posted above. To run our application, note that the `app.py` runs as per usual.Hello, get_customers!
" @app.route("/get/Hello, get!
" @app.route("/add", methods=['POST']) def add_customer(customer): return "Hello, add!
" ``` Now that we've built our URLs using routes, next we need to understand HTTP methods ## HTTP Methods There are a number of HTTP methods for web services, popular ones being `POST` and `GET` `GET` method is for general retrieval of data `POST` method is for passing data to our service Let's setup each of our routes with dedicated HTTP Methods ``` @app.route("/", methods=['GET']) def get_customers(): return "Hello, get_customers!
" @app.route("/get/Hello, get!
" @app.route("/add", methods=['POST']) def add_customer(): return "Hello, add!
" ``` Now we can see if we access `http://localhost:5000/add` We get: `Method Not Allowed` , because it only accepts `POST` and browser by default, does `GET` Let's fill out our routes ### Get Customers ``` @app.route("/", methods=['GET']) def get_customers(): customers = getCustomers() return json.dumps(customers) ``` ### Get Customer ``` @app.route("/get/
Code is over [here](../../../storage/redis/clustering/readme.md)
## Python Dev Environment
The same as Part 1+2+3+4, we start with a [dockerfile](./dockerfile) where we declare our version of `python`.
```
FROM python:3.9.6-alpine3.13 as dev
WORKDIR /work
```
Let's build and start our container:
```
cd python\introduction\part-5.database.redis
docker build --target dev . -t python
docker run -it -v ${PWD}:/work -p 5000:5000 --net redis python sh
/work # python --version
Python 3.9.6
```
## Our application
We're going to use what we've learnt in part 1,2 & 3 and create
our customer app that handles customer data
Firstly we have to import our dependencies:
```
import os.path
import csv
import json
from flask import Flask
from flask import request
```
Then we have a class to define what a customer looks like:
```
class Customer:
def __init__(self, c="",f="",l=""):
self.customerID = c
self.firstName = f
self.lastName = l
def fullName(self):
return self.firstName + " " + self.lastName
```
And also set a global variable for the location of our videos `json` file:
```
dataPath = "./customers.json"
```
Then we need a function which returns our customers:
```
def getCustomers():
if os.path.isfile(dataPath):
with open(dataPath, newline='') as customerFile:
data = customerFile.read()
customers = json.loads(data)
return customers
else:
return {}
```
Here is a function to return a specific customer:
```
def getCustomer(customerID):
customers = getCustomers()
if customerID in customers:
return customers[customerID]
else:
return {}
```
And finally a function for updating our customers:
```
def updateCustomers(customers):
with open(dataPath, 'w', newline='') as customerFile:
customerJSON = json.dumps(customers)
customerFile.write(customerJSON)
```
In the previous episode, we've created a `json` file to hold all our customers.
We've learnt how to read and write to file and temporarily use the file for our storage.
Let's create a file called `customers.json` :
```
{
"a": {
"customerID": "a",
"firstName": "James",
"lastName": "Baker"
},
"b": {
"customerID": "b",
"firstName": "Jonathan",
"lastName": "D"
},
"c": {
"customerID": "c",
"firstName": "Aleem",
"lastName": "Janmohamed"
},
"d": {
"customerID": "d",
"firstName": "Ivo",
"lastName": "Galic"
},
"e": {
"customerID": "e",
"firstName": "Joel",
"lastName": "Griffiths"
},
"f": {
"customerID": "f",
"firstName": "Michael",
"lastName": "Spinks"
},
"g": {
"customerID": "g",
"firstName": "Victor",
"lastName": "Savkov"
}
}
```
Now that we have our customer data and functions to read and update our customer data, let's define our `Flask` application:
```
app = Flask(__name__)
```
We create our route to get all customers:
```
@app.route("/", methods=['GET'])
def get_customers():
customers = getCustomers()
return json.dumps(customers)
```
A route to get one customer by ID:
```
@app.route("/get/
Documentation [here](https://redis.io/topics/replication)
### Configuration
```
#persistence
dir /data
dbfilename dump.rdb
appendonly yes
appendfilename "appendonly.aof"
```
### redis-0 Configuration
```
protected-mode no
port 6379
#authentication
masterauth a-very-complex-password-here
requirepass a-very-complex-password-here
```
### redis-1 Configuration
```
protected-mode no
port 6379
slaveof redis-0 6379
#authentication
masterauth a-very-complex-password-here
requirepass a-very-complex-password-here
```
### redis-2 Configuration
```
protected-mode no
port 6379
slaveof redis-0 6379
#authentication
masterauth a-very-complex-password-here
requirepass a-very-complex-password-here
```
```
# remember to update above in configs!
docker network create redis
cd .\storage\redis\clustering\
#redis-0
docker run -d --rm --name redis-0 `
--net redis `
-v ${PWD}/redis-0:/etc/redis/ `
redis:6.0-alpine redis-server /etc/redis/redis.conf
#redis-1
docker run -d --rm --name redis-1 `
--net redis `
-v ${PWD}/redis-1:/etc/redis/ `
redis:6.0-alpine redis-server /etc/redis/redis.conf
#redis-2
docker run -d --rm --name redis-2 `
--net redis `
-v ${PWD}/redis-2:/etc/redis/ `
redis:6.0-alpine redis-server /etc/redis/redis.conf
```
## Example Application
Run example application in video, to show application writing to the master
```
cd .\storage\redis\applications\client\
docker build . -t aimvector/redis-client:v1.0.0
docker run -it --net redis `
-e REDIS_HOST=redis-0 `
-e REDIS_PORT=6379 `
-e REDIS_PASSWORD="a-very-complex-password-here" `
-p 80:80 `
aimvector/redis-client:v1.0.0
```
## Test Replication
Technically written data should now be on the replicas
```
# go to one of the clients
docker exec -it redis-2 sh
redis-cli
auth "a-very-complex-password-here"
keys *
```
## Running Sentinels
Documentation [here](https://redis.io/topics/sentinel)
```
#********BASIC CONFIG************************************
port 5000
sentinel monitor mymaster redis-0 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1
sentinel auth-pass mymaster a-very-complex-password-here
#********************************************
```
Starting Redis in sentinel mode
```
cd .\storage\redis\clustering\
docker run -d --rm --name sentinel-0 --net redis `
-v ${PWD}/sentinel-0:/etc/redis/ `
redis:6.0-alpine `
redis-sentinel /etc/redis/sentinel.conf
docker run -d --rm --name sentinel-1 --net redis `
-v ${PWD}/sentinel-1:/etc/redis/ `
redis:6.0-alpine `
redis-sentinel /etc/redis/sentinel.conf
docker run -d --rm --name sentinel-2 --net redis `
-v ${PWD}/sentinel-2:/etc/redis/ `
redis:6.0-alpine `
redis-sentinel /etc/redis/sentinel.conf
docker logs sentinel-0
docker exec -it sentinel-0 sh
redis-cli -p 5000
info
sentinel master mymaster
# clean up
docker rm -f redis-0 redis-1 redis-2
docker rm -f sentinel-0 sentinel-1 sentinel-2
```
---
--- METRICS ---
- Files Extracted: 99
- Estimated Token Budget: ~125719 tokens
- Canonical Reference: https://codewiki.google/github.com/marcel-dempers/docker-development-youtube-series