homelab Documentation

File: README.md Copy Chapter

Khue's Homelab

Features • Get Started • Documentation

This project utilizes Infrastructure as Code and GitOps to automate provisioning, operating, and updating self-hosted services in my homelab.
It can be used as a highly customizable framework to build your own homelab.

What is a homelab?

Homelab is a laboratory at home where you can self-host, experiment with new technologies, practice for certifications, and so on.
For more information, please see the r/homelab introduction and the
Home Operations Discord community (formerly known as k8s-at-home).

If you encounter an issue, please create a bug report
(avoid asking for support about issues specific to this project in other communication channels).

Overview Copy Chapter

Project status: ALPHA

This project is still in the experimental stage, and I don't use anything critical on it.
Expect breaking changes that may require a complete redeployment.
A proper upgrade path is planned for the stable release.
More information can be found in the roadmap below.

Hardware Copy Section

• 4 × NEC SFF PC-MK26ECZDR (Japanese version of the ThinkCentre M700):
  • CPU: Intel Core i5-6600T @ 2.70GHz
  • RAM: 16GB
  • SSD: 128GB
• TP-Link TL-SG108 switch:
  • Ports: 8
  • Speed: 1000Mbps

Features Copy Section

• Common applications: Gitea, Jellyfin, Paperless...
• Automated bare metal provisioning with PXE boot
• Automated Kubernetes installation and management
• Installing and managing applications using GitOps
• Automatic rolling upgrade for OS and Kubernetes
• Automatically update apps (with approval)
• Modular architecture, easy to add or remove features/components
• Automated certificate management
• Automatically update DNS records for exposed services
• VPN (Tailscale or Wireguard)
• Expose services to the internet securely with Cloudflare Tunnel
• CI/CD platform
• Private container registry
• Distributed storage
• Support multiple environments (dev, prod)
• Monitoring and alerting
• Automated backup and restore
• Single sign-on
• Infrastructure testing

Some demo videos and screenshots are shown here.
They can't capture all the project's features, but they are sufficient to get a concept of it.

Demo
[
Deploy with a single command (after updating the configuration files)
[
PXE boot
Observe network traffic with Hubble, built on top of Cilium and eBPF
Homepage powered by... Homepage
Monitoring dashboard powered by Grafana
Git server powered by Gitea
Matrix chat server
Continuous integration with Woodpecker CI
Continuous deployment with ArgoCD
ntfy displaying received alerts
Self-hosted AI powered by Ollama (experimental, not very fast because I don't have a GPU)

Tech stack Copy Section

Get Started Copy Chapter

• Try it out locally without any hardware (just 4 commands!)
• Deploy on real hardware for production workload

Roadmap Copy Chapter

See roadmap and open issues for a list of proposed features and known issues.

Contributing Copy Chapter

Any contributions you make are greatly appreciated.

Please see contributing guide for more information.

License Copy Chapter

Copyright © 2020 - 2024 Khue Doan

Distributed under the GPLv3 License.
See license page or LICENSE.md file for more information.

Acknowledgements Copy Chapter

References:

• Ephemeral PXE server inspired by Minimal First Machine in the DC
• ArgoCD usage and monitoring configuration in locmai/humble
• README template
• Run the same Cloudflare Tunnel across many cloudflared processes
• MAC address environment variable in GRUB config
• Official k3s systemd service file
• Official Cloudflare Tunnel examples
• Initialize GitOps repository on Gitea and integrate with Tekton by RedHat
• SSO configuration from xUnholy/k8s-gitops
• Pre-commit config from k8s-at-home/flux-cluster-template
• Diátaxis technical documentation framework
• Official Terratest examples
• Self-host an automated Jellyfin media streaming stack
• App Template Helm chart by bjw-s
• Various application configurations in onedr0p/home-ops

Here is a list of the contributors who have helped to improve this project.
Big shout-out to them!

• @locmai
• @MatthewJohn
• @karpfediem
• @linhng98
• @elliotblackburn
• @dotdiego
• @Crimrose
• @eventi
• @Bourne-ID
• @akwan
• @trangmaiq
• @tangowithfoxtrot
• @raedkit
• @ClashTheBunny
• @retX0
• @zalsader
• @serpro69
• @llajas
• @zalsader

If you feel you're missing from this list, please feel free to add yourself in a PR.

Stargazers over time Copy Chapter

File: docs/concepts/certificate-management.md Copy Chapter

Certificate management

Certificates are generated and managed by cert-manager with Let's Encrypt.
By default certificates are valid for 90 days and will be renewed after 60 days.

cert-manager watches Ingress resources across the cluster. When you create an Ingress with a supported annotation:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  name: foo
spec:
  rules:
    - host: foo.example.com
      # ...
  tls:
    - hosts:
        - foo.example.com
      secretName: foo-tls-certificate

flowchart LR
  User -- 6 --> Ingress

  subgraph cluster[Homelab cluster]
    Ingress --- Secret
    Ingress -. 1 .-> Certificate
    Certificate -. 5 .-> Secret
    Certificate -- 2 --> CertificateRequest -- 3 --> Order -- 4 --> Challenge
  end

  Order -.- ACMEServer[ACME server]

  subgraph dnsprovider[DNS provider]
    TXT
  end

  Challenge -- 4.a --> TXT
  ACMEServer -.- Challenge
  ACMEServer -. 4.b .-> TXT

1. cert-manager creates a corresponding Certificate resources
2. Based on the Certificate resource, cert-manager creates a CertificateRequest resource to request a signed certificate from the configured ClusterIssuer
3. The CertificateRequest will create an order with an ACME server (we use Let's Encrypt), which is represented by the Order resource
4. Then cert-manager will perform a DNS-01 Challenge:
   1. Create a DNS TXT record (contains a computed key)
   2. The ACME server retrieve this key via a DNS lookup and validate that we own the domain for the requested certificate
5. cert-manager stores the certificate (typically tls.crt and tls.key) in the Secret specified in the Ingress configuration
6. Now you can access the HTTPS website with a valid certificate

A much more detailed diagram can be found in the official documentation under certificate lifecycle.

File: docs/concepts/development-shell.md Copy Chapter

Development shell

The development shell makes it easy to get all of the dependencies needed to interact with the homelab.

Prerequisites Copy Chapter

!!! info

NixOS users can skip this step.

Install Nix using one of the following methods:

• Official Nix installer
• Determinate Nix Installer

If you're using the official installer, add the following to your
~/.config/nix/nix.conf to enable Flakes:

experimental-features = nix-command flakes

How to open it Copy Chapter

Run the following command:

nix develop

It will open a shell with all the dependencies defined in ./flake.nix:

[khuedoan@ryzentower:~/Documents/homelab]$ which kubectl
/nix/store/0558zzzqynzw7rx9dp2i7jymvznd1cqx-kubectl-1.30.1/bin/kubectl

!!! tip

If you have [`direnv`](https://direnv.net) installed, you can run `direnv
allow` once and it will automatically enter the Nix shell every time you
`cd` into the project.

File: docs/concepts/pxe-boot.md Copy Chapter

PXE boot

flowchart TD
  subgraph controller[Initial controller]
    Ansible
    dhcp[DHCP server]
    tftp[TFTP server]
    http[HTTP server]
  end

  machine[Bare metal machine]

  Ansible -. 1 .-> machine
  machine <-. 2, 3 .-> dhcp
  machine <-. 4, 5 .-> tftp
  machine <-. 6, 7 .-> http

1. Ansible: Hey MAC address xx:xx:xx:xx:xx:xx, wake up!
2. Machine: Hello everyone, I just woke up in network mode, could someone please show me how to boot?
3. DHCP server: I hear you, here's your IP address, proceed to the next server to obtain your bootloader.
4. Machine: Hello, could you please send me my bootloader?
5. TFTP server: Here you go. Grab your boot configuration, kernel, and initial ramdisk as well.
6. Machine: Hi, I just booted into my bootloader, and my boot parameters instructed me to get the installation instructions, packages, etc. from this site.
7. HTTP server: It's all yours.
8. Machine: Great, now I can install the OS and reboot!

Here's how it looks like in action:

File: docs/concepts/secrets-management.md Copy Chapter

Secrets management

Overview Copy Chapter

• Global secrets are stored in the global-secrets namespace.
• Integrate with GitOps using External Secrets Operator.
• Secrets that can be generated are automatically generated and stored in the global-secrets namespace.

!!! info

Despite the name _External_ Secrets Operator, global secrets are created in the same cluster and synced
to other namespaces using the [Kubernetes provider](https://external-secrets.io/latest/provider/kubernetes).

While not supported by default in this project, you can also use other external providers such as HashiCorp Vault,
AWS Secret Manager, Google Cloud Secret Manager, Azure Key Vault, 1Password, etc.

flowchart TD
  subgraph global-secrets-namespace[global-secrets namespace]
    secret-generator[Secret Generator] -- generate if not exist --> source-secrets[Source Secrets]
  end

  subgraph app-namespace[application namespace]
    ExternalSecret -- create --> Secret
    App -- read --> Secret
  end

  ClusterSecretStore -- read --> source-secrets
  ExternalSecret --- ClusterSecretStore

Randomly generated secrets Copy Chapter

This is useful when you want to generate random secrets like admin password and store in global secrets.

--8<--
./platform/global-secrets/files/secret-generator/config.yaml
--8<--

Extra third-party secrets Copy Chapter

For third-party secrets that you don't control, add them to external/terraform.tfvars under the extra_secrets key,
then run make external.

They will be available as a Secret named external in the global-secrets namespace.
You can use it with ExternalSecret just like any other global secret.

How secrets are pulled from global secrets to other namespaces Copy Chapter

When you apply an ExternalSecret object, for example:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: gitea-admin-secret
  namespace: gitea
spec:
  data:
  - remoteRef:
      conversionStrategy: Default
      key: gitea.admin
      property: password
    secretKey: password
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: global-secrets
  target:
    creationPolicy: Owner
    deletionPolicy: Retain
    template:
      data:
        password: '{{ .password }}'
        username: gitea_admin
      engineVersion: v2

This will create a corresponding Kubernetes secret:

kubectl describe secrets -n gitea gitea-admin-secret

Name:         gitea-admin-secret
Namespace:    gitea
Labels:       <none>
Annotations:  reconcile.external-secrets.io/data-hash: <REDACTED>

Type:  Opaque

Data
====
password:  32 bytes
username:  11 bytes

Please see the official documentation for more information:

• External Secrets Operator
• API specification

File: docs/concepts/testing.md Copy Chapter

Testing infrastructure code

We use Terratest for automated tests.
The tests are written in Go and can be found at ./test.

TODO: more docs here (PR welcomed)

File: docs/getting-started/install-pre-commit-hooks.md Copy Chapter

Install pre-commit hooks

Git hook scripts are useful for identifying simple issues before commiting changes.

Install pre-commit first, one-liner for Arch users:

sudo pacman -S python-pre-commit

Then install git hook scripts:

make git-hooks

If you want to enable pre-commit on all repositories without enabling it manually, see automatically enabling pre-commit on repositories.

File: docs/getting-started/user-onboarding.md Copy Chapter

User onboarding

=== "For user"

## Create user

Ask an admin to create your account, provide the following information:

- [ ] Full name (John Doe)
- [ ] Select a username (`johndoe`)
- [ ] Email address (`[email protected]`)

## Install companion apps

For all users:

- [ ] A password manager (I personally recommend [Bitwarden](https://bitwarden.com/download))
- [ ] A [Matrix chat client](https://matrix.org/clients) (optional, you can use the web version)

For technical users:

- [ ] [Docker](https://docs.docker.com/engine/install)
- [ ] [Nix](https://nixos.org/download) and [direnv](https://direnv.net) (optional, but highly recommended)
- [ ] [Lens](https://k8slens.dev) (optional, you can use the included `kubectl` or `k9s` command in the tools container)

=== "For admin"

Run the following script:

```sh
./scripts/onboard-user johndoe "John Doe" "[email protected]"
```

Let the user scan the QR code or follow the link to set up passkeys or password + TOTP.

File: docs/how-to-guides/troubleshooting/pxe-boot.md Copy Chapter

PXE boot

PXE server logs Copy Chapter

To view PXE server (includes DHCP, TFTP and HTTP server) logs:

./scripts/pxe-logs

!!! tip

You can view the logs of one or more containers selectively, for example:

```sh
./scripts/pxe-logs dnsmasq
./scripts/pxe-logs http
```

Nodes not booting from the network Copy Chapter

• Plug a monitor and a keyboard to one of the bare metal node if possible to make the debugging process easier
• Check if the controller (PXE server) is on the same subnet with bare metal nodes (sometimes Wifi will not work or conflict with wired Ethernet, try to turn it off)
• Check if bare metal nodes are configured to boot from the network
• Check if Wake-on-LAN is enabled
• Check if the operating system ISO file is mounted
• Check the controller's firewall config to make sure that the following ports are open:
  • DHCP (67/68)
  • TFTP (69)
  • HTTP (80)
• Check PXE server Docker logs
• Check if the servers are booting to the correct OS (Fedora Server installer instead of the previously installed OS), if not try to select it manually or remove the previous OS boot entry
• Examine the network boot process with Wireshark or Termshark