## File: README.md # Khue's Homelab **[Features](#features) • [Get Started](#get-started) • [Documentation](https://homelab.khuedoan.com)** [](https://github.com/khuedoan/homelab/tags) [](https://homelab.khuedoan.com) [](https://www.gnu.org/licenses/gpl-3.0.html) [](https://github.com/khuedoan/homelab) This project utilizes [Infrastructure as Code](https://en.wikipedia.org/wiki/Infrastructure_as_code) and [GitOps](https://www.weave.works/technologies/gitops) to automate provisioning, operating, and updating self-hosted services in my homelab. It can be used as a highly customizable framework to build your own homelab. > **What is a homelab?** > > Homelab is a laboratory at home where you can self-host, experiment with new technologies, practice for certifications, and so on. > For more information, please see the [r/homelab introduction](https://www.reddit.com/r/homelab/wiki/introduction) and the > [Home Operations Discord community](https://discord.gg/home-operations) (formerly known as [k8s-at-home](https://k8s-at-home.com)). If you encounter an issue, please create [a bug report](https://github.com/khuedoan/homelab/issues/new?template=bug_report.md) (avoid asking for support about issues specific to this project in other communication channels). ## Overview Project status: **ALPHA** This project is still in the experimental stage, and I don't use anything critical on it. Expect breaking changes that may require a complete redeployment. A proper upgrade path is planned for the stable release. More information can be found in [the roadmap](#roadmap) below. ### Hardware - 4 × NEC SFF `PC-MK26ECZDR` (Japanese version of the ThinkCentre M700): - CPU: `Intel Core i5-6600T @ 2.70GHz` - RAM: `16GB` - SSD: `128GB` - TP-Link `TL-SG108` switch: - Ports: `8` - Speed: `1000Mbps` ### Features - [x] Common applications: Gitea, Jellyfin, Paperless... - [x] Automated bare metal provisioning with PXE boot - [x] Automated Kubernetes installation and management - [x] Installing and managing applications using GitOps - [x] Automatic rolling upgrade for OS and Kubernetes - [x] Automatically update apps (with approval) - [x] Modular architecture, easy to add or remove features/components - [x] Automated certificate management - [x] Automatically update DNS records for exposed services - [x] VPN (Tailscale or Wireguard) - [x] Expose services to the internet securely with [Cloudflare Tunnel](https://www.cloudflare.com/products/tunnel/) - [x] CI/CD platform - [x] Private container registry - [x] Distributed storage - [x] Support multiple environments (dev, prod) - [x] Monitoring and alerting - [x] Automated backup and restore - [x] Single sign-on - [x] Infrastructure testing Some demo videos and screenshots are shown here. They can't capture all the project's features, but they are sufficient to get a concept of it. | Demo | | :--: | | [ | | Deploy with a single command (after updating the configuration files) | | [ | | PXE boot | | [![][hubble-demo]][hubble-demo] | | Observe network traffic with Hubble, built on top of [Cilium](https://cilium.io) and eBPF | | [![][homepage-demo]][homepage-demo] | | Homepage powered by... [Homepage](https://gethomepage.dev) | | [![][grafana-demo]][grafana-demo] | | Monitoring dashboard powered by [Grafana](https://grafana.com) | | [![][gitea-demo]][gitea-demo] | | Git server powered by [Gitea](https://gitea.io/en-us) | | [![][matrix-demo]][matrix-demo] | | [Matrix](https://matrix.org/) chat server | | [![][woodpecker-demo]][woodpecker-demo] | | Continuous integration with [Woodpecker CI](https://woodpecker-ci.org) | | [![][argocd-demo]][argocd-demo] | | Continuous deployment with [ArgoCD](https://argoproj.github.io/cd) | | [![][alert-demo]][alert-demo] | | [ntfy](https://ntfy.sh) displaying received alerts | | [![][ai-demo]][ai-demo] | | Self-hosted AI powered by [Ollama](https://ollama.com) (experimental, not very fast because I don't have a GPU) | [deploy-demo]: https://asciinema.org/a/xkBRkwC6e9RAzVuMDXH3nGHp7.svg [pxe-demo]: https://user-images.githubusercontent.com/27996771/157303477-df2e7410-8f02-4648-a86c-71e6b7e89e35.png [hubble-demo]: https://github.com/khuedoan/homelab/assets/27996771/9c6677d0-3564-47c0-852b-24b6a554b4a3 [homepage-demo]: https://github.com/khuedoan/homelab/assets/27996771/d0eaf620-be08-48d8-8420-40bcaa86093b [grafana-demo]: https://github.com/khuedoan/homelab/assets/27996771/ad937b26-e9bc-4761-83ae-1c7f512ea97f [gitea-demo]: https://github.com/khuedoan/homelab/assets/27996771/c245534f-88d9-4565-bde8-b39f60ccee9e [matrix-demo]: https://user-images.githubusercontent.com/27996771/149448510-7163310c-2049-4ccd-901d-f11f605bfc32.png [woodpecker-demo]: https://github.com/khuedoan/homelab/assets/27996771/5d887688-d20a-44c8-8f77-0c625527dfe4 [argocd-demo]: https://github.com/khuedoan/homelab/assets/27996771/527e2529-4fe1-4664-ab8a-b9eb3c492d20 [alert-demo]: https://github.com/khuedoan/homelab/assets/27996771/c922f755-e911-4ca0-9d4a-6e552d387f18 [ai-demo]: https://github.com/khuedoan/homelab/assets/27996771/d77ba511-00b7-47c3-9032-55679a099e70 ### Tech stack | Logo | Name | Description | | --- | --- | --- | | | [Ansible](https://www.ansible.com) | Automate bare metal provisioning and configuration | | | [ArgoCD](https://argoproj.github.io/cd) | GitOps tool built to deploy applications to Kubernetes | | | [cert-manager](https://cert-manager.io) | Cloud native certificate management | | | [Cilium](https://cilium.io) | eBPF-based Networking, Observability and Security (CNI, LB, Network Policy, etc.) | | | [Cloudflare](https://www.cloudflare.com) | DNS and Tunnel | | | [Docker](https://www.docker.com) | Ephemeral PXE server | | | [ExternalDNS](https://github.com/kubernetes-sigs/external-dns) | Synchronizes exposed Kubernetes Services and Ingresses with DNS providers | | | [Fedora Server](https://getfedora.org/en/server) | Base OS for Kubernetes nodes | | | [Gitea](https://gitea.com) | Self-hosted Git service | | | [Grafana](https://grafana.com) | Observability platform | | | [Helm](https://helm.sh) | The package manager for Kubernetes | | | [K3s](https://k3s.io) | Lightweight distribution of Kubernetes | | | [Kanidm](https://kanidm.com) | Modern and simple identity management platform | | | [Kubernetes](https://kubernetes.io) | Container-orchestration system, the backbone of this project | | | [Loki](https://grafana.com/oss/loki) | Log aggregation system | | | [NGINX](https://www.nginx.com) | Kubernetes Ingress Controller | | | [Nix](https://nixos.org) | Convenient development shell | | | [ntfy](https://ntfy.sh) | Notification service to send notifications to your phone or desktop | | | [Prometheus](https://prometheus.io) | Systems monitoring and alerting toolkit | | | [Renovate](https://www.whitesourcesoftware.com/free-developer-tools/renovate) | Automatically update dependencies | | | [Rook Ceph](https://rook.io) | Cloud-Native Storage for Kubernetes | | | [Tailscale](https://tailscale.com) | VPN without port forwarding | | | [Wireguard](https://www.wireguard.com) | Fast, modern, secure VPN tunnel | | | [Woodpecker CI](https://woodpecker-ci.org) | Simple yet powerful CI/CD engine with great extensibility | | | [Zot Registry](https://zotregistry.dev) | Private container registry | ## Get Started - [Try it out locally](https://homelab.khuedoan.com/installation/sandbox) without any hardware (just 4 commands!) - [Deploy on real hardware](https://homelab.khuedoan.com/installation/production/prerequisites) for production workload ## Roadmap See [roadmap](https://homelab.khuedoan.com/reference/roadmap) and [open issues](https://github.com/khuedoan/homelab/issues) for a list of proposed features and known issues. ## Contributing Any contributions you make are greatly appreciated. Please see [contributing guide](https://homelab.khuedoan.com/reference/contributing) for more information. ## License Copyright © 2020 - 2024 Khue Doan Distributed under the GPLv3 License. See [license page](https://homelab.khuedoan.com/reference/license) or `LICENSE.md` file for more information. ## Acknowledgements References: - [Ephemeral PXE server inspired by Minimal First Machine in the DC](https://speakerdeck.com/amcguign/minimal-first-machine-in-the-dc) - [ArgoCD usage and monitoring configuration in locmai/humble](https://github.com/locmai/humble) - [README template](https://github.com/othneildrew/Best-README-Template) - [Run the same Cloudflare Tunnel across many `cloudflared` processes](https://developers.cloudflare.com/cloudflare-one/tutorials/many-cfd-one-tunnel) - [MAC address environment variable in GRUB config](https://askubuntu.com/questions/1272400/how-do-i-automate-network-installation-of-many-ubuntu-18-04-systems-with-efi-and) - [Official k3s systemd service file](https://github.com/k3s-io/k3s/blob/master/k3s.service) - [Official Cloudflare Tunnel examples](https://github.com/cloudflare/argo-tunnel-examples) - [Initialize GitOps repository on Gitea and integrate with Tekton by RedHat](https://github.com/redhat-scholars/tekton-tutorial/tree/master/triggers) - [SSO configuration from xUnholy/k8s-gitops](https://github.com/xUnholy/k8s-gitops) - [Pre-commit config from k8s-at-home/flux-cluster-template](https://github.com/k8s-at-home/flux-cluster-template) - [Diátaxis technical documentation framework](https://diataxis.fr) - [Official Terratest examples](https://github.com/gruntwork-io/terratest/tree/master/test) - [Self-host an automated Jellyfin media streaming stack](https://zerodya.net/self-host-jellyfin-media-streaming-stack) - [App Template Helm chart by bjw-s](https://bjw-s-labs.github.io/helm-charts/docs/app-template) - [Various application configurations in onedr0p/home-ops](https://github.com/onedr0p/home-ops) Here is a list of the contributors who have helped to improve this project. Big shout-out to them! - [@locmai](https://github.com/locmai) - [@MatthewJohn](https://github.com/MatthewJohn) - [@karpfediem](https://github.com/karpfediem) - [@linhng98](https://github.com/linhng98) - [@elliotblackburn](https://github.com/elliotblackburn) - [@dotdiego](https://github.com/dotdiego) - [@Crimrose](https://github.com/Crimrose) - [@eventi](https://github.com/eventi) - [@Bourne-ID](https://github.com/Bourne-ID) - [@akwan](https://github.com/akwan) - [@trangmaiq](https://github.com/trangmaiq) - [@tangowithfoxtrot](https://github.com/tangowithfoxtrot) - [@raedkit](https://github.com/raedkit) - [@ClashTheBunny](https://github.com/ClashTheBunny) - [@retX0](https://github.com/retX0) - [@zalsader](https://github.com/zalsader) - [@serpro69](https://github.com/serpro69) - [@llajas](https://github.com/llajas) - [@zalsader](https://github.com/zalsader) If you feel you're missing from this list, please feel free to add yourself in a PR. ## Stargazers over time [](https://starchart.cc/khuedoan/homelab) --- ## File: docs/concepts/certificate-management.md # Certificate management Certificates are generated and managed by [cert-manager](https://cert-manager.io) with [Let's Encrypt](https://letsencrypt.org). By default certificates are valid for 90 days and will be renewed after 60 days. cert-manager watches `Ingress` resources across the cluster. When you create an `Ingress` with a [supported annotation](https://cert-manager.io/docs/usage/ingress/#supported-annotations): ```yaml hl_lines="5 13 14" apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: cert-manager.io/cluster-issuer: letsencrypt-prod name: foo spec: rules: - host: foo.example.com # ... tls: - hosts: - foo.example.com secretName: foo-tls-certificate ``` ```mermaid flowchart LR User -- 6 --> Ingress subgraph cluster[Homelab cluster] Ingress --- Secret Ingress -. 1 .-> Certificate Certificate -. 5 .-> Secret Certificate -- 2 --> CertificateRequest -- 3 --> Order -- 4 --> Challenge end Order -.- ACMEServer[ACME server] subgraph dnsprovider[DNS provider] TXT end Challenge -- 4.a --> TXT ACMEServer -.- Challenge ACMEServer -. 4.b .-> TXT ``` 1. cert-manager creates a corresponding `Certificate` resources 2. Based on the `Certificate` resource, cert-manager creates a `CertificateRequest` resource to request a signed certificate from the configured `ClusterIssuer` 3. The `CertificateRequest` will create an order with an ACME server (we use Let's Encrypt), which is represented by the `Order` resource 4. Then cert-manager will perform a [DNS-01](https://cert-manager.io/docs/configuration/acme/dns01) `Challenge`: 1. Create a DNS TXT record (contains a computed key) 2. The ACME server retrieve this key via a DNS lookup and validate that we own the domain for the requested certificate 7. cert-manager stores the certificate (typically `tls.crt` and `tls.key`) in the `Secret` specified in the `Ingress` configuration 8. Now you can access the HTTPS website with a valid certificate A much more detailed diagram can be found in the official documentation under [certificate lifecycle](https://cert-manager.io/docs/concepts/certificate/#certificate-lifecycle). --- ## File: docs/concepts/development-shell.md # Development shell The development shell makes it easy to get all of the dependencies needed to interact with the homelab. ## Prerequisites !!! info NixOS users can skip this step. Install Nix using one of the following methods: - [Official Nix installer](https://nixos.org/download) - [Determinate Nix Installer](https://docs.determinate.systems/getting-started/#installer) If you're using the official installer, add the following to your `~/.config/nix/nix.conf` to enable [Flakes](https://nixos.wiki/wiki/Flakes): ```conf experimental-features = nix-command flakes ``` ## How to open it Run the following command: ```sh nix develop ``` It will open a shell with all the dependencies defined in `./flake.nix`: ``` [khuedoan@ryzentower:~/Documents/homelab]$ which kubectl /nix/store/0558zzzqynzw7rx9dp2i7jymvznd1cqx-kubectl-1.30.1/bin/kubectl ``` !!! tip If you have [`direnv`](https://direnv.net) installed, you can run `direnv allow` once and it will automatically enter the Nix shell every time you `cd` into the project. --- ## File: docs/concepts/pxe-boot.md # PXE boot ```mermaid flowchart TD subgraph controller[Initial controller] Ansible dhcp[DHCP server] tftp[TFTP server] http[HTTP server] end machine[Bare metal machine] Ansible -. 1 .-> machine machine <-. 2, 3 .-> dhcp machine <-. 4, 5 .-> tftp machine <-. 6, 7 .-> http ``` 1. Ansible: Hey MAC address `xx:xx:xx:xx:xx:xx`, wake up! 2. Machine: Hello everyone, I just woke up in network mode, could someone please show me how to boot? 3. DHCP server: I hear you, here's your IP address, proceed to the next server to obtain your bootloader. 4. Machine: Hello, could you please send me my bootloader? 5. TFTP server: Here you go. Grab your boot configuration, kernel, and initial ramdisk as well. 6. Machine: Hi, I just booted into my bootloader, and my boot parameters instructed me to get the installation instructions, packages, etc. from this site. 7. HTTP server: It's all yours. 8. Machine: Great, now I can install the OS and reboot! Here's how it looks like in action: --- ## File: docs/concepts/secrets-management.md # Secrets management ## Overview - Global secrets are stored in the `global-secrets` namespace. - Integrate with GitOps using [External Secrets Operator](https://external-secrets.io). - Secrets that can be generated are automatically generated and stored in the `global-secrets` namespace. !!! info Despite the name _External_ Secrets Operator, global secrets are created in the same cluster and synced to other namespaces using the [Kubernetes provider](https://external-secrets.io/latest/provider/kubernetes). While not supported by default in this project, you can also use other external providers such as HashiCorp Vault, AWS Secret Manager, Google Cloud Secret Manager, Azure Key Vault, 1Password, etc. ```mermaid flowchart TD subgraph global-secrets-namespace[global-secrets namespace] secret-generator[Secret Generator] -- generate if not exist --> source-secrets[Source Secrets] end subgraph app-namespace[application namespace] ExternalSecret -- create --> Secret App -- read --> Secret end ClusterSecretStore -- read --> source-secrets ExternalSecret --- ClusterSecretStore ``` ## Randomly generated secrets This is useful when you want to generate random secrets like admin password and store in global secrets. ```yaml title="./platform/global-secrets/files/secret-generator/config.yaml" hl_lines="2-6" --8<-- ./platform/global-secrets/files/secret-generator/config.yaml --8<-- ``` ## Extra third-party secrets For third-party secrets that you don't control, add them to `external/terraform.tfvars` under the `extra_secrets` key, then run `make external`. They will be available as a Secret named `external` in the `global-secrets` namespace. You can use it with `ExternalSecret` just like any other global secret. ## How secrets are pulled from global secrets to other namespaces When you apply an `ExternalSecret` object, for example: ```yaml hl_lines="4 21-23" apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: gitea-admin-secret namespace: gitea spec: data: - remoteRef: conversionStrategy: Default key: gitea.admin property: password secretKey: password refreshInterval: 1h secretStoreRef: kind: ClusterSecretStore name: global-secrets target: creationPolicy: Owner deletionPolicy: Retain template: data: password: '{{ .password }}' username: gitea_admin engineVersion: v2 ``` This will create a corresponding Kubernetes secret: `kubectl describe secrets -n gitea gitea-admin-secret` ```yaml hl_lines="1 8-11" Name: gitea-admin-secret Namespace: gitea Labels: Annotations: reconcile.external-secrets.io/data-hash: Type: Opaque Data ==== password: 32 bytes username: 11 bytes ``` Please see the official documentation for more information: - [External Secrets Operator](https://external-secrets.io) - [API specification](https://external-secrets.io/latest/spec) --- ## File: docs/concepts/testing.md # Testing infrastructure code We use [Terratest](https://terratest.gruntwork.io) for automated tests. The tests are written in Go and can be found at `./test`. TODO: more docs here (PR welcomed) --- ## File: docs/getting-started/install-pre-commit-hooks.md # Install pre-commit hooks Git hook scripts are useful for identifying simple issues before commiting changes. Install [pre-commit](https://pre-commit.com/#install) first, one-liner for Arch users: ```sh sudo pacman -S python-pre-commit ``` Then install git hook scripts: ```sh make git-hooks ``` If you want to enable pre-commit on all repositories without enabling it manually, see [automatically enabling pre-commit on repositories](https://pre-commit.com/#automatically-enabling-pre-commit-on-repositories). --- ## File: docs/getting-started/user-onboarding.md # User onboarding === "For user" ## Create user Ask an admin to create your account, provide the following information: - [ ] Full name (John Doe) - [ ] Select a username (`johndoe`) - [ ] Email address (`johndoe@example.com`) ## Install companion apps For all users: - [ ] A password manager (I personally recommend [Bitwarden](https://bitwarden.com/download)) - [ ] A [Matrix chat client](https://matrix.org/clients) (optional, you can use the web version) For technical users: - [ ] [Docker](https://docs.docker.com/engine/install) - [ ] [Nix](https://nixos.org/download) and [direnv](https://direnv.net) (optional, but highly recommended) - [ ] [Lens](https://k8slens.dev) (optional, you can use the included `kubectl` or `k9s` command in the tools container) === "For admin" Run the following script: ```sh ./scripts/onboard-user johndoe "John Doe" "johndoe@example.com" ``` Let the user scan the QR code or follow the link to set up passkeys or password + TOTP. --- ## File: docs/how-to-guides/troubleshooting/pxe-boot.md # PXE boot ## PXE server logs To view PXE server (includes DHCP, TFTP and HTTP server) logs: ```sh ./scripts/pxe-logs ``` !!! tip You can view the logs of one or more containers selectively, for example: ```sh ./scripts/pxe-logs dnsmasq ./scripts/pxe-logs http ``` ## Nodes not booting from the network - Plug a monitor and a keyboard to one of the bare metal node if possible to make the debugging process easier - Check if the controller (PXE server) is on the same subnet with bare metal nodes (sometimes Wifi will not work or conflict with wired Ethernet, try to turn it off) - Check if bare metal nodes are configured to boot from the network - Check if Wake-on-LAN is enabled - Check if the operating system ISO file is mounted - Check the controller's firewall config to make sure that the following ports are open: - DHCP (67/68) - TFTP (69) - HTTP (80) - Check PXE server Docker logs - Check if the servers are booting to the correct OS (Fedora Server installer instead of the previously installed OS), if not try to select it manually or remove the previous OS boot entry - Examine the network boot process with [Wireshark](https://www.wireshark.org) or [Termshark](https://termshark.io)