homelab

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

★ 9 Open GitHub ↗

GitOps Kubernetes cluster for experiments

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Homelab

K3s cluster running self-hosted services.

>[!NOTE]
> You can find detailed write-ups on my blog:  
> [K3s initial setup](https://gogov.dev/blog/homelab-initial-setup)  
> [Adding Observability with Prometheus & Grafana](https://gogov.dev/blog/homelab-observability)  
> [GitOps, FluxCD Edition](https://gogov.dev/blog/homelab-gitops-fluxcd)  
> [Moving toward virtualization and other design decisions](https://gogov.dev/blog/design-decisions)  
> [Manual to Makefile - Terraform, KVM, Ansible](https://gogov.dev/blog/homelab-terraform-libvirt)  
> [The Complete Pipeline - End-to-end IaC GitOps](https://gogov.dev/blog/end-to-end-iac-gitops)  
> [Implementing SOPS - GitOps secrets management](https://gogov.dev/blog/sops-secret-management)  
> [Networking Overhaul & Production Migration](https://gogov.dev/blog/homelab-networking-overhaul)  
> [NAS Introduction and more networking issues](https://gogov.dev/blog/homelab-nas-introduction)  
> [Networking: A Retrospective in Pain](https://gogov.dev/blog/homelab-networking-overview)  
> [Diagnosing an Intel i219-V Hardware Hang](https://gogov.dev/blog/diagnosing-hardware-hang)  
> (WIP) [Kyverno implementation and policies]()

## Setup

> [!WARNING]
> This setup is no longer easily reproducible as it requires external dependencies. Although still
> very much doable, it requires significant effort and multiple services outside the cluster itself - **NAS Storage**, **Cloudflare Account**, and a **Personal Domain with DNS managed by Cloudflare**.

### Prerequisites:

> [!IMPORTANT]
> The following is expected:  
> **Production** host permanent IP: **192.168.0.111**  
> **Staging** host permanent IP: **192.168.0.109** (optional, if setting up dual environments)  
> **NAS** permanent IP: **192.168.0.104** (can be replicated without external provisioner, but requires tweaking)  
> If your setup differs in any way, you should adjust those values in all relevant places - deployments, persistent volume claims, env script etc.
1. **Prepare Fedora Host:** 
    - Install a fresh Fedora (KDE and Server tested)
    - Connected via Ethernet (required for bridge)
    - Ensure SSH is active and your key is added
    - Ensure virtualization is enabled at BIOS level
2. **GitHub PAT**: 
    - Provide your GitHub PAT in: **provisioning/ansible/roles/flux/tasks/main.yaml**   
(preferably via Ansible Vault secret as currently implemented)
3. **Terraform Variables**: 
    - Provide/create **terraform.tfvars** in   
**provisioning/terraform/production** AND **provisioning/terraform/staging**:

```sh
# Secrets
server_password = "YOUR_VM_PASSWORD"
ssh_keys = [
  "YOUR_PUBLIC_SSH_KEY(S)" 
]
# Variables
node_count    = 3                # Number of VMs
vm_memory     = "4096"           # VM RAM
vm_vcpu       = 2                # VM CPU
user_name     = "server"         # VM User (don't change)
hostname_base = "k3s-node"       # VM base hostname
env           = "production"     # or "staging" respectively
host_ip       = "192.168.X.XXX"  # Host IP; 192.168.0.111 / 192.168.0.109
```

4. **Execute Pipeline:**
```bash
source scripts/env.sh  # Sets up your shell environment

# Add ENV=production or leave blank for production
# Add ENV=staging for staging

# make provision-host           = production
# make provision-host ENV=stage = staging

make provision-host    # Configures KVM/libvirt & Tailscale
make apply             # Provisions VMs via Terraform
make provision         # Bootstraps K3s & FluxCD
```

5. **Host network stability note (optional, bridged setup)**

On Fedora hosts running a libvirt bridge (`br0`), relying on DHCP for the host interface can lead to intermittent connectivity issues during network reconfiguration or interface resets.
    To make the setup resilient, the host bridge interface is recommended to be configured with a static IP:

```bash
sudo nmcli connection modify br0 \
  ipv4.method manual \
  ipv4.addresses 192.168.0.111/24 \
  ipv4.gateway 192.168.0.1 \
  ipv4.dns "192.168.0.1 1.1.1.1 8.8.8.8"
```

6. **External Access (Cloudflare Tunnel)**

After provisioning, services are exposed via Cloudflare Tunnel - no port forwarding required.

**Prerequisites:**
    - Cloudflare Account
    - Domain DNS managed by Cloudflare
    - Age key available (stored in Ansible Vault)

**Setup:**
    1. Create a tunnel in Cloudflare Zero Trust → Networks → Tunnels
    2. Copy the tunnel token
    3. Add token to `infrastructure/base/networking/secret.sops.yaml` (encrypted via SOPS)
    4. Add public hostnames in Cloudflare dashboard per service
    5. Add corresponding CNAME records in Cloudflare DNS per service

**DNS records:**
    | Type | Name | Value |
    |------|------|-------|
    | CNAME | grafana | `<tunnel-id>.cfargotunnel.com` |
    | CNAME | jellyfin | `<tunnel-id>.cfargotunnel.com` |
    | CNAME | etc. | `<tunnel-id>.cfargotunnel.com` |

## Architecture

>[!NOTE]
>In-progress.

## Repo Structure
```
.
├── clusters/                # GitOps via FluxCD
│   ├── production/
│   └── staging/
├── infrastructure/          # Cluster-wide setup
│   ├── base/                # Base Configuration
│       ├── kyverno/
│       ├── namespaces/
│       ├── networking/
│       └── storage/
│   ├── production/          # Production specific overlays
│   └── staging/             # Staging specific overlays
├── monitoring/              # Monitoring stack
│   ├── base/
│      ├── grafana/
│      ├── prometheus/
│      ├── kube-state-metrics/
│      ├── node-exporter/
│      └── alertmanager/
│   ├── production/
│   └── staging/       
├── policies/                # Kyverno policies
│   ├── base/
│   ├── production/
│   └── staging/        
├── provisioning/            # IaC and Configuration
│   ├── terraform/  
│      ├── modules/
│      ├── production/
│      └── staging/        
│   └── ansible/    
│      ├── inventory/
│      ├── playbook/
│      └── roles/      
├── resources/
├── scripts/
├── services/
│   ├── base/
│       ├── jellyfin/
│       ├── karakeep/
│       └── silverbullet/
│   ├── production/
│   └── staging/
├── Makefile                 # Main orchestrator
    
```

## Tooling overview

### Hardware
| Logo | Device | Role |
|:-:|-----|-------------|
| ![Lenovo](https://cdn.simpleicons.org/lenovo?size=32) | Lenovo Thinkpad T14 Gen 1 | Production |
| ![Lenovo](https://cdn.simpleicons.org/lenovo?size=32) | Lenovo Thinkcentre M700 | Staging |
| ![HP](https://cdn.simpleicons.org/hp?size=32) | HP EliteDesk 800 G2 SFF 2x1TB WD Red | NAS |
| ![Lenovo](https://cdn.simpleicons.org/lenovo?size=32) | Lenovo Legion 5 Slim | Workstation |

### Infrastructure
| Logo | Name | Description |
|:-:|-----|-------------|
| ![Fedora](https://cdn.simpleicons.org/fedora?size=32) | Fedora | Linux Distribution used on Host, VMs and Workstation |
| ![TrueNAS](https://cdn.simpleicons.org/truenas?size=32) | TrueNAS | Open-source unified storage operating system based on OpenZFS |
| ![QEMU](https://cdn.simpleicons.org/qemu?size=32) | QEMU/KVM | Hypervisor for running virtual machines |
| ![K3s](https://cdn.simpleicons.org/k3s?size=32) | K3s | Lightweight Kubernetes engine |
| ![FluxCD](https://cdn.simpleicons.org/flux?size=32) | FluxCD | GitOps tool for managing Kubernetes declaratively |
| ![Terraform](https://cdn.simpleicons.org/terraform?size=32) | Terraform | IaC tool for provisioning infrastructure declaratively |
| ![Ansible](https://cdn.simpleicons.org/ansible/f00?size=32) | Ansible | Automation tool for post-provisioning configuration and orchestration |
| ![SOPS](https://cdn.simpleicons.org/privateinternetaccess/000?size=32) | SOPS | Secret OPerationS - tool for managing secrets |
| ![Cilium](https://cdn.simpleicons.org/cilium/size=32) | Cilium | Solution for providing, securing, and observing network connectivity |
| <img src="https://raw.githubusercontent.com/kyverno/artwork/5be18d691ae2b42beb898ffc1312024975749bd8/Kyverno.svg" width="32" height="32" /> | Kyverno | Unified Policy as Code solution |
| <img src="https://cdn.jsdelivr.net/gh/homarr-labs/dashboard-icons/png/cloudflare-zero-trust.png" width="32" height="32" />  | Cloudflare | Zero-Trust Tunnel for exposing services securely on a the internet |

### Monitoring
| Logo | Name | Description |
|:-:|-----|-------------|
| ![Prometheus](https://cdn.simpleicons.org/prometheus?size=32) | Prometheus | Scrapes infrastructure metrics for visualization in Grafana |
| ![AlertManager](https://cdn.simpleicons.org/prometheus/f51d1d?size=32) | AlertManager | Sends notifications and alerts based on Prometheus metrics |
| ![Grafana](https://cdn.simpleicons.org/grafana?size=32) | Grafana | Dashboard and visualization for metrics collected by Prometheus |
| ![kube-state-metrics](https://cdn.simpleicons.org/cncf/5EC5EB?size=32) | kube-state-metrics | Exposes Kubernetes cluster-level metrics for Prometheus |
| ![node-exporter](https://cdn.simpleicons.org/prometheus?size=32) | node-exporter | Collects host-level metrics for Prometheus |
| ![K9s](https://cdn.simpleicons.org/kubernetes?size=32) | k9s | CLI tool to interactively view Kubernetes resources |
| ![K9s](https://cdn.simpleicons.org/uptimekuma?size=32) | Uptime Kuma | Alternative monitoring running as TrueNAS App |

### Services
| Logo | Name | Description |
|:-:|-----|-------------|
| ![Jellyfin](https://cdn.simpleicons.org/jellyfin?size=32) | Jellyfin | Media streaming service | 
| <img src="https://repository-images.githubusercontent.com/459944886/a6e61d23-9090-4cc4-946d-c5d9c189240f" width="32" height="32" /> | SilverBullet.md | Programmable browser-based Markdown editor |
| ![Karakeep](https://cdn.simpleicons.org/karakeep?size=32) | Karakeep | Bookmark manager |

### Backup Strategy

The homelab follows a **3-2-1 backup strategy**:

| Logo | Name | Description |
|:-:|-----|-------------|
| ![TrueNAS](https://cdn.simpleicons.org/truenas?size=32) | TrueNAS | Primary local storage - 2x 1TB WD Red ZFS mirror |
| <img src="https://images.icon-icons.com/61/PNG/128/lightbrown_external_drive_usb_folder_12286.png" width="32" height="32" /> | External SSD | Secondary local copy for critical data |
| ![Backblaze](https://cdn.simpleicons.org/backblaze?size=32) | Backblaze B2 | Offsite cloud storage via TrueNAS Cloud Sync |

>[!TIP]
> Infrastructure configs and other version-controlled assets also live on GitHub, adding a fourth copy for the most critical data.

## Up next / To do list

- Current focus is stability and reliabilty, no new features planned
- Focus observability and alerting

## Goal
Learn **Kubernetes** and its ecosystem by breaking things in a controlled environment.

homelab

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README