doran/docs/hetzner-k3s-bootstrap.md

Hetzner + k3s + self-hosted Git/CI bootstrap

Goal: provision and deploy everything from this repo to a single Hetzner machine with no manual server login.

Stack

• Terraform provisions the Hetzner Cloud VM, private network, and firewall
• cloud-init installs Tailscale first when configured, then installs k3s automatically
• Kubernetes manifests deploy:
  • Redpanda
  • trading system services
  • private registry
  • Forgejo
  • ingress-nginx
  • cert-manager
  • ACME issuers
• local bootstrap script:
  • runs Terraform
  • optionally creates DNS records via Cloudflare or Porkbun
  • writes overlay secrets/host patches from local env
  • applies the Hetzner single-node k8s overlay
  • builds the current app image locally
  • fetches the real kubeconfig from the node
  • imports the bootstrap image into k3s for the first rollout

Files

• infra/terraform/hetzner/
• deploy/k8s/base/
• deploy/k8s/overlays/hetzner-single-node/
• scripts/hetzner/bootstrap.sh
• scripts/hetzner/configure-cloudflare-dns.sh
• scripts/hetzner/destroy.sh
• scripts/k8s/logs.sh
• .forgejo/workflows/deploy.yml

Required local tools

• terraform
• kubectl
• docker
• curl
• python3

Required local env

Start from:

cp scripts/hetzner/bootstrap-secrets.env.example scripts/hetzner/bootstrap-secrets.env
source scripts/hetzner/bootstrap-secrets.env

Required values:

• HCLOUD_TOKEN
• SSH_PUBLIC_KEY_PATH
• PUBLIC_DOMAIN
• BASE_DOMAIN
• recommended Tailscale values:
  • TAILSCALE_AUTH_KEY
  • TAILSCALE_CONTROL_PLANE_HOSTNAME
• FORGEJO_DOMAIN
• FORGEJO_ROOT_URL
• REGISTRY_DOMAIN
• LETSENCRYPT_EMAIL
• REGISTRY_USERNAME
• REGISTRY_PASSWORD
• NEAR_INTENTS_API_KEY
• FORGEJO_RUNNER_REGISTRATION_TOKEN

Optional for automatic DNS:

• Cloudflare:
  • CLOUDFLARE_API_TOKEN
  • CLOUDFLARE_ZONE_ID
• Porkbun:
  • PORKBUN_API_KEY
  • PORKBUN_SECRET_API_KEY

Bootstrap

bash scripts/hetzner/bootstrap.sh

Outputs:

• Hetzner VM created
• Tailscale joined if configured
• k3s installed
• kubeconfig written to .state/hetzner/kubeconfig.yaml
• overlay secrets and ingress host patches rendered from local env
• namespaces, Redpanda, app deployments, Forgejo, registry, ingress, cert-manager, and issuers applied
• bootstrap image built and first rollout triggered

Tailscale-first admin access

Recommended mode:

• public firewall exposes only 80/443
• admin access uses Tailscale
• Kubernetes API uses the Tailscale hostname when TAILSCALE_CONTROL_PLANE_HOSTNAME is set

TF_ADMIN_CIDR_BLOCKS remains only as a fallback if you intentionally want public admin/API exposure.

DNS and TLS

If DNS provider credentials are present, bootstrap updates:

• ${BASE_DOMAIN}
• git.${BASE_DOMAIN}
• registry.${BASE_DOMAIN}

Supported scripted providers:

• Cloudflare
• Porkbun

TLS is handled in-cluster by cert-manager using Let's Encrypt issuers and the rendered ingress hosts.

Observe the cluster

KUBECONFIG=.state/hetzner/kubeconfig.yaml kubectl get pods -A
bash scripts/k8s/logs.sh

Self-hosted CI/CD handoff

After bootstrap:

1. open Forgejo at https://${FORGEJO_DOMAIN}
2. seed or mirror this repo into Forgejo
3. add Forgejo Actions secrets:
   • KUBECONFIG_B64
   • REGISTRY_USERNAME
   • REGISTRY_PASSWORD
4. add Forgejo Actions variable:
   • REGISTRY_HOST=${REGISTRY_DOMAIN}
5. push to main

The workflow then:

• builds the image
• pushes it to https://${REGISTRY_DOMAIN}
• updates the app deployments in unrip
• waits for rollout

Destroy everything

bash scripts/hetzner/destroy.sh

Current limitations

• Forgejo admin bootstrap and repo seeding are still operator-driven after the first cluster bootstrap.
• bootstrap and CI authentication paths should still be hardened before production use.
• routine deploys are intended to be registry-native through Forgejo Actions, but that still needs a real-world verification pass.