doran/deploy/k8s/overlays/hetzner-single-node

Hetzner single-node overlay

This overlay turns the shared platform and unrip project bases into a concrete first-node bootstrap target for the Terraform-provisioned k3s VM.

The checked-in overlay is the declarative template. For first-cluster bootstrap, scripts/hetzner/bootstrap.sh renders a generated overlay under .state/hetzner/generated-overlay/ and applies that generated copy as the source of truth for the run.

Two ways to use this overlay

1. Recommended: scripts/hetzner/bootstrap.sh

This is the intended operator workflow for a fresh Hetzner cluster. The bootstrap script renders secret and patch inputs from local env and pass, creates imperative registry secrets, and applies a generated Kustomize overlay.

That generated overlay now imports the platform resources from deploy/k8s/platform/base/kustomization.yaml, so new checked-in platform components such as observability manifests are included automatically during bootstrap instead of being silently skipped by a hard-coded file list.

Bootstrap overwrites these operator-worktree files on each run:

• deploy/k8s/overlays/hetzner-single-node/secrets/unrip.env
• deploy/k8s/overlays/hetzner-single-node/secrets/forgejo.env
• deploy/k8s/overlays/hetzner-single-node/secrets/observability.env

Bootstrap also renders and applies generated copies of these patch files under .state/hetzner/generated-overlay/ instead of modifying the checked-in overlay files directly:

• ingress-hosts.patch.yaml
• issuer-email.patch.yaml
• storage-class.patch.yaml

Secret/config sources when using bootstrap:

• from pass or direct env overrides via scripts/hetzner/bootstrap-secrets.env:
  • HCLOUD_TOKEN
  • TAILSCALE_AUTH_KEY
  • CLOUDFLARE_API_TOKEN
  • CLOUDFLARE_ZONE_ID
  • PORKBUN_API_KEY
  • PORKBUN_SECRET_API_KEY
  • REGISTRY_PASSWORD
  • NEAR_INTENTS_API_KEY
  • FORGEJO_ADMIN_PASSWORD
  • optional GRAFANA_ADMIN_PASSWORD (bootstrap generates one if omitted)
• from plain env/non-secret config in scripts/hetzner/bootstrap-secrets.env:
  • PUBLIC_DOMAIN, BASE_DOMAIN, FORGEJO_DOMAIN, FORGEJO_ROOT_URL, REGISTRY_DOMAIN, GRAFANA_DOMAIN, GRAFANA_ROOT_URL, HEADLAMP_DOMAIN
  • default hostname model under PUBLIC_DOMAIN: git.${PUBLIC_DOMAIN}, registry.${PUBLIC_DOMAIN}, grafana.${PUBLIC_DOMAIN}, headlamp.${PUBLIC_DOMAIN}
  • LETSENCRYPT_EMAIL
  • REGISTRY_USERNAME
  • FORGEJO_ADMIN_USERNAME, FORGEJO_ADMIN_EMAIL
  • optional GRAFANA_ADMIN_USERNAME (defaults to admin)
  • optional project overrides such as PROJECT_NAME, PROJECT_NAMESPACE, and PROJECT_SECRET_ENV_BASENAME

Bootstrap materializes Kubernetes inputs like this:

• secrets/unrip.env gets NEAR_INTENTS_API_KEY
• secrets/forgejo.env gets only root_url and domain
• secrets/observability.env gets grafana_admin_user, grafana_admin_password, and grafana_root_url
• generated overlay Kustomize secret generators create observability-secrets in namespace observability alongside the project and Forgejo secrets
• registry-secrets in namespace registry is created imperatively from REGISTRY_USERNAME and REGISTRY_PASSWORD
• <project>-registry-creds image pull secret is created imperatively in the project namespace from the same registry credentials

Note: the Forgejo runner no longer reads runner_registration_token from forgejo-secrets. scripts/hetzner/bootstrap.sh generates a one-time runner token in-cluster, registers the runner, and writes /data/forgejo-runner/.runner on the shared Forgejo PVC before restarting the runner deployment.

2. Manual: kubectl apply -k

Use this only if you intentionally want to manage the checked-in overlay inputs yourself. In manual mode, the checked-in overlay remains the source of truth; in bootstrap mode, the generated overlay is the source of truth for what gets applied.

Before apply, create or edit real local input files:

cp deploy/k8s/overlays/hetzner-single-node/secrets/unrip.env.example deploy/k8s/overlays/hetzner-single-node/secrets/unrip.env
cp deploy/k8s/overlays/hetzner-single-node/secrets/forgejo.env.example deploy/k8s/overlays/hetzner-single-node/secrets/forgejo.env
cp deploy/k8s/overlays/hetzner-single-node/secrets/observability.env.example deploy/k8s/overlays/hetzner-single-node/secrets/observability.env
cp deploy/k8s/overlays/hetzner-single-node/secrets/registry.htpasswd.example deploy/k8s/overlays/hetzner-single-node/secrets/registry.htpasswd

Then update:

• ingress hosts in ingress-hosts.patch.yaml for Forgejo, Registry, Grafana, and Headlamp
• ACME email in issuer-email.patch.yaml
• project secret values in secrets/unrip.env
• Forgejo secret values in secrets/forgejo.env (root_url and domain only)
• observability secret values in secrets/observability.env (grafana_admin_user, grafana_admin_password, grafana_root_url)

Important manual-mode caveat:

• kubectl apply -k deploy/k8s/overlays/hetzner-single-node creates only the Kustomize-managed secrets from the checked-in files (unrip-secrets, forgejo-secrets, observability-secrets, and registry-secrets when secrets/registry.htpasswd exists)
• it does not create the project docker-registry pull secret
• if you skip scripts/hetzner/bootstrap.sh, you must create that pull secret separately before expecting image pulls or CI builds to work

Apply

Bootstrap path:

bash scripts/hetzner/bootstrap.sh

Manual path:

kubectl apply -k deploy/k8s/overlays/hetzner-single-node

What gets installed

• shared platform namespaces for registry, ingress, cert-manager, Forgejo, and observability
• project namespace unrip
• Redpanda plus a topic bootstrap job inside unrip
• app worker deployments referencing unrip-secrets
• Forgejo and Forgejo runner referencing forgejo-secrets
• private registry workload, which still requires the imperative registry-secrets auth secret to be created separately unless you used scripts/hetzner/bootstrap.sh
• nginx ingress and ACME issuers for TLS
• observability ingress for Grafana and Headlamp, plus local-path PVC overrides for Grafana and Loki

Observability UI exposure policy

• Grafana and Headlamp are both wired into the Hetzner ingress/domain model.
• Use grafana.${PUBLIC_DOMAIN} / headlamp.${PUBLIC_DOMAIN} or explicit GRAFANA_DOMAIN / HEADLAMP_DOMAIN values.
• Grafana is the historical log search UI backed by Loki.
• Headlamp is the Kubernetes cluster UI for workloads, events, and pod logs.
• Grafana is authenticated through observability-secrets; Headlamp is authenticated with the generated Kubernetes service-account token that bootstrap stores in pass when HEADLAMP_ADMIN_TOKEN_PASS is configured.

For future projects, do not reuse unrip; create a new project namespace and matching <project>-config, <project>-secrets, and <project>-registry-creds resources.