# Hetzner single-node overlay

This overlay turns the shared platform and `unrip` project bases into a concrete first-node bootstrap target for the Terraform-provisioned k3s VM.

The checked-in overlay is the declarative template. For first-cluster bootstrap, `scripts/hetzner/bootstrap.sh` renders a generated overlay under `.state/hetzner/generated-overlay/` and applies that generated copy as the source of truth for the run.

## Two ways to use this overlay

### 1. Recommended: `scripts/hetzner/bootstrap.sh`
This is the intended operator workflow for a fresh Hetzner cluster. The bootstrap script renders secret and patch inputs from local env and `pass`, creates imperative registry secrets, and applies a generated Kustomize overlay.

That generated overlay now imports the platform resources from `deploy/k8s/platform/base/kustomization.yaml`, so new checked-in platform components such as observability manifests are included automatically during bootstrap instead of being silently skipped by a hard-coded file list.

Bootstrap overwrites these operator-worktree files on each run:
- `deploy/k8s/overlays/hetzner-single-node/secrets/unrip.env`
- `deploy/k8s/overlays/hetzner-single-node/secrets/forgejo.env`
- `deploy/k8s/overlays/hetzner-single-node/secrets/observability.env`

Bootstrap also renders and applies generated copies of these patch files under `.state/hetzner/generated-overlay/` instead of modifying the checked-in overlay files directly:
- `ingress-hosts.patch.yaml`
- `issuer-email.patch.yaml`
- `storage-class.patch.yaml`

Secret/config sources when using bootstrap:
- from `pass` or direct env overrides via `scripts/hetzner/bootstrap-secrets.env`:
  - `HCLOUD_TOKEN`
  - `TAILSCALE_AUTH_KEY`
  - `CLOUDFLARE_API_TOKEN`
  - `CLOUDFLARE_ZONE_ID`
  - `PORKBUN_API_KEY`
  - `PORKBUN_SECRET_API_KEY`
  - `REGISTRY_PASSWORD`
  - `NEAR_INTENTS_API_KEY`
  - `FORGEJO_ADMIN_PASSWORD`
  - optional `GRAFANA_ADMIN_PASSWORD` (bootstrap generates one if omitted)
- from plain env/non-secret config in `scripts/hetzner/bootstrap-secrets.env`:
  - `PUBLIC_DOMAIN`, `BASE_DOMAIN`, `FORGEJO_DOMAIN`, `FORGEJO_ROOT_URL`, `REGISTRY_DOMAIN`, `GRAFANA_DOMAIN`, `GRAFANA_ROOT_URL`, `HEADLAMP_DOMAIN`
  - default hostname model under `PUBLIC_DOMAIN`: `git.${PUBLIC_DOMAIN}`, `registry.${PUBLIC_DOMAIN}`, `grafana.${PUBLIC_DOMAIN}`, `headlamp.${PUBLIC_DOMAIN}`
  - `LETSENCRYPT_EMAIL`
  - `REGISTRY_USERNAME`
  - `FORGEJO_ADMIN_USERNAME`, `FORGEJO_ADMIN_EMAIL`
  - optional `GRAFANA_ADMIN_USERNAME` (defaults to `admin`)
  - optional project overrides such as `PROJECT_NAME`, `PROJECT_NAMESPACE`, and `PROJECT_SECRET_ENV_BASENAME`

Bootstrap materializes Kubernetes inputs like this:
- `secrets/unrip.env` gets `NEAR_INTENTS_API_KEY`
- `secrets/forgejo.env` gets only `root_url` and `domain`
- `secrets/observability.env` gets `grafana_admin_user`, `grafana_admin_password`, and `grafana_root_url`
- generated overlay Kustomize secret generators create `observability-secrets` in namespace `observability` alongside the project and Forgejo secrets
- `registry-secrets` in namespace `registry` is created imperatively from `REGISTRY_USERNAME` and `REGISTRY_PASSWORD`
- `<project>-registry-creds` image pull secret is created imperatively in the project namespace from the same registry credentials

Note: the Forgejo runner no longer reads `runner_registration_token` from `forgejo-secrets`. `scripts/hetzner/bootstrap.sh` generates a one-time runner token in-cluster, registers the runner, and writes `/data/forgejo-runner/.runner` on the shared Forgejo PVC before restarting the runner deployment.

### 2. Manual: `kubectl apply -k`
Use this only if you intentionally want to manage the checked-in overlay inputs yourself. In manual mode, the checked-in overlay remains the source of truth; in bootstrap mode, the generated overlay is the source of truth for what gets applied.

Before apply, create or edit real local input files:

```bash
cp deploy/k8s/overlays/hetzner-single-node/secrets/unrip.env.example deploy/k8s/overlays/hetzner-single-node/secrets/unrip.env
cp deploy/k8s/overlays/hetzner-single-node/secrets/forgejo.env.example deploy/k8s/overlays/hetzner-single-node/secrets/forgejo.env
cp deploy/k8s/overlays/hetzner-single-node/secrets/observability.env.example deploy/k8s/overlays/hetzner-single-node/secrets/observability.env
cp deploy/k8s/overlays/hetzner-single-node/secrets/registry.htpasswd.example deploy/k8s/overlays/hetzner-single-node/secrets/registry.htpasswd
```

Then update:
- ingress hosts in `ingress-hosts.patch.yaml` for Forgejo, Registry, Grafana, and Headlamp
- ACME email in `issuer-email.patch.yaml`
- project secret values in `secrets/unrip.env`
- Forgejo secret values in `secrets/forgejo.env` (`root_url` and `domain` only)
- observability secret values in `secrets/observability.env` (`grafana_admin_user`, `grafana_admin_password`, `grafana_root_url`)

Important manual-mode caveat:
- `kubectl apply -k deploy/k8s/overlays/hetzner-single-node` creates only the Kustomize-managed secrets from the checked-in files (`unrip-secrets`, `forgejo-secrets`, `observability-secrets`, and `registry-secrets` when `secrets/registry.htpasswd` exists)
- it does **not** create the project docker-registry pull secret
- if you skip `scripts/hetzner/bootstrap.sh`, you must create that pull secret separately before expecting image pulls or CI builds to work

## Apply
Bootstrap path:
```bash
bash scripts/hetzner/bootstrap.sh
```

Manual path:
```bash
kubectl apply -k deploy/k8s/overlays/hetzner-single-node
```

## What gets installed
- shared platform namespaces for registry, ingress, cert-manager, Forgejo, and observability
- project namespace `unrip`
- Redpanda plus a topic bootstrap job inside `unrip`
- app worker deployments referencing `unrip-secrets`
- Forgejo and Forgejo runner referencing `forgejo-secrets`
- private registry workload, which still requires the imperative `registry-secrets` auth secret to be created separately unless you used `scripts/hetzner/bootstrap.sh`
- nginx ingress and ACME issuers for TLS
- observability ingress for Grafana and Headlamp, plus local-path PVC overrides for Grafana and Loki

## Observability UI exposure policy
- Grafana and Headlamp are both wired into the Hetzner ingress/domain model.
- Use `grafana.${PUBLIC_DOMAIN}` / `headlamp.${PUBLIC_DOMAIN}` or explicit `GRAFANA_DOMAIN` / `HEADLAMP_DOMAIN` values.
- Grafana is the historical log search UI backed by Loki.
- Headlamp is the Kubernetes cluster UI for workloads, events, and pod logs.
- Grafana is authenticated through `observability-secrets`; Headlamp is authenticated with the generated Kubernetes service-account token that bootstrap stores in `pass` when `HEADLAMP_ADMIN_TOKEN_PASS` is configured.

For future projects, do not reuse `unrip`; create a new project namespace and matching `<project>-config`, `<project>-secrets`, and `<project>-registry-creds` resources.