# Hetzner self-hosted CI/CD runbook

This is the operator runbook for the handoff from local bootstrap to self-hosted Forgejo-based deployment.

## Bootstrap prerequisites
From your workstation:

```bash
cp scripts/hetzner/bootstrap-secrets.env.example scripts/hetzner/bootstrap-secrets.env
source scripts/hetzner/bootstrap-secrets.env
bash scripts/hetzner/bootstrap.sh
```

After that you should have:
- `.state/hetzner/kubeconfig.yaml`
- Forgejo reachable at `https://${FORGEJO_DOMAIN}`
- Registry reachable at `https://${REGISTRY_DOMAIN}`
- private admin/control-plane access over Tailscale if configured

## Verify the cluster
```bash
export KUBECONFIG=$PWD/.state/hetzner/kubeconfig.yaml
kubectl get nodes -o wide
kubectl get pods -A
kubectl -n forgejo get deploy,pods,svc,ingress
kubectl -n registry get deploy,pods,svc,ingress
kubectl -n unrip get deploy,pods
```

## Seed the repo into Forgejo
Create the target repo in Forgejo, then from your workstation:

```bash
git remote add forgejo https://${FORGEJO_DOMAIN}/<owner>/<repo>.git
git push forgejo main
```

## Configure Forgejo Actions secrets and variables
Create these repository secrets in Forgejo:
- `KUBECONFIG_B64`
- `REGISTRY_USERNAME`
- `REGISTRY_PASSWORD`

Create these repository variables:
- `REGISTRY_HOST=${REGISTRY_DOMAIN}`
- optional: `PROJECT_NAME=unrip`
- optional: `PROJECT_NAMESPACE=unrip`
- optional: `PROJECT_DEPLOYMENTS=near-intents-ingest,dummy-reactor,dummy-executor,dummy-consumer`

Generate `KUBECONFIG_B64` from the bootstrap kubeconfig:

```bash
base64 -w0 .state/hetzner/kubeconfig.yaml
```

## Workflow behavior
The workflow in `.forgejo/workflows/deploy.yml` now:
1. installs `buildah` and `kubectl` on the Forgejo runner
2. checks out the repo with the Forgejo job token
3. loads kubeconfig from `KUBECONFIG_B64`
4. logs into the private registry
5. builds `registry.<domain>/<project-name>:${GIT_SHA}` with `buildah`
6. pushes the image
7. updates each deployment listed in `PROJECT_DEPLOYMENTS` inside `PROJECT_NAMESPACE`
8. waits for rollout after each image update

Default behavior if you do not set project variables:
- `PROJECT_NAME=unrip`
- `PROJECT_NAMESPACE=unrip`
- `PROJECT_DEPLOYMENTS=near-intents-ingest,dummy-reactor,dummy-executor,dummy-consumer`

For a future project, reuse the same workflow by changing only the Forgejo repository variables instead of copying the workflow.

The first bootstrap deploy is different from routine CI:
- bootstrap fetches the real kubeconfig from the node and imports a local bootstrap image directly into k3s
- routine CI is intended to push versioned images to the private registry

## Trigger deploys
Push to `main` in Forgejo:

```bash
git push forgejo main
```

## Observe deploys
```bash
export KUBECONFIG=$PWD/.state/hetzner/kubeconfig.yaml
kubectl -n unrip rollout status deployment/near-intents-ingest --timeout=300s
kubectl -n unrip rollout status deployment/dummy-reactor --timeout=300s
kubectl -n unrip rollout status deployment/dummy-executor --timeout=300s
kubectl -n unrip rollout status deployment/dummy-consumer --timeout=300s
kubectl -n unrip get pods -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -n 50
```

## DNS and TLS
If DNS automation was enabled during bootstrap, A records for the base, Forgejo, and registry hosts are already managed from the repo-side bootstrap.

Currently supported DNS providers:
- Cloudflare
- Porkbun

TLS is issued by cert-manager using the rendered Let's Encrypt email and ingress hosts.

## Current limitations
- Forgejo admin bootstrap and repository creation are not yet API-automated.
- Forgejo repository secrets/variables still need to be populated before the first real deploy run.
- The runner currently uses host-mode jobs and installs `buildah`/`kubectl` at job start, which is functional but not yet optimized.