Hetzner self-hosted CI/CD runbook

This is the operator runbook for the handoff from local bootstrap to self-hosted Forgejo-based deployment.

Bootstrap prerequisites

From your workstation:

cp scripts/hetzner/bootstrap-secrets.env.example scripts/hetzner/bootstrap-secrets.env
source scripts/hetzner/bootstrap-secrets.env
bash scripts/hetzner/bootstrap.sh

After that you should have:

.state/hetzner/kubeconfig.yaml
Forgejo reachable at https://${FORGEJO_DOMAIN}
Registry reachable at https://${REGISTRY_DOMAIN}
private admin/control-plane access over Tailscale if configured

Verify the cluster

export KUBECONFIG=$PWD/.state/hetzner/kubeconfig.yaml
kubectl get nodes -o wide
kubectl get pods -A
kubectl -n forgejo get deploy,pods,svc,ingress
kubectl -n registry get deploy,pods,svc,ingress
kubectl -n unrip get deploy,pods

Seed the repo into Forgejo

Create the target repo in Forgejo, then from your workstation:

git remote add forgejo https://${FORGEJO_DOMAIN}/<owner>/<repo>.git
git push forgejo main

Configure Forgejo Actions secrets and variables

Create these repository secrets in Forgejo:

KUBECONFIG_B64
REGISTRY_USERNAME
REGISTRY_PASSWORD

Create these repository variables:

REGISTRY_HOST=${REGISTRY_DOMAIN}
optional: PROJECT_NAME=unrip
optional: PROJECT_NAMESPACE=unrip
optional: PROJECT_DEPLOYMENTS=near-intents-ingest,dummy-reactor,dummy-executor,dummy-consumer

Generate KUBECONFIG_B64 from the bootstrap kubeconfig:

base64 -w0 .state/hetzner/kubeconfig.yaml

Workflow behavior

The workflow in .forgejo/workflows/deploy.yml now:

installs buildah and kubectl on the Forgejo runner
checks out the repo with the Forgejo job token
loads kubeconfig from KUBECONFIG_B64
logs into the private registry
builds registry.<domain>/<project-name>:${GIT_SHA} with buildah
pushes the image
updates each deployment listed in PROJECT_DEPLOYMENTS inside PROJECT_NAMESPACE
waits for rollout after each image update

Default behavior if you do not set project variables:

PROJECT_NAME=unrip
PROJECT_NAMESPACE=unrip
PROJECT_DEPLOYMENTS=near-intents-ingest,dummy-reactor,dummy-executor,dummy-consumer

For a future project, reuse the same workflow by changing only the Forgejo repository variables instead of copying the workflow.

The first bootstrap deploy is different from routine CI:

bootstrap fetches the real kubeconfig from the node and imports a local bootstrap image directly into k3s
routine CI is intended to push versioned images to the private registry

Trigger deploys

Push to main in Forgejo:

git push forgejo main

Observe deploys

export KUBECONFIG=$PWD/.state/hetzner/kubeconfig.yaml
kubectl -n unrip rollout status deployment/near-intents-ingest --timeout=300s
kubectl -n unrip rollout status deployment/dummy-reactor --timeout=300s
kubectl -n unrip rollout status deployment/dummy-executor --timeout=300s
kubectl -n unrip rollout status deployment/dummy-consumer --timeout=300s
kubectl -n unrip get pods -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -n 50

DNS and TLS

If DNS automation was enabled during bootstrap, A records for the base, Forgejo, and registry hosts are already managed from the repo-side bootstrap.

Currently supported DNS providers:

Cloudflare
Porkbun

TLS is issued by cert-manager using the rendered Let's Encrypt email and ingress hosts.

Current limitations

Forgejo admin bootstrap and repository creation are not yet API-automated.
Forgejo repository secrets/variables still need to be populated before the first real deploy run.
The runner currently uses host-mode jobs and installs buildah/kubectl at job start, which is functional but not yet optimized.

3.7 KiB Raw Blame History