forked from infra/ansible
forgejo: Redesign runnerconfig
Signed-off-by: David Kirwan <davidkirwanirl@gmail.com>
This commit is contained in:
parent
068605d67d
commit
1ffd0cd8a5
GPG key ID:
A5893AB6474AC37D
35 changed files with 357 additions and 12 deletions
95
roles/openshift-apps/forgejo/README.md
Normal file
95
roles/openshift-apps/forgejo/README.md
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
# Forgejo OpenShift App Role
|
||||
|
||||
Deploys Fedora Forgejo (git forge) on OpenShift: namespace, storage, PostgreSQL, Valkey, Helm release, CI runners, and control-host backups.
|
||||
|
||||
**Playbook:** [`playbooks/openshift-apps/forgejo.yml`](../../../playbooks/openshift-apps/forgejo.yml)
|
||||
|
||||
## Entry point and execution model
|
||||
|
||||
- **Hosts:** `os_control_stg:os_control` (OpenShift control nodes), `gather_facts: false`
|
||||
- **Where work runs:** On the control host under `/root/ocp4/openshift-apps/forgejo/` — templates are rendered there, then applied with `kubernetes.core.k8s`, `oc apply`, or `kubernetes.core.helm`
|
||||
- **Pattern:** Hybrid OpenShift app — raw manifests plus the upstream [forgejo-helm](https://codeberg.org/fedora/forgejo-helm) chart (not the composable `openshift/*` roles)
|
||||
- **Secrets/config:** Production vs staging uses `env` (`production` / `staging`) and ternaries in `templates/values.yaml.j2`. Sensitive vars (passwords, OIDC keys, runner secrets) come from private vars (`/srv/private/ansible/vars.yml`), not this repo
|
||||
|
||||
## Task flow (`tasks/main.yaml`)
|
||||
|
||||
### 1. Control-host backup infrastructure
|
||||
|
||||
Before cluster resources:
|
||||
|
||||
- Creates `/root/ocp4/openshift-apps/forgejo/backups/{script,dump,log}/`
|
||||
- Copies backup scripts from `files/`
|
||||
- Cron on the control host:
|
||||
- **02:00** — `run_forgejo_backup.sh` (rsync scripts into Forgejo pod, DB dump, pull dump back)
|
||||
- **03:00** — `run_forgejo_pruning.sh` (prune old dumps)
|
||||
|
||||
Backup dumps are NFS-mounted on `os-control01` (see `inventory/host_vars/os-control01.rdu3.fedoraproject.org`).
|
||||
|
||||
### 2. Kubernetes foundation
|
||||
|
||||
| Step | Task file | Deploys |
|
||||
|------|-----------|---------|
|
||||
| Namespace | `create-forgejo-namespace.yaml` | `forgejo` namespace |
|
||||
| Storage | `create-forgejo-pvc.yaml` | PV + PVC `gitea-shared-storage` (NetApp NFS `forgejo-volume`, RWX 100Gi) |
|
||||
| Database | `create-postgres-operator-config.yaml` | Crunchy `PostgresCluster` `forgejo-ha` (PG 17, 2 replicas, pgBackRest); 30s wait |
|
||||
| Cache | `deploy-valkey.yaml` | 6-replica Valkey StatefulSet + Service + ConfigMap + Secret (`valkey` tag, `oc apply`) |
|
||||
| App | `call-helm.yaml` | Forgejo Helm release |
|
||||
|
||||
Custom theme ConfigMap task is **commented out** in `main.yaml` (`create-custom-theme-configmap.yaml`). Theme CSS is in `files/themes/` and referenced in Helm values (`fedora-auto`, etc.).
|
||||
|
||||
### 3. Forgejo application (Helm)
|
||||
|
||||
`call-helm.yaml`:
|
||||
|
||||
1. Renders `templates/values.yaml.j2` → `helm-values.yaml`
|
||||
2. Clones https://codeberg.org/fedora/forgejo-helm.git to `/tmp/forgejo-helm_repo`
|
||||
3. Runs `helm dependency update`
|
||||
4. Deploys release `forgejo` in namespace `forgejo` via `kubernetes.core.helm`
|
||||
|
||||
Notable Helm settings:
|
||||
|
||||
- **Image:** Quay (`forgejo_prod_quay_image` / `forgejo_stg_quay_image`), rootless
|
||||
- **Route:** OpenShift Route (not Ingress); `forgejo_hostname` / `forgejo_stg_hostname`
|
||||
- **Persistence:** Pre-created PVC `gitea-shared-storage` (`persistence.create: false`)
|
||||
- **DB:** External Crunchy Postgres, not chart PostgreSQL
|
||||
- **Cache:** External Valkey via `redis+cluster://` in `gitea.config.cache` (chart `redis-cluster` subchart enabled; live cache is the separate Valkey StatefulSet)
|
||||
- **Auth:** OpenID Connect to Fedora Accounts with `group-team-map` for org/team RBAC
|
||||
- **Admin:** Private vars; `passwordMode: keepUpdated`
|
||||
|
||||
### 4. CI runners (Forgejo Actions)
|
||||
|
||||
| Step | Task file | Purpose |
|
||||
|------|-----------|---------|
|
||||
| Load runners | `load-forgejo-runners.yaml` | Merges `runners/<env>/*.yml` with private registration secrets |
|
||||
| Runner vars | `copy-forgejo-runner-vars.yaml` | Renders runner vars from `forgejo-runners.yaml.j2` |
|
||||
| Runner secret | `create-forgejo-runner-config-secret.yaml` | K8s Secret for runner VM |
|
||||
| VM RBAC | `create-forgejo-runnerhost-sa-rbac.yaml` | ServiceAccount, Role, RoleBinding, token Secret |
|
||||
| Runner VM | `create-forgejo-runnerhost-vm.yaml` | KubeVirt VM + headless Service |
|
||||
| Registration | `register-runners.yaml` | `forgejo-cli actions register` in Forgejo pod |
|
||||
|
||||
Runner host:
|
||||
|
||||
- KubeVirt VM `forgejo-runnerhost-vm` (Fedora `u1.xlarge`, 30Gi)
|
||||
- Packages in `defaults/main.yml` (`ansible`, `podman`, `python3-kubernetes`, etc.)
|
||||
- SA token via serial disk `SATOKEN000001`
|
||||
- Cloud-init runs **ansible-pull** from [ansible-role-forgejo-runner](https://codeberg.org/fedora/ansible-role-forgejo-runner) (in-repo clone is commented out)
|
||||
- `load-forgejo-runners.yaml` merges `runners/<env>/*.yml` with secret dicts from private vars, then `register-runners.yaml` registers them (idempotent; tolerates duplicate key)
|
||||
|
||||
## Variables
|
||||
|
||||
- `defaults/main.yml` — namespace, headless service name, runnerhost packages
|
||||
- `vars/main.yml` — `openshift_user_id` for PV ownership
|
||||
|
||||
Most operational variables live in private/group vars (hostnames, DB, OIDC, runner secrets, Valkey password, etc.).
|
||||
|
||||
## Related code
|
||||
|
||||
- **`roles/openshift-apps/dist-git`** — reuses `/root/ocp4/openshift-apps/forgejo/` for theme ConfigMaps
|
||||
- **`.forgejo/workflows/`** — CI for this ansible repo runs on this Forgejo instance
|
||||
|
||||
## Design notes
|
||||
|
||||
1. Mixed deploy paths: `kubernetes.core.k8s`, `oc apply` (Valkey), and Helm
|
||||
2. Runner registration is in Ansible; runner daemons are configured via ansible-pull on the VM
|
||||
3. Backups are control-host cron jobs using `oc rsync` / `oc exec`, not in-cluster CronJobs
|
||||
4. Same playbook runs prod and staging; `env` selects all branching
|
||||
38
roles/openshift-apps/forgejo/runners/README.md
Normal file
38
roles/openshift-apps/forgejo/runners/README.md
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
# Forgejo runner definitions
|
||||
|
||||
Each runner has one YAML file under `production/` or `staging/`. These files are public (this repo) and contain everything **except** the registration secret.
|
||||
|
||||
## File format
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: releng-1
|
||||
scope: releng
|
||||
labels:
|
||||
- releng-1
|
||||
image: code.forgejo.org/forgejo/runner:11
|
||||
```
|
||||
|
||||
- **name** — Unique runner name (must match the filename stem).
|
||||
- **scope** — Forgejo org scope for `forgejo-cli actions register --scope` (omit or use `global` for instance-wide runners).
|
||||
- **labels** — Workflow `runs-on` labels. Quote values that contain colons.
|
||||
- **image** — Runner container image on the runner host.
|
||||
- **capacity** — Optional concurrent job limit (passed through to runner host config).
|
||||
|
||||
## Secrets (private ansible repo)
|
||||
|
||||
Registration secrets stay in `/srv/private/ansible/vars.yml` as a dict keyed by runner name:
|
||||
|
||||
```yaml
|
||||
forgejo_runner_secrets:
|
||||
releng-1: "01c729e8f65dea42ddaf335dbfcf3841c71e58c8"
|
||||
infra-1: "c4ebefbdc76b023bd7c7ef06a483fae045bb2f3d"
|
||||
# ...
|
||||
|
||||
forgejo_stg_runner_secrets:
|
||||
# staging runners, same shape
|
||||
```
|
||||
|
||||
Generate a new secret with: `openssl rand -hex 20`
|
||||
|
||||
`tasks/load-forgejo-runners.yaml` merges each definition file with its secret before templating and CLI registration.
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: ai-ml-1
|
||||
scope: ai-ml
|
||||
labels:
|
||||
- podman
|
||||
image: code.forgejo.org/forgejo/runner:12
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: apps-1
|
||||
scope: apps
|
||||
labels:
|
||||
- podman
|
||||
image: code.forgejo.org/forgejo/runner:11
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
---
|
||||
name: atomic-desktops-1
|
||||
scope: atomic-desktops
|
||||
labels:
|
||||
- "docker:docker://node:22-bookworm"
|
||||
- "docker-slim:docker://node:22-alpine"
|
||||
image: code.forgejo.org/forgejo/runner:11
|
||||
7
roles/openshift-apps/forgejo/runners/production/ci-1.yml
Normal file
7
roles/openshift-apps/forgejo/runners/production/ci-1.yml
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
---
|
||||
name: ci-1
|
||||
scope: ci
|
||||
labels:
|
||||
- "docker:docker://node:22-bookworm"
|
||||
- "docker-slim:docker://node:22-alpine"
|
||||
image: code.forgejo.org/forgejo/runner:11
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: commops-1
|
||||
scope: commops
|
||||
labels:
|
||||
- podman
|
||||
image: code.forgejo.org/forgejo/runner:11
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: council-1
|
||||
scope: council
|
||||
labels:
|
||||
- podman
|
||||
image: code.forgejo.org/forgejo/runner:12
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: dei-1
|
||||
scope: dei
|
||||
labels:
|
||||
- podman
|
||||
image: code.forgejo.org/forgejo/runner:12
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: design-1
|
||||
scope: design
|
||||
labels:
|
||||
- podman
|
||||
image: code.forgejo.org/forgejo/runner:12
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: docs-1
|
||||
scope: docs
|
||||
labels:
|
||||
- podman
|
||||
image: code.forgejo.org/forgejo/runner:12
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: forge-1
|
||||
scope: forge
|
||||
labels:
|
||||
- podman
|
||||
image: code.forgejo.org/forgejo/runner:11
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: infra-1
|
||||
scope: infra
|
||||
labels:
|
||||
- infra-1
|
||||
image: code.forgejo.org/forgejo/runner:11
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: marketing-1
|
||||
scope: marketing
|
||||
labels:
|
||||
- podman
|
||||
image: code.forgejo.org/forgejo/runner:12
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: mentored-projects-1
|
||||
scope: mentored-projects
|
||||
labels:
|
||||
- podman
|
||||
image: code.forgejo.org/forgejo/runner:12
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: mindshare-1
|
||||
scope: mindshare
|
||||
labels:
|
||||
- podman
|
||||
image: code.forgejo.org/forgejo/runner:11
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: operations-1
|
||||
scope: operations
|
||||
labels:
|
||||
- podman
|
||||
image: code.forgejo.org/forgejo/runner:12
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: podcast-1
|
||||
scope: podcast
|
||||
labels:
|
||||
- podman
|
||||
image: code.forgejo.org/forgejo/runner:12
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: quality-1
|
||||
scope: quality
|
||||
labels:
|
||||
- fedora
|
||||
image: code.forgejo.org/forgejo/runner:11
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: releng-1
|
||||
scope: releng
|
||||
labels:
|
||||
- releng-1
|
||||
image: code.forgejo.org/forgejo/runner:11
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: releng-2
|
||||
scope: releng
|
||||
labels:
|
||||
- docker
|
||||
image: code.forgejo.org/forgejo/runner:11
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: security-1
|
||||
scope: security
|
||||
labels:
|
||||
- podman
|
||||
image: code.forgejo.org/forgejo/runner:12
|
||||
6
roles/openshift-apps/forgejo/runners/staging/cle-1.yml
Normal file
6
roles/openshift-apps/forgejo/runners/staging/cle-1.yml
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: cle-1
|
||||
scope: cle
|
||||
labels:
|
||||
- cle-1
|
||||
image: code.forgejo.org/forgejo/runner:11
|
||||
6
roles/openshift-apps/forgejo/runners/staging/docs-1.yml
Normal file
6
roles/openshift-apps/forgejo/runners/staging/docs-1.yml
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: docs-1
|
||||
scope: docs
|
||||
labels:
|
||||
- docs-1
|
||||
image: code.forgejo.org/forgejo/runner:11
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: fedora-1
|
||||
scope: global
|
||||
labels:
|
||||
- fedora-1
|
||||
image: code.forgejo.org/forgejo/runner:11
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: fedora-2
|
||||
scope: global
|
||||
labels:
|
||||
- fedora-2
|
||||
image: code.forgejo.org/forgejo/runner:11
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: fedora-3
|
||||
scope: global
|
||||
labels:
|
||||
- fedora-3
|
||||
image: code.forgejo.org/forgejo/runner:11
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: fedora-4
|
||||
scope: global
|
||||
labels:
|
||||
- fedora-4
|
||||
image: code.forgejo.org/forgejo/runner:11
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: fedora-5
|
||||
scope: global
|
||||
labels:
|
||||
- fedora-5
|
||||
image: code.forgejo.org/forgejo/runner:11
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
name: fedora-6
|
||||
scope: global
|
||||
labels:
|
||||
- fedora-6
|
||||
image: code.forgejo.org/forgejo/runner:11
|
||||
8
roles/openshift-apps/forgejo/runners/staging/test.yml
Normal file
8
roles/openshift-apps/forgejo/runners/staging/test.yml
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
---
|
||||
name: test
|
||||
scope: playground
|
||||
labels:
|
||||
- "docker:docker://node:22-bookworm"
|
||||
- "docker-slim:docker://node:22-alpine"
|
||||
image: code.forgejo.org/forgejo/runner:11
|
||||
capacity: 4
|
||||
40
roles/openshift-apps/forgejo/tasks/load-forgejo-runners.yaml
Normal file
40
roles/openshift-apps/forgejo/tasks/load-forgejo-runners.yaml
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
---
|
||||
# Build forgejo_runners from per-runner YAML files in runners/<env>/ plus secrets in private vars.
|
||||
# Private vars (forgejo_runner_secrets / forgejo_stg_runner_secrets) must be a dict keyed by runner name:
|
||||
# forgejo_runner_secrets:
|
||||
# releng-1: "0123...hex40"
|
||||
|
||||
- name: Find forgejo runner definition files
|
||||
ansible.builtin.find:
|
||||
paths: "{{ role_path }}/runners/{{ env }}"
|
||||
patterns: "*.yml"
|
||||
register: _forgejo_runner_def_files
|
||||
|
||||
- name: Build forgejo_runners from definitions and private secrets
|
||||
ansible.builtin.set_fact:
|
||||
forgejo_runners: "{{ forgejo_runners | default([]) + [_runner] }}"
|
||||
loop: "{{ _forgejo_runner_def_files.files | sort(attribute='path') }}"
|
||||
vars:
|
||||
_def: "{{ lookup('ansible.builtin.file', item.path) | from_yaml }}"
|
||||
_secrets: "{{ (env == 'production') | ternary(forgejo_runner_secrets, forgejo_stg_runner_secrets) }}"
|
||||
_runner: "{{ _def | combine({'secret': _secrets[_def.name]}) }}"
|
||||
when: _forgejo_runner_def_files.matched > 0
|
||||
|
||||
- name: Set empty forgejo_runners when no definition files exist
|
||||
ansible.builtin.set_fact:
|
||||
forgejo_runners: []
|
||||
when: _forgejo_runner_def_files.matched == 0
|
||||
|
||||
- name: Validate each runner has a secret in private vars
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- _secrets[item.name] is defined
|
||||
- _secrets[item.name] | length > 0
|
||||
fail_msg: >-
|
||||
Missing secret for runner '{{ item.name }}'.
|
||||
Add it to {{ (env == 'production') | ternary('forgejo_runner_secrets', 'forgejo_stg_runner_secrets') }}
|
||||
in the ansible private repo.
|
||||
loop: "{{ forgejo_runners }}"
|
||||
vars:
|
||||
_secrets: "{{ (env == 'production') | ternary(forgejo_runner_secrets, forgejo_stg_runner_secrets) }}"
|
||||
when: forgejo_runners | length > 0
|
||||
|
|
@ -57,6 +57,7 @@
|
|||
# - include_tasks: create-custom-theme-configmap.yaml
|
||||
- include_tasks: deploy-valkey.yaml
|
||||
- include_tasks: call-helm.yaml
|
||||
- include_tasks: load-forgejo-runners.yaml
|
||||
- include_tasks: copy-forgejo-runner-vars.yaml
|
||||
- include_tasks: create-forgejo-runner-config-secret.yaml
|
||||
- include_tasks: create-forgejo-runnerhost-sa-rbac.yaml
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@
|
|||
namespace: forgejo
|
||||
pod: "{{ forgejo_pods.resources[0].metadata.name }}"
|
||||
command: "forgejo forgejo-cli actions register --name {{ item['name'] }} --labels {{ item['labels'] | join(',') }} --secret {{ item['secret'] }}{{ ' --scope ' + item['scope'] if item['scope'] is defined and item['scope'] != 'global' else '' }}"
|
||||
loop: "{{ (env == 'production') | ternary(forgejo_runner_secrets, forgejo_stg_runner_secrets) }}"
|
||||
loop: "{{ forgejo_runners }}"
|
||||
register: forgejo_cli_outputs
|
||||
failed_when: >-
|
||||
(forgejo_cli_outputs.rc is defined
|
||||
|
|
|
|||
|
|
@ -4,18 +4,11 @@
|
|||
# Internal URL (accessible from VM): http://forgejo-http.forgejo.svc.cluster.local:3000
|
||||
forgejo_instance_url: "{{ (env == 'production') | ternary(forgejo_runner_instance_url, forgejo_stg_runner_instance_url) }}"
|
||||
|
||||
# Forgejo runners to register and configure
|
||||
# Note: Generate secrets with: openssl rand -hex 20
|
||||
# Forgejo runners to register and configure (built by load-forgejo-runners.yaml).
|
||||
# Non-secret fields live in roles/openshift-apps/forgejo/runners/<env>/*.yml;
|
||||
# registration secrets live in private vars (forgejo_runner_secrets / forgejo_stg_runner_secrets).
|
||||
#
|
||||
# Runner configuration:
|
||||
# - name: Unique runner name
|
||||
# - labels: List of label names (the runner container will automatically add schema:arg based on its environment)
|
||||
# These are the label names that workflows will use in "runs-on"
|
||||
# - image: Container image to use for the runner
|
||||
# The image determines what execution environment and tools are available
|
||||
# - secret: 40-character hex string for idempotent registration
|
||||
#
|
||||
forgejo_runners: {{ (env == 'production') | ternary(forgejo_runner_secrets, forgejo_stg_runner_secrets) }}
|
||||
forgejo_runners: {{ forgejo_runners }}
|
||||
|
||||
# Configuration for the zabbix agent running on the runnerhost
|
||||
forgejo_runnerhost_zabbix_agent_identity: {{ zabbix_tls_psk_identity }}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue