yumrepos: fix rhel10-infra-stg repo also

Signed-off-by: Kevin Fenzi <kevin@scrye.com>

.ai_review/project.md

@@ -0,0 +1,159 @@
+## Project Overview
+
+**Purpose:** Ansible automation for the entire Fedora Project infrastructure — managing hundreds of bare-metal hosts, VMs, and OpenShift-deployed applications across production and staging environments.
+**Type:** Infrastructure as Code (Ansible)
+**Domain:** Linux distribution infrastructure (build systems, package repositories, CI/CD, web services, identity management)
+**Key Components:** `roles/` (134 reusable roles), `playbooks/` (groups, hosts, openshift-apps, manual), `inventory/` (host/group vars with constructed plugin)
+
+## Technology Stack
+
+### Versions (current as of 2026-03-05)
+- **Ansible** — core automation engine; playbooks target Fedora and RHEL/CentOS hosts
+- **yamllint** v1.35.1 — pre-commit hook for YAML validation
+- **ansible-lint** — CI linting (skips: `yaml`, `role-name[path]`, `var-naming[no-role-prefix]`, `no-changed-when`, `ignore-errors`)
+- **OpenShift 4** (OCP4) — hosts ~60 containerized applications via `openshift-apps/` playbooks
+
+### Target Platforms
+- **Fedora** (current cycle: F43 stable, F44 branched, F45 rawhide)
+- **RHEL/CentOS** 8+ (EPEL 10, current minor: 10.3)
+- **OpenShift 4** cluster (rdu3 datacenter)
+
+### Dev Tools
+- **Linting:** yamllint (pre-commit) + ansible-lint (CI)
+- **CI:** Forgejo Actions on `quay.io/fedora/fedora:latest` — runs yamllint and ansible-lint on changed files only
+- **Control host:** batcave01 (`/srv/web/infra/ansible` public, `/srv/private/ansible` private)
+
+## Resource Organization
+
+### Structure
+```
+├── main.yml                    # Master playbook — imports all group/host playbooks
+├── playbooks/
+│   ├── groups/                 # 59 group playbooks (one per service group)
+│   ├── hosts/                  # 2 host-specific playbooks (FQDN.yml)
+│   ├── openshift-apps/         # 60 OCP4 application deployments
+│   ├── manual/                 # 39 admin-run-only playbooks
+│   └── include/                # Shared proxy/virt/cert playbook fragments
+├── roles/                      # 134 roles
+│   ├── base/                   # Applied to ALL hosts (packages, SSH, SELinux, nftables)
+│   ├── openshift/              # OCP4 primitives (project, object, keytab, route, rollout)
+│   └── openshift-apps/         # Per-app roles (templates, files for OCP4 deployments)
+├── inventory/
+│   ├── inventory/              # Host definitions
+│   ├── group_vars/             # 150 group variable files (incl. _stg variants)
+│   ├── host_vars/              # Per-host overrides
+│   └── zzz-inventory.config    # Constructed inventory plugin (dynamic groups by datacenter, distro, vmhost)
+├── vars/
+│   ├── global.yml              # Global vars (paths, SSL config, base packages)
+│   ├── all/                    # Release cycle vars (Fedora versions, freeze states, EPEL)
+│   ├── apps/                   # Per-application vars (bodhi, mirrormanager, etc.)
+│   ├── Fedora.yml              # Distro-specific packages/services
+│   └── RedHat.yml / CentOS.yml
+├── tasks/                      # Reusable task snippets (cloud, postfix, yumrepos, etc.)
+├── handlers/                   # restart_services.yml
+├── library/                    # Custom modules (delete_old_oci_images.py, virt_boot, etc.)
+├── callback_plugins/           # fedora_messaging_callback.py, logdetail.py
+├── files/                      # Static files/templates organized by service
+└── scripts/                    # Admin utility scripts (auth-keys-from-fas, freezelist, etc.)
+```
+
+### Module/Role Structure
+
+**Standard role layout:** `tasks/main.yml`, `templates/`, `files/`, `handlers/main.yml`, `meta/main.yml`
+
+**OpenShift app pattern** — the dominant pattern for new services:
+1. Playbook in `playbooks/openshift-apps/<app>.yml` targets `os_control[0]:os_control_stg[0]`
+2. Uses composable `openshift/*` roles (`project`, `object`, `keytab`, `secret-file`, `imagestream`, `route`, `rollout`)
+3. App-specific templates in `roles/openshift-apps/<app>/templates/`
+4. Staging vs production controlled by `env` variable and `env_suffix`
+
+**Group playbook pattern** — for traditional VM-based services:
+1. Playbook in `playbooks/groups/<group>.yml` with `hosts:` matching inventory group
+2. Must include standard `vars_files` triplet (see Review Guidance)
+
+### Critical Resources
+
+- **`vars/all/`** — Fedora release cycle variables. Changed every ~6 months during branching/release. Incorrect values break builds, composes, and Bodhi across the entire infrastructure.
+- **`roles/base/tasks/main.yml`** (700+ lines) — Applied to every managed host. Changes here have maximum blast radius.
+- **`inventory/group_vars/`** — 150 files controlling per-group behavior. Many have `_stg` counterparts for staging.
+- **`main.yml`** — Master playbook importing all groups. Nightly `--check --diff` cron runs all playbooks here.
+
+## Review Guidance
+
+### What Reviewers Must Know
+- **All playbooks must be idempotent.** They can be run at any time by the nightly cron. The checked-in state must always be the desired state.
+- **Standard vars_files triplet is required** in all group/host playbooks:
+  ```yaml
+  vars_files:
+    - /srv/web/infra/ansible/vars/global.yml
+    - "{{ private }}/vars.yml"
+    - /srv/web/infra/ansible/vars/{{ ansible_distribution }}.yml
+  ```
+  Plus `include_vars` for `vars/all/` when release cycle vars are needed.
+- **Hardcoded paths are standard** — paths like `/srv/web/infra/ansible/` and `/srv/private/ansible/` are the production layout on batcave01. Don't suggest making them relative or configurable.
+- **Use `yml` not `yaml`** for Ansible files (per STYLEGUIDE). The exception is `vars/all/*.yaml` which uses `.yaml` historically.
+- **Add `.j2` extension** to all Jinja2 templates.
+- **YAML indentation is 2 spaces.** Line length is not enforced.
+- **Prefer readable multi-line module args** over single-line `module: name=x arg=y` format.
+- **Staging uses `_stg` suffixed group_vars** and `env_suffix` variable (empty string for prod, `.stg` for staging).
+- **Tags `packages` and `config`** should be applied to relevant tasks. `build` and `rollout` tags use `never` to prevent accidental execution.
+- **OpenShift apps target `os_control[0]:os_control_stg[0]`** — always the first control node. Don't suggest targeting all control nodes.
+
+### Do NOT Flag (Known False Positives)
+- `ansible-lint` skip list includes `no-changed-when` and `ignore-errors` — these are intentionally suppressed project-wide
+- `yaml` rule category is skipped in ansible-lint — yamllint handles YAML validation separately
+- Hardcoded absolute paths in playbooks (e.g., `/srv/web/infra/ansible/...`) — this is the expected deployment layout
+- `mock_modules` and `mock_roles` in `.ansible-lint` — used to pass syntax checks without all dependencies
+- Octal values forbidden in yamllint — intentional policy to avoid ambiguous YAML parsing
+- `when: env == "production"` / `when: env == "staging"` conditional duplication in openshift-apps — standard pattern for different scaling/config per environment
+
+### Common Pitfalls
+- **Forgetting to update `vars/all/` during release transitions** — these variables control Fedora version numbers, freeze states, and EPEL branches. Multiple files must be updated together (e.g., branching requires changes to `FedoraBranched.yaml`, `00-FedoraCycleNumber.yaml`, `FedoraBranchedBodhi.yaml`, and `Frozen.yaml`).
+- **Not testing with staging first** — staging group_vars (`*_stg`) should be updated before production. Changes that work in staging may still break production due to different scaling or secrets.
+- **Breaking idempotency** — a task that makes changes on every run will generate noise in the nightly `--check --diff` report and mask real drift.
+- **Wrong file extension for templates** — placing a `.yml` file in `templates/` instead of `.yml.j2` means Jinja2 expressions won't be rendered.
+- **ansible-lint file/role misclassification** — the `.ansible-lint` `kinds` section maps `tasks/*.yml` and `vars/*.yml` explicitly. New directories with tasks may need similar mappings.
+
+## Internal & Proprietary
+
+- **`/srv/private/ansible/`** — Private vars (secrets, credentials) stored on batcave01. Referenced as `{{ private }}/vars.yml`. Never committed to this repo.
+- **`callback_plugins/fedora_messaging_callback.py`** — Custom Ansible callback that publishes play results to Fedora's AMQP message bus. Don't suggest replacing with standard callback plugins.
+- **`callback_plugins/logdetail.py`** — Custom detailed logging callback for the nightly check-diff runs.
+- **`library/virt_boot`** / **`library/delete_old_oci_images.py`** — Custom Ansible modules for VM management and OCI image cleanup. Not upstream modules.
+- **`scripts/auth-keys-from-fas`** — Fetches SSH authorized keys from Fedora Account System (FAS). Referenced by `auth_keys_from_fas` global variable.
+- **Constructed inventory plugin** (`zzz-inventory.config`) — Dynamically creates groups by datacenter (`rdu3`), distro, vmhost, and virtualization role. The `zzz-` prefix ensures it loads last.
+
+---
+<!-- MANUAL SECTIONS - DO NOT MODIFY THIS LINE -->
+
+## Architecture & Design Decisions
+
+- **Mostly flat role directory with some nesting**: Most roles live directly under `roles/`, but several use subdirectory namespacing — `openshift/` (OCP4 primitives), `openshift-apps/` (per-app deployments), `awx/`, `openqa/`, `rabbit/`, among others.
+- **OpenShift apps via Ansible, not Helm/Kustomize**: OCP4 applications are deployed through Ansible roles that template and apply OpenShift objects. This keeps all infrastructure in one tool and one repo.
+- **Staging/production parity through group_vars**: Rather than separate inventories, staging hosts are in the same inventory with `_stg` group_vars files providing overrides. The `env` and `env_suffix` variables control behavior.
+- **Release cycle managed through simple YAML vars**: Fedora's complex release lifecycle (rawhide, branched, stable, EOL) is encoded in `vars/all/` as a set of interdependent variables rather than a database or API. This is intentional — the variables are updated manually during each release milestone by the release engineering team.
+
+## Business Logic
+
+- **Fedora release cycle states**: Three main states — unbranched (rawhide only), branched (rawhide + pre-release), and post-release. Controlled by `FedoraBranched`, `FedoraCycleNumber`, `FedoraBranchedBodhi` (preenable/prebeta/postbeta), and freeze flags. These cascade through templates across the entire infrastructure.
+- **EPEL minor version lifecycle**: EPEL 10+ has minor versions that move through states: `epel_minor` (built against CentOS), `epel_branched_minor` (branched, built against CentOS snapshot), `epel_z_minor` (built against RHEL). Up to three active minor versions at once.
+- **Critical path applications**: forge, pagure, mirrormanager, bodhi, koji, dist-git, and ~20 others require two-reviewer PRs and coordinated downtime scheduling for risky changes.
+- **Nightly check-diff**: All playbooks under `playbooks/{groups,hosts}` are run nightly with `--check --diff`. The ideal state is zero changes reported.
+
+## Domain-Specific Context
+
+- **batcave01** — The Ansible control host. All playbooks are run from here via `sudo -i ansible-playbook`.
+- **env / env_suffix** — `env` is `"production"` or `"staging"`. `env_suffix` is `""` for prod, `".stg"` for staging. Used throughout to construct hostnames, queue names, and paths.
+- **FAS (Fedora Account System)** — Identity provider for the Fedora community. SSH keys, group memberships, and permissions come from FAS/IPA.
+- **Koji** — Fedora's build system. Build hosts (buildvm, buildhw) are managed here. Koji hub is VM-based (`playbooks/groups/koji-hub.yml` + `roles/koji_hub`), not on OpenShift.
+- **Bodhi** — Fedora's update management system. Runs on OpenShift with complex RabbitMQ messaging integration.
+- **dist-git / Pagure** — Package source repositories. The lookaside cache and git hosting are managed by separate roles.
+- **RabbitMQ / fedora-messaging** — AMQP message bus connecting Fedora services. Certificates managed per-service via `openshift/secret-file` role.
+
+## Special Cases
+
+- **`playbooks/openshift-apps/` uses `gather_facts: false`** — OCP4 playbooks target the control node to run `oc` commands, not the apps themselves. Facts aren't needed and would slow execution.
+- **`vars/all/*.yaml` uses `.yaml` extension** despite STYLEGUIDE mandating `.yml` — historical exception, don't "fix" this.
+- **`ansible-lint` runs in offline mode** (`offline: true`) — dependencies aren't installed during linting. `mock_modules` and `mock_roles` paper over missing dependencies.
+- **Some playbooks are excluded from ansible-lint** (e.g., `copr-db.yml`, `list-vms-per-host.yml`) due to known issues with hardcoded paths or unicode errors.
+- **`linux-system-roles.network`** is mocked in ansible-lint — it's an external role not available during CI.

.ansible-lint

@@ -67,3 +67,4 @@ skip_list:
   - role-name[path]
   - var-naming[no-role-prefix]
   - no-changed-when
+  - ignore-errors

.forgejo/workflows/ai-review.yaml

@@ -0,0 +1,17 @@
+---
+name: AI Code Review
+on:
+  pull_request_target:
+    types: [labeled]
+
+jobs:
+  ai-review:
+    runs-on: infra-1
+    if: forgejo.event.label.name == 'ai-review-please'
+    uses: quality/workflows/.forgejo/workflows/ai-review.yml@main
+    with:
+      pr: ${{ forgejo.event.pull_request.number }}
+      config: |
+        include_mr_summary: false
+    secrets:
+      GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}

.forgejo/workflows/ci.yaml

@@ -0,0 +1,71 @@
+---
+name: Linter
+on:
+  pull_request:
+    types: [opened, synchronize]
+
+jobs:
+  yamllint:
+    runs-on: infra-1
+    container:
+      image: quay.io/fedora/fedora:latest
+    steps:
+      - name: Install testing tools
+        run: |
+          dnf install -y yamllint nodejs git
+
+      - name: Checkout code
+        uses: https://code.forgejo.org/actions/checkout@v6
+        with:
+          fetch-depth: 0
+
+      - name: Get changed files
+        id: changed-files
+        uses: https://code.forgejo.org/tj-actions/changed-files@v47
+
+      - name: Run yamllint
+        if: steps.changed-files.outputs.any_changed == 'true'
+        run: |
+          for file in ${{ steps.changed-files.outputs.all_changed_files }}; do
+            if [[ "$file" == *.yml || "$file" == *.yaml ]]; then
+              yamllint "$file"
+            fi
+          done
+
+  ansible-lint:
+    runs-on: infra-1
+    container:
+      image: quay.io/fedora/fedora:latest
+    steps:
+      - name: Install testing tools
+        run: |
+          dnf install -y python3-pip nodejs git
+          pip3 install ansible-core ansible-lint
+
+      - name: Install ansible collections
+        run: |
+          ansible-galaxy collection install \
+          community.zabbix \
+          community.general \
+          ansible.posix
+
+      - name: Checkout code
+        uses: https://code.forgejo.org/actions/checkout@v6
+        with:
+          fetch-depth: 0
+
+      - name: Create symlink to overcome absolute paths
+        run: |
+          mkdir -p /srv/web/infra/
+          ln -s /workspace/infra/ansible /srv/web/infra/ansible
+
+      - name: Get changed files
+        id: changed-files
+        uses: https://code.forgejo.org/tj-actions/changed-files@v47
+
+      - name: Run ansible-lint
+        if: steps.changed-files.outputs.any_changed == 'true'
+        run: |
+          for file in ${{ steps.changed-files.outputs.all_changed_files }}; do
+            ansible-lint "$file"
+          done

.forgejo/workflows/yamllint-diff.yaml

@@ -0,0 +1,23 @@
+---
+name: Differential yamllint
+
+"on":
+  push:
+  pull_request:
+    branches: [main]
+
+jobs:
+  yamllint-job:
+    runs-on: infra-1
+    steps:
+      - name: Repository checkout
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 100
+
+      - name: VCS Diff Lint
+        uses: https://github.com/fedora-copr/vcs-diff-lint-action@v1
+        id: VCS_Diff_Lint
+        with:
+          linter_tags: |
+            yamllint

.gitignore

@@ -1,5 +1,11 @@
 *.swp
 *.pyc
 
+# Cursor workspace
+.cursor
+
 # emacs projectile
 .projectile
+
+# ansible-lint directory
+.ansible/

.pre-commit-config.yaml

@@ -1,22 +1,22 @@
----
-# See https://pre-commit.com for more information
-# See https://pre-commit.com/hooks.html for more hooks
-repos:
+---		
+# See https://pre-commit.com for more information		
+# See https://pre-commit.com/hooks.html for more hooks		
+repos:		
 
-# -   repo: https://github.com/pre-commit/pre-commit-hooks
-#    rev: v3.2.0
-#    hooks:
-#    -   id: trailing-whitespace
-#    -   id: end-of-file-fixer
-#    -   id: check-yaml
-#    -   id: check-added-large-files
+# -   repo: https://github.com/pre-commit/pre-commit-hooks		
+#    rev: v3.2.0		
+#    hooks:		
+#    -   id: trailing-whitespace		
+#    -   id: end-of-file-fixer		
+#    -   id: check-yaml		
+#    -   id: check-added-large-files		
 
-- repo: https://github.com/adrienverge/yamllint
-  rev: v1.35.1
-  hooks:
-  - id: yamllint
+- repo: https://github.com/adrienverge/yamllint		
+  rev: v1.35.1		
+  hooks:		
+  - id: yamllint		
 
-#  -   repo: https://github.com/ansible/ansible-lint
-#     rev: v24.12.1
-#     hooks:
-#     -   id: ansible-lint
+#  -   repo: https://github.com/ansible/ansible-lint		
+#     rev: v24.12.1		
+#     hooks:		
+#     -   id: ansible-lint

.yamllint.yaml

@@ -23,4 +23,8 @@ rules:
     # level: warning
   truthy:
     allowed-values: ['true', 'false', 'yes', 'no']
+
+ignore:
+  - '*/templates/*'
+  - '*/openshift-apps/*/templates/*'
 ...

.zuul.yaml

@@ -1,6 +0,0 @@
----
-- project:
-    check:
-      jobs:
-        - fi-ansible-lint-diff
-        - fi-yamllint-diff

AGENTS.md

@@ -0,0 +1,108 @@
+# AGENTS.md
+
+This file provides guidance to AI coding agents when working with code in this repository.
+
+## Project Overview
+
+Ansible automation for the entire Fedora Project infrastructure. Manages hundreds of bare-metal hosts, VMs, and OpenShift 4 applications across production and staging environments. The control host is **batcave01** (`/srv/web/infra/ansible` public, `/srv/private/ansible` private).
+
+Repository is hosted at https://forge.fedoraproject.org/infra/ansible
+
+## Linting
+
+Run yamllint on changed files:
+```sh
+yamllint path/to/file.yml
+```
+
+Run ansible-lint on changed files:
+```sh
+ansible-lint path/to/file.yml
+```
+
+CI runs both linters on changed files only (Forgejo Actions on `quay.io/fedora/fedora:latest`). The `community.zabbix` collection is installed before ansible-lint runs.
+
+### Linting configuration
+
+- **yamllint** (`.yamllint.yaml`): 2-space indentation (warning level), line-length disabled, octal values forbidden, truthy values restricted to `true/false/yes/no`, templates directories ignored.
+- **ansible-lint** (`.ansible-lint`): Runs in offline mode. Skipped rules: `yaml`, `role-name[path]`, `var-naming[no-role-prefix]`, `no-changed-when`, `ignore-errors`. Uses `mock_modules` and `mock_roles` to pass syntax checks without all dependencies.
+
+## Architecture
+
+### Directory structure
+
+- `playbooks/groups/` — One playbook per service group (multi-host). Filename should be descriptive.
+- `playbooks/hosts/` — One playbook per unique host. Filename MUST be `FQDN.yml`.
+- `playbooks/openshift-apps/` — ~60 OCP4 application deployments.
+- `playbooks/manual/` — Admin-only playbooks, never run by cron.
+- `playbooks/include/` — Shared playbook fragments (proxy, virt, cert).
+- `roles/` — 134 roles. Flat structure with some namespacing (`openshift/`, `openshift-apps/`, `awx/`, `rabbit/`).
+- `roles/base/` — Applied to ALL managed hosts. Changes here have maximum blast radius.
+- `inventory/` — Host definitions, `group_vars/` (150 files), `host_vars/`, and `zzz-inventory.config` (constructed inventory plugin, loads last due to `zzz-` prefix).
+- `vars/all/` — Fedora release cycle variables. Changed every ~6 months during branching/release. Incorrect values break builds across the entire infrastructure.
+- `vars/global.yml` — Global vars (paths, SSL, base packages).
+- `tasks/` — Reusable task snippets included in playbooks.
+- `library/` — Custom Ansible modules (`delete_old_oci_images.py`, `virt_boot`, etc.).
+- `callback_plugins/` — Custom callbacks (`fedora_messaging_callback.py`, `logdetail.py`). Don't suggest replacing these.
+- `main.yml` — Master playbook importing all group/host playbooks. Used with `-t tag` to run specific tags across all hosts. Nightly `--check --diff` cron runs all playbooks.
+
+### Two main deployment patterns
+
+**OpenShift app pattern** (dominant for new services):
+1. Playbook in `playbooks/openshift-apps/<app>.yml` targets `os_control[0]:os_control_stg[0]`
+2. Uses composable `openshift/*` roles: `project`, `object`, `keytab`, `secret-file`, `imagestream`, `route`, `rollout`
+3. App templates in `roles/openshift-apps/<app>/templates/`
+4. Uses `gather_facts: false` (runs `oc` commands on control node, not on apps)
+
+**Group playbook pattern** (traditional VM services):
+1. Playbook in `playbooks/groups/<group>.yml` with `hosts:` matching inventory group
+2. Must include standard vars_files (see below)
+
+### Staging vs Production
+
+Same inventory, not separate inventories. Staging hosts use `_stg` suffixed group_vars files. Controlled by:
+- `env` — `"production"` or `"staging"`
+- `env_suffix` — `""` for prod, `".stg"` for staging
+
+## Fedora AI-Assisted Contribution Policy
+
+This repository is part of the Fedora Project and subject to the [Fedora AI-Assisted Contributions Policy](https://docs.fedoraproject.org/en-US/council/policy/ai-contribution-policy/). Key requirements:
+
+- **Accountability**: The human contributor is always the author and is fully accountable for the entirety of AI-assisted contributions. All submissions must meet project standards for quality, license compliance, and utility.
+- **Transparency**: Use of AI tools MUST be disclosed when the significant part of the contribution is taken from a tool without changes. For git contributions, use an `Assisted-by:` commit message trailer (e.g., `Assisted-by: ChatGPTv5` or `Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>`).
+- **No AI-only judgments**: AI MUST NOT be used as the sole or final arbiter for substantive or subjective judgments on contributions, nor for evaluating a person's standing in the community. Automated objective validation (CI/CD, testing, linting) is permitted.
+
+When generating commit messages, always include the `Assisted-by:` trailer naming the specific AI model and version used.
+
+## Coding Conventions
+
+### Required vars_files in all group/host playbooks
+```yaml
+vars_files:
+  - /srv/web/infra/ansible/vars/global.yml
+  - "{{ private }}/vars.yml"
+  - /srv/web/infra/ansible/vars/{{ ansible_distribution }}.yml
+```
+These hardcoded paths are the production layout on batcave01. Do not make them relative or configurable.
+
+### Style rules
+- Use `.yml` not `.yaml` for file extensions. Exception: `vars/all/*.yaml` is historical — don't "fix" this.
+- Add `.j2` extension to all Jinja2 templates. A `.yml` file in `templates/` without `.j2` won't render Jinja expressions.
+- 2-space YAML indentation. Line length is not enforced.
+- Prefer readable multi-line module args over single-line `module: name=x arg=y` format.
+- Standard tags: `packages` (installs/removes packages), `config` (installs config files).
+- `build` and `rollout` tags use `never` to prevent accidental execution.
+- All playbooks must be **idempotent** — they can be run at any time by the nightly cron.
+
+### Common pitfalls
+- Forgetting to update multiple files in `vars/all/` together during release transitions (e.g., branching requires `FedoraBranched.yaml`, `00-FedoraCycleNumber.yaml`, `FedoraBranchedBodhi.yaml`, and `Frozen.yaml`).
+- Breaking idempotency — generates noise in nightly `--check --diff` and masks real drift.
+- OpenShift apps must target `os_control[0]:os_control_stg[0]`, not all control nodes.
+
+### Domain terminology
+- **batcave01** — Ansible control host
+- **FAS/IPA** — Fedora Account System (identity provider)
+- **Koji** — Build system (VM-based, not OpenShift)
+- **Bodhi** — Update management (runs on OpenShift)
+- **dist-git/src.fedoraproject.org** — Package source repositories
+- **fedora-messaging/RabbitMQ** — AMQP message bus connecting services

README.md

@@ -1,19 +1,19 @@
 Fedora Infrastructure
 =====================
 
-Welcome! This is the Fedora Infrastructure Ansible Pagure project.
+Welcome! This is the Fedora Infrastructure Ansible Forge project.
 
 Pull requests and forks can be made against this repository hosted
-at https://pagure.io/fedora-infra/ansible
+at <https://forge.fedoraproject.org/infra/ansible>
 
 This repository is also mirrored for production runs to
-https://infrastructure.fedoraproject.org/infra/ansible/
+<https://infrastructure.fedoraproject.org/infra/ansible/>
 but this is the working repository where changes are made.
 
 If you would like to help out with Fedora Infrastructure, see:
 
-* https://docs.fedoraproject.org/en-US/infra/gettingstarted/
-* https://docs.fedoraproject.org/en-US/infra/apprentice/
+* <https://docs.fedoraproject.org/en-US/infra/gettingstarted/>
+* <https://docs.fedoraproject.org/en-US/infra/apprentice/>
 
 Ansible repository/structure
 ----------------------------
@@ -100,19 +100,19 @@ Contributing and Licensing
 Contributions to this repository are subject to the Fedora Project
 Contributor Agreement. If no license is specified, the MIT license is used, otherwise
 the contribution is under the specified acceptable Fedora License.
-See https://docs.fedoraproject.org/en-US/legal/fpca/
+See <https://docs.fedoraproject.org/en-US/legal/fpca/>
 for more information.
 
 Contributing Pull Requests
 --------------------------
 
 If found a way to improve this repository or fix an issue found in our
-infrastructure tracker (see https://forge.fedoraproject.org/infra/tickets)
+infrastructure tracker (see <https://forge.fedoraproject.org/infra/tickets>)
 open a pull-request.
 
 You either should have capability to run the playbooks after they have been reviewed,
 and merged or find the person responsible and work with them to make sure the changes
-will be aplied afterwards. 
+will be aplied afterwards.
 
 We are currently working on a simple to use list of Point Of Contanct people for the applications
 here, untill it is done, you can, look at people that recently edited the ansible files,
@@ -120,7 +120,7 @@ or if you belong to sysadmin group, view the /etc/ansible_utils/rbac.yaml locate
 where you can see the groups of people that have capabilities to run the relevant playbooks.
 
 For example, to upgrade Release Monitoring, you need to run playbook openshift-apps/release-monitoring.yaml.
-People in sysadmin-releasemonitoring have that capability, and you cand find the members in https://accounts.fedoraproject.org/group/sysadmin-releasemonitoring/
+People in sysadmin-releasemonitoring have that capability, and you cand find the members in <https://accounts.fedoraproject.org/group/sysadmin-releasemonitoring/>
 
 If the application in question is not on the critical path it should be sufficient,
 if person responsible for the application reviews the PR.
@@ -131,13 +131,12 @@ at least two different people should review the PR.
 If there is any risk at all, that the application of the changes would induce downtime,
 work closely with other to ensure that the downtime is properly scheduled:
 
-- there is an issue in https://forge.fedoraproject.org/infra/tickets specifying the downtime
-- there is an email sent to the devel-list
-- https://status.fedoraproject.org is updated (see https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/status-fedora/)
+* there is an issue in <https://forge.fedoraproject.org/infra/tickets> specifying the downtime
+* there is an email sent to the devel-list
+* <https://status.fedoraproject.org> is updated (see <https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/status-fedora/>)
 
-Applications on critical path: pagure, mirrormanager, toddlers, bodhi, noggin, mdapi, rpmautospec, pagure-dist-git, mirror_from_pagure, fedora-messaging, dist-git, PDC/FPDC, FMN, sigul
+Applications on critical path: forge, pagure, mirrormanager, toddlers, bodhi, noggin, mdapi, rpmautospec, pagure-dist-git, mirror_from_pagure, fedora-messaging, dist-git, PDC/FPDC, FMN, sigul
 robosignatory, tag2distrepo, ci-resultsdb-listener, stylo, mirrorlist
-resultsdb, Nagios, koschei, wiki / mediawiki, wiki / moin, waiverdb, 
-greenwave, ODCS, Mailman3 / HK, mailman 2, OSBS, pungi, koji, MBS, 
+resultsdb, Nagios, koschei, wiki / mediawiki, wiki / moin, waiverdb,
+greenwave, ODCS, Mailman3 / HK, mailman 2, OSBS, pungi, koji, MBS,
 IPA, rabbitmq, geoip,ipsilon
-

files/common/RPM-GPG-KEY-INFRASTRUCTURE-SIGUL-20260709

@@ -0,0 +1,29 @@
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+
+mQINBGpQExgBEADP5+9qybH7gg2aapWmy7dBLL64j45ALFyYtZCFprsFALW4OdIH
+5jQwnt/k3ErygvVA2HTsN85f2vTnBUeRyCcZhgx6wLbfxSMqI1eCvWfTgTQ+Xyd/
+TB8eDUW5hVPtW3M/9lCNm3gW8FFsY8eo62gbKQFMEHqv4jzvycJHC+dLSJwqDhRo
+8QXcw8Xa3yyD1i4x/CJsxgnkbAuNN3FvJ0ALdc4QkeCslTV0R/APZYojOsdOxxj9
+6OM8KsHTofoM90x7uDH5SBYfszA/dV38s7IIRMdhQVff7Vyru1Wg19xWAV1DB/XD
+fIarhFV+UuzjdGBdYHCh8dJ7f+l3IvzeYtNDdgf1uho6TPU79OaGF5BQdyq3h3Ep
+n7MvP7kDmf3zp/169ED1KafBzKlCxCB1f6C5icaddl4GXHhsSnLVU6GNWVDYzs49
+PT1iJquGqQIVrdvZPVRxF9EajtOy2ihZrGuyu0CfEhmlxbxodRhl4QWxHtqI9VQf
+qr8hNt3PJjQ9ZXF+39u65aV8VmsZm0ifwLjXf028KHYEn/ZxIADJpf6cs3tST/FC
+NUdWydaRy15wHZGAW4DjITJYyOaxS/O88c0AYiPpyIzjcdM5X6yPd/Ks7jNp4jix
+xq5+in4kT6Ob6i/vLGPiMOg2lgoDyk9Se4Xnfj0mji8BA+0Kd8822UQZtQARAQAB
+tEBGZWRvcmEgSW5mcmFzdHJ1Y3R1cmUgKGluZnJhc3RydWN0dXJlKSA8YWRtaW5A
+ZmVkb3JhcHJvamVjdC5vcmc+iQJSBBMBCAA8FiEEZsXzm0iCia3GFtyfH6GpJwHY
+P8QFAmpQExgCGw8FCwkIBwIDIgIBBhUKCQgLAgQWAgMBAh4HAheAAAoJEB+hqScB
+2D/ET3wP/RvrubZbpK5TGSnLnxbOx74RfjSW7B01cKuLEnZV4VMMlqZONTpvMPBT
+t69aifiuVYsD1Pynj+WA0HduZdLnoczrlIYAqeAPwGOPz/ogmsEeTtnlCLzAT8Gr
+VIT0MR/jhP/IUVuZQ0t7wY2Ezl7yYRnYiB4y8ZuofF8hmby363bceMsX0T/aDQNy
+brbWKF38Ag8FQfzhTnYHnlBVUdo0Sdu94uINoW0jh6Y+ml+zusx4rP+BBqEvSQIG
+xYEbWaDxxGyrCIIADz6yz6/0w6HYBh8LgqPOfRKqo/0KTe4ZuzX3Ao8r03ZIiQj4
+7N+dA8bC9lPgHsoa6hMXamR6Mh1m/561RIzPSN8bkygg3d+Og4sglGI0FojFmdF7
+C/WHZwXpMnLkaDIQgju0JCVO9Blut85u7u0WM5AcoEC7ODsv+kxAIFiTAZVf2zh2
+1fV+lDyMVOtbykdn7YjbRvFGj3++2CJq+jP3S0Pnsc34eKsY+bw0pmCV2Wu1VPeA
+V8hnpeGqz2vyhG8TqSXZnSgWaVR5T52GCNiohqPBMaOrNFa5xoV+FYPAZbPEJf/A
+m85HsYgzmE9DbExX9JxrPpPiCmBe/DDmweP7slq+016OUHafR0shOBXVH5OFLU+w
+esbMjQFiccYct/oGbfUZVIyDBxAL3Wq8yGzzI4sskR5HqoYIeNYb
+=2YPV
+-----END PGP PUBLIC KEY BLOCK-----

files/common/epel10.repo

@@ -1,9 +1,10 @@
 [epel]
 name=Extras Packages for Enterprise Linux $releasever - $basearch
-#  This is a bit too magic, esp. as we are using the infra. repo. and explicitly
-# point to a 10.x variant ... might as well do the same here.
+# We aren't using mirrormanager here, but pointing direct to our repos
+# So, we want to just point to '10' here and depend on the link that exists
+# to point us to the newest minor release.
 # baseurl=https://infrastructure.fedoraproject.org/pub/epel/$releasever${releasever_minor:+z}/Everything/$basearch/
-baseurl=https://infrastructure.fedoraproject.org/pub/epel/10.1/Everything/$basearch/
+baseurl=https://infrastructure.fedoraproject.org/pub/epel/10z/Everything/$basearch/
 enabled=1
 gpgcheck=1
 gpgkey=https://infrastructure.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-10
@@ -11,7 +12,7 @@ gpgkey=https://infrastructure.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-10
 [epel-testing]
 name=Extras Packages for Enterprise Linux $releasever - $basearch
 # baseurl=https://infrastructure.fedoraproject.org/pub/epel/testing/$releasever${releasever_minor:+z}/Everything/$basearch/
-baseurl=https://infrastructure.fedoraproject.org/pub/epel/testing/10.1/Everything/$basearch/
+baseurl=https://infrastructure.fedoraproject.org/pub/epel/testing/10z/Everything/$basearch/
 enabled=0
 gpgcheck=1
 gpgkey=https://infrastructure.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-10

files/common/rhel-infra-tags-20260709.repo

@@ -0,0 +1,6 @@
+[infrastructure-tags]
+name=Fedora Infrastructure tag $releasever - $basearch
+baseurl=https://kojipkgs.fedoraproject.org/repos-dist/epel$releasever-infra/latest/$basearch/
+enabled=1
+gpgcheck=1
+gpgkey=https://infrastructure.fedoraproject.org/repo/infra/RPM-GPG-KEY-INFRA-TAGS-20260709

files/common/rhel-infra-tags-stg-20260709.repo

@@ -0,0 +1,6 @@
+[infrastructure-tags-stg]
+name=Fedora Infrastructure tag $releasever - $basearch
+baseurl=https://kojipkgs.fedoraproject.org/repos-dist/epel$releasever-infra-stg/latest/$basearch/
+enabled=1
+gpgcheck=1
+gpgkey=https://infrastructure.fedoraproject.org/repo/infra/RPM-GPG-KEY-INFRA-TAGS-20260709

files/common/rhel10.repo

@@ -12,24 +12,25 @@ gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta,file:///etc/pki/rpm-gpg
 enabled=1
 gpgcheck=1
 
-# NOTE: When you change the 10.x here also change it in the epel10.repo file.
+# These are not using mirrormanager, so point to '10' which will use the link
+# to the latest minor release.
 [rhel10-BaseOS]
 name = rhel10 BaseOS $basearch
-baseurl=https://infrastructure.fedoraproject.org/repo/rhel/rhel10/10.1/repos/$basearch/baseos
+baseurl=https://infrastructure.fedoraproject.org/repo/rhel/rhel10/10/repos/$basearch/baseos
 gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta,file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
 enabled=1
 gpgcheck=1
 
 [rhel10-AppStream]
 name = rhel10 AppStream $basearch
-baseurl=https://infrastructure.fedoraproject.org/repo/rhel/rhel10/10.1/repos/$basearch/appstream
+baseurl=https://infrastructure.fedoraproject.org/repo/rhel/rhel10/10/repos/$basearch/appstream
 gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta,file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
 enabled=1
 gpgcheck=1
 
 [rhel10-CRB]
 name = rhel10 CodeReadyBuilder $basearch
-baseurl=https://infrastructure.fedoraproject.org/repo/rhel/rhel10/10.1/repos/$basearch/crb
+baseurl=https://infrastructure.fedoraproject.org/repo/rhel/rhel10/10/repos/$basearch/crb
 gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta,file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
 enabled=1
 gpgcheck=1

files/download/download-sync.cron.ib01

@@ -1,4 +1,2 @@
 # Run quick mirror fedora every 10minutes
 */10 * * * * root flock -n -E0 /tmp/download-sync -c '/root/quick-fedora-mirror/quick-fedora-mirror -c /root/quick-fedora-mirror/quick-fedora-mirror.conf'
-## Need to run this to keep centos mirror up to date.
-00 1,7,13,19 * * * root /usr/local/bin/lock-wrapper sync-up-downloads "/usr/local/bin/sync-up-centos"

files/download/download-sync.cron.iso01

@@ -1,5 +1,3 @@
 # Run quick mirror fedora every 10minutes
 MAILTO=root@fedoraproject.org
 */10 * * * * root flock -n -E0 /tmp/download-sync -c '/root/quick-fedora-mirror/quick-fedora-mirror -c /root/quick-fedora-mirror.conf'
-## Need to run this to keep centos mirror up to date.
-00 1,7,13,19 * * * root /usr/local/bin/lock-wrapper sync-up-downloads "/usr/local/bin/sync-up-centos"

files/download/sync-up-downloads.sh.ib01

@@ -8,7 +8,7 @@
 RSYNC='/usr/bin/rsync'
 RS_OPT="-avSHP  --numeric-ids"
 RS_DEADLY="--delete --delete-excluded --delete-delay --delay-updates"
-ALT_EXCLUDES=""
+ALT_EXCLUDES="--exclude 44_Beta*" # 2026-04-16 - tmp. running out of space
 EPL_EXCLUDES=""
 FED_EXCLUDES=""
 

files/httpd/fedorahosted-redirects.conf

@@ -39,7 +39,7 @@ RewriteRule ^/released/i/m/imapsync https://pagure.io/releases/imapsync [R=301]
 RewriteRule ^/released/imapsync https://pagure.io/releases/imapsync [R=301]
 
 RewriteRule ^/fedora-infrastructure/report https://forge.fedoraproject.org/infra/tickets [R=301]
-RewriteRule ^/fedora-infrastructure/ticket/(.*) https://forge.fedoraproject.org/infra/tickets/$1 [R=301]
+RewriteRule ^/fedora-infrastructure/ticket/(.*) https://forge.fedoraproject.org/infra/tickets/issues/$1 [R=301]
 RewriteRule ^/fedora-infrastructure https://forge.fedoraproject.org/infra/tickets [R=301]
 
 RewriteRule ^/fesco/report https://pagure.io/fesco/issues [R=301]
@@ -90,13 +90,13 @@ RewriteRule ^/gfs2-utils/report https://pagure.io/gfs2-utils/issues [R=301]
 RewriteRule ^/gfs2-utils/ticket/(.*) https://pagure.io/gfs2-utils/issue/$1 [R=301]
 RewriteRule ^/gfs2-utils https://pagure.io/gfs2-utils [R=301]
 
-RewriteRule ^/elections/report https://pagure.io/elections/issues [R=301]
-RewriteRule ^/elections/ticket/(.*) https://pagure.io/elections/issue/$1 [R=301]
-RewriteRule ^/elections https://pagure.io/elections [R=301]
+RewriteRule ^/elections/report https://forge.fedoraproject.org/apps/elections/issues [R=301]
+RewriteRule ^/elections/ticket/(.*) https://forge.fedoraproject.org/apps/elections/issues/$1 [R=301]
+RewriteRule ^/elections https://forge.fedoraproject.org/apps/elections [R=301]
 
-RewriteRule ^/fedocal/report https://pagure.io/fedocal/issues [R=301]
-RewriteRule ^/fedocal/ticket/(.*) https://pagure.io/fedocal/issue/$1 [R=301]
-RewriteRule ^/fedocal https://pagure.io/fedocal [R=301]
+RewriteRule ^/fedocal/report https://forge.fedoraproject.org/apps/fedocal/issues [R=301]
+RewriteRule ^/fedocal/ticket/(.*) https://forge.fedoraproject.org/apps/fedocal/issue/$1 [R=301]
+RewriteRule ^/fedocal https://forge.fedoraproject.org/apps/fedocal [R=301]
 
 RewriteRule ^/FedoraReview/report https://pagure.io/FedoraReview/issues [R=301]
 RewriteRule ^/FedoraReview/ticket/(.*) https://pagure.io/FedoraReview/issue/$1 [R=301]

files/scripts/file-retention-iso-date

@@ -0,0 +1,94 @@
+#!/bin/bash
+# vim: ts=4:sw=4:expandtab:tw=100
+
+# This script manages a set of backup files which are prefixed with ISO dates  (YYYY-MM-DD).
+# - As new files come in, we want to delete the old ones.
+#   - RETENTION_DAYS decides how old is "old".
+# - But if no new files come in (e.g. the backup process is broken), we don't want to
+#   end up deleting _all_ the files, just because they're old.
+#   - MIN_KEEP decides how many files to keep.
+#
+# Usage: ./retain.sh --dry-run <directory>
+
+set -euo pipefail
+
+DRY_RUN=false
+DIR=""
+
+# Parse arguments (extremely verbosely to satisfy the LLM reviewing this PR)
+while [[ $# -gt 0 ]]; do
+    case "$1" in
+        --dry-run)
+            DRY_RUN=true
+            shift
+            ;;
+        -*)
+            echo "Error: Unknown flag '$1'" >&2
+            exit 1
+            ;;
+        *)
+            if [[ -z "$DIR" ]]; then
+                DIR="$1"
+                shift
+            else
+                echo "Error: Multiple positional arguments provided. Expected only DIR." >&2
+                exit 1
+            fi
+            ;;
+    esac
+done
+
+# Validate required argument
+if [[ -z "$DIR" ]]; then
+    echo "Error: DIR argument is required." >&2
+    echo "Usage: $0 [--dry-run] DIR" >&2
+    exit 1
+fi
+
+RETENTION_DAYS="${RETENTION_DAYS:-31}"
+MIN_KEEP="${MIN_KEEP:-31}"
+CUTOFF=$(date -d "${RETENTION_DAYS} days ago" +%Y-%m-%d)
+
+# Collect all dated files (basename matches YYYY-MM-DD-*)
+mapfile -t DATED_FILES < <(
+    find "$DIR" -maxdepth 1 -type f -name "[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]-*" \
+        | sort -r # sort descending by filename (date first)
+)
+
+total=${#DATED_FILES[@]}
+
+$DRY_RUN && echo "*** DRY RUN — no files will be deleted ***"
+echo ""
+
+if [[ $total -eq 0 ]]; then
+    echo "No dated files found in '$DIR'."
+    exit 0
+fi
+
+deleted=0
+skipped_retention=0
+
+for i in "${!DATED_FILES[@]}"; do
+    filepath="${DATED_FILES[$i]}"
+    filename=$(basename "$filepath")
+    file_date="${filename:0:10}"  # extract YYYY-MM-DD
+
+    # Always keep the MIN_KEEP most-recent files (indices 0..MIN_KEEP-1)
+    if [[ $i -lt $MIN_KEEP ]]; then
+        echo "KEEP (newest ${MIN_KEEP}): $filename"
+        continue
+    fi
+
+    # Delete if the file's date is before the cutoff
+    if [[ "$file_date" < "$CUTOFF" ]]; then
+        echo "DELETE (expired): $filename"
+        $DRY_RUN || rm -- "$filepath"
+        deleted="$((deleted + 1))"
+    else
+        echo "KEEP (within retention): $filename"
+        skipped_retention="$((skipped_retention + 1))"
+    fi
+done
+
+echo ""
+echo "Deleted: $deleted files (dry-run=$DRY_RUN).  Kept by min-count: $(( total < MIN_KEEP ? total : MIN_KEEP )).  Kept by date window: $skipped_retention."

files/scripts/openshift-app-image-report/openshift-app-image-notes.yml

@@ -0,0 +1,22 @@
+---
+# Optional annotations for openshift-app-image-report.py.
+#
+# The scanner reads the OpenShift app files directly. Use this file for cases
+# that static scanning cannot infer, such as Jinja variables, external
+# Dockerfiles, or manually maintained tags.
+
+# fill in like
+# annotations:
+#   - app: <roles/openshift-apps/THIS>
+#     match: <string to match in image/source or path>
+#     os_version: <replacemet version text>
+#     kind: <optional replacement kind>
+#     note: <display note>
+# EXAMPLE:
+# annotations:
+#   - app: transtats
+#     match: registry.fedoraproject.org/fedora:34
+#     os_version: Fedora 34
+#     note: Inline Dockerfile base image; should be reviewed for EOL.
+
+annotations: []

files/scripts/openshift-app-image-report/openshift-app-image-report.py

@@ -0,0 +1,522 @@
+#! /usr/bin/python3
+
+import argparse
+import os
+import re
+import sys
+from dataclasses import dataclass
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:
+    yaml = None
+
+
+SCRIPT_DIR = Path(__file__).resolve().parent
+NOTES_FILE = SCRIPT_DIR / "openshift-app-image-notes.yml"
+
+
+def find_repo_root(start):
+    for path in (start, *start.parents):
+        if (path / "roles" / "openshift-apps").is_dir():
+            return path
+    print("Error: could not find roles/openshift-apps above script path", file=sys.stderr)
+    sys.exit(2)
+
+
+REPO_ROOT = find_repo_root(SCRIPT_DIR)
+APP_ROOT = REPO_ROOT / "roles" / "openshift-apps"
+
+IMAGE_PREFIXES = (
+    "docker.io/",
+    "docker-registry.",
+    "fedora:",
+    "image-registry.",
+    "quay.io/",
+    "redis:",
+    "registry.access.redhat.com/",
+    "registry.fedoraproject.org/",
+    "solr:",
+    "valkey/",
+    "busybox",
+)
+
+
+@dataclass(frozen=True)
+class Finding:
+    app: str
+    env: str
+    kind: str
+    image: str
+    os_version: str
+    floating: bool
+    unresolved: bool
+    path: str
+    line: int
+    note: str = ""
+
+
+def _strip_value(value):
+    value = value.strip()
+    if value.startswith(("'", '"')) and value.endswith(("'", '"')):
+        value = value[1:-1]
+    return value.strip()
+
+
+def _looks_like_image(value):
+    value = _strip_value(value)
+    if not value:
+        return False
+    if value.startswith(IMAGE_PREFIXES):
+        return True
+    if "/" in value and ":" in value:
+        return True
+    if ":" in value and not value.startswith(("http://", "https://")):
+        return True
+    return False
+
+
+def _tag_from_image(image):
+    if "@" in image:
+        return image.rsplit("@", 1)[1]
+    # Only inspect the image-name tail so registry ports are not treated as tags.
+    tail = image.rsplit("/", 1)[-1]
+    if ":" not in tail:
+        return ""
+    return tail.rsplit(":", 1)[1]
+
+
+def _is_floating(image):
+    if "{{" in image:
+        return False
+    tag = _tag_from_image(image)
+    return tag in ("", "latest")
+
+
+def _python_version(text):
+    match = re.search(r"python[-:]?(\d)(\d{1,2})", text)
+    if not match:
+        return ""
+    major, minor = match.groups()
+    return f"Python {major}.{minor}"
+
+
+def _infer_os_version(image):
+    low = image.lower()
+
+    match = re.search(
+        r"(?:registry\.fedoraproject\.org/|quay\.io/fedora/)?fedora:(\d+|latest)", low
+    )
+    if match:
+        return f"Fedora {match.group(1)}"
+
+    match = re.search(r"ubi(\d+)", low)
+    if match:
+        parts = [f"UBI {match.group(1)}"]
+        pyver = _python_version(low)
+        if pyver:
+            parts.append(pyver)
+        nginx = re.search(r"nginx[-:]?(\d)(\d{2})", low)
+        if nginx:
+            parts.append(f"nginx {nginx.group(1)}.{nginx.group(2)}")
+        return " / ".join(parts)
+
+    pyver = _python_version(low)
+    if pyver:
+        return pyver
+
+    if low.startswith("busybox"):
+        return "busybox"
+    if low.startswith("redis:"):
+        return "redis"
+    if "bitnami" in low and re.search(r"(^|[/:_-])redis($|[/:_.-])", low):
+        return "redis"
+    if low.startswith("solr:"):
+        return "Solr"
+    if low.startswith("valkey/") or low.startswith("valkey:"):
+        return "Valkey"
+
+    return "?"
+
+
+def _source_kind(line, value):
+    stripped = line.strip()
+    if stripped.startswith("FROM "):
+        return "dockerfile-from"
+    if stripped.startswith("image:"):
+        return "runtime-image"
+    if stripped.startswith("dockerfilePath:"):
+        return "dockerfile-path"
+    if stripped.startswith("uri:"):
+        return "git-source"
+    if stripped.startswith("ref:"):
+        return "git-ref"
+    if stripped.startswith("name:"):
+        if value.startswith("image-registry.") or value.startswith("docker-registry."):
+            return "internal-image"
+        return "image-or-builder"
+    return "image-reference"
+
+
+def _env_from_jinja_condition(line):
+    if "env" not in line:
+        return None
+    if re.search(r"env\s*==\s*['\"]staging['\"]", line):
+        return "staging"
+    if re.search(r"env\s*==\s*['\"]production['\"]", line):
+        return "production"
+    return "conditional"
+
+
+def _opposite_env(env):
+    if env == "staging":
+        return "production"
+    if env == "production":
+        return "staging"
+    return "conditional"
+
+
+def _update_env_scope(line, env_stack, current_env):
+    stripped = line.strip()
+    if not stripped.startswith("{%"):
+        return current_env
+
+    if re.match(r"{%-?\s*if\b", stripped):
+        next_env = _env_from_jinja_condition(stripped)
+        env_stack.append((current_env, next_env))
+        return next_env or current_env
+
+    if re.match(r"{%-?\s*elif\b", stripped):
+        next_env = _env_from_jinja_condition(stripped)
+        if env_stack:
+            previous_env, _old_env = env_stack[-1]
+            env_stack[-1] = (previous_env, next_env)
+        return next_env or current_env
+
+    if re.match(r"{%-?\s*else\s*-?%}", stripped):
+        if not env_stack:
+            return "conditional"
+        _previous_env, if_env = env_stack[-1]
+        if if_env is None:
+            return current_env
+        return _opposite_env(if_env)
+
+    if re.match(r"{%-?\s*endif\s*-?%}", stripped):
+        if not env_stack:
+            return "all"
+        previous_env, _if_env = env_stack.pop()
+        return previous_env
+
+    return current_env
+
+
+def _line_env(line, current_env):
+    if "{{" in line and "env" in line and "ternary" in line:
+        return "conditional"
+    return current_env
+
+
+def _iter_scan_files(app_dir):
+    for path in sorted(app_dir.rglob("*")):
+        if not path.is_file():
+            continue
+        rel_parts = path.relative_to(app_dir).parts
+        if not rel_parts or rel_parts[0] not in ("files", "templates", "vars"):
+            continue
+        name = path.name.lower()
+        suffix = path.suffix.lower()
+        if suffix in (".yml", ".yaml", ".j2", ".toml", ".cfg"):
+            yield path
+            continue
+        if "dockerfile" in name or "containerfile" in name:
+            yield path
+
+
+def _scan_line(line):
+    stripped = line.strip()
+    if not stripped or stripped.startswith("#"):
+        return None
+
+    if stripped.startswith("FROM "):
+        parts = stripped.split()
+        if len(parts) > 1:
+            return parts[1], True
+        return None
+
+    for key in ("image:", "name:", "dockerfilePath:", "uri:", "ref:"):
+        if not stripped.startswith(key):
+            continue
+        value = _strip_value(stripped[len(key) :])
+        if key == "dockerfilePath:":
+            return value, False
+        if key in ("uri:", "ref:"):
+            if "github.com" in value or "gitlab" in value or "{{" in value:
+                return value, False
+            return None
+        if key == "image:" and "{{" in value:
+            return value, True
+        if _looks_like_image(value):
+            return value, True
+        return None
+
+    return None
+
+
+def scan_apps(app_filter=None):
+    findings = []
+    wanted = set(app_filter or [])
+    for app_dir in sorted(APP_ROOT.iterdir()):
+        if not app_dir.is_dir():
+            continue
+        app = app_dir.name
+        if wanted and app not in wanted:
+            continue
+        for path in _iter_scan_files(app_dir):
+            rel_path = path.relative_to(REPO_ROOT).as_posix()
+            try:
+                lines = path.read_text(encoding="utf-8").splitlines()
+            except UnicodeDecodeError:
+                continue
+            env_stack = []
+            current_env = "all"
+            for lineno, line in enumerate(lines, 1):
+                current_env = _update_env_scope(line, env_stack, current_env)
+                scanned = _scan_line(line)
+                if not scanned:
+                    continue
+                value, is_image = scanned
+                unresolved = "{{" in value or "{%" in value
+                if line.strip().startswith("dockerfilePath:"):
+                    unresolved = True
+                kind = _source_kind(line, value)
+                os_version = _infer_os_version(value) if is_image else "?"
+                findings.append(
+                    Finding(
+                        app=app,
+                        env=_line_env(line, current_env),
+                        kind=kind,
+                        image=value,
+                        os_version=os_version,
+                        floating=_is_floating(value) if is_image else False,
+                        unresolved=unresolved,
+                        path=rel_path,
+                        line=lineno,
+                    )
+                )
+    return findings
+
+
+def load_annotations(notes_file):
+    if not notes_file.exists():
+        return []
+    if yaml is None:
+        print(
+            "Error: python3-yaml is required to read openshift-app-image-notes.yml",
+            file=sys.stderr,
+        )
+        sys.exit(2)
+    with notes_file.open(encoding="utf-8") as stream:
+        data = yaml.safe_load(stream) or {}
+    annotations = data.get("annotations", [])
+    if not isinstance(annotations, list):
+        print(
+            "Error: annotations must be a list in openshift-app-image-notes.yml",
+            file=sys.stderr,
+        )
+        sys.exit(2)
+    return annotations
+
+
+def apply_annotations(findings, annotations):
+    by_key = {}
+    for item in annotations:
+        if not isinstance(item, dict):
+            continue
+        app = item.get("app")
+        match = item.get("match", "")
+        note = item.get("note", "")
+        os_version = item.get("os_version", "")
+        kind = item.get("kind", "")
+        if not app or not match:
+            continue
+        by_key[(app, match)] = (note, os_version, kind)
+
+    used = set()
+    updated = []
+    for finding in findings:
+        note = finding.note
+        os_version = finding.os_version
+        kind = finding.kind
+        for (app, match), values in by_key.items():
+            if app != finding.app:
+                continue
+            if match not in finding.image and match not in finding.path:
+                continue
+            used.add((app, match))
+            ann_note, ann_os, ann_kind = values
+            note = ann_note or note
+            os_version = ann_os or os_version
+            kind = ann_kind or kind
+        updated.append(
+            Finding(
+                app=finding.app,
+                env=finding.env,
+                kind=kind,
+                image=finding.image,
+                os_version=os_version,
+                floating=finding.floating,
+                unresolved=finding.unresolved,
+                path=finding.path,
+                line=finding.line,
+                note=note,
+            )
+        )
+    return updated, sorted(set(by_key) - used)
+
+
+def filter_findings(findings, args):
+    filtered = findings
+    if args.env:
+        wanted = set(args.env)
+        env_matches = set(wanted)
+        if "staging" in wanted or "production" in wanted:
+            env_matches.add("all")
+        filtered = [item for item in filtered if item.env in env_matches]
+    if args.kind:
+        wanted = set(args.kind)
+        filtered = [item for item in filtered if item.kind in wanted]
+    if args.floating_only:
+        filtered = [item for item in filtered if item.floating]
+    if args.unresolved_only:
+        filtered = [item for item in filtered if item.unresolved]
+    return filtered
+
+
+def _short_path(finding):
+    prefix = f"roles/openshift-apps/{finding.app}/"
+    path = finding.path
+    if path.startswith(prefix):
+        path = path[len(prefix) :]
+    return f"{path}:{finding.line}"
+
+
+def _clip(value, width):
+    if len(value) <= width:
+        return value
+    if width <= 1:
+        return value[:width]
+    return value[: width - 3] + "..."
+
+
+def print_table(findings):
+    columns = (
+        ("APP", 22),
+        ("ENV", 11),
+        ("KIND", 16),
+        ("OS/VERSION", 20),
+        ("FLOAT", 5),
+        ("PATH", 58),
+        ("NOTE", 30),
+        ("IMAGE/SOURCE", None),
+    )
+    rows = []
+    for finding in findings:
+        rows.append(
+            (
+                finding.app,
+                finding.env,
+                finding.kind,
+                finding.os_version,
+                "yes" if finding.floating else "",
+                _short_path(finding),
+                finding.note,
+                ("?" if finding.unresolved else "") + finding.image,
+            )
+        )
+
+    print(
+        "  ".join(
+            name if width is None else name.ljust(width) for name, width in columns
+        )
+    )
+    print(
+        "  ".join(
+            "-" * len(name) if width is None else "-" * width for name, width in columns
+        )
+    )
+    for row in rows:
+        print(
+            "  ".join(
+                value if width is None else _clip(value, width).ljust(width)
+                for value, (_name, width) in zip(row, columns)
+            )
+        )
+    print(f"\n{len(rows)} findings")
+
+
+def parse_args(argv):
+    parser = argparse.ArgumentParser(
+        description="Print OpenShift app image and base-image references found in roles/openshift-apps.",
+    )
+    parser.add_argument(
+        "--app",
+        action="append",
+        help="Only report one app. May be used more than once.",
+    )
+    parser.add_argument(
+        "--kind",
+        action="append",
+        help="Only report one finding kind. May be used more than once.",
+    )
+    parser.add_argument(
+