infra/ansible
23
7
Fork 53
Code Pull requests 11 Activity Actions

ELNBuildSync: support staging RabbitMQ #3400

Merged
james merged 4 commits from sgallagh/infra-ansible:elnbuildsync-deploy into main 2026-06-11 18:13:33 +00:00
Conversation 8 Commits 4 Files changed 5 +129 -122
sgallagh commented 2026-06-09 16:53:36 +00:00
Contributor
Copy link

Signed-off-by: Stephen Gallagher sgallagh@redhat.com

rh-pre-commit.version: 2.4.0
rh-pre-commit.check-secrets: ENABLED

Signed-off-by: Stephen Gallagher <sgallagh@redhat.com> rh-pre-commit.version: 2.4.0 rh-pre-commit.check-secrets: ENABLED
ELNBuildSync: support staging RabbitMQ
Some checks failed
Linter / yamllint (pull_request) Successful in 28s
Details
Linter / ansible-lint (pull_request) Failing after 59s
Details
AI Code Review / ai-review (pull_request_target) Successful in 46s
Details
076f51b02c
 
Signed-off-by: Stephen Gallagher <sgallagh@redhat.com>

rh-pre-commit.version: 2.4.0
rh-pre-commit.check-secrets: ENABLED
sgallagh requested review from james 2026-06-09 16:53:49 +00:00
james commented 2026-06-09 17:29:12 +00:00
Member
Copy link

This looks fine to me, probably worth someone with more openshift knowledge having a look too.

This looks fine to me, probably worth someone with more openshift knowledge having a look too.
forgejo-actions commented 2026-06-09 19:17:56 +00:00
Copy link

AI Code Review

Detailed Code Review

The PR successfully parameterizes the RabbitMQ configuration to support a staging environment by introducing environment-specific suffixes. However, there are critical logic errors in the extracted fedora-messaging-config.toml that will cause the staging deployment to fail or misbehave. Specifically, the AMQP connection string is missing the environment suffix for the username, and the consumer bindings still hardcode the prod environment for routing keys, meaning the staging consumer will attempt to listen to production messages instead of staging messages.

Additionally, there is an architectural concern with how the TOML file is injected into the ConfigMap. Standard file loading functions read file contents as raw strings without evaluating embedded Jinja tags (like {{ env_suffix }}). This needs to be parsed as a template to ensure the variables are populated correctly before being embedded into the ConfigMap YAML.

📂 File Reviews

📄 `roles/openshift-apps/elnbuildsync/templates/fedora-messaging-config.toml` - Extracted RabbitMQ configuration to a separate TOML file with Jinja templating.
  • Critical [Bug]: The AMQP URL username does not match the provisioned user. The playbook creates the user elnbuildsync{{ env_suffix }}, but the URL hardcodes elnbuildsync.
    • Suggestion: Include the {{ env_suffix }} variable in the username section of the AMQP URL to ensure authentication succeeds in staging.
amqp_url = "amqps://elnbuildsync{{ env_suffix }}:@rabbitmq{{ env_suffix }}.fedoraproject.org/%2Fpubsub"
  • Critical [Bug]: The consumer routing keys have hardcoded the .prod. environment scope, completely breaking staging functionality as the staging app will bind to production messages.
    • Suggestion: Parameterize the routing keys to use {{ env_short }} (or standard wildcard # as defined in the playbook queue setup).
routing_keys = [
    "org.fedoraproject.{{ env_short }}.buildsys.repo.init",
    "org.fedoraproject.{{ env_short }}.buildsys.repo.done",
    "org.fedoraproject.{{ env_short }}.buildsys.tag",
    "org.fedoraproject.{{ env_short }}.buildsys.task.state.change"
    ]
📄 `roles/openshift-apps/elnbuildsync/templates/fedora-messaging-config.yml.j2` - Updated ConfigMap template to load the TOML configuration file and environment-aware certificates.
  • Major [Architecture]: Using load_file to embed a file containing Jinja variables ({{ env_suffix }}) will likely embed the raw strings instead of evaluating them, as standard file lookups do not recursively process templates.
    • Suggestion: Use Ansible's standard lookup('template', ...) functionality to ensure the TOML file's variables are fully evaluated before embedding.
  config.toml: |-
    {{ lookup('template', 'fedora-messaging-config.toml') | indent(4) }}

Questions:

  • ❓ Does the load_file function in your environment natively perform recursive Jinja rendering? If not, the variables inside the TOML won't be replaced.

✅ Summary

  • Overall Assessment: Several critical bugs were found regarding environment suffix application in the AMQP URL, hardcoded production routing keys, and potential template-rendering issues.
  • Priority Issues:
    • Fix the hardcoded .prod. in the TOML routing keys to ensure the staging consumer listens to staging events.
    • Fix the missing {{ env_suffix }} for the username in the amqp_url configuration string.
    • Verify that the TOML file injected into the ConfigMap correctly evaluates the nested Jinja variables; replace load_file with lookup('template') if needed.
  • Minor Suggestions:
    • Align the routing_keys bindings defined in fedora-messaging-config.toml strictly with the queue_routing_keys defined in the playbook to prevent desyncs.

🤖 AI Code Review | Generated with ai-code-review | Model: gemini-3.1-pro-preview

⚠️ AI-generated suggestions may be incorrect. Verify before applying. Not a replacement for human review.

## AI Code Review ### Detailed Code Review The PR successfully parameterizes the RabbitMQ configuration to support a staging environment by introducing environment-specific suffixes. However, there are critical logic errors in the extracted `fedora-messaging-config.toml` that will cause the staging deployment to fail or misbehave. Specifically, the AMQP connection string is missing the environment suffix for the username, and the consumer bindings still hardcode the `prod` environment for routing keys, meaning the staging consumer will attempt to listen to production messages instead of staging messages. Additionally, there is an architectural concern with how the TOML file is injected into the ConfigMap. Standard file loading functions read file contents as raw strings without evaluating embedded Jinja tags (like `{{ env_suffix }}`). This needs to be parsed as a template to ensure the variables are populated correctly before being embedded into the ConfigMap YAML. #### 📂 File Reviews <details> <summary><strong>📄 `roles/openshift-apps/elnbuildsync/templates/fedora-messaging-config.toml`</strong> - Extracted RabbitMQ configuration to a separate TOML file with Jinja templating.</summary> - **Critical** [Bug]: The AMQP URL username does not match the provisioned user. The playbook creates the user `elnbuildsync{{ env_suffix }}`, but the URL hardcodes `elnbuildsync`. - *Suggestion:* Include the `{{ env_suffix }}` variable in the username section of the AMQP URL to ensure authentication succeeds in staging. ```` amqp_url = "amqps://elnbuildsync{{ env_suffix }}:@rabbitmq{{ env_suffix }}.fedoraproject.org/%2Fpubsub" ```` - **Critical** [Bug]: The consumer routing keys have hardcoded the `.prod.` environment scope, completely breaking staging functionality as the staging app will bind to production messages. - *Suggestion:* Parameterize the routing keys to use `{{ env_short }}` (or standard wildcard `#` as defined in the playbook queue setup). ```` routing_keys = [ "org.fedoraproject.{{ env_short }}.buildsys.repo.init", "org.fedoraproject.{{ env_short }}.buildsys.repo.done", "org.fedoraproject.{{ env_short }}.buildsys.tag", "org.fedoraproject.{{ env_short }}.buildsys.task.state.change" ] ```` </details> <details> <summary><strong>📄 `roles/openshift-apps/elnbuildsync/templates/fedora-messaging-config.yml.j2`</strong> - Updated ConfigMap template to load the TOML configuration file and environment-aware certificates.</summary> - **Major** [Architecture]: Using `load_file` to embed a file containing Jinja variables (`{{ env_suffix }}`) will likely embed the raw strings instead of evaluating them, as standard file lookups do not recursively process templates. - *Suggestion:* Use Ansible's standard `lookup('template', ...)` functionality to ensure the TOML file's variables are fully evaluated before embedding. ```` config.toml: |- {{ lookup('template', 'fedora-messaging-config.toml') | indent(4) }} ```` **Questions:** - ❓ Does the `load_file` function in your environment natively perform recursive Jinja rendering? If not, the variables inside the TOML won't be replaced. </details> ### ✅ Summary - **Overall Assessment:** Several critical bugs were found regarding environment suffix application in the AMQP URL, hardcoded production routing keys, and potential template-rendering issues. - **Priority Issues:** - Fix the hardcoded `.prod.` in the TOML routing keys to ensure the staging consumer listens to staging events. - Fix the missing `{{ env_suffix }}` for the username in the `amqp_url` configuration string. - Verify that the TOML file injected into the ConfigMap correctly evaluates the nested Jinja variables; replace `load_file` with `lookup('template')` if needed. - **Minor Suggestions:** - Align the `routing_keys` bindings defined in `fedora-messaging-config.toml` strictly with the `queue_routing_keys` defined in the playbook to prevent desyncs. --- 🤖 **AI Code Review** | Generated with [ai-code-review](https://gitlab.com/redhat/edge/ci-cd/ai-code-review) | **Model:** `gemini-3.1-pro-preview` ⚠️ *AI-generated suggestions may be incorrect. Verify before applying. Not a replacement for human review.*
sgallagh commented 2026-06-09 19:21:10 +00:00
Author
Contributor
Copy link

I'm not sure why ansible-lint is complaining. https://forge.fedoraproject.org/infra/ansible/src/branch/main/roles/rabbit/user/tasks/main.yml exists...

I'm not sure why ansible-lint is complaining. https://forge.fedoraproject.org/infra/ansible/src/branch/main/roles/rabbit/user/tasks/main.yml exists...
kevin requested changes 2026-06-09 23:11:44 +00:00
kevin left a comment
Copy link

One minor comment...

One minor comment...
roles/openshift-apps/elnbuildsync/templates/fedora-messaging-config.yml.j2
@ -6,3 +6,3 @@
config.toml: |-
{{ ebs_fedora_messaging_config | indent(4) }}
{{ load_file('fedora-messaging-config.toml') | indent(4) }}
cacert.pem: |-
kevin commented 2026-06-09 23:09:20 +00:00
Owner
Copy link

Instead of inlining these... set them as secrets, ie, look at for example:
playbooks/openshift-apps/fedocal.yml

and the openshift/secret-file role calls there.
Then they are secrets, which I think might be better than adding them here?

Instead of inlining these... set them as secrets, ie, look at for example: playbooks/openshift-apps/fedocal.yml and the openshift/secret-file role calls there. Then they are secrets, which I think might be better than adding them here?
sgallagh commented 2026-06-10 12:40:36 +00:00
Author
Contributor
Copy link

I'm not entirely sure what you're saying here. This entire YAML is defining a Kubernetes Secret object. It's doing what the openshift/secret-file role does under the hood, except it's adding all of the certificates to a single Secret (and thus directory) rather than individual directories.

I can switch to using openshift/secret-file, but it seems less efficient. And if you want me to give ebs-config.yml.j2 the same treatment, that will actually require code changes to EBS, because they expect them to both to be in a default location /etc/elnbuildsync.

I'm not entirely sure what you're saying here. This entire YAML is defining a Kubernetes Secret object. It's doing what the `openshift/secret-file` role does under the hood, except it's adding all of the certificates to a single Secret (and thus directory) rather than individual directories. I *can* switch to using `openshift/secret-file`, but it seems less efficient. And if you want me to give `ebs-config.yml.j2` the same treatment, that will actually require code changes to EBS, because they expect them to both to be in a default location `/etc/elnbuildsync`.
sgallagh marked this conversation as resolved
kevin commented 2026-06-10 16:31:48 +00:00
Owner
Copy link

Ahhh. I didn't look at enough context there. :( Sorry...

no, this approch seems fine to me. +1

Ahhh. I didn't look at enough context there. :( Sorry... no, this approch seems fine to me. +1
👍 1
sgallagh force-pushed elnbuildsync-deploy from 076f51b02c
Some checks failed
Linter / yamllint (pull_request) Successful in 28s
Details
Linter / ansible-lint (pull_request) Failing after 59s
Details
AI Code Review / ai-review (pull_request_target) Successful in 46s
Details
to d2a3de5315
Some checks failed
Linter / yamllint (pull_request) Successful in 27s
Details
Linter / ansible-lint (pull_request) Failing after 1m8s
Details
2026-06-10 19:04:56 +00:00 Compare
ELNBuildSync: Use different config branch on staging
Some checks failed
Linter / yamllint (pull_request) Successful in 28s
Details
Linter / ansible-lint (pull_request) Failing after 1m4s
Details
9462629749
 
Also fix missing variable reference for the main role deployment
playbook.

Signed-off-by: Stephen Gallagher <sgallagh@redhat.com>

rh-pre-commit.version: 2.4.0
rh-pre-commit.check-secrets: ENABLED
sgallagh commented 2026-06-10 20:56:21 +00:00
Author
Contributor
Copy link

I think I've fixed the AI-detected issues and I've also enhanced the Deployment to use a different config file depending on whether the environment is production or staging.

I think I've fixed the AI-detected issues and I've also enhanced the Deployment to use a different config file depending on whether the environment is production or staging.
james merged commit 92d93d29f0 into main 2026-06-11 18:13:33 +00:00
abompard commented 2026-06-19 07:43:31 +00:00
Member
Copy link

Hey @sgallagh , it looks like the distrobuildsync-eln user in RabbitMQ is not unused (and deprecated in favor or elnbuildsync), is it correct? Can I remove it from RabbitMQ?

Hey @sgallagh , it looks like the `distrobuildsync-eln` user in RabbitMQ is not unused (and deprecated in favor or `elnbuildsync`), is it correct? Can I remove it from RabbitMQ?
sgallagh commented 2026-06-30 12:50:34 +00:00
Author
Contributor
Copy link

That's probably correct, but if possible can we not change anything until I have ELNBuildSync up and running on Fedora Infrastructure, just in case I'm mistaken?

The new user elnbuildsync is definitely correct for the new deployment, but I'm not 100% sure right now that the distrobuildsync-eln user isn't in use by the old/current deployment.

That's probably correct, but if possible can we not change anything until I have ELNBuildSync up and running on Fedora Infrastructure, just in case I'm mistaken? The new user `elnbuildsync` is definitely correct for the new deployment, but I'm not 100% sure right now that the `distrobuildsync-eln` user isn't in use by the old/current deployment.
Sign in to join this conversation.
Reviewers
No reviewers
james
kevin
Labels
Clear labels   
ai-review-please
label for requesting AI code review

  
freeze-break-request

   
post-freeze
  
Backlog Status
Needs Review
 Work items that have been created or updated but need review, refinement, or approval before they can be considered ready for sprint planning.

  
Backlog Status
Ready
 Work items that have been reviewed, refined, and are ready to be committed to a sprint. These items have clear requirements and can be worked on immediately when capacity is available.

  
chore
Work that needs to be done but doesn't add new functionality or fix bugs. Maintenance tasks that keep the system running smoothly and the codebase clean.

  
documentation
Work focused on creating, updating, or improving documentation for users, developers, or system administrators.

  
points
01
 A task that is simple, small, and has very little uncertainty. It should be easy enough for any team member to pick up and complete quickly, without a lot of discussion or research.

  
points
02
 This is a task that is slightly more complex or larger than a "1." It might involve two or three simple steps, but the outcome is still very clear and there's little uncertainty.

  
points
03
 Medium-sized task with some noticeable complexity. It might require a bit more thought or collaboration. The steps are well-defined, but it's not a trivial change.

  
points
05
 A significant piece of work. It has more complexity and a higher degree of uncertainty than smaller tasks. It might require some research, a design discussion, or collaboration across multiple teams.

  
points
08
 Complex task with a lot of uncertainty. It's often the largest size that a team will consider for a single sprint. It likely requires a detailed breakdown and multiple team members to complete.

  
points
13
 Any task that feels like it's a "13" or more is considered an Epic. These are too large and uncertain to be worked on in a single sprint and should be broken down.

  
Priority
High
 This is for work that is critical and needs to be addressed in the next sprint. It's often for urgent bugs, important features, or a major piece of technical debt that is blocking other work.

  
Priority
Low
 This is for work that is good to do but not urgent. It's a candidate for being worked on when there's extra time or when it rises in importance. It can stay in the backlog for a longer period.

  
Priority
Medium
 This is for important work that should be done soon, but isn't as urgent as a high-priority item. It will likely be considered for the next 1-2 sprints.

  
Sprint Status
Blocked
 Work items that cannot progress due to external dependencies, blockers, or issues that need resolution.

  
Sprint Status
Done
 Work items that have been completed, reviewed, and integrated. No further work is needed within the current sprint.

  
Sprint Status
In Progress
 Work items currently being worked on by a team member. Active development, research, or implementation is happening.

  
Sprint Status
Review
 Work items that are complete but need review, testing, or approval before being marked as done.

  
Sprint Status
To Do
 Work items that are committed to the sprint but haven't been started yet. These items are ready to be picked up by team members.

  
Technical Debt
Work that addresses shortcuts, workarounds, or suboptimal implementations that were done previously to meet deadlines. These items improve code quality, maintainability, or performance without adding new features.

  
Work Item
Bug
 Issues where the system is not working as intended or designed. Bugs represent deviations from expected behavior.

  
Work Item
Epic
 Large initiatives that are broken down into multiple smaller work items (user stories, tasks, bugs). Epics represent significant features or projects that span multiple sprints.

  
Work Item
Spike
 Time-boxed research, investigation, or proof-of-concept work to reduce uncertainty or explore solutions. Spikes help inform future development decisions.

  
Work Item
Task
 Used to break down larger user stories into smaller, manageable implementation pieces. They focus on the specific technical steps needed to deliver the user story's value.

  
Work Item
User Story
 User-facing features or functionality that can be framed from a user perspective, describing what someone wants to accomplish and why.

No labels
ai-review-please
freeze-break-request
post-freeze
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
5 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
infra/ansible!3400
No description provided.