infra/ansible
23
2
Fork 44
Code Pull requests 13 Activity Actions

proxies-reverseproxy: set keepalive=on ttl=10 for koji #3173

Merged
kevin merged 1 commit from victorkoycheff/ansible:issue-12913-koji-502 into main 2026-03-11 20:36:50 +00:00
Conversation 4 Commits 1 Files changed 1 +3 -1
victorkoycheff commented 2026-02-25 17:25:41 +00:00
Contributor
Copy link

This PR addresses the intermittent 502 Bad Gateway errors users experience during long-running Koji connections (e.g., watch-task, watch-logs, or chainbuild).

The Problem:
Currently, the Apache proxy has a KeepAliveTimeout of 15s and the Koji backend has a KeepAliveTimeout of 16s. However, because the two layers of keep-alive are untethered, a race condition occurs. mod_proxy occasionally attempts to reuse an idle connection from its backend pool at the exact millisecond the Koji backend is closing it, resulting in a 502.

The Fix:
By adding keepalive=on ttl=10 to the proxyopts for the Koji proxies, we force Apache to proactively expire and drop idle backend connections after 10 seconds. This guarantees the proxy will never attempt to reuse a connection that is approaching the backend's 16-second threshold, completely eliminating the race condition.

Reference: Apache mod_proxy documentation

Fixes infra/tickets#12913

This PR addresses the intermittent 502 Bad Gateway errors users experience during long-running Koji connections (e.g., `watch-task`, `watch-logs`, or `chainbuild`). **The Problem:** Currently, the Apache proxy has a `KeepAliveTimeout` of 15s and the Koji backend has a `KeepAliveTimeout` of 16s. However, because the two layers of keep-alive are untethered, a race condition occurs. `mod_proxy` occasionally attempts to reuse an idle connection from its backend pool at the exact millisecond the Koji backend is closing it, resulting in a 502. **The Fix:** By adding `keepalive=on ttl=10` to the `proxyopts` for the Koji proxies, we force Apache to proactively expire and drop idle backend connections after 10 seconds. This guarantees the proxy will never attempt to reuse a connection that is approaching the backend's 16-second threshold, completely eliminating the race condition. Reference: [Apache mod_proxy documentation](https://httpd.apache.org/docs/2.4/mod/mod_proxy.html) Fixes https://forge.fedoraproject.org/infra/tickets/issues/12913
kevin commented 2026-02-25 22:55:57 +00:00
Owner
Copy link

I'm not fully convinced that this is the place where the keepalive is mismatched. It's a pretty complex path:

request -> proxy httpd -> anubis -> proxy httpd -> koji httpd

But it's possible at least. :)

Since we are in beta freeze this will need to wait until we are unfrozen, but we could try it then.

Or if we can duplicate it in staging we could try there.

I'm not fully convinced that this is the place where the keepalive is mismatched. It's a pretty complex path: request -> proxy httpd -> anubis -> proxy httpd -> koji httpd But it's possible at least. :) Since we are in beta freeze this will need to wait until we are unfrozen, but we could try it then. Or if we can duplicate it in staging we could try there.
victorkoycheff force-pushed issue-12913-koji-502 from d1912a75f4
Some checks failed
Linter / yamllint (pull_request) Failing after 23s
Details
Linter / ansible-lint (pull_request) Failing after 26s
Details
to 947cfd0b2c
Some checks failed
Linter / yamllint (pull_request) Failing after 29s
Details
Linter / ansible-lint (pull_request) Failing after 30s
Details
AI Code Review / ai-review (pull_request_target) Successful in 29s
Details
2026-02-26 07:46:53 +00:00 Compare
victorkoycheff commented 2026-02-26 07:51:05 +00:00
Author
Contributor
Copy link

Yeah, it's definitely a pretty complex path. :)

My thinking was that the race condition happens right at that very last hop to the koji backend, so dropping the connection at 10s there might just do the trick... Empirically, I've seen this exact tweak clear up similar 502s where the backend and proxy timeouts were fighting each other.

I also just force-pushed a fix for a silly yaml syntax error that was failing the linter checks.

We can definitely just leave this until the freeze is lifted and try to duplicate it in staging then.

Yeah, it's definitely a pretty complex path. :) My thinking was that the race condition happens right at that very last hop to the koji backend, so dropping the connection at 10s there might just do the trick... Empirically, I've seen this exact tweak clear up similar 502s where the backend and proxy timeouts were fighting each other. I also just force-pushed a fix for a silly yaml syntax error that was failing the linter checks. We can definitely just leave this until the freeze is lifted and try to duplicate it in staging then.
forgejo-actions commented 2026-03-11 19:13:16 +00:00
Copy link

AI Code Review

📋 MR Summary

Adds proxy keepalive and TTL options to Koji reverse proxy configurations to resolve intermittent 502 Bad Gateway errors.

  • Key Changes:
    • Added proxyopts: "keepalive=on ttl=10" to the Koji production balancer configuration.
    • Added proxyopts: "keepalive=on ttl=10" to the Koji staging balancer configuration.
    • Fixed Ansible syntax indentation for the varnish_url variable.
  • Impact: proxies-reverseproxy.yml
  • Risk Level: 🟢 Low - The changes are straightforward Apache mod_proxy configuration updates with standard, documented values that prevent known race conditions without introducing new logical risks.

Detailed Code Review

The implementation aligns perfectly with the intent described in the PR and previous discussions. Setting keepalive=on ttl=10 is the standard and correct approach to prevent mod_proxy from reusing connections that the backend is about to close. The indentation fix for varnish_url is a nice minor correction.

📂 File Reviews

📄 `playbooks/include/proxies-reverseproxy.yml` - Updated proxy configuration to include keepalive options for Koji and fixed syntax indentation for variables.
  • Suggestion [Style]: The change from - varnish_url to varnish_url corrects the Ansible variable dictionary structure, which is good practice.

✅ Summary

  • Overall Assessment: No critical issues identified. The fix effectively addresses the reported 502 errors.

🤖 AI Code Review | Generated with ai-code-review | Model: gemini-3.1-pro-preview

⚠️ AI-generated suggestions may be incorrect. Verify before applying. Not a replacement for human review.

## AI Code Review ### 📋 MR Summary Adds proxy keepalive and TTL options to Koji reverse proxy configurations to resolve intermittent 502 Bad Gateway errors. - **Key Changes:** - Added `proxyopts: "keepalive=on ttl=10"` to the Koji production balancer configuration. - Added `proxyopts: "keepalive=on ttl=10"` to the Koji staging balancer configuration. - Fixed Ansible syntax indentation for the `varnish_url` variable. - **Impact:** proxies-reverseproxy.yml - **Risk Level:** 🟢 Low - The changes are straightforward Apache mod_proxy configuration updates with standard, documented values that prevent known race conditions without introducing new logical risks. ### Detailed Code Review The implementation aligns perfectly with the intent described in the PR and previous discussions. Setting `keepalive=on ttl=10` is the standard and correct approach to prevent `mod_proxy` from reusing connections that the backend is about to close. The indentation fix for `varnish_url` is a nice minor correction. #### 📂 File Reviews <details> <summary><strong>📄 `playbooks/include/proxies-reverseproxy.yml`</strong> - Updated proxy configuration to include keepalive options for Koji and fixed syntax indentation for variables.</summary> - **Suggestion** [Style]: The change from `- varnish_url` to `varnish_url` corrects the Ansible variable dictionary structure, which is good practice. </details> ### ✅ Summary - **Overall Assessment:** No critical issues identified. The fix effectively addresses the reported 502 errors. --- 🤖 **AI Code Review** | Generated with [ai-code-review](https://gitlab.com/redhat/edge/ci-cd/ai-code-review) | **Model:** `gemini-3.1-pro-preview` ⚠️ *AI-generated suggestions may be incorrect. Verify before applying. Not a replacement for human review.*
kevin commented 2026-03-11 20:36:43 +00:00
Owner
Copy link

ok, lets try (staging first)...

ok, lets try (staging first)...
kevin merged commit 35a1b3223b into main 2026-03-11 20:36:50 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
ai-review-please

label for requesting AI code review

  
freeze-break-request

   
post-freeze
  
Backlog Status
Needs Review
Work items that have been created or updated but need review, refinement, or approval before they can be considered ready for sprint planning.

  
Backlog Status
Ready
Work items that have been reviewed, refined, and are ready to be committed to a sprint. These items have clear requirements and can be worked on immediately when capacity is available.

  
chore

Work that needs to be done but doesn't add new functionality or fix bugs. Maintenance tasks that keep the system running smoothly and the codebase clean.

  
documentation

Work focused on creating, updating, or improving documentation for users, developers, or system administrators.

  
points
01
A task that is simple, small, and has very little uncertainty. It should be easy enough for any team member to pick up and complete quickly, without a lot of discussion or research.

  
points
02
This is a task that is slightly more complex or larger than a "1." It might involve two or three simple steps, but the outcome is still very clear and there's little uncertainty.

  
points
03
Medium-sized task with some noticeable complexity. It might require a bit more thought or collaboration. The steps are well-defined, but it's not a trivial change.

  
points
05
A significant piece of work. It has more complexity and a higher degree of uncertainty than smaller tasks. It might require some research, a design discussion, or collaboration across multiple teams.

  
points
08
Complex task with a lot of uncertainty. It's often the largest size that a team will consider for a single sprint. It likely requires a detailed breakdown and multiple team members to complete.

  
points
13
Any task that feels like it's a "13" or more is considered an Epic. These are too large and uncertain to be worked on in a single sprint and should be broken down.

  
Priority
High
This is for work that is critical and needs to be addressed in the next sprint. It's often for urgent bugs, important features, or a major piece of technical debt that is blocking other work.

  
Priority
Low
This is for work that is good to do but not urgent. It's a candidate for being worked on when there's extra time or when it rises in importance. It can stay in the backlog for a longer period.

  
Priority
Medium
This is for important work that should be done soon, but isn't as urgent as a high-priority item. It will likely be considered for the next 1-2 sprints.

  
Sprint Status
Blocked
Work items that cannot progress due to external dependencies, blockers, or issues that need resolution.

  
Sprint Status
Done
Work items that have been completed, reviewed, and integrated. No further work is needed within the current sprint.

  
Sprint Status
In Progress
Work items currently being worked on by a team member. Active development, research, or implementation is happening.

  
Sprint Status
Review
Work items that are complete but need review, testing, or approval before being marked as done.

  
Sprint Status
To Do
Work items that are committed to the sprint but haven't been started yet. These items are ready to be picked up by team members.

  
Technical Debt

Work that addresses shortcuts, workarounds, or suboptimal implementations that were done previously to meet deadlines. These items improve code quality, maintainability, or performance without adding new features.

  
Work Item
Bug
Issues where the system is not working as intended or designed. Bugs represent deviations from expected behavior.

  
Work Item
Epic
Large initiatives that are broken down into multiple smaller work items (user stories, tasks, bugs). Epics represent significant features or projects that span multiple sprints.

  
Work Item
Spike
Time-boxed research, investigation, or proof-of-concept work to reduce uncertainty or explore solutions. Spikes help inform future development decisions.

  
Work Item
Task
Used to break down larger user stories into smaller, manageable implementation pieces. They focus on the specific technical steps needed to deliver the user story's value.

  
Work Item
User Story
User-facing features or functionality that can be framed from a user perspective, describing what someone wants to accomplish and why.

No labels
ai-review-please
freeze-break-request
post-freeze
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
infra/ansible!3173
No description provided.