infra/ansible
23
7
Fork 53
Code Pull requests 11 Activity Actions

httpd/koji: set MaxKeepAliveRequests 0 to fix 502 race condition #3213

Merged
kevin merged 2 commits from victorkoycheff/ansible:issue-12913-koji-502 into main 2026-04-30 00:54:15 +00:00
Conversation 1 Commits 2 Files changed 3 +3 -2
victorkoycheff commented 2026-03-12 07:42:29 +00:00
Contributor
Copy link

This PR is the second (and final) half of the fix for the Koji 502 Bad Gateway errors reported in #12913.

What was happening:
My previous PR (#3173) added ttl=10 to the proxyopts. That successfully fixed the idle timeout race condition, but people were still occasionally hitting 502s under heavy load.

Turns out there was a second race condition at play: the request count limit. The backend Apache configurations (httpd.conf.j2, kojiweb.conf.j2) had MaxKeepAliveRequests 500 hardcoded. If a client polled aggressively enough (bypassing the 10-second idle TTL), the proxy would reuse the same connection until it hit exactly 500 requests. At that moment, the backend Apache server would ruthlessly close the TCP connection. If the proxy tried to hand off request #501 at that exact millisecond, we'd get a 502.

The Fix:
This PR sets MaxKeepAliveRequests 0 (unlimited) on the proxy and Koji hub/web templates.
By removing the request limit on the backend, we ensure the backend never hangs up on the proxy unexpectedly. The ttl=10 setting from the previous PR remains in place and now functions perfectly as the sole lifecycle manager for these connections, gracefully closing them only when they are genuinely idle.

Testing:
I verified this locally using a Python script built to spam the hello endpoint. Without this fix, the script reliably hits a 502 between 500-1500 requests.

Fixes #12913

This PR is the second (and final) half of the fix for the Koji 502 Bad Gateway errors reported in #12913. **What was happening:** My previous PR (#3173) added `ttl=10` to the proxyopts. That successfully fixed the *idle timeout* race condition, but people were still occasionally hitting 502s under heavy load. Turns out there was a second race condition at play: the **request count limit**. The backend Apache configurations (`httpd.conf.j2`, `kojiweb.conf.j2`) had `MaxKeepAliveRequests 500` hardcoded. If a client polled aggressively enough (bypassing the 10-second idle TTL), the proxy would reuse the same connection until it hit exactly 500 requests. At that moment, the backend Apache server would ruthlessly close the TCP connection. If the proxy tried to hand off request #501 at that exact millisecond, we'd get a 502. **The Fix:** This PR sets `MaxKeepAliveRequests 0` (unlimited) on the proxy and Koji hub/web templates. By removing the request limit on the backend, we ensure the backend never hangs up on the proxy unexpectedly. The `ttl=10` setting from the previous PR remains in place and now functions perfectly as the *sole* lifecycle manager for these connections, gracefully closing them only when they are genuinely idle. **Testing:** I verified this locally using a Python script built to spam the `hello` endpoint. Without this fix, the script reliably hits a 502 between 500-1500 requests. Fixes #12913
proxies-reverseproxy: set keepalive=on ttl=10 for koji
Some checks failed
Linter / yamllint (pull_request) Failing after 29s
Details
Linter / ansible-lint (pull_request) Failing after 30s
Details
AI Code Review / ai-review (pull_request_target) Successful in 29s
Details
947cfd0b2c
 
This fixes intermittent 502 Bad Gateway errors during long-running
koji connections (like watch-task or watch-logs).

For more details on proxy keepalive and ttl, see:
https://httpd.apache.org/docs/2.4/mod/mod_proxy.html

Fixes #12913

Signed-off-by: Victor Koycheff <victorkoycheff@gmail.com>
victorkoycheff force-pushed issue-12913-koji-502 from cd59b81cfc
Some checks failed
Linter / yamllint (pull_request) Failing after 26s
Details
Linter / ansible-lint (pull_request) Failing after 30s
Details
to c686b58853
All checks were successful
Linter / yamllint (pull_request) Successful in 26s
Details
Linter / ansible-lint (pull_request) Successful in 41s
Details
2026-03-12 20:40:44 +00:00 Compare
kevin commented 2026-04-30 00:53:53 +00:00
Owner
Copy link

So, I am not convinced that this will fix the underlying issue.

However, I am willing to give it a try now that we are out of freeze.

So, I am not convinced that this will fix the underlying issue. However, I am willing to give it a try now that we are out of freeze.
kevin merged commit e56db32aa6 into main 2026-04-30 00:54:15 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
ai-review-please
label for requesting AI code review

  
freeze-break-request

   
post-freeze
  
Backlog Status
Needs Review
 Work items that have been created or updated but need review, refinement, or approval before they can be considered ready for sprint planning.

  
Backlog Status
Ready
 Work items that have been reviewed, refined, and are ready to be committed to a sprint. These items have clear requirements and can be worked on immediately when capacity is available.

  
chore
Work that needs to be done but doesn't add new functionality or fix bugs. Maintenance tasks that keep the system running smoothly and the codebase clean.

  
documentation
Work focused on creating, updating, or improving documentation for users, developers, or system administrators.

  
points
01
 A task that is simple, small, and has very little uncertainty. It should be easy enough for any team member to pick up and complete quickly, without a lot of discussion or research.

  
points
02
 This is a task that is slightly more complex or larger than a "1." It might involve two or three simple steps, but the outcome is still very clear and there's little uncertainty.

  
points
03
 Medium-sized task with some noticeable complexity. It might require a bit more thought or collaboration. The steps are well-defined, but it's not a trivial change.

  
points
05
 A significant piece of work. It has more complexity and a higher degree of uncertainty than smaller tasks. It might require some research, a design discussion, or collaboration across multiple teams.

  
points
08
 Complex task with a lot of uncertainty. It's often the largest size that a team will consider for a single sprint. It likely requires a detailed breakdown and multiple team members to complete.

  
points
13
 Any task that feels like it's a "13" or more is considered an Epic. These are too large and uncertain to be worked on in a single sprint and should be broken down.

  
Priority
High
 This is for work that is critical and needs to be addressed in the next sprint. It's often for urgent bugs, important features, or a major piece of technical debt that is blocking other work.

  
Priority
Low
 This is for work that is good to do but not urgent. It's a candidate for being worked on when there's extra time or when it rises in importance. It can stay in the backlog for a longer period.

  
Priority
Medium
 This is for important work that should be done soon, but isn't as urgent as a high-priority item. It will likely be considered for the next 1-2 sprints.

  
Sprint Status
Blocked
 Work items that cannot progress due to external dependencies, blockers, or issues that need resolution.

  
Sprint Status
Done
 Work items that have been completed, reviewed, and integrated. No further work is needed within the current sprint.

  
Sprint Status
In Progress
 Work items currently being worked on by a team member. Active development, research, or implementation is happening.

  
Sprint Status
Review
 Work items that are complete but need review, testing, or approval before being marked as done.

  
Sprint Status
To Do
 Work items that are committed to the sprint but haven't been started yet. These items are ready to be picked up by team members.

  
Technical Debt
Work that addresses shortcuts, workarounds, or suboptimal implementations that were done previously to meet deadlines. These items improve code quality, maintainability, or performance without adding new features.

  
Work Item
Bug
 Issues where the system is not working as intended or designed. Bugs represent deviations from expected behavior.

  
Work Item
Epic
 Large initiatives that are broken down into multiple smaller work items (user stories, tasks, bugs). Epics represent significant features or projects that span multiple sprints.

  
Work Item
Spike
 Time-boxed research, investigation, or proof-of-concept work to reduce uncertainty or explore solutions. Spikes help inform future development decisions.

  
Work Item
Task
 Used to break down larger user stories into smaller, manageable implementation pieces. They focus on the specific technical steps needed to deliver the user story's value.

  
Work Item
User Story
 User-facing features or functionality that can be framed from a user perspective, describing what someone wants to accomplish and why.

No labels
ai-review-please
freeze-break-request
post-freeze
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
infra/ansible!3213
No description provided.