502 error from long running koji connections (watch-task, chainbuild, watch-logs)

kevin commented

2025-11-13 18:51:41 +00:00

Owner

Since the firewall cluster upgade eariler this week, users have been reporting koji cli giving a:

502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub

This is easily duplicated by picking a large/long running build and running a watch-logs on it.
The error doesn't actually affect the underlying tasks, it just errors on the client end, and the user can re-run their watch to resume (but then it will fail again)

The traceback from this looks like this here (rawhide koji):

2025-11-11 20:03:54,480 [DEBUG] koji: Opening new requests session
Traceback (most recent call last):
  File "/usr/bin/koji", line 331, in <module>
    rv = locals()[command].__call__(options, session, args)
  File "/usr/lib/python3.14/site-packages/koji_cli/commands.py", line 6559, in anon_handle_watch_task
    return watch_tasks(session, tasks, quiet=options.quiet,
                       poll_interval=goptions.poll_interval, topurl=goptions.topurl)
  File "/usr/lib/python3.14/site-packages/koji_cli/lib.py", line 361, in watch_tasks
    for child in session.getTaskChildren(task_id):
                 ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/usr/lib/python3.14/site-packages/koji/__init__.py", line 2539, in __call__
    return self.__func(self.__name, args, opts)
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.14/site-packages/koji/__init__.py", line 3123, in _renew_expired_session
    return func(self, *args, **kwargs)
  File "/usr/lib/python3.14/site-packages/koji/__init__.py", line 3149, in _callMethod
    return self._sendCall(handler, headers, request)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.14/site-packages/koji/__init__.py", line 3042, in _sendCall
    raise e
  File "/usr/lib/python3.14/site-packages/koji/__init__.py", line 3038, in _sendCall
    return self._sendOneCall(handler, headers, request)
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.14/site-packages/koji/__init__.py", line 3086, in _sendOneCall
    r.raise_for_status()
    ~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.14/site-packages/requests/models.py", line 1026, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub

So, seems likely something in the new firewall os is somehow being more aggressive in closing connections.
It's worth noting that I see the 502 in the proxy access logs, but there's no error in error log or anything in koji hub log

@mikem have any thoughts on this? would koji be trying to reuse a closed connection here? I can look at adding some kind of keepalives, but we didn't need them before and I would prefer to reason out what to add before we try too much.

Since the firewall cluster upgade eariler this week, users have been reporting koji cli giving a: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub This is easily duplicated by picking a large/long running build and running a watch-logs on it. The error doesn't actually affect the underlying tasks, it just errors on the client end, and the user can re-run their watch to resume (but then it will fail again) The traceback from this looks like this here (rawhide koji): ``` 2025-11-11 20:03:54,480 [DEBUG] koji: Opening new requests session Traceback (most recent call last): File "/usr/bin/koji", line 331, in <module> rv = locals()[command].__call__(options, session, args) File "/usr/lib/python3.14/site-packages/koji_cli/commands.py", line 6559, in anon_handle_watch_task return watch_tasks(session, tasks, quiet=options.quiet, poll_interval=goptions.poll_interval, topurl=goptions.topurl) File "/usr/lib/python3.14/site-packages/koji_cli/lib.py", line 361, in watch_tasks for child in session.getTaskChildren(task_id): ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/usr/lib/python3.14/site-packages/koji/__init__.py", line 2539, in __call__ return self.__func(self.__name, args, opts) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/koji/__init__.py", line 3123, in _renew_expired_session return func(self, *args, **kwargs) File "/usr/lib/python3.14/site-packages/koji/__init__.py", line 3149, in _callMethod return self._sendCall(handler, headers, request) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/koji/__init__.py", line 3042, in _sendCall raise e File "/usr/lib/python3.14/site-packages/koji/__init__.py", line 3038, in _sendCall return self._sendOneCall(handler, headers, request) ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.14/site-packages/koji/__init__.py", line 3086, in _sendOneCall r.raise_for_status() ~~~~~~~~~~~~~~~~~~^^ File "/usr/lib/python3.14/site-packages/requests/models.py", line 1026, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub ``` related/other mentions: https://pagure.io/fedora-infrastructure/issue/12814#comment-993194 https://pagure.io/releng/issue/13087 So, seems likely something in the new firewall os is somehow being more aggressive in closing connections. It's worth noting that I see the 502 in the proxy access logs, but there's no error in error log or anything in koji hub log @mikem have any thoughts on this? would koji be trying to reuse a closed connection here? I can look at adding some kind of keepalives, but we didn't need them before and I would prefer to reason out what to add before we try too much.

kevin commented

2025-11-13 18:54:17 +00:00

Author

Owner

Oops. The releng ticket is not related. It's an unrelated outage this morning.

mikem commented

2025-11-18 14:54:00 +00:00

would koji be trying to reuse a closed connection here? I can look at adding some kind of keepalives

Koji uses a python-requests session for all hub connections, which automatically uses keep-alive. Also keep-alive is the default in the http standard since http/1.1. Koji should automatically detect when a connection gets closed and make a new one. However, the symptom here is not a connection closing, but an active connection returning a 502.

What is probably happening is that the http connection from the proxy to the backend hub is timing out before the the http connection from the client to the proxy. What are the keep-alive timeout settings on each? You probably want to make sure the proxy keep-alive timeout is shorter than the backend.

> would koji be trying to reuse a closed connection here? I can look at adding some kind of keepalives Koji uses a python-requests session for all hub connections, which automatically uses keep-alive. Also keep-alive is the default in the http standard since http/1.1. Koji should automatically detect when a connection gets closed and make a new one. However, the symptom here is not a connection closing, but an active connection returning a 502. What is probably happening is that the http connection from the proxy to the backend hub is timing out before the the http connection from the client to the proxy. What are the keep-alive timeout settings on each? You probably want to make sure the proxy keep-alive timeout is shorter than the backend.

mikem commented

2025-11-18 14:57:02 +00:00

Fwiw, koji clients can probably work around this by setting anon_retry=true in their koji config, but I think the real fix should be in the proxy setup.

Fwiw, koji clients can probably work around this by setting `anon_retry=true` in their koji config, but I think the real fix should be in the proxy setup.

kevin commented

2025-11-19 22:55:50 +00:00

Author

Owner

I set the hub to 16s, and the proxies are 15s... but something is still not working right. I was still able to get a 502.

mikem commented

2025-11-20 00:21:27 +00:00

Well, worth a shot at least.
I still haven't replicated it on my end

Well, worth a shot at least. I still haven't replicated it on my end

mikem commented

2025-11-20 15:00:44 +00:00

I did get it to happen once here, but takes a while

kevin commented

2025-11-22 21:22:13 +00:00

Author

Owner

ok, we had a mass update/reboot cycle and since then I cannot get it to happen here.

I have been watching a webkitgtk build for 2+ hours now and no problems.

Can you all see if you can duplicate it now?

ok, we had a mass update/reboot cycle and since then I cannot get it to happen here. I have been watching a webkitgtk build for 2+ hours now and no problems. Can you all see if you can duplicate it now?

mikem commented

2025-11-24 16:05:27 +00:00

I can still get it to happen if I let something poll the hub of several hours. Most recently,

2025-11-24 01:51:42,545 [ERROR] koji: HTTPError: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub

I can still get it to happen if I let something poll the hub of several hours. Most recently, ``` 2025-11-24 01:51:42,545 [ERROR] koji: HTTPError: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub ```

adamwill commented

2025-11-25 00:47:52 +00:00

Member

Saw it multiple times this morning, including within seconds of doing a fedpkg build.

[adamw@toolbx fedora-toolbox-43 os-autoinst (f42 %)]$ fedpkg build
Building os-autoinst-5^20250707gitd55ec72-6.fc42 for f42-candidate
Created task: 139261749
Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=139261749
Watching tasks (this may be safely interrupted)...
139261749 build (f42-candidate, /rpms/os-autoinst.git:125dca21d6bed9f2bc17b194b99b3f4e23f7771a): free
139261749 build (f42-candidate, /rpms/os-autoinst.git:125dca21d6bed9f2bc17b194b99b3f4e23f7771a): free -> open (buildvm-a64-25.rdu3.fedoraproject.org)
Could not execute build: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub
[adamw@toolbx fedora-toolbox-43 os-autoinst (f42 %)]$ koji watch-task 139261749
Watching tasks (this may be safely interrupted)...
139261749 build (f42-candidate, /rpms/os-autoinst.git:125dca21d6bed9f2bc17b194b99b3f4e23f7771a): open (buildvm-a64-25.rdu3.fedoraproject.org)
  139261760 buildSRPMFromSCM (/rpms/os-autoinst.git:125dca21d6bed9f2bc17b194b99b3f4e23f7771a): open (buildvm-a64-05.rdu3.fedoraproject.org)
2025-11-24 11:59:46,060 [ERROR] koji: HTTPError: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub
[adamw@toolbx fedora-toolbox-43 os-autoinst (f42 %)]$ koji watch-task 139261749
Watching tasks (this may be safely interrupted)...
139261749 build (f42-candidate, /rpms/os-autoinst.git:125dca21d6bed9f2bc17b194b99b3f4e23f7771a): closed
  139261981 tagBuild (noarch): closed
  139261803 buildArch (os-autoinst-5^20250707gitd55ec72-6.fc42.src.rpm, ppc64le): closed
  139261804 buildArch (os-autoinst-5^20250707gitd55ec72-6.fc42.src.rpm, s390x): closed
  139261801 buildArch (os-autoinst-5^20250707gitd55ec72-6.fc42.src.rpm, x86_64): closed
  139261802 buildArch (os-autoinst-5^20250707gitd55ec72-6.fc42.src.rpm, aarch64): closed
  139261760 buildSRPMFromSCM (/rpms/os-autoinst.git:125dca21d6bed9f2bc17b194b99b3f4e23f7771a): closed
2025-11-24 12:12:30,826 [ERROR] koji: HTTPError: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub

Saw it multiple times this morning, including within seconds of doing a `fedpkg build`. ``` [adamw@toolbx fedora-toolbox-43 os-autoinst (f42 %)]$ fedpkg build Building os-autoinst-5^20250707gitd55ec72-6.fc42 for f42-candidate Created task: 139261749 Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=139261749 Watching tasks (this may be safely interrupted)... 139261749 build (f42-candidate, /rpms/os-autoinst.git:125dca21d6bed9f2bc17b194b99b3f4e23f7771a): free 139261749 build (f42-candidate, /rpms/os-autoinst.git:125dca21d6bed9f2bc17b194b99b3f4e23f7771a): free -> open (buildvm-a64-25.rdu3.fedoraproject.org) Could not execute build: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub [adamw@toolbx fedora-toolbox-43 os-autoinst (f42 %)]$ koji watch-task 139261749 Watching tasks (this may be safely interrupted)... 139261749 build (f42-candidate, /rpms/os-autoinst.git:125dca21d6bed9f2bc17b194b99b3f4e23f7771a): open (buildvm-a64-25.rdu3.fedoraproject.org) 139261760 buildSRPMFromSCM (/rpms/os-autoinst.git:125dca21d6bed9f2bc17b194b99b3f4e23f7771a): open (buildvm-a64-05.rdu3.fedoraproject.org) 2025-11-24 11:59:46,060 [ERROR] koji: HTTPError: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub [adamw@toolbx fedora-toolbox-43 os-autoinst (f42 %)]$ koji watch-task 139261749 Watching tasks (this may be safely interrupted)... 139261749 build (f42-candidate, /rpms/os-autoinst.git:125dca21d6bed9f2bc17b194b99b3f4e23f7771a): closed 139261981 tagBuild (noarch): closed 139261803 buildArch (os-autoinst-5^20250707gitd55ec72-6.fc42.src.rpm, ppc64le): closed 139261804 buildArch (os-autoinst-5^20250707gitd55ec72-6.fc42.src.rpm, s390x): closed 139261801 buildArch (os-autoinst-5^20250707gitd55ec72-6.fc42.src.rpm, x86_64): closed 139261802 buildArch (os-autoinst-5^20250707gitd55ec72-6.fc42.src.rpm, aarch64): closed 139261760 buildSRPMFromSCM (/rpms/os-autoinst.git:125dca21d6bed9f2bc17b194b99b3f4e23f7771a): closed 2025-11-24 12:12:30,826 [ERROR] koji: HTTPError: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub ```

mikem commented

2025-11-25 15:02:27 +00:00

@adamwill do you have a custom poll_interval setting for koji?

adamwill commented

2025-11-25 17:06:01 +00:00

Member

I don't believe so. If I did, where would it be set? The string poll doesn't appear in my /etc/koji.conf.

I don't believe so. If I did, where would it be set? The string `poll` doesn't appear in my `/etc/koji.conf`.

mikem commented

2025-11-25 17:12:36 +00:00

I don't believe so. If I did, where would it be set?

It's not something most people adjust, but I wanted to make sure for replication purposes. Koji pulls settings from ~/.koji/config, /etc/koji.conf, and /etc/koji.conf.d*

> I don't believe so. If I did, where would it be set? It's not something most people adjust, but I wanted to make sure for replication purposes. Koji pulls settings from ~/.koji/config, /etc/koji.conf, and /etc/koji.conf.d*

mikem commented

2025-11-25 17:32:31 +00:00

I've been playing with this and while I can't make it replicate readily, I have noticed that backend connection changes during the course of keepalive (at least according to the X-Fedora-Appserver header).

header: X-Fedora-Appserver: koji02.rdu3.fedoraproject.org
header: X-Fedora-Proxyserver: proxy01.rdu3.fedoraproject.org
header: Keep-Alive: timeout=15, max=450

...

header: X-Fedora-Appserver: koji01.rdu3.fedoraproject.org
header: X-Fedora-Proxyserver: proxy01.rdu3.fedoraproject.org
header: Keep-Alive: timeout=15, max=449

This implies the two layers of keepalive are untethered. Perhaps the proxy is just grabbing a connection from a pool each time (unless it is not using keepalive?). So my suggestion to make the inner keepalive timeout longer doesn't solve the issue (though perhaps a longer timeout might reduce the problem).

I've been playing with this and while I can't make it replicate readily, I have noticed that backend connection changes during the course of keepalive (at least according to the `X-Fedora-Appserver` header). ``` header: X-Fedora-Appserver: koji02.rdu3.fedoraproject.org header: X-Fedora-Proxyserver: proxy01.rdu3.fedoraproject.org header: Keep-Alive: timeout=15, max=450 ... header: X-Fedora-Appserver: koji01.rdu3.fedoraproject.org header: X-Fedora-Proxyserver: proxy01.rdu3.fedoraproject.org header: Keep-Alive: timeout=15, max=449 ``` This implies the two layers of keepalive are untethered. Perhaps the proxy is just grabbing a connection from a pool each time (unless it is not using keepalive?). So my suggestion to make the inner keepalive timeout longer doesn't solve the issue (though perhaps a longer timeout might reduce the problem).

mikem commented

2025-12-02 19:34:01 +00:00

If I spam the hub with hello calls (a no-op) as fast as I can, it takes about 10-15 min to hit it. It seems to happen with both proxy01 and proxy10. If I disable keepalive, it takes longer to hit it, but this is probably just because keepalive allows for a faster rate of calls.

Unfortunately, this doesn't really give any great insight into what is happening.

Here is my best guess -- the proxy is using a pool of keepalive connections to the backends and not quite handling the inherent race condition in the http keepalive spec. The race happens when the keepalive timeout happens just barely after the client (in this case the proxy) decides to reuse the connection. It fails to connect and passes the error on.

You can see a different example of this race here -- https://github.com/mikem23/keepalive-race
(this code is specific to python-requests, but the general race could happen with different tools).

If I spam the hub with `hello` calls (a no-op) as fast as I can, it takes about 10-15 min to hit it. It seems to happen with both proxy01 and proxy10. If I disable keepalive, it takes longer to hit it, but this is probably just because keepalive allows for a faster rate of calls. Unfortunately, this doesn't really give any great insight into what is happening. Here is my best guess -- the proxy is using a pool of keepalive connections to the backends and not quite handling the inherent race condition in the http keepalive spec. The race happens when the keepalive timeout happens just barely after the client (in this case the proxy) decides to reuse the connection. It fails to connect and passes the error on. You can see a different example of this race here -- https://github.com/mikem23/keepalive-race (this code is specific to python-requests, but the general race could happen with different tools).

decathorpe commented

2025-12-11 15:58:18 +00:00

If this is a race condition, then it seems I'm quite unlucky, because I appear to have been hitting it consistently a few times per minute for the past hour or so. :(

mikem commented

2025-12-11 16:15:39 +00:00

As noted above, folks can work around this by setting anon_retry=true in their koji config.

vondruch commented

2025-12-18 10:14:31 +00:00

Just FTR, my experience in past two days is that Zuul is basically unusable for Ruby builds due to this 🤷

https://fedora.softwarefactory-project.io/zuul/build/41e6b1e4e3ef42ed8fb1e372c097e440

Just FTR, my experience in past two days is that Zuul is basically unusable for Ruby builds due to this 🤷 https://fedora.softwarefactory-project.io/zuul/build/41e6b1e4e3ef42ed8fb1e372c097e440

mikem commented

2025-12-18 17:11:43 +00:00

While this is an infra issue, not a Koji issue, the anon_retry setting for Koji can help. I'm going to change the default to True in the upcoming release. https://pagure.io/koji/pull-request/4517

In the meantime, any systems that are running into this should be able to just set anon_retry=true in their koji configs.

While this is an infra issue, not a Koji issue, the anon_retry setting for Koji can help. I'm going to change the default to True in the upcoming release. https://pagure.io/koji/pull-request/4517 In the meantime, any systems that are running into this should be able to just set `anon_retry=true` in their koji configs.

mikem commented

2025-12-18 17:33:25 +00:00

My script to replicate this is hitting it much faster today. I'm getting it in about 45s instead of the 10-15m I reported before.

Most recently, the script hit it on call number 570.

Elapsed time: 45.40 sec
INFO:koji.test:Sending request #570
send: b'POST /kojihub HTTP/1.1\r\nHost: koji.fedoraproject.org\r\nUser-Agent: koji-gateway-test/1\r\nAccept-Encoding: gzip, deflate, br, zstd\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Type: text/xml\r\nContent-Length: 99\r\n\r\n'
send: b"<?xml version='1.0'?>\n<methodCall>\n<methodName>hello</methodName>\n<params>\n</params>\n</methodCall>\n"
reply: 'HTTP/1.1 502 Bad Gateway\r\n'
header: Date: Thu, 18 Dec 2025 17:20:54 GMT
header: Server: Apache
header: Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
header: Content-Length: 0
header: AppTime: D=18509
header: X-Fedora-ProxyServer: proxy01.rdu3.fedoraproject.org
header: Keep-Alive: timeout=15, max=432
header: Connection: Keep-Alive
DEBUG:urllib3.connectionpool:https://koji.fedoraproject.org:443 "POST /kojihub HTTP/1.1" 502 0

If I look at the access logs on the hub, I only see entries for the successful calls. The unsuccessful one does not appear to be reaching either of the backend hubs. (Granted this is not surprising, but nice to confirm)

[root@koji01 httpd][PROD-RDU3]# grep -c -P '18/Dec/2025:17:(18|19|20).*koji-gateway-test' access_log
291
[root@koji02 httpd][PROD-RDU3]# grep -c -P '18/Dec/2025:17:(18|19|20).*koji-gateway-test' access_log
278

291 + 278 = 569
I.e. one short of the total number of calls in the test run.

My script to replicate this is hitting it much faster today. I'm getting it in about 45s instead of the 10-15m I reported before. Most recently, the script hit it on call number 570. ``` Elapsed time: 45.40 sec INFO:koji.test:Sending request #570 send: b'POST /kojihub HTTP/1.1\r\nHost: koji.fedoraproject.org\r\nUser-Agent: koji-gateway-test/1\r\nAccept-Encoding: gzip, deflate, br, zstd\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Type: text/xml\r\nContent-Length: 99\r\n\r\n' send: b"<?xml version='1.0'?>\n<methodCall>\n<methodName>hello</methodName>\n<params>\n</params>\n</methodCall>\n" reply: 'HTTP/1.1 502 Bad Gateway\r\n' header: Date: Thu, 18 Dec 2025 17:20:54 GMT header: Server: Apache header: Strict-Transport-Security: max-age=31536000; includeSubDomains; preload header: Content-Length: 0 header: AppTime: D=18509 header: X-Fedora-ProxyServer: proxy01.rdu3.fedoraproject.org header: Keep-Alive: timeout=15, max=432 header: Connection: Keep-Alive DEBUG:urllib3.connectionpool:https://koji.fedoraproject.org:443 "POST /kojihub HTTP/1.1" 502 0 ``` If I look at the access logs on the hub, I only see entries for the successful calls. The unsuccessful one does not appear to be reaching either of the backend hubs. (Granted this is not surprising, but nice to confirm) ``` [root@koji01 httpd][PROD-RDU3]# grep -c -P '18/Dec/2025:17:(18|19|20).*koji-gateway-test' access_log 291 [root@koji02 httpd][PROD-RDU3]# grep -c -P '18/Dec/2025:17:(18|19|20).*koji-gateway-test' access_log 278 ``` 291 + 278 = 569 I.e. one short of the total number of calls in the test run.

mikem commented

2025-12-18 17:35:59 +00:00

None of the access.log entries for my test calls show anything other than a 200 result

[root@koji01 httpd][PROD-RDU3]# grep gateway.test access_log|awk '$9 != 200'

(and same on koji02)

None of the access.log entries for my test calls show anything other than a 200 result ``` [root@koji01 httpd][PROD-RDU3]# grep gateway.test access_log|awk '$9 != 200' ``` (and same on koji02)

zlopez commented

2026-01-16 11:04:44 +00:00

Owner

@mikem Is this still happening?

vondruch commented

2026-01-16 13:05:17 +00:00

Was there some change? I think it was happening e.g. 3 days ago:

https://src.fedoraproject.org/rpms/ruby/pull-request/226#comment-301595

Was there some change? I think it was happening e.g. 3 days ago: https://src.fedoraproject.org/rpms/ruby/pull-request/226#comment-301595

vondruch commented

2026-01-19 09:55:33 +00:00

More recent:

https://src.fedoraproject.org/rpms/cucumber-messages/pull-request/2#comment-302505

More recent: https://src.fedoraproject.org/rpms/cucumber-messages/pull-request/2#comment-302505

kevin commented

2026-01-19 22:08:27 +00:00

Author

Owner

That is almost surely related to the mass rebuild... nothing to do with this ticket.

music commented

2026-01-19 22:41:07 +00:00

I was still seeing this when I finally tried the workaround from https://pagure.io/fedora-infrastructure/issue/12913#comment-993744 last week. I will comment out the workaround and report back if/when I see this again.

kevin commented

2026-01-20 00:25:40 +00:00

Author

Owner

To be clear, I haven't done anything to fix this issue, was just saying that those PR issues looked like timeouts caused by the mass rebuild in ci.

But it would indeed be good to know if it's still there or not.

To be clear, I haven't done anything to fix this issue, was just saying that those PR issues looked like timeouts caused by the mass rebuild in ci. But it would indeed be good to know if it's still there or not.

mikem commented

2026-01-20 19:30:18 +00:00

fwiw, https://pagure.io/koji/pull-request/4517 was merged and will be in the upcoming 1.36 Koji release. This doesn't fix whatever underlying issue is happening with Fedora's proxy, but it should lessen the impact for the Koji cli.

mikem commented

2026-01-20 19:32:13 +00:00

Is this still happening?

I ran my replication script just now. It reproduced the issue in 1m1.953s (1068 calls).

> Is this still happening? I ran my replication script just now. It reproduced the issue in 1m1.953s (1068 calls).

mikem commented

2026-01-21 13:44:15 +00:00

And this morning, 2m17.555s (2316 calls)

rjones commented

2026-02-20 14:01:26 +00:00

I'm definitely seeing this today, eg:

$ fedpkg build --target=f45-build-side-130113
Found a gating.yaml file in the repo and it is properly configured
Using ocaml-ocamlbuild.spec
Building ocaml-ocamlbuild-0.16.1-5.fc45 for f45-build-side-130113
Created task: 142543104
Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=142543104
Watching tasks (this may be safely interrupted)...
142543104 build (f45-build-side-130113, /rpms/ocaml-ocamlbuild.git:ee70a0b94c0e0915c8f01d027c4acd0fbf6f3d15): free
142543104 build (f45-build-side-130113, /rpms/ocaml-ocamlbuild.git:ee70a0b94c0e0915c8f01d027c4acd0fbf6f3d15): free -> open (buildvm-a64-11.rdu3.fedoraproject.org)
  142543128 waitrepo (f45-build-side-130113, ): free
  142543128 waitrepo (f45-build-side-130113, ): free -> open (buildvm-s390x-08.s390.fedoraproject.org)
  142543236 buildSRPMFromSCM (/rpms/ocaml-ocamlbuild.git:ee70a0b94c0e0915c8f01d027c4acd0fbf6f3d15): free
  142543128 waitrepo (f45-build-side-130113, ): open (buildvm-s390x-08.s390.fedoraproject.org) -> closed
  1 free  1 open  1 done  0 failed
  142543236 buildSRPMFromSCM (/rpms/ocaml-ocamlbuild.git:ee70a0b94c0e0915c8f01d027c4acd0fbf6f3d15): free -> open (buildvm-a64-47.rdu3.fedoraproject.org)
  142543318 buildArch (ocaml-ocamlbuild-0.16.1-5.fc45.src.rpm, aarch64): free
  142543319 buildArch (ocaml-ocamlbuild-0.16.1-5.fc45.src.rpm, ppc64le): free
  142543316 buildArch (ocaml-ocamlbuild-0.16.1-5.fc45.src.rpm, x86_64): free
  142543320 buildArch (ocaml-ocamlbuild-0.16.1-5.fc45.src.rpm, s390x): free
  142543236 buildSRPMFromSCM (/rpms/ocaml-ocamlbuild.git:ee70a0b94c0e0915c8f01d027c4acd0fbf6f3d15): open (buildvm-a64-47.rdu3.fedoraproject.org) -> closed
  4 free  1 open  2 done  0 failed
  142543318 buildArch (ocaml-ocamlbuild-0.16.1-5.fc45.src.rpm, aarch64): free -> open (buildvm-a64-04.rdu3.fedoraproject.org)
Could not execute build: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub

While it's recoverable when doing builds by hand, it breaks my automation for the OCaml rebuild.

I'm definitely seeing this today, eg: ``` $ fedpkg build --target=f45-build-side-130113 Found a gating.yaml file in the repo and it is properly configured Using ocaml-ocamlbuild.spec Building ocaml-ocamlbuild-0.16.1-5.fc45 for f45-build-side-130113 Created task: 142543104 Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=142543104 Watching tasks (this may be safely interrupted)... 142543104 build (f45-build-side-130113, /rpms/ocaml-ocamlbuild.git:ee70a0b94c0e0915c8f01d027c4acd0fbf6f3d15): free 142543104 build (f45-build-side-130113, /rpms/ocaml-ocamlbuild.git:ee70a0b94c0e0915c8f01d027c4acd0fbf6f3d15): free -> open (buildvm-a64-11.rdu3.fedoraproject.org) 142543128 waitrepo (f45-build-side-130113, ): free 142543128 waitrepo (f45-build-side-130113, ): free -> open (buildvm-s390x-08.s390.fedoraproject.org) 142543236 buildSRPMFromSCM (/rpms/ocaml-ocamlbuild.git:ee70a0b94c0e0915c8f01d027c4acd0fbf6f3d15): free 142543128 waitrepo (f45-build-side-130113, ): open (buildvm-s390x-08.s390.fedoraproject.org) -> closed 1 free 1 open 1 done 0 failed 142543236 buildSRPMFromSCM (/rpms/ocaml-ocamlbuild.git:ee70a0b94c0e0915c8f01d027c4acd0fbf6f3d15): free -> open (buildvm-a64-47.rdu3.fedoraproject.org) 142543318 buildArch (ocaml-ocamlbuild-0.16.1-5.fc45.src.rpm, aarch64): free 142543319 buildArch (ocaml-ocamlbuild-0.16.1-5.fc45.src.rpm, ppc64le): free 142543316 buildArch (ocaml-ocamlbuild-0.16.1-5.fc45.src.rpm, x86_64): free 142543320 buildArch (ocaml-ocamlbuild-0.16.1-5.fc45.src.rpm, s390x): free 142543236 buildSRPMFromSCM (/rpms/ocaml-ocamlbuild.git:ee70a0b94c0e0915c8f01d027c4acd0fbf6f3d15): open (buildvm-a64-47.rdu3.fedoraproject.org) -> closed 4 free 1 open 2 done 0 failed 142543318 buildArch (ocaml-ocamlbuild-0.16.1-5.fc45.src.rpm, aarch64): free -> open (buildvm-a64-04.rdu3.fedoraproject.org) Could not execute build: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub ``` While it's recoverable when doing builds by hand, it breaks my automation for the OCaml rebuild.

victorkoycheff referenced this issue from infra/ansible

2026-02-25 17:25:41 +00:00

proxies-reverseproxy: set keepalive=on ttl=10 for koji #3173

victorkoycheff commented

2026-02-25 17:29:49 +00:00

Hey everyone, I've opened a PR to fix this.

As @mikem suspected, this is a mod_proxy keep-alive race condition. Because the proxy and backend keep-alive pools are untethered, the proxy occasionally tries to hand off a pooled connection to a client right as the Koji backend (which has a 16s timeout) closes it.

The PR adds keepalive=on ttl=10 to the Koji proxyopts in the reverse proxy playbook. This forces Apache to proactively expire its pooled backend connections after 10 seconds, well before the Koji hub ever gets the chance to close them. There should be no more 502s on long-running tasks.

infra/ansible#3173

Also check this:
Reference: Apache mod_proxy documentation

Hey everyone, I've opened a PR to fix this. As @mikem suspected, this is a `mod_proxy` keep-alive race condition. Because the proxy and backend keep-alive pools are untethered, the proxy occasionally tries to hand off a pooled connection to a client right as the Koji backend (which has a 16s timeout) closes it. The PR adds `keepalive=on ttl=10` to the Koji `proxyopts` in the reverse proxy playbook. This forces Apache to proactively expire its pooled backend connections after 10 seconds, well before the Koji hub ever gets the chance to close them. There should be no more 502s on long-running tasks. https://forge.fedoraproject.org/infra/ansible/pulls/3173 Also check this: Reference: [Apache mod_proxy documentation](https://httpd.apache.org/docs/2.4/mod/mod_proxy.html)

❤️ 1 🎉 1

music commented

2026-02-25 20:54:03 +00:00

Thanks! Excited to see progress on this.

👍 1

kevin commented

2026-03-11 21:07:08 +00:00

Author

Owner

I've merged this and am deploying to staging.

If folks could test against staging that would be great.

I've merged this and am deploying to staging. If folks could test against staging that would be great.

mikem commented

2026-03-11 21:39:08 +00:00

If folks could test against staging that would be great.

Caveat. I've never pointed my script at stage, so I don't have a proper baseline to compare against there.
That said, it's been running for about 20 minutes without error. (In comparison, it encountered the issue on prod in less than 5 minutes just beforehand).

So, stage seems fine, but I technically do not know if the problem was manifesting there before.

> If folks could test against staging that would be great. Caveat. I've never pointed my script at stage, so I don't have a proper baseline to compare against there. That said, it's been running for about 20 minutes without error. (In comparison, it encountered the issue on prod in less than 5 minutes just beforehand). So, stage seems fine, but I technically do not know if the problem was manifesting there before.

kevin commented

2026-03-11 21:59:17 +00:00

Author

Owner

Yeah, I think it was... and I've been testing here too and it seems ok.

So, lets try production! Deploying now...

Yeah, I think it was... and I've been testing here too and it seems ok. So, lets try production! Deploying now...

kevin commented

2026-03-11 22:26:41 +00:00

Author

Owner

All deployed. Test away.

kevin commented

2026-03-11 23:04:48 +00:00

Author

Owner

I still hit one...

2026-03-11 15:58:13,415 [ERROR] koji: HTTPError: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub

I still hit one... 2026-03-11 15:58:13,415 [ERROR] koji: HTTPError: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub

victorkoycheff commented

2026-03-12 07:48:11 +00:00

ok, so looking closer at mikem's debug logs where max=450 was showing up, that wasn't a static limit - it was actually counting down from 500.

It turns out our backend Apache configs (kojiweb and the proxy httpd.conf specifically) had MaxKeepAliveRequests 500 hardcoded. If a script spams the hub fast enough to bypass the 10-second idle ttl we just added, it hits exactly 500 requests and the backend just ruthlessly drops the connection. If the proxy tries to hand off a new request to that pooled connection at that exact second, we get the 502.

I've put up a new PR to set MaxKeepAliveRequests 0 (unlimited) on the backend templates so it never hangs up on the proxy unexpectedly. Our previous ttl=10 fix is still completely necessary to clean up idle connections, so the two together should finally kill this bug.

PR is here: infra/ansible#3213

Let's see if that works

ok, so looking closer at mikem's debug logs where max=450 was showing up, that wasn't a static limit - it was actually counting down from 500. It turns out our backend Apache configs (kojiweb and the proxy httpd.conf specifically) had MaxKeepAliveRequests 500 hardcoded. If a script spams the hub fast enough to bypass the 10-second idle ttl we just added, it hits exactly 500 requests and the backend just ruthlessly drops the connection. If the proxy tries to hand off a new request to that pooled connection at that exact second, we get the 502. I've put up a new PR to set MaxKeepAliveRequests 0 (unlimited) on the backend templates so it never hangs up on the proxy unexpectedly. Our previous ttl=10 fix is still completely necessary to clean up idle connections, so the two together should finally kill this bug. PR is here: https://forge.fedoraproject.org/infra/ansible/pulls/3213 Let's see if that works

mikem commented

2026-03-12 14:04:16 +00:00

fwiw, I ran my script against prod for 40m yesterday after the deployment and did not see an error

mikem commented

2026-03-12 14:49:53 +00:00

BUT, today is a different story. My script is hitting the issue in less than a minute reliably (against prod)

mikem commented

2026-03-12 15:39:59 +00:00

I'm also seeing a number of aborted connections this morning. That seems new.

Got 5 errors in 176.59 sec
  ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
  502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub
  502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub
  502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub
  502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub
Average error time 35.32 sec

I'm also seeing a number of aborted connections this morning. That seems new. ``` Got 5 errors in 176.59 sec ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub Average error time 35.32 sec ```

kevin commented

2026-03-12 20:25:36 +00:00

Author

Owner

So, that might be related to the kojipkgs outage / issue this morning, as that was likely filling up connection slots for other applications since they all use the same proxies.

Can you retest now?

So, that might be related to the kojipkgs outage / issue this morning, as that was likely filling up connection slots for other applications since they all use the same proxies. Can you retest now?

victorkoycheff commented

2026-03-12 20:40:24 +00:00

@mikem @kevin If we are going to retest I suggest also merging infra/ansible#3213 too so we can double whammy the issue and fix it once and for all

@mikem @kevin If we are going to retest I suggest also merging https://forge.fedoraproject.org/infra/ansible/pulls/3213 too so we can double whammy the issue and fix it once and for all

kevin commented

2026-03-12 20:49:28 +00:00

Author

Owner

well, I would kind of like @mikem to confirm he's still seeing it, but only after 500 requests before we do that...
because if the underlying issue isn't solved, thats not going to help.

well, I would kind of like @mikem to confirm he's still seeing it, but only after 500 requests before we do that... because if the underlying issue isn't solved, thats not going to help.

mikem commented

2026-03-12 21:28:00 +00:00

Still seeing it on prod, roughly every 20s right now

mikem commented

2026-03-12 21:28:46 +00:00

Only gateway errors though. Not seeing the aborted connections anymore, at least so far.

mikem commented

2026-03-12 22:52:52 +00:00

Actually I did get one aborted connection in the run. In about 10m, got 19 gateway errors and 1 aborted connection.

mikem commented

2026-03-13 20:24:43 +00:00

Anyway, happy to test whenever

gtb commented

2026-04-17 13:19:18 +00:00

FWIW, I am still seeing occasional Could not execute scratch_build: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub during the watch-logs part of a few recent fedpkg scratch-builds, although the scratch build does seem to complete.

FWIW, I am still seeing occasional `Could not execute scratch_build: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub `during the watch-logs part of a few recent fedpkg scratch-builds, although the scratch build does seem to complete.

mikem commented

2026-04-17 14:00:11 +00:00

Yes, still happening quite readily if I run my test.

Also, I now see a number of connection aborted errors in the mix (about 1 in 5 of them them morning).

Yes, still happening quite readily if I run my test. Also, I now see a number of connection aborted errors in the mix (about 1 in 5 of them them morning).

kevin commented

2026-04-17 16:28:07 +00:00

Author

Owner

FWIW, there was a DDOS against out wiki this morning and those pass through the same proxies, so might have caused instablity in other applications.

adamwill referenced this issue

2026-04-25 07:09:42 +00:00

Testing Farm gets intermittent 502s from koji.fedoraproject.org #13292

kevin commented

2026-04-30 02:25:03 +00:00

Author

Owner

ok, I merged infra/ansible#3213 and it did seem to help in staging, so I pushed to prod too.

Can everyone test and see if things seem better?

ok, I merged https://forge.fedoraproject.org/infra/ansible/pulls/3213 and it did seem to help in staging, so I pushed to prod too. Can everyone test and see if things seem better?

mikem commented

2026-04-30 16:26:43 +00:00

I ran my script and got plenty of errors still. Got 50 errors in 422.66 sec

	 41   502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub
	  9   ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

I ran my script and got plenty of errors still. `Got 50 errors in 422.66 sec` ``` 41 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub 9 ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) ```

adamwill commented

2026-04-30 16:32:20 +00:00

Member

Right now was possibly a bad time to run it, as we were rebooting stuff for copyfail. Can you try again?

mikem commented

2026-04-30 18:22:32 +00:00

INFO:koji.test:Got 40 errors in 600.02 sec
INFO:koji.test:  31: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub
INFO:koji.test:  9: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
INFO:koji.test:Average error time 15.00 sec

``` INFO:koji.test:Got 40 errors in 600.02 sec INFO:koji.test: 31: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub INFO:koji.test: 9: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) INFO:koji.test:Average error time 15.00 sec ```

mikem commented

2026-04-30 18:29:49 +00:00

Perhaps not surprising, but a data point: I get the gateway errors even if I disable keepalive on the client end (though of course more slowly)

mikem commented

2026-04-30 18:53:49 +00:00

Looking at infra/ansible!3213 I see it sets MaxKeepAliveRequests 0 in multiple http conf templates -- httpd.conf, kojihub.conf, and kojiweb.conf.

If I look at the config on koji01, I do not see this setting applied in /etc/httpd/conf/httpd.conf, just the other two.

Looking at infra/ansible!3213 I see it sets `MaxKeepAliveRequests 0` in multiple http conf templates -- httpd.conf, kojihub.conf, and kojiweb.conf. If I look at the config on koji01, I **do not** see this setting applied in /etc/httpd/conf/httpd.conf, just the other two.

mikem commented

2026-04-30 19:07:34 +00:00

It might be useful to add %k to LogFormat for the httpd access logs (unless that breaks something?)

It might be useful to add [%k to LogFormat](https://httpd.apache.org/docs/current/mod/mod_log_config.html) for the httpd access logs (unless that breaks something?)

kevin commented

2026-04-30 23:10:49 +00:00

Author

Owner

Yeah, koji01 includes all the /etc/httpd/conf.d/*.conf files, so that gets applied in kojihub/kojiweb.conf... httpd.conf is on the proxy side.

Yeah, adding %k would be good...

Yeah, koji01 includes all the /etc/httpd/conf.d/*.conf files, so that gets applied in kojihub/kojiweb.conf... httpd.conf is on the proxy side. Yeah, adding %k would be good...

kevin commented

2026-05-11 17:04:26 +00:00

Author

Owner

I just used claude to look at all the things we tried here and what could be missing and it found one thing.

I have fixed that... can folks please retest? I can push the fix to ansible if it turns out to be the fix.

Running the test script and so far: Got 0 errors in 367.63 sec
and did a watch-logs on a big build and it's going along fine.

I just used claude to look at all the things we tried here and what could be missing and it found one thing. I have fixed that... can folks please retest? I can push the fix to ansible if it turns out to be the fix. Running the test script and so far: Got 0 errors in 367.63 sec and did a watch-logs on a big build and it's going along fine.

mikem commented

2026-05-11 18:28:58 +00:00

I'm still seeing errors at the moment

2026-05-11 14:25:55,815 [INFO] koji.test: Got 5 errors in 56.38 sec
2026-05-11 14:25:55,815 [INFO] koji.test:   5: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub
2026-05-11 14:25:55,815 [INFO] koji.test: Average error time 11.28 sec

I'm still seeing errors at the moment ``` 2026-05-11 14:25:55,815 [INFO] koji.test: Got 5 errors in 56.38 sec 2026-05-11 14:25:55,815 [INFO] koji.test: 5: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub 2026-05-11 14:25:55,815 [INFO] koji.test: Average error time 11.28 sec ```

kevin commented

2026-05-11 18:33:41 +00:00

Author

Owner

Ah bummer. ok. I will keep digging...

I pushed a number of changes out now:

The proxyops that Victor set in #12913 (comment) were not actually being applied. Fixed the template.
There was no timeout set on the hub (but this is cosmetic).
The proxy to anubis didn't have keepalive set right, set now.

I'll see if I can find more. :(

Ah bummer. ok. I will keep digging... I pushed a number of changes out now: 1. The proxyops that Victor set in https://forge.fedoraproject.org/infra/tickets/issues/12913#issuecomment-557246 were not actually being applied. Fixed the template. 2. There was no timeout set on the hub (but this is cosmetic). 3. The proxy to anubis didn't have keepalive set right, set now. I'll see if I can find more. :(

kevin commented

2026-05-11 19:00:56 +00:00

Author

Owner

ok, found a few more things.

Can you retest? So far here no errors... but you seem to get them a lot quicker/more often than I do for some reason. ;(

ok, found a few more things. Can you retest? So far here no errors... but you seem to get them a lot quicker/more often than I do for some reason. ;(

mikem commented

2026-05-11 21:26:47 +00:00

2026-05-11 17:25:16,448 [INFO] Got 6 errors in 97.76 sec
2026-05-11 17:25:16,448 [INFO]   5: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub
2026-05-11 17:25:16,448 [INFO]   1: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
2026-05-11 17:25:16,448 [INFO] Average error time 16.29 sec

``` 2026-05-11 17:25:16,448 [INFO] Got 6 errors in 97.76 sec 2026-05-11 17:25:16,448 [INFO] 5: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub 2026-05-11 17:25:16,448 [INFO] 1: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) 2026-05-11 17:25:16,448 [INFO] Average error time 16.29 sec ```

kevin commented

2026-05-11 21:57:13 +00:00

Author

Owner

ok. I got annoyed and spent more time adding debugging and digging and now I think I see the actual issue.

It's the return path.

backend koji sever sends a 200 reply back -> anubis logs:

May 11 21:26:19 proxy01.rdu3.fedoraproject.org anubis[2948855]: {"time":"2026-05-11T21:26:19.11382
903Z","level":"INFO","msg":"http: proxy error: EOF"}

So, it's reading from the backend the reply (which was a 200) and gets a EOF and then sends back to the client the 502.

Unfortunately, anubis doesn't log the connection/host that got this, but matching up verbose apache logs shows the times match up, so I am pretty sure this is it.,

I'm trying to figure out what is causing that now. This may be causing other weird anubis things as well...

ok. I got annoyed and spent more time adding debugging and digging and now I think I see the actual issue. It's the return path. backend koji sever sends a 200 reply back -> anubis logs: ``` May 11 21:26:19 proxy01.rdu3.fedoraproject.org anubis[2948855]: {"time":"2026-05-11T21:26:19.11382 903Z","level":"INFO","msg":"http: proxy error: EOF"} ``` So, it's reading from the backend the reply (which was a 200) and gets a EOF and then sends back to the client the 502. Unfortunately, anubis doesn't log the connection/host that got this, but matching up verbose apache logs shows the times match up, so I am pretty sure this is it., I'm trying to figure out what is causing that now. This may be causing other weird anubis things as well...

kevin commented

2026-05-12 00:05:54 +00:00

Author

Owner

ok. I tweaked all the keepalives and other things I can think of.

Filed upstream: https://github.com/TecharoHQ/anubis/issues/1624

ok. I tweaked all the keepalives and other things I can think of. Filed upstream: https://github.com/TecharoHQ/anubis/issues/1624

mikem commented

2026-05-12 15:00:30 +00:00

Getting less errors, but still getting them this morning

2026-05-12 10:58:03,719 [INFO] koji.test: Got 16 errors in 600.02 sec
2026-05-12 10:58:03,719 [INFO] koji.test:   4: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
2026-05-12 10:58:03,719 [INFO] koji.test:   12: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub
2026-05-12 10:58:03,719 [INFO] koji.test: Average error time 37.50 sec

Getting less errors, but still getting them this morning ``` 2026-05-12 10:58:03,719 [INFO] koji.test: Got 16 errors in 600.02 sec 2026-05-12 10:58:03,719 [INFO] koji.test: 4: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) 2026-05-12 10:58:03,719 [INFO] koji.test: 12: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub 2026-05-12 10:58:03,719 [INFO] koji.test: Average error time 37.50 sec ```

mikem commented

2026-05-12 21:29:25 +00:00

Just got a gateway error updating a forgejo PR. Related?

mikem commented

2026-05-12 21:30:42 +00:00

Quite ironically, submitting the above comment also gave me a gateway error

kevin commented

2026-05-12 21:32:46 +00:00

Author

Owner

Quite possibly. I see anubis EOF messages, but it's hard to match them up to the request... they don't say what host/url/session/anything is involved.

So, yes, this could be happening on other services as well. ;(

Quite possibly. I see anubis EOF messages, but it's hard to match them up to the request... they don't say what host/url/session/anything is involved. So, yes, this could be happening on other services as well. ;(

kevin referenced this issue

2026-05-12 21:35:13 +00:00

Testing Farm gets intermittent 502s from koji.fedoraproject.org #13292

kevin commented

2026-05-14 22:02:37 +00:00

Author

Owner

ok. I patched anubis to log more for these cases. It didn't help too much, but reminded me that these are all POSTs...

so on proxy01.fedoraproject.org I set it to just bypass anubis for POSTs (we already just allow them in anubis policy).

Can you test against just proxy01 and see if you still can get it to fail?

ok. I patched anubis to log more for these cases. It didn't help too much, but reminded me that these are all POSTs... so on proxy01.fedoraproject.org I set it to just bypass anubis for POSTs (we already just allow them in anubis policy). Can you test against just proxy01 and see if you still can get it to fail?

kevin referenced this issue from a commit

2026-05-14 23:32:24 +00:00

proxies / website: on src/koji/riscv-koji have POSTs bypass anubis

kevin referenced this issue from infra/ansible

2026-05-14 23:32:34 +00:00

proxies / website: on src/koji/riscv-koji have POSTs bypass anubis #3352

kevin referenced this issue from a commit

2026-05-14 23:56:35 +00:00

proxies / website: on src/koji/riscv-koji have POSTs bypass anubis

mikem commented

2026-05-15 14:21:46 +00:00

This morning it seems about the same for me

2026-05-15 10:20:09,232 [INFO] koji.test: Got 5 errors in 42.98 sec
2026-05-15 10:20:09,232 [INFO] koji.test:   5: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub
2026-05-15 10:20:09,232 [INFO] koji.test: Average error time 8.60 sec

This morning it seems about the same for me ``` 2026-05-15 10:20:09,232 [INFO] koji.test: Got 5 errors in 42.98 sec 2026-05-15 10:20:09,232 [INFO] koji.test: 5: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub 2026-05-15 10:20:09,232 [INFO] koji.test: Average error time 8.60 sec ```

mikem commented

2026-05-15 14:27:41 +00:00

Oh, you said proxy01.fedoraproject.org. I don't know what url to hit there?

Oh, you said `proxy01.fedoraproject.org`. I don't know what url to hit there?

mikem commented

2026-05-15 14:35:05 +00:00

nm, I figured it out ;)

mikem commented

2026-05-15 15:47:52 +00:00

Zero gateway errors in on proxy01 this morning

2026-05-15 10:54:35,718 [INFO] koji.test: Got 3 errors in 1200.01 sec
2026-05-15 10:54:35,718 [INFO] koji.test:   3: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
2026-05-15 10:54:35,718 [INFO] koji.test: Average error time 400.00 sec
2026-05-15 10:54:35,718 [WARNING] koji.test: Exiting due to timeout

Zero gateway errors in on proxy01 this morning ``` 2026-05-15 10:54:35,718 [INFO] koji.test: Got 3 errors in 1200.01 sec 2026-05-15 10:54:35,718 [INFO] koji.test: 3: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) 2026-05-15 10:54:35,718 [INFO] koji.test: Average error time 400.00 sec 2026-05-15 10:54:35,718 [WARNING] koji.test: Exiting due to timeout ```

kevin referenced this issue from a commit

2026-05-15 16:39:09 +00:00

proxies / website: on src/koji/riscv-koji have POSTs bypass anubis

gtb commented

2026-05-15 17:39:49 +00:00

FWIW, https://github.com/fedora-copr/copr/issues/4110 seems to report a similar solution (disable anubis for POSTs).

kevin commented

2026-05-15 18:29:01 +00:00

Author

Owner

ok. I have rolled out this workaround to prod for koji, riscv-koji and src (Those were the only ones where I was seeing this).

Can folks test and see if things look 'fixed'?

Yeah, that copr problem seems like it might be similar... I'm not sure if they implemented anything though.

ok. I have rolled out this workaround to prod for koji, riscv-koji and src (Those were the only ones where I was seeing this). Can folks test and see if things look 'fixed'? Yeah, that copr problem seems like it might be similar... I'm not sure if they implemented anything though.

kevin self-assigned this

2026-05-18 03:17:50 +00:00

kevin added this to the Sprint ( 2026-05-04 to 2026-05-18 ) project

2026-05-18 03:17:54 +00:00

kevin added

and removed