infra/tickets

Fork 1

Packit 503 responses from Pagure API #13123

New issue

Closed

opened 2026-02-06 12:30:44 +00:00 by lbarczio · 15 comments

lbarczio commented

2026-02-06 12:30:44 +00:00

Packit uses the Pagure API on src.fedoraproject.org for CI/CD automation (setting commit statuses, fetching PR info, posting comments, etc.). With a larger number of dist-git repos being onboarded, we started receiving more HTTP 503 responses that include a Retry-After header. We handle these by respecting the header and retrying, so it's mostly not breaking the UX. We are not entirely sure if this is related to rate limiting for our user or general service load, but since we plan to switch to default for CI soon (currently there are around 2000 packages opted in, we plan to switch to all on Feb 16), we would like to understand this better before scaling up. If this is rate limiting, we're wondering if there's a possibility to configure higher limits for the packit user, or if there are any other recommendations that could help. Thanks for any guidance on this!

[Packit](https://packit.dev/) uses the Pagure API on src.fedoraproject.org for CI/CD automation (setting commit statuses, fetching PR info, posting comments, etc.). With a larger number of dist-git repos being onboarded, we started receiving more HTTP 503 responses that include a Retry-After header. We handle these by respecting the header and retrying, so it's mostly not breaking the UX. We are not entirely sure if this is related to rate limiting for our user or general service load, but since we plan to switch to default for CI soon (currently there are around 2000 packages opted in, we plan to switch to all on Feb 16), we would like to understand this better before scaling up. If this is rate limiting, we're wondering if there's a possibility to configure higher limits for the [packit user](https://src.fedoraproject.org/user/packit), or if there are any other recommendations that could help. Thanks for any guidance on this!

zlopez added the

Priority

High

day-to-day

labels

2026-02-09 09:56:30 +00:00

kevin commented

2026-02-13 01:10:29 +00:00

Owner

It's not deliberate rate limiting that I know of.

Is there a way for us to look for these requests in logs? Whats the user-agent that it's using and/or ip address/network it's coming from?

It's not deliberate rate limiting that I know of. Is there a way for us to look for these requests in logs? Whats the user-agent that it's using and/or ip address/network it's coming from?

lbarczio commented

2026-02-13 07:41:39 +00:00

Author

It's not deliberate rate limiting that I know of.

Thanks for confirming.

Is there a way for us to look for these requests in logs? Whats the user-agent that it's using and/or ip address/network it's coming from?

User agent is packit.dev/packit-prod (prod+packit@packit.dev). I can see one of the latest exception on Feb 12, 2026 6:02:37 PM UTC.

> It's not deliberate rate limiting that I know of. Thanks for confirming. > Is there a way for us to look for these requests in logs? Whats the user-agent that it's using and/or ip address/network it's coming from? User agent is `packit.dev/packit-prod (prod+packit@packit.dev)`. I can see one of the latest exception on `Feb 12, 2026 6:02:37 PM UTC`.

kevin commented

2026-02-13 18:19:51 +00:00

Owner

I'm not finding much in logs. ;(

Here's the ones I see for the 12th:

ipaddress - - [12/Feb/2026:14:57:16 +0000] "POST /api/0/rpms/python-specfile/pull-request/723/flag HTTP/1.1" 503 286 "-" "packit.dev/packit-prod (prod+packit@packit.dev)"
ipaddress - - [12/Feb/2026:15:31:58 +0000] "POST /api/0/rpms/python-specfile/pull-request/732/flag HTTP/1.1" 503 286 "-" "packit.dev/packit-stg (stg+packit@packit.dev)"
ipaddress - - [12/Feb/2026:17:42:23 +0000] "POST /api/0/rpms/packit/pull-request/1620/flag HTTP/1.1" 503 284 "-" "packit.dev/packit-prod (prod+packit@packit.dev)"
ipaddress - - [12/Feb/2026:17:53:53 +0000] "GET /api/0/rpms/python-cliff/tree/rawhide HTTP/1.1" 503 286 "-" "packit.dev/packit-prod (prod+packit@packit.dev)"
18.213.120.162 - - [12/Feb/2026:19:10:09 +0000] "GET /api/0/rpms/rust-ctrlc/git/branches HTTP/1.1" 503 286 "-" "packit.dev/packit-prod (prod+packit@packit.dev)"

Some of these are POST's and to fix a forge issue I just allowed them globally. So, perhaps things are better on those?
Not sure on the GET ones, will try and find out more.

If you have more/details on more that would help.

I'm not finding much in logs. ;( Here's the ones I see for the 12th: ``` ipaddress - - [12/Feb/2026:14:57:16 +0000] "POST /api/0/rpms/python-specfile/pull-request/723/flag HTTP/1.1" 503 286 "-" "packit.dev/packit-prod (prod+packit@packit.dev)" ipaddress - - [12/Feb/2026:15:31:58 +0000] "POST /api/0/rpms/python-specfile/pull-request/732/flag HTTP/1.1" 503 286 "-" "packit.dev/packit-stg (stg+packit@packit.dev)" ipaddress - - [12/Feb/2026:17:42:23 +0000] "POST /api/0/rpms/packit/pull-request/1620/flag HTTP/1.1" 503 284 "-" "packit.dev/packit-prod (prod+packit@packit.dev)" ipaddress - - [12/Feb/2026:17:53:53 +0000] "GET /api/0/rpms/python-cliff/tree/rawhide HTTP/1.1" 503 286 "-" "packit.dev/packit-prod (prod+packit@packit.dev)" 18.213.120.162 - - [12/Feb/2026:19:10:09 +0000] "GET /api/0/rpms/rust-ctrlc/git/branches HTTP/1.1" 503 286 "-" "packit.dev/packit-prod (prod+packit@packit.dev)" ``` Some of these are POST's and to fix a forge issue I just allowed them globally. So, perhaps things are better on those? Not sure on the GET ones, will try and find out more. If you have more/details on more that would help.

kevin added the

points

label

2026-02-13 18:20:02 +00:00

kevin added this to the Sprint ( 2026-02-09 to 2026-02-22 ) project

2026-02-13 18:20:05 +00:00

kevin self-assigned this

2026-02-13 18:20:07 +00:00

lbarczio commented

2026-02-17 12:14:34 +00:00

Author

Thanks you looking into it. More of the events from our Sentry:

| 2026-02-17T01:39:59.052Z | http | httplib | error |  | {"http.fragment":"","http.method":"POST","http.query":"","http.response.status_code":503,"reason":"Service Unavailable","thread.id":"140517256861248","url":"https://src.fedoraproject.org/api/0/rpms/python-mkdocs-click/pull-request/4/flag"} |
| 2026-02-17T01:39:59.053Z | info | message | error | Rate limit hit (status 503). Waiting 5s until reset (Retry-After header) |  |

| 2026-02-17T00:13:39.289Z | http | httplib | error |  | {"http.fragment":"","http.method":"GET","http.query":"","http.response.status_code":503,"reason":"Service Unavailable","thread.id":"140219705366080","url":"https://src.fedoraproject.org/api/0/rpms/rust-toml0.9/tree/rawhide"} |
| 2026-02-17T00:13:39.290Z | info | message | error | Rate limit hit (status 503). Waiting 5s until reset (Retry-After header) |  |

| 2026-02-16T15:45:41.740Z | http | httplib | error |  | {"http.fragment":"","http.method":"GET","http.query":"","http.response.status_code":503,"reason":"Service Unavailable","thread.id":"139792385398336","url":"https://src.fedoraproject.org/api/0/rpms/spirv-tools/tree/rawhide"} |
| 2026-02-16T15:45:41.741Z | info | message | error | Rate limit hit (status 503). Waiting 5s until reset (Retry-After header) |  |

| 2026-02-16T11:33:02.726Z | http | httplib | error |  | {"http.fragment":"","http.method":"POST","http.query":"","http.response.status_code":503,"reason":"Service Unavailable","thread.id":"139683746367040","thread.name":"MainThread","url":"https://src.fedoraproject.org/api/0/rpms/license-validate/pull-request/10"} |
| 2026-02-16T11:33:02.727Z | info | message | error | Rate limit hit (status 503). Waiting 5s until reset (Retry-After header) |  |

| 2026-02-16T07:19:08.057Z | http | httplib | error |  | {"http.fragment":"","http.method":"POST","http.query":"","http.response.status_code":503,"reason":"Service Unavailable","thread.id":"140378068203072","url":"https://src.fedoraproject.org/api/0/rpms/catch/pull-request/6/flag"} |
| 2026-02-16T07:19:08.057Z | info | message | error | Rate limit hit (status 503). Waiting 5s until reset (Retry-After header) |  |

| 2026-02-14T18:56:08.617Z | http | httplib | error |  | {"http.fragment":"","http.method":"GET","http.query":"","http.response.status_code":503,"reason":"Service Unavailable","thread.id":"139881209251392","url":"https://src.fedoraproject.org/api/0/rpms/pimcommon/git/branches"} |
| 2026-02-14T18:56:08.618Z | info | message | error | Rate limit hit (status 503). Waiting 5s until reset (Retry-After header) |  |

| 2026-02-14T10:52:02.792Z | http | httplib | error |  | {"http.fragment":"","http.method":"GET","http.query":"","http.response.status_code":503,"reason":"Service Unavailable","thread.id":"140700861584960","url":"https://src.fedoraproject.org/api/0/rpms/kf6-kimageformats/tree/rawhide"} |
| 2026-02-14T10:52:02.793Z | info | message | error | Rate limit hit (status 503). Waiting 5s until reset (Retry-After header) |  |

Do you need some other data?

Thanks you looking into it. More of the events from our Sentry: ``` | 2026-02-17T01:39:59.052Z | http | httplib | error | | {"http.fragment":"","http.method":"POST","http.query":"","http.response.status_code":503,"reason":"Service Unavailable","thread.id":"140517256861248","url":"https://src.fedoraproject.org/api/0/rpms/python-mkdocs-click/pull-request/4/flag"} | | 2026-02-17T01:39:59.053Z | info | message | error | Rate limit hit (status 503). Waiting 5s until reset (Retry-After header) | | ``` ``` | 2026-02-17T00:13:39.289Z | http | httplib | error | | {"http.fragment":"","http.method":"GET","http.query":"","http.response.status_code":503,"reason":"Service Unavailable","thread.id":"140219705366080","url":"https://src.fedoraproject.org/api/0/rpms/rust-toml0.9/tree/rawhide"} | | 2026-02-17T00:13:39.290Z | info | message | error | Rate limit hit (status 503). Waiting 5s until reset (Retry-After header) | | ``` ``` | 2026-02-16T15:45:41.740Z | http | httplib | error | | {"http.fragment":"","http.method":"GET","http.query":"","http.response.status_code":503,"reason":"Service Unavailable","thread.id":"139792385398336","url":"https://src.fedoraproject.org/api/0/rpms/spirv-tools/tree/rawhide"} | | 2026-02-16T15:45:41.741Z | info | message | error | Rate limit hit (status 503). Waiting 5s until reset (Retry-After header) | | ``` ``` | 2026-02-16T11:33:02.726Z | http | httplib | error | | {"http.fragment":"","http.method":"POST","http.query":"","http.response.status_code":503,"reason":"Service Unavailable","thread.id":"139683746367040","thread.name":"MainThread","url":"https://src.fedoraproject.org/api/0/rpms/license-validate/pull-request/10"} | | 2026-02-16T11:33:02.727Z | info | message | error | Rate limit hit (status 503). Waiting 5s until reset (Retry-After header) | | ``` ``` | 2026-02-16T07:19:08.057Z | http | httplib | error | | {"http.fragment":"","http.method":"POST","http.query":"","http.response.status_code":503,"reason":"Service Unavailable","thread.id":"140378068203072","url":"https://src.fedoraproject.org/api/0/rpms/catch/pull-request/6/flag"} | | 2026-02-16T07:19:08.057Z | info | message | error | Rate limit hit (status 503). Waiting 5s until reset (Retry-After header) | | ``` ``` | 2026-02-14T18:56:08.617Z | http | httplib | error | | {"http.fragment":"","http.method":"GET","http.query":"","http.response.status_code":503,"reason":"Service Unavailable","thread.id":"139881209251392","url":"https://src.fedoraproject.org/api/0/rpms/pimcommon/git/branches"} | | 2026-02-14T18:56:08.618Z | info | message | error | Rate limit hit (status 503). Waiting 5s until reset (Retry-After header) | | ``` ``` | 2026-02-14T10:52:02.792Z | http | httplib | error | | {"http.fragment":"","http.method":"GET","http.query":"","http.response.status_code":503,"reason":"Service Unavailable","thread.id":"140700861584960","url":"https://src.fedoraproject.org/api/0/rpms/kf6-kimageformats/tree/rawhide"} | | 2026-02-14T10:52:02.793Z | info | message | error | Rate limit hit (status 503). Waiting 5s until reset (Retry-After header) | | ``` Do you need some other data?

kevin commented

2026-02-17 23:03:47 +00:00

Owner

Ah ha. That does help.

So, I think this is our varnish layer... (at least all the Retries).

not yet sure why, but looking...

Ah ha. That does help. So, I think this is our varnish layer... (at least all the Retries). not yet sure why, but looking...

kevin commented

2026-02-18 23:03:06 +00:00

Owner

So, I am not sure, but think this might be a varnish bug with http/2...

is there any way you could try just using http1.1 to see if it does or doesn't show the problem?

So, I am not sure, but think this might be a varnish bug with http/2... is there any way you could try just using http1.1 to see if it does or doesn't show the problem?

kevin modified the project from Sprint ( 2026-02-09 to 2026-02-22 ) to Sprint ( 2026-02-23 to 2026-03-08 )

2026-02-22 22:37:18 +00:00

lbarczio commented

2026-02-23 07:07:59 +00:00

Author

I am going to look into that and discuss this change with the team, and will get back to you. Thanks!

lbarczio commented

2026-02-24 07:40:05 +00:00

Author

Hi! Checked this and we are using python's request library for requests to Pagure, so it's already http/1.1.

Hi! Checked this and we are using python's `request` library for requests to Pagure, so it's already http/1.1.

kevin commented

2026-02-25 19:05:56 +00:00

Owner

ok. Humf. I only was seeing http/2 issues... but clearly I missed something.

So, when you retry after 5 seconds it usually works? Or still persists?
Also, do you ever stop retrying? or if it keeps sending you the 503 you keep retrying every 5s?

As far as I could tell this is in the varnish talking to haproxy and getting a 503, but... haproxy isnt logging the backend as down, so the 503 has to be coming from the backend itself and recovers fast enough to avoid the health check on haproxy. If thats true, I would expect most of your retries to work on the retry?

We could look at removing haproxy from the picture here, since there's only one backend and we could talk to it directly via varnish. That would rule out this being a haproxy issue somehow. Of course we are in beta freeze so thats a pretty invasive change and would need a freeze break.

ok. Humf. I only was seeing http/2 issues... but clearly I missed something. So, when you retry after 5 seconds it usually works? Or still persists? Also, do you ever stop retrying? or if it keeps sending you the 503 you keep retrying every 5s? As far as I could tell this is in the varnish talking to haproxy and getting a 503, but... haproxy isnt logging the backend as down, so the 503 has to be coming from the backend itself and recovers fast enough to avoid the health check on haproxy. If thats true, I would expect most of your retries to work on the retry? We could look at removing haproxy from the picture here, since there's only one backend and we could talk to it directly via varnish. That would rule out this being a haproxy issue somehow. Of course we are in beta freeze so thats a pretty invasive change and would need a freeze break.

lbarczio commented

2026-02-27 15:20:04 +00:00

Author

I checked few of the logs from recent days and the code to get the details:

it seems like retry after 5 seconds usually works - it worked every time in our logs in the recent days
we do stop retrying eventually (API library has 3 retries at the HTTP level, plus Celery task-level retries with limits of 2-5 depending on the operation)

So what you are writing about varnish→haproxy→backend would align what we're seeing - the errors are transient and recover very quickly.

Currently, this seems to not be impacting our users since the retries handle it transparently. We would continue to monitor this more closely going forward and f we start seeing increased frequency or cases where multiple retries fail, we can revisit and potentially try the fix you mentioned (removing haproxy from the picture). But given the current low impact and the beta freeze, there's no urgency from our side to push for changes right now. Thanks for the investigation and guidance!

I checked few of the logs from recent days and the code to get the details: - it seems like retry after 5 seconds usually works - it worked every time in our logs in the recent days - we do stop retrying eventually (API library has 3 retries at the HTTP level, plus Celery task-level retries with limits of 2-5 depending on the operation) So what you are writing about varnish→haproxy→backend would align what we're seeing - the errors are transient and recover very quickly. Currently, this seems to not be impacting our users since the retries handle it transparently. We would continue to monitor this more closely going forward and f we start seeing increased frequency or cases where multiple retries fail, we can revisit and potentially try the fix you mentioned (removing haproxy from the picture). But given the current low impact and the beta freeze, there's no urgency from our side to push for changes right now. Thanks for the investigation and guidance!

kevin commented

2026-02-27 18:29:06 +00:00

Owner

Yeah, so after beta freeze I can try and remove haproxy from the path and see if that helps any...

if not we can look more at the varnish -> backend part, as this must be happening in there.

Yeah, so after beta freeze I can try and remove haproxy from the path and see if that helps any... if not we can look more at the varnish -> backend part, as this must be happening in there.

kevin removed this from the Sprint ( 2026-02-23 to 2026-03-08 ) project

2026-03-09 14:53:01 +00:00

kevin referenced this issue from a commit

2026-03-12 00:23:02 +00:00

proxies / src.fp.o: drop haproxy from src

kevin referenced this issue from infra/ansible

2026-03-12 00:23:14 +00:00

proxies / src.fp.o: drop haproxy from src #3211

kevin commented

2026-03-12 00:36:19 +00:00

Owner

PR for dropping haproxy from the src.fp.o path: infra/ansible#3211