From the logs on the proxies I can see the reason the redirects
for attachments are failing is the certificate chain for
pagure.io is now deeper than it was before:
[Fri Aug 07 23:23:10.874410 2026] [ssl:error] [pid 2283452:tid 2288622] [remote 2620:52:6:1161::32:443] AH02040: Certificate Verification: Certificate Chain too long (chain has 3 certificates, but maximum allowed are only 2)
I don't know why this changed, but I guess it's something to do
with preparation for pagure.io retirement? Anyway, assuming the
change is expected, this fixes the redirects (I tested by hand
patching proxy04 and forcing my box to always use proxy04 for
forge.fp.o, it works fine).
Resolves: forge/forge#703
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is a grab bag of things done today to try and mitigate a massive
scraper hit against src.fedoraproject.org.
- increase memory on riscv-koji (it was getting OOM killed)
- add memory and cpu to pkgs to process more
- increase wsgi procs and threads to allow pkgs to process more
- increase max requests on proxies, since we were hitting the old limits
- block Lightpanda useragent
- tell varnish on proxies to only wait 10s for pkgs connections,
this lets us 503 them sooner and prevents pile ups waiting.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
Add document start marker (---) to 28 YAML files that were missing it
to comply with yamllint requirements for proper YAML document structure.
Signed-off-by: Ryan Lerch <rlerch@redhat.com>
We have been seeing some 502's on forge also, that might be the same
thing we were seeing with src and koji, so lets try and bypass POSTs
here too.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
See infra/tickets#12913
and https://github.com/TecharoHQ/anubis/issues/1624
We are seeing sporadic EOF errors when koji/src/riscv-koji send a 200
reply back through anubis.
Since we just allow POST in anubis anyhow, bypass it entirely for them
to avoid the EOF issue until we can sort it out more.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
Clanker lead me astray here. We can't pass options on the proxypass here
because we are using a balancer, so we need to pass them on the balancer
members.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
I got claude to dig into the things we tried to fix this issue and look
for things we missed. It found a few things:
1. In commit 35a1b3223b Victor Koycheff added some proxyopts for
keepalive, etc. However, unfortunately, while the variable was set
the template wasn't looking at that, so they were never actually
set in the website. ;( So, we fix that by passing the variable in the
right template here.
2. kojihub didn't have a keepalive set on the backend. (but this is
likely cosmetic since the problem is at the proxy layer). We fix
this by adding one anyhow.
3. The pass thru anubis didn't have keepalive set right, so we do
that in the template. This may fix other 502 issues with other
applications also.
Calude used 57,508 tokens looking at all this. ;)
Assisted-By: claude
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This acts as the second half of the fix for the 502 Bad Gateway errors
on long-running koji connections.
Fixes#12913
Signed-off-by: Victor Koycheff <victorkoycheff@gmail.com>
This is the same file used on primary koji. It disallows robots
instead of just asking them to slow their roll like the default
robots.txt does.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This worked in staging, so drop the staging conditional and just apply
it in prod too. It should allow websocket connections via the anubis
proxing.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
Re-enable anubis for openqa-lab01 and then try and pass it a proxy
statement to pass websockets correctly via the proxy.
disclaimer: claude pointed me in this direction.
Possible fix for infra/tickets#13252
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This also relocates the `httpchecks` roles to the right place,
as putting it in it's own role was a bit of hack. Having it in
the proxy role is better.
Signed-off-by: Greg Sutcliffe <fedora@emeraldreverie.org>
This also removes the Nagios config - it'll be looking in the wrong
place. A Zabbix replacement check will follow in the next commit.
Signed-off-by: Greg Sutcliffe <fedora@emeraldreverie.org>
