This is a grab bag of things done today to try and mitigate a massive
scraper hit against src.fedoraproject.org.
- increase memory on riscv-koji (it was getting OOM killed)
- add memory and cpu to pkgs to process more
- increase wsgi procs and threads to allow pkgs to process more
- increase max requests on proxies, since we were hitting the old limits
- block Lightpanda useragent
- tell varnish on proxies to only wait 10s for pkgs connections,
this lets us 503 them sooner and prevents pile ups waiting.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
Add document start marker (---) to 28 YAML files that were missing it
to comply with yamllint requirements for proper YAML document structure.
Signed-off-by: Ryan Lerch <rlerch@redhat.com>
Right now requests go:
client -> httpd on proxy -> anubis -> httpd on proxy -> varnish -> haproxy -> pkgs01
but haproxy is pretty useless in this case.
There is only one backend (pkgs01) so no load balancing, and doing a
liveness check is also pointless because if its down the request will
fail anyhow.
It might be tha haproxy ovehead is causing varnish to return retries
sometimes ( infra/tickets#13123 )
So, this drops it out for this.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
I can't recall why we disabled this at the time, but it's been disabled
for 11 years, so the likelyhood of us needing it now is low.
So, lets just delete it.
See https://pagure.io/fedora-infrastructure/issue/13002 for discussion.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
Moved to openshift in 2022 (see e.g. ab4db44) and was dropped from
the haproxy config in 55056c6, nothing listens on 10022 on the
proxies these days.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
See cb22afd (removed ask config in 2018). It's been gone since
then. Nothing listens on 10021 on the proxies.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
See 5413a21 (noting it was moved to openshift in 2020) and
55056c6 (removed it from haproxy config).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Was moved to openshift long ago. All of these URLs are now just
302s to bodhi.fedoraproject.org in the httpd config. Nothing
listens on port 10009 on the proxies.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I'm pretty sure this is a vestige of FAS2. 'fas01' resolves on
*most* proxies, but to an IP address that doesn't exist. On
proxy01.stg (but not proxy02.stg for some reason?) it does not
resolve at all, and this causes varnish to fail to start up.
Let's just get rid of it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
These fonts are kind of hit a lot, no need to send to the pkgs01 backend
for them, we can just cache them in varnish for clients.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This seems to cause a problem for some super large archives,
as seen in https://pagure.io/fedora-infrastructure/issue/12930
So, lets just tell varnish not to cache these sorts of archive
files and just pass them through.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
varnish won't cache things with cookies (as they might be dependent on
those). So, lets drop cookies for static files that dont care about
that.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
The scrapers are now downloading js and css files at a really high rate,
and thats causing a lot of load on pkgs01.
So, lets see if we can just move src behind the varnish on proxies.
This should allow it to return those pretty static files a lot
faster and not cause load on the backend.
However, putting varnish in the path might mess up something, so
lets just test in staging first.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
Currently we are not caching anything under /compose/ in varnish.
This was desired in case there was some change that got cached when
there were updates.
However, we are hitting an issue currently where clients are requesting
large amounts of "/compose/iot/repo/objects/ABC/" directory listings.
These directories have nearly 65k entries in them and when apache has
to stat 65k files to get attributes it takes a long time.
Sometimes they seem to just get stuck because there are so many.
Caching this tree should allow varnish to cache things and prevent
high load issues.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This was done in trying to get caching more stable for s390x builds.
Pushing it now to match the deployed state.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This was done last week to help out with builds that were failing
due to s390x cache issues. Moving to memory based seems to have
helped this stop happening, possibly along with other changes.
I'm pushing it now because it reflects the current state
after outages last week.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This will unify all the handlers to use first uppercase letter for
ansible-lint to stop complaining.
I went through all `notify:` occurrences and fixed them by running
```
set TEXT "text_to_replace"; set REPLACEMENT "replacement_text"; git grep
-rlz "$TEXT" . | xargs -0 sed -i "s/$TEXT/$REPLACEMENT/g"
```
Then I went through all the changes and removed the ones that wasn't
expected to be changed.
Fixes https://pagure.io/fedora-infrastructure/issue/12391
Signed-off-by: Michal Konecny <mkonecny@redhat.com>
fix 1900 failures of the following case issue:
`name[casing]: All names should start with an uppercase letter.`
Signed-off-by: Ryan Lerch <rlerch@redhat.com>
Using `git grep el6` and `git grep el7` and variants like EL-7 or
el-7, I found various entries and files which were no longer needed
with the current ansible. I updated text or tests to later versions of
RHEL as needed.
found entries for the fedora ami's for the original cloud and removed
those entries also.
Signed-off-by: Stephen Smoogen <ssmoogen@redhat.com>
New mailman staging deployment is giving 302 as a default response instead of
301. Let's reflect that in varnish config.
Signed-off-by: Michal Konecny <mkonecny@redhat.com>
We can cache any other file in the repo, even the repository metadata.
But repomd.xml is the smallest part (glue) that needs to be always
up2date to correctly reference the actual repofiles.
Fixes: https://pagure.io/fedora-infrastructure/issue/10531
