pagure will generate a zip or gz archive of any particular thing on a
fork. scrapers crawl this and generate an archive for every single
tag in a fork.
Since we don't really need these in forks, lets forbid that.
Users that somehow need an archive for a fork can clone locally and
generate one. Non forks we are leaving enabled at this time as they
can/do use this for releases.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
The blame endpoint gets crawled a lot by scrapers and it's very
intensive for pagure. It has to run the blame in the git repo, which
can take a while and requests pile up.
While right now load is fine with it, it's might be causing
sporadic 503's that we are seeing cloing things. If there's a bunch
of blame requests cloging up the queue.
Users can git clone and run a blame locally if they need to.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This adds a setup where we have podman pulling the anubis pod and
running it and then proxying requests through it for weighing.
The pod runs as an unpriv 'anubis' user and uses a systemd service file
to manage it.
Changes then are made in httpd config to terminate ssl and then proxy
via anubis. The flow is:
request -> httpd -> anubis/pod -> httpd (pagure wsgi app).
This is already manually setup on stg.pagure.io and this pr
should not change pagure.io any.
If/when we wish to enable this for pagure.io, we just need to adjust the
0_pagure.conf file and the playbook to include the role.
Note that we will want to use a different approach for proxies/other
applications. There we can use the native fedora anubis package, and
will need to adjust templates to handle the middle proxying.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
For now, we should be able to drop these blocks.
I checked logs and there have been no recent hits to these.
Of course they could become active again, but we can always put it back
in place. There's been requests from folks wanting to submit pr's that
their forks are blocked by these.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
Update the latest project blocks that were added.
Also, extend the bot block to docs.pagure.io, which was
being hammered by scrapers.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
These have to be in "s in order to do a string comparison, since
they were not, they were never matching anything. ;(
Fix them all up, and also block a few more repos on pagure that are
getting heavily crawled.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
Today the pagure started to be unresponsive. After few hours of
debugging we narrowed the issue to worker limit. We tweaked the value to
something that should be more responsive and it helped.
Here is the PR to make the change permanent.
Thanks @gwmngilfen and @nphillip for help with that.
Looked at logs of servers being hit by the 'non-responsive' bots and
the following were hit heavily every day multiple times a day:
100006 nagios.fedoraproject.org-access.log
102150 koschei.fedoraproject.org-access.log
162296 lists.fedoraproject.org-access.log
495776 fedoraproject.org-access.log
850471 dl.fedoraproject.org-access.log
Added bloks to dl.fedoraproject to try and lower its hit rate. Others
need review from people who know their internals more.
Signed-off-by: Stephen Smoogen <ssmoogen@redhat.com>
I'll try this in stg first and then roll to prod if all looks ok.
I don't see any reason why it wouldn't work off hand.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
There's no reason to not just use one letsencrypt cert for stg.pagure.
Also clean up logic in the web config and make sure all the servernames
are handled correctly.
Once this works, will roll this to production.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
