dkirwan/infra-ansible
1
0
Fork 0
forked from infra/ansible
Code Pull requests Activity

download: block infinite crawler looping

Browse source 
The /pub/alt/virtio-win directory had in it some 11 year old things.
One was a readme noting that it moved 11 years ago and nothing was
still here. The others were links to .

The scrapers, being as dumb as posts followed all those links over and
over again to the tune of millions per day.
I removed the links, but of course they were still trying, so
lets be a bit more aggressive and just 403 them all.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This commit is contained in:
Kevin Fenzi 2026-02-12 16:54:16 -08:00
commit fefbc356f0
1 changed files with 5 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

5
roles/download/templates/httpd/dl.fedoraproject.org.conf
View file

@ -32,6 +32,11 @@
SSLProtocol {{ ssl_protocols }}
SSLCipherSuite {{ ssl_ciphers }}
RewriteEngine On
RewriteCond %{REQUEST_URI} ^/pub/alt/virtio-win/.*$
RewriteRule .* - [F]
# proxy all requests to anubis after ssl termination
RequestHeader set "X-Real-Ip" expr=%{REMOTE_ADDR}