Some folks were getting posts failing against forge.
See forge/forge#401
So, lets just explicitly allow POSTs through anubis as
this should be fine for normal people and should not be something
that scrapers normally do.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
The backend uses urlretrieve() to download comps.xml from the frontend
at /coprs/<user>/<project>/chroot/<chroot>/comps/. Since this path is
under /coprs/ which is protected by Anubis, the backend receives the
JavaScript challenge page instead of the actual comps data.
This HTML then gets saved as comps.xml and included in the repository
metadata, causing users to download Anubis challenge HTML instead of
the actual comps.xml.gz file.
Anubis was accidentally enabled for all traffic (/) instead of just
the /coprs/ web UI. This caused unnecessary bot challenges for API
clients, dnf/yum, and other automated tools.
Use Anubis BASE_PREFIX to cleanly protect only specific endpoints:
- Frontend: /coprs/ (web UI)
- Dist-git: /{{ cgit_uri }}/ (package browser)
https://anubis.techaro.lol/docs/admin/installation#using-base-prefix
The python productmd library hits kojipkgs to find recent
composes/compose information. It's not easy for it to adjust on the
client end, but it does send a user-agent header like "productmd/1.49"
So, lets just allowlist it for now.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
Checking for the presence of dead.package by trying to download the
dead.package file and checking for an HTTP 404 or 200 is an easy way to
check if a package is retied in distgit.
This practice is used in the wild in tools like fedora-misc-package-utilities.
This was an attempt to block the scrapers, but sadly it breaks humans.
We cannot challenge everything under forks because people need to push
pr's and those get blocked.
So, go back to just weighting those higher...
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
We had this in place when scrapers were hitting libreoffice really hard.
They don't seem to be anymore, so drop it for now.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This was causing problems when we last upgraded anubis because
the refs/summary were getting challenges from anubis for ostree users.
So, lets try and just allow those. They are all small files, so
it shouldn't be much in the way of load.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This copies the excellect copr/anubis role to a more generic top level
one. It allows for using a different env per ipa_host_group (but we
could also just add more cases in there) or a default one if that isn't
found.
It moves the defaults and policies to template so we can include a
static key from ansible-private if we like.
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
