Loading Fedora sites sometimes failed with "Internal Server Error: administrator has misconfigured Anubis." #13192
Labels
No labels
announcement
anubis
authentication
aws
backlog
blocked
bodhi
ci
cloud
communishift
copr
database
day-to-day
dc-move
deprecated
dev
discourse
dns
downloads
easyfix
epel
firmitas
forgejo_migration
Gain
High
Gain
Low
Gain
Medium
gitlab
greenwave
hardware
help wanted
high-trouble
koji
koschei
lists
low-trouble
medium-trouble
mirrorlists
monitoring
Needs investigation
odcs
OpenShift
ops
outage
packager_workflow_blocker
pagure
permissions
Priority
Needs Review
Priority
Next Meeting
Priority
🔥 URGENT 🔥
Priority
Waiting on Assignee
Priority
Waiting on External
Priority
Waiting on Reporter
rabbitmq
release-monitoring
releng
request-for-resources
s390x
security
SMTP
sprint-0
sprint-1
src.fp.o
staging
unfreeze
waiverdb
websites-general
wiki
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
No milestone
No project
No assignees
7 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
infra/tickets#13192
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description of request
When I've tried to load Fedora sites like bodhi, koji, https://lists.fedoraproject.org/archives/list/devel%40lists.fedoraproject.org/ and https://src.fedoraproject.org/ in the last two weeks in Firefox 150.0a1 in a Fedora 44 KDE installation, the pages failed to load about 30% of the time. Errors were shown like
"Oh noes!
Internal Server Error: administrator has misconfigured Anubis. Please contact the administrator and ask them to look for the logs around: fast.
H4sIAAAAAAAA/wTA
UQ7CMAgA0H9PQfZP
Umc6KbeBBOuHYZSq
sbf3faWniS9IMzWG
R1dLhl++QLTD1KcM
hi0+vlbKO7lcW4yz
Thz1ILyX6kiHn0j7
3sJvhaL5dvkHAAD/
/7DWgcVXAAAA
"
A URL when this happened was https://lists.fedoraproject.org/.within.website/x/cmd/anubis/api/pass-challenge?id=019cdb5f-d568-705a-86ab-8229ca308c9a&response=0000430e06b7201cfc07a6a83a6fb889a75c3ddce74cadfd602204cea9036c9e&nonce=111349&redir=https%3A%2F%2Flists.fedoraproject.org%2Farchives%2Flist%2Fdevel%2540lists.fedoraproject.org%2F&elapsedTime=1086
When I loaded the sites again, the pages usually appeared normally.
I assume that this is related to specific proxy and when reloading you are hitting different one. Would be good to know which proxy actually misbehaves like this.
I have seen this very sporadically in the past. I'm not fully sure what causes it, but it's not actually any misconfiguration.
It seems to happen when anubis sends a challenge, but then the browser rejects setting the cookie or doesn't generate the challenge right. Do you have any cookie denying extensions? Does the problem persist on reloading?
Can you look and see if you have a techaro.lol-anubis-auth cookie?
Finally, we were behind one anubis version, I'll look at updating in case we are hitting a bug of some kind thats already been fixed.
I don't have cookie denying extensions, but I have Extended Tracking Protection set to Custom with All cross-site cookies blocked and Private Browsing enabled permanently. The problem doesn't usually persist on reloading. I don't see any cookies under Manage Browsing Data in Settings, but I think that's because I have Private Browsing enabled. First-party cookies should still be accepted. If techaro.lol-anubis-auth is considered a third-party cookie it would probably be blocked. I can try to allow third-party cookies and see if that helps. Thanks.
The problem seemed to happen when the Anubis page showed Calculating... and the progress bar was mostly complete. Sometimes there was a message like Calculations are taking too long...
The problem was less frequent (<10%) when I allowed third-party cookies or selected Isolate cross-site cookies in Enhanced Tracking Protection in Settings. I only saw the error once when allowing third-party cookies. There might be a race condition involving the cookie anubis is using, though I don't know if it's blocked or not. The message I mentioned was Verification is taking longer than expected... Maybe the combination of the verification being slow and the cookie being set or not was involved.
When the problem happened, there were cookies techaro.lol-anubis-cookie-verification and one with a hash for a name, but there wasn't techaro.lol-anubis-auth in Storage in Developer Tools. When the page loaded normally, techaro.lol-anubis-auth, techaro.lol-anubis-cookie-verification and one with a hash for a name were shown. The cookies had the domains like bodhi.fedoraproject.org, so I guess they were first-party cookies. I'm not sure why blocking third-party cookies would affect them.
following up, it started happening again frequently today, i hit this page around 4 times in the last 8 hours
link from browser: https://forge.fedoraproject.org/.within.website/x/cmd/anubis/api/pass-challenge?id=019d77c2-5e40-7591-89ac-b6e56265d062&response=0000a3da60f8a5c013c1b23fbc18b47f700ddc6b27a675d0525c33cdb3babd68&nonce=74954&redir=https%3A%2F%2Fforge.fedoraproject.org%2Finfra%2Ftickets%2Fissues&elapsedTime=1243
@smoliicek wrote in #13192 (comment):
Can confirm that this link really throws this error every time I click on it.
Okay, so I did some digging and this is the error message that Anubis produces:
internal error: store: key not found: "challenge:019d77c2-5e40-7591-89ac-b6e56265d062"(note: the challenge ID changes every time, this is from my request and the link i provided)My only idea is that
User hits proxy -> anubis challenge generated -> browser solves challenge, however sends it to another proxy -> get this error
The error message is also encoded in the error page itself, I wrote a Go program for decoding it, it is published on my people page - https://smoliicek.fedorapeople.org/anubis_decode.go
i was thinking, one of this could be a solution to this problem:
what do you guys think?
I'm not sure how it could jump to another proxy in the middle of a challenge flow. That should all be taking place over the same existing connection I thought. ;(
The only idea I could come up with is that somehow anubis wasn't able to store the solved challenge in memory, but when I looked it seemed like there wasn't particular memory pressure or anything. ;(
Those options all seem untenable to me. ;( Our proxy network is spread out accross the world. Making them all use the same redis backend would make them dependent on the network between them and that backend. If we are going to do that we might as well drop all of them and just have one central place for proxies, which would be a shame.
Making sessions stick to one proxy is also kinda bad, because we have multiple proxies in each region so people can get to one in the event one is down. If they are tied to one and it goes down that would be not too great. Also, that would have to be one cookie per backend right?
Can you confirm this is where more than 1 proxy is involved? ie, set /etc/hosts to point to just one proxy for the service you are testing, and see if you can get the error?
I wonder at this point if we perhaps should look at an upstream issue so they could tell us whats really going on here.
I'm still seeing these errors with a similar frequency. The upstream issue https://github.com/TecharoHQ/anubis/issues/1141 mentioned similar errors "Internal Server Error: administrator has misconfigured Anubis. Please contact the administrator and ask them to look for the logs around: fast." and ""store: key not found: redis: nil" The reporter seemed to conclude that the problem was "The error was DragonflyDB was accepting writes all writes, but not persisting most of them. Some examples as above, were persisted no problem - but often keys were just silently dropped."
I didn't see this problem with other sites using anubis like https://gitlab.freedesktop.org/mesa/mesa or https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/?h=v6.19.13
Alright I seem to be able to reproduce this by doing the following repeatedly until the error shows. It takes about 5-10% of the time:
(Open an incognito window)
1- Go to https://src.fedoraproject.org/rpms/freerdp
2- Click on Builds Status (This will forward to koji window with anubis)
Rinse and repeat.
This is the headers and URL of one of the times I noticed:
cURL request: curl 'https://koji.fedoraproject.org/.within.website/x/cmd/anubis/api/pass-challenge?id=019e04ca-d58b-70f4-814d-d7108efb1e60&response=0000bf30f8ed582f5778fb1710a1774db27c13883be694a64132ef7a5d043d97&nonce=76178&redir=https%3A%2F%2Fkoji.fedoraproject.org%2Fkoji%2Fsearch%3Ftype%3Dpackage%26match%3Dglob%26terms%3Dfreerdp&elapsedTime=276'
-H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8'
-H 'accept-language: en-US,en;q=0.9'
-H 'cache-control: no-cache'
-b 'techaro.lol-anubis-cookie-verification=019e04ca-d58b-70f4-814d-d7108efb1e60'
-H 'pragma: no-cache'
-H 'priority: u=0, i'
-H 'referer: https://koji.fedoraproject.org/koji/search?type=package&match=glob&terms=freerdp'
-H 'sec-ch-ua: "Brave";v="147", "Not.A/Brand";v="8", "Chromium";v="147"'
-H 'sec-ch-ua-mobile: ?0'
-H 'sec-ch-ua-platform: "Linux"'
-H 'sec-fetch-dest: document'
-H 'sec-fetch-mode: navigate'
-H 'sec-fetch-site: same-origin'
-H 'sec-fetch-user: ?1'
-H 'sec-gpc: 1'
-H 'upgrade-insecure-requests: 1'
-H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36'
Response Headers:
apptime: D=685
content-type: text/html; charset=utf-8
date: Thu, 07 May 2026 23:34:40 GMT
server: Apache
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-fedora-proxyserver: proxy10.rdu3.fedoraproject.org
I was referred to this ticket in the infra channel.
I get more and more often locked out of Fedora services from Anubis ("Oh noes!"). My feeling is that the issues occur mostly when I'm at home with 5G Internet (not sure if the latency makes a difference or so?), but sometimes I need to refresh many times and then it works at some time, but sometimes it seems to not work at all. I have comparable issues with some kernel.org pages. I use our default Firefox build. Strict mode (with the recommended "fix major issues" enabled), ublock, noscript (having a page and its linked domains as "always trusted" in noscript does either not always mitigate the issue or has no impact at all). My cookies of fedoraproject.org are permanent.
The difference to the initial case of this ticket: I have no "internal server error". In my case, Anubis refers to me having cookies disabled, which obviously is not true as it works at some time when I keep refreshing (mostly at least ^^). Let me know if I shall open a new ticket about it (I could imagine both errors have a common origin given that it definitely is no cookie problem).
Supplement: I add a screenshot when I experience the issue the next time
I haven't seen this error in the last two weeks or so, so it might've been fixed. @py0xc3 the problem you described sounds different since you didn't see the same error. I'd report your problem in a new issue.
Yeah, I think that makes sense. Different error messages do not always mean it's a different cause (I remained skeptics because there is definitely no cookie issue). But I think I just became able to reproduce my issue. I will open a new ticket, and reference this one to allow the team to consider both possibilities though.
Thanks :)
I now get this. It can be a coincident, but it has a positive correlation with being for the first time on the Internet using a WiFi I have not used before. I re-opened a firefox that had previously pages open when it closed, 3 of them forge tickets. The latter three all ended up with this error, but different base64 values. The result is always the same if I do F5/refresh. But when I start "from scratch" opening forge from a new tab and navigate to the tickets I need, it works. So not critical and it can be mitigated. Just reporting its still there :) Happy to report more if you have questions
The screenshots with the three values:
I got this error here:
https://forge.fedoraproject.org/.within.website/x/cmd/anubis/api/pass-challenge?id=019f8df4-f9c2-745a-b2d3-94299b1267b1&response=0000774e78431e59e2316a72db85f083304a96fe2b785a02b6cc53090f057111&nonce=46892&redir=https%3A%2F%2Fforge.fedoraproject.org%2Fjoin%2FWelcomeToFedora%2Fpulls%2F197&elapsedTime=194