Anubis is discarding actions in Forge, force-reloading pages, losing people's work #401
Labels
No labels
Backlog Status
Needs Review
Backlog Status
Ready
Chore
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
No milestone
No project
No assignees
5 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
forge/forge#401
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
I saw this first on Feb 6th, and I still see it now (Feb 11th). Every few minutes working with Forge, one of my actions get discarded, and the whole page reloads. For example, a new comment is not added (is thrown away), an edited comment is not saved (is thrown away), a new label is not added, a + button to add a PR comment to a certain line doesn't do anything, etc. Usually the whole page turns yellow for a fraction of a second, before it reloads (and loses my work). Or in some cases, where it doesn't reload, some parts turn yellow and the action doesn't work. After a page reload, I see Anubis quickly flash and then Forge works OK for some time (2-5 minutes?) before this happens again, and again.
Today I found Anubis red-handed. When trying to preview a comment, this is what I received:
Please note that I captured the whole page. I couldn't scroll up any more than this. Anubis replaces some of the Javascript response the page made, and of course the result is undefined - sometimes it reloads, sometimes it re-renders parts of the page (like this example), sometimes it shows "JSON parse error" bars at the top, like this:
This is Firefox's error console:
After this happened to me, I see it again a few minutes later when editing a comment, and then I saw it four times during reporting this very ticket. I couldn't attach a screenshot, I received errors. I opened Forge in another tab, saw Anubis, and then I could attach screenshots here. A few minutes later (still typing this), it happened again for another screenshot (this time I could upload it, but couldn't retrieve it's URL). And a few minutes later, I almost lost this text by previewing it. And when submitting this ticket, I again couldn't (had to open Forge in a new tab to let Anubis run).
So basically, every few minutes, Anubis starts replacing any Javascript calls from an open Forge page, and injecting itself into them instead. Instead of just affecting Forge page loads, it also affects any asynchronous communication that Forge does, breaking everything.
It's extremely hard to work with Forge for me at this moment, and I often lose work.
This may be better as an infrastructure ticket? As anubis is in front of forge and forge has not much control here. ;(
I have indeed seen reports of and seen myself anubis being much more aggressive. It seems to have started this week, but we made no changes I can think of that would affect things. ;(
I will try and figure out whats causing it.
Have you seen this on any other sites? Or only forge?
Indeed, I've definitely been getting more Anubis prompts over the past week or so.
In case it helps:
I have firefox-147.0.2-1.fc42.x86_64
I do have adblock.
I also have Consent-O-Matic enabled.
would it be possible to run with the network inspector up when you start seeing lots of these?
It would be nice to see what proxy you are hitting, and help me isolate it.
I dug through logs but didn't really find much that looked likely. ;(
Also, if you all would like to DM me your ip address I can try and look at logs for those around the approximate times you were seeing issues?
I have just encountered a very similar issue. While writing a comment on a ticket, the Anubis action caused me to lose nearly twenty minutes of work by discarding the text I had entered.
I was not reloading the page or navigating away intentionally. I simply clicked on “Comment”, after which the browser displayed a prompt asking whether I wanted to leave the page. I confirmed the prompt, and the entire comment content was lost.
This behavior is problematic, as it leads to silent data loss during normal interaction with the site. Using an external editor as a workaround is not reliable either, since pasted content can be discarded again at any time due to the same mechanism.
My IP address is not public, but I am connecting from the Red Hat Brno office. I believe traffic from these networks should not be subject to blocking or aggressive protection mechanisms.
@kevin wrote in #401 (comment):
Sorry :( We can't move it, it seems. Should I file it again?
Only Forge.
I reproduced it on staging Forge. There should be much less traffic, so hopefully you can better search the log in there?
I was testing with ticket https://forge.stg.fedoraproject.org/quality/test/issues/1 . I typed some text into a new comment, and waited, until this appeared as yellow text in Firefox's network console:
That is the indication that Anubis broke Forge, I believe. Then I hit the Preview button next to me new comment. The first time I was unlucky and the whole page refreshed. The second time I was luckier and Anubis replaced just the preview window (similar to the screenshot above). The request was going to https://forge.stg.fedoraproject.org/quality/test/markup , but instead of a proper response, Anubis content was returned. I believe this was the first improper response that was shown by Firefox's network inspector (but something else in the background must've been affected already, so that the error shown above was printed first). I saved the request as HAR (which seems to be a json dump of it), and also all requests as HAR. It's here:
anubis har
(the smaller file is the one request, the bigger file are all requests)
Does that help? Should I do something else with the network inspector?
Private IP at home, unfortunately, or RH Brno office IP when in the office (today).
@kparal wrote in #401 (comment):
Not to hijack this too much, but I reckon this is occurring on other services as well, most likely Pagure. Ideally the fix should be a bit more generic.
At the very least why is Anubis not disabled when there is an authenticated user?
@kparal wrote in #401 (comment):
I suppose it might be more visible there... so sure.
Well, I was hoping you could look at the headers and get the proxy that you were hitting. That would allow me to look at the logs on just that proxy instead of trying to find it in the 30+ of them. ;(
ok.
@lecris wrote in #401 (comment):
Well, pagure is a completely different instance and it's own different anubis (container vs package).
So, thats why I was asking if anyone saw it on other sites.
Are you just suspecting it happens there? or you have seen it there?
This was as far as I could tell 100% impossible with pagure. It always gives a cookie and the only way you can tell someone logged in vs not is to parse that cookie and see if it's a valid auth. anubis cannot do that.
That said, it looks like forge might only set a cookie when a user is authenticated... so we may be able to do this here.
Alas, forgejo seems to always set that cookie (like pagure does), so I am not sure how anubis can tell if someone is logged in or not. I suppose we could test for the cookie anyhow, with the idea that scrapers won't persist cookies...
In any case, it sounds like the issues here are:
So, in staging I have set anubis to allow any POST, which I think is whats getting caught here.
Can you all see if you can see it anymore in staging? If it looks better there I can roll this out to prod too.
I went ahead and rolled this out to prod too since I didn't think it could break anything...
Hi, I tested it today again in https://forge.stg.fedoraproject.org/quality/test/issues/1 , and it seems to be working well! 🎉 No Anubis interfering with activities on that page, it works as expected. Thanks!
Let's keep this open for a short while to gather more feedback.
Awesome. Glad to hear it.