Remove remnants of ABRT #503
Labels
No labels
a11y
btrfs
Closed As
Can't fix
Closed As
Deferred to upstream
Closed As
Fixed
Closed As
Won't fix
default-apps
easyfix
experience
help-wanted
installation
meeting
meeting-request
nvidia
packaging
pending-action
qa
testing
user-docs
wg-docs
wg-meta
No milestone
No project
No assignees
9 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
workstation/tickets#503
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
In #130 we removed the ABRT GUI, but left the non-GUI portions -- the automatic bug reporting framework -- intact. I thought this was a good plan because it allowed us to keep most of ABRT working exactly as before: we'd only lose the ability to manually report bugs to Bugzilla, which is the part that needed the most work.
Unfortunately, Kamil has pointed out that automatic reporting to the FAF server is nonfunctional because the FAF server has not been updated to handle Fedora 43. So... that's bad. ABRT is really not doing anything anymore. Additionally, when I powered off Fedora just now, I noticed that shutting down hung for 45 seconds because abrtd.service was nonresponsive. It's the second time I've noticed this recently. So keeping ABRT around is harmful. Safe to say that my plan to keep ABRT backend and remove only the GUI has already failed.
Proposal: remove the non-GUI portions of ABRT as well.
@catanzaro, it's not doing nothing. Reporting a crash via
abrt reportremains incredibly effective, even without the FAF server available whatsoever. 1To elaborate, if I were to report SIGSEGV manually, I would not even bother with RHBZ, and I'd likely generally not bother with upstream, for most. With
abrt report, I can upload a stack trace ofthread apply all bt full, and an assembly dump at the end (for which I've no idea how to complete ingdb), alongside my entirejournalctl, and myriad additional environment information, in a minute, to RHBZ.Consequently, remove the FAF packages, if you've no desire to merely upgrade the FAF server. If you do, you'll need to retire the FAF server, however, which shall be quite a significant reduction in analytical capabilities, merely so that the versions are aligned.
Regardless, do not remove
^abrt.*:I have filed a plea at
discussion.fedoraproject.org/t/180667to gain more eyes on this. If nothing comes of it, I suppose that no maintainer exists.However, I shall reiterate that that does not render
abrtuseless, yet; solely FAF, and that is if the extent of its deterioration is not merely a version mismatch.discussion.fedoraproject.org/t/176353/4↩︎@catanzaro wrote in #503 (comment):
I'm not sure about his. I only said that FAF doesn't seem to be aware of F43. Maybe the auto-reports are still being processed correctly, and will be categorized once F43 release info is added... I just don't know.
I think there's still value in having at least CLI ABRT around:
/var/spool/abrt.For completeness, we need to say that
coredumpctlalready captures the crashes as well (but just coredumps, not all the surrounding info like package versions, logs, etc) and can process them usingcoredumpctl debug. It is even more work and more expertise required, though.@kparal,
bugs.kde.org/show_bug.cgi?id=511524#c5shall provide this for all CoreDumpD reports, for KDE Plasma users in F44. However, it does not include any of the environment information thatabrtcaptures, nor does it assist when a bug is triaged as downstream, whichabrtwould provide.Additionally, it cannot retroactively access crashes that require superuser authentication, so it cannot report them, if the user misses the relevant notification, when it occurs (if it creates one for such crashes). 1 Users must manually report these.
It's also more easily breakable than
abrt, perdiscuss.kde.org/t/44078.discuss.kde.org/t/44476↩︎If somebody updates FAF server to handle Fedora 43, then I will close this ticket. I opened this ticket on the premise that the FAF server is completely abandoned. If somebody wants to keep it alive, then great, because the automatic crash reporting functionality is extremely valuable.
OK, but if you're going to open up a terminal and run
abrt reporton the command line, then you probably also know how to install abrt, right? Usingabrt reportis much harder than installing abrt, actually, because it requires a Red Hat Bugzilla API key.It only makes sense to have it installed by default if it actually does something useful without the need for command line incantations.
@kparal wrote in #503 (comment):
If you think this is substantially more useful than coredumpctl, then feel free to install abrt?
Installing abrt is easier than reporting to Bugzilla via GUI, and certainly easier than reporting to Bugzilla using CLI!
Surely the ABRT daemon is a dependency of ABRT GUI. Although if we are talking about Silverblue, then yes this would be relevant.
I agree that the notifications are valuable, but I'm afraid the notifications are provided by the GUI component.
It looks like notifications are handled by abrt-applet (abrt-gui rpm; https://github.com/abrt/abrt/tree/master/src/applet) and GUI app by gnome-abrt (gnome-abrt rpm; https://github.com/abrt/gnome-abrt). Currently you cannot install and use abrt-applet rpm package without gnome-abrt one. Maybe there is a low-cost way to the decouple them?
Inside abrt-applet there are checks for "g_gnome_abrt_available" variable. Maybe only packaging change is required to decouple them?
@catanzaro wrote in #503 (comment):
All of your responses assume that you know how to repeat the crash. I'm trying to see it as a QA person helping a community person in a discussion forum. If the crash is already captured (i.e. at least abrt daemons are running), then we can navigate the person to retrieve valuable information (by looking at the dirs/using abrt cli/installing abrt gui/etc). But if no abrt daemons are running by default, then we either need to guide the person through coredumpctl + gdb (not easy, and still doesn't contain all the info that abrt captures), or tell them "sorry, install abrt and hope that it happens again".
@catanzaro, have you tried this? I ask because you may be surprised how inaccurate that is:
When I last attempted to diagnose
discussion.fedoraproject.org/t/159313/2, installingabrton Plasma Mobile required > 40 minutes of effort (if I was ever successful). 1 Unfortunately, I don't recall why, although I considered myself to be fairly familiar withabrtat the time, and was certainly more competent than the computer-illiterate shall be. I presume that some identities may lack relevant services, or that they are not set toenabledupon installation.Thereafter, configuring an RHBZ key (via
gnome-abrt, at least) is trivial, for it provides the URI directly to the page, whereupon copying and pasting the key is trivial. Even if not, enough discussions exist at Fedora's Discourse instance that cite the relevant URI.@kparal, I agree. Probably 70 % of the crashes I record are unique, yet most are triaged and remediated with upstream changes, so I know that they're not useless. Additionally, my brother would not report a single one if he had to reproduce them.
Tangentially, one of the phenomenal aspects of the FAF server is that even without
abrtable to report, it still submits traces. You can see, ingithub.com/terrapkg/packages/issues/9732, how it submits them even for a crash of a non-Fedora package!Konqi may provide this for KDE Developers, at their Sentry instance, but FAF is more akin to Mozilla's bug reporter, in that the user can observe whether they are alone in experiencing this.
bugs.kde.org/show_bug.cgi?id=504184#c1↩︎Honestly, I don't see much point in this debate. I'm fine with keeping abrt command line so long as somebody updates the FAF server. If nobody is willing to do that, then surely it is time to move on.
Fortunately, abrt only has to be installed once. It won't remove itself. If the crash only happens once and never again, then problem solved! If you are guiding users through how to use abrt command line to report a crash to Bugzilla, you can definitely include installation instructions for abrt.
The instructions for getting a stack trace without abrt are basically:
I'm pretty sure that's easier than figuring out how to work abrt. The main benefit was debuginfo installation, but nowadays debuginfod fortunately handles that.
sudo dnf install abrt?systemctl enable abrtd.service && systemctl start abrtd.service?@catanzaro wrote in #503 (comment):
I'm not sure why you continue associating these two things with each other?
Using abrt to process crashes / generating backtraces from coredumps locally and reporting bugs works fine. The only thing that appears to be (partially) broken is submitting coredumps to the retrace server to generate a backtrace, and I'm not even sure whether it's "broken" or whether things actually work but the web UI just doesn't allow filtering for them.
The retrace server functionality is an extremely valuable reason to have the package installed by default... if it were working.
"Command line tool is potentially useful" is not a valuable reason to have the package installed by default. Command line users can install command line tools.
@catanzaro, did you read what @decathorpe 1 and @kparal 2 stated last? I ask because — to reiterate — whether the server can even be purported to be dysfunctional requires additional, as yet not conducted, investigation. It may merely return incorrect results for searches. That distinction matters, because if it continues to collect accurate data, it continues to perform a useful function.
That you suggest that I might need to manually configure the services post installation demonstrates otherwise; that should be part of the installation script. Being able to type some basic commands into a shell doesn't make someone technically competent in system administration, so I don't understand why you continuously summarise this as if it does.
#issuecomment-376839↩︎#issuecomment-376390↩︎I did read the above comments. I was simply unpersuaded.
I talked to Michal Srb, who is/was the last maintainer of ABRT, IIUIC. I asked him to provide a public response here, not just to me privately, but because I want to speed things up, I'll relay his response here myself.
In short, according to Michal, ABRT is dead. It is no longer a part of RHEL, and the ABRT team got disbanded. There are no people (meaning RH employees) that still work on ABRT. The fact that FAF doesn't have F43 support and nobody cared too much means that people simply don't use it, according to him. There's a lot of work needed to make the project useful for maintainers and upstreams. He supports dropping ABRT from Fedora completely, and replace it with something lighter and less over-engineered. He also states that ABRT would be dead and dropped from Fedora with the Bugzilla->Forge transition anyway, since there are no developers left.
Everything in italics are Michal's views, not mine. I asked him if he could make a public call on devel list, describe the situation and ask for community contributors to step forward for ABRT maintenance and development, if they wish to save ABRT. I don't know if that will happen or not.
That was helpful. Thanks.
Anyway: discussed this at today's meeting. We all agreed that it's a shame to lose ABRT. We will wait a couple more weeks to see if there is discussion on devel list, or if volunteers emerge. Then we'll revisit this to decide whether or not to remove the ABRT service.
Some notable points:
Here is a example of a f43 bug filed via abrt this week:
https://bugzilla.redhat.com/show_bug.cgi?id=2437635
This still looks useful to me.
Surely users should install and enable ABRT if they wish to report bugs to Red Hat Bugzilla. Without the GUI app, I suspect almost nobody is going to do this. Keeping an unmaintained system service installed and running by default to facilitate the possibility that somebody might want to report a bug using the abrt command line is overkill.
@catanzaro, I believe that I dissuaded that notion in
#issuecomment-376875. If you are, again, “unpersuaded”, merely repeating exactly what you've aforestated invites repeated conversation.If the crash notification remains, I disagree. AOSP provides crash notifications, but no mechanism to report them, and in consequence, I constantly observe posts from users that are asking how to report such crashes to their relevant developers.
Unique crashes cannot be retroactively reported if that service is inactive at the time of the crash.
We don't yet have consensus on whether to remove the ABRT service, so let's keep it for F44. The removal of the GUI will act as a "jump scare" to give users a heads-up that ABRT is in danger. Jens will continue to investigate whether a new maintainer can be found. I previously posted a call for help on devel@ and will post an update there.
We'll revisit this in a month or two. I propose removing it for F45 if no new maintainer can be found.
Please at least keep crash notifications GUI (abrt-applet) installed by default so people get to know when something has just crashed – #503 (comment)
To repeat what I said during today's call: a call for new maintainers needs to have the existing maintainers on board and willing to support the transition. The call also needs to spell out the value of taking on the position, and what would be involved. And presumably there are two parts to handing over the maintenance, since there's both the retrace server and abrt itself that need maintaining.
I would welcome a simpler crash catcher to handle notifications, if anybody wishes to write one. ABRT is too complicated to keep around unmaintained for just that one purpose.
@catanzaro Sure. What I meant was if you want to keep ABRT for F44, then keep the notification itself for F44 too.
I'm afraid the notification is probably provided by gnome-abrt, which is gone in F44.
@catanzaro Please take a look at the linked comment #503 (comment)
If somebody wants to work on decoupling abrt-applet from gnome-abrt, I would have no objection. (But the benefit would be very short term if we then proceed with removing ARBT service in F45.)
(It looks like they're already adequately decoupled upstream, and this is only a packaging issue.)
@tomaszgasior wrote in #503 (comment):
Please don't. It is annoying, confusing for end-users, and frankly, not very useful.
@asciiwolf, it's very useful when a device doesn't possess sufficient storage to retain coredumps for long enough for the user to remember to check. It also demonstrates the cause of a crash, which might not be evident even a few minutes afterward, if the user doesn't take note of what they did.
To be honest, most of the crash notifications that I got in past few years were completely useless and some were even extremely annoying (for example recurring video thumbnailer crashes), disturbing the otherwise great distraction-free GNOME design.
@asciiwolf, I can't possibly debate something as vague as “distraction-free GNOME design”; if something crashes, that's not part of normal operation. It should be reported, and for that to be feasible, the user must know that it has crashed, and have an idea of how.
FWIW, the gnome-abrt notifications do not serve this purpose since they do not show any details. Just that an app has crashed.
On a second thought, some basic crash notifier tool could be useful - if it won't be too intrusive and will actually provide useful output (at least the crash signal name).
However, what I'm more worried about is the automatic crash reporting tool and its backend (FAF). I don't think that removing it without a proper replacement is a good idea.
Nobody thinks that having no crash reporting tool is a good idea. But the current tool does not work anymore, and zero developers have volunteered to maintain it. If somebody was willing to maintain ABRT and get it working again, then we could simply close this ticket and move on.
I'm not sure what you are expecting to happen without any developers.
Well, we could fix FAF for F43 and the upcoming F44. That would at least give us some time.
Anyway, my comment was mostly a reply to @rokejulianlockhart. I am unfortunately not able to help with this since my time is already too limited.
If anybody fixes that, then I will close this ticket and we can keep ABRT.
I've just looked at the state of faf and want to share my opinion here: to keep faf alive needs new maintainers, it's not just fixing it for F43 and F44.
This looks like exactly the same situation where I ended up maintaining Bodhi... a "core" system in Fedora infrastructure which was maintained by RH employee is suddendly left without maintainers and keeps going at best effort.
My suggestion here is to let it die and use a suitable existing alternative (if any) or search for possible interest for its usage in other Linux distros to build a new group of maintainers.
@catanzaro wrote in #503 (comment):
Even if the FAF server is decommissioned: It's only necessary for one (small) part of abrt functionality (remote backtrace generation and a statistics frontend). Generating backtraces from coredumps locally and reporting bugs still works fine without it.
I guess I'm just going to keep repeating myself again and again:
Keeping ABRT installed by default is only useful if it is able to report bugs automatically.
@mattia, what
github.com/NixOS/nixpkgs/issues/40550#issuecomment-3500245168references exists.@catanzaro wrote in #503 (comment):
I would like to propose that we revert that change.
@petersen, might that be the purview of another ticket (like
issues/130)?We can handle it here. Next week's meeting is busy, so I'll schedule Jens's proposal for our meeting on April 7.
We discussed this at today's meeting anyway, because Jens pointed out there is very little time remaining in the Fedora 44 schedule, and waiting another week to make a decision would not be great. We held a vote on whether to bring back ABRT GUI, gnome-abrt, for Fedora 44. The proposal failed (+2, 1, -5), so we will ship Fedora 44 with only the ABRT service.
Action item: Michael to invite Jef to a future Working Group meeting to discuss the fate of ABRT.