Give ABRT some ❤ #130
Labels
No labels
a11y
btrfs
Closed As
Can't fix
Closed As
Deferred to upstream
Closed As
Fixed
Closed As
Won't fix
default-apps
easyfix
experience
help-wanted
installation
meeting
meeting-request
nvidia
packaging
pending-action
qa
testing
user-docs
wg-docs
wg-meta
No milestone
No project
No assignees
14 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
workstation/tickets#130
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
ABRT is very important for improving the quality of Fedora Workstation. So let's give it some ❤.
ABRT has two completely separate crash reporting paths currently, so let's talk about them separately.
❤ Automatic crash reports with truncated backtraces
This is already working more or less well.
The main problem here is that crash reports never seem to make their way from the retrace server to Bugzilla unless a user manually reports the crash to Bugzilla. Ideally, once a crash is reported by, say, 100 users, a Bugzilla report would be automatically created. (Mega bonus points if it could learn to create bug reports directly in GNOME GitLab.) I think we used to have this functionality, but if so, it's broken.
But on the whole, automatic crash reporting is in good shape.
❤ Semi-automated manual crash reports with full backtraces
Automatic crash reports, while important and helpful, don't contain full backtraces (to protect the user's privacy, since full backtraces sometimes contain sensitive data in stack frames). Without a full backtrace, resolving a crash using just the automatic crash report can be quite difficult. Often, the only practical solution is to wait for a manual ABRT bug report to arrive, since ABRT's manual bug reports to Red Hat Bugzilla include detailed backtraces.
Sadly, ABRT's manual bug report process suffers from a very large number of quality issues.
Highest-priority issues:
Try to reduce false-positives when flagging sensitive wordsandPossible sensitive data detected is almost always a false positiveRetrace server should not failABRT very frequently fails to report crashes, and does not allow reporting failure to reportandCan GNOME-Shell/Wayland or "bigger" programs even reported in any way?Other major issues:
Don't offer to restart GNOME ShellABRT deletes core dumps far too aggressively, even while user is trying to report the crashEach of these issues is, on its own, a serious quality problem for Fedora Workstation. Combined, it's really quite a lot.
Pet peeve quality issues:
Remove preferencesCoredump references 204 debuginfo files, 224 of them are not installedkernel-core failure reported for userspace crashesThe WG should work with the ABRT developers to understand their priorities, figure out how we can reduce the number of quality issues, and determine whether we have a plausible path forward for ABRT ❤.
❤ Future wishlist
I do not think this will end well at all. I think we're going to see people just ignoring them upstream like they do downstream.
CC @ekulik
In practice, this does happen in some GNOME projects. But on the whole, we are far better at responding to bug reports upstream than we are downstream. Let's continue this thread of discussion in #131.
I believe most issues are pure engineering, but we could probably do something about the overall quality of reports if we were able to define what makes a good report.
FAF could also be involved in reporting to bug trackers for some finer-grained controls of whatever (I don’t know, attachments to include, something), but then there’s a level of indirection and some challenges to overcome to prevent spam (my thinking here is if we don’t require reporters to have a Bugzilla account, but now I see how that could be annoying, not being able to query for information).
I think the reports that ABRT uploads are generally already of good quality. The problems I identified above are mainly user and developer experience issues.
Probably auto-filing bugs upstream and linking them to downstream bugs in RHBZ would work better than straight-up avoiding filing bugs downstream.
The branching of Rawhide is always a special time of year and there’s always a bit of lag between that and us adding the version and repository. If that still happens after, say, the Bodhi activation point, it’s a bigger problem.
Please, I created #131 for this, let's discuss it there.
Metadata Update from @catanzaro:
Metadata Update from @chrismurphy:
Discussed at today's meeting. Ernestas will provide us with occasional progress updates in this issue, and we'll revisit a few months from now.
Metadata Update from @catanzaro:
https://github.com/abrt/abrt/pull/1481 should address this. https://github.com/abrt/abrt/issues/1475, if (properly) implemented, should prevent the issue from happening altogether. Might aim for F33 with the latter.
In the end, we’ve decided to wait until we get the new hardware to redeploy everything on RHEL 8. With the amount of crud on the current server, we cannot guarantee that we would be able to replicate the deployment and be able to roll back.
I don’t have the dates, but it’s still “soon”.
Bug 1878317 - doesn't offer option to process stack traces locally is now a Fedora 33 beta blocker.
The decision to classify this bug as an "AcceptedBlocker" was made on the grounds that it "hinders execution of required Beta test plans or dramatically reduces test coverage"...
Metadata Update from @chrismurphy:
There are two more. Barring some miracle, we will almost surely slip another week.
Metadata Update from @chrismurphy:
Adding the F34 milestone, mostly to ensure that we continue to track this.
Metadata Update from @aday:
Metadata Update from @catanzaro:
Discussed again at today's WG meeting.
Metadata Update from @catanzaro:
We've split the "major issues" list in the first comment into two halves, "highest-priority" and "other major issues." The highest-priority list includes the three bullet points we found most important. It's actually five issues, but two of those are closely-related to other issues.
He'll attend next week, so I'll put this topic first on the agenda for next week.
Metadata Update from @catanzaro:
Kevin mentioned the bugzilla dashboard which I didn't know about.
https://bugzilla.redhat.com/page.cgi?id=productdashboard.html&tab=summary&product=Fedora&bug_status=open&assignee_table_length=25
As for QA there isn't a systematic way of using bugzilla for learning about bugs or problem areas. In no particular order problem areas are uncovered by: blocker bug app, openqa testing, and the various compose reports that get sent to the devel@ list.
@adamwill
Well, also significant in that context are human mailing list posts (and IRC discussion, forum discussion, etc). We do follow those forums and jump on any problems that seem to need jumping on. Major issues tend to 'bubble up' from Bugzilla to the mailing lists especially. Crash reports are often useful for figuring out what is going on with a bug at that point, or at least identifying duplicates and getting a sense of the scale of the problem.
We do haphazardly use the Bugzilla reports too. For instance a year or two back I just decided to try and triage all gnome-shell bug reports, and I did a bunch of work digging through abrt crash reports there (which made up a large chunk of all the bug reports). Some of them did turn out to be quite significant bugs we could isolate and fix, IIRC.
There is one kind of systematic GNOME-specific issue here, which is that the backtraces abrt produces often aren't much use in diagnosing crashes in the Shell because the bug really happened "somewhere else", e.g. in the javascript code. But even then, abrt does attach a snippet of the system logs from around the time of the crash, which can help.
Metadata Update from @catanzaro:
Discussed at today's WG meeting.
Other high-priority and major issues:
@msuchy, would January 5 work OK for the next WG meeting? (If not, we can use a later date.)
@catanzaro I prefer date week later. On 5th I have planned PTO - it may or not happen. It depends on current local pandemic situation. 12th is definitely ok with me.
12th is booked too, so let's plan for the 17th.
Hi @msuchy, I guess you still want to attend? Does the 17th sound good?
I understand you are no longer managing ABRT team anymore. Perhaps we should invite the new manager as well? Who would that be? (Of course, the whole team is welcome to attend.)
I can attend. I will also notify the new tech lead who is @msrb
I will attend as well ;)
Metadata Update from @catanzaro:
Discussed at today's meeting. ABRT developers accepted some of my suggestions in Possible sensitive data detected is almost always a false positive.
Partially discussed today, though we were short on time. Miroslav has posted a short summary in the issue.
Metadata Update from @catanzaro:
Metadata Update from @catanzaro:
Current status is retrace jobs seem to always fail. Bug was reported in October last year, but it's still broken.
I can confirm
https://github.com/abrt/retrace-server/issues/428
We discussed this issue during today's WG call, and there's a desire to somehow resolve it during the F37 cycle.
Obvious options that are available to us:
Next steps:
We should aim to talk about this again in no less than a month.
Metadata Update from @aday:
Hi everyone!
Jens reached out to us (abrt) via e-mail and I'd like to shed some light on the progress here.
We tried to address this issue in libreport 2.15.1 (released 04/2021) and it should be available in all f34+ releases. Do people still experience this problem?
Yes, this is a big one. Although Matej G. fixed a ton of problems in retrace server last year, it just randomly keeps breaking. We have monitoring for the service that shows that the failure rate is usually on the lower-ish side (~5%), but from time to time there is a day when the instance is completely broken and everything fails.
Since debuginfod is available by default in f35+, we decided that we will switch to that for backtrace generation instead of retrace server.
There is already a WIP PR with the change here: https://github.com/abrt/abrt/pull/1600
The change is planned for this quarter.
We patched libreport in all f34+ releases so it should now require API key instead of username+password.
However, if we are talking about existing images that bundle older version of libreport, then there is unfortunately nothing the ABRT team (or anybody) can do -- it's technically not possible to update the library there.
New f36 images should already include the patched version and I'd hope that for older releases people already reported the majority of problems that they encountered during installation and thus we shouldn't be missing much (?)
There is one big area where we unfortunately did not make any direct progress last year -- abrt reporting too many duplicates.
We started modernizing the bugzilla reporting codebase that will allow us to implement changes around Bugzilla workflow much quicker. This effort should also finish this quarter and then we will try to address the problem with too many duplicates.
Please let me/us know if you have questions or suggestions.
Thanks!
I experience it quite often. Some environment variables trigger it, even though they could be easily ignored. I used to report some false positives in the past, but then gave up. Should I continue to file those?
Thanks for the update, @msaju ! It's great to hear that progress is being made on these issues.
The working group discussed this again yesterday and is looking forward to testing any improvements. We also discussed the possibility of updating the gnome-abrt UI. I'd be happy to work on updated designs, if that would help.
@kparal We keep trying to improve the user experience so please do report any false positives you encounter.
@aday I'm glad to hear that. We would be happy to cooperate on improving the UI as well.
Another one that would be good to fix: https://bugzilla.redhat.com/show_bug.cgi?id=1120859
@mgrabovs - I've created an updated set of gnome-abrt mockups. These are just a start and will need plenty of discussion and collaboration, so just let me know if and when you'd like to make any changes in this direction.
Metadata Update from @aday:
Thank you so much for the mockups, @aday. They look great. We'll get in touch once we're ready for the transition.
Metadata Update from @aday:
We briefly discussed this issue during today's WG call.
We agreed that the reports from ABRT can often include useful information. Unfortunately the ongoing issues with private reports and duplicates mean that they're difficult to digest. Since this issue has been going on for so long, we are reluctant to continue to let it slide.
We've therefore agreed to propose a PR to remove the privacy filtering from ABRT. @mclasen has volunteered to try and find someone to do that.
Metadata Update from @aday:
As pointed by @msrb in https://pagure.io/fedora-workstation/issue/130#comment-791156 the problematic keywords (as reported in https://github.com/abrt/abrt/issues/1399) were fixed and are already part of all stable Fedora releases.
I just received a private bug report yesterday (here so the private reports are still a problem. I'd like the functionality to report private bugs removed altogether, or at minimum, hidden so that it's not easy to do via the ABRT UI.
This seems dangerous and I'd be concerned about it. We absolutely have had real cases of people submitting bug reports that contained their actual passwords.
I'm not suggesting that we remove the function to highlight possible sensitive data. But the private bug reports have got to stop.
Oh, okay, yeah, if we provide a viable alternative way to deal with possibly-sensitive data I'd be less concerned.
It seems that we are waiting on finding a volunteer to take this forward.
We discussed this issue at today's WG meeting. The WG feels that the data we get from abrt is valuable, but there are concerns about the maintenance level.
We plan on talking to the relevant individuals about this, and then on possibly opening up a wider discussion on the subject.
The switch to debuginfod will likely affect the issues being discussed here, so we're going to need to try that and then reassess. Is there an ETA on when it will be released, or when we'll be able to test it?
Hello there. I apologize for the delay -- I got swamped with other work. The update is incoming on Monday ;)
Here we go:
The update with debuginfod-enabled ABRT has been pushed to Rawhide:
https://bodhi.fedoraproject.org/updates/FEDORA-2022-bedb590873
I've also prepared COPR with the same update for f37:
https://copr.fedorainfracloud.org/coprs/g/abrt/abrt-debuginfod/
I can try to build it for f36 if that would help with testing.
Please let us know if you encounter any issues. Thanks! ;)
After installing the packages from the copr and rebooting my computer, I open gnome-chess and then run
killall -SEGV gnome-chessto trigger a fake crash. Then I try to Report the crash using ABRT. It fails:So that's not good. I was able to successfully "report" a nautilus crash though, which wound up taking me to https://bugzilla.redhat.com/show_bug.cgi?id=2129705, although it discarded my comment presumably because I didn't write anything, so nothing really happened. That's a strange user experience.
Will continue to test and see how it works.
Experience just now with the debuginfod version of ABRT: the app correctly showed a notification for an Inkscape crash. When I tried to report it, it failed to generate the report with
Error: No segments found in coredump './coredump'.I think I've seen this same error once before with abrt from the copr. I haven't been able to successfully report an issue with it yet.
@msrb fyi: gnome-abrt has been orphaned and will be retired if nobody takes ownership. Should probably be owned by the abrt-team account.
I'm seeing the same error message (which prevents me from reporting any issues through GNOME abrt) and have reported it to the retrace server: https://github.com/abrt/retrace-server/issues/484
Is there a workaround until the bug gets fixed?
So I noticed that my Epiphany crashes are still being detected as kernel failures due to this bug that has been on the list for a long time. Unfortunately ABRT developers have not made significant progress on the list of issues in the first comment. And we don't have manpower to assist ABRT team with improving ABRT at this time.
Proposal: remove ABRT it for Fedora 39 and aim to bring it back in the future.
Metadata Update from @catanzaro:
Metadata Update from @catanzaro:
Metadata Update from @catanzaro:
Hi @msuchy @msrb , the Workstation WG is concerned about slow progress on the issues identified in the first comment. We have two parallel proposals:
Any thoughts on this?
It might be possible to close some of the existing tickets that we're tracking that relate to the retrace server, if that is no longer used.
Metadata Update from @catanzaro:
I no longer work in the ABRT team. This is at @msrb discretion now
Oops, sorry, I knew that. I meant to ping @mgrabovs.
Hi @catanzaro
I like the plan. There is usually at least one bugfix release every quarter, so things should be, hopefully, going in the right direction. Although slowly... :/
It would be amazing if you could find somebody to work on the UI part. I was thinking whether we could try to find a contributor via GSoC to work on that, but that opportunity has passed for this year.
Fedora 38 is the first release which uses debuginfod instead of retrace server by default. The same update has been submitted for F37 as well, so it leaves us with just F36 (EOL'ed soon) that still relies on the retrace server.
OK, so sounds like this is a plan. We'll check back in July (Q3).
Metadata Update from @aday:
Reminder: we'll check back in early July for a progress update on these issues.
Hi @aday and @msrb, would Tuesday, July 11 be a good date for ABRT follow-up discussion? Workstation WG meetings are Tuesdays at 10:00 AM EDT (14:00 UTC). (We could do a separate meeting at a different time if that time slot doesn't work well for you, but that's the time where we can have the full Working Group present.)
Metadata Update from @catanzaro:
Metadata Update from @catanzaro:
@catanzaro Tuesday, July 11 works for me ;)
[Edit - I commented in the wrong ticket. Please ignore.]
Metadata Update from @aday:
Apologies for the previous comment - I picked the wrong tab by mistake.
We discussed this issue on 11 July. Some fixes have been made from @catanzaro 's list, and it is hoped that these will improve the situation. The plan is to continue trying to make progress on a quarterly basis.
The ABRT team have told us that they don't have the time or expertise to work on the
gnome-abrtUI, so I've created a separate ticket to track what to do about that - that's #386.Let's plan to have another meeting with ABRT developers in October for another progress update.
Small status update: After upgrading to Fedora 39, which is unsurprisingly not as stable as Fedora 38, and using the new ABRT for several days to report many crashes, I'm pretty confident that the debuginfod integration has dramatically improved both the speed and the reliability of ABRT. Nice.
Ironically, for me, debuginfod is what made me mostly stop using abrt - I just backtrace things with coredumpctl or gdb directly now, since gdb has debuginfod integration. the main benefit of abrt for me was the retrace server, heh.
Hi @msrb, seems we all forgot about October progress update. How does November sound? Are you able to attend another Workstation WG meeting?
Metadata Update from @catanzaro:
Hi @msrb, any update?
Metadata Update from @catanzaro:
Hey, I apologize for the radio silence here lately. I did not have much time to look into these things. However, a new release is coming to Fedora this week 😊
Metadata Update from @catanzaro:
Metadata Update from @ngompa:
I suggest it's time to take a pragmatic approach here. We keep ABRT and automatic problem reporting, but remove GNOME ABRT from the default install. Users who want the user interface for reviewing problems and reporting to Bugzilla will have to install it manually.
It would be nice to keep the GNOME ABRT tool, but it would require a significant redesign and major functionality enhancements (at minimum support for Flatpaks and for reporting problems directly to upstream GNOME GItLab rather than Red Hat Bugzilla) and we don't have contributors interested on working on these currently. This ticket has been open for four years now and further progress is unlikely. We should add it to our list of "nice to do, if we had more contributors."
Metadata Update from @catanzaro:
+1 from me. I was hoping to maybe get somebody to work on this as part of the GSoC, but Fedora project wasn't selected this year.
We do not have consensus to remove the ABRT UI.
Action items:
This issue has been open for 4 years now. I fear there's a strong chance it will still be open 4 years later if Red Hat doesn't fund more work on ABRT and no new volunteers emerge. But we all agree that crash reporting is very important, so the Working Group's reluctance to remove the app is understandable.
Metadata Update from @catanzaro:
I think the potential middle ground could be to keep the UI enabled pre-GA. Some people (including me) upgrade to the upcoming release once the Beta is out, and sometimes even sooner. The assumption here is that most blatant crashes will be caught and (easily) reported by people who upgrade and test stuff early, before the final release is out.
How would we do this, though? We would have to modify the desktop file to hide the application shortly before final freeze. But then it's gone for everybody and you have to manually unhide it if you want it. Not sure that's a good idea.
I asked @sshil to start looking at this with a view to initially moving gnome-abrt to gtk4/libadwaita.
Don't want to promise too much, but let's see how far we can get for F41 and later.
That's great. Note there are mockups here.
https://github.com/abrt/gnome-abrt/pull/353 by @sshil was merged upstream and hopefully there will be a release to appear in Rawhide soon.
There are some polish fixes waiting to be merged in https://github.com/abrt/gnome-abrt/pull/368 . As part of this I've updated the mockups, since they had fallen behind the latest design patterns:
Metadata Update from @catanzaro:
Agreed: ABRT's user interface (gnome-abrt), which provides manual crash report functionality, will be removed in Fedora 44 unless development activity increases substantially in the next few months. We will keep the underlying ABRT daemon, which provides the automatic crash reporting functionality.
The Working Group is, in practice, a policy group. We're not actually very good at working and implementing stuff, so keeping the ABRT UI would require new volunteers. Help welcome. It requires substantial work:
Also agreed: https://retrace.fedoraproject.org/faf/summary/ contains a treasure trove of crash telemetry. It's a shame that Fedora developers rarely look at it. This data could be quite useful for improving software quality. This would be a good opportunity to integrate AI, which could identify crash patterns or even suggest fixes.
Metadata Update from @catanzaro:
I've sent a mail to @msrb and will also plan to notify devel@ in order to attract more attention. You never know, maybe some volunteers will emerge.
In the meantime, I'm adding the pending-action tag here since we have approved the removal and need to remember to implement it.
Metadata Update from @catanzaro:
@catanzaro,
gnome-abrtis the primary reason that I utilise Fedora. I realise that my opinions do not provide development assistance. However, it would force me to reconsider.(You can still install it if you want. It just would not be installed by default anymore.)
Ah! That's fine. Thanks, and apologies.
Mailing list thread
https://pagure.io/fedora-comps/pull-request/1239
Pending action: test an install, confirm that automatic crash reporting still works.
Confirmed fixed: ABRT GUI is gone, but ABRT service remains and automatic problem reporting appears to still be enabled. The GUI is present in both gnome-initial-setup and gnome-control-center, indicating the D-Bus service still works. I also triggered a dummy crash and ABRT recorded it:
I'm not sure how to test to be certain that it definitely got reported, but I assume everything is working. Let's close.
Metadata Update from @catanzaro:
@catanzaro, am I correct that it's solely not pre-installed, in the default images, any more? I ask because I presume that it's still available in the repositories. If so, I'm not sure that I'd close this ticket, because the GTK4 port still remains very progressed, yet importantly incomplete.
That's not a report, yet. You need to invoke
abrt report.Correct.
Well it's clear enough that nobody intends to work on this. This issue has been open for 6 years already. I don't think there's value in keeping it open forever.
This working group is responsible for preinstalled applications, not every application in the Fedora repos.
abrt reportlooks like the manual report to Bugzilla workflow, not the automatic reports to retrace.fedoraproject.org. Automatic reports should still (hopefully) work.@catanzaro, I believe that the GTK4 revamp of all but the preferences interface was introduced for F43, but reverted, due to the lack of a GTK4 preferences interface, so the work was, very recently, being done.
Checking the gnome-abrt git history, I see that zero commits landed since we specified that gnome-abrt would be removed unless development pace increases significantly. There is one interesting pull request, but it also had zero changes during this time. This issue report's 6th birthday is approaching. It's time to give up and move on.
If somebody seriously intends to fix up gnome-abrt, I would be thrilled, because removing ABRT cripples our most valuable source of crash reports and is possibly the worst product quality decision we've ever made. The developers who used to work on crash report infrastructure were a force multipler, allowing all other developers to improve software quality. But... well, clearly nobody is left. It's over. Too bad.
Sorry but I feel obliged to comment:
TBH I felt this was a rather rushed final decision: I barely had a moment to understand what you were proposing in the WG meeting on Tuesday, and had meant to confirm the plan...
There has been some work upstream both by Sudip in my team who ported gnome-abrt to gtk4 and Adwaita and also the open MR by Bilel - perhaps the changes are not happening as fast or to the level some people may desire but at least there have been some improvements in the last couple years so it is not like nothing has happened for 6 years. :-)
Anyway leaving this feedback here for the record.
@catanzaro wrote in #130 (comment):
Unfortunately the retrace server is still not even aware of Fedora 43 (yes, 43, reported a long time ago). So that's the current sad state of ABRT maintenance 🙁
We didn't make any decisions on Tuesday. I removed ABRT on Monday, and provided a quick status update that I had done so at our meeting on Tuesday.
Our agreement which we approved in October was: "will be removed in Fedora 44 unless development activity increases substantially in the next few months." Since then, 0 commits in gnome-abrt including 0 updated commits in Bilal's merge request. There have been 3 commits in abrt itself. I don't think it's controversial to say that development has not increased substantially.
It took more than 5 and a half years to reach this agreement. ABRT has obviously been abandoned. All of the gnome-abrt work you reference happened before the agreement; it really looks like nothing has happened since. At this point, it's clear that nobody is going to volunteer to maintain ABRT, and Red Hat is not going to assign anybody to do so. I would be very enthusiastic to reintroduce ABRT if this changes, but this seems quite unlikely.
Created #503 for removing the rest of ABRT.
Michael, do I understand correctly that with ABRT GUI gone from Workstation, there will be no notification whatsoever when an application crashes? So basically an app window will suddenly disappear (or not start at all), but the user won't be made aware that this was an unexpected app crash?
If that's so, I consider it a very sad outcome. Not being able to (graphically) report a bug is one thing, but not informing the user about a crash at all is even worse. There's benefit in knowing that something crashed, especially when the user then e.g. asks in discussion forums and wants to describe the situation.
Any chance of at least having an "App/process FOO crashed, sorry!" notification handler (without any further actions) as a replacement?
@kparal, if so,
discuss.kde.org/t/4821at least provides this on KDE Plasma:However, without access to FAF, it doesn't provide information that is useful to a non-technical user, unless we expect them to understand
coredumpctl debug's YAML-like preamble, even indrkonqi-coredump-gui.Correct.
I agree it's an extremely sad outcome.
It would need to watch the system journal for new core dumps, just like ABRT. I guess it could go into gnome-settings-daemon, but really the best approach would be to resurrect ABRT and do it there. tbh anything is possible if developers were interested in working on crash handling. With no interested developers, nothing is possible. Honestly this seems like it would be a fun project, if only I didn't have everything else that I work on to work on....