Bugzilla "report upstream" script causes conflicts with ABRT reports

kparal commented

2025-03-20 11:35:21 +00:00

Today I wanted to report a crash in Nautilus using ABRT. However, I received this error during ABRT reporting:

--- Running report_Bugzilla ---
Checking for duplicates
Creating a new bug...
Adding External URL to bug 2353858
Adding attachments to bug 2353858
Failed to attach 'environ' to bug # 2353858.
Server says: 409 Conflict
{"error":true,"documentation":"https://bugzilla.redhat.com/docs/en/html/api/index.html","code":31113,"message":"Red Hat Bugzilla's database reported a query serialization error. Most likely this occurred because another user or process attempted to change the same data that you were attempting to change. Please press Back and retry the transaction. <pre> UPDATE bugs SET delta_ts = ? WHERE bug_id = ? at /var/www/html/bugzilla/Bugzilla/Bug.pm line 1523. </pre>"}
Traceback (most recent call last):
  File "/usr/bin/reporter-bugzilla", line 833, in <module>
    attachment, attachments_bug_id, response.json()['ids'][0])
                                    ^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'json'
('report_Bugzilla' exited with 1)

The important part is:

Red Hat Bugzilla's database reported a query serialization error. Most likely this occurred because another user or process attempted to change the same data that you were attempting to change.

The bug was only partially reported, with the most important attachments missing:
https://bugzilla.redhat.com/show_bug.cgi?id=2353858

I believe the problem is in comment 4, which is a "report upstream" script that Workstation runs for most/selected GNOME components.

The immediate response of the script, before the bug was fully reported, caused a request conflict in Bugzilla, and prevented ABRT from fully reporting it.

I see this as a problem, because important attachments (e.g. a full stacktrace) are missing. That severely limits the usefulness, if the user wanted to report the crash upstream.

The easiest solution is probably not to trigger the script immediately, but delayed (a minute or two should be enough). Or make it run in batches (e.g. every 30 minutes, every hour) on all bugs detected during that interval. This would not eliminate, but minimize the likelihood of such issues.

Today I wanted to report a crash in Nautilus using ABRT. However, I received this error during ABRT reporting: ``` --- Running report_Bugzilla --- Checking for duplicates Creating a new bug... Adding External URL to bug 2353858 Adding attachments to bug 2353858 Failed to attach 'environ' to bug # 2353858. Server says: 409 Conflict {"error":true,"documentation":"https://bugzilla.redhat.com/docs/en/html/api/index.html","code":31113,"message":"Red Hat Bugzilla's database reported a query serialization error. Most likely this occurred because another user or process attempted to change the same data that you were attempting to change. Please press Back and retry the transaction. <pre> UPDATE bugs SET delta_ts = ? WHERE bug_id = ? at /var/www/html/bugzilla/Bugzilla/Bug.pm line 1523. </pre>"} Traceback (most recent call last): File "/usr/bin/reporter-bugzilla", line 833, in <module> attachment, attachments_bug_id, response.json()['ids'][0]) ^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'json' ('report_Bugzilla' exited with 1) ``` The important part is: > Red Hat Bugzilla's database reported a query serialization error. Most likely this occurred because another user or process attempted to change the same data that you were attempting to change. The bug was only partially reported, with the most important attachments missing: https://bugzilla.redhat.com/show_bug.cgi?id=2353858 I believe the problem is in [comment 4](https://bugzilla.redhat.com/show_bug.cgi?id=2353858#c4), which is a "report upstream" script that Workstation runs for most/selected GNOME components. The immediate response of the script, before the bug was fully reported, caused a request conflict in Bugzilla, and prevented ABRT from fully reporting it. I see this as a problem, because important attachments (e.g. a full stacktrace) are missing. That severely limits the usefulness, if the user wanted to report the crash upstream. The easiest solution is probably not to trigger the script immediately, but delayed (a minute or two should be enough). Or make it run in batches (e.g. every 30 minutes, every hour) on all bugs detected during that interval. This would not eliminate, but minimize the likelihood of such issues.

tpopela commented

2025-03-20 12:54:12 +00:00

Owner

@blc do you know it it's possible to add some timeout as Kamil is suggesting? This is about the following Bugzilla rule - https://bugzilla.redhat.com/page.cgi?id=ruleengine%2Fdetails%2Findex.html&rule_name=Further%20Upstream%20First%3A%20GNOME

blc commented

2025-03-20 22:45:31 +00:00

I think Kamil is right and this is a multi-update situation.

There is no "sleep" feature in BRE, but I could change the rule to run once per day instead of at time of bug update. I'm loathe to do this though because it would mean that whoever files a bug will get the feedback they filed in the wrong place as late as 24 hours after, long since they thought they'd done their part.

Currently this rule runs first in BRE. I could have it run later.... but there are only 3 rules total so I don't think it would slow down a lot.

Probably the best solution would be to split the rule in 2, allowing a different behavior depending on some characteristic of the initial bug report. As a bad example, I could have the current rule only trigger when anybody but Kamil files the issue, and a second rule that causes the rule to run once a day when it's Kamil who files the issue. This isn't actually a good solution, but it's basically the kind of capability that BRE has.

A more realistic rule would be to check something like the "QA Whiteboard" for something like "PauseBRE", then the tool that this issue is being filed with would set "QA Whiteboard" to "PauseBRE" on creation, attach any files, add any comments, then clear the "PauseBRE" value from "QA Whiteboard". All the rules that might cause a problem would then look for this value and not trigger on update if the value is there. Or we could use a flag. In this circumstance I would add a rule that runs every day that automatically clear the "PauseBRE" value. Or a flag. You get the idea. Let me know if there is or easily could be some value in the initial BZ that we could use to pause the rules engine and I can take it from there.

I think Kamil is right and this is a multi-update situation. There is no "sleep" feature in BRE, but I could change the rule to run once per day instead of at time of bug update. I'm loathe to do this though because it would mean that whoever files a bug will get the feedback they filed in the wrong place as late as 24 hours after, long since they thought they'd done their part. Currently this rule runs first in BRE. I could have it run later.... but there are only 3 rules total so I don't think it would slow down a lot. Probably the best solution would be to split the rule in 2, allowing a different behavior depending on some characteristic of the initial bug report. As a *bad* example, I could have the current rule only trigger when anybody but Kamil files the issue, and a second rule that causes the rule to run once a day when it's Kamil who files the issue. This isn't actually a good solution, but it's basically the kind of capability that BRE has. A more realistic rule would be to check something like the "QA Whiteboard" for something like "PauseBRE", then the tool that this issue is being filed with would set "QA Whiteboard" to "PauseBRE" on creation, attach any files, add any comments, then clear the "PauseBRE" value from "QA Whiteboard". All the rules that might cause a problem would then look for this value and not trigger on update if the value is there. Or we could use a flag. In this circumstance I would add a rule that runs every day that automatically clear the "PauseBRE" value. Or a flag. You get the idea. Let me know if there is or easily could be some value in the initial BZ that we could use to pause the rules engine and I can take it from there.

kparal commented

2025-03-21 08:51:13 +00:00

Author

All ABRT-reported bugs have [abrt] in the Summary (title). They also have abrt_hash: in the Whiteboard. Can we use one of these to move the rule to the "update once per day" bucket?

For manual bug submission, I think the immediate response is not a big problem, the reporter can deal with it even if a conflict is encountered (Bugzilla offers you several options to continue). We just need to handle automatic reports better, which means ABRT (and I'm not aware of anything else). So detecting ABRT keywords could be a good enough solution.

All ABRT-reported bugs have `[abrt]` in the Summary (title). They also have `abrt_hash:` in the Whiteboard. Can we use one of these to move the rule to the "update once per day" bucket? For manual bug submission, I think the immediate response is not a big problem, the reporter can deal with it even if a conflict is encountered (Bugzilla offers you several options to continue). We just need to handle automatic reports better, which means ABRT (and I'm not aware of anything else). So detecting ABRT keywords could be a good enough solution.

catanzaro commented

2025-03-21 15:31:53 +00:00

Owner

All ABRT-reported bugs have [abrt] in the Summary (title). They also have abrt_hash: in the Whiteboard. Can we use one of these to move the rule to the "update once per day" bucket?

This sounds like an excellent short-term solution.

A more realistic rule would be to check something like the "QA Whiteboard" for something like "PauseBRE", then the tool that this issue is being filed with would set "QA Whiteboard" to "PauseBRE" on creation, attach any files, add any comments, then clear the "PauseBRE" value from "QA Whiteboard".

And this sounds like an excellent long-term solution. This would need an ABRT bug report.

> All ABRT-reported bugs have `[abrt]` in the Summary (title). They also have `abrt_hash:` in the Whiteboard. Can we use one of these to move the rule to the "update once per day" bucket? This sounds like an excellent short-term solution. > A more realistic rule would be to check something like the "QA Whiteboard" for something like "PauseBRE", then the tool that this issue is being filed with would set "QA Whiteboard" to "PauseBRE" on creation, attach any files, add any comments, then clear the "PauseBRE" value from "QA Whiteboard". And this sounds like an excellent long-term solution. This would need an ABRT bug report.

catanzaro commented

2025-10-28 16:55:33 +00:00

Owner

Hi @blc, since you maintain the Bugzilla rules, I wonder how you would prefer to move forward here. Maybe we could start with checking for abrt_hash:, process it once daily, and hope that users don't submit bug reports at the time it processes?

This is going to be a terrible user experience regardless, but so long as ABRT is reporting issues only on Red Hat Bugzilla, and we want to receive issues only on GNOME GitLab, the user experience is not going to be good. Also, it requires a Bugzilla API key, which will block all but the most advanced users from using the tool, so I think we don't have to worry too much about the user experience.

Hi @blc, since you maintain the Bugzilla rules, I wonder how you would prefer to move forward here. Maybe we could start with checking for `abrt_hash:`, process it once daily, and hope that users don't submit bug reports at the time it processes? This is going to be a terrible user experience regardless, but so long as ABRT is reporting issues only on Red Hat Bugzilla, and we want to receive issues only on GNOME GitLab, the user experience is not going to be good. Also, it requires a Bugzilla API key, which will block all but the most advanced users from using the tool, so I think we don't have to worry too much about the user experience.

catanzaro commented

2026-04-28 19:25:27 +00:00

Owner

Ping @blc and @tpopela, would be nice to fix this regardless of future plans to move away from Red Hat Bugzilla.

tpopela commented

2026-04-29 07:48:56 +00:00

Owner

I wouldn't invest more time into improving the current BRE if it will go away anyway.

catanzaro commented

2026-04-29 14:01:02 +00:00

Owner

Then let's disable it globally and declare it a failure. We shouldn't break ABRT for people who choose to install and use it.

👍 2

kparal commented

2026-04-30 06:57:03 +00:00

Author

I tried reporting a (fake) gnome-text-editor crash today, and it broke the reporting in exactly the same way. It looks like this access conflict is quite reliable, there's low chance that all the attachments are uploaded before the BRE kicks in.

OTOH with ABRT being gone from Workstation, this is going to bother much fewer people than before. And it has been at least a year already.

I tried reporting a (fake) gnome-text-editor crash today, and it broke the reporting in exactly the same way. It looks like this access conflict is quite reliable, there's low chance that all the attachments are uploaded before the BRE kicks in. OTOH with ABRT being gone from Workstation, this is going to bother much fewer people than before. And it has been at least a year already.

tpopela commented

2026-04-30 07:38:55 +00:00

Owner

OTOH with ABRT being gone from Workstation, this is going to bother much fewer people than before. And it has been at least a year already.

I had the exactly same thoughts. Also if we will switch over to use templates (see https://pagure.io/fesco/issue/3568#comment-1015514) then that will also resolve this issue.

> OTOH with ABRT being gone from Workstation, this is going to bother much fewer people than before. And it has been at least a year already. I had the exactly same thoughts. Also if we will switch over to use templates (see https://pagure.io/fesco/issue/3568#comment-1015514) then that will also resolve this issue.

catanzaro commented

2026-05-05 17:41:13 +00:00

Owner

Ping again @blc. We would like to disable the autoresponder entirely: just completely delete it. Can you do that please? This is necessary regardless of the above problem with ABRT due to: https://pagure.io/fesco/issue/3568

catanzaro referenced this issue

2026-05-05 17:44:49 +00:00

Red Hat Bugzilla: GNOME packages are in bad shape #131

tpopela commented

2026-05-06 05:34:06 +00:00

Owner

We would like to disable ..

From what I see is that you would like to disable it.

> We would like to disable .. From what I see is that you would like to disable it.

😕 1

catanzaro referenced this issue

2026-05-08 17:11:22 +00:00

Red Hat Bugzilla: GNOME packages are in bad shape #131

catanzaro referenced this issue

2026-05-12 15:31:55 +00:00

Red Hat Bugzilla: GNOME packages are in bad shape #131

catanzaro commented

2026-05-12 15:32:41 +00:00

Owner

Agreed:

Disable the autoresponder for new bug reports. Users creating a new bug report will see our issue report template instead (#131).
If possible, keep the autoresponder for when component is changed, since these users will not have seen our issue report template. If that's not possible with BRE, then delete the autoresponder.

We also agreed to keep the autoresponder enabled for reports created by ABRT if the incompatibility with ABRT can be fixed, but after reviewing the previous discussion, I don't think it's worth attempting to do this so long as ABRT remains unmaintained. So let's not worry about that.

Agreed: * Disable the autoresponder for new bug reports. Users creating a new bug report will see our issue report template instead (#131). * If possible, keep the autoresponder for when component is changed, since these users will not have seen our issue report template. If that's not possible with BRE, then delete the autoresponder. We also agreed to keep the autoresponder enabled for reports created by ABRT if the incompatibility with ABRT can be fixed, but after reviewing the previous discussion, I don't think it's worth attempting to do this so long as ABRT remains unmaintained. So let's not worry about that.

blc commented

2026-05-12 16:35:30 +00:00

Hey folks, I have disabled the BRE rule for now. Sadly the system largely only works with current ticket state. More sophisticated transition comparisons aren't available.

catanzaro commented

2026-05-12 18:05:05 +00:00

Owner

OK, that's fine. Thank you! We will rely on the issue template only, then. Closing.

catanzaro closed this issue

2026-05-12 18:05:05 +00:00

Rows
Columns

Bugzilla "report upstream" script causes conflicts with ABRT reports #475