quality/blockerbugs

Fork 0

Synchronize vote decisions between discussions tickets and Bugzilla #149

New issue

Open

opened 2020-11-16 15:18:19 +00:00 by kparal · 6 comments

kparal commented

2020-11-16 15:18:19 +00:00

Owner

There are issues to consider:

We only detect Bugzilla changes in ~30 minutes intervals, and so the vote in the ticket can change between a blocker was accepted in Bugzilla and before we detected it. The saved vote record (example) could then be incorrect. This doesn't occur when we close the vote in Pagure instead, because that is immediate (there can still be a race with multiple people sending a comment at roughly the same time, but the likelihood is much smaller). I place a quite high emphasis on having a trusted record of who voted how, and I'm not sure if this is OK.
The voting system in Pagure has a superset of information compared to the system in Bugzilla. For example, we can reject a BetaBlocker while accepting the FinalBlocker in Pagure. That can't be done with the Bugzilla system. So if we want to close the vote in Bugzilla, we won't be able to propagate these finer details. (We could think about redoing the Bugzilla keywords system, but that's a big change in itself).
It is definitely easier to provide a comment (especially multi-line) along with the decision in Bugzilla than to do it in Pagure and then pass it through. We currently have AGREED (Accepted|Rejected)TRACKER... [JUSTIFICATION] command, but that only allows a single-line justification, and for multi-line justifications the syntax would need to get more complex 😞
It's alluring to close the vote in the canonical location where the blocker data is stored (Bugzilla), rather than to do it in a different system and then hope all technology works as it is supposed to.

I'm not sure if there is a good solution here. Having two (or three) disjoint systems brings disadvantages. I'd love to hear your take on this, @adamwill (and anyone else's).

Note: This idea came up in the blockerbugs discussion tickets feedback.

Instead of manually updating the voting decision in both places (discussion tickets and Bugzilla), it would be nice to be able to update it in just one place and it would propagate to the other one. My initial idea was to close the vote in Pagure and let it propagate to Bugzilla. However, @adamwill suggested the opposite way. There are issues to consider: * We only detect Bugzilla changes in ~30 minutes intervals, and so the vote in the ticket can change between a blocker was accepted in Bugzilla and before we detected it. The saved vote record ([example](https://pagure.io/fedora-qa/blocker-review/issue/171#comment-696874)) could then be incorrect. This doesn't occur when we close the vote in Pagure instead, because that is immediate (there can still be a race with multiple people sending a comment at roughly the same time, but the likelihood is much smaller). I place a quite high emphasis on having a trusted record of who voted how, and I'm not sure if this is OK. * The voting system in Pagure has a superset of information compared to the system in Bugzilla. For example, we can reject a `BetaBlocker` while accepting the `FinalBlocker` in Pagure. That can't be done with the Bugzilla system. So if we want to close the vote in Bugzilla, we won't be able to propagate these finer details. (We could think about redoing the Bugzilla keywords system, but that's a big change in itself). * It is definitely easier to provide a comment (especially multi-line) along with the decision in Bugzilla than to do it in Pagure and then pass it through. We currently have `AGREED (Accepted|Rejected)TRACKER... [JUSTIFICATION]` command, but that only allows a single-line justification, and for multi-line justifications the syntax would need to get more complex :disappointed: * It's alluring to close the vote in the canonical location where the blocker data is stored (Bugzilla), rather than to do it in a different system and then hope all technology works as it is supposed to. I'm not sure if there is a good solution here. Having two (or three) disjoint systems brings disadvantages. I'd love to hear your take on this, @adamwill (and anyone else's). *Note: This idea came up in the [blockerbugs discussion tickets feedback](https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org/thread/CRCPMAHAFCK6M7AP5CBK7TBOHZY7SSBR/).*

kparal commented

2020-11-16 15:18:32 +00:00

Author

Owner

Issue tagged with: next

adamwill commented

2020-11-17 01:11:41 +00:00

Owner

"It's alluring to close the vote in the canonical location where the blocker data is stored (Bugzilla), rather than to do it in a different system and then hope all technology works as it is supposed to."

Yes. This is my prime reason for suggesting it that way around.

kparal commented

2020-11-19 10:53:52 +00:00

Author

Owner

@adamwill What are your thoughts on the first two bullet points in the ticket description? Do you consider it not a big issue?

If we decided to close votes in Bugzilla, we'd get rid of AGREED command in Pagure tickets. REVOTE could only be used for ongoing unclosed votes. Closing and reopening votes would only be possible through Bugzilla. There would be a delay to both.

I see a problem in how to decide which votes to allow in Pagure and which to consider closed (accepted or rejected). Currently, because everything occurs in Pagure (and later is forwarded to Bugzilla, sometimes not completely because of the keyword limitations), it's easy - you either see Accepted/Rejected (and your votes no longer count), or you can vote for any other tracker. It is clear to people what they're supposed to vote for. However, if we rely on Bugzilla to supply this information, there are problems. It might be unclear what to vote for.

Some changes are simple:

Bug contained Blocks: BetaBlocker, later it was removed -> Pagure ticket still allows voting for BetaBlocker, even though the nomination was retracted.
Bug contained Blocks: BetaBlocker, later it was removed and Whiteboard: RejectedBlocker was added -> Pagure ticket shows Rejected BetaBlocker and doesn't allow voting for it any more. The same version for Accepted.

But there are some pretty common cases in which it's not simple to define the requested behavior:
3. Bug contained Blocks: BetaBlocker FinalBlocker, later it was changed to Blocks: FinalBlocker Whiteboard: AcceptedBlocker. Does this mean the BetaBlocker was rejected, retracted, or is still in consideration? Because all options are valid sometimes. How should the Pagure ticket display this and behave when people vote?

You might say - just don't care about the semantics, and allow people to vote all the time. The smaller downside is that people might feel confused when the summary doesn't clearly indicate the current status. The bigger downside is that if we allow voting all the time, we'll then have a big problem to count the votes correctly. If it is not clear when exactly the vote started and became valid, counting all the votes in the ticket (including some very old ones) might show invalid results. We need to have a clear line "this is when voting starts" and "this is when voting ends".

The problem is even more visible in a case where a vote is completed (e.g. a blocker is rejected) and then started again after some time (e.g. it is re-proposed because of new data). So far, the whole counting process was a stateless machine, we didn't have any extra info stored anywhere. We always recounted the whole ticket (all comments), and those AGREED and REVOTE commands told us when to stop counting or reset the score. However, if we get rid of them, we have a problem. We must have a way to ignore the old votes. We can parse the Bugzilla ticket, which would slow down the blockerbot (what if Bugzilla is down at the moment?), and more importantly it's easy to learn the current state of tags from Bugzilla, but it's pretty difficult to learn the history - when exactly a bug was rejected, when it was re-proposed. Especially because humans sometimes make the change in several comments, make mistakes which they later correct, etc. It would be a hell to implement. Another approach is to save some metadata ourselves in blockerbugs webapp, but it is still far more complex then the current stateless machine. And far more error-prone. The last approach that comes to my mind is to inject our own commands as part of the blockerbot comments, when a change in Bugzilla is detected. So not only would it inform "BetaBlocker is now rejected", but it would also include AGREED RejectedBetaBlocker in its comment. We would then use these commands (no longer accessible to public, just for internal processing) to correctly count the votes. This is doable. But this still doesn't solve case 3) above.

I also considered removing the idea of "people are allowed to vote for anything (not accepted or rejected already) anytime". That would really simplify our handling when relying on Bugzilla as the source of current voting state. There would be a clear "Vote open for BetaBlocker" comment when the bug is proposed and "Vote closed for BetaBlocker" comment when there is some resolution or the nomination is retracted. However, that would prevent us from voting for a BetaBlocker exactly in the case 3) above. We would need a full syntax in Bugzilla that allows us to separate a BetaBlocker resolution from a FinalBlocker resolution.

So, ehm, sigh? The Bugzilla -> Pagure sync is currently doable, but it'll probably require lots of work and in some cases the voting might produce incorrect results. If we really want to keep this sync direction, I'm starting to think that it might be worth redoing our Bugzilla voting system first.

@adamwill How do you feel about changing the current Bugzilla voting system to e.g. flags (Blocks: F34Tracker Flags: fedora_beta_blocker=?/+/-) or extending whiteboard words (Blocks: F34Tracker Whiteboard: AcceptedBetaBlocker) or something else more expressive than what we currently have today?

@adamwill What are your thoughts on the first two bullet points in the ticket description? Do you consider it not a big issue? If we decided to close votes in Bugzilla, we'd get rid of `AGREED` command in Pagure tickets. `REVOTE` could only be used for ongoing unclosed votes. Closing and reopening votes would only be possible through Bugzilla. There would be a delay to both. I see a problem in how to decide which votes to allow in Pagure and which to consider closed (accepted or rejected). Currently, because everything occurs in Pagure (and later is forwarded to Bugzilla, sometimes not completely because of the keyword limitations), it's easy - you either see Accepted/Rejected (and your votes no longer count), or you can vote for any other tracker. It is clear to people what they're supposed to vote for. However, if we rely on Bugzilla to supply this information, there are problems. It might be unclear what to vote for. Some changes are simple: 1. Bug contained `Blocks: BetaBlocker`, later it was removed -> Pagure ticket still allows voting for BetaBlocker, even though the nomination was retracted. 2. Bug contained `Blocks: BetaBlocker`, later it was removed and `Whiteboard: RejectedBlocker` was added -> Pagure ticket shows `Rejected BetaBlocker` and doesn't allow voting for it any more. The same version for `Accepted`. But there are some pretty common cases in which it's not simple to define the requested behavior: 3. Bug contained `Blocks: BetaBlocker FinalBlocker`, later it was changed to `Blocks: FinalBlocker` `Whiteboard: AcceptedBlocker`. Does this mean the BetaBlocker was rejected, retracted, or is still in consideration? Because all options are valid sometimes. How should the Pagure ticket display this and behave when people vote? You might say - just don't care about the semantics, and allow people to vote all the time. The smaller downside is that people might feel confused when the summary doesn't clearly indicate the current status. The bigger downside is that if we allow voting all the time, we'll then have a big problem to count the votes correctly. If it is not clear when exactly the vote started and became valid, counting all the votes in the ticket (including some very old ones) might show invalid results. We need to have a clear line "this is when voting starts" and "this is when voting ends". The problem is even more visible in a case where a vote is completed (e.g. a blocker is rejected) and then started again after some time (e.g. it is re-proposed because of new data). So far, the whole counting process was a stateless machine, we didn't have any extra info stored anywhere. We always recounted the whole ticket (all comments), and those `AGREED` and `REVOTE` commands told us when to stop counting or reset the score. However, if we get rid of them, we have a problem. We must have a way to ignore the old votes. We can parse the Bugzilla ticket, which would slow down the blockerbot (what if Bugzilla is down at the moment?), and more importantly it's easy to learn the current state of tags from Bugzilla, but it's pretty difficult to learn the history - when exactly a bug was rejected, when it was re-proposed. Especially because humans sometimes make the change in several comments, make mistakes which they later correct, etc. It would be a hell to implement. Another approach is to save some metadata ourselves in blockerbugs webapp, but it is still far more complex then the current stateless machine. And far more error-prone. The last approach that comes to my mind is to inject our own commands as part of the blockerbot comments, when a change in Bugzilla is detected. So not only would it inform "BetaBlocker is now rejected", but it would also include `AGREED RejectedBetaBlocker` in its comment. We would then use these commands (no longer accessible to public, just for internal processing) to correctly count the votes. This is doable. But this still doesn't solve case 3) above. I also considered removing the idea of "people are allowed to vote for anything (not accepted or rejected already) anytime". That would really simplify our handling when relying on Bugzilla as the source of current voting state. There would be a clear "Vote open for BetaBlocker" comment when the bug is proposed and "Vote closed for BetaBlocker" comment when there is some resolution or the nomination is retracted. However, that would prevent us from voting for a BetaBlocker exactly in the case 3) above. We would need a full syntax in Bugzilla that allows us to separate a BetaBlocker resolution from a FinalBlocker resolution. So, ehm, sigh? The Bugzilla -> Pagure sync is currently doable, but it'll probably require lots of work and in some cases the voting might produce incorrect results. If we really want to keep this sync direction, I'm starting to think that it might be worth redoing our Bugzilla voting system first. @adamwill How do you feel about changing the current Bugzilla voting system to e.g. flags (`Blocks: F34Tracker Flags: fedora_beta_blocker=?/+/-`) or extending whiteboard words (`Blocks: F34Tracker` `Whiteboard: AcceptedBetaBlocker`) or something else more expressive than what we currently have today?

adamwill commented

2020-11-26 16:48:00 +00:00

Owner

"The voting system in Pagure has a superset of information compared to the system in Bugzilla" - I think you made a good case on the list for changing this on Bugzilla side, and this between-release time is the ideal time to do it. So I would support just doing that. I'll try and remember to put it on the agenda for Monday's meeting. Just adjusting the whiteboard syntax doesn't require action from anyone outside our team, only extending what blockerbugs can cope with and revising the wiki policy, so I think we could do it quite rapidly.

"We only detect Bugzilla changes in ~30 minutes intervals" - why? Why not use fedora-messaging messages, which typically get sent more or less immediately? But even so, this doesn't seem insurmountable: surely we can get the timestamp of the comment out of Bugzilla, and adjust parsing of the Pagure-side issue to ignore things from after that time? (OK, that's probably new code too). "The voting system in Pagure has a superset of information compared to the system in Bugzilla" - I think you made a good case on the list for changing this on Bugzilla side, and this between-release time is the ideal time to do it. So I would support just doing that. I'll try and remember to put it on the agenda for Monday's meeting. Just adjusting the whiteboard syntax doesn't require action from anyone outside our team, only extending what blockerbugs can cope with and revising the wiki policy, so I think we could do it quite rapidly.

kparal commented

2020-12-01 15:01:35 +00:00

Author

Owner

"We only detect Bugzilla changes in ~30 minutes intervals" - why? Why not use fedora-messaging messages, which typically get sent more or less immediately?

Because nobody implemented it, it's that easy :-)

But even so, this doesn't seem insurmountable: surely we can get the timestamp of the comment out of Bugzilla, and adjust parsing of the Pagure-side issue to ignore things from after that time? (OK, that's probably new code too).

Yes. But I tried to point out that Bugzilla will tell you "comment 7 contained $text", but won't tell you "the state of all fields was $value after comment 7 was submitted". And so you have to go comment by comment, and manually piece together the state of fields you're interested it.

Also, if we go by timestamps, it's then not obvious why some Pagure comment was not counted. It's obvious for the blockerbot, but how do we make it obvious to the human voters? Let's say that there are 7 comments in the ticket right now, but everything from comment 5 onward was submitted after the timestamp. If blockerbot parses Bugzilla after each Pagure comment is submitted, it'll be slow, but we can immediately close the vote and say "comment X was not counted because the vote was closed in Bugzilla at $timestamp". If we process the votes immediately as we do currently, and only update Bugzilla every 30 minutes, the votes will be initially counted, but then we'll disregard them and again submit a comment "comments X, Y, Z were not counted...". In both cases the user experience is worsened, unfortunately, and we have more wiggly parts that can break.

I'm not saying we can't do it, I'm just trying to find a way which is not a headache for us implementing it and a worse experience for the voters. Which includes identifying obstacles and there seems to be quite many here.

I think you made a good case on the list for changing this on Bugzilla side, and this between-release time is the ideal time to do it.

OK. Let's discuss it on the list. I'll try to think about all the BBA use cases when discussing that. Also, I'd like us to consider all the possible implementation paths in Bugzilla (whiteboard, keywords, flags, something else?), when we're already changing stuff. Some of that might be much easier to work with in a programmable manner than others.

> "We only detect Bugzilla changes in ~30 minutes intervals" - why? Why not use fedora-messaging messages, which typically get sent more or less immediately? Because nobody implemented it, it's that easy :-) > But even so, this doesn't seem insurmountable: surely we can get the timestamp of the comment out of Bugzilla, and adjust parsing of the Pagure-side issue to ignore things from after that time? (OK, that's probably new code too). Yes. But I tried to point out that Bugzilla will tell you "comment 7 contained $text", but won't tell you "the state of all fields was $value after comment 7 was submitted". And so you have to go comment by comment, and manually piece together the state of fields you're interested it. Also, if we go by timestamps, it's then not obvious why some Pagure comment was not counted. It's obvious for the blockerbot, but how do we make it obvious to the human voters? Let's say that there are 7 comments in the ticket right now, but everything from comment 5 onward was submitted after the timestamp. If blockerbot parses Bugzilla after each Pagure comment is submitted, it'll be slow, but we can immediately close the vote and say "comment X was not counted because the vote was closed in Bugzilla at $timestamp". If we process the votes immediately as we do currently, and only update Bugzilla every 30 minutes, the votes will be initially counted, but then we'll disregard them and again submit a comment "comments X, Y, Z were not counted...". In both cases the user experience is worsened, unfortunately, and we have more wiggly parts that can break. I'm not saying we can't do it, I'm just trying to find a way which is not a headache for us implementing it and a worse experience for the voters. Which includes identifying obstacles and there seems to be quite many here. > I think you made a good case on the list for changing this on Bugzilla side, and this between-release time is the ideal time to do it. OK. Let's discuss it on the list. I'll try to think about all the BBA use cases when discussing that. Also, I'd like us to consider all the possible implementation paths in Bugzilla (whiteboard, keywords, flags, something else?), when we're already changing stuff. Some of that might be much easier to work with in a programmable manner than others.

kparal commented

2025-09-03 12:39:33 +00:00

Author

Owner

Metadata Update from @kparal: