[Epic] Improve Heroes of Fedora Quality tools and workflows #904

New issue

Open

opened 2026-05-18 15:08:19 +00:00 by kparal · 1 comment

kparal commented

2026-05-18 15:08:19 +00:00

Owner

Copy link

I'd like to use this ticket to collect improvement ideas for these statistics, so that we have a list of tasks to do (which can then be handled in individual tickets and PRs).

Related links: QA:Recognition, "Heroes of Fedora" at CommunityBlog and at kparal's blog.

After a few years of pause, I recently published stats for Fedora 44 (#897).

Ideas:

Publish the stats regularly. I'd like to see:
- Stats for each new Fedora release (related just to that release).
  Note: In the past, we used to split the release stats into Beta stats and Final stats. While that has PR advantages (reminds and motivates people after Beta to join Final testing), I'd probably drop it, because it generates additional load during the time where we struggle with available time the most.
- Quarterly stats (general numbers for all contributions)
Figure out where to publish the stats to have the best reach for audience. It might be the CommunityBlog, or maybe Fedora Discussion announcement category, or maybe something else (or multiple combined). Also take into consideration if we can generate an input format for that article, or whether it would involve manual copy&pasting and hand-editing of (especially) tables in some interactive tools (like the Wordpress UI). We want this process to be very efficient, if we are to publish the stats frequently and regularly, not a manual tedious process.
Separate data collection from output format generation. Currently most of our tools generate HTML that was tailored to some ancient Wordpress version. It's not easy to output it in a different format when needed. I'd like the tools to be able to produce something like CSV and/or json with raw data, all using the same format. And then we'd have a separate formatter script, that would convert it to specialized HTML or whatever we need for publishing (it would not just generate the output, but also handle setting thresholds for how many top contributors to include, etc).
Use a template to put the whole article together. We don't want to hand-craft each article, that's time consuming. We don't want the article to feel too generated either. But if we make it sound like the F44 stats, that could be good enough? Make sure the template does a good job of encouraging people to join, and provides the most important links so that people can learn how.
Update QA:Recognition. Some of the recommendations are likely outdated now, especially if we don't want to hand-craft each article. Also mention Schedules as a means to determine the right dates.
Drop remaining RH-branding code from scripts. It was well-meant, but it's not a recommended approach, let's just drop the code.
Try to synchronize on using FAS logins or full names. Some scripts produce this, some produce that. It might not be possible to achieve complete unification (e.g. Bugzilla users != FAS accounts), but it should be possible to make it more similar. Maybe the "real name discovery" could even be a separate step outside of the raw data generation, as a standalone post-process script.
Do something with referenced bugs often producing misleading numbers. For test cases validation, we add a "Referenced bugs" column. This only captures links to RH Bugzilla, nowhere else. In this case, mostly of the referenced bugs are likely to go to RHBZ, but not all. It provides a nice extra data that shows we also care about bug reports, not just the pure number of test cases executed. But in the test days stats, the referenced bugs section is completely off. Nowadays, most bugs likely point to upstream, and we don't track that at all. Which means the data are extremely misleading. They don't even render properly in the top testers table, and I haven't included it in the F44 stats post. We should either consider upstream bugs as well (at least the most common upstreams), or drop the feature. No numbers might be better than bad numbers.
In Bodhi stats, let's link the updates number to the actual comments feed of that user (that might encourage providing good feedback), e.g. https://bodhi.fedoraproject.org/comments/?user=kparal
In Bodhi stats, let's allow to query multiple releases together (or make individual calls and combine the numbers, but Bodhi seems to support multiple releases= params). The use case is that for quarterly stats, we'd like to query supported releases (and perhaps Branched), but not Rawhide (it has a ton of automated updates where people don't/can't submit feedback anyway, but it makes the query process extremely long when pulling in all Rawhide updates). It would also be good to support releases=__current__ param to quickly select all supported releases.
In Bugzilla stats, allow to generate combined stats for the whole release, not just relevant to a particular one milestone (like Beta or Final).
In Testdays stats, allow to print full output or set a threshold, not just the top (hardcoded) 10.
There's a little bit of discrepancy when collecting testdays stats for a particular release (it doesn't apply to quarter stats). Some testdays are done in Branched timeframe, but using a stable release as the baseline, even though the functionality is not tied to a particular release, e.g. these GRUB, GRUB part 2 and Shim test days. Similarly Kernel test weeks are also not tied to a specific release. It's a bit hard to assign it a wiki category then (F43? F44? F45? All of them?), which influences the stats numbers. We should re-think what exactly we want those categories to track (Is it related to that particular release, or did the test day occur during its development?), and whether there might be a different approach to this.
Figure out if we can add some additional stats to the currently collected set. For example, what about Bugzilla quarterly stats, do they make sense? And what about something completely new?
Look if we can award (even more) badges to our contributors.

Feel free to come up with more ideas, if you have any. Thanks.

If we want to approach new audience to help us out with testing Fedora, one area of improvement is to publish contributor statistics, which can motivate people to participate (or maybe expand their contribution to new areas), and also it helps our team to be more visible. I'd like to use this ticket to collect improvement ideas for these statistics, so that we have a list of tasks to do (which can then be handled in individual tickets and PRs). Related links: [QA:Recognition](https://fedoraproject.org/wiki/QA:Recognition), "Heroes of Fedora" at [CommunityBlog](https://communityblog.fedoraproject.org/?s=Heroes+of+Fedora) and at [kparal's blog](https://kparal.wordpress.com/category/fedora-qa/heroes-of-fedora-testing/). After a few years of pause, I recently published [stats for Fedora 44](https://kparal.wordpress.com/2026/05/15/heroes-of-fedora-quality-for-fedora-44/) (#897). Ideas: - [ ] Publish the stats regularly. I'd like to see: * Stats for each new Fedora release (related just to that release). Note: In the past, we used to split the release stats into Beta stats and Final stats. While that has PR advantages (reminds and motivates people after Beta to join Final testing), I'd probably drop it, because it generates additional load during the time where we struggle with available time the most. * Quarterly stats (general numbers for all contributions) - [ ] Figure out where to publish the stats to have the best reach for audience. It might be the CommunityBlog, or maybe Fedora Discussion announcement category, or maybe something else (or multiple combined). Also take into consideration if we can generate an input format for that article, or whether it would involve manual copy&pasting and hand-editing of (especially) tables in some interactive tools (like the Wordpress UI). We want this process to be very efficient, if we are to publish the stats frequently and regularly, not a manual tedious process. - [ ] Separate data collection from output format generation. Currently most of our tools generate HTML that was tailored to some ancient Wordpress version. It's not easy to output it in a different format when needed. I'd like the tools to be able to produce something like CSV and/or json with raw data, all using the same format. And then we'd have a separate formatter script, that would convert it to specialized HTML or whatever we need for publishing (it would not just generate the output, but also handle setting thresholds for how many top contributors to include, etc). - [ ] Use a template to put the whole article together. We don't want to hand-craft each article, that's time consuming. We don't want the article to feel too generated either. But if we make it sound like the [F44 stats](https://kparal.wordpress.com/2026/05/15/heroes-of-fedora-quality-for-fedora-44/), that could be good enough? Make sure the template does a good job of encouraging people to join, and provides the most important links so that people can learn how. - [ ] Update [QA:Recognition](https://fedoraproject.org/wiki/QA:Recognition). Some of the recommendations are likely outdated now, especially if we don't want to hand-craft each article. Also mention [Schedules](https://fedorapeople.org/groups/schedule/) as a means to determine the right dates. - [ ] Drop remaining RH-branding code from scripts. It was well-meant, but it's not a recommended approach, let's just drop the code. - [ ] Try to synchronize on using FAS logins or full names. Some scripts produce this, some produce that. It might not be possible to achieve complete unification (e.g. Bugzilla users != FAS accounts), but it should be possible to make it more similar. Maybe the "real name discovery" could even be a separate step outside of the raw data generation, as a standalone post-process script. - [ ] Do something with referenced bugs often producing misleading numbers. For test cases validation, we add a "Referenced bugs" column. This only captures links to RH Bugzilla, nowhere else. In this case, mostly of the referenced bugs are likely to go to RHBZ, but not all. It provides a nice extra data that shows we also care about bug reports, not just the pure number of test cases executed. But in the test days stats, the referenced bugs section is completely off. Nowadays, most bugs likely point to upstream, and we don't track that at all. Which means the data are extremely misleading. They don't even render properly in the top testers table, and I haven't included it in the F44 stats post. We should either consider upstream bugs as well (at least the most common upstreams), or drop the feature. No numbers might be better than bad numbers. - [ ] In Bodhi stats, let's link the updates number to the actual comments feed of that user (that might encourage providing good feedback), e.g. <https://bodhi.fedoraproject.org/comments/?user=kparal> - [ ] In Bodhi stats, let's allow to query multiple releases together (or make individual calls and combine the numbers, but Bodhi seems to support multiple `releases=` params). The use case is that for quarterly stats, we'd like to query supported releases (and perhaps Branched), but not Rawhide (it has a ton of automated updates where people don't/can't submit feedback anyway, but it makes the query process extremely long when pulling in all Rawhide updates). It would also be good to support [`releases=__current__`](https://bodhi.fedoraproject.org/updates/?releases=__current__) param to quickly select all supported releases. - [ ] In Bugzilla stats, allow to generate combined stats for the whole release, not just relevant to a particular one milestone (like Beta or Final). - [ ] In Testdays stats, allow to print full output or set a threshold, not just the top (hardcoded) 10. - [ ] There's a little bit of discrepancy when collecting testdays stats for a particular release (it doesn't apply to quarter stats). Some testdays are done in Branched timeframe, but using a stable release as the baseline, even though the functionality is not tied to a particular release, e.g. these [GRUB](https://fedoraproject.org/wiki/Test_Day:2025-12-15_GRUB_out_of_memory_verification), [GRUB part 2](https://fedoraproject.org/wiki/Test_Day:2026-02-09_GRUB_out_of_memory_fix_verification_part_2) and [Shim](https://fedoraproject.org/wiki/Test_Day:2026-01-12_Multi-signed_shim) test days. Similarly [Kernel test weeks](https://fedoraproject.org/wiki/Test_Day:2026-02-22_Kernel_6.19_Test_Week) are also not tied to a specific release. It's a bit hard to assign it a wiki category then (F43? F44? F45? All of them?), which influences the stats numbers. We should re-think what exactly we want those categories to track (Is it related to that particular release, or did the test day occur during its development?), and whether there might be a different approach to this. - [ ] Figure out if we can add some additional stats to the currently collected set. For example, what about Bugzilla quarterly stats, do they make sense? And what about something completely new? - [ ] Look if we can award (even more) badges to our contributors. Feel free to come up with more ideas, if you have any. Thanks.