quality/paperwork
3
0
Issues 9 Projects Activity

openQA failure monitoring 2026 sprint 8 #37

New issue
Closed
opened 2026-04-21 01:36:35 +00:00 by adamwill · 15 comments
adamwill commented 2026-04-21 01:36:35 +00:00
Owner
Copy link

This is a process ticket to account for the ongoing task of openQA test failure monitoring and assessment in the agile/scrum process. We will create a ticket like this for each sprint. The ticket can potentially be used for co-ordination if that seems useful.

This is a process ticket to account for the ongoing task of openQA test failure monitoring and assessment in the agile/scrum process. We will create a ticket like this for each sprint. The ticket can potentially be used for co-ordination if that seems useful.
adamwill added this to the Sprint 8 project 2026-04-21 01:36:42 +00:00
adamwill commented 2026-04-21 22:28:21 +00:00
Author
Owner
Copy link
* reported another [ELN ppc64le dep issue](https://github.com/fedora-eln/eln/issues/487) * [updated a needle](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/218e266e40976eee84f23a08e373d3121f1c6038) for a new Firefox release
lruzicka commented 2026-04-22 12:06:08 +00:00
Owner
Copy link
  • re-ran a bunch tests for 20260421 and 20260422, some of the continued to fail repetitiously
  • Added a real screen change check to replace a simple sleep which were not helping on aarch64
  • Added a workaround to unstuck map files and force the to reload
* re-ran a bunch tests for 20260421 and 20260422, some of the continued to fail repetitiously * Added a real [screen change](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/f033d2c08882911865872ec08c3c109682f211d2) check to replace a simple `sleep` which were not helping on `aarch64` * Added [a workaround](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/ed3ab381392db35e709ab33eb83a8b36ed58df16) to unstuck map files and force the to reload
adamwill commented 2026-04-22 15:11:36 +00:00
Author
Owner
Copy link

Added a real screen change check to replace a simple sleep which were not helping on aarch64

Eh, I think this was working 99% of the time. Most of the failures I see are on blivet tests, where various clicks in blivet just...don't work. It's always on aarch64, never x86_64. There's probably a real, subtle bug lurking in GTK or something but it'd be a nightmare to debug, I think.

I've actually tried a few mitigations for this but there's just so many clicks in blivet, and some of them are annoyingly awkward to retry :(

> Added a real screen change check to replace a simple sleep which were not helping on aarch64 Eh, I think this was working 99% of the time. Most of the failures I see are on blivet tests, where various clicks in blivet just...don't work. It's *always* on aarch64, never x86_64. There's probably a real, subtle bug lurking in GTK or something but it'd be a nightmare to debug, I think. I've actually tried a few mitigations for this but there's just so many clicks in blivet, and some of them are annoyingly awkward to retry :(
adamwill commented 2026-04-22 16:26:35 +00:00
Author
Owner
Copy link

For the notification test we need to find a way to robustify it or just disable it again :( I'm wondering if having the test run as a new user, created after we do the package futzing, would help, so the user environment is fresh...

(or I guess there's an actual bug here? but...I suspect that's not it.)

* reported a [critical dep issue in samba](https://bodhi.fedoraproject.org/updates/FEDORA-2026-12549c6a9b#comment-4624717) * restarted a whole ton of flakes, mainly aarch64 but also the notification test sometimes just not finding any updates * [dropped concurrency on aarch64 workers from 60 to 45](https://forge.fedoraproject.org/infra/ansible/pulls/3298) to see if it maybe helps with the aarch64 flakes For the notification test we need to find a way to robustify it or just disable it again :( I'm wondering if having the test run as a new user, created after we do the package futzing, would help, so the user environment is fresh... (or I guess there's an actual bug here? but...I suspect that's not it.)
adamwill commented 2026-04-24 05:45:40 +00:00
Author
Owner
Copy link
* reported [unexpected installer changes](https://github.com/fedora-eln/eln/issues/490) in ELN with new release package, adjusted openQA for the package set change: [expect the new default package set](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/6131eb77a50d9fe2876034e6e55555f8134202e6), [bump disk sizes to cope with it](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/8095768e69617a2823c33217b2693feb837a1f54) * reported [broken webui installer theming](https://github.com/fedora-eln/eln/issues/493) in ELN with the new release package, made some new needles for minor button appearance changes but not committing them in case this gets fixed and the needles turn out to be unneeded * reported ELN installs [ultimately crash on a bootloader error](https://github.com/fedora-eln/eln/issues/494) after working around all other issues
adamwill commented 2026-04-24 05:50:51 +00:00
Author
Owner
Copy link

investigated gnome-initial-setup running on ELN GNOME live along with yselkowitz, wound up with a livesys-scripts PR

investigated gnome-initial-setup running on ELN GNOME live along with yselkowitz, wound up with a [livesys-scripts PR](https://pagure.io/livesys-scripts/pull-request/37)
adamwill commented 2026-04-27 23:34:55 +00:00
Author
Owner
Copy link
* kernel tests [still failing on podman quadlets](https://bodhi.fedoraproject.org/updates/FEDORA-2026-23d7d9ae9a#comment-4629794), tried the test with SELinux disabled at request of jforbes, it passed, he will take it from there
adamwill commented 2026-04-28 01:52:20 +00:00
Author
Owner
Copy link
  • tweaked things to work with ELN changing BaseOS to default to graphical desktop (ongoing) - scheduler, tests
  • updated a couple of maps needles
* tweaked things to work with ELN changing BaseOS to default to graphical desktop (ongoing) - [scheduler](https://forge.fedoraproject.org/quality/fedora_openqa/pulls/139), [tests](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/pulls/537) * updated a couple of [maps needles](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/9ecc999ebf683c5c385f697381b87f02943c75d2)
adamwill commented 2026-04-30 07:00:30 +00:00
Author
Owner
Copy link
  • disabled the podman quadlet test as we established it's the test that's broken not the kernel or SELinux
  • restarted some flakes
  • more staring at rmdepcheck failures on ELN, realized we need to be fancier about the ELN repo setup, sent alternative fixes
* [disabled](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/4e94e5ef66ee88f495203a48eeffc68e8c9e3484) the podman quadlet test as we established it's the test that's broken not the kernel or SELinux * restarted some flakes * more staring at rmdepcheck failures on ELN, [realized](https://forge.fedoraproject.org/quality/rmdepcheck/issues/37) we need to be fancier about the ELN repo setup, sent [alternative](https://forge.fedoraproject.org/quality/rmdepcheck/pulls/38) [fixes](https://forge.fedoraproject.org/quality/rmdepcheck/pulls/39)
adamwill commented 2026-05-01 06:47:02 +00:00
Author
Owner
Copy link
  • still trying to nail down rmdepcheck on ELN to behave exactly as yaakov wants, latest attempt is a WIP, will polish it more tomorrow
* still trying to nail down rmdepcheck on ELN to behave exactly as yaakov wants, [latest attempt](https://forge.fedoraproject.org/quality/rmdepcheck/pulls/40) is a WIP, will polish it more tomorrow
adamwill commented 2026-05-02 07:26:45 +00:00
Author
Owner
Copy link
* investigated ELN boot failures, turns out it's caused by [wrongly-signed kernel modules](https://bugzilla.redhat.com/show_bug.cgi?id=2464626). investigation ongoing
lruzicka commented 2026-05-04 10:35:59 +00:00
Owner
Copy link

checking failing tests, restarting them, adding needles, etc.

checking failing tests, restarting them, adding needles, etc.
adamwill commented 2026-05-04 20:36:12 +00:00
Author
Owner
Copy link
* reported [gpgme package split affecting ELN deps](https://github.com/fedora-eln/eln/issues/506)
lruzicka commented 2026-05-05 14:40:54 +00:00
Owner
Copy link
* reported https://gitlab.freedesktop.org/pipewire/pipewire/-/work_items/5259
adamwill commented 2026-05-05 15:21:33 +00:00
Author
Owner
Copy link

Sprint done.

Sprint done.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels
  
ai-review-please
Request AI review for a pull request

  
Backlog Status
Needs Review
 Work items that have been created or updated but need review, refinement, or approval before they can be considered ready for sprint planning.

  
Backlog Status
Ready
 Work items that have been reviewed, refined, and are ready to be committed to a sprint. These items have clear requirements and can be worked on immediately when capacity is available.

  
chore
Work that needs to be done but doesn't add new functionality or fix bugs. Maintenance tasks that keep the system running smoothly and the codebase clean.

  
documentation
Work focused on creating, updating, or improving documentation for users, developers, or system administrators.

  
points
01
 A task that is simple, small, and has very little uncertainty. It should be easy enough for any team member to pick up and complete quickly, without a lot of discussion or research.

  
points
02
 This is a task that is slightly more complex or larger than a "1." It might involve two or three simple steps, but the outcome is still very clear and there's little uncertainty.

  
points
03
 Medium-sized task with some noticeable complexity. It might require a bit more thought or collaboration. The steps are well-defined, but it's not a trivial change.

  
points
05
 A significant piece of work. It has more complexity and a higher degree of uncertainty than smaller tasks. It might require some research, a design discussion, or collaboration across multiple teams.

  
points
08
 Complex task with a lot of uncertainty. It's often the largest size that a team will consider for a single sprint. It likely requires a detailed breakdown and multiple team members to complete.

  
points
13
 Any task that feels like it's a "13" or more is considered an Epic. These are too large and uncertain to be worked on in a single sprint and should be broken down.

  
Priority
Critical
 This is for work that is urgent and of critical importance and so and must be addressed immediately with a higher priority than all other work, even if not originally planned for the current sprint.

  
Priority
High
 This is for work that is critical and needs to be addressed in the next sprint. It's often for urgent bugs, important features, or a major piece of technical debt that is blocking other work.

  
Priority
Low
 This is for work that is good to do but not urgent. It's a candidate for being worked on when there's extra time or when it rises in importance. It can stay in the backlog for a longer period.

  
Priority
Medium
 This is for important work that should be done soon, but isn't as urgent as a high-priority item. It will likely be considered for the next 1-2 sprints.

  
Sprint Status
Blocked
 Work items that cannot progress due to external dependencies, blockers, or issues that need resolution.

  
Sprint Status
Done
 Work items that have been completed, reviewed, and integrated. No further work is needed within the current sprint.

  
Sprint Status
In Progress
 Work items currently being worked on by a team member. Active development, research, or implementation is happening.

  
Sprint Status
Review
 Work items that are complete but need review, testing, or approval before being marked as done.

  
Sprint Status
To Do
 Work items that are committed to the sprint but haven't been started yet. These items are ready to be picked up by team members.

  
Technical Debt
Work that addresses shortcuts, workarounds, or suboptimal implementations that were done previously to meet deadlines. These items improve code quality, maintainability, or performance without adding new features.

  
Work Item
Bug
 Issues where the system is not working as intended or designed. Bugs represent deviations from expected behavior.

  
Work Item
Epic
 Large initiatives that are broken down into multiple smaller work items (user stories, tasks, bugs). Epics represent significant features or projects that span multiple sprints.

  
Work Item
Spike
 Time-boxed research, investigation, or proof-of-concept work to reduce uncertainty or explore solutions. Spikes help inform future development decisions.

  
Work Item
Task
 Used to break down larger user stories into smaller, manageable implementation pieces. They focus on the specific technical steps needed to deliver the user story's value.

  
Work Item
User Story
 User-facing features or functionality that can be framed from a user perspective, describing what someone wants to accomplish and why.

No labels
ai-review-please
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Sprint 8
Assignees
Clear assignees
No assignees
adamwill
lruzicka
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
quality/paperwork#37
No description provided.