quality/planning
3
0
Issues 12 Projects Activity

openQA failure monitoring 2026 sprint 14 #87

New issue
Closed
opened 2026-08-11 17:41:54 +00:00 by adamwill · 17 comments
adamwill commented 2026-08-11 17:41:54 +00:00
Owner
Copy link

This is a process ticket to account for the ongoing task of openQA test failure monitoring and assessment in the agile/scrum process. We will create a ticket like this for each sprint. The ticket can potentially be used for co-ordination if that seems useful.

This is a process ticket to account for the ongoing task of openQA test failure monitoring and assessment in the agile/scrum process. We will create a ticket like this for each sprint. The ticket can potentially be used for co-ordination if that seems useful.
adamwill added this to the Sprint 14 project 2026-08-11 17:42:00 +00:00
adamwill commented 2026-08-18 00:41:59 +00:00
Author
Owner
Copy link
  • fixed tests for repo configs moving to /usr/share
  • fixed kiwi_build test for repo GPG keys moving to /usr/share
  • standard needle updates for branching (new identification needles, background needles, pre-release note needles)
  • worked around freeipa webui user entry field not always being active
  • worked around console_shutdown failures when tty switching
  • gave ostree installs longer to complete as they were just barely timing out after the move to webui
  • updated firefox needles for new release
  • fixed a couple of issues in ostree_build on f45+
  • (re-)generated 45 and 46 base images for branching
* [fixed tests](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/4c2e1407c818f810fa5a179c044836eea10c1ebd) for repo configs moving to /usr/share * [fixed kiwi_build test](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/8815ba4679f472950e43611fc437bac73bd4b345) for repo GPG keys moving to /usr/share * standard needle updates for branching (new identification needles, background needles, pre-release note needles) * [worked around](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/b60a7f3f824abbc16b3d8aeafe7fc3dece372293) freeipa webui user entry field not always being active * [worked around](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/7575713bec68173738c93cec83ec8af7f6d2737a) console_shutdown failures when tty switching * [gave ostree installs longer to complete](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/f04640f1ee3f204d274ca0a475721f0eb0487701) as they were just barely timing out after the move to webui * [updated](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/609af5abb110af4e42c20d5fc9aba591d8fb4a01) firefox needles for new release * [fixed](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/d4a0113c72a6f1d0ad4b33612b913934038e157b) a couple of issues in ostree_build on f45+ * (re-)generated 45 and 46 base images for branching
adamwill commented 2026-08-18 15:58:24 +00:00
Author
Owner
Copy link
* investigated and reported a dnf5 update that causes [packagekitd to crash](https://bodhi.fedoraproject.org/updates/FEDORA-2026-008372390d#comment-4737561)
adamwill commented 2026-08-19 00:16:02 +00:00
Author
Owner
Copy link
* reported an update with [incorrect version](https://bodhi.fedoraproject.org/updates/FEDORA-2026-57e97d4977#comment-4737975)
adamwill commented 2026-08-19 18:52:02 +00:00
Author
Owner
Copy link
  • reported anaconda-webui f45 and f46 builds having dep issues on s390x
  • investigated failures on FreeIPA upgrade tests for F44 updates, realized FreeIPA on F43 had been bumped to a higher EVR than F44/F45/F46, this causes upgrades to fail; did bumped builds for 44, 45 and 46 and added them to workarounds until they go stable
  • reported dep issue in ELN update (ELN needs to pull in a split-out source package)
* reported anaconda-webui [f45](https://bodhi.fedoraproject.org/updates/FEDORA-2026-832676488a#comment-4739088) and [f46](https://bodhi.fedoraproject.org/updates/FEDORA-2026-f16d8af1dc#comment-4739089) builds having dep issues on s390x * investigated failures on FreeIPA upgrade tests for F44 updates, realized FreeIPA on F43 had been bumped to a higher EVR than F44/F45/F46, this causes upgrades to fail; did bumped builds for [44](https://bodhi.fedoraproject.org/updates/FEDORA-2026-a8d45d82aa), [45](https://bodhi.fedoraproject.org/updates/FEDORA-2026-b85e45ea00) and [46](https://bodhi.fedoraproject.org/updates/FEDORA-2026-06fdf04532) and [added them to workarounds](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/01f2173907f9bfd8d9b263e3a57d33d2820d62ae) until they go stable * [reported](https://bodhi.fedoraproject.org/updates/FEDORA-2026-3b110939fd#comment-4739087) dep issue in ELN update (ELN needs to pull in a split-out source package)
adamwill commented 2026-08-19 18:52:50 +00:00
Author
Owner
Copy link

reporting for lruzicka: he fixed the scale view check in the gnome-maps test and updated many needles to fix test failures in quality/os-autoinst-distri-fedora@7e2252187a and three subsequent commits

reporting for lruzicka: he [fixed the scale view check in the gnome-maps test](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/e6ef6661c5f4acd7c95c6f58700fbb315155eec4) and updated many needles to fix test failures in https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/7e2252187ad0cc98d5d531055faa5f989c335d2d and three subsequent commits
lruzicka commented 2026-08-24 12:51:41 +00:00
Owner
Copy link

Somehow, the scale view was using ctrl-s instead of ctrl-shift-s, so the scale module was failing today. I fixed it and made it pass. I restarted a lot of tests that failed due to mistypes or slowness of the GUI and I could bring the count of failures down significantly.

Somehow, the scale view was using `ctrl-s` instead of `ctrl-shift-s`, so the scale module was failing today. I fixed it and made it pass. I restarted a lot of tests that failed due to mistypes or slowness of the GUI and I could bring the count of failures down significantly.
adamwill commented 2026-08-24 17:12:22 +00:00
Author
Owner
Copy link

I changed it to ctrl-s because I saw failures where ctrl-shift-s didn't work, and I tested on my own system and found ctrl-s works but ctrl-shift-s doesn't.

edit: aha, it looks like it's a flatpak thing. The current flatpak version needs ctrl-s. The current packaged version (on F45/Rawhide at least, I guess) needs ctrl-shift-s.

I [changed it to ctrl-s](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/03de19ab985b7382ee543792ad63565d4c6e712c) because I saw failures where ctrl-shift-s didn't work, and I tested on my own system and found ctrl-s works but ctrl-shift-s doesn't. edit: aha, it looks like it's a flatpak thing. The current flatpak version needs ctrl-s. The current packaged version (on F45/Rawhide at least, I guess) needs ctrl-shift-s.
lruzicka commented 2026-08-24 18:11:30 +00:00
Owner
Copy link

@adamwill wrote in #87 (comment):

I changed it to ctrl-s because I saw failures where ctrl-shift-s didn't work, and I tested on my own system and found ctrl-s works but ctrl-shift-s doesn't.

edit: aha, it looks like it's a flatpak thing. The current flatpak version needs ctrl-s. The current packaged version (on F45/Rawhide at least, I guess) needs ctrl-shift-s.

oh my, shall I report it?

@adamwill wrote in https://forge.fedoraproject.org/quality/planning/issues/87#issuecomment-1345282: > I [changed it to ctrl-s](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/03de19ab985b7382ee543792ad63565d4c6e712c) because I saw failures where ctrl-shift-s didn't work, and I tested on my own system and found ctrl-s works but ctrl-shift-s doesn't. > > edit: aha, it looks like it's a flatpak thing. The current flatpak version needs ctrl-s. The current packaged version (on F45/Rawhide at least, I guess) needs ctrl-shift-s. oh my, shall I report it?
adamwill commented 2026-08-24 18:13:42 +00:00
Author
Owner
Copy link

no, it'll just be the usual version desync thing - the flatpak is older. they changed the shortcut somewhere between the version that's in the flatpak, and the version in f45/rawhide. flatpaks are built from the current stable release so this happens somewhat frequently. when the flatpak gets rebased on f45 shortly before f45 release, the difference should go away.

I've changed the test to use different shortcuts depending on the CANNED variable, hopefully that'll sort it finally.

no, it'll just be the usual version desync thing - the flatpak is older. they changed the shortcut somewhere between the version that's in the flatpak, and the version in f45/rawhide. flatpaks are built from the current stable release so this happens somewhat frequently. when the flatpak gets rebased on f45 shortly before f45 release, the difference should go away. I've [changed the test](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/fe20687422ca28847497d80a03a603a6d8b8a3c3) to use different shortcuts depending on the CANNED variable, hopefully that'll sort it finally.
adamwill commented 2026-08-24 22:36:49 +00:00
Author
Owner
Copy link
* reported anaconda-webui update [boots to a PIN screen](https://bodhi.fedoraproject.org/updates/FEDORA-2026-ea7c31fcfb#comment-4744943) * [hacked around](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/b2196d812da02c3acf34f369a4850fb0073746e9) an [update that broke libreoffice](https://bodhi.fedoraproject.org/updates/FEDORA-2026-4cf69c401e) * [tweaked](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/8b4cdbc174c2371cf28b01e028f9f19ac7a4ec17) my 'wait for language select box to go active' workaround to retry the click if it doesn't work first time
adamwill commented 2026-08-25 17:03:29 +00:00
Author
Owner
Copy link
  • reported qt6 updates booting to a black screen - F44, F45, F46
* reported qt6 updates booting to a black screen - [F44](https://bodhi.fedoraproject.org/updates/FEDORA-2026-d13f91ad0e#comment-4745982), [F45](https://bodhi.fedoraproject.org/updates/FEDORA-2026-5013479509#comment-4745986), [F46](https://bodhi.fedoraproject.org/updates/FEDORA-2026-c550ab8b1b#comment-4745988)
adamwill commented 2026-08-26 00:20:45 +00:00
Author
Owner
Copy link
  • reported kernel updates failing to boot on aarch64 and breaking x86_64 BIOS installs - Bodhi, BZ
  • reported updated qt6 updates now crashing
* reported kernel updates failing to boot on aarch64 and breaking x86_64 BIOS installs - [Bodhi](https://bodhi.fedoraproject.org/updates/FEDORA-2026-f9c6b692f1#comment-4742806), [BZ](https://bugzilla.redhat.com/show_bug.cgi?id=2523734) * reported updated qt6 updates [now crashing](https://bodhi.fedoraproject.org/updates/FEDORA-2026-d13f91ad0e#comment-4746126)
lruzicka commented 2026-08-26 11:54:49 +00:00
Owner
Copy link
  • helped Katerina find specific test jobs for the bug 2521241
  • checked failed tests on updates, provided needles where neccessary
* helped Katerina find specific test jobs for the bug 2521241 * checked failed tests on updates, provided needles where neccessary
adamwill commented 2026-08-27 00:08:14 +00:00
Author
Owner
Copy link
* lots of investigation on the mysterious 'discover doesn't start' bug, got enough to [file it as a bug](https://bugzilla.redhat.com/show_bug.cgi?id=2524692), now thanks to Simo we have a working theory of the cause (pkcs11-provider change) * built a silverblue [installer image](https://adamwill.fedorapeople.org/06880571-149504956-Silverblue-ostree-x86_64.iso) with a debugging build of rust-bootupd to investigate the [install failure bug](https://github.com/coreos/bootupd/issues/1139) * also built an image with more debugging to investigate the [ostree webui installs sometimes get stuck](https://bugzilla.redhat.com/show_bug.cgi?id=2521241) bug
adamwill commented 2026-08-27 21:41:29 +00:00
Author
Owner
Copy link
* further research on [discover doesn't start](https://bugzilla.redhat.com/show_bug.cgi?id=2524692#c9), [bootupd 0.2.36 breaks installs](https://github.com/coreos/bootupd/issues/1139#issuecomment-5444917757), and [ostree webui installs get stuck](https://bugzilla.redhat.com/show_bug.cgi?id=2521241#c5) bugs * updated [more KDE needles](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/9b86482d712a84354dc1cfaabd3b2a5779610031) * reported an [ntfs-3g update which broke deps](https://bodhi.fedoraproject.org/updates/FEDORA-2026-c14e47bb59#comment-4749802)
adamwill commented 2026-09-04 00:34:33 +00:00
Author
Owner
Copy link
  • reported multiple rawhide kernel builds having apparently broken input on aarch64 - first report
  • reported krb5 updates breaking freeipa (same issue across releases)
  • reported systemd-coredumpd broken in a systemd update due to selinux denial
  • reported GNOME 51 RC updates seemingly breaking anaconda gtkui interaction
  • extended a timeout to avoid some flaky failures
  • worked around broken openh264 rawhide repo for ostree rebase test
  • made desktop_notifications a soft fail on x86_64 (as well as aarch64) on f43 as it's also become flaky and I don't have enough time to work out why
  • updated several needles for f45 candidate composes (some checks only run on candidates, not nightlies)
* reported multiple rawhide kernel builds having apparently broken input on aarch64 - [first report](https://bodhi.fedoraproject.org/updates/FEDORA-2026-e5176fe863#comment-4755014) * [reported](https://bodhi.fedoraproject.org/updates/FEDORA-2026-b887a90f4b#comment-4756801) krb5 updates breaking freeipa (same issue across releases) * [reported](https://bodhi.fedoraproject.org/updates/FEDORA-2026-0d926540f9#comment-4759821) systemd-coredumpd broken in a systemd update due to selinux denial * [reported](https://bodhi.fedoraproject.org/updates/FEDORA-2026-494d8c11bd#comment-4760062) GNOME 51 RC updates seemingly breaking anaconda gtkui interaction * [extended a timeout](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/5fe2307cda5e025941e78ac78c00df38af6e8a7c) to avoid some flaky failures * [worked around](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/3dc05e2f47a5f57d460e42b4f2ede8bb3bda0698) broken openh264 rawhide repo for ostree rebase test * [made desktop_notifications a soft fail](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/7a639655f13fbf088117e73e29605d419e9af663) on x86_64 (as well as aarch64) on f43 as it's also become flaky and I don't have enough time to work out why * [updated several needles](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/ad1736bd11d719e56f4dc6f806f78a9dcb97b2dd) for f45 candidate composes (some checks only run on candidates, not nightlies)
adamwill commented 2026-09-04 01:13:30 +00:00
Author
Owner
Copy link
  • updated console initial setup needles for kmscon (now it finally works again)
* [updated](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/518e3c73063f192f271d9385e4f47aef76fc6372) console initial setup needles for kmscon (now it finally works again)
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels
  
ai-review-please
Request AI review for a pull request

  
Backlog Status
Needs Review
 Work items that have been created or updated but need review, refinement, or approval before they can be considered ready for sprint planning.

  
Backlog Status
Ready
 Work items that have been reviewed, refined, and are ready to be committed to a sprint. These items have clear requirements and can be worked on immediately when capacity is available.

  
chore
Work that needs to be done but doesn't add new functionality or fix bugs. Maintenance tasks that keep the system running smoothly and the codebase clean.

  
documentation
Work focused on creating, updating, or improving documentation for users, developers, or system administrators.

  
points
01
 A task that is simple, small, and has very little uncertainty. It should be easy enough for any team member to pick up and complete quickly, without a lot of discussion or research.

  
points
02
 This is a task that is slightly more complex or larger than a "1." It might involve two or three simple steps, but the outcome is still very clear and there's little uncertainty.

  
points
03
 Medium-sized task with some noticeable complexity. It might require a bit more thought or collaboration. The steps are well-defined, but it's not a trivial change.

  
points
05
 A significant piece of work. It has more complexity and a higher degree of uncertainty than smaller tasks. It might require some research, a design discussion, or collaboration across multiple teams.

  
points
08
 Complex task with a lot of uncertainty. It's often the largest size that a team will consider for a single sprint. It likely requires a detailed breakdown and multiple team members to complete.

  
points
13
 Any task that feels like it's a "13" or more is considered an Epic. These are too large and uncertain to be worked on in a single sprint and should be broken down.

  
pr2jira
Sync this pull request to Jira for Agility tracking. Only use for PRs without a corresponding ticket.

  
Priority
Critical
 This is for work that is urgent and of critical importance and so and must be addressed immediately with a higher priority than all other work, even if not originally planned for the current sprint.

  
Priority
High
 This is for work that is critical and needs to be addressed in the next sprint. It's often for urgent bugs, important features, or a major piece of technical debt that is blocking other work.

  
Priority
Low
 This is for work that is good to do but not urgent. It's a candidate for being worked on when there's extra time or when it rises in importance. It can stay in the backlog for a longer period.

  
Priority
Medium
 This is for important work that should be done soon, but isn't as urgent as a high-priority item. It will likely be considered for the next 1-2 sprints.

  
Sprint Status
Blocked
 Work items that cannot progress due to external dependencies, blockers, or issues that need resolution.

  
Sprint Status
Done
 Work items that have been completed, reviewed, and integrated. No further work is needed within the current sprint.

  
Sprint Status
In Progress
 Work items currently being worked on by a team member. Active development, research, or implementation is happening.

  
Sprint Status
Review
 Work items that are complete but need review, testing, or approval before being marked as done.

  
Sprint Status
To Do
 Work items that are committed to the sprint but haven't been started yet. These items are ready to be picked up by team members.

  
Technical Debt
Work that addresses shortcuts, workarounds, or suboptimal implementations that were done previously to meet deadlines. These items improve code quality, maintainability, or performance without adding new features.

  
Work Item
Bug
 Issues where the system is not working as intended or designed. Bugs represent deviations from expected behavior.

  
Work Item
Epic
 Large initiatives that are broken down into multiple smaller work items (user stories, tasks, bugs). Epics represent significant features or projects that span multiple sprints.

  
Work Item
Spike
 Time-boxed research, investigation, or proof-of-concept work to reduce uncertainty or explore solutions. Spikes help inform future development decisions.

  
Work Item
Task
 Used to break down larger user stories into smaller, manageable implementation pieces. They focus on the specific technical steps needed to deliver the user story's value.

  
Work Item
User Story
 User-facing features or functionality that can be framed from a user perspective, describing what someone wants to accomplish and why.

No labels
ai-review-please
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
pr2jira
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Sprint 14
Assignees
Clear assignees
No assignees
adamwill
lruzicka
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
quality/planning#87
No description provided.