quality/tickets
9
3
Fork 0
Code Issues 64 Projects Activity Actions

Write guide to investigating openQA failures #896

New issue
Open
opened 2026-04-29 23:32:54 +00:00 by adamwill · 1 comment
adamwill commented 2026-04-29 23:32:54 +00:00
Owner
Copy link

We should document the basic procedure, expectations and techniques for investigating openQA test failures. This will help more people do it and provide cover instead of me mostly doing it all the time.

Key thing to note is priority: gating tests on Fedora updates (which is most of them) are most important, then non-gating Fedora update tests, then Fedora compose tests, then ELN update tests, then non-Fedora compose tests (IoT, CoreOS etc).

Also must note the core principle that all update test failures must be addressed. Flakes must be restarted. Real failures must be reported and a comment added linking to the report. Bugs in the test process must be addressed and related failures restarted. If a failure is introduced between tested updates, or by a waiver or something, it must be untagged (or, in rare cases, the test nobbled or disabled). Basically there should be no rows in https://openqa.fedoraproject.org/group_overview/2?limit_builds=100 without either the green (all passed) or grey (all failures commented) badges.

We should document the basic procedure, expectations and techniques for investigating openQA test failures. This will help more people do it and provide cover instead of me mostly doing it all the time. Key thing to note is priority: gating tests on Fedora updates (which is most of them) are most important, then non-gating Fedora update tests, then Fedora compose tests, then ELN update tests, then non-Fedora compose tests (IoT, CoreOS etc). Also must note the core principle that *all* update test failures must be addressed. Flakes must be restarted. Real failures must be reported and a comment added linking to the report. Bugs in the test process must be addressed and related failures restarted. If a failure is introduced between tested updates, or by a waiver or something, it must be untagged (or, in rare cases, the test nobbled or disabled). Basically there should be no rows in https://openqa.fedoraproject.org/group_overview/2?limit_builds=100 without either the green (all passed) or grey (all failures commented) badges.
lruzicka commented 2026-04-30 07:51:49 +00:00
Owner
Copy link

Yes, because I should have a talk on Flock with this exact topic. So it would be extremely helpful.

Yes, because I should have a talk on Flock with this exact topic. So it would be extremely helpful.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
No results found.
Labels
Clear labels   
agile
related to Scrum or other Agile methodology

  
anacondawebui
tickets relating to the anaconda 'web ui' interface refresh

  
arm

   
blockerfe
Issues relating to the blocker / freeze exception process

  
Closed As
Duplicate
 Closed with the reason of Duplicate

  
Closed As
Fixed
 Closed with the reason of Fixed

  
Closed As
Invalid
 Closed with the reason of Invalid

  
Closed As
Wontfix
 Closed with the reason of Wontfix

  
Closed As
Worksforme
 Closed with the reason of Worksforme

  
coreos
issue related to coreos

  
criteria
release criteria-related issues

  
defect

   
easyfix

   
enhancement

   
iot
Issues related to IoT

  
meeting
a topic for the next QA meeting

  
meta
issues in the issue tracker itself (like, whooooaaa, man)

  
onboarding call
issues that are used to track new contributor onboarding

  
proventesters

   
retrospective
issues filed as a result of QA Retrospective feedback

  
silverblue
Issues related to silverblue

  
sponsor
requests to become a FAS group sponsor

  
test cases

   
test days

   
wiki
  
ai-review-please
Request AI review for a pull request

  
Backlog Status
Needs Review
 Work items that have been created or updated but need review, refinement, or approval before they can be considered ready for sprint planning.

  
Backlog Status
Ready
 Work items that have been reviewed, refined, and are ready to be committed to a sprint. These items have clear requirements and can be worked on immediately when capacity is available.

  
chore
Work that needs to be done but doesn't add new functionality or fix bugs. Maintenance tasks that keep the system running smoothly and the codebase clean.

  
documentation
Work focused on creating, updating, or improving documentation for users, developers, or system administrators.

  
points
01
 A task that is simple, small, and has very little uncertainty. It should be easy enough for any team member to pick up and complete quickly, without a lot of discussion or research.

  
points
02
 This is a task that is slightly more complex or larger than a "1." It might involve two or three simple steps, but the outcome is still very clear and there's little uncertainty.

  
points
03
 Medium-sized task with some noticeable complexity. It might require a bit more thought or collaboration. The steps are well-defined, but it's not a trivial change.

  
points
05
 A significant piece of work. It has more complexity and a higher degree of uncertainty than smaller tasks. It might require some research, a design discussion, or collaboration across multiple teams.

  
points
08
 Complex task with a lot of uncertainty. It's often the largest size that a team will consider for a single sprint. It likely requires a detailed breakdown and multiple team members to complete.

  
points
13
 Any task that feels like it's a "13" or more is considered an Epic. These are too large and uncertain to be worked on in a single sprint and should be broken down.

  
pr2jira
Sync this pull request to Jira for Agility tracking. Only use for PRs without a corresponding ticket.

  
Priority
Critical
 This is for work that is urgent and of critical importance and so and must be addressed immediately with a higher priority than all other work, even if not originally planned for the current sprint.

  
Priority
High
 This is for work that is critical and needs to be addressed in the next sprint. It's often for urgent bugs, important features, or a major piece of technical debt that is blocking other work.

  
Priority
Low
 This is for work that is good to do but not urgent. It's a candidate for being worked on when there's extra time or when it rises in importance. It can stay in the backlog for a longer period.

  
Priority
Medium
 This is for important work that should be done soon, but isn't as urgent as a high-priority item. It will likely be considered for the next 1-2 sprints.

  
Sprint Status
Blocked
 Work items that cannot progress due to external dependencies, blockers, or issues that need resolution.

  
Sprint Status
Done
 Work items that have been completed, reviewed, and integrated. No further work is needed within the current sprint.

  
Sprint Status
In Progress
 Work items currently being worked on by a team member. Active development, research, or implementation is happening.

  
Sprint Status
Review
 Work items that are complete but need review, testing, or approval before being marked as done.

  
Sprint Status
To Do
 Work items that are committed to the sprint but haven't been started yet. These items are ready to be picked up by team members.

  
Technical Debt
Work that addresses shortcuts, workarounds, or suboptimal implementations that were done previously to meet deadlines. These items improve code quality, maintainability, or performance without adding new features.

  
Work Item
Bug
 Issues where the system is not working as intended or designed. Bugs represent deviations from expected behavior.

  
Work Item
Epic
 Large initiatives that are broken down into multiple smaller work items (user stories, tasks, bugs). Epics represent significant features or projects that span multiple sprints.

  
Work Item
Spike
 Time-boxed research, investigation, or proof-of-concept work to reduce uncertainty or explore solutions. Spikes help inform future development decisions.

  
Work Item
Task
 Used to break down larger user stories into smaller, manageable implementation pieces. They focus on the specific technical steps needed to deliver the user story's value.

  
Work Item
User Story
 User-facing features or functionality that can be framed from a user perspective, describing what someone wants to accomplish and why.

No labels
agile
anacondawebui
arm
blockerfe
Closed As
Duplicate
Closed As
Fixed
Closed As
Invalid
Closed As
Wontfix
Closed As
Worksforme
coreos
criteria
defect
easyfix
enhancement
iot
meeting
meta
onboarding call
proventesters
retrospective
silverblue
sponsor
test cases
test days
wiki
ai-review-please
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
pr2jira
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
adamwill
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
quality/tickets#896
No description provided.