releng/tickets
15
3
Fork 2
Code Issues 86 Pull requests Projects Releases Packages Wiki Activity Actions

Untag haveged-1.9.24-1.fc45 #13393

New issue
Closed
opened 2026-06-21 15:08:56 +00:00 by adamwill · 10 comments
adamwill commented 2026-06-21 15:08:56 +00:00
Member
Copy link

Describe the issue

Please untag haveged-1.9.24-1.fc45 from Rawhide. I suspect it's causing the failures we've been seeing for the FreeIPA server upgrade test on all Rawhide updates.

When do you need this? (YYYY/MM/DD)

ASAP.

When is this no longer needed or useful? (YYYY/MM/DD)

When the bug is fixed (assuming the untag confirms haveged is really at fault).

If we cannot complete this, what is the impact? [Dependencies/Blocker]

FreeIPA upgrade tests will continue to fail and all Rawhide updates will be gated.

Checklist

  • I have checked existing issues for duplicates
  • I have filled out all the fields above
  • I have provided relevant links or references (if applicable)
## Describe the issue Please untag haveged-1.9.24-1.fc45 from Rawhide. I suspect it's causing the failures we've been seeing for the FreeIPA server upgrade test on all Rawhide updates. ## When do you need this? (YYYY/MM/DD) ASAP. ## When is this no longer needed or useful? (YYYY/MM/DD) When the bug is fixed (assuming the untag confirms haveged is really at fault). ## If we cannot complete this, what is the impact? [Dependencies/Blocker] FreeIPA upgrade tests will continue to fail and all Rawhide updates will be gated. --- ## Checklist - [X] I have checked existing issues for duplicates - [X] I have filled out all the fields above - [X] I have provided relevant links or references (if applicable)
kevin commented 2026-06-21 15:13:20 +00:00
Owner
Copy link

How far back? Just that one?

Wed May 20 15:54:20 2026 haveged-1.9.22-1.fc45 tagged into f45 by bodhi
Wed Jun 17 18:37:33 2026 haveged-1.9.23-1.fc45 tagged into f45 by bodhi
Thu Jun 18 20:02:56 2026 haveged-1.9.23-2.fc45 tagged into f45 by bodhi [still active]
Thu Jun 18 22:14:14 2026 haveged-1.9.23-3.fc45 tagged into f45 by bodhi [still active]
Fri Jun 19 08:04:45 2026 haveged-1.9.24-1.fc45 tagged into f45 by bodhi [still active]

The previous 2 still active were built just the day before...

How far back? Just that one? Wed May 20 15:54:20 2026 haveged-1.9.22-1.fc45 tagged into f45 by bodhi Wed Jun 17 18:37:33 2026 haveged-1.9.23-1.fc45 tagged into f45 by bodhi Thu Jun 18 20:02:56 2026 haveged-1.9.23-2.fc45 tagged into f45 by bodhi [still active] Thu Jun 18 22:14:14 2026 haveged-1.9.23-3.fc45 tagged into f45 by bodhi [still active] Fri Jun 19 08:04:45 2026 haveged-1.9.24-1.fc45 tagged into f45 by bodhi [still active] The previous 2 still active were built just the day before...
kevin self-assigned this 2026-06-21 15:13:54 +00:00
kevin commented 2026-06-21 15:21:57 +00:00
Owner
Copy link

I guess I'll just do haveged-1.9.24-1.fc45 and we can leave this open to go from there.

I guess I'll just do haveged-1.9.24-1.fc45 and we can leave this open to go from there.
adamwill commented 2026-06-21 15:59:20 +00:00
Author
Member
Copy link

Sorry, I had to run out. Yes, please go one at a time. I think the previous builds were likely ok because of when the tests started failing, but we'll see.

I've confirmed this is definitely haveged - I forced openQA to run on the f44 update and it failed. I'm on phone for the rest of the night but I'll try to check in. Can you do a wait-repo for the untag? I can't do it from phone. Then we can re-run a failed rawhide test and see what happens.

Sorry, I had to run out. Yes, please go one at a time. I *think* the previous builds were likely ok because of when the tests started failing, but we'll see. I've confirmed this is definitely haveged - I forced openQA to run on the f44 update and it failed. I'm on phone for the rest of the night but I'll try to check in. Can you do a wait-repo for the untag? I can't do it from phone. Then we can re-run a failed rawhide test and see what happens.
kevin commented 2026-06-21 16:03:45 +00:00
Owner
Copy link

done. latest build is haveged-1.9.23-3.fc45

done. latest build is haveged-1.9.23-3.fc45
adamwill commented 2026-06-22 00:13:11 +00:00
Author
Member
Copy link

So something is weird here. The untag didn't work - tests are still failing. And the 'last known good' test seems to have used haveged-1.9.23-3.fc45 , at least going by the logs. But every other test I've tried indicates that haveged is the issue. See the forced test of the 1.9.24 build for f44, which fails the same way. I also did some local tests: I installed F44, deployed it as a FreeIPA server, took a snapshot, upgraded to F45, and it reproduced the bug (booted to emergency mode after upgrade). Then I restored the snapshot and did the same upgrade only with haveged skipped; this time the upgraded system boots fine, and I verified the only non-upgraded package is haveged.

So something is weird here, but maybe we should untag a few more and see what happens. I'm also testing a run with haveged omitted (we use it to ensure there's enough entropy for the FreeIPA deployment, it's not a strict requirement).

So something is weird here. The untag didn't work - tests are still failing. And the ['last known good' test](https://openqa.fedoraproject.org/tests/4883625) seems to have used haveged-1.9.23-3.fc45 , at least going by the logs. But every other test I've tried indicates that haveged is the issue. See the [forced test of the 1.9.24 build for f44](https://openqa.stg.fedoraproject.org/tests/6567612), which fails the same way. I also did some local tests: I installed F44, deployed it as a FreeIPA server, took a snapshot, upgraded to F45, and it reproduced the bug (booted to emergency mode after upgrade). Then I restored the snapshot and did the same upgrade only with haveged skipped; this time the upgraded system boots fine, and I verified the only non-upgraded package is haveged. So something is weird here, but maybe we should untag a few more and see what happens. I'm also testing a run with haveged omitted (we use it to ensure there's enough entropy for the FreeIPA deployment, it's not a strict requirement).
adamwill commented 2026-06-22 00:24:40 +00:00
Author
Member
Copy link

The run with haveged left out seems to have passed. I guess I'll take it out in prod at least temporarily too.

The [run with haveged left out](https://openqa.stg.fedoraproject.org/tests/6567753#) seems to have passed. I guess I'll take it out in prod at least temporarily too.
adamwill commented 2026-06-22 14:40:17 +00:00
Author
Member
Copy link

I took haveged out of the test in prod and it's now passing, which further suggests haveged is the issue.

I suppose we should try untagging more build(s) and re-testing on staging with haveged added back to the test?

I took haveged out of the test [in prod](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/ce142e04f4e02165e5fbd4fc12da939f91cfeccf) and it's now passing, which further suggests haveged is the issue. I suppose we should try untagging more build(s) and re-testing on staging with haveged added back to the test?
adamwill commented 2026-06-23 10:32:27 +00:00
Author
Member
Copy link

Oh, hey, actually: I just tried running the test with haveged enabled again on staging, and it works now. So it looks like just untagging 1.9.24 might have worked after all, possibly I re-tested too early before or something.

I'll re-test that a 1.9.24 upgrade still fails, and if so, we're probably OK here.

Oh, hey, actually: I just tried running the test with haveged enabled again on staging, and it works now. So it looks like just untagging 1.9.24 might have worked after all, possibly I re-tested too early before or something. I'll re-test that a 1.9.24 upgrade still fails, and if so, we're probably OK here.
jnsamyak commented 2026-06-23 10:58:16 +00:00
Owner
Copy link

fun things happens in my night, but let me know if I could do something here, I'm around!

fun things happens in my night, but let me know if I could do something here, I'm around!
adamwill commented 2026-06-24 08:51:06 +00:00
Author
Member
Copy link

I think all is good now. The untag was correct, I've also worked around the problem on openQA test side, I pinned down the bug more precisely, the maintainer wrote a fix, and now we're testing it. There shouldn't be a need to retag, a fixed build should supersede the untagged build.

I think all is good now. The untag was correct, I've also worked around the problem on openQA test side, I pinned down the bug more precisely, the maintainer wrote a fix, and now we're testing it. There shouldn't be a need to retag, a fixed build should supersede the untagged build.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
modular-tags-at-branchtime
product-release
createrepoc
bz-epel
branch-eol-sop
multiline-sop
weird-loop
container-branching
unretire
blocking-for-f27
json-json-json
orphan-management
find-failures-owners
cvsadmin
twoweek-stable
rawhide-stable
Labels
Clear labels   
after freeze

   
automation

   
backlog
Tasks that will take longer then ~4 hours to do

  
blocked

   
change-ack
Changes that have releng work and are accepted

  
change-nak
Changes that have releng work and have been rejected

  
change-noreleng
Changes that do not have any releng work

  
changes
  
Closed As
Can't Fix
 Closed with the reason of Can't Fix

  
Closed As
Duplicate
 Closed with the reason of Duplicate

  
Closed As
Fixed
 Closed with the reason of Fixed

  
Closed As
Fixed with Explanation
 Closed with the reason of Fixed with Explanation

  
Closed As
Get back later
 Closed with the reason of Get back later

  
Closed As
Grooming
 Closed with the reason of Grooming

  
Closed As
Insufficient data
 Closed with the reason of Insufficient data

  
Closed As
Invalid
 Closed with the reason of Invalid

  
Closed As
It's all good
 Closed with the reason of It's all good

  
Closed As
taiga
 Closed with the reason of taiga

  
Closed As
upstream
 Closed with the reason of upstream

  
day-to-day
These are small, self-contained tasks that are part of normal operations but are async from sprint planning.

  
dev

   
docs

   
easyfix

   
epel

   
f26

   
f27
Fedora 27

  
f28
Fedora 28

  
f29
Fedora 29

  
f30

   
f31

   
f32

   
f33
Fedora 33

  
f34

   
f35
Fedora 35

  
f36

   
f37

   
f38

   
f39
Fedora 39

  
f40

   
f41
Fedora 41

  
f42
Fedora 42

  
f43
Fedora 43

  
f44
Fedora 44

  
f45
Fedora 45

  
fedora

   
groomed

   
high-gain

   
high-trouble

   
in-progress

   
in-review

   
investigation

   
legal

   
low-gain

   
low-trouble

   
mass rebuild

   
medium-gain

   
medium-trouble

   
meeting

   
mini-initiative

   
new_artifact
New artifact for fedora release

  
ops

   
pdc_retirement

   
rawhide

   
RCA
Root cause analysis

  
review

   
script

   
sidetarget

   
sprint-0

   
sprint-1
Releng Sprint 1 Tracker

  
sprint-2
Sprint 2

  
sprint-3
Releng Sprint 3 Tracker

  
sprint-4
RelEng Spring 4 Tracker

  
sprint-5
RelEng Sprint Tracker 5 (Nov 18 - Dec 02)

  
unfrozen
Revisit after freeze

  
waiting on external
  
Backlog Status
Needs Review
 Needs to go through refinement

  
Backlog Status
Ready
 Ready for sprint

  
chore

   
documentation
  
points
01
 A task that is simple, small, and has very little uncertainty. It should be easy enough for any team member to pick up and complete quickly, without a lot of discussion or research.

  
points
02
 This is a task that is slightly more complex or larger than a "1." It might involve two or three simple steps, but the outcome is still very clear and there's little uncertainty.

  
points
03
 Medium-sized task with some noticeable complexity. It might require a bit more thought or collaboration. The steps are well-defined, but it's not a trivial change.

  
points
05
 A significant piece of work. It has more complexity and a higher degree of uncertainty than smaller tasks. It might require some research, a design discussion, or collaboration across multiple teams.

  
points
08
 Complex task with a lot of uncertainty. It's often the largest size that a team will consider for a single sprint. It likely requires a detailed breakdown and multiple team members to complete.

  
points
13
 Any task that feels like it's a "13" or more is considered an Epic. These are too large and uncertain to be worked on in a single sprint and should be broken down.

  
Priority
High
 This is for work that is critical and needs to be addressed in the next sprint. It's often for urgent bugs, important features, or a major piece of technical debt that is blocking other work.

  
Priority
Low
 This is for work that is good to do but not urgent. It's a candidate for being worked on when there's extra time or when it rises in importance. It can stay in the backlog for a longer period.

  
Priority
Medium
 This is for important work that should be done soon, but isn't as urgent as a high-priority item. It will likely be considered for the next 1-2 sprints.

  
Sprint Status
Blocked
 Work items that cannot progress due to external dependencies, blockers, or issues that need resolution.

  
Sprint Status
Done
 Work items that have been completed, reviewed, and integrated. No further work is needed within the current sprint.

  
Sprint Status
In Progress
 Work items currently being worked on by a team member. Active development, research, or implementation is happening.

  
Sprint Status
Review
 Work items that are complete but need review, testing, or approval before being marked as done.

  
Sprint Status
To Do
 Work items that are committed to the sprint but haven't been started yet. These items are ready to be picked up by team members.

  
Technical Debt
Work that addresses shortcuts, workarounds, or suboptimal implementations that were done previously to meet deadlines. These items improve code quality, maintainability, or performance without adding new features.

  
Work Item
Bug
 Issues where the system is not working as intended or designed. Bugs represent deviations from expected behavior.

  
Work Item
Epic
 Large initiatives that are broken down into multiple smaller work items (user stories, tasks, bugs). Epics represent significant features or projects that span multiple sprints.

  
Work Item
Spike
 Time-boxed research, investigation, or proof-of-concept work to reduce uncertainty or explore solutions. Spikes help inform future development decisions.

  
Work Item
Task
 Used to break down larger user stories into smaller, manageable implementation pieces. They focus on the specific technical steps needed to deliver the user story's value.

  
Work Item
User Story
 User-facing features or functionality that can be framed from a user perspective, describing what someone wants to accomplish and why.

No labels
after freeze
automation
backlog
blocked
change-ack
change-nak
change-noreleng
changes
Closed As
Can't Fix
Closed As
Duplicate
Closed As
Fixed
Closed As
Fixed with Explanation
Closed As
Get back later
Closed As
Grooming
Closed As
Insufficient data
Closed As
Invalid
Closed As
It's all good
Closed As
taiga
Closed As
upstream
day-to-day
dev
docs
easyfix
epel
f26
f27
f28
f29
f30
f31
f32
f33
f34
f35
f36
f37
f38
f39
f40
f41
f42
f43
f44
f45
fedora
groomed
high-gain
high-trouble
in-progress
in-review
investigation
legal
low-gain
low-trouble
mass rebuild
medium-gain
medium-trouble
meeting
mini-initiative
new_artifact
ops
pdc_retirement
rawhide
RCA
review
script
sidetarget
sprint-0
sprint-1
sprint-2
sprint-3
sprint-4
sprint-5
unfrozen
waiting on external
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Sprint ( 2026-06-15 to 2026-06-28 )
Assignees
Clear assignees
No assignees
kevin
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
releng/tickets#13393
No description provided.