[release-process] F45 Cycle: Mass Rebuild (scheduled July 15th, 2026 14 UTC) #13406
Labels
No labels
after freeze
automation
backlog
blocked
change-ack
change-nak
change-noreleng
changes
Closed As
Can't Fix
Closed As
Duplicate
Closed As
Fixed
Closed As
Fixed with Explanation
Closed As
Get back later
Closed As
Grooming
Closed As
Insufficient data
Closed As
Invalid
Closed As
It's all good
Closed As
taiga
Closed As
upstream
day-to-day
dev
docs
easyfix
epel
f26
f27
f28
f29
f30
f31
f32
f33
f34
f35
f36
f37
f38
f39
f40
f41
f42
f43
f44
f45
fedora
groomed
high-gain
high-trouble
in-progress
in-review
investigation
legal
low-gain
low-trouble
mass rebuild
medium-gain
medium-trouble
meeting
mini-initiative
new_artifact
ops
pdc_retirement
rawhide
RCA
review
script
sidetarget
sprint-0
sprint-1
sprint-2
sprint-3
sprint-4
sprint-5
unfrozen
waiting on external
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
No milestone
No project
7 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
releng/tickets#13406
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
As per the Fedora 45 release schedule [1], the mass rebuild is scheduled to happen on Wednesday, 2026-07-15.
For the driving changes, please refer to the relevant wiki page [2].
If you need to exclude a package from the rebuild, you can do one of the following:
noautobuildfile to the root of your dist-git repository.Either method will cause the mass rebuild script to skip the package.
If you have questions, comment on this ticket or ask in the Release Engineering Matrix room [4].
[1] https://fedorapeople.org/groups/schedule/f-45/f-45-releng-tasks.html
[2] https://fedoraproject.org/wiki/Fedora_45_Mass_Rebuild#Driving_Features
[3] https://forge.fedoraproject.org/releng/tooling/src/branch/main/release-process/mass-rebuilds/mass_rebuilds_info.py
[4] https://matrix.to/#/#releng:fedoraproject.org
Before the Mass Rebuild
Communication
Koji Setup
Release Engineering Script Updates
Footnotes:
[1] https://fedoraproject.org/wiki/Fedora_45_Mass_Rebuild
[2] https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/GRFRJG3X6VHX4R5QJUTKPBGJRO5AO22L/
[3] https://koji.fedoraproject.org/koji/taginfo?tagID=143719
[4] https://koji.fedoraproject.org/koji/buildtargetinfo?targetID=61249
[5] infra/tickets#13459
[6] releng/tooling#13041
During the Mass Rebuild
Footnotes:
[1] https://kojipkgs.fedoraproject.org/mass-rebuild/f45-need-rebuild.html
[2] https://kojipkgs.fedoraproject.org/mass-rebuild/f45-failures.html
[3] https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/6VHXCNHJYDJQHMN7LWR6F4YV3OQH5WB5/
After the Mass Rebuild
Footnotes:
[1] https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/IYGKLVOR7HE2PS5FKA2OZVN7XYPGTD25/
[2] #13426
PR for reworked SOP:
infra/docs#531
f45-rebuildTag #13459F45 Cycle: Mass Rebuildto [release-process] F45 Cycle: Mass Rebuild (scheduled July 15th, 2026 14 UTC)@fabio mentioned in releng matrix channel:
It seems like there are toolchain changes after all. The change proposal [1] was submitted before the deadline, but for one reason or another the FESCo ticket [2] was only filed today. FESCo approved the proposal but the changes haven't yet landed in Rawhide. Without the toolchain updates being in place we cannot start the mass rebuild, so there's a real possibility it will have to be delayed.
I reached out to the change owner (Carlos O'Donell) directly asking for more details.
[1] https://fedoraproject.org/w/index.php?title=Changes/GNUToolchainF45
[2] fesco/tickets#3641
The glibc update in https://bodhi.fedoraproject.org/updates/FEDORA-2026-54318fdaca is good enough for a mass rebuild, it was updated from the upstream development branch by Frederic on the 7th.
The gcc update in https://bodhi.fedoraproject.org/updates/FEDORA-2026-9a7eae5a85 is good enough for a mass rebuild, it was updated from the upstream release branch by Jakub on the 3rd.
The binutils update in https://bodhi.fedoraproject.org/updates/FEDORA-2026-aa53457f70 is undergoing final review by Siddhesh and Nick. The previous build https://bodhi.fedoraproject.org/updates/FEDORA-2026-547ddbb113 is in theory good enough for a mass rebuild.
The gdb update in https://bodhi.fedoraproject.org/updates/FEDORA-2026-4a670576d0 is good enough for a mass rebuild, it was updated by Kevin on the 13th.
I think we are green from the system toolchain perspective except the binutils update is stuck under review, but we're working on that.
@siddhesh @nickc How is the binutils update going?
Okay, so just binutils-2.46.90-1.fc45 needs to make it to stable. I see it still is in testing.
The rebuild was already supposed to be running at this point, but there are still two unresolved issues:
So it's pretty clear we will have to delay the start.
@codonell @decathorpe (and I suppose also @siddhesh and @nickc)
Could you please post a comment here once we're good to go? Thank you.
I poked the pending LLVM PR: https://src.fedoraproject.org/rpms/llvm/pull-request/620
I don't think it's important to land the LLVM change before the mass rebuild.
@patrikp wrote in #13406 (comment):
It's in stable now.
Reminder that we should check the list of gated updates before merging, and not merge rebuilds that correspond to updates that failed gating for what look like valid/worrisome reasons.
Also note there is a large KDE update currently pending on a side tag; it looks like the mass rebuild is rebuilding things that were intended for that side tag, but won't necessarily build them against each other, if it's just building in alphabetical order?
Looking through gated updates...
re/ nettle: there's a nettle3.10 compat package, so at least gnutls built against nettle 3 from the current buildroot won't not be installable.
yeah, saw that after. So it'll probably be okay, but not what the packager intended.
@patrikp, mass rebuild is completed. We need to mass-merge builds to tag to the main tag, but before that, when you come in tomorrow, please make sure to check @adamwill's comment above, untag the builds, and do the needful. Once the tag is merged, send out an announcement on the list.
FYI: We keep need-rebuilds and find-failures HTML pages open for approximately two weeks, so folks have time to view.
Post announcement, I'll file the FTBFS bugs after creating new bug pages.
Additionally, three builds were stuck during the mass rebuild, having run times of more than 50 hours. Two were stuck at the ppc64le builder, and one was stuck with x86_64.
It would be good to add these to the completion email so folks know why their respective builds were canceled.
robosignatory changes has been merged, and autosign playbook ran successfully, clear for tagging packages @patrikp
@adamwill wrote in #13406 (comment):
First off, thank you very much for keeping such a watchful eye, I literally don't know what we'd do without you.
These three have been untagged from
f45-rebuild(or rather, the builds that were done as part of mass rebuild):It seems like the build actually succeeded. In any case:
$ koji list-tagged f45-rebuild | grep -E primecountreturns no results. No primecount builds are currently in the rebuild tag so it shouldn't be of any concern in terms of running the mass tagging script.Untagged to be safe:
So I believe we should be all good to run the mass tagging script, which I'm going to do now.
So, when trying to run the mass tagging script, it fails with this error:
ERROR: User patrikp lacks f45 access permission. Reason: tag requires autosign permissionWhich makes sense as I don't have the permission and usually use the --force flag to tag/untag builds.
I will bring this up in the weekly meeting which conveniently starts in less than half an hour.
It was brought to my attention that, since I have admin powers in Koji, I can just add the permission myself (imagine that).
I ran the script and it finished:
Let me send the final email now.
Total tagged builds: 816?
@adamwill wrote in #13406 (comment):
I think this is a bug in the script. It does the tagging in batches of 1000. The last line is just the remainder after the last full batch of 1000.
21816 (all builds) - 816 (the remainder) = 21000 (multiple of 1000). We're good.
Announcement email sent and the first comment with the checklist updated.
Next up is filing FTBFS bugs, but that is tracked in a separate ticket, so I believe we can close this one.