Fedora 44 Mass Rebuild Tracker #13150
Labels
No labels
after freeze
automation
backlog
blocked
change-ack
change-nak
change-noreleng
changes
Closed As
Can't Fix
Closed As
Duplicate
Closed As
Fixed
Closed As
Fixed with Explanation
Closed As
Get back later
Closed As
Grooming
Closed As
Insufficient data
Closed As
Invalid
Closed As
It's all good
Closed As
taiga
Closed As
upstream
day-to-day
dev
docs
easyfix
epel
f26
f27
f28
f29
f30
f31
f32
f33
f34
f35
f36
f37
f38
f39
f40
f41
f42
f43
f44
f45
fedora
groomed
high-gain
high-trouble
in-progress
in-review
investigation
legal
low-gain
low-trouble
mass rebuild
medium-gain
medium-trouble
meeting
mini-initiative
new_artifact
ops
pdc_retirement
rawhide
RCA
review
script
sidetarget
sprint-0
sprint-1
sprint-2
sprint-3
sprint-4
sprint-5
unfrozen
waiting on external
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
No milestone
No project
9 participants
Notifications
Due date
No due date set.
Depends on
Reference
releng/tickets#13150
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The Fedora 44 schedule[1] has a mass rebuild scheduled for Wed 2026-01-14, We need to plan and coordinate all tasks in preparation for it. For the driving changes please refer[2].
Please note that if you need to exclude any packages from the rebuild,
you can use the PKG_SKIP_LIST or add a noautobuild file to the root of
your distgit repository. This will instruct the mass rebuild script to
skip these packages. Previously, some OCaml packages were rebuilt, but
they can now be opted out using this method if required.
[1] https://fedorapeople.org/groups/schedule/f-44/f-44-key-tasks.html
[2] https://fedoraproject.org/wiki/Fedora_44_Mass_Rebuild#Driving_Features
2)Wiki page is ready: https://fedoraproject.org/wiki/Fedora_44_Mass_Rebuild#Packages_status
7)Scripts are updated: releng/tooling#13011
5)Requested autosign for rebuild tag: https://pagure.io/fedora-infrastructure/issue/13014
The above comments' numbers are in reference to the steps in our mass rebuild SOP [1].
We now also have the tag and build target, i.e.:
Step 4: Create a Tag to Contain the Mass Rebuild
https://koji.fedoraproject.org/koji/taginfo?tagID=116591
Step 6: Create the Koji Target for the Mass Rebuild
https://koji.fedoraproject.org/koji/buildtargetinfo?targetID=54820
The announcement (step 3) will be sent out tomorrow, Jan 7th.
[1] https://docs.fedoraproject.org/en-US/infra/release_guide/sop_mass_rebuild/
Notification email sent to
devel-announce. 👍https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/X57NJC5RYAWXV4EON7BPYZO6LV5VPOLC/
I sent an email to the owners of the driving changes to see if we're on track for the mass rebuild on Wednesday.
Hi, see "Depends On" on the change ticket: https://bugzilla.redhat.com/show_bug.cgi?id=2418384 there are still 2 packages currently FTBFS (dnf5 and libyui) due to GCC/C++, but from Ruby POV, everything was rebuilt and is part of Fedora at this point in time, now just tying up the change process.
tl;dr: everything OK from Ruby 4.0 point of view and there are no blockers.
Thanks.
I checked in with the GCC and GLIBC teams and we are Green light, good to go with the mass rebuild for those teams.
I'm still checking in BINUTILS and GDB.
Once I'm done I'll provide status on the GNU Toolchain go/no-go for mass rebuild.
Confirmed with the GDB team, we're green light, good to go.
Still checking in with BINUTILS.
Confirmed with the BINUTILS team that we're green light, good to go.
The teams will put some final polish on the builds.
Sorry, we are late on the GCC side, we were blocked for a few hours yesterday with https://bugzilla.redhat.com/141058262 glibc issue and couldn't start the build earlier.
We'd really like to see https://koji.fedoraproject.org/koji/taskinfo?taskID=141058262 in rawhide for the mass rebuild,
it fixes a lot of important issues, but most importantly https://gcc.gnu.org/PR123396 which doesn't affect just gcc itself, but more importantly could affect anything that uses libtool.
It has finished building on all arches but s390x by now (and I've checked the %check results for those arches and on x86_64 tested in mock the -latomic by default linking as well as -lgcc_s), but s390x build servers are underpowered and I'm afraid it can take 4-5 more hours to finish (and then some time to get tagged into f44, even if I wave the 5+ hrs long extra QA).
We are keeping an eye on the build. We will see about tagging it to f44 as soon as it is done.
CC: @patrikp @amedvede @kevin
For the record, mass rebuild has NOT started yet, we are waiting for gcc the build to finish.
I think the build won't take too long from this point, I think the current libgomp make check is one of the last larger ones in %check, the really large like libstdc++ or gcc/g++ are already complete.
The build is almost done, writing rpms now. Shall I do the update and waive it or do you want to tag it into f44 yourself?
There's a blocker with boost now ( #13171 ) please hold mass rebuild start until thats sorted.
@jakub wrote in #13150 (comment):
Do the update, why does it need waving?
https://bodhi.fedoraproject.org/updates/FEDORA-2026-824fdb31e5
@kevin wrote in #13150 (comment):
Because the test gating takes 5-6 more hours and in 99% cases fails due to infrastructure errors and one needs to wave it after those 5-6hrs.
koji latest-pkg --quiet f44 gcc
gcc-16.0.1-0.2.fc44 f44 jakub
@jakub openQA tests do not "in 99% cases fails due to infrastructure errors" and do not take 5-6 hours, they take around 2 hours. If you have an unreliable Fedora CI test in your gating configuration, remove it, but please do not waive openQA failures as a matter of course.
Currently almost all Rawhide updates are failing openQA tests because of the boost 1.90 mess...which itself was caused by inappropriate waiving of openQA failures.
The GCC openQA tests do try to rebuild a couple of packages with the new gcc (kernel, emacs, ...).
And at least lately, they fail pretty much all the time, with errors like
Could not execute build: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub
Often I ask for rerun of the tests, it churns for 5-6 hours and it results in this again.
See e.g.
https://osci-jenkins-1.ci.fedoraproject.org/job/scratch-build-test/1950/console
https://bodhi.fedoraproject.org/updates/FEDORA-2025-504d65eec8
https://bodhi.fedoraproject.org/updates/FEDORA-2025-d98c5fb638
https://bodhi.fedoraproject.org/updates/FEDORA-2025-f6a6021345
Ok,
https://bodhi.fedoraproject.org/updates/FEDORA-2025-eb758075aa
was fine.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-ca2de2f916
failed again, etc. Unfortunately no logs for the month+ old ones.
Note, in gcc itself we have %check with millions of tests and I have scripts to check it separately from OpenQA.
That's not an openQA test. openQA tests are generic (not specific to any particular package) and their resultsdb names all start with
update.. The test you're referring to seems to bebaseos-qe.koji-build.scratch-build.validation, which is not an openQA test. I'm not actually sure what that is or how it's triggered - typical Fedora CI tests have resultsdb names starting withfedora-ci., notbaseos-qe., and I can't find that test in https://src.fedoraproject.org/rpms/gcc/blob/rawhide/f/plans or https://gitlab.com/redhat/centos-stream/tests/gcc.git . (edit: looks like it is something from Fedora CI , it seems like it's configured to run against some subset of build chain-y packages, we'd have to ask that team for more details). edit 2: I've filed https://github.com/fedora-ci/scratch-build-test/issues/12 .More to the point, though, it's only gating because it's configured as a gating test here. If you don't actually want your updates to be gated on that test, just take that line out; it's entirely under your control. There's no point having it configured as gating but then waiving failures of it.
The 502 errors are tracked at https://pagure.io/fedora-infrastructure/issue/12913 , with a workaround mentioned at https://pagure.io/fedora-infrastructure/issue/12913#comment-993744 .
On another note:
This time let's please avoid merging rebuilds of known-bad builds. That is, before merging anything from the rebuild tag, check https://bodhi.fedoraproject.org/updates/?status=testing&release=F44 and look through any updates more than a few hours old, and figure why they're not stable. It's likely because a gating test failed. If so, try and determine whether the failure was a genuine one, and if so, probably do not merge the rebuild as it will contain the same bug. In particular, please assume any rebuild of an update that's gated by a failed openQA test should not be merged. This currently includes the kernel, which is gated because recent builds cause openQA to hit a kernel panic on graphical desktops. It also includes shadow-utils, whose latest build breaks the installer's user creation.
When an update is gated on a Fedora CI test it's more often the case that the test is just broken or something, in which case landing the rebuild might be OK. But it's best to go case-by-case.
I think things are ready to fire off the mass rebuild... unless anyone chimes up with anything else.
Okay so I'll try to fire it off now.
Here's my list of things that definitely should not currently be merged from a mass rebuild:
Other currently-gated updates are less clear-cut. Several are caused by bad test suite configurations. Others are failing their test suites, but for reasons that likely aren't specific to the gated update. I've posted follow-up comments on most of them.
I fired off the mass-rebuild, and we can monitor the builds from here.
https://kojipkgs.fedoraproject.org/mass-rebuild/f44-need-rebuild.html
https://kojipkgs.fedoraproject.org/mass-rebuild/f44-failures.html
There are some abrupt exit signals for rebuild file which were causing issue, I have addressed them there, we need to get them in and test it out in stg once, releng/tooling#13013.
Fixing FTBFS and building in the side-tag is OK, it appears. At least the scripts have picked-up my fix for python-PyMuPDF ;-)
[ docutils 0.22 may give other users of rst2pdf the same trouble]
I guess we should not build unrelated updates there just because we're impationt, right?
@mjg wrote in #13150 (comment):
you do not need to hold anything or do anything differently due to the mass rebuild. If you need/want to push a rawhide update, go ahead as normal. Then your build will be 'newer' and the one from the mass rebuild will just not be used.
For transparency before merging, as @adamwill suggested, I'm untagging the following obvious error builds
Yeah, kernel-headers is fine. I'm not sure why kernel wasn't rebuilt as part of the mass rebuild, though.
Please also do NOT tag firefox-147.0-2.fc44 , I have just confirmed that it's broken the same as the 147.0.1 Rawhide update.
Done: firefox-147.0-2.fc44 has been untagged as well from the f44-rebuild tag! We are waiting on the signing to finish, once it is done, we will do the merge.
So I think work is done here, email has been sent out, closing this.
Here's the updated SOP:
infra/docs#481