Fedora 44 Mass Rebuild Tracker #13150

Closed
opened 2026-01-05 15:19:43 +00:00 by amedvede · 35 comments
Member

The Fedora 44 schedule[1] has a mass rebuild scheduled for Wed 2026-01-14, We need to plan and coordinate all tasks in preparation for it. For the driving changes please refer[2].

Please note that if you need to exclude any packages from the rebuild,
you can use the PKG_SKIP_LIST or add a noautobuild file to the root of
your distgit repository. This will instruct the mass rebuild script to
skip these packages. Previously, some OCaml packages were rebuilt, but
they can now be opted out using this method if required.

[1] https://fedorapeople.org/groups/schedule/f-44/f-44-key-tasks.html
[2] https://fedoraproject.org/wiki/Fedora_44_Mass_Rebuild#Driving_Features

The Fedora 44 schedule[1] has a mass rebuild scheduled for Wed 2026-01-14, We need to plan and coordinate all tasks in preparation for it. For the driving changes please refer[2]. Please note that if you need to exclude any packages from the rebuild, you can use the PKG_SKIP_LIST or add a noautobuild file to the root of your distgit repository. This will instruct the mass rebuild script to skip these packages. Previously, some OCaml packages were rebuilt, but they can now be opted out using this method if required. [1] https://fedorapeople.org/groups/schedule/f-44/f-44-key-tasks.html [2] https://fedoraproject.org/wiki/Fedora_44_Mass_Rebuild#Driving_Features
Author
Member
2)Wiki page is ready: https://fedoraproject.org/wiki/Fedora_44_Mass_Rebuild#Packages_status
Author
Member

7)Scripts are updated: releng/tooling#13011

7)Scripts are updated: https://forge.fedoraproject.org/releng/tooling/pulls/13011
Author
Member

5)Requested autosign for rebuild tag: https://pagure.io/fedora-infrastructure/issue/13014

5)Requested autosign for rebuild tag: https://pagure.io/fedora-infrastructure/issue/13014
Member

The above comments' numbers are in reference to the steps in our mass rebuild SOP [1].
We now also have the tag and build target, i.e.:

Step 4: Create a Tag to Contain the Mass Rebuild
https://koji.fedoraproject.org/koji/taginfo?tagID=116591

Step 6: Create the Koji Target for the Mass Rebuild
https://koji.fedoraproject.org/koji/buildtargetinfo?targetID=54820

The announcement (step 3) will be sent out tomorrow, Jan 7th.

[1] https://docs.fedoraproject.org/en-US/infra/release_guide/sop_mass_rebuild/

The above comments' numbers are in reference to the steps in our mass rebuild SOP [1]. We now also have the tag and build target, i.e.: Step 4: Create a Tag to Contain the Mass Rebuild https://koji.fedoraproject.org/koji/taginfo?tagID=116591 Step 6: Create the Koji Target for the Mass Rebuild https://koji.fedoraproject.org/koji/buildtargetinfo?targetID=54820 The announcement (step 3) will be sent out tomorrow, Jan 7th. [1] https://docs.fedoraproject.org/en-US/infra/release_guide/sop_mass_rebuild/
Member
Notification email sent to `devel-announce`. 👍 https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/X57NJC5RYAWXV4EON7BPYZO6LV5VPOLC/
Member

I sent an email to the owners of the driving changes to see if we're on track for the mass rebuild on Wednesday.

I sent an email to the owners of the driving changes to see if we're on track for the mass rebuild on Wednesday.

Hi, see "Depends On" on the change ticket: https://bugzilla.redhat.com/show_bug.cgi?id=2418384 there are still 2 packages currently FTBFS (dnf5 and libyui) due to GCC/C++, but from Ruby POV, everything was rebuilt and is part of Fedora at this point in time, now just tying up the change process.

tl;dr: everything OK from Ruby 4.0 point of view and there are no blockers.

Thanks.

Hi, see "Depends On" on the change ticket: https://bugzilla.redhat.com/show_bug.cgi?id=2418384 there are still 2 packages currently FTBFS (dnf5 and libyui) due to GCC/C++, but from Ruby POV, everything was rebuilt and is part of Fedora at this point in time, now just tying up the change process. tl;dr: everything OK from Ruby 4.0 point of view and there are no blockers. Thanks.

I checked in with the GCC and GLIBC teams and we are Green light, good to go with the mass rebuild for those teams.

I'm still checking in BINUTILS and GDB.

Once I'm done I'll provide status on the GNU Toolchain go/no-go for mass rebuild.

I checked in with the GCC and GLIBC teams and we are Green light, good to go with the mass rebuild for those teams. I'm still checking in BINUTILS and GDB. Once I'm done I'll provide status on the GNU Toolchain go/no-go for mass rebuild.

Confirmed with the GDB team, we're green light, good to go.

Still checking in with BINUTILS.

Confirmed with the GDB team, we're green light, good to go. Still checking in with BINUTILS.

Confirmed with the BINUTILS team that we're green light, good to go.

The teams will put some final polish on the builds.

Confirmed with the BINUTILS team that we're green light, good to go. The teams will put some final polish on the builds.

Sorry, we are late on the GCC side, we were blocked for a few hours yesterday with https://bugzilla.redhat.com/141058262 glibc issue and couldn't start the build earlier.
We'd really like to see https://koji.fedoraproject.org/koji/taskinfo?taskID=141058262 in rawhide for the mass rebuild,
it fixes a lot of important issues, but most importantly https://gcc.gnu.org/PR123396 which doesn't affect just gcc itself, but more importantly could affect anything that uses libtool.
It has finished building on all arches but s390x by now (and I've checked the %check results for those arches and on x86_64 tested in mock the -latomic by default linking as well as -lgcc_s), but s390x build servers are underpowered and I'm afraid it can take 4-5 more hours to finish (and then some time to get tagged into f44, even if I wave the 5+ hrs long extra QA).

Sorry, we are late on the GCC side, we were blocked for a few hours yesterday with https://bugzilla.redhat.com/141058262 glibc issue and couldn't start the build earlier. We'd really like to see https://koji.fedoraproject.org/koji/taskinfo?taskID=141058262 in rawhide for the mass rebuild, it fixes a lot of important issues, but most importantly https://gcc.gnu.org/PR123396 which doesn't affect just gcc itself, but more importantly could affect anything that uses libtool. It has finished building on all arches but s390x by now (and I've checked the %check results for those arches and on x86_64 tested in mock the -latomic by default linking as well as -lgcc_s), but s390x build servers are underpowered and I'm afraid it can take 4-5 more hours to finish (and then some time to get tagged into f44, even if I wave the 5+ hrs long extra QA).
Owner

We are keeping an eye on the build. We will see about tagging it to f44 as soon as it is done.

CC: @patrikp @amedvede @kevin

We are keeping an eye on the build. We will see about tagging it to f44 as soon as it is done. CC: @patrikp @amedvede @kevin
Member

For the record, mass rebuild has NOT started yet, we are waiting for gcc the build to finish.

For the record, mass rebuild has NOT started yet, we are waiting for gcc the build to finish.

I think the build won't take too long from this point, I think the current libgomp make check is one of the last larger ones in %check, the really large like libstdc++ or gcc/g++ are already complete.

I think the build won't take too long from this point, I think the current libgomp make check is one of the last larger ones in %check, the really large like libstdc++ or gcc/g++ are already complete.

The build is almost done, writing rpms now. Shall I do the update and waive it or do you want to tag it into f44 yourself?

The build is almost done, writing rpms now. Shall I do the update and waive it or do you want to tag it into f44 yourself?
Owner

There's a blocker with boost now ( #13171 ) please hold mass rebuild start until thats sorted.

@jakub wrote in #13150 (comment):

The build is almost done, writing rpms now. Shall I do the update and waive it or do you want to tag it into f44 yourself?

Do the update, why does it need waving?

There's a blocker with boost now ( https://forge.fedoraproject.org/releng/tickets/issues/13171 ) please hold mass rebuild start until thats sorted. @jakub wrote in https://forge.fedoraproject.org/releng/tickets/issues/13150#issuecomment-261458: > The build is almost done, writing rpms now. Shall I do the update and waive it or do you want to tag it into f44 yourself? Do the update, why does it need waving?
https://bodhi.fedoraproject.org/updates/FEDORA-2026-824fdb31e5

@kevin wrote in #13150 (comment):

Do the update, why does it need waving?

Because the test gating takes 5-6 more hours and in 99% cases fails due to infrastructure errors and one needs to wave it after those 5-6hrs.

@kevin wrote in https://forge.fedoraproject.org/releng/tickets/issues/13150#issuecomment-261460: > Do the update, why does it need waving? Because the test gating takes 5-6 more hours and in 99% cases fails due to infrastructure errors and one needs to wave it after those 5-6hrs.

koji latest-pkg --quiet f44 gcc
gcc-16.0.1-0.2.fc44 f44 jakub

koji latest-pkg --quiet f44 gcc gcc-16.0.1-0.2.fc44 f44 jakub
Contributor

@jakub openQA tests do not "in 99% cases fails due to infrastructure errors" and do not take 5-6 hours, they take around 2 hours. If you have an unreliable Fedora CI test in your gating configuration, remove it, but please do not waive openQA failures as a matter of course.

Currently almost all Rawhide updates are failing openQA tests because of the boost 1.90 mess...which itself was caused by inappropriate waiving of openQA failures.

@jakub openQA tests do not "in 99% cases fails due to infrastructure errors" and do not take 5-6 hours, they take around 2 hours. If you have an unreliable Fedora CI test in your gating configuration, remove it, but please do not waive openQA failures as a matter of course. Currently almost all Rawhide updates are failing openQA tests because of the boost 1.90 mess...which *itself* was caused by inappropriate waiving of openQA failures.

The GCC openQA tests do try to rebuild a couple of packages with the new gcc (kernel, emacs, ...).
And at least lately, they fail pretty much all the time, with errors like
Could not execute build: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub
Often I ask for rerun of the tests, it churns for 5-6 hours and it results in this again.
See e.g.
https://osci-jenkins-1.ci.fedoraproject.org/job/scratch-build-test/1950/console
https://bodhi.fedoraproject.org/updates/FEDORA-2025-504d65eec8
https://bodhi.fedoraproject.org/updates/FEDORA-2025-d98c5fb638
https://bodhi.fedoraproject.org/updates/FEDORA-2025-f6a6021345
Ok,
https://bodhi.fedoraproject.org/updates/FEDORA-2025-eb758075aa
was fine.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-ca2de2f916
failed again, etc. Unfortunately no logs for the month+ old ones.

The GCC openQA tests do try to rebuild a couple of packages with the new gcc (kernel, emacs, ...). And at least lately, they fail pretty much all the time, with errors like Could not execute build: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub Often I ask for rerun of the tests, it churns for 5-6 hours and it results in this again. See e.g. https://osci-jenkins-1.ci.fedoraproject.org/job/scratch-build-test/1950/console https://bodhi.fedoraproject.org/updates/FEDORA-2025-504d65eec8 https://bodhi.fedoraproject.org/updates/FEDORA-2025-d98c5fb638 https://bodhi.fedoraproject.org/updates/FEDORA-2025-f6a6021345 Ok, https://bodhi.fedoraproject.org/updates/FEDORA-2025-eb758075aa was fine. https://bodhi.fedoraproject.org/updates/FEDORA-2025-ca2de2f916 failed again, etc. Unfortunately no logs for the month+ old ones.

Note, in gcc itself we have %check with millions of tests and I have scripts to check it separately from OpenQA.

Note, in gcc itself we have %check with millions of tests and I have scripts to check it separately from OpenQA.
Contributor

That's not an openQA test. openQA tests are generic (not specific to any particular package) and their resultsdb names all start with update.. The test you're referring to seems to be baseos-qe.koji-build.scratch-build.validation, which is not an openQA test. I'm not actually sure what that is or how it's triggered - typical Fedora CI tests have resultsdb names starting with fedora-ci., not baseos-qe., and I can't find that test in https://src.fedoraproject.org/rpms/gcc/blob/rawhide/f/plans or https://gitlab.com/redhat/centos-stream/tests/gcc.git . (edit: looks like it is something from Fedora CI , it seems like it's configured to run against some subset of build chain-y packages, we'd have to ask that team for more details). edit 2: I've filed https://github.com/fedora-ci/scratch-build-test/issues/12 .

More to the point, though, it's only gating because it's configured as a gating test here. If you don't actually want your updates to be gated on that test, just take that line out; it's entirely under your control. There's no point having it configured as gating but then waiving failures of it.

The 502 errors are tracked at https://pagure.io/fedora-infrastructure/issue/12913 , with a workaround mentioned at https://pagure.io/fedora-infrastructure/issue/12913#comment-993744 .

That's not an openQA test. openQA tests are generic (not specific to any particular package) and their resultsdb names all start with `update.`. The test you're referring to seems to be `baseos-qe.koji-build.scratch-build.validation`, which is not an openQA test. I'm not actually sure what that is or how it's triggered - typical Fedora CI tests have resultsdb names starting with `fedora-ci.`, not `baseos-qe.`, and I can't find that test in https://src.fedoraproject.org/rpms/gcc/blob/rawhide/f/plans or https://gitlab.com/redhat/centos-stream/tests/gcc.git . (edit: looks like it is [something from Fedora CI](https://github.com/fedora-ci/scratch-build-test) , it seems like it's configured to run against some subset of build chain-y packages, we'd have to ask that team for more details). edit 2: I've filed https://github.com/fedora-ci/scratch-build-test/issues/12 . More to the point, though, it's only gating because it's [configured as a gating test here](https://src.fedoraproject.org/rpms/gcc/blob/rawhide/f/gating.yaml#_8). If you don't actually want your updates to be gated on that test, just take that line out; it's entirely under your control. There's no point having it configured as gating but then waiving failures of it. The 502 errors are tracked at https://pagure.io/fedora-infrastructure/issue/12913 , with a workaround mentioned at https://pagure.io/fedora-infrastructure/issue/12913#comment-993744 .
Contributor

On another note:

This time let's please avoid merging rebuilds of known-bad builds. That is, before merging anything from the rebuild tag, check https://bodhi.fedoraproject.org/updates/?status=testing&release=F44 and look through any updates more than a few hours old, and figure why they're not stable. It's likely because a gating test failed. If so, try and determine whether the failure was a genuine one, and if so, probably do not merge the rebuild as it will contain the same bug. In particular, please assume any rebuild of an update that's gated by a failed openQA test should not be merged. This currently includes the kernel, which is gated because recent builds cause openQA to hit a kernel panic on graphical desktops. It also includes shadow-utils, whose latest build breaks the installer's user creation.

When an update is gated on a Fedora CI test it's more often the case that the test is just broken or something, in which case landing the rebuild might be OK. But it's best to go case-by-case.

On another note: This time let's *please* avoid merging rebuilds of known-bad builds. That is, *before* merging anything from the rebuild tag, check https://bodhi.fedoraproject.org/updates/?status=testing&release=F44 and look through any updates more than a few hours old, and figure why they're not stable. It's likely because a gating test failed. If so, try and determine whether the failure was a genuine one, and if so, probably do *not* merge the rebuild as it will contain the same bug. In particular, please assume any rebuild of an update that's gated by a failed openQA test should not be merged. This currently includes the kernel, which is gated because recent builds cause openQA to hit a kernel panic on graphical desktops. It also includes [shadow-utils](https://bodhi.fedoraproject.org/updates/FEDORA-2026-20c03e4acb), whose latest build breaks the installer's user creation. When an update is gated on a Fedora CI test it's more often the case that the test is just broken or something, in which case landing the rebuild might be OK. But it's best to go case-by-case.
Owner

I think things are ready to fire off the mass rebuild... unless anyone chimes up with anything else.

I think things are ready to fire off the mass rebuild... unless anyone chimes up with anything else.
Owner

Okay so I'll try to fire it off now.

Okay so I'll try to fire it off now.
Contributor

Here's my list of things that definitely should not currently be merged from a mass rebuild:

Other currently-gated updates are less clear-cut. Several are caused by bad test suite configurations. Others are failing their test suites, but for reasons that likely aren't specific to the gated update. I've posted follow-up comments on most of them.

Here's my list of things that definitely should not currently be merged from a mass rebuild: * kernel - it is gated by openQA tests because [openQA hits a kernel panic in vmmouse](https://bodhi.fedoraproject.org/updates/FEDORA-2025-491003512b#comment-4468315), @jforbes is currently bisecting this * shadow-utils - it is gated by openQA because it [breaks any install where no root password is set](https://github.com/shadow-maint/shadow/issues/1483#issuecomment-3757398138) * authselect - it is gated because it [breaks FreeIPA uninstall](https://bodhi.fedoraproject.org/updates/FEDORA-2026-8a3906ef4b#comment-4516638) Other currently-gated updates are less clear-cut. Several are caused by bad test suite configurations. Others are failing their test suites, but for reasons that likely aren't specific to the gated update. I've posted follow-up comments on most of them.
Owner

I fired off the mass-rebuild, and we can monitor the builds from here.

https://kojipkgs.fedoraproject.org/mass-rebuild/f44-need-rebuild.html
https://kojipkgs.fedoraproject.org/mass-rebuild/f44-failures.html

There are some abrupt exit signals for rebuild file which were causing issue, I have addressed them there, we need to get them in and test it out in stg once, releng/tooling#13013.

I fired off the mass-rebuild, and we can monitor the builds from here. https://kojipkgs.fedoraproject.org/mass-rebuild/f44-need-rebuild.html https://kojipkgs.fedoraproject.org/mass-rebuild/f44-failures.html There are some abrupt exit signals for rebuild file which were causing issue, I have addressed them there, we need to get them in and test it out in stg once, https://forge.fedoraproject.org/releng/tooling/pulls/13013.

Fixing FTBFS and building in the side-tag is OK, it appears. At least the scripts have picked-up my fix for python-PyMuPDF ;-)
[ docutils 0.22 may give other users of rst2pdf the same trouble]

I guess we should not build unrelated updates there just because we're impationt, right?

Fixing FTBFS and building in the side-tag is OK, it appears. At least the scripts have picked-up my fix for python-PyMuPDF ;-) [ docutils 0.22 may give other users of rst2pdf the same trouble] I guess we should *not* build unrelated updates there just because we're impationt, right?
Owner

@mjg wrote in #13150 (comment):

Fixing FTBFS and building in the side-tag is OK, it appears. At least the scripts have picked-up my fix for python-PyMuPDF ;-) [ docutils 0.22 may give other users of rst2pdf the same trouble]

I guess we should not build unrelated updates there just because we're impationt, right?

you do not need to hold anything or do anything differently due to the mass rebuild. If you need/want to push a rawhide update, go ahead as normal. Then your build will be 'newer' and the one from the mass rebuild will just not be used.

@mjg wrote in https://forge.fedoraproject.org/releng/tickets/issues/13150#issuecomment-264534: > Fixing FTBFS and building in the side-tag is OK, it appears. At least the scripts have picked-up my fix for python-PyMuPDF ;-) [ docutils 0.22 may give other users of rst2pdf the same trouble] > > I guess we should _not_ build unrelated updates there just because we're impationt, right? you do not need to hold anything or do anything differently due to the mass rebuild. If you need/want to push a rawhide update, go ahead as normal. Then your build will be 'newer' and the one from the mass rebuild will just not be used.
Owner

For transparency before merging, as @adamwill suggested, I'm untagging the following obvious error builds

  • authselect-1.7.0-3.fc44
  • shadow-utils-4.19.0-4.fc44
  • I don't see a kernel build; I suspect the kernel-headers is a different set altogether.
koji list-tagged f44-rebuild --latest | grep "kernel"  

R-IRkernel-1.3.2-16.fc44                  f44-rebuild           releng
config-kernel-0.3-6.fc44                  f44-rebuild           releng
gap-pkg-jupyterkernel-1.5.1-8.fc44        f44-rebuild           releng
kernel-headers-6.19.0-0.rc5.38.fc44.1     f44-rebuild           releng
kernel-srpm-macros-1.0-28.fc44            f44-rebuild           releng
kernelshark-2.3.1-8.fc44                  f44-rebuild           releng
ocaml-intrinsics-kernel-0.17.1-8.fc44     f44-rebuild           releng
python-ipykernel-6.29.3-17.fc44           f44-rebuild           releng
python-jupyter-kernel-singular-0.9.9-25.fc44  f44-rebuild           releng
python-jupyter-kernel-test-0.7.0-10.fc44  f44-rebuild           releng
python-metakernel-0.30.4-2.fc44           f44-rebuild           releng
python-spyder-kernels-3.1.1-2.fc44        f44-rebuild           releng

For transparency before merging, as @adamwill suggested, I'm untagging the following obvious error builds - authselect-1.7.0-3.fc44 - shadow-utils-4.19.0-4.fc44 - I don't see a kernel build; I suspect the kernel-headers is a different set altogether. ``` koji list-tagged f44-rebuild --latest | grep "kernel" R-IRkernel-1.3.2-16.fc44 f44-rebuild releng config-kernel-0.3-6.fc44 f44-rebuild releng gap-pkg-jupyterkernel-1.5.1-8.fc44 f44-rebuild releng kernel-headers-6.19.0-0.rc5.38.fc44.1 f44-rebuild releng kernel-srpm-macros-1.0-28.fc44 f44-rebuild releng kernelshark-2.3.1-8.fc44 f44-rebuild releng ocaml-intrinsics-kernel-0.17.1-8.fc44 f44-rebuild releng python-ipykernel-6.29.3-17.fc44 f44-rebuild releng python-jupyter-kernel-singular-0.9.9-25.fc44 f44-rebuild releng python-jupyter-kernel-test-0.7.0-10.fc44 f44-rebuild releng python-metakernel-0.30.4-2.fc44 f44-rebuild releng python-spyder-kernels-3.1.1-2.fc44 f44-rebuild releng ```
Contributor

Yeah, kernel-headers is fine. I'm not sure why kernel wasn't rebuilt as part of the mass rebuild, though.

Please also do NOT tag firefox-147.0-2.fc44 , I have just confirmed that it's broken the same as the 147.0.1 Rawhide update.

Yeah, kernel-headers is fine. I'm not sure why kernel wasn't rebuilt as part of the mass rebuild, though. Please also do ***NOT*** tag firefox-147.0-2.fc44 , I have just [confirmed that it's broken](https://openqa.stg.fedoraproject.org/tests/5793364#step/desktop_browser/29) the same as the [147.0.1 Rawhide update](https://bodhi.fedoraproject.org/updates/FEDORA-2026-1fd8fcdf6f).
Owner

Done: firefox-147.0-2.fc44 has been untagged as well from the f44-rebuild tag! We are waiting on the signing to finish, once it is done, we will do the merge.

Done: firefox-147.0-2.fc44 has been untagged as well from the f44-rebuild tag! We are waiting on the signing to finish, once it is done, we will do the merge.
Owner

So I think work is done here, email has been sent out, closing this.

So I think work is done here, email has been sent out, closing this.
Member

Here's the updated SOP:
infra/docs#481

Here's the updated SOP: https://forge.fedoraproject.org/infra/docs/pulls/481
Sign in to join this conversation.
No milestone
9 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
releng/tickets#13150
No description provided.