Fedora 43 Mass Rebuild Tracker

jnsamyak commented

2025-07-21 14:40:14 +00:00

Owner

The Fedora 43 schedule[1] has a mass rebuild scheduled for Wed 2025-07-23, We need to plan and coordinate all tasks in preparation for it. For the driving changes please refer[2].

Please note that if you need to exclude any packages from the rebuild,
you can use the PKG_SKIP_LIST or add a noautobuild file to the root of
your distgit repository. This will instruct the mass rebuild script to
skip these packages. Previously, some OCaml packages were rebuilt, but
they can now be opted out using this method if required.

[1] https://fedorapeople.org/groups/schedule/f-43/f-43-key-tasks.html
[2] https://fedoraproject.org/wiki/Fedora_43_Mass_Rebuild#Driving_Features

The Fedora 43 schedule[1] has a mass rebuild scheduled for Wed 2025-07-23, We need to plan and coordinate all tasks in preparation for it. For the driving changes please refer[2]. Please note that if you need to exclude any packages from the rebuild, you can use the PKG_SKIP_LIST or add a noautobuild file to the root of your distgit repository. This will instruct the mass rebuild script to skip these packages. Previously, some OCaml packages were rebuilt, but they can now be opted out using this method if required. [1] https://fedorapeople.org/groups/schedule/f-43/f-43-key-tasks.html [2] https://fedoraproject.org/wiki/Fedora_43_Mass_Rebuild#Driving_Features

jnsamyak commented

2025-07-21 14:40:55 +00:00

Author

Owner

Metadata Update from @jnsamyak:

Issue tagged with: high-gain, high-trouble

**Metadata Update from @jnsamyak**: - Issue tagged with: high-gain, high-trouble

jnsamyak commented

2025-07-21 14:41:05 +00:00

Author

Owner

Metadata Update from @jnsamyak:

Issue assigned to jnsamyak

**Metadata Update from @jnsamyak**: - Issue assigned to jnsamyak

dreua commented

2025-07-21 20:17:34 +00:00

Contributor

That should be "2025-07-23" I guess?

jnsamyak commented

2025-07-22 06:47:17 +00:00

Author

Owner

@dreua, correct!

jnsamyak commented

2025-07-22 17:17:58 +00:00

Author

Owner

The tags & targets have been added, and the PR has been opened for the autosigning those!
I'll hold on to those details tomorrow to check before starting the process, it should be scheduled for 14 UTC.

PRs to review:

cc: @kevin @patrikp @humaton @amedvede

The tags & targets have been added, and the PR has been opened for the autosigning those! I'll hold on to those details tomorrow to check before starting the process, it should be scheduled for 14 UTC. PRs to review: - https://pagure.io/fedora-infra/ansible/pull-request/2761 - https://pagure.io/releng/pull-request/12834 cc: @kevin @patrikp @humaton @amedvede

jnsamyak commented

2025-07-23 10:33:51 +00:00

Author

Owner

While investigating the mass retirement script today I found out these are included in the need-rebuild but not in the pkg_skip_list, is there any specific reasons we just are missing out?

❌ Repo not found: Fedora-AtomicHost
❌ Repo not found: Fedora-AtomicHost-Vagrant
❌ Repo not found: Fedora-COSMIC-Disk
❌ Repo not found: Fedora-COSMIC-Live
❌ Repo not found: Fedora-Cloud-Atomic
❌ Repo not found: Fedora-Cloud-Atomic-Vagrant
❌ Repo not found: Fedora-Cloud-Base
❌ Repo not found: Fedora-Cloud-Base-Vagrant
❌ Repo not found: Fedora-Docker-Base
❌ Repo not found: Fedora-FEX-RootFS
❌ Repo not found: Fedora-KDE-Desktop-Disk
❌ Repo not found: Fedora-KDE-Desktop-Live
❌ Repo not found: Fedora-KDE-Mobile-Disk
❌ Repo not found: Fedora-KDE-Mobile-Live
❌ Repo not found: Fedora-LXQt-Disk
❌ Repo not found: Fedora-MiracleWM-Disk
❌ Repo not found: Fedora-MiracleWM-Live
❌ Repo not found: Fedora-Python-Classroom-Disk
❌ Repo not found: Fedora-Python-Classroom-Vagrant-VirtualBox
❌ Repo not found: Fedora-Python-Classroom-Vagrant-libvirt
❌ Repo not found: Fedora-Scientific-Vagrant
❌ Repo not found: Fedora-Scientific-Vagrant-VirtualBox
❌ Repo not found: Fedora-Scientific-Vagrant-libvirt
❌ Repo not found: Fedora-Scientific_KDE-Disk
❌ Repo not found: Fedora-Server-Guest-Generic
❌ Repo not found: Fedora-Server-Host-Generic
❌ Repo not found: Fedora-SoaS-Disk
❌ Repo not found: Fedora-WSL-Base
❌ Repo not found: Fedora-Workstation-Disk
❌ Repo not found: Fedora-Xfce-Disk
❌ Repo not found: Fedora-i3-Live
❌ Repo not found: andy-super-great-park
❌ Repo not found: fedora-container-image
❌ Repo not found: module-build-macros
❌ Repo not found: python3-doctutils

While investigating the mass retirement script today I found out these are included in the need-rebuild but not in the pkg_skip_list, is there any specific reasons we just are missing out? ``` ❌ Repo not found: Fedora-AtomicHost ❌ Repo not found: Fedora-AtomicHost-Vagrant ❌ Repo not found: Fedora-COSMIC-Disk ❌ Repo not found: Fedora-COSMIC-Live ❌ Repo not found: Fedora-Cloud-Atomic ❌ Repo not found: Fedora-Cloud-Atomic-Vagrant ❌ Repo not found: Fedora-Cloud-Base ❌ Repo not found: Fedora-Cloud-Base-Vagrant ❌ Repo not found: Fedora-Docker-Base ❌ Repo not found: Fedora-FEX-RootFS ❌ Repo not found: Fedora-KDE-Desktop-Disk ❌ Repo not found: Fedora-KDE-Desktop-Live ❌ Repo not found: Fedora-KDE-Mobile-Disk ❌ Repo not found: Fedora-KDE-Mobile-Live ❌ Repo not found: Fedora-LXQt-Disk ❌ Repo not found: Fedora-MiracleWM-Disk ❌ Repo not found: Fedora-MiracleWM-Live ❌ Repo not found: Fedora-Python-Classroom-Disk ❌ Repo not found: Fedora-Python-Classroom-Vagrant-VirtualBox ❌ Repo not found: Fedora-Python-Classroom-Vagrant-libvirt ❌ Repo not found: Fedora-Scientific-Vagrant ❌ Repo not found: Fedora-Scientific-Vagrant-VirtualBox ❌ Repo not found: Fedora-Scientific-Vagrant-libvirt ❌ Repo not found: Fedora-Scientific_KDE-Disk ❌ Repo not found: Fedora-Server-Guest-Generic ❌ Repo not found: Fedora-Server-Host-Generic ❌ Repo not found: Fedora-SoaS-Disk ❌ Repo not found: Fedora-WSL-Base ❌ Repo not found: Fedora-Workstation-Disk ❌ Repo not found: Fedora-Xfce-Disk ❌ Repo not found: Fedora-i3-Live ❌ Repo not found: andy-super-great-park ❌ Repo not found: fedora-container-image ❌ Repo not found: module-build-macros ❌ Repo not found: python3-doctutils ```

matiwari commented

2025-07-23 12:14:46 +00:00

A recent update to gettext-0.25.1, has been pushed to rawhide :

A recent update to gettext-0.25.1, has been pushed to rawhide : - https://bodhi.fedoraproject.org/updates/FEDORA-2025-c4d76ee10c - https://koji.fedoraproject.org/koji/taskinfo?taskID=135147460

jnsamyak commented

2025-07-23 12:59:56 +00:00

Author

Owner

@matiwari, I see it's pending for now, we have exactly an hour to run the rebuild!

petersen commented

2025-07-23 13:32:06 +00:00

A recent update to gettext-0.25.1, has been pushed to rawhide :

https://bodhi.fedoraproject.org/updates/FEDORA-2025-c4d76ee10c

Unfortunately it is still gated mostly due to some CI infra issues, though we would like to use it for the mass rebuild if possible.
Dunno if you want to tag into the buildroot anyway?: it has been tested in a couple of copr mass rebuilds anyway.

Actually I think it is probably stuck in the signing queue?

Also I believe @fberat is about to drop in a late autoconf patch: autoconf.2.72-6.fc43

> A recent update to gettext-0.25.1, has been pushed to rawhide : > - https://bodhi.fedoraproject.org/updates/FEDORA-2025-c4d76ee10c ~~Unfortunately it is still gated mostly due to some CI infra issues, though we would like to use it for the mass rebuild if possible. Dunno if you want to tag into the buildroot anyway?: it has been tested in a couple of copr mass rebuilds anyway.~~ _Actually I think it is probably stuck in the signing queue?_ Also I believe @fberat is about to drop in a late autoconf patch: autoconf.2.72-6.fc43

churchyard commented

2025-07-23 13:41:29 +00:00

Contributor

Any chance you could kick the robosignatory before the rebuild starts? The rawhide updates have not progressed for a while

https://admin.fedoraproject.org/collectd/bin/graph.cgi?hostname=rabbitmq_slash_pubsub;plugin=queues;plugin_instance=robosignatory;type=messages;begin=-43200

Any chance you could kick the robosignatory before the rebuild starts? The rawhide updates have not progressed for a while https://admin.fedoraproject.org/collectd/bin/graph.cgi?hostname=rabbitmq_slash_pubsub;plugin=queues;plugin_instance=robosignatory;type=messages;begin=-43200

fberat commented

2025-07-23 13:42:23 +00:00

Also I believe @fberat is about to drop in a late autoconf patch: autoconf.2.72-6.fc43

The build is running:
https://koji.fedoraproject.org/koji/taskinfo?taskID=135152050

Edit: https://bodhi.fedoraproject.org/updates/FEDORA-2025-d9c3ce5e46

> Also I believe @fberat is about to drop in a late autoconf patch: autoconf.2.72-6.fc43 The build is running: https://koji.fedoraproject.org/koji/taskinfo?taskID=135152050 Edit: https://bodhi.fedoraproject.org/updates/FEDORA-2025-d9c3ce5e46

jnsamyak commented

2025-07-23 14:40:13 +00:00

Author

Owner

@kevin just confirmed the vault was crashed, he restarted it, so it should be back soon! I'll see if everything looks fine will try to fire in 20 mins?

fberat commented

2025-07-23 14:41:39 +00:00

@jnsamyak Any chance both gettext and autoconf updates are taken in before the mass-rebuild kicks in ?

jnsamyak commented

2025-07-23 14:55:34 +00:00

Author

Owner

@fberat, I'll wait for these to sign and see if it gets into stable timely! :D

jnsamyak commented

2025-07-23 15:08:39 +00:00

Author

Owner

gettext -> stable
autoconf -> stable

@fberat all is good and done, we should be okay now!

gettext -> stable autoconf -> stable @fberat all is good and done, we should be okay now!

jnsamyak commented

2025-07-23 15:43:30 +00:00

Author

Owner

Mass Rebuild has been fired!

pghmcfc commented

2025-07-24 07:44:51 +00:00

It looks like there is a problem with debuginfo in golang packages:

e.g.
https://koji.fedoraproject.org/koji/taskinfo?taskID=135189702
error: Empty %files file /builddir/build/BUILD/golang-github-cloudflare-redoctober-0-build/redoctober-6a74ccc611f655e1ee9a648062a6ececfb6f2bb8/debugsourcefiles.list

https://koji.fedoraproject.org/koji/taskinfo?taskID=135180495
error: Empty %files file /builddir/build/BUILD/geoipupdate-7.1.1-build/geoipupdate-7.1.1/debugsourcefiles.list

It looks like there is a problem with debuginfo in golang packages: e.g. https://koji.fedoraproject.org/koji/taskinfo?taskID=135189702 error: Empty %files file /builddir/build/BUILD/golang-github-cloudflare-redoctober-0-build/redoctober-6a74ccc611f655e1ee9a648062a6ececfb6f2bb8/debugsourcefiles.list https://koji.fedoraproject.org/koji/taskinfo?taskID=135180495 error: Empty %files file /builddir/build/BUILD/geoipupdate-7.1.1-build/geoipupdate-7.1.1/debugsourcefiles.list

mjw commented

2025-07-24 13:11:50 +00:00

It looks like there is a problem with debuginfo in golang packages:

e.g.
https://koji.fedoraproject.org/koji/taskinfo?taskID=135189702
error: Empty %files file /builddir/build/BUILD/golang-github-cloudflare-redoctober-0-build/redoctober-6a74ccc611f655e1ee9a648062a6ececfb6f2bb8/debugsourcefiles.list

https://koji.fedoraproject.org/koji/taskinfo?taskID=135180495
error: Empty %files file /builddir/build/BUILD/geoipupdate-7.1.1-build/geoipupdate-7.1.1/debugsourcefiles.list

That is upstream debugedit bug https://sourceware.org/bugzilla/show_bug.cgi?id=33204
golang 1.25 generates dwarf5 by default that is incompatible with debugedit

> It looks like there is a problem with debuginfo in golang packages: > > e.g. > https://koji.fedoraproject.org/koji/taskinfo?taskID=135189702 > error: Empty %files file /builddir/build/BUILD/golang-github-cloudflare-redoctober-0-build/redoctober-6a74ccc611f655e1ee9a648062a6ececfb6f2bb8/debugsourcefiles.list > > https://koji.fedoraproject.org/koji/taskinfo?taskID=135180495 > error: Empty %files file /builddir/build/BUILD/geoipupdate-7.1.1-build/geoipupdate-7.1.1/debugsourcefiles.list > That is upstream debugedit bug https://sourceware.org/bugzilla/show_bug.cgi?id=33204 golang 1.25 generates dwarf5 by default that is incompatible with debugedit

mcrha commented

2025-07-25 09:31:49 +00:00

Does the mass rebuild .spec file update script not understand %autochangelog (yet)?

For example gnome-software had this in the .spec file:

%changelog
* Wed Jul 23 2025 Fedora Release Engineering <releng@fedoraproject.org> - 49~alpha-2
- Rebuilt for https://fedoraproject.org/wiki/Fedora_43_Mass_Rebuild

%autochangelog

I expect the mass rebuild will not add the lines when the %autochangelog is used.

Does the mass rebuild .spec file update script not understand `%autochangelog` (yet)? For example gnome-software had this in the .spec file: ``` %changelog * Wed Jul 23 2025 Fedora Release Engineering <releng@fedoraproject.org> - 49~alpha-2 - Rebuilt for https://fedoraproject.org/wiki/Fedora_43_Mass_Rebuild %autochangelog ``` I expect the mass rebuild will not add the lines when the `%autochangelog` is used.

dreua commented

2025-07-25 10:00:53 +00:00

Contributor

@mcrha it does understand %autochangelog and in combination with %autorelease it even creates an empty commit (example). Not sure why it didn't work in this case, maybe would be best to create a separate issue to look into it. (Are there more examples where you have seen this?)

@mcrha it does understand %autochangelog and in combination with %autorelease it even creates an empty commit ([example](https://src.fedoraproject.org/rpms/pdfarranger/c/850411b4a742a3a2b7414f74f13e1155f5c5cefa?branch=rawhide)). Not sure why it didn't work in this case, maybe would be best to create a separate issue to look into it. (Are there more examples where you have seen this?)

mcrha commented

2025-07-25 10:13:53 +00:00

It looks like packages which use only %autochangelog are affected. The gnome-shell is fine, but evolution is not. Where do I fill the bug, just here or the script has its own home, please?

It looks like packages which use _only_ `%autochangelog` are affected. The gnome-shell is fine, but evolution is not. Where do I fill the bug, just here or the script has its own home, please?

dreua commented

2025-07-25 12:11:26 +00:00

Contributor

You're right, it certainly looks like autochangelog without autorelease is triggering this. From a quick grep through "rpm-specs-latest" it looks like about 88 (false positives are possible) packages are affected by this. Many already have some added changelog entries above the %autochangelog entry from other rebuilds.

[... rpm-specs]$ grep -L "autorelease" $(grep -l "autochangelog" *) | wc -l
88

I'm not an expert here but from a look at the scripts I'd say the mass rebuild is using rpmdev-bumpspec for this task which has its own source repo here:
https://pagure.io/rpmdevtools/issues

You're right, it certainly looks like autochangelog without autorelease is triggering this. From a quick grep through "rpm-specs-latest" it looks like about 88 (false positives are possible) packages are affected by this. Many already have some added changelog entries above the %autochangelog entry from other rebuilds. ``` [... rpm-specs]$ grep -L "autorelease" $(grep -l "autochangelog" *) | wc -l 88 ``` I'm not an expert here but from a look at the scripts I'd say the mass rebuild is using rpmdev-bumpspec for this task which has its own source repo here: https://pagure.io/rpmdevtools/issues

mcrha commented

2025-07-28 07:11:04 +00:00

Thanks for the pointer. I filled https://pagure.io/rpmdevtools/issue/127 , though the project does not look like the most active, thus we'll see whether it'll get fixed any time soon.

Thanks for the pointer. I filled https://pagure.io/rpmdevtools/issue/127 , though the project does not look like the most active, thus we'll see whether it'll get fixed any time _soon_.

jnsamyak commented

2025-07-28 09:02:47 +00:00

Author

Owner

Mass rebuild is finished, everything is signed and tagged to f43; There were some blockers which shouldn't have been tagged (so I have untagged those); Finally, a rawhide is fired now, ✌️ hopefully nothing breaks!

Mass rebuild is finished, everything is signed and tagged to f43; There were some blockers which shouldn't have been tagged (so I have untagged those); Finally, a rawhide is fired now, :v: hopefully nothing breaks!

jnsamyak commented

2025-07-30 06:54:40 +00:00

Author

Owner

We have our first compose on Monday!

Approximately 23010 builds have been tagged into f43;
There are currently 1236 failed builds that need to be addressed
by the package maintainers.  FTBFS bugs will be filed shortly.

We have our first compose on Monday! ``` Approximately 23010 builds have been tagged into f43; There are currently 1236 failed builds that need to be addressed by the package maintainers. FTBFS bugs will be filed shortly. ```

jnsamyak commented

2025-07-30 06:54:41 +00:00

Author

Owner

Metadata Update from @jnsamyak:

Issue close_status updated to: Fixed
Issue status updated to: Closed (was: Open)

**Metadata Update from @jnsamyak**: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)

Rows
Columns

Fedora 43 Mass Rebuild Tracker #12833