releng/tickets

Fork 2

[release-process] File FTBFS Bug post Mass Rebuild #13426

New issue

Closed

opened 2026-07-13 14:44:37 +00:00 by jnsamyak · 10 comments

jnsamyak commented

2026-07-13 14:44:37 +00:00

Owner

Describe the issue

After every mass rebuild, we file FTBFS bugs for the packages that failed to build during mass rebuild.

This should be run after the mass rebuild builds are merged into main tag.

When do you need this? (YYYY/MM/DD)

Post F45 Mass Rebuild

When is this no longer needed or useful? (YYYY/MM/DD)

F45 EOL

If we cannot complete this, what is the impact? [Dependencies/Blocker]

oh no, can't happen :D We need to tell maintainers for failing packages so that can fix it before we do freezes

ref: #13406

Checklist

I have checked existing issues for duplicates
I have filled out all the fields above
I have provided relevant links or references (if applicable)

## Describe the issue After every mass rebuild, we file FTBFS bugs for the packages that failed to build during mass rebuild. This should be run after the mass rebuild builds are merged into main **tag.** ## When do you need this? (YYYY/MM/DD) Post F45 Mass Rebuild ## When is this no longer needed or useful? (YYYY/MM/DD) F45 EOL ## If we cannot complete this, what is the impact? [Dependencies/Blocker] oh no, can't happen :D We need to tell maintainers for failing packages so that can fix it before we do freezes ref: https://forge.fedoraproject.org/releng/tickets/issues/13406 --- ## Checklist - [x] I have checked existing issues for duplicates - [x] I have filled out all the fields above - [x] I have provided relevant links or references (if applicable)

jnsamyak added the

labels

2026-07-13 14:44:37 +00:00

jnsamyak self-assigned this

2026-07-13 14:44:37 +00:00

jnsamyak added this to the Sprint ( 2026-07-13 to 2026-07-26 ) project

2026-07-13 14:44:37 +00:00

patrikp referenced this issue

2026-07-15 17:59:55 +00:00

F45 Cycle: Mass Rebuild (scheduled July 15th, 2026 14 UTC) #13406

pnemade commented

2026-07-20 08:20:02 +00:00

Member

I have hunspell-* packages which are still in need-rebuild status. What will happen to them, as I consider them as not FTBFS packages, how to know reason about why they not submitted to koji?

jnsamyak commented

2026-07-20 15:07:16 +00:00

Author

Owner

Ok might be a problem with need-rebuild script:

I see mostly all of them are build

massbuild_attempt4.out:Committing changes for hunspell-xh
massbuild_attempt4.out:Pushing changes for hunspell-xh
massbuild_attempt4.out:To ssh://pkgs.fedoraproject.org/rpms/hunspell-xh
massbuild_attempt4.out:Getting git url for hunspell-xh
massbuild_attempt4.out:Building hunspell-xh
massbuild_attempt4.out:  Successful submission: hunspell-xh  taskID: 147862038
massbuild_attempt4.out:Checking out hunspell-yi
massbuild_attempt4.out:Cloning into 'hunspell-yi'...
massbuild_attempt4.out:Bumping /root/massbuild/hunspell-yi/hunspell-yi.spec
massbuild_attempt4.out:Committing changes for hunspell-yi
massbuild_attempt4.out:Pushing changes for hunspell-yi
massbuild_attempt4.out:To ssh://pkgs.fedoraproject.org/rpms/hunspell-yi
massbuild_attempt4.out:Getting git url for hunspell-yi
massbuild_attempt4.out:Building hunspell-yi
massbuild_attempt4.out:  Successful submission: hunspell-yi  taskID: 147862039
massbuild_attempt4.out:Checking out hunspell-zu
massbuild_attempt4.out:Cloning into 'hunspell-zu'...
massbuild_attempt4.out:Bumping /root/massbuild/hunspell-zu/hunspell-zu.spec
massbuild_attempt4.out:Committing changes for hunspell-zu
massbuild_attempt4.out:Pushing changes for hunspell-zu
massbuild_attempt4.out:To ssh://pkgs.fedoraproject.org/rpms/hunspell-zu
massbuild_attempt4.out:Getting git url for hunspell-zu
massbuild_attempt4.out:Building hunspell-zu
massbuild_attempt4.out:  Successful submission: hunspell-zu  taskID: 147862041
massbuild_attempt4.out:Checking out mingw-hunspell
massbuild_attempt4.out:Cloning into 'mingw-hunspell'...
massbuild_attempt4.out:Bumping /root/massbuild/mingw-hunspell/mingw-hunspell.spec
massbuild_attempt4.out:Committing changes for mingw-hunspell
massbuild_attempt4.out:Pushing changes for mingw-hunspell
massbuild_attempt4.out:To ssh://pkgs.fedoraproject.org/rpms/mingw-hunspell
massbuild_attempt4.out:Getting git url for mingw-hunspell
massbuild_attempt4.out:Building mingw-hunspell
massbuild_attempt4.out:  Successful submission: mingw-hunspell  taskID: 147876195
massbuild_attempt4.out:Checking out pyhunspell
massbuild_attempt4.out:Cloning into 'pyhunspell'...
massbuild_attempt4.out:Bumping /root/massbuild/pyhunspell/pyhunspell.spec
massbuild_attempt4.out:Committing changes for pyhunspell
massbuild_attempt4.out:Pushing changes for pyhunspell
massbuild_attempt4.out:To ssh://pkgs.fedoraproject.org/rpms/pyhunspell
massbuild_attempt4.out:Getting git url for pyhunspell
massbuild_attempt4.out:Building pyhunspell
massbuild_attempt4.out:  Successful submission: pyhunspell  taskID: 147898050

the logs are much bigger sharing some, can you list down specific packages please so i can check

Ok might be a problem with need-rebuild script: I see mostly all of them are build ``` massbuild_attempt4.out:Committing changes for hunspell-xh massbuild_attempt4.out:Pushing changes for hunspell-xh massbuild_attempt4.out:To ssh://pkgs.fedoraproject.org/rpms/hunspell-xh massbuild_attempt4.out:Getting git url for hunspell-xh massbuild_attempt4.out:Building hunspell-xh massbuild_attempt4.out: Successful submission: hunspell-xh taskID: 147862038 massbuild_attempt4.out:Checking out hunspell-yi massbuild_attempt4.out:Cloning into 'hunspell-yi'... massbuild_attempt4.out:Bumping /root/massbuild/hunspell-yi/hunspell-yi.spec massbuild_attempt4.out:Committing changes for hunspell-yi massbuild_attempt4.out:Pushing changes for hunspell-yi massbuild_attempt4.out:To ssh://pkgs.fedoraproject.org/rpms/hunspell-yi massbuild_attempt4.out:Getting git url for hunspell-yi massbuild_attempt4.out:Building hunspell-yi massbuild_attempt4.out: Successful submission: hunspell-yi taskID: 147862039 massbuild_attempt4.out:Checking out hunspell-zu massbuild_attempt4.out:Cloning into 'hunspell-zu'... massbuild_attempt4.out:Bumping /root/massbuild/hunspell-zu/hunspell-zu.spec massbuild_attempt4.out:Committing changes for hunspell-zu massbuild_attempt4.out:Pushing changes for hunspell-zu massbuild_attempt4.out:To ssh://pkgs.fedoraproject.org/rpms/hunspell-zu massbuild_attempt4.out:Getting git url for hunspell-zu massbuild_attempt4.out:Building hunspell-zu massbuild_attempt4.out: Successful submission: hunspell-zu taskID: 147862041 massbuild_attempt4.out:Checking out mingw-hunspell massbuild_attempt4.out:Cloning into 'mingw-hunspell'... massbuild_attempt4.out:Bumping /root/massbuild/mingw-hunspell/mingw-hunspell.spec massbuild_attempt4.out:Committing changes for mingw-hunspell massbuild_attempt4.out:Pushing changes for mingw-hunspell massbuild_attempt4.out:To ssh://pkgs.fedoraproject.org/rpms/mingw-hunspell massbuild_attempt4.out:Getting git url for mingw-hunspell massbuild_attempt4.out:Building mingw-hunspell massbuild_attempt4.out: Successful submission: mingw-hunspell taskID: 147876195 massbuild_attempt4.out:Checking out pyhunspell massbuild_attempt4.out:Cloning into 'pyhunspell'... massbuild_attempt4.out:Bumping /root/massbuild/pyhunspell/pyhunspell.spec massbuild_attempt4.out:Committing changes for pyhunspell massbuild_attempt4.out:Pushing changes for pyhunspell massbuild_attempt4.out:To ssh://pkgs.fedoraproject.org/rpms/pyhunspell massbuild_attempt4.out:Getting git url for pyhunspell massbuild_attempt4.out:Building pyhunspell massbuild_attempt4.out: Successful submission: pyhunspell taskID: 147898050 ``` the logs are much bigger sharing some, can you list down specific packages please so i can check

decathorpe commented

2026-07-20 15:59:01 +00:00

Contributor

I tried to query koji for failed buildSRPMFromSCM tasks that happened during the mass rebuild:

import koji
session = koji.ClientSession("https://koji.fedoraproject.org/kojihub")
result = session.callMethod("listTasks", opts={"method": "buildSRPMFromSCM", "state": [koji.TASK_STATES["FAILED"]], "createdAfter": "2026-07-15 00:00:00", "owner": 3445})
packages = [item["request"].split(".org/rpms/", 1)[1].split(".git")[0] for item in result] # yes this is ugly but it worked

Results (54 packages):

Charliecloud
bazelisk
co2mon
falkon-pdfreader
glob2
golang-modernc-ccorpus2
hunspell-am
httpry
hunspell-csb
hunspell-ast
hunspell-az
hugin
hunspell-bn
httpcomponents-core
hunspell-af
httrack
httperf
hstr
ht-alegreya-fonts
hstsparser
hipfft
ht-alegreya-sans-fonts
hfg-gmuend-openmoji-fonts
hercules
httprobe
hugo
hunspell-ak
hunspell-ca
human-theme-gtk
hunspell-da
httpdump
hunspell-el
hunspell-cy
lcmaps
kolf
krb5
liboil
liboggz
mhwaveedit
openbox-theme-mistral-thin-dark
openbox-theme-mistral-thin
openttd
openttd-opengfx
openxr
opentype-sanitizer
openvino
openvkl
openxr-simple-playground
perl-App-grindperl
psi4
python-rtslib
python3-prctl
slick-greeter
zonefs-tools

I tried to query koji for failed buildSRPMFromSCM tasks that happened during the mass rebuild: ```python import koji session = koji.ClientSession("https://koji.fedoraproject.org/kojihub") result = session.callMethod("listTasks", opts={"method": "buildSRPMFromSCM", "state": [koji.TASK_STATES["FAILED"]], "createdAfter": "2026-07-15 00:00:00", "owner": 3445}) packages = [item["request"].split(".org/rpms/", 1)[1].split(".git")[0] for item in result] # yes this is ugly but it worked ``` Results (54 packages): Charliecloud bazelisk co2mon falkon-pdfreader glob2 golang-modernc-ccorpus2 hunspell-am httpry hunspell-csb hunspell-ast hunspell-az hugin hunspell-bn httpcomponents-core hunspell-af httrack httperf hstr ht-alegreya-fonts hstsparser hipfft ht-alegreya-sans-fonts hfg-gmuend-openmoji-fonts hercules httprobe hugo hunspell-ak hunspell-ca human-theme-gtk hunspell-da httpdump hunspell-el hunspell-cy lcmaps kolf krb5 liboil liboggz mhwaveedit openbox-theme-mistral-thin-dark openbox-theme-mistral-thin openttd openttd-opengfx openxr opentype-sanitizer openvino openvkl openxr-simple-playground perl-App-grindperl psi4 python-rtslib python3-prctl slick-greeter zonefs-tools

pnemade commented

2026-07-20 16:02:51 +00:00

Member

I read the output of https://kojipkgs.fedoraproject.org/mass-rebuild/f45-need-rebuild.html with my username pnemade

hunspell-af
hunspell-ak
hunspell-am
hunspell-ast
hunspell-az
hunspell-bn
hunspell-ca
hunspell-csb
hunspell-cy
hunspell-da
hunspell-el

I read the output of https://kojipkgs.fedoraproject.org/mass-rebuild/f45-need-rebuild.html with my username pnemade > hunspell-af > hunspell-ak > hunspell-am > hunspell-ast > hunspell-az > hunspell-bn > hunspell-ca > hunspell-csb > hunspell-cy > hunspell-da > hunspell-el

kevin commented

2026-07-20 16:35:14 +00:00

Owner

Yeah, those seem to have failed due to some kind of infrastructure issue where they couldn't clone the git repo.

You should be able to just do 'fedpkg build' in all of them and they should disappear from the needs-rebuild page.

Yeah, those seem to have failed due to some kind of infrastructure issue where they couldn't clone the git repo. You should be able to just do 'fedpkg build' in all of them and they should disappear from the needs-rebuild page.

pnemade commented

2026-07-20 16:37:42 +00:00

Member

@kevin wrote in #13426 (comment):

Yeah, those seem to have failed due to some kind of infrastructure issue where they couldn't clone the git repo.

You should be able to just do 'fedpkg build' in all of them and they should disappear from the needs-rebuild page.

Thanks. Will be building now.

@kevin wrote in https://forge.fedoraproject.org/releng/tickets/issues/13426#issuecomment-1067223: > Yeah, those seem to have failed due to some kind of infrastructure issue where they couldn't clone the git repo. > > You should be able to just do 'fedpkg build' in all of them and they should disappear from the needs-rebuild page. Thanks. Will be building now.

decathorpe commented

2026-07-21 10:29:39 +00:00

Contributor

I've gone ahead and resubmitted all builds where tasks failed in buildSRPMFromSCM that didn't already have a new build or changes newer than the mass rebuild.

Most failures were due to git clone / fedpkg sources flukes, but some are caused by actual broken spec files or broken / missing sources. All of these are actual issues so they should get FTBFS bugs filed for them:

package	buildSRPMFromSCM failure reason
Charliecloud	duplicate package; Both charliecloud and Charliecloud exist, and both have `Name: charliecloud` in their spec. I suppose the capitalized version should be retired? koji is confused about both existing too.
co2mon	source checksum mismatch
falkon-pdfreader	source checksum mismatch
glob2	spec file fails to parse
golang-modernc-ccorpus2	sources not present in lookaside cache (?)
mhwaveedit	sources not present in lookaside cache (?)
openbox-theme-mistral-thin	source checksum mismatch
openbox-theme-mistral-thin-dark	source checksum mismatch
psi4	sources not uploaded to lookaside cache (?)
python-rtslib	sources not uploaded to lookaside cache (?)
python3-prctl	wrong source package name, spec file has `Name: python-prctl`
psi4	sources not uploaded to lookaside cache (?)
slick-greeter	missing patch / patch not committed to dist-git
zonefs-tools	sources not uploaded to lookaside cache (?)

Some of the packages I resubmitted are now proper FTBFS, not sure if they'll show up on the f45-failures list now - these should also get FTBFS bugs:

hugo
lcmaps
openvino

I've gone ahead and resubmitted all builds where tasks failed in buildSRPMFromSCM that didn't already have a new build or changes newer than the mass rebuild. Most failures were due to git clone / fedpkg sources flukes, but some are caused by actual broken spec files or broken / missing sources. All of these are actual issues so they should get FTBFS bugs filed for them: | package | buildSRPMFromSCM failure reason | | --- | --- | | Charliecloud | duplicate package; Both charliecloud and Charliecloud exist, and both have `Name: charliecloud` in their spec. I suppose the capitalized version should be retired? koji is confused about both existing too. | | co2mon | source checksum mismatch | | falkon-pdfreader | source checksum mismatch | | glob2 | spec file fails to parse | | golang-modernc-ccorpus2 | sources not present in lookaside cache (?) | | mhwaveedit | sources not present in lookaside cache (?) | | openbox-theme-mistral-thin | source checksum mismatch | | openbox-theme-mistral-thin-dark | source checksum mismatch | | psi4 | sources not uploaded to lookaside cache (?) | | python-rtslib | sources not uploaded to lookaside cache (?) | | python3-prctl | wrong source package name, spec file has `Name: python-prctl` | | psi4 | sources not uploaded to lookaside cache (?) | | slick-greeter | missing patch / patch not committed to dist-git | | zonefs-tools | sources not uploaded to lookaside cache (?) | Some of the packages I resubmitted are now proper FTBFS, not sure if they'll show up on the f45-failures list now - these should also get FTBFS bugs: - hugo - lcmaps - openvino

jnsamyak referenced this issue from releng/tooling

2026-07-21 17:08:08 +00:00

[bugfiling] FTBFS bug filing post F45 Mass Rebuild - Update f46 ftbfs bz tracker #13047

jnsamyak commented

2026-07-21 17:08:39 +00:00

Author

Owner

I'll need to check, but for starters, two new bugzilla tickets have been opened:

Fedora 46 Fails To install: https://bugzilla.redhat.com/show_bug.cgi?id=2503729
Fedora 46 FTBFS Tracker: https://bugzilla.redhat.com/show_bug.cgi?id=2503726

PR for adding to tooling before I start filing bugs: releng/tooling#13047

I'll need to check, but for starters, two new bugzilla tickets have been opened: - Fedora 46 Fails To install: https://bugzilla.redhat.com/show_bug.cgi?id=2503729 - Fedora 46 FTBFS Tracker: https://bugzilla.redhat.com/show_bug.cgi?id=2503726 PR for adding to tooling before I start filing bugs: https://forge.fedoraproject.org/releng/tooling/pulls/13047

jnsamyak commented

2026-07-22 04:56:52 +00:00

Author

Owner

The bugs for failures were filed, I'll manually look for filing the one @decathorpe mentioned now.

❤️ 1

jnsamyak commented

2026-07-22 05:06:28 +00:00

Author

Owner

That too is now done, thanks. Closing these one now.

jnsamyak closed this issue