releng/tickets

Fork 2

Mass rebuild scripts should check Bodhi and not bump/build anything that failed gating #13428

New issue

Open

opened 2026-07-16 18:44:37 +00:00 by adamwill · 5 comments

adamwill commented

2026-07-16 18:44:37 +00:00

Member

Describe the issue

Every cycle we do a mass rebuild, and every time I wind up hastily scrolling through the Bodhi list of gated builds while the rebuild is going on, or after it finishes, and trying to flag up cases where we're rebuilding something that was gated because it failed tests.

This is dumb and we shouldn't do it. What should happen is that the mass rebuild scripts should check Bodhi and omit any package if the most recent build of it failed gating. Or, I guess it could equally just check whether current dist-git state matches the most recently-tagged build, and omit the package if it doesn't. Whatever: we should not blindly rebuild and merge packages that have failed gating tests, that is a bad idea.

When do you need this? (YYYY/MM/DD)

Before the F46 mass rebuild, I guess.

When is this no longer needed or useful? (YYYY/MM/DD)

When we stop doing blind mass rebuilds?

If we cannot complete this, what is the impact? [Dependencies/Blocker]

We will continue to merge builds of known-broken states.

## Describe the issue Every cycle we do a mass rebuild, and every time I wind up hastily scrolling through the Bodhi list of gated builds while the rebuild is going on, or after it finishes, and trying to flag up cases where we're rebuilding something that was gated because it failed tests. This is dumb and we shouldn't do it. What should happen is that the mass rebuild scripts should check Bodhi and omit any package if the most recent build of it failed gating. Or, I guess it could equally just check whether current dist-git state matches the most recently-tagged build, and omit the package if it doesn't. Whatever: we should not blindly rebuild and merge packages that have failed gating tests, that is a bad idea. ## When do you need this? (YYYY/MM/DD) Before the F46 mass rebuild, I guess. ## When is this no longer needed or useful? (YYYY/MM/DD) When we stop doing blind mass rebuilds? ## If we cannot complete this, what is the impact? [Dependencies/Blocker] We will continue to merge builds of known-broken states.

adamwill referenced this issue from quality/tickets

2026-07-16 23:52:53 +00:00

Check gated updates and flag up any which might cause problems in the mass rebuild #914

jnsamyak commented

2026-07-17 02:21:04 +00:00

Owner

Okay, I think these points to mass tagging script we use after the mass rebuild https://forge.fedoraproject.org/releng/tooling/src/branch/main/release-process/mass-rebuilds/mass_tag.py;

I'll try to brainstorm this idea on how we can achieve that.

Okay, I think these points to mass tagging script we use after the mass rebuild https://forge.fedoraproject.org/releng/tooling/src/branch/main/release-process/mass-rebuilds/mass_tag.py; I'll try to brainstorm this idea on how we can achieve that.

jnsamyak added the

labels

2026-07-17 03:02:58 +00:00

jnsamyak self-assigned this

2026-07-17 03:03:08 +00:00

jnsamyak commented

2026-07-17 04:22:03 +00:00

Owner

Ahh, read the ticket again, maybe after all we can tweak the rebuild script only.

clone pkg
  → latest tagged in desttag?
       no  → rebuild
       yes → HEAD matches tagged source (+ releng-only commits)?
              yes → rebuild
              no  → skip + report

WDYT @adamwill, I think adding bodhi check in the script will slow things down, also it would be bringing one failure mode as we add one more service for rebuild (what if bodhi spots responding, error, etc.).

Ahh, read the ticket again, maybe after all we can tweak the rebuild script only. ``` clone pkg → latest tagged in desttag? no → rebuild yes → HEAD matches tagged source (+ releng-only commits)? yes → rebuild no → skip + report ``` WDYT @adamwill, I think adding bodhi check in the script will slow things down, also it would be bringing one failure mode as we add one more service for rebuild (what if bodhi spots responding, error, etc.).

adamwill commented

2026-07-17 06:05:27 +00:00

Author

Member

what does "latest tagged in desttag" mean?

jnsamyak commented

2026-07-17 06:13:51 +00:00

Owner

here: https://forge.fedoraproject.org/releng/tooling/src/branch/main/release-process/mass-rebuilds/mass_rebuilds_info.py#L129

jnsamyak added this to the Current Sprint ( 2026-07-27 to 2026-08-09 ) project

2026-07-19 13:32:12 +00:00

jnsamyak referenced this issue from a commit

2026-08-10 13:57:14 +00:00

Skip mass rebuilds when dist-git is ahead of the tagged build

jnsamyak referenced this issue from releng/tooling

2026-08-10 13:57:39 +00:00

Skip mass rebuilds when dist-git is ahead of the tagged build #13051

jnsamyak commented

2026-08-10 13:58:02 +00:00

Owner

A starting point to discuss this issue: releng/tooling#13051 (Update: We need more discussion to happen here, moving it to next sprint. The PR is here but again the work is not done unless the PR is merged.)

A starting point to discuss this issue: https://forge.fedoraproject.org/releng/tooling/pulls/13051 (Update: We need more discussion to happen here, moving it to next sprint. The PR is here but again the work is not done unless the PR is merged.)

jnsamyak modified the project from Current Sprint ( 2026-07-27 to 2026-08-09 ) to Current Sprint ( 2026-08-10 to 2026-08-23 )

2026-08-10 14:22:31 +00:00