Fix issue where updates-testing builds are never untagged pre GA

yselkowitz commented

2023-11-08 20:43:17 +00:00

Contributor

Describe the issue
waycheck-0.1.3-1.fc39 was not properly handled when it went from testing to stable:

$ koji list-tagged --inherit --quiet f39-updates-testing-pending waycheck
waycheck-0.1.3-1.fc39                     f39                   ngompa
waycheck-0.1.3-1.fc39                     f39-updates-testing   ngompa
waycheck-0.2.0-1.fc39                     f39                   ngompa
waycheck-1.0.0-1.fc39                     f39-updates           ngompa
$ koji latest-build --quiet f39-updates-testing-pending waycheck
waycheck-0.1.3-1.fc39                     f39-updates-testing   ngompa

This is causing flatpak-module-tools to think 0.1.3-1.fc39 is the version to target for building flatpaks, where it should be 1.0.0-1.fc39, and this will continue to occur (whenever the actual latest release is stable) for the duration of F39.

To that end, please koji untag-build f39-updates-testing waycheck-0.1.3-1.fc39

When do you need this? (YYYY/MM/DD)
Not urgent.
When is this no longer needed or useful? (YYYY/MM/DD)
March 2024, when we start migrating flatpaks to F40.
If we cannot complete your request, what is the impact?
It will take manual intervention to build the correct NVR of any future F39 updates to this package (unless done while itself still in testing) for F39 flatpaks.

fyi @otaylor

* Describe the issue waycheck-0.1.3-1.fc39 was not properly handled when it went from testing to stable: ``` $ koji list-tagged --inherit --quiet f39-updates-testing-pending waycheck waycheck-0.1.3-1.fc39 f39 ngompa waycheck-0.1.3-1.fc39 f39-updates-testing ngompa waycheck-0.2.0-1.fc39 f39 ngompa waycheck-1.0.0-1.fc39 f39-updates ngompa $ koji latest-build --quiet f39-updates-testing-pending waycheck waycheck-0.1.3-1.fc39 f39-updates-testing ngompa ``` This is causing flatpak-module-tools to think 0.1.3-1.fc39 is the version to target for building flatpaks, where it should be 1.0.0-1.fc39, and this will continue to occur (whenever the actual latest release is stable) for the duration of F39. To that end, please `koji untag-build f39-updates-testing waycheck-0.1.3-1.fc39` * When do you need this? (YYYY/MM/DD) Not urgent. * When is this no longer needed or useful? (YYYY/MM/DD) March 2024, when we start migrating flatpaks to F40. * If we cannot complete your request, what is the impact? It will take manual intervention to build the correct NVR of any future F39 updates to this package (unless done while itself still in testing) for F39 flatpaks. fyi @otaylor

phsmoura commented

2023-11-09 19:11:04 +00:00

Metadata Update from @phsmoura:

Issue tagged with: low-gain, low-trouble, ops

**Metadata Update from @phsmoura**: - Issue tagged with: low-gain, low-trouble, ops

kalev commented

2023-11-15 21:39:06 +00:00

Contributor

I don't think there's an expectation that things tagged with f39-updates-testing are always going to be higher version than what's tagged with f39 or f39-updates. Certainly bodhi doesn't enforce it in any way. Whenever a build gets karma fast enough to get queued for a stable push before it gets a chance to get pushed to testing, the stable tag gets the new build and nothing changes for the testing tag.

Another aspect is that during the pre-release phase, bodhi never untags builds from updates-testing when they migrate to stable, keeping them tagged with both. This is because bodhi doesn't control the base repo composes (which happen on cron, and might fail for several days in a row sometimes) and bodhi wants to ensure there's an overlap in time so that builds don't disappear from updates-testing and take several days before they appear again in the stable repo.

I don't think there's an expectation that things tagged with f39-updates-testing are always going to be higher version than what's tagged with f39 or f39-updates. Certainly bodhi doesn't enforce it in any way. Whenever a build gets karma fast enough to get queued for a stable push before it gets a chance to get pushed to testing, the stable tag gets the new build and nothing changes for the testing tag. Another aspect is that during the pre-release phase, bodhi never untags builds from updates-testing when they migrate to stable, keeping them tagged with both. This is because bodhi doesn't control the base repo composes (which happen on cron, and might fail for several days in a row sometimes) and bodhi wants to ensure there's an overlap in time so that builds don't disappear from updates-testing and take several days before they appear again in the stable repo.

kevin commented

2023-12-01 17:38:29 +00:00

Owner

Yeah, so I think we should add a releng step here to clean this up at/before GA, or perhaps better: get bodhi to clean it up after a few days.

I am going to manually untag the f38/f39 ones today. That is, those builds that are in both f39-updates-testing and f39 (so, it's pointless for them to be in updates-testing)

I am going to leave f37 alone as it goes EOL next week and I don't want to mess with it.

CC: @humaton @jnsamyak

Yeah, so I think we should add a releng step here to clean this up at/before GA, or perhaps better: get bodhi to clean it up after a few days. I am going to manually untag the f38/f39 ones today. That is, those builds that are in both f39-updates-testing and f39 (so, it's pointless for them to be in updates-testing) I am going to leave f37 alone as it goes EOL next week and I don't want to mess with it. CC: @humaton @jnsamyak

kevin commented

2023-12-01 18:23:13 +00:00

Owner

CC: @mattia also

mattia commented

2023-12-02 09:14:19 +00:00

I think the culprit is in github.com/fedora-infra/bodhi@b6abafcdf2/bodhi-server/bodhi/server/tasks/composer.py (L595-L599)

When a release is frozen, stable composes are "skipped" by bodhi-composer, so instead of moving builds from one tag to the other, only the newer tag is added.
From a first glance, I cannot see why this difference was introduced.

I think the culprit is in https://github.com/fedora-infra/bodhi/blob/b6abafcdf2a09246d178ea5b985c3c6e7b80c7ad/bodhi-server/bodhi/server/tasks/composer.py#L595-L599 When a release is frozen, stable composes are "skipped" by bodhi-composer, so instead of moving builds from one tag to the other, only the newer tag is added. From a first glance, I cannot see why this difference was introduced.

kevin commented

2023-12-03 22:44:13 +00:00

Owner

The difference is because of what @kalev said in the comment above. ;) In the time after updates-testing is enabled, but before GA, we don't want updates going from testing to stable to possibly 'disappear' from updates-testing before they appear in a compose. I guess the ideal solution would be to have the branched compose process untag builds from updates-testing if they are in the base tag. (after the compose completes).

kevin commented

2026-04-27 18:54:28 +00:00

Owner

Hit again with uv-0.10.6-1.fc44... it was moved stable a while back, but is still in testing and its signed copies got removed.

jnsamyak added this to the Backlog project

2026-04-30 08:46:19 +00:00

jnsamyak added

and removed

labels

2026-04-30 09:21:44 +00:00

jnsamyak commented

2026-04-30 10:00:40 +00:00

Owner

Okay, so one part of it I think can be handled by dealing with adding an if-else loop into the current one, by checking the state of the release, need to look into how, but when we do this, we could actually be able to continue to use the newer tag instead of updating as line github.com/fedora-infra/bodhi@b6abafcdf2/bodhi-server/bodhi/server/tasks/composer.py (L596)

Okay, so one part of it I think can be handled by dealing with adding an if-else loop into the current one, by checking the state of the release, need to look into how, but when we do this, we could actually be able to continue to use the newer tag instead of updating as line https://github.com/fedora-infra/bodhi/blob/b6abafcdf2a09246d178ea5b985c3c6e7b80c7ad/bodhi-server/bodhi/server/tasks/composer.py#L596

kevin commented

2026-05-04 17:56:46 +00:00

Owner

I'm not sure I follow there, but I think if we have bodhi check when doing testing composes for things that are already in the base tag it could just untag them from testing?

(That might be what you are saying... :)

I'm not sure I follow there, but I think if we have bodhi check when doing testing composes for things that are already in the base tag it could just untag them from testing? (That might be what you are saying... :)

Rows
Columns

Fix issue where updates-testing builds are never untagged pre GA #11775