Some packages tagged with "fXX-updates-testing" even after their bodhi update is claimed stable

mcrha commented

2026-05-07 07:37:53 +00:00

Description of request

The https://mdapi.fedoraproject.org/branches reports f41 as one of the active branches, but it's dead for almost six months.

The main page says for the f44 branch "Primary metadata was previously updated 3 minutes ago." (2025-05-07), but when I look at for example https://mdapi.fedoraproject.org/f44/pkg/evolution or https://mdapi.fedoraproject.org/f44/srcpkg/evolution , then it says the latest version is 3.60.0, part of updates-testing, but there is a higher version 3.60.1 in f44-updates https://koji.fedoraproject.org/koji/buildinfo?buildID=2976478 ; the https://koji.fedoraproject.org/koji/buildinfo?buildID=2958810 is part of f44 and f44-updates-testing, even though Bodhi pushed the update to stable https://bodhi.fedoraproject.org/updates/FEDORA-2026-9e90b1c6ba . This is not necessarily mdapi issue, but, maybe, it can catch on these things and not expect the updates-testing being always the latest version.

### Description of request The https://mdapi.fedoraproject.org/branches reports f41 as one of the active branches, but it's dead for almost six months. The main page says for the f44 branch "Primary metadata was previously updated 3 minutes ago." (2025-05-07), but when I look at for example https://mdapi.fedoraproject.org/f44/pkg/evolution or https://mdapi.fedoraproject.org/f44/srcpkg/evolution , then it says the latest version is 3.60.0, part of updates-testing, but there is a higher version 3.60.1 in `f44-updates` https://koji.fedoraproject.org/koji/buildinfo?buildID=2976478 ; the https://koji.fedoraproject.org/koji/buildinfo?buildID=2958810 is part of f44 and f44-updates-testing, even though Bodhi pushed the update to stable https://bodhi.fedoraproject.org/updates/FEDORA-2026-9e90b1c6ba . This is not necessarily mdapi issue, but, maybe, it can catch on these things and not expect the updates-testing being always the latest version.

james added the

labels

2026-05-07 19:06:12 +00:00

james commented

2026-05-07 19:06:43 +00:00

Member

@t0xic0der

t0xic0der self-assigned this

2026-05-08 04:47:50 +00:00

t0xic0der added

and removed

labels

2026-05-08 04:48:59 +00:00

t0xic0der commented

2026-05-08 04:53:53 +00:00

Member

I retriaged this issue ticket as the context already constitutes of at least two issue tickets worth of work.

@mcrha, could you please report these issues to the upstream https://github.com/fedora-infra/mdapi/issues?

The old branch removal thing is something that I can get to right away, as that only requires purging.

For the second one though, we could use some more example of packages where this ends up happening too.

At the moment, we assume that updates-testing would very likely be more recent than updates versions.

If we find more cases where that is not the case, we need to delve deeper to add a version check condition.

This would mean that we are going through all the databases to get the most recent version before sharing.

And since, we are making use of Sqlite3 backend for the same, we might expect a performance hit on queries.

I retriaged this issue ticket as the context already constitutes of at least two issue tickets worth of work. @mcrha, could you please report these issues to the upstream https://github.com/fedora-infra/mdapi/issues? The old branch removal thing is something that I can get to right away, as that only requires purging. For the second one though, we could use some more example of packages where this ends up happening too. At the moment, we assume that `updates-testing` would very likely be more recent than `updates` versions. If we find more cases where that is not the case, we need to delve deeper to add a version check condition. This would mean that we are going through all the databases to get the most recent version before sharing. And since, we are making use of Sqlite3 backend for the same, we might expect a performance hit on queries.

mcrha commented

2026-05-11 07:26:14 +00:00

Author

Could it be any package which received new upstream releases before a bodhi update was moved to stable? The fishy thing about the evolution package is that the corresponding 3.60.0 update is marked stable, thus the corresponding build should not be marked as f44-updates-testing, right?

t0xic0der commented

2026-05-11 08:34:32 +00:00

Member

@mcrha 3.60.0 was what was found in the updates-testing for evolution and hence was reported. The service does not look further into why something was versioned as such, but if we have more examples, we might just consider doing that.

@mcrha `3.60.0` was what was found in the `updates-testing` for `evolution` and hence was reported. The service does not look further into why something was versioned as such, but if we have more examples, we might just consider doing that.

mcrha commented

2026-05-11 09:14:46 +00:00

Author

Yes, I understood that. What I suspect is that the untag from f44-updates-testing after the update was claimed stable failed for some reason, which consequently broke the mdapi code as a side effect.

Then, maybe, this bug is filled at the correct place, because there should be investigated why the untag did not happen on the server.

For other example, see https://koji.fedoraproject.org/koji/buildinfo?buildID=2960504 and https://koji.fedoraproject.org/koji/buildinfo?buildID=2978168

Yes, I understood that. What I suspect is that the untag from f44-updates-testing after the update was claimed stable failed for some reason, which consequently broke the mdapi code as a side effect. Then, maybe, this bug is filled at the correct place, because there should be investigated why the untag did not happen on the server. For other example, see https://koji.fedoraproject.org/koji/buildinfo?buildID=2960504 and https://koji.fedoraproject.org/koji/buildinfo?buildID=2978168

t0xic0der commented

2026-05-11 10:38:04 +00:00

Member

Fair. For the first part though, I have filed the bug upstream on your behalf https://github.com/fedora-infra/mdapi/issues/414 but for the second part, maybe we should reword the description so that it does not look like a problem at MDAPI's end.

mcrha changed title from ~~mdapi providing old data~~ to Some packages tagged with "fXX-updates-testing" even after their bodhi update is claimed stable

2026-05-11 11:11:44 +00:00

mcrha commented

2026-05-11 11:12:41 +00:00

Author

Thanks for filling it. I thought you'd do it even without it, I'm sorry for my misunderstanding.

I re-titled the issue.

Thanks for filling it. I thought you'd do it even without it, I'm sorry for my misunderstanding. I re-titled the issue.

kevin commented

2026-05-11 15:52:11 +00:00

Owner

I think this is just a duplicate of releng/tickets#11775

TLDR: During freezes we don't untag updates from testing when they go stable, because of the timing of composes if we did that they would 'disappear' for a while before the next base compose happened and appear there.

I think this is just a duplicate of https://forge.fedoraproject.org/releng/tickets/issues/11775 TLDR: During freezes we don't untag updates from testing when they go stable, because of the timing of composes if we did that they would 'disappear' for a while before the next base compose happened and appear there.

mcrha commented

2026-05-11 16:57:41 +00:00

Author

the same with glycin in f43, it shows 2.0.3 (i686 for whatever reason, https://mdapi.fedoraproject.org/f43/srcpkg/glycin ), while the latest there is 2.0.8

@t0xic0der okay, it would be better to make mdapi reliable; I provided some examples, the issue kevin links has more; it's two years old issue. Do you want me to open a bug at https://github.com/fedora-infra/mdapi/issues to make it more cautious, to not trust updates-testing?

the same with `glycin` in f43, it shows 2.0.3 (i686 for whatever reason, https://mdapi.fedoraproject.org/f43/srcpkg/glycin ), while the latest there is 2.0.8 @t0xic0der okay, it would be better to make mdapi reliable; I provided some examples, the issue kevin links has more; it's two years old issue. Do you want me to open a bug at https://github.com/fedora-infra/mdapi/issues to make it more cautious, to not trust updates-testing?

t0xic0der commented

2026-05-12 04:17:14 +00:00

Member

Ok, for the first part, I have shipped https://github.com/fedora-infra/mdapi/pull/415 that should be available on the staging environment shortly. @mcrha, if you can help with testing it out before it hits the production, that would be awesome.

For the second part, I still think that making changes at the foundational level would be more scaleable than having changes in ecosystem applications, just because the untagging never comes to fruition due to the variants getting skipped. From the conversations listed in releng/tickets#11775, I see that they seem to have figured out what causes this problem. I would much rather let the changes happen there and have its consequences trickle down to MDAPI, than going ahead with a change at my end.

Ok, for the first part, I have shipped https://github.com/fedora-infra/mdapi/pull/415 that should be available on the staging environment shortly. @mcrha, if you can help with testing it out before it hits the production, that would be awesome. For the second part, I still think that making changes at the foundational level would be _more scaleable_ than having changes in ecosystem applications, just because the untagging never comes to fruition due to the variants getting skipped. From the conversations listed in https://forge.fedoraproject.org/releng/tickets/issues/11775, I see that they seem to have figured out what causes this problem. I would _much rather_ let the changes happen there and have its consequences trickle down to MDAPI, than going ahead with a change at my end.

mcrha commented

2026-05-12 07:10:55 +00:00

Author

if you can help with testing it out before it hits the production, that would be awesome.

I do not know how I would do that, I do not manage fedora infrastructure servers, to be able to apply it "for testing purposes". It seems you received some confirmation there meanwhile.

I would much rather let the changes happen there and have its consequences trickle down to MDAPI, than going ahead with a change at my end.

I understand this policy, better to fix the root cause than to add workarounds. In this case, you still need to consult (up to?) three tags, do you not? It's fXX-updates-testing, fXX-updates and fXX. Being part of fXX-updates-testing does not mean the package is available for the end users, they need to enable the updates-testing repo to get packages from there, where the "testing" explains why it's so. That is, when I want to get the latest version available for the users in Fedora XX, I'm not interested in packages in the updates-testing, which may or may not make it to the users (depending how the testing will go).

> if you can help with testing it out before it hits the production, that would be awesome. I do not know how I would do that, I do not manage fedora infrastructure servers, to be able to apply it "for testing purposes". It seems you received some confirmation there meanwhile. > I would much rather let the changes happen there and have its consequences trickle down to MDAPI, than going ahead with a change at my end. I understand this policy, better to fix the root cause than to add workarounds. In this case, you still need to consult (up to?) three tags, do you not? It's `fXX-updates-testing`, `fXX-updates` and `fXX`. Being part of `fXX-updates-testing` does not mean the package is available for the end users, they need to enable the updates-testing repo to get packages from there, where the "testing" explains why it's so. That is, when I want to get the latest version available for the users in Fedora XX, I'm not interested in packages in the updates-testing, which may or may not make it to the users (depending how the testing will go).

kevin commented

2026-05-13 19:41:54 +00:00

Owner

FYI, the staging mdapi is available at: https://mdapi.stg.fedoraproject.org/

t0xic0der commented

2026-05-18 06:19:01 +00:00

Member

The scheduled purging of obsolete metadata works. Once this is deployed on production, this ticket can be closed.

t0xic0der commented

2026-05-18 06:32:24 +00:00

Member

https://github.com/fedora-infra/mdapi/pull/416 was merged and confirmed to be working. Closing this as complete.

t0xic0der closed this issue

2026-05-18 06:32:24 +00:00

Rows
Columns

Some packages tagged with "fXX-updates-testing" even after their bodhi update is claimed stable #13332

Description of request