bluez package incorrectly versioned, test failures inappropriately waived

adamwill commented

2026-09-16 22:12:35 +00:00

Member

I'd like to bring this update to FPC's attention: https://bodhi.fedoraproject.org/updates/FEDORA-2026-77f32a44ce

The sequence of events, AFAICT:

@hadess bumped bluez to a git snapshot. The versioning he used was not compliant with the guidelines - he did not use e.g. Version: 5.87^20260909git8750129efca8, Release: 1%{?dist} (new style) or Version: 5.87, Release: 8.20260909git8750129efca8%{?dist} (old style). He used Version: 5.87+1.git8750129efca8, Release: 1%{?dist}, which is completely non-standard and non-compliant.
@pbrobinson bumped to a later snapshot. He changed the git commit hash from 8750129efca8 to 789f6e154 and the release to 2%{?dist}, but did not change the +1 in the Version at all. That meant that his build was versioned 5.87+1.git789f6e154-2.fc46, which is lower than the previous build, 5.87+1.git8750129efca8-1.fc46 (you can confirm this with rpmdev-vercmp). (This situation is exactly why using non-standard versioning approaches is a bad idea; maybe you know what you did, but the next person to touch the package probably doesn't).
Lots of openQA tests failed on the update, correctly, because there is a specific check which catches this. E.g. https://openqa.fedoraproject.org/tests/5326957#step/_advisory_post/37 . There are a variety of causes of the "installed newer than update" state caught here, but "the update is incorrectly lower versioned than the current stable package" is certainly one of them.
@pbrobinson then waived all the failures with the flippant message "whatevah". I find this disappointing behavior from a long-established proven packager. He does not appear to have made any effort to contact the quality team for help understanding the failures, as the Bodhi web UI specifically suggests (on updates with failed gating tests).

This needs to be cleaned up, somehow, for future builds, and I'd like to ask the FPC to formally request that @hadess and @pbrobinson pay more attention to the packaging guidelines and do not inappropriately waive test failures. They've both been around long enough to know better.

I'd like to bring this update to FPC's attention: https://bodhi.fedoraproject.org/updates/FEDORA-2026-77f32a44ce The sequence of events, AFAICT: 1. @hadess [bumped bluez to a git snapshot](https://src.fedoraproject.org/rpms/bluez/c/86e689ec80200a8443ee646b4f67ff411eda6497?branch=rawhide). The versioning he used was not compliant with [the guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/#_snapshots) - he did not use e.g. `Version: 5.87^20260909git8750129efca8`, `Release: 1%{?dist}` (new style) or `Version: 5.87`, `Release: 8.20260909git8750129efca8%{?dist}` (old style). He used `Version: 5.87+1.git8750129efca8`, `Release: 1%{?dist}`, which is completely non-standard and non-compliant. 2. @pbrobinson [bumped to a later snapshot](https://src.fedoraproject.org/rpms/bluez/c/d790badb0f4e80001b7267ddd441d5caa8e1ee1b?branch=rawhide). He changed the git commit hash from 8750129efca8 to 789f6e154 and the release to `2%{?dist}`, but did not change the `+1` in the `Version` at all. That meant that his build was versioned `5.87+1.git789f6e154-2.fc46`, which is *lower* than the previous build, `5.87+1.git8750129efca8-1.fc46` (you can confirm this with rpmdev-vercmp). (This situation is *exactly* why using non-standard versioning approaches is a bad idea; maybe you know what you did, but the next person to touch the package probably doesn't). 3. Lots of openQA tests failed on the update, correctly, because there is a specific check which catches this. E.g. https://openqa.fedoraproject.org/tests/5326957#step/_advisory_post/37 . There are a variety of causes of the "installed newer than update" state caught here, but "the update is incorrectly lower versioned than the current stable package" is certainly one of them. 4. @pbrobinson then [waived all the failures](https://waiverdb.fedoraproject.org/api/v1.0/waivers/?subject_identifier=FEDORA-2026-77f32a44ce) with the flippant message "whatevah". I find this disappointing behavior from a long-established proven packager. He does not appear to have made any effort to contact the quality team for help understanding the failures, as the Bodhi web UI specifically suggests (on updates with failed gating tests). This needs to be cleaned up, somehow, for future builds, and I'd like to ask the FPC to formally request that @hadess and @pbrobinson pay more attention to the packaging guidelines and do not inappropriately waive test failures. They've both been around long enough to know better.

ngompa commented

2026-09-16 23:24:50 +00:00

Owner

I think this is actually something for FESCo rather than FPC? This is about technical contributor conduct, and that needs to be bumped up to FESCo.

👍 1

adamwill referenced this issue from quality/planning

2026-09-17 00:34:56 +00:00

openQA failure monitoring 2026 sprint 15 #104

pbrobinson commented

2026-09-17 08:08:52 +00:00

Member

4. @pbrobinson then [waived all the failures](https://waiverdb.fedoraproject.org/api/v1.0/waivers/?subject_identifier=FEDORA-2026-77f32a44ce) with the flippant message "whatevah". I find this disappointing behavior from a long-established proven packager. He does not appear to have made any effort to contact the quality team for help understanding the failures, as the Bodhi web UI specifically suggests (on updates with failed gating tests).

I waived them because I couldn't actually see what was failing because it just sat there with a spinner, if the infrastructure just sits there and makes you wait for ever for something and you need to get something done I have no idea how long you're expected to wait. I only actually pushed the update because hadess asked as he's out.

> 4. @pbrobinson then [waived all the failures](https://waiverdb.fedoraproject.org/api/v1.0/waivers/?subject_identifier=FEDORA-2026-77f32a44ce) with the flippant message "whatevah". I find this disappointing behavior from a long-established proven packager. He does not appear to have made any effort to contact the quality team for help understanding the failures, as the Bodhi web UI specifically suggests (on updates with failed gating tests). I waived them because I couldn't actually see what was failing because it just sat there with a spinner, if the infrastructure just sits there and makes you wait for ever for something and you need to get something done I have no idea how long you're expected to wait. I only actually pushed the update because hadess asked as he's out.

pbrobinson commented

2026-09-17 08:15:28 +00:00

Member

long-established proven packager.

I would also point out I am a bluez maintainer so this has nothing to do with proven packagers

> long-established proven packager. I would also point out I am a bluez maintainer so this has nothing to do with proven packagers

churchyard commented

2026-09-17 11:26:53 +00:00

Owner

I agree this is a FESCo matter. However, technically, all packagers should pay attention to the packaging guidelines and not inappropriately waive test failures. Not sure if formally saying that to anyone makes a difference :/

pbrobinson commented

2026-09-17 12:53:36 +00:00

Member

Which would be fine if you can actually get the details of the issues form the platform in any level of time that is useful. If I have to wait 5 mins for a spinner to update actual details I have moved on and forgotten what I was even doing there.

pbrobinson commented

2026-09-17 13:26:48 +00:00

Member

I apologise not paying more attention to the versioning, I didn't make the change to move to snapshots and when bumping the git rev to deal with a CVE as hadess asked if I could deal as he was out I just assumed it was correct. I will fix this.
The 'whatev' comment in the waiving is the default comment I use when the system is too slow to provide actual details (you can look back through the history). It's not meant to be flippant to anyone or any Fedora process. I apologise if it comes across as such, it's more a tongue-in-cheek poke at the CI system in frustration with the speed of the CI system's process.

1) I apologise not paying more attention to the versioning, I didn't make the change to move to snapshots and when bumping the git rev to deal with a CVE as hadess asked if I could deal as he was out I just assumed it was correct. I will fix this. 2) The 'whatev' comment in the waiving is the default comment I use when the system is too slow to provide actual details (you can look back through the history). It's not meant to be flippant to anyone or any Fedora process. I apologise if it comes across as such, it's more a tongue-in-cheek poke at the CI system in frustration with the speed of the CI system's process.

adamwill commented

2026-09-17 15:20:44 +00:00

Author

Member

I just loaded the update page and it loaded all the test results in about a second.

It can vary because the detailed view is produced on the front end (while it's loading, you just see the overall gating status); your browser requests the status from greenwave with a 'verbose=1' option (IIRC, this is all from memory) which causes it to fetch all the results from resultsdb and forward them to your browser, which then constructs the table. If there's any slow link in that chain it'll be slow. It can take some time on updates with a lot of packages because we have to do multiple greenwave requests, but this is a single-package update so that would not be the case. I'd guess you just ran into some kind of infra wobble.

Regardless, I don't think "oh I'll just waive the failures, I'm a busy guy" is an appropriate response. You could come and ask us. We know how to just go look up the results directly in openQA (you just go to the front page, click "Fedora Updates", expand the view to 100 or 400 builds if necessary, and search for the update ID), and we could have done that and told you what the problem was.

I just loaded the update page and it loaded all the test results in about a second. It can vary because the detailed view is produced on the front end (while it's loading, you just see the overall gating status); your browser requests the status from greenwave with a 'verbose=1' option (IIRC, this is all from memory) which causes it to fetch all the results from resultsdb and forward them to your browser, which then constructs the table. If there's any slow link in that chain it'll be slow. It can take some time on updates with a lot of packages because we have to do multiple greenwave requests, but this is a single-package update so that would not be the case. I'd guess you just ran into some kind of infra wobble. Regardless, I don't think "oh I'll just waive the failures, I'm a busy guy" is an appropriate response. You could come and ask us. We know how to just go look up the results directly in openQA (you just go to the front page, click "Fedora Updates", expand the view to 100 or 400 builds if necessary, and search for the update ID), and we could have done that and told you what the problem was.

james commented

2026-09-17 18:46:34 +00:00

Owner

So FPC discussed this for basically the entire meeting this week, feel free to read what was said directly: https://meetbot.fedoraproject.org/meeting-1_matrix_fedoraproject-org/2026-09-17/fpc.2026-09-17-16.01.log.html

I'll try to summarize it nicely here, but I am human.

What we can do

Probably the main thing we can actually do, that might help here in the future, is to simplify the versioning guidelines.

We currently allow ~6 different formats for how to version a snapshot for an rpm. In general FPC tries to formally write down best practices of what packagers do, so sometimes we hit things like this where people tried a lot of things and there wasn't an obvious reason to disallow some of them so we just try to limit growth. But I can see how here it looks a lot like "there's a lot of ways to do it, so why not my slightly different version that I think works fine".

@carlwgeorge and @ngompa are going to work on the changes, and it's very likely we'll have a single format that everyone should use with a "perfect is the enemy of good, just do it this way for everyone's sake" like wording. Possible that an autoQA test could complain "this looks like a snapshot, but isn't using the one true format".

We also discussed having some kind of macro for producing snapshot versions, but there's a weird dumpster fire there even if we could get everyone to use it.

What we can't do

We don't have an enforcement arm, can't give out demerit points, or even anything like debian janitors ... so it's long been known that after a new package is approved FPC can't really do much (and technically it's the reviewer following our "rules" that enforce things for new packages).

It'd be nice if all the packagers bowed to collective experience, or even just accepted that everyone else's time was as valuable as theirs.

“Hope” is the thing with feathers -
That perches in the soul

Things we would probably support

It seems very likely that some autoQA tests shouldn't be able to be immediately waived, even by very experienced proven packagers, Eg. "this update is actually older than the current version"

If someone proposes that some tests become "needs a fix (Eg. epoch bump for lower version), nobody can waive it" or even having a bigger category of "these need to be waived by a QA person" I think it's very likely that FPC would support either of those changes.

Final notes

I'm going to leave this issue open for a bit, so people can respond or complain about what we said, but again "we" can't do a lot here.

So FPC discussed this for basically the entire meeting this week, feel free to read what was said directly: https://meetbot.fedoraproject.org/meeting-1_matrix_fedoraproject-org/2026-09-17/fpc.2026-09-17-16.01.log.html I'll try to summarize it nicely here, but I am human. ### What we can do Probably the main thing we can actually do, that might help here in the future, is to simplify the versioning guidelines. We currently allow ~6 different formats for how to version a snapshot for an rpm. In general FPC tries to formally write down best practices of what packagers do, so sometimes we hit things like this where people tried a lot of things and there wasn't an obvious reason to disallow some of them so we just try to limit growth. But I can see how here it looks a lot like "there's a lot of ways to do it, so why not my slightly different version that I think works fine". @carlwgeorge and @ngompa are going to work on the changes, and it's very likely we'll have a single format that everyone should use with a "perfect is the enemy of good, just do it this way for everyone's sake" like wording. Possible that an autoQA test could complain "this looks like a snapshot, but isn't using the one true format". We also discussed having some kind of macro for producing snapshot versions, but there's a weird dumpster fire there even if we could get everyone to use it. ### What we can't do We don't have an enforcement arm, can't give out demerit points, or even anything like debian janitors ... so it's long been known that after a new package is approved FPC can't really do much (and technically it's the reviewer following our "rules" that enforce things for new packages). It'd be nice if all the packagers bowed to collective experience, or even just accepted that everyone else's time was as valuable as theirs. “Hope” is the thing with feathers - That perches in the soul ### Things we would probably support It seems very likely that some autoQA tests shouldn't be able to be immediately waived, even by very experienced proven packagers, Eg. "this update is actually older than the current version" If someone proposes that some tests become "needs a fix (Eg. epoch bump for lower version), nobody can waive it" or even having a bigger category of "these need to be waived by a QA person" I think it's very likely that FPC would support either of those changes. ### Final notes I'm going to leave this issue open for a bit, so people can respond or complain about what we said, but again "we" can't do a lot here.

adamwill commented

2026-09-17 18:58:17 +00:00

Author

Member

If someone proposes that some tests become "needs a fix (Eg. epoch bump for lower version), nobody can waive it" or even having a bigger category of "these need to be waived by a QA person" I think it's very likely that FPC would support either of those changes.

I'm happy with this. I've had it in my mind for a while that we should make it so openQA tests, specifically, can't just be waived by anybody.

We do need to check the actual capabilities of waiverdb, greenwave and Bodhi around different policies, though.

> If someone proposes that some tests become "needs a fix (Eg. epoch bump for lower version), nobody can waive it" or even having a bigger category of "these need to be waived by a QA person" I think it's very likely that FPC would support either of those changes. I'm happy with this. I've had it in my mind for a while that we should make it so openQA tests, specifically, can't just be waived by anybody. We do need to check the actual capabilities of waiverdb, greenwave and Bodhi around different policies, though.

pbrobinson commented

2026-09-18 15:59:28 +00:00

Member

The sequence of events, AFAICT:

1. @hadess [bumped bluez to a git snapshot](https://src.fedoraproject.org/rpms/bluez/c/86e689ec80200a8443ee646b4f67ff411eda6497?branch=rawhide). The versioning he used was not compliant with [the guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/#_snapshots) - he did not use e.g. `Version: 5.87^20260909git8750129efca8`, `Release: 1%{?dist}` (new style) or `Version: 5.87`, `Release: 8.20260909git8750129efca8%{?dist}` (old style). He used `Version: 5.87+1.git8750129efca8`, `Release: 1%{?dist}`, which is completely non-standard and non-compliant.

I've been looking at this piece to bring it back into compliance (new style)

So @hadess in the spec referenced this packaging spec URL possibly not realising the packaging docs have moved to a new location and based on this, in particular the piece covering the Snapshot Packages and to quote:

kismet-1.0-1%{?dist} (this is the formal release of kismet 1.0)
kismet-1.0-2%{?dist} (this is a bugfix build to the 1.0 release)
kismet-1.0+cvs20050515-1%{?dist} (move to a post-release cvs checkout, note the reset of the Release section)
kismet-1.0+cvs20050515-2%{?dist} (bugfix to the post-release cvs checkout)
kismet-1.0+cvs20050517-1%{?dist} (new cvs checkout, note the reset of the release tag)

So I believe he believed he was in compliance, and in dealing with package update (there was a CVE he had notified me of pre PTO) before I started my dayjob, I didn't look to closely as I assumed it was correct.

I think also the packaging committee needs to clean up wayward drafts that have been dealt with so search engines or AI assistants don't get the wrong end of the stick too.

> The sequence of events, AFAICT: > > 1. @hadess [bumped bluez to a git snapshot](https://src.fedoraproject.org/rpms/bluez/c/86e689ec80200a8443ee646b4f67ff411eda6497?branch=rawhide). The versioning he used was not compliant with [the guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/#_snapshots) - he did not use e.g. `Version: 5.87^20260909git8750129efca8`, `Release: 1%{?dist}` (new style) or `Version: 5.87`, `Release: 8.20260909git8750129efca8%{?dist}` (old style). He used `Version: 5.87+1.git8750129efca8`, `Release: 1%{?dist}`, which is completely non-standard and non-compliant. I've been looking at this piece to bring it back into compliance (new style) So @hadess in the spec referenced [this packaging spec URL](https://fedoraproject.org/wiki/PackagingDrafts/TildeVersioning) possibly not realising the packaging docs have moved to a new location and based on this, in particular the piece covering the [Snapshot Packages](https://fedoraproject.org/wiki/PackagingDrafts/TildeVersioning#Snapshot_Packages) and to quote: ``` kismet-1.0-1%{?dist} (this is the formal release of kismet 1.0) kismet-1.0-2%{?dist} (this is a bugfix build to the 1.0 release) kismet-1.0+cvs20050515-1%{?dist} (move to a post-release cvs checkout, note the reset of the Release section) kismet-1.0+cvs20050515-2%{?dist} (bugfix to the post-release cvs checkout) kismet-1.0+cvs20050517-1%{?dist} (new cvs checkout, note the reset of the release tag) ``` So I believe he believed he was in compliance, and in dealing with package update (there was a CVE he had notified me of pre PTO) before I started my dayjob, I didn't look to closely as I assumed it was correct. I think also the packaging committee needs to clean up wayward drafts that have been dealt with so search engines or AI assistants don't get the wrong end of the stick too.

pbrobinson commented

2026-09-18 21:03:44 +00:00

Member

So I've pushed an updated build with in conformance and it still fails in a bazillion ways and looking through a random chuck of them and they all fail with:

# Test died: Package(s) from update not installed when it should have been! See script output at fedora/lib/utils.pm line 1407, <$isotovideo> line 1.

And I'm confused what that means and how I'm supposed to interpret it.

Also can someone explain why podman, podman_client, kiwi_build, realmd_join_sssd, domain controllers and god knows what else have to do with bluez?

@adamwill can you please go into details why these errors have anything to do with a bluetooth update and why the perl error that I see on every error on the screenshot in OpenQA that says Failed and how a average packager is supposed to debug that?

So I've pushed an updated build with in conformance and it still fails in a bazillion ways and looking through a random chuck of them and they all fail with: ``` # Test died: Package(s) from update not installed when it should have been! See script output at fedora/lib/utils.pm line 1407, <$isotovideo> line 1. ``` And I'm confused what that means and how I'm supposed to interpret it. Also can someone explain why podman, podman_client, kiwi_build, realmd_join_sssd, domain controllers and god knows what else have to do with bluez? @adamwill can you please go into details why these errors have anything to do with a bluetooth update and why the perl error that I see on every error on the screenshot in OpenQA that says Failed and how a average packager is supposed to debug that?

Screenshot From 2026-09-18 21-49-41.png

179 KiB

adamwill commented

2026-09-19 00:27:06 +00:00

Author

Member

I commented on the update already, three hours before you posted this. The correct versioning sorts lower than the incorrect versioning. So I think our choices are to stick with the incorrect versioning but bump the digit behind the + with each build, or declare that these are not "post-5.87 snapshots" but "pre-5.88 snapshots" and version as 5.88~20260918gitb5ed56d8c-<N>.fc46, because that should sort higher than the incorrect versions.

The openQA check that's failing is a sort of final step sanity check that's run on almost all the tests, no matter what the test was for. That's why you see a bunch of failures in apparently-unrelated tests. Once the test is otherwise successful, we check what packages that are in the update are installed on the test system. Then we check their versions. If any package that's in the update is present on the system, but the installed version is not the version from the update, that's a fail. The message can't really be much more specific because there are various ways this can occur, though it does say "See script output", which you can see on the previous frame - https://openqa.fedoraproject.org/tests/5346440#step/_advisory_post/35 for e.g., where you can see "Installed: is newer than update: and update is not stable or obsolete", indicating the installed package is higher versioned than the package from the update. So you'll see a failure for any test where bluez happens to be installed on the SUT and this check is run (there are a few tests where it's not run for various reasons).

I added this check long ago after a few updates passed testing but broke stuff when they landed; it turned out they had versioning/dependency issues which caused them never to get installed during the tests, so the tests passed but did not actually test the packages from the update. When these updates went stable and replaced the previous, working versions (that had worked in the tests), things broke.

I [commented on the update](https://bodhi.fedoraproject.org/updates/FEDORA-2026-1e1db3562e#comment-4779475) already, three hours before you posted this. The correct versioning sorts lower than the incorrect versioning. So I think our choices are to stick with the incorrect versioning but bump the digit behind the `+` with each build, or declare that these are not "post-5.87 snapshots" but "pre-5.88 snapshots" and version as `5.88~20260918gitb5ed56d8c-<N>.fc46`, because that should sort higher than the incorrect versions. The openQA check that's failing is a sort of final step sanity check that's run on almost all the tests, no matter what the test was for. That's why you see a bunch of failures in apparently-unrelated tests. Once the test is otherwise successful, we check what packages that are in the update are installed on the test system. Then we check their versions. If any package that's in the update is present on the system, but the installed version is *not* the version from the update, that's a fail. The message can't really be much more specific because there are various ways this can occur, though it does say "See script output", which you can see on the previous frame - https://openqa.fedoraproject.org/tests/5346440#step/_advisory_post/35 for e.g., where you can see "Installed: <x> is newer than update: <y> and update is not stable or obsolete", indicating the installed package is higher versioned than the package from the update. So you'll see a failure for any test where bluez happens to be installed on the SUT and this check is run (there are a few tests where it's not run for various reasons). I added this check long ago after a few updates passed testing but broke stuff when they landed; it turned out they had versioning/dependency issues which caused them *never to get installed* during the tests, so the tests passed but did not actually test the packages from the update. When these updates went stable and replaced the previous, working versions (that had worked in the tests), things broke.

pbrobinson commented

2026-09-19 06:53:03 +00:00

Member

@adamwill wrote in #1569 (comment):

I commented on the update already, three hours before you posted this. The correct versioning sorts lower than the incorrect versioning. So I think our choices are to stick with the incorrect versioning but bump the digit behind the + with each build, or declare that these are not "post-5.87 snapshots" but "pre-5.88 snapshots" and version as 5.88~20260918gitb5ed56d8c-<N>.fc46, because that should sort higher than the incorrect versions.

I saw your comment there, you opened this ticket so I thought we should keep the conversation on the associated ticket.

Presumably the other fix is to bump the epoch.

@adamwill wrote in https://forge.fedoraproject.org/packaging/guidelines/issues/1569#issuecomment-1358396: > I [commented on the update](https://bodhi.fedoraproject.org/updates/FEDORA-2026-1e1db3562e#comment-4779475) already, three hours before you posted this. The correct versioning sorts lower than the incorrect versioning. So I think our choices are to stick with the incorrect versioning but bump the digit behind the `+` with each build, or declare that these are not "post-5.87 snapshots" but "pre-5.88 snapshots" and version as `5.88~20260918gitb5ed56d8c-<N>.fc46`, because that should sort higher than the incorrect versions. I saw your comment there, you opened this ticket so I thought we should keep the conversation on the associated ticket. Presumably the other fix is to bump the epoch.

adamwill commented

2026-09-19 07:00:08 +00:00

Author

Member

Yeah but I don't think that's a good idea as it just makes the issue permanent. If you don't like the 'make it a 5.88 pre' idea, then I think just stick with the 'wrong' versioning, but increment the counter so we don't have the problem of sorting on the commit hash.

pbrobinson commented

2026-09-19 07:22:02 +00:00

Member

So without me actually going into rpm code please explain what is making this versioning break.

If we look at the bits right behind the 5.87 bit comparing the last tagged version in f45 to the recent onest:

Tagged:
bluez-5.87+1.git8750129efca8-1.fc45

My first. So here git7 is lower than git8:
bluez-5.87+1.git789f6e154-2.fc45

My second:
bluez-5.87^20260918gitb5ed56d8c-3.fc46

The 2 here is bigger than 1 in both cases so is rpm evaluating ^ as a higher char than +? I presume the former has a higher ACSII number than the later and hence is causing the upgrade issue?

I don't get your "So I think our choices are to stick with the incorrect versioning but bump the digit behind the + with each build" point, there is no longer a plus.

So without me actually going into rpm code please explain what is making this versioning break. If we look at the bits right behind the 5.87 bit comparing the last tagged version in f45 to the recent onest: Tagged: bluez-5.87+1.git8750129efca8-1.fc45 My first. So here git7 is lower than git8: bluez-5.87+1.git789f6e154-2.fc45 My second: bluez-5.87^20260918gitb5ed56d8c-3.fc46 The 2 here is bigger than 1 in both cases so is rpm evaluating ^ as a higher char than +? I presume the former has a higher ACSII number than the later and hence is causing the upgrade issue? I don't get your "So I think our choices are to stick with the incorrect versioning but bump the digit behind the + with each build" point, there is no longer a plus.

adamwill commented

2026-09-19 19:58:52 +00:00

Author

Member

Well yeah, I assume RPM decides + beats ^. I think ^ and ~ were made "canonical" for RPM upstream so that's why we're meant to use them; they're treated 'specially' so they should always work the way we'd want for pre/post-releases. + is just...a random character that gets evaluated...somehow. But rpmdev-vercmp uses RPM's actual comparison mechanism so it always gives the correct result:

[adamw@toolbx fedora-toolbox-44 kmscon (rawhide)]$ rpmdev-vercmp 0 5.87+1.git789f6e154 2.fc45 0 5.87^20260918gitb5ed56d8c 3.fc46
0:5.87+1.git789f6e154-2.fc45 > 0:5.87^20260918gitb5ed56d8c-3.fc46

from which it's pretty clear that yes, it must decide + is 'higher' than ^, otherwise we would not get that result. So the fact that hadess put the + in there means we essentially can only "win" with:

another character that sorts higher than a +
a + and a higher number after it
something higher than 5.87 before the +

I don't get your "So I think our choices are to stick with the incorrect versioning but bump the digit behind the + with each build" point, there is no longer a plus.

I meant go back to hadess' incorrect 5.87+1.git<hash> style and just bump the 1 to 2 (and then remember to bump it to 3 for the build after that, and so on). If there's no cleaner option, we're just stuck with it until you reach 5.88 at which point we're clear and you can use the right format for any future snapshots. I think it's arguable whether that's the best option, or the "call it a 5.88 pre-release snapshot" plan is the best option.

Well yeah, I assume RPM decides + beats ^. I think ^ and ~ were made "canonical" for RPM upstream so that's why we're meant to use them; they're treated 'specially' so they should always work the way we'd want for pre/post-releases. + is just...a random character that gets evaluated...somehow. But `rpmdev-vercmp` uses RPM's actual comparison mechanism so it always gives the correct result: ``` [adamw@toolbx fedora-toolbox-44 kmscon (rawhide)]$ rpmdev-vercmp 0 5.87+1.git789f6e154 2.fc45 0 5.87^20260918gitb5ed56d8c 3.fc46 0:5.87+1.git789f6e154-2.fc45 > 0:5.87^20260918gitb5ed56d8c-3.fc46 ``` from which it's pretty clear that yes, it must decide + is 'higher' than ^, otherwise we would not get that result. So the fact that hadess put the + in there means we essentially can only "win" with: 1. another character that sorts higher than a + 2. a + and a higher number after it 3. something higher than 5.87 before the + > I don't get your "So I think our choices are to stick with the incorrect versioning but bump the digit behind the + with each build" point, there is no longer a plus. I meant go back to hadess' incorrect `5.87+1.git<hash>` style and just bump the 1 to 2 (and then remember to bump it to 3 for the build after that, and so on). If there's no cleaner option, we're just stuck with it until you reach 5.88 at which point we're clear and you can use the right format for any future snapshots. I think it's arguable whether that's the best option, or the "call it a 5.88 pre-release snapshot" plan is the best option.

carlwgeorge commented

2026-09-21 05:37:06 +00:00

Owner

In RPM sorting, ., _, and + are functionally equivalent separators (e.g. 5.87+1 is the same thing as 5.87.1). ~ and ^ trigger special lower/higher logic off the rest of the version to their left, but don't override additional segments to their left. + isn't really "higher" than ^, they're just handled differently, but I can see how if you squint just right that's how they seem. I guess it's higher in the same way that 1.0.1 is higher than 1.0^1.

https://rpm.org/docs/latest/man/rpm-version.7

My recommendation for a guidelines-compliant solution is 1:5.87^1.b5ed56d8c.

Epoch 1 to sort higher than everything else so far (not a big deal, this is the kind of thing it exists for).
Base version of 5.87.
Snapshot format of <number>.<revision> following a ^. Additional snapshots from the same base version should increment this number for proper sorting.

In RPM sorting, `.`, `_`, and `+` are functionally equivalent separators (e.g. `5.87+1` is the same thing as `5.87.1`). `~` and `^` trigger special lower/higher logic off the rest of the version to their left, but _don't override additional segments to their left_. `+` isn't really "higher" than `^`, they're just handled differently, but I can see how if you squint just right that's how they seem. I guess it's higher in the same way that `1.0.1` is higher than `1.0^1`. https://rpm.org/docs/latest/man/rpm-version.7 My recommendation for a guidelines-compliant solution is `1:5.87^1.b5ed56d8c`. * Epoch `1` to sort higher than everything else so far (not a big deal, this is the kind of thing it exists for). * Base version of `5.87`. * Snapshot format of `<number>.<revision>` following a `^`. Additional snapshots from the same base version should increment this number for proper sorting.

carlwgeorge commented

2026-09-23 03:52:54 +00:00

Owner

or declare that these are not "post-5.87 snapshots" but "pre-5.88 snapshots" and version as 5.88~20260918gitb5ed56d8c-<N>.fc46

The risk with this approach is that it requires the next upstream version to be 5.88 to sort correctly. If upstream releases 5.87.1, it will need an epoch to sort correctly.

> or declare that these are not "post-5.87 snapshots" but "pre-5.88 snapshots" and version as `5.88~20260918gitb5ed56d8c-<N>.fc46` The risk with this approach is that it requires the next upstream version to be 5.88 to sort correctly. If upstream releases 5.87.1, it will need an epoch to sort correctly.

adamwill commented

2026-09-23 05:22:31 +00:00

Author

Member

That's true. I was assuming knowledge of upstream's intentions there, and of course replace 5.88 with whatever the next version will in fact be.

hadess commented

2026-10-07 13:08:22 +00:00

Member

I'm back from PTO. Can somebody TLDR this for me?

Have the draft docs been removed to point to the correct ones that should be used now?

The reason why we're shipping snapshots is that the upstream release process is currently not in the hands of the folks actually maintaining the code, and I'm part of that upstream with no say as to when a release is made.

I applied the same versioning scheme to the centos and RHEL packages for bluez, in all the branches, so I would likely have to make changes there as well to match Fedora and not create myself even more work.

I'm back from PTO. Can somebody TLDR this for me? Have the draft docs been removed to point to the correct ones that should be used now? The reason why we're shipping snapshots is that the upstream release process is currently not in the hands of the folks actually maintaining the code, and I'm part of that upstream with no say as to when a release is made. I applied the same versioning scheme to the centos and RHEL packages for bluez, in all the branches, so I would likely have to make changes there as well to match Fedora and not create myself even more work.

adamwill commented

2026-10-07 15:57:31 +00:00

Author

Member

tl;dr is that the mess still exists. Peter tried to convert to the correct versioning in this update but it fails tests as it compares lower than the incorrect versioning which is in stable. I suggested several possible ways forward but it doesn't look like he went with any yet.

The options AFAICS are:

Use correct versioning, add an epoch
Decide that these are 'pre-5.88' or 'pre-5.87.1' or whatever snapshots, not 'post-5.87' snapshots, and use correct versioning: I tested that e.g. 5.88~20260918gitb5ed56d8c-<N>.fc46 sorts higher than the current stable version. You can also do 5.88~0.20260918gitb5ed56d8c-<N>.fc46, i.e. add an rev counter ahead of the date, if you may need multiple snapshots on the same date
Stick with the incorrect versioning until a new release happens, then don't do it again in future. But to do this you would need to increment the counter you put in the version with each build so we are not relying on the git commit hashes as the decider. So, do a build versioned bluez-5.87+2.gitb5ed56d8c-2.fc46 (or whatever snapshot you want to use), then bump it to 3, 4, and so on with future post-5.87 snapshots, until a stable release happens and we can forget about this mess.

I don't think the committee has ventured an opinion on which option is preferable.

tl;dr is that the mess still exists. Peter tried to convert to the correct versioning in [this update](https://bodhi.fedoraproject.org/updates/FEDORA-2026-1e1db3562e) but it fails tests as it compares lower than the incorrect versioning which is in stable. I suggested several possible ways forward but it doesn't look like he went with any yet. The options AFAICS are: 1) Use correct versioning, add an epoch 2) Decide that these are 'pre-5.88' or 'pre-5.87.1' or whatever snapshots, not 'post-5.87' snapshots, and use correct versioning: I tested that e.g. `5.88~20260918gitb5ed56d8c-<N>.fc46` sorts higher than the current stable version. You can also do `5.88~0.20260918gitb5ed56d8c-<N>.fc46`, i.e. add an rev counter ahead of the date, if you may need multiple snapshots on the same date 3) Stick with the incorrect versioning until a new release happens, then don't do it again in future. But to do this you would need to increment the counter you put in the version with each build so we are not relying on the git commit hashes as the decider. So, do a build versioned `bluez-5.87+2.gitb5ed56d8c-2.fc46` (or whatever snapshot you want to use), then bump it to 3, 4, and so on with future post-5.87 snapshots, until a stable release happens and we can forget about this mess. I don't think the committee has ventured an opinion on which option is preferable.

carlwgeorge commented

2026-10-07 18:01:34 +00:00

Owner

Have the draft docs been removed to point to the correct ones that should be used now?

The only correct docs for this are the official guidelines. Anything else is unofficial and should probably be ignored. The PackagingDrafts/TildeVersioning that was linked earlier has a big yellow box at the top with a warning sign stating "do not rely on the information on this page". Since it's a wiki, anyone can expand that warning with more text if they feel like that isn't clear enough. Some of the other non-draft guidelines pages on the wiki have done exactly that, with a link to the equivalent page in the official docs (e.g. Packaging:LicensingGuidelines).

I don't think the committee has ventured an opinion on which option is preferable.

The committee as a whole hasn't ventured an opinion. I have in this comment as an individual but also as a committee member. Option 1 is the safest bet. Option 2 and 3 rely on correctly predicting what the next upstream version will be, and if predicted incorrectly will need an epoch to get back on track anyways.

The committee has been discussing simplifying the snapshot guidelines in our last two meeting and in #1570. Hopefully once that discussion concludes the snapshot guidelines will be easier to follow and easier to get right.

> Have the draft docs been removed to point to the correct ones that should be used now? The only correct docs for this are the [official guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/). Anything else is unofficial and should probably be ignored. The [PackagingDrafts/TildeVersioning](https://fedoraproject.org/wiki/PackagingDrafts/TildeVersioning) that was linked earlier has a big yellow box at the top with a warning sign stating "do not rely on the information on this page". Since it's a wiki, anyone can expand that warning with more text if they feel like that isn't clear enough. Some of the other non-draft guidelines pages on the wiki have done exactly that, with a link to the equivalent page in the official docs (e.g. [Packaging:LicensingGuidelines](https://fedoraproject.org/wiki/Packaging:LicensingGuidelines)). > I don't think the committee has ventured an opinion on which option is preferable. The committee as a whole hasn't ventured an opinion. I have in [this comment](https://forge.fedoraproject.org/packaging/guidelines/issues/1569#issuecomment-1358648) as an individual but also as a committee member. Option 1 is the safest bet. Option 2 and 3 rely on correctly predicting what the next upstream version will be, and if predicted incorrectly will need an epoch to get back on track anyways. The committee has been discussing simplifying the snapshot guidelines in our last two meeting and in #1570. Hopefully once that discussion concludes the snapshot guidelines will be easier to follow and easier to get right.

adamwill commented

2026-10-07 18:10:37 +00:00

Author

Member

Option 2 and 3 rely on correctly predicting what the next upstream version will be, and if predicted incorrectly will need an epoch to get back on track anyways.

I mean, I'd flip that around and say "we may as well try fixing it without an Epoch first, since we can always add an Epoch later if nothing else works".

> Option 2 and 3 rely on correctly predicting what the next upstream version will be, and if predicted incorrectly will need an epoch to get back on track anyways. I mean, I'd flip that around and say "we may as well try fixing it without an Epoch first, since we can always add an Epoch later if nothing else works".

Rows
Columns

bluez package incorrectly versioned, test failures inappropriately waived #1569

What we can do

What we can't do

Things we would probably support

Final notes