F45 System-Wide Change: Build x86-64-v3 Packages #13310

Open
opened 2026-04-10 02:28:31 +00:00 by lleyton · 10 comments

Please review the Change to Build x86-64-v3 Packages.

This will need release engineering assistance to enable and deliver x86_64_v3 repositories and images for x86_64_v3.

Please review the Change to [Build x86-64-v3 Packages](https://fedoraproject.org/wiki/Changes/Build_x86-64-v3_Packages). This will need release engineering assistance to enable and deliver x86_64_v3 repositories and images for x86_64_v3.
Owner

I'm not sure of the scope here. Are you saying every single package would additionally be built for v3? So, there would be v3 repos, v3 deliverable images, etc? Thats a vast amount of duplication, storage, cpu time, images, etc.

Perhaps you might look at https://fedoraproject.org/wiki/Changes/Optimized_Binaries_for_the_AMD64_Architecture_v2 and see if it could be extended for your needs?

I'm not sure of the scope here. Are you saying every single package would additionally be built for v3? So, there would be v3 repos, v3 deliverable images, etc? Thats a vast amount of duplication, storage, cpu time, images, etc. Perhaps you might look at https://fedoraproject.org/wiki/Changes/Optimized_Binaries_for_the_AMD64_Architecture_v2 and see if it could be extended for your needs?
Member

@kevin wrote in #13310 (comment):

I'm not sure of the scope here. Are you saying every single package would additionally be built for v3? So, there would be v3 repos, v3 deliverable images, etc? Thats a vast amount of duplication, storage, cpu time, images, etc.

That does sound like the idea. It wouldn't be the first time we've done it, it's just been quite a while. The duplication is probably worth it though, since it makes everything "cleanly" x86_64-v3.

Perhaps you might look at https://fedoraproject.org/wiki/Changes/Optimized_Binaries_for_the_AMD64_Architecture_v2 and see if it could be extended for your needs?

The optimized binaries change is unworkable on two fronts:

  • It's basically path hacking, which is going to be a security issue
  • We can't selectively enable an architecture in Koji for this

And to be honest, I'd rather we didn't lie about the x86_64 subarches in our packages like RHEL does. So at least on that front, this Change appeals to me because the RPM/DNF stack can do the right thing, and Mock can do things like require x86_64-v3 hosts for EPEL 10 and have it work reliably.

@kevin wrote in https://forge.fedoraproject.org/releng/tickets/issues/13310#issuecomment-615147: > I'm not sure of the scope here. Are you saying every single package would additionally be built for v3? So, there would be v3 repos, v3 deliverable images, etc? Thats a vast amount of duplication, storage, cpu time, images, etc. > That does sound like the idea. It wouldn't be the first time we've done it, it's just been quite a while. The duplication is probably worth it though, since it makes everything "cleanly" x86_64-v3. > Perhaps you might look at https://fedoraproject.org/wiki/Changes/Optimized_Binaries_for_the_AMD64_Architecture_v2 and see if it could be extended for your needs? The optimized binaries change is unworkable on two fronts: * It's basically path hacking, which is going to be a security issue * We can't selectively enable an architecture in Koji for this And to be honest, I'd rather we didn't lie about the x86_64 subarches in our packages like RHEL does. So at least on that front, this Change appeals to me because the RPM/DNF stack can do the right thing, and Mock can do things like require x86_64-v3 hosts for EPEL 10 and have it work reliably.
Member

I see no benchmarks for the upside here. Esp. Interesting would be what happens if we "just" compile all libraries twice and use the existing ld infra. to pick the better version.

I see no kind of data on how many users are on either side of this. Wikipedia says v3 was released ~13 years ago. So the obvious question is how bad would it be to just make x86_64 v3+ for 45?

Seems like there should be a pointer or something for users to see how this affects them (Eg. /lib64/ld-linux-x86-64.so.2 --help | grep -E "v[0-9]" shows you what your CPU supports AIUI).

I don't see any mention of mock/containers/anaconda/ostree/bootc. This all seems like a lot of effort for not much if those are all staying v2 only (mock looks like you are trying to ignore it and use config -- this seems like a bad idea). Are you planning on both v2 and v3 rpms having the same NVRA? This also has to extend to other tools that might assume x86_64 == x86_64 (Eg. anything doing binary hashes might be very confused).

If we need to provide v2 support, why would v2 be the default? (Eg. if everything defaulted to v3, but we provide a legacy x86_64 install+repo ... we could then probably get away with ignoring support in a lot of the above things.

The policy change is cute, but someone needs to own that as part of the change proposal AND ALSO that person needs to audit and work to change/fix existing specfiles that are now broken.

If the v2 hacks are bad what about some kind of fat binaries? And maybe some simple way to use strip to get v3 only? We still get 2x the storage almost everywhere, but we don't have a lot of the other problems.

Also v4 is already out there and coming up to 10 years, do you expect to want to do something similar for that soon? Would that be to support v2, v3, and v4 or just move to only v3 and v4?

I see no benchmarks for the upside here. Esp. Interesting would be what happens if we "just" compile all libraries twice and use the existing ld infra. to pick the better version. I see no kind of data on how many users are on either side of this. Wikipedia says v3 was [released ~13 years ago](https://en.wikipedia.org/wiki/X86-64#Microarchitecture_levels). So the obvious question is how bad would it be to just make x86_64 v3+ for 45? Seems like there should be a pointer or something for users to see how this affects them (Eg. `/lib64/ld-linux-x86-64.so.2 --help | grep -E "v[0-9]"` shows you what your CPU supports AIUI). I don't see any mention of mock/containers/anaconda/ostree/bootc. This all seems like a lot of effort for not much if those are all staying v2 only (mock looks like you are trying to ignore it and use config -- this seems like a bad idea). Are you planning on both v2 and v3 rpms having the same NVRA? This also has to extend to other tools that might assume x86_64 == x86_64 (Eg. anything doing binary hashes might be very confused). If we need to provide v2 support, why would v2 be the default? (Eg. if everything defaulted to v3, but we provide a legacy x86_64 install+repo ... we could then probably get away with ignoring support in a lot of the above things. The policy change is cute, but someone needs to own that as part of the change proposal **AND ALSO** that person needs to audit and work to change/fix existing specfiles that are now broken. If the v2 hacks are bad what about some kind of fat binaries? And maybe some simple way to use strip to get v3 only? We still get 2x the storage almost everywhere, but we don't have a lot of the other problems. Also v4 is already out there and coming up to 10 years, do you expect to want to do something similar for that soon? Would that be to support v2, v3, and v4 or just move to only v3 and v4?
Member

@james wrote in #13310 (comment):

I see no benchmarks for the upside here. Esp. Interesting would be what happens if we "just" compile all libraries twice and use the existing ld infra. to pick the better version.

There are some benchmarks (CachyOS, Ubuntu), though I don't consider them compelling enough to only ship x86_64-v3.

This also puts us in reasonably competitive footing with Arch Linux (who did something similar in 2021) and Ubuntu (who did this with Ubuntu 25.10).

I see no kind of data on how many users are on either side of this. Wikipedia says v3 was released ~13 years ago. So the obvious question is how bad would it be to just make x86_64 v3+ for 45?

It is bad enough that AlmaLinux restored x86_64-v2 as an alternate build. AlmaLinux has already identified all the changes required across the stack to support dual x86_64 subarches. They would just be merged into Fedora so that it works here too.

Seems like there should be a pointer or something for users to see how this affects them (Eg. /lib64/ld-linux-x86-64.so.2 --help | grep -E "v[0-9]" shows you what your CPU supports AIUI).

I don't see any mention of mock/containers/anaconda/ostree/bootc. This all seems like a lot of effort for not much if those are all staying v2 only (mock looks like you are trying to ignore it and use config -- this seems like a bad idea). Are you planning on both v2 and v3 rpms having the same NVRA? This also has to extend to other tools that might assume x86_64 == x86_64 (Eg. anything doing binary hashes might be very confused).

They don't have the same NEVRA. RPM treats each x86_64 subarch as a completely different architecture. There is x86_64 (for v1), x86_64_v2 (for v2), x86_64_v3 (for v3), and x86_64_v4 (for v4). RPM and libsolv also can test and consider compatibility with the host system for the architecture when packages are correctly labeled with the architectures (ie not like RHEL which does v3 as x86_64). This is already used in openSUSE, for example.

If we need to provide v2 support, why would v2 be the default? (Eg. if everything defaulted to v3, but we provide a legacy x86_64 install+repo ... we could then probably get away with ignoring support in a lot of the above things.

The policy change is cute, but someone needs to own that as part of the change proposal AND ALSO that person needs to audit and work to change/fix existing specfiles that are now broken.

If the v2 hacks are bad what about some kind of fat binaries? And maybe some simple way to use strip to get v3 only? We still get 2x the storage almost everywhere, but we don't have a lot of the other problems.

To be honest, I'd prefer FatELF, but who is going to bring that into the toolchain (where the ELF stuff lives)?

Also, Fedora isn't x86_64-v2. It's x86_64-v1.

Also v4 is already out there and coming up to 10 years, do you expect to want to do something similar for that soon? Would that be to support v2, v3, and v4 or just move to only v3 and v4?

As for x86_64-v4, Intel killed it when they dropped AVX512 from their latest CPUs. It now only works on AMD CPUs. At this point, there is no commercial or community viability to x86_64-v4 anymore.

@james wrote in https://forge.fedoraproject.org/releng/tickets/issues/13310#issuecomment-615381: > I see no benchmarks for the upside here. Esp. Interesting would be what happens if we "just" compile all libraries twice and use the existing ld infra. to pick the better version. > There are some benchmarks ([CachyOS](https://sunnyflunk.github.io/2023/01/15/x86-64-v3-Mixed-Bag-of-Performance.html), [Ubuntu](https://www.phoronix.com/review/ubuntu-x86-64-v3-benchmark)), though I don't consider them compelling enough to only ship x86_64-v3. This also puts us in reasonably competitive footing with Arch Linux ([who did something similar in 2021](https://rfc.archlinux.page/0002-x86-64-v3-microarchitecture/)) and Ubuntu ([who did this with Ubuntu 25.10](https://discourse.ubuntu.com/t/introducing-architecture-variants-amd64v3-now-available-in-ubuntu-25-10/71312)). > I see no kind of data on how many users are on either side of this. Wikipedia says v3 was [released ~13 years ago](https://en.wikipedia.org/wiki/X86-64#Microarchitecture_levels). So the obvious question is how bad would it be to just make x86_64 v3+ for 45? > It is bad enough that AlmaLinux restored x86_64-v2 as an alternate build. AlmaLinux has already identified all the changes required across the stack to support dual x86_64 subarches. They would just be merged into Fedora so that it works here too. > Seems like there should be a pointer or something for users to see how this affects them (Eg. `/lib64/ld-linux-x86-64.so.2 --help | grep -E "v[0-9]"` shows you what your CPU supports AIUI). > > I don't see any mention of mock/containers/anaconda/ostree/bootc. This all seems like a lot of effort for not much if those are all staying v2 only (mock looks like you are trying to ignore it and use config -- this seems like a bad idea). Are you planning on both v2 and v3 rpms having the same NVRA? This also has to extend to other tools that might assume x86_64 == x86_64 (Eg. anything doing binary hashes might be very confused). They don't have the same NEVRA. RPM treats each x86_64 subarch as a completely different architecture. There is x86_64 (for v1), x86_64_v2 (for v2), x86_64_v3 (for v3), and x86_64_v4 (for v4). RPM and libsolv also can test and consider compatibility with the host system for the architecture when packages are correctly labeled with the architectures (ie not like RHEL which does v3 as x86_64). This is already used in openSUSE, for example. > If we need to provide v2 support, why would v2 be the default? (Eg. if everything defaulted to v3, but we provide a legacy x86_64 install+repo ... we could then probably get away with ignoring support in a lot of the above things. > > The policy change is cute, but someone needs to own that as part of the change proposal **AND ALSO** that person needs to audit and work to change/fix existing specfiles that are now broken. > > If the v2 hacks are bad what about some kind of fat binaries? And maybe some simple way to use strip to get v3 only? We still get 2x the storage almost everywhere, but we don't have a lot of the other problems. To be honest, I'd prefer [FatELF](https://icculus.org/fatelf/), but who is going to bring that into the toolchain (where the ELF stuff lives)? Also, Fedora isn't x86_64-v2. It's x86_64-v1. > Also v4 is already out there and coming up to 10 years, do you expect to want to do something similar for that soon? Would that be to support v2, v3, and v4 or just move to only v3 and v4? As for x86_64-v4, Intel killed it when they dropped AVX512 from their latest CPUs. It now only works on AMD CPUs. At this point, there is no commercial or community viability to x86_64-v4 anymore.
Member

I don't see any mention of mock/containers/anaconda/ostree/bootc.

Mock was fixed by AlmaLinux (it'll be in Mock 6.8). Podman supports x86_64 variants with Debian-style syntax: --platform linux/amd64/vX. The bootc stuff inherits that. Anaconda will handle whatever DNF tells it to do. RPM-OSTree is an interesting open question, though.

> I don't see any mention of mock/containers/anaconda/ostree/bootc. Mock was fixed by AlmaLinux (it'll be in Mock 6.8). Podman supports x86_64 variants with Debian-style syntax: `--platform linux/amd64/vX`. The bootc stuff inherits that. Anaconda will handle whatever DNF tells it to do. RPM-OSTree is an interesting open question, though.
Member

Are you planning on both v2 and v3 rpms having the same NVRA?

To be clear here, after drinking some more tea ... we used to have i686 vs. i386, which mapped to arch and basearch in rpm/etc. terminology. From what I can see you want to do the same thing but it'll be x86_64 as basearch and x86_64_v3 as arch ... so rpm filenames (or rpm -q etc) will now show bash-5.3.0-2.fc43.x86_64_v3.rpm etc.

Following this previous path implies a few other things for how this can move forwards. Eg. in Fedora-10 we mostly compiled everything as i386 but for glibc/kernel/openssl we also had i686 packages ... and then by Fedora-11 the common arch was i586, and by Fedora-12 i686.

NOTE: This still has all the problems for containers/etc. where we aren't just shipping packages anymore.

> Are you planning on both v2 and v3 rpms having the same NVRA? To be clear here, after drinking some more tea ... we used to have i686 vs. i386, which mapped to arch and basearch in rpm/etc. terminology. From what I can see you want to do the same thing but it'll be x86_64 as basearch and x86_64_v3 as arch ... so rpm filenames (or `rpm -q` etc) will now show bash-5.3.0-2.fc43.x86_64_v3.rpm etc. Following this previous path implies a few other things for how this can move forwards. Eg. in Fedora-10 we mostly compiled everything as i386 but for glibc/kernel/openssl we also had i686 packages ... and then by Fedora-11 the common arch was i586, and by Fedora-12 i686. **NOTE:** This still has all the problems for containers/etc. where we aren't just shipping packages anymore.
Member

Containers support x86_64 arch variants (at least with the Red Hat container tools). I'm not sure about Docker, but Podman, Skopeo, etc. all do.

Containers support x86_64 arch variants (at least with the Red Hat container tools). I'm not sure about Docker, but Podman, Skopeo, etc. all do.
Owner

Hi. Can you all please take this discussion to the discussion or mailing list threads on the change?

This ticket is for the releng impact. There could be a lot, we don't know how much exactly yet.

Higher level discussion about the change shouldn't be here.

Hi. Can you all please take this discussion to the discussion or mailing list threads on the change? This ticket is for the releng impact. There could be a lot, we don't know how much exactly yet. Higher level discussion about the change shouldn't be here.
Member

@ngompa wrote in #13310 (comment):

@james wrote in #13310 (comment):

I see no benchmarks for the upside here. Esp. Interesting would be what happens if we "just" compile all libraries twice and use the existing ld infra. to pick the better version.

There are some benchmarks (CachyOS

Yeh, those numbers are underwhelming. Intentionally ignoring anything phoronix related.

This also puts us in reasonably competitive footing with Arch Linux

You mean PR feature competitive, right?

not like RHEL which does v3 as x86_64

I mean ... this doesn't seem like the worst model to follow, and us changing this makes it a bit confusing for rhel in the future.

To be honest, I'd prefer FatELF, but who is going to bring that into the toolchain (where the ELF stuff lives)?

Yeh, I assume it'll be a bunch of work but it only needs to be done in the compiler/linker and then everything just works after (and it's compatible with rhel). Although fat binaries will be fat ;).

Also, Fedora isn't x86_64-v2. It's x86_64-v1.

Yeh, thanks for the correction. It wasn't super obvious to me as I was typing, I should have double checked and reviewed what I'd wrote.

Also correct me if I'm wrong but the CachyOS benchmarks are v3 vs. v1, right?

Also v4 is already out there and coming up to 10 years, do you expect to want to do something similar for that soon? Would that be to support v2, v3, and v4 or just move to only v3 and v4?

As for x86_64-v4, Intel killed it when they dropped AVX512 from their latest CPUs. It now only works on AMD CPUs. At this point, there is no commercial or community viability to x86_64-v4 anymore.

This very much feels like a negative reason to try doing it the subarch way. We are creating a bunch of changes/problems so that we'll have v1 and v3 for a limited time, before going full v3, and maybe v5 years from now?

It's been a while since I worked deeply with rpm, and I've probably tried to block a bunch of things from my mind (including subarch) ... but I'm pretty sure everyone disliked it then, and we were very much package focused at that point.

@ngompa wrote in https://forge.fedoraproject.org/releng/tickets/issues/13310#issuecomment-615399: > @james wrote in #13310 (comment): > > > I see no benchmarks for the upside here. Esp. Interesting would be what happens if we "just" compile all libraries twice and use the existing ld infra. to pick the better version. > > There are some benchmarks ([CachyOS](https://sunnyflunk.github.io/2023/01/15/x86-64-v3-Mixed-Bag-of-Performance.html) Yeh, those numbers are underwhelming. Intentionally ignoring anything phoronix related. > This also puts us in reasonably competitive footing with Arch Linux You mean PR feature competitive, right? > not like RHEL which does v3 as x86_64 I mean ... this doesn't seem like the worst model to follow, and us changing this makes it a bit confusing for rhel in the future. > To be honest, I'd prefer [FatELF](https://icculus.org/fatelf/), but who is going to bring that into the toolchain (where the ELF stuff lives)? Yeh, I assume it'll be a bunch of work but it only needs to be done in the compiler/linker and then everything just works after (and it's compatible with rhel). Although fat binaries will be fat ;). > Also, Fedora isn't x86_64-v2. It's x86_64-v1. Yeh, thanks for the correction. It wasn't super obvious to me as I was typing, I should have double checked and reviewed what I'd wrote. Also correct me if I'm wrong but the CachyOS benchmarks are v3 vs. v1, right? > > Also v4 is already out there and coming up to 10 years, do you expect to want to do something similar for that soon? Would that be to support v2, v3, and v4 or just move to only v3 and v4? > > As for x86_64-v4, Intel killed it when they dropped AVX512 from their latest CPUs. It now only works on AMD CPUs. At this point, there is no commercial or community viability to x86_64-v4 anymore. This very much feels like a negative reason to try doing it the subarch way. We are creating a bunch of changes/problems so that we'll have v1 and v3 for a limited time, before going full v3, and maybe v5 years from now? It's been a while since I worked deeply with rpm, and I've probably tried to block a bunch of things from my mind (including subarch) ... but I'm pretty sure everyone disliked it then, and we were very much package focused at that point.
Member

@kevin wrote in #13310 (comment):

Hi. Can you all please take this discussion to the discussion or mailing list threads on the change?

Yeh, my bad ... I'll try to reply on discourse today with a synopsis of my open questions and some of the things neal's answered.

@kevin wrote in https://forge.fedoraproject.org/releng/tickets/issues/13310#issuecomment-615409: > Hi. Can you all please take this discussion to the discussion or mailing list threads on the change? Yeh, my bad ... I'll try to reply on discourse today with a synopsis of my open questions and some of the things neal's answered.
Sign in to join this conversation.
No milestone
No project
No assignees
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
releng/tickets#13310
No description provided.