Side by side minimal installs have identical UUIDs

pat42smith commented

2024-06-02 03:54:10 +00:00

(This was originally posted as an issue against the installer: https://github.com/AsahiLinux/asahi-installer/issues/265 . Reposting here at the request of marcan.)

I tried to install two copies of Fedora Asahi 39 on a single M2 Mac Mini, both minimal installs. This led to various bad outcomes, such as booting in to one of them showing the other's root filesystem mounted at /. And the boot process not completing, but leaving me at a prompt "Enter root password for maintenance", and after rebooting, "Press enter for maintenance" (That's not the exact wording of the two messages, but captures the gist; I'm writing just from memory.)

On examination, I found the two installs were using the same set of 3 filesystem UUIDs for /, /boot, and /boot/efi. I am guessing that this is the cause of the above problems. Also, I would guess the UUIDs are hard coded in the filesystem images used by the installer, and that similar problems would occur whenever doing side by side installs of the same type.

On the other hand, I have tried doing a minimal install beside a KDE install. And on another occasion, a minimal install beside a server install. Both cases seemed to be fine. When I checked the UUIDs in the minimal+server case, they were all different. I imagine this would true with any pair of different type installs, but haven't tested any further.

How bad could the consequences of this be? I don't know, but I'd worry about the possibility that someone might have had an install for some time, then decide to install another copy beside it, using the same type, and wind up losing all their data from the original install.

What could be done about this? I suppose the ideal would be to randomize the UUIDs on each install. Failing that, perhaps the installer could display a warning?

(This was originally posted as an issue against the installer: https://github.com/AsahiLinux/asahi-installer/issues/265 . Reposting here at the request of marcan.) I tried to install two copies of Fedora Asahi 39 on a single M2 Mac Mini, both minimal installs. This led to various bad outcomes, such as booting in to one of them showing the other's root filesystem mounted at /. And the boot process not completing, but leaving me at a prompt "Enter root password for maintenance", and after rebooting, "Press enter for maintenance" (That's not the exact wording of the two messages, but captures the gist; I'm writing just from memory.) On examination, I found the two installs were using the same set of 3 filesystem UUIDs for /, /boot, and /boot/efi. I am guessing that this is the cause of the above problems. Also, I would guess the UUIDs are hard coded in the filesystem images used by the installer, and that similar problems would occur whenever doing side by side installs of the same type. On the other hand, I have tried doing a minimal install beside a KDE install. And on another occasion, a minimal install beside a server install. Both cases seemed to be fine. When I checked the UUIDs in the minimal+server case, they were all different. I imagine this would true with any pair of different type installs, but haven't tested any further. How bad could the consequences of this be? I don't know, but I'd worry about the possibility that someone might have had an install for some time, then decide to install another copy beside it, using the same type, and wind up losing all their data from the original install. What could be done about this? I suppose the ideal would be to randomize the UUIDs on each install. Failing that, perhaps the installer could display a warning?

pat42smith commented

2024-06-02 04:25:59 +00:00

Author

Conan-Kudo writes at https://github.com/AsahiLinux/asahi-installer/issues/265#issuecomment-2143417487:

It never got implemented because I don't think either @davide125 or I knew that was a thing we should do.

As far as I know, btrfstune -u can do it, but only for an offline btrfs filesystem. Maybe it's possible to change it so it's allowed for online filesystems?

("it" referring to randomizing the UUIDs at first boot.)

Also tune2fs -U for /boot and mlabel -N for /boot/efi. But with those, maybe it's practical to just unmount them, do the change, and remount them.

Maybe the first boot could use a special initramfs that changes the root filesystem UUID before mounting it? Seems like a fair bit of work, though. :-(

Conan-Kudo writes at https://github.com/AsahiLinux/asahi-installer/issues/265#issuecomment-2143417487: > It never got implemented because I don't think either @davide125 or I knew that was a thing we should do. > As far as I know, btrfstune -u <deviceid> can do it, but only for an offline btrfs filesystem. Maybe it's possible to change it so it's allowed for online filesystems? ("it" referring to randomizing the UUIDs at first boot.) Also tune2fs -U for /boot and mlabel -N for /boot/efi. But with those, maybe it's practical to just unmount them, do the change, and remount them. Maybe the first boot could use a special initramfs that changes the root filesystem UUID before mounting it? Seems like a fair bit of work, though. :-(

pat42smith commented

2024-06-02 04:33:45 +00:00

Author

Marcan writes at https://github.com/AsahiLinux/asahi-installer/issues/265#issuecomment-2143339648:

Also note that different versions/builds also have different UUIDs, so installing e.g. F40 and F39 side by side should work, as well as installing with enough time passing in between that we pushed new image builds. You can also grab nightly image builds from https://fedora-asahi-remix.org, each of which has a different UUID.

That's nice to know, in the sense that this problem is a bit less likely to occur, and there are more workarounds. But it still leaves the side by side install as a case that sometimes works and sometimes fails in ways likely to baffle a naive user.

Marcan writes at https://github.com/AsahiLinux/asahi-installer/issues/265#issuecomment-2143339648: > Also note that different versions/builds also have different UUIDs, so installing e.g. F40 and F39 side by side should work, as well as installing with enough time passing in between that we pushed new image builds. You can also grab nightly image builds from https://fedora-asahi-remix.org, each of which has a different UUID. That's nice to know, in the sense that this problem is a bit less likely to occur, and there are more workarounds. But it still leaves the side by side install as a case that sometimes works and sometimes fails in ways likely to baffle a naive user.

ngompa commented

2024-06-02 05:13:18 +00:00

Owner

I think we're going to need to track getting this limitation removed from btrfs-progs and come up with a workaround. It's probably not acceptable to not have a way to auto-rotate and fixup UUIDs on first boot.

I think we're going to need to track getting this limitation removed from `btrfs-progs` and come up with a workaround. It's probably not acceptable to not have a way to auto-rotate and fixup UUIDs on first boot.

marcan commented

2024-06-02 05:37:22 +00:00

I suspect that limitation might involve the kernel (i.e. no way to tell the kernel the UUID changed for whatever purposes it uses it for while mounted) so I wouldn't hold my breath that that will be that easy to fix.

I think, if we play it right, we can probably build this functionality into the asahi-scripts dracut stuff, in a vaguely generic way that might work for other distros (opt-in though, so nobody gets caught enabling it without testing). We can do it with the filesystems unmounted there, without relying on anyone else. It's going to be finicky and subtle, but doable I think.

I suspect that limitation might involve the kernel (i.e. no way to tell the kernel the UUID changed for whatever purposes it uses it for while mounted) so I wouldn't hold my breath that that will be that easy to fix. I think, if we play it right, we can probably build this functionality into the asahi-scripts dracut stuff, in a vaguely generic way that might work for other distros (opt-in though, so nobody gets caught enabling it without testing). We can do it with the filesystems unmounted there, without relying on anyone else. It's going to be finicky and subtle, but doable I think.

pat42smith commented

2024-06-02 05:42:58 +00:00

Author

Marcan writes at https://github.com/AsahiLinux/asahi-installer/issues/265#issuecomment-2143339648:

This is a known regression for the Fedora images. On Arch we used to randomize the UUIDs on first boot, but that never got implemented for Fedora (maybe it was because it was hard to do online with btrfs?). It is not practical to randomize the UUIDs at installation time, since they need to be changed in bootloader scripts etc.

I'm guessing there are two issues here?

Not wanting the installer to know the places that need to be changed.
Difficulty in working with ext4 and btrfs under MacOS.

Perhaps instead the installer could just do a global search and replace to change the common UUIDs to unique ones? This could be done to the filesystem images before or during the copy to the partitions. Or to the partitions themselves, after the copy. Both binary and text forms of the UUIDs might have to be replaced.

Since the /boot/efi serial number ("UUID") is just 32 bits, it might be worthwhile when generating the image to check that the serial number doesn't occur anywhere it shouldn't (e.g by counting the number of occurrences).

Marcan writes at https://github.com/AsahiLinux/asahi-installer/issues/265#issuecomment-2143339648: > This is a known regression for the Fedora images. On Arch we used to randomize the UUIDs on first boot, but that never got implemented for Fedora (maybe it was because it was hard to do online with btrfs?). It is not practical to randomize the UUIDs at installation time, since they need to be changed in bootloader scripts etc. I'm guessing there are two issues here? - Not wanting the installer to know the places that need to be changed. - Difficulty in working with ext4 and btrfs under MacOS. Perhaps instead the installer could just do a global search and replace to change the common UUIDs to unique ones? This could be done to the filesystem images before or during the copy to the partitions. Or to the partitions themselves, after the copy. Both binary and text forms of the UUIDs might have to be replaced. Since the /boot/efi serial number ("UUID") is just 32 bits, it might be worthwhile when generating the image to check that the serial number doesn't occur anywhere it shouldn't (e.g by counting the number of occurrences).

ngompa commented

2024-06-02 05:53:39 +00:00

Owner

@marcan:

I suspect that limitation might involve the kernel (i.e. no way to tell the kernel the UUID changed for whatever purposes it uses it for while mounted) so I wouldn't hold my breath that that will be that easy to fix.

I think, if we play it right, we can probably build this functionality into the asahi-scripts dracut stuff, in a vaguely generic way that might work for other distros (opt-in though, so nobody gets caught enabling it without testing). We can do it with the filesystems unmounted there, without relying on anyone else. It's going to be finicky and subtle, but doable I think.

Yeah, a dracut module is probably the way to go, and we can structurally handle the ESP, the boot partition, and the Btrfs volume all in one go there.

@marcan: > I suspect that limitation might involve the kernel (i.e. no way to tell the kernel the UUID changed for whatever purposes it uses it for while mounted) so I wouldn't hold my breath that that will be that easy to fix. > > I think, if we play it right, we can probably build this functionality into the asahi-scripts dracut stuff, in a vaguely generic way that might work for other distros (opt-in though, so nobody gets caught enabling it without testing). We can do it with the filesystems unmounted there, without relying on anyone else. It's going to be finicky and subtle, but doable I think. Yeah, a dracut module is probably the way to go, and we can structurally handle the ESP, the boot partition, and the Btrfs volume all in one go there.

marcan commented

2024-06-02 07:21:42 +00:00

Perhaps instead the installer could just do a global search and replace to change the common UUIDs to unique ones? This could be done to the filesystem images before or during the copy to the partitions. Or to the partitions themselves, after the copy. Both binary and text forms of the UUIDs might have to be replaced.

This isn't feasible, besides being slow there are issues with checksums, compression, etc. You need to do it from a context with real FS drivers (hence dracut). Forget about hacks like this, they aren't going to work and are extremely ugly and brittle even if they do.

> Perhaps instead the installer could just do a global search and replace to change the common UUIDs to unique ones? This could be done to the filesystem images before or during the copy to the partitions. Or to the partitions themselves, after the copy. Both binary and text forms of the UUIDs might have to be replaced. This isn't feasible, besides being slow there are issues with checksums, compression, etc. You need to do it from a context with real FS drivers (hence dracut). Forget about hacks like this, they aren't going to work and are extremely ugly and brittle even if they do.

dcavalca commented

2024-08-07 13:36:23 +00:00

Owner

After talking to upstream, supporting online updates of the uuid is technically possible, but it's non-trivial to implement and would be operationally expensive, as the uuid is encoded in all of the metadata. For the time being we're probably better off going the dracut module route for this.

Rows
Columns

Side by side minimal installs have identical UUIDs #9