root mount options from /etc/fstab are not taken into account anymore with composefs (symptom: `systemd-remount-fs.service` fails to start)

Siosm commented

2025-03-17 17:09:32 +00:00

(Migrated from gitlab.com)

Symptom

The systemd-remount-fs.service service fails to start:

Apr 11 11:25:25 fedora systemd-remount-fs[837]: mount: /: fsconfig system call failed: overlay: No changes allowed in reconfigure.
Apr 11 11:25:25 fedora systemd-remount-fs[837]:        dmesg(1) may have more information after failed mount system call.
Apr 11 11:25:25 fedora systemd[1]: Starting systemd-remount-fs.service - Remount Root and Kernel File Systems...
Apr 11 11:25:25 fedora systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE
Apr 11 11:25:25 fedora systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'.
Apr 11 11:25:25 fedora systemd[1]: Failed to start systemd-remount-fs.service - Remount Root and Kernel File Systems.

Workaround

If you want to keep using or enable on-demand zstd compression for your main btrfs partition, update your kernel arguments to match what's in your /etc/fstab:

$ grep 'root' /etc/fstab
# UUID=... / btrfs subvol=root,compress=zstd:1,ro 0 0

$ cat /proc/cmdline
... rootflags=subvol=root ...

$ sudo rpm-ostree kargs --delete=rootflags=subvol=root -- 
append=rootflags=subvol=root,compress=zstd:1

Then comment the entry for / in your /etc/fstab. Make sure that any future changes to the root mount point options are added to the kernel command line.
Reboot for the changes to apply.

Original issue

With composefs in F42 (https://gitlab.com/fedora/ostree/sig/-/issues/35), the mount options set for the root filesystem in /etc/fstab are not taken into account anymore as / is now an overlay mount point.

One notable option on default Atomic Desktops installation in zstd compression for btrfs: https://fedoraproject.org/wiki/Changes/BtrfsTransparentCompression

silverblue@fedora:~$ cat /etc/fstab 
# /etc/fstab
# Created by anaconda on Wed Feb 12 13:48:11 2025
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
# Updated by bootc-fstab-edit.service
UUID=15200db3-8480-4c87-93de-7127a3cb1315 / btrfs subvol=root,compress=zstd:1,ro 0 0
UUID=ff7188f3-e5e5-44c5-8763-93b7ef13a27e /boot                   ext4    defaults        1 2
UUID=523B-489E          /boot/efi               vfat    umask=0077,shortname=winnt 0 2
UUID=15200db3-8480-4c87-93de-7127a3cb1315 /home                   btrfs   subvol=home,compress=zstd:1 0 0
UUID=15200db3-8480-4c87-93de-7127a3cb1315 /var                    btrfs   subvol=var,compress=zstd:1 0 0

silverblue@fedora:~$ sudo journalctl -u bootc-fstab-edit.service
Feb 13 14:14:50 fedora bootc[702]: Updated /etc/fstab to add `ro` for `/`

Options:

Mask systemd-remount-fs.service and rely on kernel command line arguments only.
- We would have to create something that does a one time sync of /etc/fstab options to the kernel command line to migrate existing systems.
- New systems need to have those options added as kargs by Anaconda.
- This is not ideal as all custom /proc, /sys, etc. custom mount options would also be ignored.
Teach systemd-remount-fs.service to properly remount /sysroot instead of /.

Refrences & Bugzillas

See:

## Symptom The `systemd-remount-fs.service` service fails to start: ``` Apr 11 11:25:25 fedora systemd-remount-fs[837]: mount: /: fsconfig system call failed: overlay: No changes allowed in reconfigure. Apr 11 11:25:25 fedora systemd-remount-fs[837]: dmesg(1) may have more information after failed mount system call. Apr 11 11:25:25 fedora systemd[1]: Starting systemd-remount-fs.service - Remount Root and Kernel File Systems... Apr 11 11:25:25 fedora systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE Apr 11 11:25:25 fedora systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'. Apr 11 11:25:25 fedora systemd[1]: Failed to start systemd-remount-fs.service - Remount Root and Kernel File Systems. ``` ## Workaround * If you want to keep using or enable on-demand zstd compression for your main btrfs partition, update your kernel arguments to match what's in your `/etc/fstab`: ``` $ grep 'root' /etc/fstab # UUID=... / btrfs subvol=root,compress=zstd:1,ro 0 0 $ cat /proc/cmdline ... rootflags=subvol=root ... $ sudo rpm-ostree kargs --delete=rootflags=subvol=root -- append=rootflags=subvol=root,compress=zstd:1 ``` * Then comment the entry for `/` in your `/etc/fstab`. Make sure that any future changes to the root mount point options are added to the kernel command line. * Reboot for the changes to apply. ## Original issue With composefs in F42 (https://gitlab.com/fedora/ostree/sig/-/issues/35), the mount options set for the root filesystem in /etc/fstab are not taken into account anymore as `/` is now an overlay mount point. One notable option on default Atomic Desktops installation in zstd compression for btrfs: https://fedoraproject.org/wiki/Changes/BtrfsTransparentCompression ``` silverblue@fedora:~$ cat /etc/fstab # /etc/fstab # Created by anaconda on Wed Feb 12 13:48:11 2025 # # Accessible filesystems, by reference, are maintained under '/dev/disk/'. # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info. # # After editing this file, run 'systemctl daemon-reload' to update systemd # units generated from this file. # # Updated by bootc-fstab-edit.service UUID=15200db3-8480-4c87-93de-7127a3cb1315 / btrfs subvol=root,compress=zstd:1,ro 0 0 UUID=ff7188f3-e5e5-44c5-8763-93b7ef13a27e /boot ext4 defaults 1 2 UUID=523B-489E /boot/efi vfat umask=0077,shortname=winnt 0 2 UUID=15200db3-8480-4c87-93de-7127a3cb1315 /home btrfs subvol=home,compress=zstd:1 0 0 UUID=15200db3-8480-4c87-93de-7127a3cb1315 /var btrfs subvol=var,compress=zstd:1 0 0 silverblue@fedora:~$ sudo journalctl -u bootc-fstab-edit.service Feb 13 14:14:50 fedora bootc[702]: Updated /etc/fstab to add `ro` for `/` ``` Options: - Mask [systemd-remount-fs.service](https://www.freedesktop.org/software/systemd/man/latest/systemd-remount-fs.service.html) and rely on kernel command line arguments only. - We would have to create something that does a one time sync of `/etc/fstab` options to the kernel command line to migrate existing systems. - New systems need to have those options added as kargs by Anaconda. - This is not ideal as all custom `/proc`, `/sys`, etc. custom mount options would also be ignored. - Teach systemd-remount-fs.service to properly remount /sysroot instead of `/`. ## Refrences & Bugzillas See: - https://github.com/bootc-dev/bootc/issues/971 - https://bugzilla.redhat.com/show_bug.cgi?id=2348934 - https://bugzilla.redhat.com/show_bug.cgi?id=2332319 - https://github.com/fedora-iot/iot-distro/issues/81

👍 1

Siosm commented

2025-03-17 17:12:40 +00:00

(Migrated from gitlab.com)

changed the description

walters2 commented

2025-03-18 00:45:34 +00:00

(Migrated from gitlab.com)

We would have to create something that does a one time sync of /etc/fstab options to the kernel command line to migrate existing systems.

I was actually thinking about doing this via https://github.com/bootc-dev/bootc/pull/1157

> We would have to create something that does a one time sync of `/etc/fstab` options to the kernel command line to migrate existing systems. I was actually thinking about doing this via https://github.com/bootc-dev/bootc/pull/1157

Siosm commented

2025-03-18 18:54:47 +00:00

(Migrated from gitlab.com)

Did a first try at a systemd patch in https://github.com/systemd/systemd/compare/main...travier:systemd:remount-fs-sysroot, but from testing so far, this did not work (i.e. this did not re-mount /sysroot with zstd compression on my Silverblue 42 installation). Not sure why, maybe because there are multiple mount points already? But this was apparently working before composefs, so I don't know where the issue is.

Siosm commented

2025-03-24 18:09:50 +00:00

(Migrated from gitlab.com)

changed title from root mount options from /etc/fstab are not taken into account anymore with composefs to root mount options from /etc/fstab are not taken into account anymore with composefs{+ (symptom: systemd-remount-fs.service fails to start)+}

changed title from **root mount options from /etc/fstab are not taken into account anymore with composefs** to **root mount options from /etc/fstab are not taken into account anymore with composefs{+ (symptom: `systemd-remount-fs.service` fails to start)+}**

Siosm commented

2025-04-11 09:12:37 +00:00

(Migrated from gitlab.com)

changed the description

Siosm commented

2025-04-11 09:13:34 +00:00

(Migrated from gitlab.com)

changed the description

Siosm commented

2025-04-11 09:28:14 +00:00

(Migrated from gitlab.com)

changed the description

Siosm commented

2025-04-11 09:28:30 +00:00

(Migrated from gitlab.com)

changed the description

Siosm commented

2025-04-11 09:31:27 +00:00

(Migrated from gitlab.com)

changed the description

Siosm commented

2025-04-11 09:32:01 +00:00

(Migrated from gitlab.com)

changed the description

Siosm commented

2025-04-11 09:38:01 +00:00

(Migrated from gitlab.com)

changed the description

Siosm commented

2025-04-16 16:01:46 +00:00

(Migrated from gitlab.com)

See the discussion in:

See the discussion in: - https://discussion.fedoraproject.org/t/root-mount-options-are-ignored-in-fedora-atomic-desktops-42/148562 - https://discussion.fedoraproject.org/t/talk-mount-options-are-ignored-in-fedora-atomic-desktops-42/148874

Siosm commented

2025-04-24 16:11:51 +00:00

(Migrated from gitlab.com)

mentioned in issue #35

hrismarin commented

2025-08-25 14:28:11 +00:00

(Migrated from gitlab.com)

This is my simple workaround that I use in my container builds for Fedora Atomic Desktops. Maybe it doesn't make much sense, though.

Copy the following file to /usr/libexec/.

$ cat rm-mount-for-root-from-etc-fstab.bash 

#!126}' | cut -d '=' -f 2,3)"
    if [[ $(grep -E '^'"${root_UUID}"'[[:space:]]/[[:space:]]' /etc/fstab) && \
    $(findmnt / | grep composefs) && ! -L /boot/grub2/grub.cfg ]]; then
        sed --in-place="-BACKUP" '/^'"${root_UUID}"'[[:space:]]\/[[:space:]]/d' /etc/fstab
        touch "${stamp_file}"
    else
        exit 0
    fi
}

main "${@}"

Copy the following file to /usr/lib/systemd/system/.

$ cat rm-mount-for-root-from-etc-fstab.service 

[Unit]
Description=Remove mount options for / (root) from /etc/fstab
ConditionPathExists=!/var/lib/.rm_mount_for_root

[Service]
Type=oneshot
ExecStart=/usr/libexec/rm-mount-for-root-from-etc-fstab.bash
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

This is my simple workaround that I use in my container builds for Fedora Atomic Desktops. Maybe it doesn't make much sense, though. Copy the following file to `/usr/libexec/`. ``` $ cat rm-mount-for-root-from-etc-fstab.bash #!126}' | cut -d '=' -f 2,3)" if [[ $(grep -E '^'"${root_UUID}"'[[:space:]]/[[:space:]]' /etc/fstab) && \ $(findmnt / | grep composefs) && ! -L /boot/grub2/grub.cfg ]]; then sed --in-place="-BACKUP" '/^'"${root_UUID}"'[[:space:]]\/[[:space:]]/d' /etc/fstab touch "${stamp_file}" else exit 0 fi } main "${@}" ``` Copy the following file to `/usr/lib/systemd/system/`. ``` $ cat rm-mount-for-root-from-etc-fstab.service [Unit] Description=Remove mount options for / (root) from /etc/fstab ConditionPathExists=!/var/lib/.rm_mount_for_root [Service] Type=oneshot ExecStart=/usr/libexec/rm-mount-for-root-from-etc-fstab.bash RemainAfterExit=yes [Install] WantedBy=multi-user.target ```

documentvineyard commented

2026-03-15 06:46:26 +00:00

(Migrated from gitlab.com)

i do not expect an answer.

why was this change EVEN made? it broke a working project. and now, a year later, there is no fix, the whateverfs is not rolled back, btrfs is not replaced with something that works, there is nobody assigned.

i have tried the workaround and it broke my system.

so do not try the /fix/ unless you want to spend the next day recovering from backups.

i do not expect an answer. why was this change EVEN made? it broke a working project. and now, a year later, there is no fix, the whateverfs is not rolled back, btrfs is not replaced with something that works, there is nobody assigned. i have tried the workaround and it broke my system. so do not try the /fix/ unless you want to spend the next day recovering from backups.

Siosm commented

2026-03-17 11:02:45 +00:00

(Migrated from gitlab.com)

The change that triggered this bug is https://fedoraproject.org/wiki/Changes/ComposefsAtomicDesktops.

This issue should not break your system. In the most common case, it will only disables Btrfs compression for new files.

Please tell us what commands you tried as a workaround.

The change that triggered this bug is https://fedoraproject.org/wiki/Changes/ComposefsAtomicDesktops. This issue should not break your system. In the most common case, it will only disables Btrfs compression for new files. Please tell us what commands you tried as a workaround.

siosm commented

2026-03-27 10:35:21 +00:00

Owner

So I think the path forward for this issue is:

For new systems: Teach Anaconda to add those options as kargs
For existing systems: Add code in bootc or rpm-ostree to "convert" root kargs from /etc/fstab into real kargs

So I think the path forward for this issue is: - For new systems: Teach Anaconda to add those options as kargs - For existing systems: Add code in bootc or rpm-ostree to "convert" root kargs from `/etc/fstab` into real kargs

siosm pinned this

2026-04-21 10:54:16 +00:00

Rows
Columns

root mount options from /etc/fstab are not taken into account anymore with composefs (symptom: systemd-remount-fs.service fails to start) #72

Symptom

Workaround

Original issue

Refrences & Bugzillas

root mount options from /etc/fstab are not taken into account anymore with composefs (symptom: `systemd-remount-fs.service` fails to start) #72