Gather logs from stuck boot

kparal commented

2025-08-21 09:26:13 +00:00

Owner

Thinking about the cases where gdm doesn't start at all, like here:
https://openqa.fedoraproject.org/tests/3643039

I noticed that we basically have no gathered logs in there. Why? Could openqa try to switch to tty to gather logs?

To cover cases where there's no user to log in as (e.g. stuck boot after fresh install), we could just always add systemd.debug-shell to the kernel cmdline, and then we would have a root shell permanently available on TTY9. This way, we could always gather system logs, regardless of the system state.

Thoughts?

Thinking about the cases where gdm doesn't start at all, like here: https://openqa.fedoraproject.org/tests/3643039 I noticed that we basically have no gathered logs in there. Why? Could openqa try to switch to tty to gather logs? To cover cases where there's no user to log in as (e.g. stuck boot after fresh install), we could just always add `systemd.debug-shell` to the kernel cmdline, and then we would have a root shell permanently available on TTY9. This way, we could always gather system logs, regardless of the system state. Thoughts?

adamwill commented

2025-08-21 16:44:12 +00:00

Owner

I noticed that we basically have no gathered logs in there. Why? Could openqa try to switch to tty to gather logs?

It does. If we don't get logs, it means switching to tty didn't work.

The code is in https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/main/f/lib/installedtest.pm#_35 . It has special handling for dracut emergency mode, otherwise it switches to tty6 and tries to upload logs (if networking works and tar is available) or dump them out on the serial console (if not).

In this case, switching to tty6 doesn't work, the system is hung too hard for some reason. The second red thumbnail is openQA checking for either a logged-in console or a login prompt, and finding neither. You can see the record of this in https://openqa.fedoraproject.org/tests/3643039/logfile?filename=autoinst-log.txt , from lines 1326 (where the initial failure happens) onwards - we see select_console(testapi_console="tty6-console") to try and get to tty6, then we try and match the login prompt or a logged-in console, but fail.

A human in this case would likely reboot to runlevel 3 to get logs. We could try and enhance post_fail_hook to do that, I guess, but it might get a bit complicated.

> I noticed that we basically have no gathered logs in there. Why? Could openqa try to switch to tty to gather logs? It does. If we don't get logs, it means switching to tty didn't work. The code is in https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/main/f/lib/installedtest.pm#_35 . It has special handling for dracut emergency mode, otherwise it switches to tty6 and tries to upload logs (if networking works and tar is available) or dump them out on the serial console (if not). In this case, switching to tty6 doesn't work, the system is hung too hard for some reason. The *second* red thumbnail is openQA checking for either a logged-in console or a login prompt, and finding neither. You can see the record of this in https://openqa.fedoraproject.org/tests/3643039/logfile?filename=autoinst-log.txt , from lines 1326 (where the initial failure happens) onwards - we see `select_console(testapi_console="tty6-console")` to try and get to tty6, then we try and match the login prompt or a logged-in console, but fail. A human in this case would likely reboot to runlevel 3 to get logs. We *could* try and enhance post_fail_hook to do that, I guess, but it might get a bit complicated.

kparal commented

2025-09-02 13:39:19 +00:00

Author

Owner

Ok, if there's an automated step to try to gather logs from TTY6, that's great. In this particular case, TTY switching really doesn't work, I confirmed manually.

I wonder what OpenQA does in cases when no system user is available (after Workstation install but before gnome-initial-setup)? In that case, the systemd.debug-shell approach might be useful.

Ok, if there's an automated step to try to gather logs from TTY6, that's great. In this particular case, TTY switching really doesn't work, I confirmed manually. I wonder what OpenQA does in cases when no system user is available (after Workstation install but before gnome-initial-setup)? In that case, the `systemd.debug-shell` approach might be useful.

adamwill commented

2025-09-03 17:10:01 +00:00

Owner

it doesn't use the system user, it uses root. We create the root account on almost all test paths.

There are definitely occasional cases where the current approach doesn't work, but addressing them is complex. "just always add systemd.debug-shell to the kernel cmdline" has several problems: it's actually quite hard to reliably add kernel args such that they will be present in every test. In some situations the bootloader timeout is very short so we can't reliably do it at boot time. We can inject args after install for tests which involve an anaconda install, but some tests don't.

And beyond the practical problems, there's the philosophical one that we want our tests to resemble real-world usage as closely as possible, and it's not typical to boot with systemd.debug-shell in real-world usage; it's possible that doing it could cause a bug that otherwise wouldn't happen, or hide one that otherwise would.

So far I've preferred in such cases to reproduce the problem manually, or hack up the tests temporarily on the staging instance to get the logs out in some way. We can definitely consider ways to improve this, but I don't think it's easy.

it doesn't use the system user, it uses root. We create the root account on almost all test paths. There are definitely occasional cases where the current approach doesn't work, but addressing them is complex. "just always add systemd.debug-shell to the kernel cmdline" has several problems: it's actually quite hard to reliably add kernel args such that they will be present in every test. In some situations the bootloader timeout is very short so we can't reliably do it at boot time. We can inject args after install for tests which involve an anaconda install, but some tests don't. And beyond the practical problems, there's the philosophical one that we want our tests to resemble real-world usage as closely as possible, and it's not typical to boot with `systemd.debug-shell` in real-world usage; it's *possible* that doing it could cause a bug that otherwise wouldn't happen, or hide one that otherwise would. So far I've preferred in such cases to reproduce the problem manually, or hack up the tests temporarily on the staging instance to get the logs out in some way. We can definitely consider ways to improve this, but I don't think it's *easy*.

adamwill commented

2025-09-03 17:11:02 +00:00

Owner

Metadata Update from @adamwill:

Custom field story_points adjusted to 4

**Metadata Update from @adamwill**: - Custom field story_points adjusted to 4

kparal commented

2025-09-04 09:44:20 +00:00

Author

Owner

it's actually quite hard to reliably add kernel args such that they will be present in every test.

Yeah, I was more thinking about the cases where you boot from a pre-created disk image, but obviously that's just some fraction of the tests.

I think we can close this ticket, but I still want to ask about this:

We create the root account on almost all test paths.

How do you do this for Workstation installation?

> it's actually quite hard to reliably add kernel args such that they will be present in every test. Yeah, I was more thinking about the cases where you boot from a pre-created disk image, but obviously that's just some fraction of the tests. I think we can close this ticket, but I still want to ask about this: > We create the root account on almost all test paths. How do you do this for Workstation installation?

adamwill commented

2025-09-04 22:43:49 +00:00

Owner

Like this. It's a divergence from a stock install, yeah - but one we live with, because just a lot of the tests are designed around having a root account available and it'd be a lot of work to rejig them all to work without one.

(I mean...we could look at having root_console do sudo, but...mehhh...)

Like [this](https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/main/f/tests/_do_install_and_reboot.pm#_222). It's a divergence from a stock install, yeah - but one we live with, because just a *lot* of the tests are designed around having a root account available and it'd be a lot of work to rejig them all to work without one. (I mean...we could look at having root_console do sudo, but...mehhh...)

kparal commented

2025-09-05 06:46:08 +00:00

Author

Owner

I'm surprised it works fine without forcing a SELinux relabel. Ok, thanks. I think we can close this.

kparal commented

2025-09-16 11:08:33 +00:00

Author

Owner

"just always add systemd.debug-shell to the kernel cmdline" has several problems

For future reference, if this is really needed one day - editing the kernel args is not needed, this can be also enabled by running systemctl enable debug-shell in the installed system, or just by creating the appropriate symlink in etc. Source

> "just always add systemd.debug-shell to the kernel cmdline" has several problems For future reference, if this is really needed one day - editing the kernel args is not needed, this can be also enabled by running `systemctl enable debug-shell` in the installed system, or just by creating the appropriate symlink in etc. [Source](https://fedoraproject.org/wiki/Systemd_early_debug-shell)

adamwill commented

2025-09-17 06:35:17 +00:00

Owner

sure, but the general issue is 'there's no single, surefire way to do this for all tests'. all the tests have different circumstances.