Create test suite for Fedora Media Writer #462
No reviewers
Labels
No labels
ai-review-please
Closed As
Duplicate
Closed As
Fixed
Closed As
Invalid
easyfix
freeipa
improvetest
iot
newtest
silverblue
tracker
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
No milestone
No project
5 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
quality/os-autoinst-distri-fedora!462
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "newtest/fedora-media-writer"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
This PR adds the test suite for Fedora Media Writer, which is
an application to download and copy the installation media files
to the a USB disk which can be then used for installation.
So far, we have been testing this application with manual testing,
this is an attempt to automate it. It tests the following:
and writes it to the USB disk. It also checks that the usual
content exists on the USB disk.
Then it check that the usual content exists on the USB disk.
original state.
Fixes: https://pagure.io/fedora-qa/os-autoinst-distri-fedora/issue/450
Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
https://fedora.softwarefactory-project.io/zuul/buildset/9683833692974ee48094d3962d06b099
It seems that I do not have the
disconnect_usbyet in installation, so the Perl check above fails.Passes at https://openqa.stg.fedoraproject.org/tests/5611820#details
rebased onto
9a451c32a2Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
https://fedora.softwarefactory-project.io/zuul/buildset/8e6dfbd184da4af2ac44c8669dbbffc5
rebased onto
93606be074Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
https://fedora.softwarefactory-project.io/zuul/buildset/6439f8e87980431fbbe062f98e402fcb
Tests still fail because they're running on F41, because nobody merged https://pagure.io/fedora-project-config/pull-request/351 yet. :(
rebased onto
712ca067a8robot overlord review. edit: it came out a bit weird - it seems to think it's adding an entirely new automated test suite to the FMW application itself, which is...not what we're doing at all. still, the specific review notes are...sort of on point.
Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
https://fedora.softwarefactory-project.io/zuul/buildset/41f8115a770e4aaa8d026b6ca7acdb7e
Do we really need both
mwriter_download_finishedandmwriter_finish_button? Couldn't theuntilloop just wait for the finish button to appear? Presumably it only appears when the download is finished anyway.I think we already have a needle the same as
mwriter_shortcut_downloadedin the Nautilus tests -nautilus_directory_downloads.We already set
QEMU_APPENDfor the64bitmachine, which will be used to run this test on x86_64. Setting it in the test suite effectively overrides that setting, so when running this test, we won't pass the args set at the machine level any more:that's a bit of cargo culting from when I enabled Secure Boot testing. Best I can figure out/remember, it locks down the UEFI pflash variables drive so only the firmware should be able to write to it - this is a more accurate simulation of a 'real' system with Secure Boot, and should catch any problems caused by the OS trying to write to it or something. Skipping it for this one test is probably fine, but I thought I'd note it.
I don't love the use of
QEMU_APPENDto attach the disk as a matter of design, but AFAICS os-autoinst upstream doesn't offer this functionality so it's about all we can do. Long term it would be nice to look at adding this to os-autoinst, though, and maybe even allowing for hotplugging via QMP.1 new commit added
Fix after reviewBuild failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
https://fedora.softwarefactory-project.io/zuul/buildset/fe995dc70e1a4ad7b11296456686e76b
rebased onto
fcb4dcbd98I changed the needle things, rebased on main, however got some missing needles for
ps_*connected with Plasma settings, I am not aware to have used any of them. They might have come from the rebase though.As far as
QEMU_APPENDis concerned:sorry, yeah, the ps_ is from the plasma setup changes I merged, I must've missed running the needle check script.
For QEMU_APPEND I think it's probably OK to leave it. I mostly just wanted to write it down here so we have it as a reference if it ever becomes an issue in the future.
rebased onto
672916955fBuild failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
https://fedora.softwarefactory-project.io/zuul/buildset/36e49764811c4a0a9560cd7751ec62aa
rebased onto
656f44641cWhy do we bother with making
$mediagroupconfigurable asMEDIAvar but never actually change it? It's always 'official'. Is there a reason not to just hardcode it?Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
https://fedora.softwarefactory-project.io/zuul/buildset/84ae7a50917c4e2580eb3eeda33db56f
You also didn't address the robot overlord review. I disagree with a couple of its notes, but it has a valid one: the definition of
my $passwordintests/applications/media-writer/download_write.pmis unnecessary, it is never used.Why do we do the
mkdir -p /mnt/usbdiskoutside of the check function? Why not just do it in there so we only have the code once, not three times?I was thinking that maybe in the future, when we'd like to test a different image or something. Changing to the hardcoded version.
I expected the overlord to provide a link, then realized it was you who had provided it. Went and found the review and fixed it.
3 new commits added
Move mount point creation in the check functionHardcode mediagroupRemoveBrain fog? Fixed.
Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
https://fedora.softwarefactory-project.io/zuul/buildset/0f574f4b56df4cf1ac7c42e832e846b0
thanks!
the robot overlord stuff is all manual for now. I'm literally just doing
git push ai-test main; git push ai-test <branchname>then opening a pull request from the link (obviously,ai-testis https://github.com/AdamWill/os-autoinst-distri-fedora-ai-test). The overlord review happens automatically once the PR is opened, then I just have to link it back here.I could automate that, but it feels like a waste of effort. The stuff I'm doing on ai-code-review is meant to be a more permanent alternative, once we move this repo to forgejo.
rebased onto
13811e106cBuild failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
https://fedora.softwarefactory-project.io/zuul/buildset/71590efa76b045238507c7fe775f16f9
rebased onto
56bfde0f7aBuild failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
https://fedora.softwarefactory-project.io/zuul/buildset/aaed023263b04a8fb0f43c963eda84b5
rebased onto
e43f7eac09So now I'm looking at the results of this over the shutdown, it commonly fails with an error like this:
which I'm gonna guess happens because we get multiple instances of the test running at the same time, all trying to lock the same file. So that's going to be a problem.
I think we might need to extend os-autoinst upstream to support adding USB devices with overlay storage, just like how it handles hard disk images currently...
Build succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/0b1ba1168d39423886703446b31d6f17
rebased onto
80ec5ea718Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
https://fedora.softwarefactory-project.io/zuul/buildset/13645a3f438c4961989efcd887e3d0f6
rebased onto
abf1ab7755Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
https://fedora.softwarefactory-project.io/zuul/buildset/0d9555b1ff6d4f23907239fcef8f1ed5
rebased onto
d8cdebe87eBuild failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
https://fedora.softwarefactory-project.io/zuul/buildset/d9aeafb5cdef4c8aa2cee5bcafe6aa9e
A small note on this part:
Please consider using
-blockdev, as QEMU upstream considers-driveto be legacy and might remove it in the futureUntested translation of your current QEMU_APPEND line:
Also, this is not a "cosmetic change", although it can look like that.
-drivehas weird hacks and historical baggage, the-blockdevis a major low-level rework (from 5 years ago, now it's stable) in the QEMU block layer.While there's no urgency here just wanted to note that QEMU upstream discourages
-drivefor new applications. And OpenQA already uses-blockdevfor disk images.rebased onto
9a97fdcaf1Build succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/d1fa52c18f504a5db23e2f55b3105640
1 new commit added
Change -drive to -blockdevBuild succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/7317583777a7454d8560c8f258b474d0
1 new commit added
Try something elseBuild succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/d37b80a24cce42fbb3e4c6d50af3cd37
Everything worked until I started to experiment with
-blockdevthat is not able to provide the functionality that-drivecould. At least I could not figure that out just yet.I might have figured that out after all, I will need to check tomorrow once more, but I have seen tests passing with the latest update.
rebased onto
9a97fdcaf1Build succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/b1eeb5ed834647c19906dbe4e2685a6b
1 new commit added
Try making overlay in the variablesBuild succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/6935f13ac7ab4b62970022091174aebd
4 new commits added
Try making overlay in the variablesChange -drive to -blockdevFix the locking mechanism and add tests as requestedCreate test suite for Fedora Media WriterBuild succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/6d405b5b2c3743949df470609e51316b
2 new commits added
Make a locally working solutionRevert "Try making overlay in the variables"Build succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/49507ff856844480807b7e99658c520b
3 new commits added
Change -drive to -blockdevFix the locking mechanism and add tests as requestedCreate test suite for Fedora Media WriterBuild succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/c97e226256eb44148c318c4ce7cd0f46
So, I think I have lost all the stammina here and I feel drenched to the bone, however:
checkmd5isoandcmpper jobbasis so the test can run parallel with other similar tests-blockdevand not-driveas requested by Kashyap.Sorry, it has taken so long, but I tried numerous ways to set up the overlay, something only worked with
-drive, something only worked locally and not on Staging. When something worked and I wanted to build something else above, it failed apart again, so I had to try different combinations of the variables, using either theHDDor theQEMU_APPENDapproach, and each failure took ages to retest.Now, I am not sure I can come with something better than this.
rebased onto
f6c8cb3bceBuild succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/bbdce6e61eb5478e866b3198070f3018
I do think since it took this much work it might have been worth looking at doing it upstream instead, but congratulations :D I'll try and take a look at it soon once I can deal with all these piles of fire...
rebased onto
c72066ae32Build succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/0ad09c97a12440a88158127c33a3aa79
rebased onto
1d6a96231bBuild succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/652fcbe415b1495581d8ac996eab347e
rebased onto
41df08d878Build succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/a4c23dce815c46e5bca93bfec51a452e
rebased onto
0ebd4fdc31Build succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/baa9c28d69db46eb8fc05578ae610241
a6e6f93fd2to5a41509f095a41509f09e8f1023702@adamwill wrote in #462 (comment):
But who would be doing it? I did not feel confident enough to meddle with the upstream because of the hotplugging you mentioned. However, if you mean like add an upstream variable that would enable what I enable manually here. That might be worth trying once this has proved to work.
Sure, why not? It's a good way to learn - perl skills, and how os-autoinst works upstream. Even if you can't figure it out it's probably worth it as an exercise.
Overall this is looking good now, but I do have some nits, mainly to do with organization.
authenticatesub doesn't have anything much to do with disks, per se. It should probably move intoutils.pmnow we're calling it from elsewhere. We might also want to grep for other cases ofauth_requiredand see if any of those could use this sub too.select_subs inlib/mwriter.pmare used only once each and are quite small. Why not just in-line them where they're used?utils.pmwhere we dotoolbox -y enter- if you pass-ylike that it will automatically create the toolbox if it doesn't exist. You can specify the name to enter explicitly if necessary. Then you can doassert_script_run "true", 180;to wait until the toolbox is actually ready; it's more reliable than a magic sleep. There is alsotoolbox runfor running a single command in a specific container, which also might be useful here (again with-yto create it if necessary I believe). I think it'd be best to follow those patterns for the toolbox stuff here; I'm not sure whether it's worth de-duplicating with the other two uses since the subs would be very short if efficiently implemented, but see what you think.AI Code Review
📋 MR Summary
Implements an automated test suite for Fedora Media Writer, covering installation, downloading/writing media (Workstation & Silverblue), writing existing ISOs, and restoring the USB device.
lib/mwriter.pmhelper for FMW specific interactions and verification logictoolboxhelper inutils.pmfor Silverblue package managementdownload_write,open_write,restore,aboutmedia_writerprofile totemplates.fif.jsonusing-blockdevcompatible settingsDetailed Code Review
The PR successfully implements the requested test flows for Fedora Media Writer and addresses the infrastructure requirements (blockdev/locking). The separation of logic into
mwriter.pmis a good design choice.However, there are significant logic errors in the verification steps (restore verification and ISO selection) that need to be addressed to ensure the tests actually validate the functionality. Additionally, despite the previous review notes, the unused password variable remains in the code.
📂 File Reviews
📄 `tests/applications/media-writer/restore.pm` - Test module for restoring the USB device to its original state.
[ ... ] && echo 1 || echo 0will always exit with code 0 (success), soassert_script_runwill never fail regardless of whether the disk is empty or not.mountcommand usesscript_runinstead ofassert_script_run. If mounting fails (e.g., partition missing), the script continues, and the subsequent empty-check will pass on the empty mountpoint, causing a false positive.mkdir -p /mnt/usbdisk. This directory is already created inaaa_setup.pm, which is a prerequisite for this suite.aaa_setup.pmis guaranteed to run, matching the addressed issues list.📄 `lib/mwriter.pm` - Helper library for Media Writer interactions and verification.
ls /home/test/Downloads/*.isois risky. If previous tests left ISOs or if multiple ISOs exist, this returns multiple lines. Used incmp, this will break the syntax or compare the wrong files.$comparecommand is unquoted. If the ISO filename contains spaces or special characters, the command will fail.📄 `tests/applications/media-writer/download_write.pm` - Main test flow for downloading and writing an image.
$passwordis defined but never used. The review context stated this was removed, but it persists in the diff.📄 `lib/utils.pm` - Utility functions, adding `toolbox` helper.
toolboxsubroutine blindly passes$paramstosudo dnf. While low risk in a test environment, it allows command injection if caller data is tainted.✅ Summary
restore.pm(currently always passes).script_runtoassert_script_runfor themountcommand inrestore.pmto prevent false positives.lib/mwriter.pmto preventcmpcommand failures.$passwordvariable indownload_write.pmas previously agreed.lib/mwriter.pm.mkdircalls that are covered byaaa_setup.pm.🤖 AI Code Review | Generated with ai-code-review
Platform: Forgejo | AI Provider: gemini | Model: gemini-3-pro-preview
I don't have time to check if the robot overlord comments are accurate right now, please check them over and see what you think :)
@adamwill wrote in #462 (comment):
Done. Actually, I made the routine a little better to enable better control.
Done.
The
toolbox run -ydoes not work, it wants to have created the container before, but that's not a big deal. The problem is that when I used to run the command throughtoolbox runit would not work with the assert_script_run. I will try using it and see how it performs.cac50ff60dfb8dc856fbThe test still passes on Staging with the above fixes.
AI Code Review
📋 MR Summary
Introduction of a comprehensive test suite for Fedora Media Writer covering download, write, and restore workflows on Workstation and Silverblue.
mwriter.pmlibrary with robust device verification usingcheckisomd5andcmp.download_write.pm,open_write.pm,restore.pm,about.pm, andaaa_setup.pm.authenticateandtoolboxhelpers inlib/utils.pm.templates.fif.jsonto define themedia_writertest scheduling and dual-disk configuration.authenticate) affecting multiple existing tests. However, the logic changes improve maintainability and the new tests are isolated.Detailed Code Review
This is a well-structured PR that significantly improves testing coverage for Fedora Media Writer. The centralization of
authenticateandtoolboxlogic is a welcome refactoring. The verification strategy inmwriter.pm(usingcmpwith EOF detection andcheckisomd5) is clever and much more robust than simple file existence checks. The support for Silverblue viatoolboxencapsulation is correctly implemented.📂 File Reviews
📄 `lib/mwriter.pm` - New library for Media Writer verification logic.
Minor [Security]: The
lscommand is injected intoscript_outputwithout strict validation of the result, though unlikely to be exploitable here.Suggestion [Logic]: The logic assumes
cmpis available on the Silverblue host (not inside toolbox). Whilediffutilsis usually part of the base image, it's worth verifying this dependency holds true for minimal Silverblue installs.cmpis present on the Silverblue host system or run it inside the toolbox likecheckisomd5if missing.📄 `lib/utils.pm` - Utility updates including centralized authentication and toolbox helpers.
Minor [Performance]: In the
toolboxsubroutine,assert_script_run('true', timeout => 600)followstype_string. If the prompt detection logic fails (e.g., PS1 differs inside toolbox), this might time out or type prematurely.Suggestion [Logic]: The
authenticatesubroutine defines%defaultsinside the function scope correctly, but ensuringwait_still_screenis only called when necessary (i.e., after typing) is good practice. The current implementation looks correct.📄 `tests/applications/media-writer/aaa_setup.pm` - Setup script for installation and environment prep.
$self->root_console(tty => 3). On Silverblue,toolboxis typically designed for unprivileged users. Runningtoolbox enteras root might create a root toolbox, which is valid but distinct from the user one.toolboxoperations behave as expected when running as root in TTY3 on Silverblue.📄 `tests/applications/media-writer/restore.pm` - Test for the restore functionality.
assert_script_run('mount /dev/sda1 /mnt/usbdisk')assumes the restore operation created a specific partition layout (sda1). If the restore fails or creates a different layout (e.g., unpartitioned device), this will fail.lsblk) before attempting to mount, or handle the mount failure gracefully.📄 `tests/applications/media-writer/download_write.pm` - Main workflow test for downloading and writing media.
mwriter_finish_buttonhandles timeouts manually. If the timeout is reached, the loop exits and immediately asserts the button exists, which will fail. This creates the correct failure signal but relying onwait_screen_changeorassert_screenwith a long timeout is often cleaner than manual sleep loops.✅ Summary
restore.pm. The code is clean, follows project patterns, and adequately addresses the complexities of testing physical device emulation.tests/applications/media-writer/restore.pm,mount /dev/sda1might fail if the restored disk layout varies (e.g. whole disk vs partition). Validatelsblkoutput or try mounting alternative paths.toolboxbehavior when running as root inaaa_setup.pmto ensure it works consistently on Silverblue.lib/mwriter.pm, consider verifyingcmpavailability on the Silverblue host system.🤖 AI Code Review | Generated with ai-code-review
Platform: Forgejo | AI Provider: gemini | Model: gemini-3-pro-preview
I don't think we need to worry about anything in the AI review. Let's merge this. Thanks. If it causes problems we'll work through them...