Make sure we have some rpm-ostree coverage in OpenQA

kparal commented

2020-07-10 12:47:58 +00:00

Owner

According to Sudhir, it's important that we have some test coverage of ostree. Given what we test, it mostly makes sense to cover this on Silverblue, because it heavily uses (rpm-)ostree and we want to have Silverblue covered anyway. Ideally in OpenQA with automation.

The task here is to look at the existing test cases (manual and automated), if we already have something that matches. I suppose we don't. Then create some test cases that would cover at least the basic operations (like updating Silverblue, rolling it back, switching to a different tree/stream, something like that) and communicate with our OpenQA folks (Adam, LukasR) whether they can implement it in OpenQA.

There are already some test cases which can be used as an inspiration, like:
https://fedoraproject.org/wiki/QA:Testcase_CoreOS_switch_stream

@sumantrom @coremodule Any volunteers? Thanks!

According to Sudhir, it's important that we have some test coverage of ostree. Given what we test, it mostly makes sense to cover this on Silverblue, because it heavily uses (rpm-)ostree and we want to have Silverblue covered anyway. Ideally in OpenQA with automation. The task here is to look at the existing test cases (manual and automated), if we already have something that matches. I suppose we don't. Then create some test cases that would cover at least the basic operations (like updating Silverblue, rolling it back, switching to a different tree/stream, something like that) and communicate with our OpenQA folks (Adam, LukasR) whether they can implement it in OpenQA. There are already some test cases which can be used as an inspiration, like: https://fedoraproject.org/wiki/QA:Testcase_CoreOS_switch_stream @sumantrom @coremodule Any volunteers? Thanks!

coremodule commented

2020-07-10 15:33:29 +00:00

I think this is better suited for one of the OpenQA folks to do (@adamw , @lruzicka). Our OpenQA instance isn't the best documented system for someone to jump on and start using without a lot of questions being asked straight to one of the two mentioned above. That said, I don't think Sumantro or I would be able to figure out what is or isn't possible in OpenQA, as we've never worked with it before because it is so exclusive.

sumantrom commented

2020-07-10 16:39:18 +00:00

That said, I will need some handholding with OpenQA and the above test cases, needs some updating which I will do and add a bit more test cases in the coming week.

@kparal The test cases we have for silverblue atm are: Basic http://fedoraproject.org/wiki/QA:Testcase_base_startup http://fedoraproject.org/wiki/QA:Testcase_base_system_logging http://fedoraproject.org/wiki/QA:Testcase_Services_start http://fedoraproject.org/wiki/QA:Testcase_base_selinux http://fedoraproject.org/wiki/QA:Testcase_base_service_manipulation Silverblue specific http://fedoraproject.org/wiki/User:Sumantrom/Draft/Test_Case_install_flatpak_repo http://fedoraproject.org/wiki/QA:Testcase_desktop_update_notification http://fedoraproject.org/wiki/QA:Testcase_gnome-software-addons http://fedoraproject.org/wiki/User:Sumantrom/Draft/Testcase_silverblue_upgrade_%26_rebase http://fedoraproject.org/wiki/User:Sumantrom/Draft/Testcase_silverblue_rebase http://fedoraproject.org/wiki/User:Sumantrom/Draft/Testcase_silverblue_IDE_install http://fedoraproject.org/wiki/User:Sumantrom/Draft/Testcase_silverblue_googlechrome_install http://fedoraproject.org/wiki/User:Sumantrom/Draft/Testcase_silverblue_toolbox That said, I will need some handholding with OpenQA and the above test cases, needs some updating which I will do and add a bit more test cases in the coming week.

adamwill commented

2020-07-11 01:19:57 +00:00

Owner

We do already cover https://fedoraproject.org/wiki/QA:Testcase_RpmOstree_Package_Layering and https://fedoraproject.org/wiki/QA:Testcase_RpmOstree_Rebase with openQA, on IoT. We also have https://fedoraproject.org/wiki/QA:Testcase_RpmOstree_Upgrade in the IoT matrix and a ticket for automating it, but it's a bit tricky to implement so it's not done yet.

We should not duplicate ostree functionality tests between IoT and Silverblue. For upgrade and rebase we should just add those existing tests to the appropriate matrix for Silverblue, I guess. Any additional tests we want to write that aren't really specific to Silverblue but are just generic rpm-ostree tests should use the same naming convention and be added to the matrices for both.

We do already cover https://fedoraproject.org/wiki/QA:Testcase_RpmOstree_Package_Layering and https://fedoraproject.org/wiki/QA:Testcase_RpmOstree_Rebase with openQA, on IoT. We also have https://fedoraproject.org/wiki/QA:Testcase_RpmOstree_Upgrade in the IoT matrix and [a ticket for automating it](https://pagure.io/fedora-qa/os-autoinst-distri-fedora/issue/169), but it's a bit tricky to implement so it's not done yet. We should not duplicate ostree functionality tests between IoT and Silverblue. For upgrade and rebase we should just add those existing tests to the appropriate matrix for Silverblue, I guess. Any additional tests we want to write that aren't really specific to Silverblue but are just generic rpm-ostree tests should use the same naming convention and be added to the matrices for both.

adamwill commented

2020-07-11 01:20:32 +00:00

Owner

oh, forgot to mention, it should be fairly simple to extend openQA to run the existing layering and rebase tests on Silverblue as well as IoT, if we want to do that.

lruzicka commented

2020-07-13 16:52:04 +00:00

Owner

I will also take a look and see if I could be of any help here. To switch on some test cases to run on Silverblue is pretty easy, so we could start there and see how it entwines.

kparal commented

2020-07-14 13:19:58 +00:00

Author

Owner

Metadata Update from @kparal:

Issue tagged with: iot

**Metadata Update from @kparal**: - Issue tagged with: iot

kparal commented

2020-07-14 13:24:43 +00:00

Author

Owner

I think this is better suited for one of the OpenQA folks to do (@adamw , @lruzicka). Our OpenQA instance isn't the best documented system for someone to jump on and start using without a lot of questions being asked straight to one of the two mentioned above.

The intention of this ticket wasn't to write the automation, but figure out the current state and prepare everything needed so that OpenQA folks can write it. I.e. we'd just write the (manual) test cases, if needed.

> I think this is better suited for one of the OpenQA folks to do (@adamw , @lruzicka). Our OpenQA instance isn't the best documented system for someone to jump on and start using without a lot of questions being asked straight to one of the two mentioned above. The intention of this ticket wasn't to write the automation, but figure out the current state and prepare everything needed so that OpenQA folks can write it. I.e. we'd just write the (manual) test cases, if needed.

kparal commented

2020-07-14 13:34:55 +00:00

Author

Owner

We do already cover https://fedoraproject.org/wiki/QA:Testcase_RpmOstree_Package_Layering and https://fedoraproject.org/wiki/QA:Testcase_RpmOstree_Rebase with openQA, on IoT. We also have https://fedoraproject.org/wiki/QA:Testcase_RpmOstree_Upgrade in the IoT matrix and a ticket for automating it, but it's a bit tricky to implement so it's not done yet.

Awesome! That's much better than I expected. And I completely forgot that IoT uses rpmostree as well, that of course counts as well.

The upgrade test would certainly be needed to have this reasonably covered. I'd also like to see rollback covered.

Does anyone know of some other important rpmostree functionality that should be covered?

We should not duplicate ostree functionality tests between IoT and Silverblue. For upgrade and rebase we should just add those existing tests to the appropriate matrix for Silverblue, I guess.

We can take care of this in this ticket. We can make the rebase testcase a bit more generic, by saying the ioT mentioned in it is just an example and the same process is applicable to any edition based on rpm-ostree.

Does it make sense to place this into the Base matrix?

Any additional tests we want to write that aren't really specific to Silverblue but are just generic rpm-ostree tests should use the same naming convention and be added to the matrices for both.

Agreed. Do we have a simple way how to display the latest IoT matrix? It's not listed here:
https://fedoraproject.org/wiki/QA:Release_validation_test_plan#Current_and_recent_validation_test_events

> We do already cover https://fedoraproject.org/wiki/QA:Testcase_RpmOstree_Package_Layering and https://fedoraproject.org/wiki/QA:Testcase_RpmOstree_Rebase with openQA, on IoT. We also have https://fedoraproject.org/wiki/QA:Testcase_RpmOstree_Upgrade in the IoT matrix and a ticket for automating it, but it's a bit tricky to implement so it's not done yet. Awesome! That's much better than I expected. And I completely forgot that IoT uses rpmostree as well, that of course counts as well. The upgrade test would certainly be needed to have this reasonably covered. I'd also like to see [rollback](https://fedoraproject.org/wiki/QA:Testcase_CoreOS_switch_stream#Optional) covered. Does anyone know of some other important rpmostree functionality that should be covered? > We should not duplicate ostree functionality tests between IoT and Silverblue. For upgrade and rebase we should just add those existing tests to the appropriate matrix for Silverblue, I guess. We can take care of this in this ticket. We can make the rebase testcase a bit more generic, by saying the ioT mentioned in it is just an example and the same process is applicable to any edition based on rpm-ostree. Does it make sense to place this into the Base matrix? > Any additional tests we want to write that aren't really specific to Silverblue but are just generic rpm-ostree tests should use the same naming convention and be added to the matrices for both. Agreed. Do we have a simple way how to display the latest IoT matrix? It's not listed here: https://fedoraproject.org/wiki/QA:Release_validation_test_plan#Current_and_recent_validation_test_events

kparal commented

2020-07-14 13:43:01 +00:00

Author

Owner

http://fedoraproject.org/wiki/User:Sumantrom/Draft/Testcase_silverblue_upgrade_%26_rebase
http://fedoraproject.org/wiki/User:Sumantrom/Draft/Testcase_silverblue_rebase

These two fit the description, Sumantro, thanks for those links. As Adam said, these would be prime candidates for merging into generic ostree-related test cases which can then be used for both IoT and Silverblue (and CoreOS most probably).

We should also synchronize with CoreOS which some test cases (meant to be testday-specific) here:
https://fedoraproject.org/wiki/Category:CoreOS_Test_Cases
and the matching one is here:
https://fedoraproject.org/wiki/QA:Testcase_CoreOS_switch_stream

We still haven't finalized how and whether we'll have CoreOS included in our testing matrices or not (that's still in our backlog), but in this case the test case seem it can easily apply to all these edition.

Volunteers?

> http://fedoraproject.org/wiki/User:Sumantrom/Draft/Testcase_silverblue_upgrade_%26_rebase http://fedoraproject.org/wiki/User:Sumantrom/Draft/Testcase_silverblue_rebase These two fit the description, Sumantro, thanks for those links. As Adam said, these would be prime candidates for merging into generic ostree-related test cases which can then be used for both IoT and Silverblue (and CoreOS most probably). We should also synchronize with CoreOS which some test cases (meant to be testday-specific) here: https://fedoraproject.org/wiki/Category:CoreOS_Test_Cases and the matching one is here: https://fedoraproject.org/wiki/QA:Testcase_CoreOS_switch_stream We still haven't finalized how and whether we'll have CoreOS included in our testing matrices or not (that's still in our backlog), but in this case the test case seem it can easily apply to all these edition. Volunteers?

adamwill commented

2020-07-18 00:16:31 +00:00

Owner

Does it make sense to place this into the Base matrix?

Seems as good a place as any, yeah. Of course, Silverblue would be the only environment for it, since we don't have IoT images in the main compose.

We should also synchronize with CoreOS which some test cases (meant to be testday-specific) here:

I'd probably just replace those test cases with these 'unified' / 'generic' ones - make those pages just redirect to the tests we're planning here.

Agreed. Do we have a simple way how to display the latest IoT matrix?

https://fedoraproject.org/wiki/Test_Results:Current_IoT_General_Test

It's not listed here:
https://fedoraproject.org/wiki/QA:Release_validation_test_plan#Current_and_recent_validation_test_events

I'll update that :)

> Does it make sense to place this into the Base matrix? Seems as good a place as any, yeah. Of course, Silverblue would be the only environment for it, since we don't have IoT images in the main compose. > We should also synchronize with CoreOS which some test cases (meant to be testday-specific) here: I'd probably just replace those test cases with these 'unified' / 'generic' ones - make those pages just redirect to the tests we're planning here. > Agreed. Do we have a simple way how to display the latest IoT matrix? https://fedoraproject.org/wiki/Test_Results:Current_IoT_General_Test > It's not listed here: https://fedoraproject.org/wiki/QA:Release_validation_test_plan#Current_and_recent_validation_test_events I'll update that :)

kparal commented

2020-07-28 09:47:30 +00:00

Author

Owner

Seems as good a place as any, yeah. Of course, Silverblue would be the only environment for it, since we don't have IoT images in the main compose.

Right, so we'll have the test case listed in the Base matrix for Silverblue, and then once again in the IoT General matrix.

I'd probably just replace those test cases with these 'unified' / 'generic' ones - make those pages just redirect to the tests we're planning here.

Some of that might be tricky, because one of the points of the test day was to also validate existing documentation and link to it as much as possible. In some cases, it can be probably generalized, and for other cases, I won't mind having a generic test case (to be used during release validation) and a very similar testday-specific test case (to be used in future test days, when requirements/goals are slightly different).

@coremodule @sumantrom Can one of you please take this (see #comment-665174 and onward)? Thanks!

> Seems as good a place as any, yeah. Of course, Silverblue would be the only environment for it, since we don't have IoT images in the main compose. Right, so we'll have the test case listed in the Base matrix for Silverblue, and then once again in the IoT General matrix. > I'd probably just replace those test cases with these 'unified' / 'generic' ones - make those pages just redirect to the tests we're planning here. Some of that might be tricky, because one of the points of the test day was to also validate existing documentation and link to it as much as possible. In some cases, it can be probably generalized, and for other cases, I won't mind having a generic test case (to be used during release validation) and a very similar testday-specific test case (to be used in future test days, when requirements/goals are slightly different). @coremodule @sumantrom Can one of you please take this (see [#comment-665174](https://pagure.io/fedora-qa/issue/635#comment-665174) and onward)? Thanks!

sumantrom commented

2020-08-11 15:14:45 +00:00

@kparal I will be taking this and will be coming up with something by tomorrow. Thanks.