Adjust installation validation matrix and test cases for blivet-gui

adamwill commented

2017-04-26 21:11:19 +00:00

Owner

From Fedora 26 onward, as well as anaconda's own partitioning flows ('automatic' and 'custom', as we usually refer to them) you can ask anaconda to run blivet-gui and do your partitioning in that.

As this option is presented with about equal important to 'automatic' and 'custom', we probably ought to treat it with equal importance for testing. So we need to cover it in the installation validation matrix somehow. We should also look at the actual test cases and see if they need to be reworded a bit to be valid for both 'custom partitioning' and blivet-gui, or even if the differences between how the two work are big enough that we should have separate test cases for blivet-gui.

There's a few different ways we could do this. Someone should come up with a draft or two and send to the list for review.

See also #504 , for adjusting the release criteria.

From Fedora 26 onward, as well as anaconda's own partitioning flows ('automatic' and 'custom', as we usually refer to them) you can ask anaconda to run blivet-gui and do your partitioning in that. As this option is presented with about equal important to 'automatic' and 'custom', we probably ought to treat it with equal importance for testing. So we need to cover it in the installation validation matrix somehow. We should also look at the actual test cases and see if they need to be reworded a bit to be valid for both 'custom partitioning' and blivet-gui, or even if the differences between how the two work are big enough that we should have separate test cases for blivet-gui. There's a few different ways we could do this. Someone should come up with a draft or two and send to the list for review. See also #504 , for adjusting the release criteria.

roshi commented

2017-04-27 18:13:46 +00:00

Metadata Update from @roshi:

Issue assigned to roshi

**Metadata Update from @roshi**: - Issue assigned to roshi

roshi commented

2017-04-27 18:13:57 +00:00

@sumantrom and I are working on the testcases and any changes we need to make to the matrices for this effort. We'll be posting to the list once we have the changes in order.

sumantrom commented

2017-07-17 09:47:25 +00:00

Here's the test cases which I have drafted for the time being , I will send out an email once @adamwill and @roshi will have a look at these
https://fedoraproject.org/wiki/User:Sumantrom/Draft/partitioning_lvm
https://fedoraproject.org/wiki/User:Sumantrom/Draft/partitioning_guided_encrypted
https://fedoraproject.org/wiki/User:Sumantrom/Draft/BlivetGUI_custom_partition

Here's the test cases which I have drafted for the time being , I will send out an email once @adamwill and @roshi will have a look at these https://fedoraproject.org/wiki/User:Sumantrom/Draft/partitioning_lvm https://fedoraproject.org/wiki/User:Sumantrom/Draft/partitioning_guided_encrypted https://fedoraproject.org/wiki/User:Sumantrom/Draft/BlivetGUI_custom_partition

adamwill commented

2017-08-03 20:14:23 +00:00

Author

Owner

So, this isn't really quite what I had in mind. Two specific points, first of all: it's not right to just add a note about the blivet-gui workflow to the partitioning_guided_encrypted test, because that's explicitly a test of 'guided' partitioning, i.e. the 'Automatic' choice. We can't take a test that's specifically about 'Automatic' and then just throw in an admon about 'Advanced Custom', because it'll confuse people and we won't know what people mean when they file a result for that test - did they test the Automatic mode? Or the Advanced Custom mode? We just wouldn't know.

The BlivetGUI_custom_partition test is too vague, it reads like a general exploratory testing case, which again isn't a good fit for validation testing because we don't really know what it means when someone files a result for the test, we don't know what they actually tested.

What I was really expecting here is a set of test cases for 'Advanced Custom' mode that are counterparts of the existing tests for 'Custom' mode - the ones in the 'Custom storage configuration' section of the installation matrix:

https://fedoraproject.org/wiki/Template:Installation_test_matrix#Custom_storage_configuration

Here's how I'd go about this: copy each test case to a new page with a similar name, like copy QA:Testcase_partitioning_custom_btrfs to QA:Testcase_partitioning_advanced_custom_btrfs, and edit it appropriately. Most of the test cases use the Template:Partitioning_single_custom_blank template, so you'd also copy that as Template:Partitioning_single_advanced_custom_blank, change it appropriately (i.e. to tell the tester to select 'Advanced Custom', not 'Custom'), and change all the new test case pages to use the new template. Then you'd change the specific instruction steps in each of the new test cases to whatever the right steps are in blivet-gui, to achieve the intended layout. Then create an 'Advanced custom storage configuration' section in the matrix page, right below the 'Custom storage configuration' section, and put the new test cases into a table there.

Does that make sense? Is it clear enough? Thanks!

So, this isn't really quite what I had in mind. Two specific points, first of all: it's not right to just add a note about the blivet-gui workflow to the partitioning_guided_encrypted test, because that's *explicitly* a test of 'guided' partitioning, i.e. the 'Automatic' choice. We can't take a test that's specifically about 'Automatic' and then just throw in an admon about 'Advanced Custom', because it'll confuse people and we won't know what people mean when they file a result for that test - did they test the Automatic mode? Or the Advanced Custom mode? We just wouldn't know. The BlivetGUI_custom_partition test is too vague, it reads like a general exploratory testing case, which again isn't a good fit for validation testing because we don't really know what it *means* when someone files a result for the test, we don't know what they actually tested. What I was really expecting here is a set of test cases for 'Advanced Custom' mode that are counterparts of the existing tests for 'Custom' mode - the ones in the 'Custom storage configuration' section of the installation matrix: https://fedoraproject.org/wiki/Template:Installation_test_matrix#Custom_storage_configuration Here's how I'd go about this: copy each test case to a new page with a similar name, like copy `QA:Testcase_partitioning_custom_btrfs` to `QA:Testcase_partitioning_advanced_custom_btrfs`, and edit it appropriately. Most of the test cases use the `Template:Partitioning_single_custom_blank` template, so you'd also copy that as `Template:Partitioning_single_advanced_custom_blank`, change it appropriately (i.e. to tell the tester to select 'Advanced Custom', not 'Custom'), and change all the new test case pages to use the new template. Then you'd change the specific instruction steps in each of the new test cases to whatever the right steps are in blivet-gui, to achieve the intended layout. Then create an 'Advanced custom storage configuration' section in the matrix page, right below the 'Custom storage configuration' section, and put the new test cases into a table there. Does that make sense? Is it clear enough? Thanks!

adamwill commented

2018-02-06 22:11:39 +00:00

Author

Owner

@roshi @sumantrom ping?

adamwill commented

2018-02-06 22:12:46 +00:00

Author

Owner

Metadata Update from @adamwill:

Issue priority set to: normal
Issue set to the milestone: Undetermined Future
Issue tagged with: task, test cases, wiki

**Metadata Update from @adamwill**: - Issue priority set to: normal - Issue set to the milestone: Undetermined Future - Issue tagged with: task, test cases, wiki

adamwill commented

2018-06-03 01:09:37 +00:00

Author

Owner

Metadata Update from @adamwill: