quality/tickets

Fork 0

Toolbx testing: figure out a process to test the image from the compose under test #766

New issue

Closed

opened 2024-02-21 23:24:59 +00:00 by adamwill · 8 comments

adamwill commented

2024-02-21 23:24:59 +00:00

Owner

It would be good if we could figure out a way to test the image from the compose under test as well as or instead of the 'current stable' image. This ticket is for updating the wiki test case, we could also update the openQA test if we can figure it out.

This is a follow-up from https://pagure.io/fedora-qa/issue/716 . We did most of what was asked there, but one "hole" in the criteria and test case is that we don't really test the toolbx image that's actually created as part of the compose under test. The test case just says "Execute the command toolbox enter and toolbox run to enter the newly created Toolbox container", and that's what the openQA test does also. That will use, I think, whatever toolbx image was last sort of 'tagged stable' for the *release* being tested, so likely the one from the *previous* compose. It would be good if we could figure out a way to test the image from the compose under test as well as or instead of the 'current stable' image. This ticket is for updating the wiki test case, we could also update the openQA test if we can figure it out.

rishi commented

2024-02-22 13:41:34 +00:00

Is there an easy way to get to the artifact generated by the compose?

If I recall correctly, Image Factory produces a tarball in the so-called docker-archive or docker save format. So, one can download the tarball and do this to make it visible to toolbox and podman:

$ skopeo copy \
    docker-archive:/path/to/tarball \
    containers-storage:registry.fedoraproject.org/fedora-toolbox:40

However, I wonder how much this will be impacted by the KiwiBuiltCloudImages Change for Fedora 40.

This ticket is for updating the wiki test case

Isn't the Wiki test case also used as a hint for toolbox updates to stable Fedoras? If so, we might have to differentiate between Fedoras under development, and those that have been released and considered stable.

Is there an easy way to get to the artifact generated by the compose? If I recall correctly, Image Factory produces a tarball in the so-called `docker-archive` or `docker save` format. So, one can download the tarball and do this to make it visible to `toolbox` and `podman`: ``` $ skopeo copy \ docker-archive:/path/to/tarball \ containers-storage:registry.fedoraproject.org/fedora-toolbox:40 ``` However, I wonder how much this will be impacted by the [KiwiBuiltCloudImages](https://fedoraproject.org/wiki/Changes/KiwiBuiltCloudImages) Change for Fedora 40. > This ticket is for updating the wiki test case Isn't the Wiki [test case](https://fedoraproject.org/wiki/QA:Testcase_toolbox) also used as a hint for `toolbox` updates to stable Fedoras? If so, we might have to differentiate between Fedoras under development, and those that have been released and considered stable.

adamwill commented

2024-02-22 18:25:58 +00:00

Author

Owner

So the way we do this programmatically is via the productmd metadata, i.e. the images.json file. I maintain a thing called fedfind which, these days, mostly acts as a kinda convenience wrapper around that data with some fancy extra functions. The openQA scheduler uses that, and other test systems that want to test images can do the same or use images.json directly or whatever. So test systems can certainly find and test images that are listed there. The toolbox images listed there are the "Fedora-Container-Toolbox" images that have format tar.xz and type 'docker', so I guess they're in the format you describe. We can definitely try the way you describe, thanks.

For humans, the validation matrices list all the images in the compose - in the table at the top of https://fedoraproject.org/wiki/Test_Results:Fedora_40_Branched_20240219.n.0_Installation for e.g. you can see a "Container_Toolbox docker" row. (The tool that generates the pages uses fedfind to produce this table). We can tell human testers to refer to that row to find an image to test.

Yes, the test case will have to cover both paths, the 'simple just run toolbox enter' path and the more complex way to test a specific image.

So the way we do this programmatically is via the productmd metadata, i.e. [the images.json file](https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20240220.n.0/compose/metadata/images.json). I maintain a thing called [fedfind](https://pagure.io/fedora-qa/fedfind) which, these days, mostly acts as a kinda convenience wrapper around that data with some fancy extra functions. The openQA scheduler uses that, and other test systems that want to test images can do the same or use `images.json` directly or whatever. So test systems can certainly find and test images that are listed there. The toolbox images listed there are the "Fedora-Container-Toolbox" images that have format tar.xz and type 'docker', so I guess they're in the format you describe. We can definitely try the way you describe, thanks. For humans, the validation matrices list all the images in the compose - in the table at the top of https://fedoraproject.org/wiki/Test_Results:Fedora_40_Branched_20240219.n.0_Installation for e.g. you can see a "Container_Toolbox docker" row. (The tool that generates the pages uses fedfind to produce this table). We can tell human testers to refer to that row to find an image to test. Yes, the test case will have to cover both paths, the 'simple just run `toolbox enter`' path and the more complex way to test a specific image.

rishi commented

2024-02-22 23:51:23 +00:00

So the way we do this programmatically is via the productmd metadata, i.e. the images.json file. I maintain a thing called fedfind which, these days, mostly acts as a kinda convenience wrapper around that data with some fancy extra functions. The openQA scheduler uses that, and other test systems that want to test images can do the same or use images.json directly or whatever. So test systems can certainly find and test images that are listed there.

Cool.

The toolbox images listed there are the "Fedora-Container-Toolbox" images that have format tar.xz and type 'docker', so I guess they're in the format you describe. We can definitely try the way you describe, thanks.

Yeah, it will work if you download the tarballs and do a skopeo copy to place them in containers-storage with the right fully qualified name.

For humans, the validation matrices list all the images in the compose - in the table at the top of https://fedoraproject.org/wiki/Test_Results:Fedora_40_Branched_20240219.n.0_Installation for e.g. you can see a "Container_Toolbox docker" row. (The tool that generates the pages uses fedfind to produce this table). We can tell human testers to refer to that row to find an image to test.

Yes, the test case will have to cover both paths, the 'simple just run toolbox enter' path and the more complex way to test a specific image.

nod

Thanks for working on this, @adamwill !

> So the way we do this programmatically is via the productmd metadata, i.e. [the images.json file](https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20240220.n.0/compose/metadata/images.json). I maintain a thing called [fedfind](https://pagure.io/fedora-qa/fedfind) which, these days, mostly acts as a kinda convenience wrapper around that data with some fancy extra functions. The openQA scheduler uses that, and other test systems that want to test images can do the same or use `images.json` directly or whatever. So test systems can certainly find and test images that are listed there. Cool. > The toolbox images listed there are the "Fedora-Container-Toolbox" images that have format tar.xz and type 'docker', so I guess they're in the format you describe. We can definitely try the way you describe, thanks. Yeah, it will work if you download the tarballs and do a `skopeo copy` to place them in `containers-storage` with the right fully qualified name. > For humans, the validation matrices list all the images in the compose - in the table at the top of https://fedoraproject.org/wiki/Test_Results:Fedora_40_Branched_20240219.n.0_Installation for e.g. you can see a "Container_Toolbox docker" row. (The tool that generates the pages uses fedfind to produce this table). We can tell human testers to refer to that row to find an image to test. > > Yes, the test case will have to cover both paths, the 'simple just run `toolbox enter`' path and the more complex way to test a specific image. *nod* Thanks for working on this, @adamwill !

adamwill commented

2024-02-23 00:04:33 +00:00

Author

Owner

No problem. I've updated the wiki test case and I'm working on enhancing the openQA test to do this now. Let me know if you see anything wrong. Thanks for the pointers!

edit: test run of the modified openQA test will run at https://openqa.stg.fedoraproject.org/tests/3597885 soon, cross your fingers...

No problem. I've [updated the wiki test case](https://fedoraproject.org/w/index.php?title=QA:Testcase_toolbox&diff=701708&oldid=687378) and I'm working on enhancing the openQA test to do this now. Let me know if you see anything wrong. Thanks for the pointers! edit: test run of the modified openQA test will run at https://openqa.stg.fedoraproject.org/tests/3597885 soon, cross your fingers...

adamwill commented

2024-02-23 01:16:56 +00:00

Author

Owner

why is skopeo copy so slow? I gave it five minutes to run in openQA and it timed out! let's try 10...

adamwill commented

2024-02-23 01:42:04 +00:00

Author

Owner

alright, well, 10 minutes works. https://openqa.stg.fedoraproject.org/tests/3597926#details . gonna merge all that into prod and call it good.

adamwill commented

2024-02-23 01:42:04 +00:00

Author

Owner

Metadata Update from @adamwill:

Issue close_status updated to: Fixed
Issue status updated to: Closed (was: Open)

**Metadata Update from @adamwill**: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)

rishi commented

2024-02-23 09:54:13 +00:00

Fantastic!

why is skopeo copy so slow? I gave it five minutes to run
in openQA and it timed out! let's try 10...

Yeah, it can be a bit slow. I never dug into why that is.

Fantastic! > why is skopeo copy so slow? I gave it five minutes to run > in openQA and it timed out! let's try 10... Yeah, it can be a bit slow. I never dug into why that is.