Consider testing very large updates via side tag not download #569

New issue

Open

opened 2026-06-18 09:51:29 +00:00 by adamwill · 0 comments

adamwill commented

2026-06-18 09:51:29 +00:00

Owner

Very large updates are tricky for us. We just about manage to handle GNOME / KDE-scale updates. We just barely seem to fall over on the current OpenSSL 4 update, which is 736 packages ATM, including some very large ones.

The biggest issue is downloading all the packages. We do this in every test - it seems like it would be easy to do it once outside the test flow and "inject" them somehow to all the tests, but it really is not, we've tried two different ways to do this and both had issues and the current idea is "write some kind of proxy server" which seems like a lot of effort.

That creates a huge amount of load on Koji when we test a very large update, because we're running ~700 tests simultaneously, each of which is trying to download all 736 (or whatever) builds, in batches of 20 at a time. It seems there's a point at which this just falls over.

For the OpenSSL 4 update I hacked up a dumb workaround: just run the tests on the update's side tag. And it mostly works, with a few follow-ups. So I was thinking: maybe we clean that up a bit and use it for all large (>100?) updates which are created from a side tag (which ought to be basically all of them).

The main potential issue is changes in the side tag content during the test lifecycle. The side tag content can be changed at any time, and refreshing the update content is a separate action, so at almost any time, what's in the side tag and what's in the update may not match. It can even change during the execution of a single test, so when we update from the side tag early in the process and when we check what's "in the update" at the end of the test we might see different packages. This is a problem.

My initial idea is to get the current revision of the side tag at the time we schedule the tests, and tie them to that, rather than using the /latest directory. This avoids the problem of the content changing during the test run. But I'm not sure if it's possible to be sure that the current state of the side tag repo at the time we trigger update tests (which is when the update is created or its package contents change) is the same as the content of the update. I don't know if the side tag repo is reliably regenerated every time a build is run, or every time the maintainer clicks "create update" or "save" or whatever in Bodhi. If not, the idea might be a non-starter. I'll have to look into that.

Very large updates are tricky for us. We just about manage to handle GNOME / KDE-scale updates. We just barely seem to fall over on [the current OpenSSL 4 update](https://bodhi.fedoraproject.org/updates/FEDORA-2026-54c7ad647e), which is 736 packages ATM, including some very large ones. The biggest issue is downloading all the packages. We do this in *every* test - it seems like it would be easy to do it once outside the test flow and "inject" them somehow to all the tests, but it really is not, we've tried two different ways to do this and both had issues and the current idea is "write some kind of proxy server" which seems like a lot of effort. That creates a huge amount of load on Koji when we test a very large update, because we're running ~700 tests simultaneously, each of which is trying to download all 736 (or whatever) builds, in batches of 20 at a time. It seems there's a point at which this just falls over. For the OpenSSL 4 update I hacked up a dumb workaround: just [run the tests on the update's side tag](https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/commit/cae3eb0f4373874013bfce9ecc5c19bdf621ee73). And it mostly works, with a few follow-ups. So I was thinking: maybe we clean that up a bit and use it for all large (>100?) updates which are created from a side tag (which ought to be basically all of them). The main potential issue is changes in the side tag content during the test lifecycle. The side tag content can be changed at any time, and refreshing the update content is a *separate* action, so at almost any time, what's in the side tag and what's in the update may not match. It can even change *during the execution of a single test*, so when we update from the side tag early in the process and when we check what's "in the update" at the end of the test we might see different packages. This is a problem. My initial idea is to get the current *revision* of the side tag at the time we schedule the tests, and tie them to that, rather than using the `/latest` directory. This avoids the problem of the content changing during the test run. But I'm not sure if it's possible to be sure that the current state of the *side tag repo* at the time we trigger update tests (which is when the update is created or its package contents change) is the same as the content of the *update*. I don't know if the side tag repo is reliably regenerated every time a build is run, or every time the maintainer clicks "create update" or "save" or whatever in Bodhi. If not, the idea might be a non-starter. I'll have to look into that.