infra/tickets

Fork 1

Project idea: update/build repo server #13476

New issue

Open

opened 2026-07-29 22:59:43 +00:00 by adamwill · 8 comments

adamwill commented

2026-07-29 22:59:43 +00:00

Member

Description of request

Over in https://github.com/fedora-ci/rmdepcheck-pipeline/issues/11 , @lecris and I are discussing an idea I had around how test systems access packages to test.

Right now, both openQA and Fedora CI have the test pipelines download all the packages in the thing under test (usually a Bodhi update). In openQA, every single update test does this, so we waste quite a lot of network traffic. I believe each Fedora CI Jenkins pipeline also downloads the packages separately.

My idea is to build a thing I'm thinking of as the "test assets server". This would actually be a couple of things: a message consumer and a web server (just an instance of apache or nginx or whatever), running on the same system (or as a couple of containers with a shared filesystem, whatever). They would need access to a local mount of the fedora_koji NFS share.

The message consumer would listen out for bodhi.update.status.testing.koji-build-group.build.complete messages (which are published when an update is created, or the builds in it are changed) and buildsys.build.state.change messages (which are published when a Koji build completes). In either case it would get the NVR(s), find the directory in the fedora_koji NFS share, and then for each arch present in the directory, symlink the packages from that arch, plus noarch packages if any, to a path like /var/www/tas/updates/<advisory>/<arch> for updates, and /var/www/tas/builds/<nvr>/<arch> for builds. It would then run createrepo on the directory.

That's all the message consumer does. The web server just serves out whatever is under /var/www/tas.

So a test system - or anyone else, for that matter - who wants to test update FEDORA-2026-abcd on s390x can simply define a repo with the URL https://tas.fedoraproject.org/updates/FEDORA-2026-abcd/s390x and it gets the right bits. Only packages that are actually needed for the test are downloaded, and different test systems don't have to have their own code for downloading packages (or try and share it). Testing Farm could put a caching proxy in front of this service, I guess, to reduce external network trips for multiple tests testing the same assets.

I have actually written the consumer already (but not tested it) - attached as a text file for now. It doesn't seem very complicated. Wanted to file this ticket to record the idea and see if anybody has any excellent reasons why it's a terrible idea before I move forward, and also for requesting the necessary resources I guess. Not sure if it's feasible to get openshift containers with filesystem access to the koji NFS share? If not this could run on a small VM or something I suppose.

### Description of request Over in https://github.com/fedora-ci/rmdepcheck-pipeline/issues/11 , @lecris and I are discussing an idea I had around how test systems access packages to test. Right now, both openQA and Fedora CI have the test pipelines download all the packages in the thing under test (usually a Bodhi update). In openQA, every single update test does this, so we waste quite a lot of network traffic. I believe each Fedora CI Jenkins pipeline also downloads the packages separately. My idea is to build a thing I'm thinking of as the "test assets server". This would actually be a couple of things: a message consumer and a web server (just an instance of apache or nginx or whatever), running on the same system (or as a couple of containers with a shared filesystem, whatever). They would need access to a local mount of the fedora_koji NFS share. The message consumer would listen out for `bodhi.update.status.testing.koji-build-group.build.complete` messages (which are published when an update is created, or the builds in it are changed) and `buildsys.build.state.change` messages (which are published when a Koji build completes). In either case it would get the NVR(s), find the directory in the fedora_koji NFS share, and then for each arch present in the directory, symlink the packages from that arch, plus noarch packages if any, to a path like `/var/www/tas/updates/<advisory>/<arch>` for updates, and `/var/www/tas/builds/<nvr>/<arch>` for builds. It would then run `createrepo` on the directory. That's all the message consumer does. The web server just serves out whatever is under `/var/www/tas`. So a test system - or anyone else, for that matter - who wants to test update FEDORA-2026-abcd on s390x can simply define a repo with the URL `https://tas.fedoraproject.org/updates/FEDORA-2026-abcd/s390x` and it gets the right bits. Only packages that are actually needed for the test are downloaded, and different test systems don't have to have their own code for downloading packages (or try and share it). Testing Farm could put a caching proxy in front of this service, I guess, to reduce external network trips for multiple tests testing the same assets. I have actually written the consumer already (but not tested it) - attached as a text file for now. It doesn't seem very complicated. Wanted to file this ticket to record the idea and see if anybody has any excellent reasons why it's a terrible idea before I move forward, and also for requesting the necessary resources I guess. Not sure if it's feasible to get openshift containers with filesystem access to the koji NFS share? If not this could run on a small VM or something I suppose.

tassync.txt

3 KiB

adamwill commented

2026-07-29 23:00:51 +00:00

Author

Member

As well as being more efficient, this solves an awkward problem for us to do with packages that are not built on certain arches but do generate noarch subpackages - see https://github.com/fedora-ci/rmdepcheck-pipeline/issues/11#issuecomment-5121109077 for more on that.

As well as being more efficient, this solves an awkward problem for us to do with packages that are not built on certain arches but *do* generate noarch subpackages - see https://github.com/fedora-ci/rmdepcheck-pipeline/issues/11#issuecomment-5121109077 for more on that.

adamwill commented

2026-07-29 23:06:38 +00:00

Author

Member

Hmm, the koji builds side needs a bit more thought as I intended that for testing scratch builds, but those don't actually require unique NVRs, don't publish the same kinds of messages, and aren't stored in the same way in the koji dir. But it should be workable. Make it /tasks/<taskid>/<arch> or something. We can figure out the arches from the logs dir. It all looks workable. Gotta figure out the fedmsg side.

Hmm, the koji builds side needs a bit more thought as I intended that for testing scratch builds, but those don't actually require unique NVRs, don't publish the same kinds of messages, and aren't stored in the same way in the koji dir. But it should be workable. Make it `/tasks/<taskid>/<arch>` or something. We can figure out the arches from the logs dir. It all looks workable. Gotta figure out the fedmsg side.

adamwill commented

2026-07-30 00:07:45 +00:00

Author

Member

ehh, something like this I guess.

tassync.txt

3.7 KiB

adamwill commented

2026-07-30 00:34:47 +00:00

Author

Member

Hmm, actually, could ditch the message triggering and make it on-demand instead. Have a very small webapp which you interact with a bit like koji wait-repo - you send it a request like "I want a repo for update X for arch Y" and wait. It checks if the repo is already set up and returns the URL if so. If not it does all the symlinking stuff then returns the URL. The actual repos would still be served by a regular webserver instance I think...the apache/nginx/whatever config would proxy the specific request URL to the little webapp and serve everything else out of the directory tree...

that avoids processing things we're never going to test...and also means we can do garbage collection every couple of weeks, or month, and if we need to test something old, no problem, it'll just get set up again...hmm, but then, not sure how to handle edits of updates with this approach...mmf. more thinking required.

Hmm, actually, could ditch the message triggering and make it on-demand instead. Have a very small webapp which you interact with a bit like `koji wait-repo` - you send it a request like "I want a repo for update X for arch Y" and wait. It checks if the repo is already set up and returns the URL if so. If not it does all the symlinking stuff then returns the URL. The actual repos would still be served by a regular webserver instance I think...the apache/nginx/whatever config would proxy the specific request URL to the little webapp and serve everything else out of the directory tree... that avoids processing things we're never going to test...and also means we can do garbage collection every couple of weeks, or month, and if we need to test something old, no problem, it'll just get set up again...hmm, but then, not sure how to handle edits of updates with this approach...mmf. more thinking required.

gotmax23 commented

2026-07-30 03:01:34 +00:00

but then, not sure how to handle edits of updates with this approach

What about a hybrid approach where you have a message consumer solely to sync changes for cached repos of updates that were previously requested on demand and already exist in the cache?

> but then, not sure how to handle edits of updates with this approach What about a hybrid approach where you have a message consumer solely to sync changes for cached repos of updates that were previously requested on demand and already exist in the cache?

adamwill commented

2026-07-30 06:24:18 +00:00

Author

Member

yeah, I thought about that, starts to feel like a lot of bits though. the other I figured is, the 'on request' mode could just poke Bodhi API for the update and check whether it was edited since the last time it updated the cache. that does mean an API roundtrip on every request, not sure if that's a volume of traffic we need to worry about...

smoliicek added the

Needs investigation

label

2026-07-30 19:01:56 +00:00

kevin commented

2026-07-31 18:07:06 +00:00

Owner

So, this worries me a bit as even though it's just repodata it could end up being a LOT of repodata...

It's worth noting that when you download from kojipkgs already thats cached in varnish. But of course large package sets could be outside that.

Scratch builds and single non side tag rawhide builds are all just downloading one thing right? Does handling them here get too much gain?

Larger / side tag ones I could see the advantage. I wonder if we could leverage koji's dist repos here? But that might be a lot if there's a lot of sidetag updates, perhaps just large ones (whatever that means).

So, this worries me a bit as even though it's just repodata it could end up being a LOT of repodata... It's worth noting that when you download from kojipkgs already thats cached in varnish. But of course large package sets could be outside that. Scratch builds and single non side tag rawhide builds are all just downloading one thing right? Does handling them here get too much gain? Larger / side tag ones I could see the advantage. I wonder if we could leverage koji's dist repos here? But that might be a lot if there's a lot of sidetag updates, perhaps just large ones (whatever that means).

adamwill commented

2026-08-01 05:54:31 +00:00

Author

Member

So, this worries me a bit as even though it's just repodata it could end up being a LOT of repodata...

Yeah, indeed. It can be garbage collected frequently if we go with the 'request' design though, I guess.

Scratch builds and single non side tag rawhide builds are all just downloading one thing right? Does handling them here get too much gain?

It's one source package, but in some cases, one source package can produce many binary packages, not all of which the tests necessarily need to download. And rmdepcheck never actually needs to download any packages - it only needs repodata.

Larger / side tag ones I could see the advantage. I wonder if we could leverage koji's dist repos here? But that might be a lot if there's a lot of sidetag updates, perhaps just large ones (whatever that means).

I'd like to avoid having two processes if possible; there's not much benefit to this if openQA and Fedora CI both have to keep their existing downloaders for "small" updates. But I'll think it over some more.

> So, this worries me a bit as even though it's just repodata it could end up being a LOT of repodata... Yeah, indeed. It can be garbage collected frequently if we go with the 'request' design though, I guess. > Scratch builds and single non side tag rawhide builds are all just downloading one thing right? Does handling them here get too much gain? It's one *source* package, but in some cases, one source package can produce many binary packages, not all of which the tests necessarily need to download. And rmdepcheck never actually needs to download any packages - it only needs repodata. > Larger / side tag ones I could see the advantage. I wonder if we could leverage koji's dist repos here? But that might be a lot if there's a lot of sidetag updates, perhaps just large ones (whatever that means). I'd like to avoid having two processes if possible; there's not much benefit to this if openQA and Fedora CI both have to keep their existing downloaders for "small" updates. But I'll think it over some more.