infra/tickets

Fork 1

Reserving a machine for koji `ci` channel #13069

New issue

Open

opened 2026-01-22 08:59:11 +00:00 by lecris · 9 comments

lecris commented

2026-01-22 08:59:11 +00:00

Description of request

During the mass rebuild, the default channel is occupied by the rebuilds for the whole week, often leading to deprioritized jobs to end up with a multi-day scratch-build duration. On Fedora CI this means that PR jobs are effectively down this whole time since it cannot support that long of a queue. Things may improve with packit and forgejo, but still it is not a good experience for the whole PR stack to be down completely.

The question here if it is possible to reserve a machine or a few slots which would have the ci channel priority increased compared to the default, and at least part of the queue is moving forward?

I have tried to bring this topic to rel-eng also, but did not get any feedback, so maybe they can be pinged here to try and get some feedback again in an async discussion.

### Description of request During the mass rebuild, the default channel is occupied by the rebuilds for the whole week, often leading to deprioritized jobs to end up with a multi-day scratch-build duration. On Fedora CI this means that PR jobs are effectively down this whole time since it cannot support that long of a queue. Things may improve with packit and forgejo, but still it is not a good experience for the whole PR stack to be down completely. The question here if it is possible to reserve a machine or a few slots which would have the `ci` channel priority increased compared to the default, and at least part of the queue is moving forward? I have tried to bring this topic to rel-eng also, but did not get any feedback, so maybe they can be pinged here to try and get some feedback again in an async discussion.

zlopez added the

Gain

Medium

medium-trouble

labels

2026-01-22 10:02:42 +00:00

kevin added the

points

Priority

Medium

labels

2026-01-22 23:10:35 +00:00

kevin commented

2026-01-22 23:10:43 +00:00

Owner

well, for x86_64 and aarch64 that wouldn't be a big deal. For ppc64le it's less great, and for s390x it's... not good because we have few of them and want to use them all if we can.

Even if we assign a machine per arch for just ci, it would be pretty slow still... all it takes is a few llvm's and then ugh.

I suppose we could look at making ci jobs slightly higher pri than mass rebuild jobs? That would slow the mass rebuild down, but then ci jobs would still happen. What pri do they current set?

well, for x86_64 and aarch64 that wouldn't be a big deal. For ppc64le it's less great, and for s390x it's... not good because we have few of them and want to use them all if we can. Even if we assign a machine per arch for just ci, it would be pretty slow still... all it takes is a few llvm's and then ugh. I suppose we could look at making ci jobs slightly higher pri than mass rebuild jobs? That would slow the mass rebuild down, but then ci jobs would still happen. What pri do they current set?

kevin removed the

Gain

Medium

medium-trouble

labels

2026-01-22 23:10:45 +00:00

lecris commented

2026-01-23 07:57:42 +00:00

Author

@kevin wrote in #13069 (comment):

What pri do they current set?

Iirc it is something like 23 vs 24.

I suppose we could look at making ci jobs slightly higher pri than mass rebuild jobs? That would slow the mass rebuild down, but then ci jobs would still happen.

I would prefer not to slow down mass rebuild (at least as much as possible), what options could we consider? Maybe we can add a throttle on Fedora CI side, but doing that in jenkins might be quite more effort, so maybe in the packit variant.

Even if we assign a machine per arch for just ci, it would be pretty slow still... all it takes is a few llvm's and then ugh.

Indeed could 🤞that those teams take mercy on the infra during that time, but python was having PRs then also, so probably not.

Other suggestions of how to improve UX are also welcome of course. I was considering making it more visible on packit side the state of these rebuilds, or having a warning banner in the src.fp.o about the state (I thought we do, but did not see it in this mass rebuild?)

@kevin wrote in https://forge.fedoraproject.org/infra/tickets/issues/13069#issuecomment-343070: > What pri do they current set? Iirc it is something like 23 vs 24. > I suppose we could look at making ci jobs slightly higher pri than mass rebuild jobs? That would slow the mass rebuild down, but then ci jobs would still happen. I would prefer not to slow down mass rebuild (at least as much as possible), what options could we consider? Maybe we can add a throttle on Fedora CI side, but doing that in jenkins might be quite more effort, so maybe in the packit variant. > Even if we assign a machine per arch for just ci, it would be pretty slow still... all it takes is a few llvm's and then ugh. Indeed could 🤞that those teams take mercy on the infra during that time, but python was having PRs then also, so probably not. Other suggestions of how to improve UX are also welcome of course. I was considering making it more visible on packit side the state of these rebuilds, or having a warning banner in the src.fp.o about the state (I thought we do, but did not see it in this mass rebuild?)

kevin commented

2026-04-16 16:00:22 +00:00

Owner

So, I am not sure what we can do here. I haven't come up with any brilliant ideas.

To consider though, mass rebuilds have been taking like 3-4 days and 2 of those are weekend days (we plan to start tuesday, but all recent attempts have been delayed until wed or thursday), so the length of time is not too large. Perhaps this isn't worth trying to optimize for?

So, I am not sure what we can do here. I haven't come up with any brilliant ideas. To consider though, mass rebuilds have been taking like 3-4 days and 2 of those are weekend days (we plan to start tuesday, but all recent attempts have been delayed until wed or thursday), so the length of time is not too large. Perhaps this isn't worth trying to optimize for?

ngompa commented

2026-04-16 16:33:15 +00:00

It might be worth just kicking out everything but x86_64 and aarch64 from the CI channel until we have the proper size of machine resources.

lecris commented

2026-04-16 18:15:04 +00:00

Author

Do note that ci channel has almost no consumers anymore (only 1 test that runs when one of 16 packages is updated). The triple scratch build is over now, but packit is not part of the ci channel right now.

Also, if https://github.com/fedora-infra/koji-fedoramessaging/issues/155 can be resolved, everything would be fine with any approach

Do note that `ci` channel has almost no consumers anymore (only 1 test that runs when one of 16 packages is updated). The triple scratch build is over now, but packit is not part of the `ci` channel right now. Also, if https://github.com/fedora-infra/koji-fedoramessaging/issues/155 can be resolved, everything would be fine with any approach

kevin commented

2026-04-16 22:37:04 +00:00

Owner

Yeah, I don't think it's good to add packit to the ci channel because it's not just doing ci, but also doing real builds when maintainers tell it to, right?

Yeah, I don't know how feasable that is to do... I think currently it waits for the build to complete before sending anything, but perhaps @abompard could look?

Yeah, I don't think it's good to add packit to the ci channel because it's not just doing ci, but also doing real builds when maintainers tell it to, right? Yeah, I don't know how feasable that is to do... I think currently it waits for the build to complete before sending anything, but perhaps @abompard could look?

ngompa commented

2026-04-16 23:05:34 +00:00

Splitting the packit user might help with that? packit-ci vs packit

lecris commented

2026-04-17 11:29:26 +00:00

Author

Yeah, I don't think it's good to add packit to the ci channel because it's not just doing ci, but also doing real builds when maintainers tell it to, right?

When I last checked it, we can combine the conditions of user: packit and type: scratch-build (not exact quotes, but you get the idea)

> Yeah, I don't think it's good to add packit to the ci channel because it's not just doing ci, but also doing real builds when maintainers tell it to, right? When I last checked it, we can combine the conditions of `user: packit` and `type: scratch-build` (not exact quotes, but you get the idea)

kevin referenced this issue from a commit

2026-04-17 18:31:01 +00:00

koji hub: move packit scratch builds to ci channel

kevin referenced this issue from infra/ansible

2026-04-17 18:31:07 +00:00

koji hub: move packit scratch builds to ci channel #3295

kevin commented

2026-04-17 18:31:24 +00:00

Owner

yeah, thats likely true. I could look at that after the freeze is over.

infra/ansible#3295

should do it I think.

yeah, thats likely true. I could look at that after the freeze is over. https://forge.fedoraproject.org/infra/ansible/pulls/3295 should do it I think.

kevin referenced this issue from a commit

2026-04-29 21:48:32 +00:00

koji hub: move packit scratch builds to ci channel