Consider allowing draft builds for ELN

sgallagh commented

2026-06-17 00:35:07 +00:00

Owner

@kevin asked me to bring this to FESCo. I'm requesting that we allow the use of Draft Builds in the ELN rebuild process to help us handle situations where the ELNBuildSync (EBS) service crashes or the cluster it's running on otherwise terminates it while it's in the middle of batch processing. The technical details are in the Infrastructure ticket, but I'll reiterate what I'm asking for here:

The use of draft builds to be sanctioned for ELN build batches while they are in-progress.
Draft builds be permitted for tagging in side-tags, but not official tags. (If the tag policy permits it, we can limit them more specifically to side-tags derived from eln-build).
"Promotion" of draft builds is permitted by the eln-buildsync user. (From testing I did today, this is already technically possible, but I'd prefer to have it also be sanctioned.)
A garbage collection policy to be agreed-upon with Infrastructure for abandoned draft builds.

The purpose of this would be for EBS to perform its batches as draft builds into a side-tag. Once the batch is complete, we would promote the drafts to real builds and submit a side-tag to Bodhi as an erratum. If the batch is interrupted (crash, pod reboot, etc.) the draft builds would just be left behind to be garbage-collected in a reasonable timeframe.

Draft Builds are already available on the version of Koji in production, but the current policy is essentially "no draft build may be tagged anywhere", so they don't work for our needs, which is to have them tagged into a side-tag for the benefit of subsequent builds in that tag.

I can submit this as a Change Proposal if FESCo would prefer, but as it's very ELN-centric, I'm starting with just a FESCo ticket.

If needed, I wrote up an overview of how our builds are handled today.

@kevin [asked me](https://forge.fedoraproject.org/releng/tickets/issues/13374#issuecomment-794942) to bring this to FESCo. I'm requesting that we allow the use of Draft Builds in the ELN rebuild process to help us handle situations where the ELNBuildSync (EBS) service crashes or the cluster it's running on otherwise terminates it while it's in the middle of batch processing. The technical details are in the Infrastructure ticket, but I'll reiterate what I'm asking for here: * The use of draft builds to be sanctioned for ELN build batches while they are in-progress. * Draft builds be permitted for tagging in side-tags, but *not* official tags. (If the tag policy permits it, we can limit them more specifically to side-tags derived from `eln-build`). * "Promotion" of draft builds is permitted by the `eln-buildsync` user. (From testing I did today, this is already technically possible, but I'd prefer to have it also be sanctioned.) * A garbage collection policy to be agreed-upon with Infrastructure for abandoned draft builds. The purpose of this would be for EBS to perform its batches as draft builds into a side-tag. Once the batch is complete, we would promote the drafts to real builds and submit a side-tag to Bodhi as an erratum. If the batch is interrupted (crash, pod reboot, etc.) the draft builds would just be left behind to be garbage-collected in a reasonable timeframe. Draft Builds are already available on the version of Koji in production, but the current policy is essentially "no draft build may be tagged anywhere", so they don't work for our needs, which is to have them tagged into a side-tag for the benefit of subsequent builds in that tag. I can submit this as a Change Proposal if FESCo would prefer, but as it's very ELN-centric, I'm starting with just a FESCo ticket. If needed, I wrote up an [overview](https://sgallagh.wordpress.com/2026/05/07/sausage-factory-fedora-eln-rebuild-strategy-2026-edition/) of how our builds are handled today.

gotmax23 commented

2026-06-17 07:15:28 +00:00

Owner

Thanks for the writeup! This part from the Infra ticket concerns me a bit.

The reason for this is that right now, ELNBuildSync does not have a great solution for resuming after a crash. [...] So if we could perform all of our builds as Draft Builds and then only turn them into a real build that consumes the NVR right as we are submitting them to Bodhi, that would mean that on a crash, we could just start a fresh batch and not have to worry about some of the builds already having completed.

I worry that this will just put more strain on our build resources, which is already an ongoing issue, especially for certain architectures. If EBS crashes after building 999/1000 packages in a build batch, it would be ideal if it didn't just restart and build all those packages again. But if EBS crashes are very infrequent, maybe this is fine. How difficult would it be for EBS to properly store its state so it can resume building a batch after a crash?

Thanks for the writeup! This part from the Infra ticket concerns me a bit. > The reason for this is that right now, ELNBuildSync does not have a great solution for resuming after a crash. [...] So if we could perform all of our builds as Draft Builds and then only turn them into a real build that consumes the NVR right as we are submitting them to Bodhi, that would mean that on a crash, we could just start a fresh batch and not have to worry about some of the builds already having completed. I worry that this will just put more strain on our build resources, which is already an ongoing issue, especially for certain architectures. If EBS crashes after building 999/1000 packages in a build batch, it would be ideal if it didn't just restart and build all those packages again. But if EBS crashes are very infrequent, maybe this is fine. How difficult would it be for EBS to properly store its state so it can resume building a batch after a crash?

sgallagh commented

2026-06-17 13:31:51 +00:00

Author

Owner

Thanks for the writeup! This part from the Infra ticket concerns me a bit.

The reason for this is that right now, ELNBuildSync does not have a great solution for resuming after a crash. [...] So if we could perform all of our builds as Draft Builds and then only turn them into a real build that consumes the NVR right as we are submitting them to Bodhi, that would mean that on a crash, we could just start a fresh batch and not have to worry about some of the builds already having completed.

I worry that this will just put more strain on our build resources, which is already an ongoing issue, especially for certain architectures. If EBS crashes after building 999/1000 packages in a build batch, it would be ideal if it didn't just restart and build all those packages again.

Yes, the worst-case scenario is quite bad. In theory, we could crash just before submitting the Bodhi updates for a mass-rebuild and effectively need to re-run the entire batch. That would be painful.

However, looking at https://bodhi.fedoraproject.org/users/eln-buildsync it looks like the overwhelming majority of batches contain fewer than ten packages (with the exceptions being big side-tag merges and mass-rebuilds).

But if EBS crashes are very infrequent, maybe this is fine.

EBS has crashed twice in 2026 so far as I can recall. We've had a few outages for upgrades, but in those instances we pause processing so it doesn't start any new batches and wait for any in-progress ones to conclude before we shut things down.

(That's actually a secondary benefit here: if we needed to perform emergency maintenance on EBS, such as for a critical security flaw, this would also allow us to do that immediately without awaiting the conclusion of the current batch.)

How difficult would it be for EBS to properly store its state so it can resume building a batch after a crash?

It turns out to be extremely difficult. It's not impossible, but I strongly feel that it is not at all worth the effort to implement an exceptionally complex and possibly fragile resumption strategy to mitigate a problem that occurs infrequently.

Let's take the example of a batch and see the possible states that an affected package could be in.

An AMQP message has signaled to EBS that it needs to rebuild package foo. It is added to the queue of packages to put in a batch, but the batch has not yet started.
A batch begins and a side-tag is requested. We are awaiting the completion of the side-tag creation and initial tagging.
The side-tag is available. The batch divides up the packages into one or more "slices" that indicate certain packages that must build before others. We need to know which slice was in progress.
In each slice, we may have multiple "attempts" for each package. We need to determine which attempt is the latest for each task (if any have been submitted yet) and learn if any of them are still in progress or if any of them completed/failed while the service was down. This will be round-trips with Koji.
If any of them failed, we need to see if they need to be retried or if the slice is now complete.
If all attempts and slices are complete, we need to see if the update side-tag has been created.
If the update side-tag is created, we need to check if it has all of the successful builds tagged into it correctly.
Or maybe all of that happened and we just crashed after submitting to Bodhi but before we detected that the packages went stable by checking the tag events.
(This description leaves out some places where we may also have to await repo regeneration, which adds more possible states.)

> Thanks for the writeup! This part from the Infra ticket concerns me a bit. > > > The reason for this is that right now, ELNBuildSync does not have a great solution for resuming after a crash. [...] So if we could perform all of our builds as Draft Builds and then only turn them into a real build that consumes the NVR right as we are submitting them to Bodhi, that would mean that on a crash, we could just start a fresh batch and not have to worry about some of the builds already having completed. > > I worry that this will just put more strain on our build resources, which is already an ongoing issue, especially for certain architectures. If EBS crashes after building 999/1000 packages in a build batch, it would be ideal if it didn't just restart and build all those packages again. Yes, the worst-case scenario is quite bad. In theory, we *could* crash just before submitting the Bodhi updates for a mass-rebuild and effectively need to re-run the entire batch. That would be painful. However, looking at https://bodhi.fedoraproject.org/users/eln-buildsync it looks like the overwhelming majority of batches contain fewer than ten packages (with the exceptions being big side-tag merges and mass-rebuilds). > But if EBS crashes are very infrequent, maybe this is fine. EBS has crashed twice in 2026 so far as I can recall. We've had a few outages for upgrades, but in those instances we pause processing so it doesn't start any new batches and wait for any in-progress ones to conclude before we shut things down. (That's actually a secondary benefit here: if we needed to perform emergency maintenance on EBS, such as for a critical security flaw, this would also allow us to do that immediately without awaiting the conclusion of the current batch.) > How difficult would it be for EBS to properly store its state so it can resume building a batch after a crash? It turns out to be extremely difficult. It's not impossible, but I strongly feel that it is not at all worth the effort to implement an exceptionally complex and possibly fragile resumption strategy to mitigate a problem that occurs infrequently. Let's take the example of a batch and see the possible states that an affected package could be in. * An AMQP message has signaled to EBS that it needs to rebuild package foo. It is added to the queue of packages to put in a batch, but the batch has not yet started. * A batch begins and a side-tag is requested. We are awaiting the completion of the side-tag creation and initial tagging. * The side-tag is available. The batch divides up the packages into one or more "slices" that indicate certain packages that must build before others. We need to know which slice was in progress. * In each slice, we may have multiple "attempts" for each package. We need to determine which attempt is the latest for each task (if any have been submitted yet) and learn if any of them are still in progress or if any of them completed/failed while the service was down. This will be round-trips with Koji. * If any of them failed, we need to see if they need to be retried or if the slice is now complete. * If all attempts and slices are complete, we need to see if the update side-tag has been created. * If the update side-tag is created, we need to check if it has all of the successful builds tagged into it correctly. * Or maybe all of that happened and we just crashed after submitting to Bodhi but before we detected that the packages went stable by checking the tag events. (This description leaves out some places where we may also have to await repo regeneration, which adds more possible states.)

sgallagh commented

2026-06-17 13:34:32 +00:00

Author

Owner

Also, ELN is offering to be a "canary in the coal mine" here with regards to draft builds: our use of them exercises some (but not all) of this feature in Fedora. If we want to explore more interesting uses of draft builds later on, such as merge request testing with draft builds promoted on merge, ELN will have already validated that basic functionality is working properly.

gotmax23 commented

2026-06-18 16:35:08 +00:00

Owner

Okay, thanks for the additional context. I'm amenable to allowing draft builds for this (assuming that there's not something else I'm missing here and Infra/Releng is on board), but it would be nice to do something to prevent the worse case scenarios that would tax the buildsystem for everyone, particularly:

(The one I already mentioned) A big batch (e.g., one after a large sidetag merge) crashes after most of the packages have already been built and then starts over.
The same thing gets rebuilt over and over because EBS keeps creating a batch and crashing and putting the tag message back in the queue and restarting and batching and crashing and so on.

I understand that figuring out how to auto resume from various possible failure states is difficult, but is implementing some sort of NEVRs seen check that stops processing if a certain build or batch of builds has crashed multiple times or a previously seen batch that crashed is very large more feasible?

uses of draft builds later on, such as merge request testing with draft builds promoted on merge

FWIW I'm not sure doing that is worth the complexity, since all the existing testing and gating Infra is already wired through Bodhi, but that's another conversation. In this case, though, it seems like draft builds will solve a concrete problem that ELN is having, so I'm more open to it.

Okay, thanks for the additional context. I'm amenable to allowing draft builds for this (assuming that there's not something else I'm missing here and Infra/Releng is on board), but it would be nice to do *something* to prevent the worse case scenarios that would tax the buildsystem for everyone, particularly: - (The one I already mentioned) A big batch (e.g., one after a large sidetag merge) crashes after most of the packages have already been built and then starts over. - The same thing gets rebuilt over and over because EBS keeps creating a batch and crashing and putting the tag message back in the queue and restarting and batching and crashing and so on. I understand that figuring out how to auto resume from various possible failure states is difficult, but is implementing some sort of NEVRs seen check that *stops* processing if a certain build or batch of builds has crashed multiple times or a previously seen batch that crashed is very large more feasible? > uses of draft builds later on, such as merge request testing with draft builds promoted on merge FWIW I'm not sure doing that is worth the complexity, since all the existing testing and gating Infra is already wired through Bodhi, but that's another conversation. In this case, though, it seems like draft builds will solve a concrete problem that ELN is having, so I'm more open to it.

decathorpe commented

2026-06-18 17:19:50 +00:00

Owner

I assume that with this use case, we'd never get into the situation where draft builds get invalidated, they'd only ever get promoted or the entire batch is discarded?

I assume that with this use case, we'd never get into the situation where draft builds get invalidated, they'd only ever get promoted *or* the entire batch is discarded?

sgallagh commented

2026-06-18 18:23:45 +00:00

Author

Owner

(The one I already mentioned) A big batch (e.g., one after a large sidetag merge) crashes after most of the packages have already been built and then starts over.

I'll be up-front about this: I'm not going to implement any special checking for this unless reality demonstrates that it's causing a real problem. I don't know what you'd expect for the fallback situation to be, either. Would we just skip every package from the batch? What if it's a really important one, like a Python version-bump rebuild?

I appreciate that rebuilding everything could mean a waste of CPU cycles and the equivalent power costs, but trying to design a solution for a case that should hopefully be a statistical blip would be a waste of human time. I think that's worse.

The same thing gets rebuilt over and over because EBS keeps creating a batch and crashing and putting the tag message back in the queue and restarting and batching and crashing and so on.

Similarly, I think this is overblown as well; notwithstanding the possibility that we suddenly started allowing emojis in package names, most crashes have historically been with places where we didn't adequately account for a timeout or unexpected 500 error when talking to Koji, Bodhi, etc.

If something like this were to happen, we can just ask Infra to delete a single row from the database table containing the pending builds (or lie and add a "completed_at" date on it) and EBS would stop seeing it. Realistically, if we were seeing that many errors, we'd temporarily disable EBS, fix whatever bug was causing the crash and then resume it.

I understand that figuring out how to auto resume from various possible failure states is difficult, but is implementing some sort of NEVRs seen check that stops processing if a certain build or batch of builds has crashed multiple times or a previously seen batch that crashed is very large more feasible?

I'm about 75% sure that abandoning a batch is going to cause more cleanup problems (requiring a human) than just retrying it will.

I assume that with this use case, we'd never get into the situation where draft builds get invalidated, they'd only ever get promoted or the entire batch is discarded?

I don't know what "invalidated" means in this context. Either a build would be draft for a few minutes/hours as the batch processes through and then every successful build would be promoted into a real build, or else EBS crashes and the batch would be essentially orphaned and cleaned up by garbage collection after some TBD time period.

If you are asking whether EBS would go looking for abandoned draft builds and mark them for deletion, that's something we might be able to do, but it's definitely not going to be high on my priority list unless we somehow got really unlucky and started accruing many abandoned batches.

> - (The one I already mentioned) A big batch (e.g., one after a large sidetag merge) crashes after most of the packages have already been built and then starts over. I'll be up-front about this: I'm not going to implement any special checking for this unless reality demonstrates that it's causing a real problem. I don't know what you'd expect for the fallback situation to be, either. Would we just skip every package from the batch? What if it's a really important one, like a Python version-bump rebuild? I appreciate that rebuilding everything could mean a waste of CPU cycles and the equivalent power costs, but trying to design a solution for a case that should hopefully be a statistical blip would be a waste of human time. I think that's worse. > - The same thing gets rebuilt over and over because EBS keeps creating a batch and crashing and putting the tag message back in the queue and restarting and batching and crashing and so on. Similarly, I think this is overblown as well; notwithstanding the possibility that we suddenly started allowing emojis in package names, most crashes have historically been with places where we didn't adequately account for a timeout or unexpected 500 error when talking to Koji, Bodhi, etc. If something like this were to happen, we can just ask Infra to delete a single row from the database table containing the pending builds (or lie and add a "completed_at" date on it) and EBS would stop seeing it. Realistically, if we were seeing that many errors, we'd temporarily disable EBS, fix whatever bug was causing the crash and then resume it. > I understand that figuring out how to auto resume from various possible failure states is difficult, but is implementing some sort of NEVRs seen check that *stops* processing if a certain build or batch of builds has crashed multiple times or a previously seen batch that crashed is very large more feasible? I'm about 75% sure that abandoning a batch is going to cause more cleanup problems (requiring a human) than just retrying it will. > I assume that with this use case, we'd never get into the situation where draft builds get invalidated, they'd only ever get promoted *or* the entire batch is discarded? I don't know what "invalidated" means in this context. Either a build would be draft for a few minutes/hours as the batch processes through and then every successful build would be promoted into a real build, or else EBS crashes and the batch would be essentially orphaned and cleaned up by garbage collection after some TBD time period. If you are asking whether EBS would go looking for abandoned draft builds and mark them for deletion, that's something we might be able to do, but it's definitely not going to be high on my priority list unless we somehow got really unlucky and started accruing many abandoned batches.

gotmax23 commented

2026-06-18 18:55:16 +00:00

Owner

I'm about 75% sure that abandoning a batch is going to cause more cleanup problems (requiring a human) than just retrying it will.

By skipping/abandoning a batch, do you mean ignoring it and moving on to the next batch? Because yeah, I guess then any later builds that depend on changes in the earlier builds could fail. But I meant having a check that stops processing of messages entirely until a human intervenes to prevent EBS from getting stuck in a failure state that wastes a lot of Koji resources.

I'm not going to push too hard for these things here since the EBS error rate is low and you obviously know more about EBS's architecture than I do, but I would feel more confident in this change if there was a fail-safe in place (or some movement on getting more builder resources in Koji).

> I'm about 75% sure that abandoning a batch is going to cause more cleanup problems (requiring a human) than just retrying it will. By skipping/abandoning a batch, do you mean ignoring it and moving on to the next batch? Because yeah, I guess then any later builds that depend on changes in the earlier builds could fail. But I meant having a check that stops processing of messages entirely until a human intervenes to prevent EBS from getting stuck in a failure state that wastes a lot of Koji resources. I'm not going to push too hard for these things here since the EBS error rate is low and you obviously know more about EBS's architecture than I do, but I would feel more confident in this change if there was a fail-safe in place (or some movement on getting more builder resources in Koji).

decathorpe commented

2026-06-18 22:21:17 +00:00

Owner

I assume that with this use case, we'd never get into the situation where draft builds get invalidated, they'd only ever get promoted or the entire batch is discarded?

I don't know what "invalidated" means in this context. Either a build would be draft for a few minutes/hours as the batch processes through and then every successful build would be promoted into a real build, or else EBS crashes and the batch would be essentially orphaned and cleaned up by garbage collection after some TBD time period.

Yes, this is what I meant. Apparently in CentOS Stream it's possible to do draft build A-1 from MR#1, draft build B-1 from MR#2, draft build A-2 from force-pushed PR#1, and then submit B-1 and A-2, even though B-1 was built against A-1, which never got promoted to un-draft build. I don't think this should be problem for ELN.

> > I assume that with this use case, we'd never get into the situation where draft builds get invalidated, they'd only ever get promoted *or* the entire batch is discarded? > > I don't know what "invalidated" means in this context. Either a build would be draft for a few minutes/hours as the batch processes through and then every successful build would be promoted into a real build, or else EBS crashes and the batch would be essentially orphaned and cleaned up by garbage collection after some TBD time period. Yes, this is what I meant. Apparently in CentOS Stream it's possible to do draft build A-1 from MR#1, draft build B-1 from MR#2, draft build A-2 from force-pushed PR#1, and then submit B-1 and A-2, even though B-1 was built against A-1, which never got promoted to un-draft build. I don't think this should be problem for ELN.

zbyszek commented

2026-06-21 10:17:04 +00:00

Owner

I'm requesting that we allow the use of Draft Builds in the ELN rebuild process

I think this is a great idea in general. There are interesting proposals for workflows using draft builds in Fedora proper, and using ELN as a test bed for draft build policies will give us useful experience and hopefully make it easier to decide whether and how we want to use draft builds in more places.

All four bullet points in the OP seem very reasonable.

Apparently in CentOS Stream it's possible to do draft build A-1 from MR#1, draft build B-1 from MR#2, draft build A-2 from force-pushed PR#1, and then submit B-1 and A-2, even though B-1 was built against A-1, which never got promoted to un-draft build.

That seems iffy. In particular, IIUC, it makes it impossible to do a reproducible build of B-1. So far our policy in koji was that "every build that was ever used to do another official build is available", and it's a property that we should preserve.

> I'm requesting that we allow the use of Draft Builds in the ELN rebuild process I think this is a great idea in general. There are interesting proposals for workflows using draft builds in Fedora proper, and using ELN as a test bed for draft build policies will give us useful experience and hopefully make it easier to decide whether and how we want to use draft builds in more places. All four bullet points in the OP seem very reasonable. > Apparently in CentOS Stream it's possible to do draft build A-1 from MR#1, draft build B-1 from MR#2, draft build A-2 from force-pushed PR#1, and then submit B-1 and A-2, even though B-1 was built against A-1, which never got promoted to un-draft build. That seems iffy. In particular, IIUC, it makes it impossible to do a reproducible build of B-1. So far our policy in koji was that "every build that was ever used to do another official build is available", and it's a property that we should preserve.

sgallagh commented

2026-06-22 13:34:56 +00:00

Author

Owner

Apparently in CentOS Stream it's possible to do draft build A-1 from MR#1, draft build B-1 from MR#2, draft build A-2 from force-pushed PR#1, and then submit B-1 and A-2, even though B-1 was built against A-1, which never got promoted to un-draft build.

That seems iffy. In particular, IIUC, it makes it impossible to do a reproducible build of B-1. So far our policy in koji was that "every build that was ever used to do another official build is available", and it's a property that we should preserve.

To be clear: this cannot happen in the EBS design (because everything in the buildroot is either from Rawhide or an earlier "slice" of this batch) and is therefore always submitted together. I urge people not to get sidetracked on this topic; I'd like to land the implementation of this proposal sooner, rather than later.

> > Apparently in CentOS Stream it's possible to do draft build A-1 from MR#1, draft build B-1 from MR#2, draft build A-2 from force-pushed PR#1, and then submit B-1 and A-2, even though B-1 was built against A-1, which never got promoted to un-draft build. > > That seems iffy. In particular, IIUC, it makes it impossible to do a reproducible build of B-1. So far our policy in koji was that "every build that was ever used to do another official build is available", and it's a property that we should preserve. To be clear: this *cannot* happen in the EBS design (because everything in the buildroot is either from Rawhide or an earlier "slice" of this batch) and is therefore always submitted together. I urge people not to get sidetracked on this topic; I'd like to land the [implementation of this proposal](https://github.com/fedora-eln/elnbuildsync/pull/37) sooner, rather than later.

decathorpe commented

2026-06-22 13:37:32 +00:00

Owner

I didn't want to side-track, I only wanted your confirmation of my suspicion that this issue couldn't happen with ELN builds - which you did. So I'm OK with this mechanism. +1

supakeen commented

2026-06-23 13:07:38 +00:00

Owner

I'm +1 for allowing draft builds for EBS batches to be tagged into ELN-inheriting side tags. If it's technically not feasible to be narrow I'm OK with that being 'any side-tag'.

gotmax23 commented

2026-06-23 18:53:24 +00:00

Owner

+1, it sounds like this is pretty unlikely to cause problems.

gotmax23 commented

2026-06-26 00:09:22 +00:00

Owner

~~After a week: APPROVED (+3, 0, 0)~~ (see below)

~~After a week: APPROVED (+3, 0, 0)~~ (see below)

gotmax23 added the

pending announcement

label

2026-06-26 00:09:36 +00:00

yselkowitz referenced this issue from releng/tickets

2026-06-26 00:11:43 +00:00

Consider allowing draft builds for ELN #13374

zbyszek commented

2026-06-26 06:48:49 +00:00

Owner

Hmmm, what exactly was approved? There was no specific text proposal, and the discussion was ongoing. It's not entirely clear to me that any formal voting was even happening.

sgallagh commented

2026-06-26 11:47:59 +00:00

Author

Owner

@zbyszek wrote in #3621 (comment):

Hmmm, what exactly was approved? There was no specific text proposal, and the discussion was ongoing. It's not entirely clear to me that any formal voting was even happening.

While I didn't write a literal "Proposal:" string, I thought that the four bullet points that I listed as "what I'm asking for" would have been sufficiently clear.

@zbyszek wrote in https://forge.fedoraproject.org/fesco/tickets/issues/3621#issuecomment-867736: > Hmmm, what exactly was approved? There was no specific text proposal, and the discussion was ongoing. It's not entirely clear to me that any formal voting was even happening. While I didn't write a literal "Proposal:" string, I thought that the four bullet points that I listed as "what I'm asking for" would have been sufficiently clear.

zbyszek commented

2026-06-26 15:05:31 +00:00

Owner

I didn't vote because there was no formal proposal. It sounded like an opening for a discussion.

@supakeen also had a "+1" in a comment, but apparently that didn't get counted. (And it probably shouldn't, because it's not clear that it is a vote for the proposal.)

I would prefer to restart this with a delineated proposal and a clear voting period.

I didn't vote because there was no formal proposal. It sounded like an opening for a discussion. @supakeen also had a "+1" in a comment, but apparently that didn't get counted. (And it probably shouldn't, because it's not clear that it is a vote for the proposal.) I would prefer to restart this with a delineated proposal and a clear voting period.

supakeen commented

2026-06-26 15:14:00 +00:00

Owner

My +1 was meant as a vote on "I'm requesting that we allow the use of Draft Builds in the ELN rebuild process" 🙂

gotmax23 commented

2026-06-26 15:36:36 +00:00

Owner

I counted +1s from @decathorpe, @supakeen, and @gotmax23 (myself). Did I miss someone else? I thought it was clear that we were voting on allowing draft builds in the ELN rebuild sidetags, following the steps in the bullet points in the ticket description. Since that wasn't clear to everyone (mea culpa!), then yes, I'll put forward a concrete proposal to vote on:

Proposal: FESCo permits the ELNBuildSync (EBS) service to use draft builds in Koji sidetags inherited from eln-build for the purpose of ELN rebuild batches. The Kojihub policy will be updated to grant the eln-buildsync user permissions to tag and promote the draft builds in these sidetags.

(@sgallagh, is that summarized proposal inline with what you're asking for?)

I counted `+1`s from @decathorpe, @supakeen, and @gotmax23 (myself). Did I miss someone else? I thought it was clear that we were voting on allowing draft builds in the ELN rebuild sidetags, following the steps in the bullet points in the ticket description. Since that wasn't clear to everyone (mea culpa!), then yes, I'll put forward a concrete proposal to vote on: **Proposal**: FESCo permits the ELNBuildSync (EBS) service to use draft builds in Koji sidetags inherited from `eln-build` for the purpose of ELN rebuild batches. The Kojihub policy will be updated to grant the `eln-buildsync` user permissions to tag and promote the draft builds in these sidetags. (@sgallagh, is that summarized proposal inline with what you're asking for?)

gotmax23 removed the

pending announcement

label

2026-06-26 15:37:43 +00:00

sgallagh commented

2026-06-26 16:25:38 +00:00

Author

Owner

Other than the open question of whether the tag policy is granular enough to limit it to ELN-derived side-tags, yes. I'd slightly prefer to have approval cover "any side tag" for the policy, just in case.

gotmax23 commented

2026-06-26 16:38:29 +00:00

Owner

Based on the policy example in https://docs.pagure.org/koji/draft_builds/#policies-for-draft-builds, it looks like it would be possible to limit this to the glob eln-build-side-*. I think that should satisfy the conditions in the proposal.

Based on the policy example in https://docs.pagure.org/koji/draft_builds/#policies-for-draft-builds, it looks like it would be possible to limit this to the glob `eln-build-side-*`. I think that should satisfy the conditions in the proposal.

zbyszek commented

2026-06-26 18:36:16 +00:00

Owner

+1

I counted +1s from @decathorpe, @supakeen, and @gotmax23

What about @sgallagh? (I actually don't remember our policy here. Proposals made in the meeting certainly count as +1 from the proposer. I'm not sure if that covers proposals made in the ticket too.)

+1 > I counted +1s from @decathorpe, @supakeen, and @gotmax23 What about @sgallagh? (I actually don't remember our policy here. Proposals made in the meeting certainly count as +1 from the proposer. I'm not sure if that covers proposals made in the ticket too.)

sgallagh commented

2026-06-26 19:13:57 +00:00

Author

Owner

I am not currently serving on FESCo, so my vote doesn't count.

kevin commented

2026-06-29 18:24:39 +00:00

Owner

I suppose this is ok. It seems like working around things instead of properly doing it, but I understand why that would be a lot of work.

We also do need to decide a GC policy for (unporomosted) draft builds. We don't want to keep them around forever, and by their nature there will possibly be more than 1 of them per real promoted build.

I suppose this is ok. It seems like working around things instead of properly doing it, but I understand why that would be a lot of work. We also do need to decide a GC policy for (unporomosted) draft builds. We don't want to keep them around forever, and by their nature there will possibly be more than 1 of them per real promoted build.

sgallagh commented

2026-06-29 19:24:25 +00:00

Author

Owner

We also do need to decide a GC policy for (unporomosted) draft builds. We don't want to keep them around forever, and by their nature there will possibly be more than 1 of them per real promoted build.

Absolutely. FWIW, I can't really see a need for us to keep them longer than two weeks, but I was planning to ask for 30 days out of an abundance of caution.

We try really hard to avoid crashes in EBS and I think we've had a grand total of two so far this calendar year (both due to infra issues that caused python-fedora-messaging to explode). The other situations where this could be involved is if we push out a service upgrade to EBS or the OpenShift node is terminated for some reason. In the case of EBS upgrades, we'd generally aim to "pause" activity and let any running batches complete first, so that shouldn't leave any drafts around.

So, if we assume a very paranoid rate of two crashes per month, a generous batch size of 20 builds (most are fewer than ten, but the average is pulled up by the occasional big side-tag merge) and an average of 10 MiB of data per package, we're looking at about 5 GiB worth of storage a year per year in leftover Draft Builds if we don't have a garbage-collection policy. That is almost certain to be a significant overestimation.

> We also do need to decide a GC policy for (unporomosted) draft builds. We don't want to keep them around forever, and by their nature there will possibly be more than 1 of them per real promoted build. Absolutely. FWIW, I can't really see a need for us to keep them longer than two weeks, but I was planning to ask for 30 days out of an abundance of caution. We try *really* hard to avoid crashes in EBS and I think we've had a grand total of two so far this calendar year (both due to infra issues that caused python-fedora-messaging to explode). The other situations where this could be involved is if we push out a service upgrade to EBS or the OpenShift node is terminated for some reason. In the case of EBS upgrades, we'd generally aim to "pause" activity and let any running batches complete first, so that shouldn't leave any drafts around. So, if we assume a very paranoid rate of two crashes per month, a generous batch size of 20 builds (most are fewer than ten, but the average is pulled up by the occasional big side-tag merge) and an average of 10 MiB of data per package, we're looking at about 5 GiB worth of storage a year per year in leftover Draft Builds if we don't have a garbage-collection policy. That is almost certain to be a significant overestimation.

gotmax23 added the

vote-in-progress

label

2026-06-30 18:57:17 +00:00

gotmax23 commented

2026-07-01 21:22:45 +00:00

Owner

@decathorpe, @supakeen, are you still +1 to the new proposal?

@decathorpe, @supakeen, are you still `+1` to the [new proposal](https://forge.fedoraproject.org/fesco/tickets/issues/3621#issuecomment-867805)?

ngompa commented

2026-07-01 21:28:03 +00:00

Owner

I'm +1 to the proposal, provided we can restrict it to just eln tags. If we can't, I'm worried we could have problems if it's accidentally allowed across all tags.

decathorpe commented

2026-07-01 22:20:31 +00:00

Owner

Still +1.

If it's not possible to implement this as written in koji hub policy, then doing the "next best" thing (within reason) is fine with me too.

Still +1. If it's not possible to implement this as written in koji hub policy, then doing the "next best" thing (within reason) is fine with me too.

supakeen commented

2026-07-02 03:45:34 +00:00

Owner

Yep, still +1.

gotmax23 commented

2026-07-03 16:00:44 +00:00

Owner

(counted supakeen, decathorpe, ngompa, gotmax23, zbyszek)

After one week: APPROVED (+5, 0, -0) FESCo permits the ELNBuildSync (EBS) service to use draft builds in Koji sidetags inherited from eln-build for the purpose of ELN rebuild batches. The Kojihub policy will be updated to grant the eln-buildsync user permissions to tag and promote the draft builds in these sidetags.

(counted supakeen, decathorpe, ngompa, gotmax23, zbyszek) After one week: APPROVED (+5, 0, -0) FESCo permits the ELNBuildSync (EBS) service to use draft builds in Koji sidetags inherited from `eln-build` for the purpose of ELN rebuild batches. The Kojihub policy will be updated to grant the `eln-buildsync` user permissions to tag and promote the draft builds in these sidetags.

gotmax23 added

pending announcement

and removed

vote-in-progress

labels

2026-07-03 16:00:51 +00:00