Consider allowing draft builds for ELN

sgallagh commented

2026-06-03 00:36:22 +00:00

Member

Describe the issue

In ELN, we have a double-side-tag process for creating a Bodhi update for each batch of ELN builds. Essentially, we create a first side-tag, tag all of the Rawhide builds into that buildroot, build the ELN packages, and then finally tag the successes into a new side-tag which is used to submit the Bodhi update (so we don't accidentally tag any of the Rawhide builds into ELN).

When I asked about Draft Builds today in #fedora-releng, @kevin told me "yes, you can do them today... however we have a policy that says "never allow any of these to be tagged into any tag" because there's no policy around using them."

What I'd like to be able to do is have the ELN builds in the first side-tag be built as draft builds and then "promote" them to full builds to tag them into the second side-tag for Bodhi update connection. The reason for this is that right now, ELNBuildSync does not have a great solution for resuming after a crash. I have been working on recording information in the database and logic that will let us resume from the middle of the process, but it's actually quite difficult to get right. Today I had the idea that we might actually not need to be able to fully resume, but instead could maybe just restart... so long as the NVRs haven't already been used up. So if we could perform all of our builds as Draft Builds and then only turn them into a real build that consumes the NVR right as we are submitting them to Bodhi, that would mean that on a crash, we could just start a fresh batch and not have to worry about some of the builds already having completed.

Can we use this ticket to discuss and work out a policy that would be acceptable for this (limited) use-case?

When do you need this? (YYYY/MM/DD)

The sooner, the better, but no strict deadline.

When is this no longer needed or useful? (YYYY/MM/DD)

N/A

If we cannot complete this, what is the impact? [Dependencies/Blocker]

See the description above.

Checklist

I have checked existing issues for duplicates
I have filled out all the fields above
I have provided relevant links or references (if applicable)

## Describe the issue  In ELN, we have a [double-side-tag process](https://sgallagh.wordpress.com/2026/05/07/sausage-factory-fedora-eln-rebuild-strategy-2026-edition/) for creating a Bodhi update for each batch of ELN builds. Essentially, we create a first side-tag, tag all of the Rawhide builds into that buildroot, build the ELN packages, and then finally tag the successes into a new side-tag which is used to submit the Bodhi update (so we don't accidentally tag any of the Rawhide builds into ELN). When I asked about Draft Builds today in #fedora-releng, @kevin told me "yes, you can do them today... however we have a policy that says "never allow any of these to be tagged into any tag" because there's no policy around using them." What I'd like to be able to do is have the ELN builds in the first side-tag be built as draft builds and then "promote" them to full builds to tag them into the second side-tag for Bodhi update connection. The reason for this is that right now, ELNBuildSync does not have a great solution for resuming after a crash. I have been working on recording information in the database and logic that will let us resume from the middle of the process, but it's actually quite difficult to get right. Today I had the idea that we might actually not need to be able to fully *resume*, but instead could maybe just restart... so long as the NVRs haven't already been used up. So if we could perform all of our builds as Draft Builds and then only turn them into a real build that consumes the NVR right as we are submitting them to Bodhi, that would mean that on a crash, we could just start a fresh batch and not have to worry about some of the builds already having completed. Can we use this ticket to discuss and work out a policy that would be acceptable for this (limited) use-case? ## When do you need this? (YYYY/MM/DD)  The sooner, the better, but no strict deadline. ## When is this no longer needed or useful? (YYYY/MM/DD)  N/A ## If we cannot complete this, what is the impact? [Dependencies/Blocker]  See the description above. --- ## Checklist - [X] I have checked existing issues for duplicates - [X] I have filled out all the fields above - [X] I have provided relevant links or references (if applicable)

kevin commented

2026-06-03 00:51:52 +00:00

Owner

I don't have any problem discussing this here/coming up with a plan, but whatever is proposed likely should get fesco approval.
(Since this is different from all the rest of our stuff).

It would need at least adjustments to garbage collection (how long do we keep them for), what tag(s) we allow them to be tagged into (although we would need to test and see if 'promoted' draft builds match a draft build policy, or if they are only regular builds then), and probibly other things I am not thinking of yet. ;)

I don't have any problem discussing this here/coming up with a plan, but whatever is proposed likely should get fesco approval. (Since this is different from all the rest of our stuff). It would need at least adjustments to garbage collection (how long do we keep them for), what tag(s) we allow them to be tagged into (although we would need to test and see if 'promoted' draft builds match a draft build policy, or if they are only regular builds then), and probibly other things I am not thinking of yet. ;)

sgallagh commented

2026-06-03 03:07:44 +00:00

Author

Member

@kevin wrote in #13374 (comment):

I don't have any problem discussing this here/coming up with a plan, but whatever is proposed likely should get fesco approval. (Since this is different from all the rest of our stuff).

Seems reasonable. Do you want me to file a Change Proposal for this?

It would need at least adjustments to garbage collection (how long do we keep them for),

I can't really think of any situation where we'd need to have a draft build for more than a week. If they weren't promoted to a proper build in that time, it should be safe to assume that it was created and then ELNBuildSync crashed, in which case it would have been rebuilt and this draft abandoned.

what tag(s) we allow them to be tagged into (although we would need to test and see if 'promoted' draft builds match a draft build policy, or if they are only regular builds then),

Presumably they have to be capable of getting tagged somehow today at least as the result of the build target they're built in, no? But at least in the case of ELNBuildSync, we would only need them to be tagged into side-tags derived from the eln-build tag. We'd need to be able to promote them within that side-tag and then submit that side-tag as a Bodhi update.

and probibly other things I am not thinking of yet. ;)

Sure, nothing ever goes exactly as planned.

@kevin wrote in https://forge.fedoraproject.org/releng/tickets/issues/13374#issuecomment-794942: > I don't have any problem discussing this here/coming up with a plan, but whatever is proposed likely should get fesco approval. (Since this is different from all the rest of our stuff). Seems reasonable. Do you want me to file a Change Proposal for this? > It would need at least adjustments to garbage collection (how long do we keep them for), I can't really think of any situation where we'd need to have a draft build for more than a week. If they weren't promoted to a proper build in that time, it should be safe to assume that it was created and then ELNBuildSync crashed, in which case it would have been rebuilt and this draft abandoned. > what tag(s) we allow them to be tagged into (although we would need to test and see if 'promoted' draft builds match a draft build policy, or if they are only regular builds then), Presumably they have to be capable of getting tagged *somehow* today at least as the result of the build target they're built in, no? But at least in the case of ELNBuildSync, we would only need them to be tagged into side-tags derived from the `eln-build` tag. We'd need to be able to promote them within that side-tag and then submit that side-tag as a Bodhi update. > and probibly other things I am not thinking of yet. ;) Sure, nothing ever goes exactly as planned.

patrikp added the

Backlog Status

Needs Review

label

2026-06-03 08:09:48 +00:00

patrikp added this to the Backlog project

2026-06-03 08:09:51 +00:00

kevin commented

2026-06-03 23:16:11 +00:00

Owner

@sgallagh wrote in #13374 (comment):

Seems reasonable. Do you want me to file a Change Proposal for this?

Naw. I don't think it needs that... Changes are tied to fedora releases and this is eln and seems more... outside that.

So, I'm trying to see where draft builds help. Couldn't you just on resume after a crash do the builds that are pending again and if they were successfull you catch the 'build already exists' from koji ?

It's very likely I am not thinking of all the places it could crash tho.

@sgallagh wrote in https://forge.fedoraproject.org/releng/tickets/issues/13374#issuecomment-794949: > > Seems reasonable. Do you want me to file a Change Proposal for this? Naw. I don't think it needs that... Changes are tied to fedora releases and this is eln and seems more... outside that. So, I'm trying to see where draft builds help. Couldn't you just on resume after a crash do the builds that are pending again and if they were successfull you catch the 'build already exists' from koji ? It's very likely I am not thinking of all the places it could crash tho.

sgallagh commented

2026-06-04 14:45:42 +00:00

Author

Member

"Build already exists" isn't an actual response we can get from a task, so far as I know. It would mean that for every task that fails, we also need to query Koji for the logs to see if the task_failed.log happens to have "Build already exists" in its output.

Beyond that, we don't actually know if that specific NVR already exists because it was built as part of the interrupted batch (and thus would need to be part of the Bodhi update) or already exists because the build in the current batch is genuinely trying to reuse an NVR that's already taken. With a draft build, it will fail with "build already exists" if the NVR is already taken and just ignore an earlier draft build if one occurred.

"Build already exists" isn't an actual response we can get from a task, so far as I know. It would mean that for every task that fails, we also need to query Koji for the logs to see if the task_failed.log happens to have "Build already exists" in its output. Beyond that, we don't actually know if that specific NVR already exists because it was built as part of the interrupted batch (and thus would need to be part of the Bodhi update) or already exists because the build in the current batch is genuinely trying to reuse an NVR that's already taken. With a draft build, it will fail with "build already exists" if the NVR is already taken and just ignore an earlier draft build if one occurred.

sgallagh commented

2026-06-04 14:50:55 +00:00

Author

Member

If we default to assuming that "build already exists" means that the NVR was taken prior to this batch, then we'll miss updates because we won't add them to Bodhi at the conclusion of this batch.

If we default to assuming "build already exists" means that the NVR was built in an earlier attempt at this current batch, we risk that if it WAS already taken, the Bodhi update including a reused build will fail and all of the packages in the batch won't get pushed to stable, causing us to miss even more updates.

If we default to assuming that "build already exists" means that the NVR was taken prior to this batch, then we'll miss updates because we won't add them to Bodhi at the conclusion of this batch. If we default to assuming "build already exists" means that the NVR was built in an earlier attempt at this current batch, we risk that if it WAS already taken, the Bodhi update including a reused build will fail and _all_ of the packages in the batch won't get pushed to stable, causing us to miss even more updates.

sgallagh commented

2026-06-04 20:57:56 +00:00

Author

Member

As far as a policy, I believe we only need the eln-buildsync user (and probably @sysadmin-eln) to have privileges for promotion/tagging. As far as tagging, we would need privilege to tag a draft build into a side tag (if permissions permit sufficient granularity, that could be restricted to side-tags derived from eln-build) and to promote a draft build in such a side tag into a full build.

I think a good policy in general would also be to disallow tagging of draft builds into any "release" tag. Manipulation of draft builds should be kept in side-tags and maybe purpose-built target tags in the future.

As far as a policy, I believe we only need the `eln-buildsync` user (and probably `@sysadmin-eln`) to have privileges for promotion/tagging. As far as tagging, we would need privilege to tag a draft build into a side tag (if permissions permit sufficient granularity, that could be restricted to side-tags derived from eln-build) and to promote a draft build in such a side tag into a full build. I think a good policy in general would also be to disallow tagging of draft builds into any "release" tag. Manipulation of draft builds should be kept in side-tags and maybe purpose-built target tags in the future.

ckelley commented

2026-06-04 21:24:48 +00:00

@kevin wrote in #13374 (comment):

It would need at least adjustments to garbage collection (how long do we keep them for), what tag(s) we allow them to be tagged into (although we would need to test and see if 'promoted' draft builds match a draft build policy, or if they are only regular builds then), and probibly other things I am not thinking of yet. ;)

We have been using draft builds as part of the CentOS Stream/RHEL gating pipeline for 2 years now, so we have some experience to draw on.

The key thing we try to avoid at all costs is having draft builds show up in "release" tags. This is done by setting the hub policy to only allow draft builds in "draft" tags, and side-tags. A build must be promoted within the draft/side-tag before the policy will allow it to be re-tagged. In addition, only one bot user is allowed to tag these builds into *-candidate tags. Even CI maintainers like us cannot do it, it is heavily locked down.

We also verify that the commit used to make the draft build has been merged into the MR target branch before we promote it, i.e. we insist that the commit is in dist-git. This prevents the possibility of draft builds being promoted from MR branches and making it into composes/releases while the source commit is not in dist-git, which would be very bad. This is not a problem for the proposed use case here as we are working exclusively with commits already merged, but it is something to think about for the future if you want to allow a draft build MR workflow in Fedora.

I don't recall off the top of my head what the garbage collection policy is for draft builds but one exists. I suggested a feature at one point that the promotion process could find all the un-promoted drafts for that NVR and GC them, as they have no future as the NVR was consumed, but I don't know if it was ever implemented.

For the record, I am not on the teams that take care of the hub policies so I cannot provide more detail on how it works behind the scenes - we are consumers. :-) I can of course put you in contact with the people who did implement it if you want to know more.

@kevin wrote in https://forge.fedoraproject.org/releng/tickets/issues/13374#issuecomment-794942: > It would need at least adjustments to garbage collection (how long do we keep them for), what tag(s) we allow them to be tagged into (although we would need to test and see if 'promoted' draft builds match a draft build policy, or if they are only regular builds then), and probibly other things I am not thinking of yet. ;) We have been using draft builds as part of the CentOS Stream/RHEL gating pipeline for 2 years now, so we have some experience to draw on. The key thing we try to avoid at all costs is having draft builds show up in "release" tags. This is done by setting the hub policy to only allow draft builds in "draft" tags, and side-tags. A build _must_ be promoted within the draft/side-tag before the policy will allow it to be re-tagged. In addition, only one bot user is allowed to tag these builds into *-candidate tags. Even CI maintainers like us cannot do it, it is heavily locked down. We also verify that the commit used to make the draft build has been merged into the MR target branch before we promote it, i.e. we insist that the commit is in dist-git. This prevents the possibility of draft builds being promoted from MR branches and making it into composes/releases while the source commit is not in dist-git, which would be very bad. This is not a problem for the proposed use case here as we are working exclusively with commits already merged, but it is something to think about for the future if you want to allow a draft build MR workflow in Fedora. I don't recall off the top of my head what the garbage collection policy is for draft builds but one exists. I suggested a feature at one point that the promotion process could find all the un-promoted drafts for that NVR and GC them, as they have no future as the NVR was consumed, but I don't know if it was ever implemented. For the record, I am not on the teams that take care of the hub policies so I cannot provide more detail on how it works behind the scenes - we are consumers. :-) I can of course put you in contact with the people who did implement it if you want to know more.

yselkowitz commented

2026-06-26 00:11:43 +00:00

Member

fesco/tickets#3621 was approved.

https://forge.fedoraproject.org/fesco/tickets/issues/3621 was approved.

gotmax23 commented

2026-06-26 15:39:47 +00:00

Member

I'm sorry, there was confusion in the ticket, and it is not formally approved.

yselkowitz commented

2026-07-03 16:25:42 +00:00

Member

take two:

After one week: APPROVED (+5, 0, -0) FESCo permits the ELNBuildSync (EBS) service to use draft builds in Koji sidetags inherited from eln-build for the purpose of ELN rebuild batches. The Kojihub policy will be updated to grant the eln-buildsync user permissions to tag and promote the draft builds in these sidetags.

take two: > After one week: APPROVED (+5, 0, -0) FESCo permits the ELNBuildSync (EBS) service to use draft builds in Koji sidetags inherited from` eln-build` for the purpose of ELN rebuild batches. The Kojihub policy will be updated to grant the `eln-buildsync` user permissions to tag and promote the draft builds in these sidetags.

sgallagh commented

2026-07-06 15:39:26 +00:00

Author

Member

With that approval, I'd like to request that the staging and production environments be set to allow the tagging of draft builds into tags that match the glob eln-build-side-*, please.

With that approval, I'd like to request that the staging and production environments be set to allow the tagging of draft builds into tags that match the glob `eln-build-side-*`, please.

kevin commented

2026-07-07 17:05:28 +00:00

Owner

Untested, but should just be:

diff --git a/roles/koji_hub/templates/hub.conf.j2 b/roles/koji_hub/templates/hub.conf.j2
index c39a5c530c..fc7f537dd9 100644
--- a/roles/koji_hub/templates/hub.conf.j2
+++ b/roles/koji_hub/templates/hub.conf.j2
@@ -117,17 +117,19 @@ Plugins = koji-fedoramessaging runroot_hub tag2distrepo sidetag_hub save_failed_
 {% elif koji_instance == "primary" %}
 Plugins = koji-fedoramessaging runroot_hub tag2distrepo sidetag_hub save_failed_tree flatpak kiwi image_builder
 {% else %}
 Plugins = koji-fedoramessaging runroot_hub tag2distrepo sidetag_hub save_failed_tree flatpak kiwi image_builder
 {% endif %}
 [policy]
 
 tag =
-    # We don't want to allow any draft builds to be tagged yet
+    # Allow draft builds for eln sidetags
+    is_draft && tag eln-build-side* :: allow
+    # We don't want to allow any other draft builds
     is_draft :: deny
 {% if koji_instance == "primary" %}
     user bodhi && tag *-override && package systemd-boot kernel shim grub2 pesign fwupd fwupd-efi :: allow
     has_perm autosign && fromtag *-pending && package systemd-boot kernel shim grub2 pesign fwupd fwupd-efi :: allow
     has_perm autosign && fromtag *-candidate && package systemd-boot kernel shim grub2 pesign fwupd fwupd-efi :: allow
     has_perm secure-boot && package systemd-boot kernel shim grub2 pesign fwupd fwupd-efi :: allow
     # CoreOS continuous builds, https://pagure.io/releng/issue/8165
     operation tag && tag f*-coreos-continuous && has_perm coreos-continuous :: allow

Untested, but should just be: ``` diff --git a/roles/koji_hub/templates/hub.conf.j2 b/roles/koji_hub/templates/hub.conf.j2 index c39a5c530c..fc7f537dd9 100644 --- a/roles/koji_hub/templates/hub.conf.j2 +++ b/roles/koji_hub/templates/hub.conf.j2 @@ -117,17 +117,19 @@ Plugins = koji-fedoramessaging runroot_hub tag2distrepo sidetag_hub save_failed_ {% elif koji_instance == "primary" %} Plugins = koji-fedoramessaging runroot_hub tag2distrepo sidetag_hub save_failed_tree flatpak kiwi image_builder {% else %} Plugins = koji-fedoramessaging runroot_hub tag2distrepo sidetag_hub save_failed_tree flatpak kiwi image_builder {% endif %} [policy] tag = - # We don't want to allow any draft builds to be tagged yet + # Allow draft builds for eln sidetags + is_draft && tag eln-build-side* :: allow + # We don't want to allow any other draft builds is_draft :: deny {% if koji_instance == "primary" %} user bodhi && tag *-override && package systemd-boot kernel shim grub2 pesign fwupd fwupd-efi :: allow has_perm autosign && fromtag *-pending && package systemd-boot kernel shim grub2 pesign fwupd fwupd-efi :: allow has_perm autosign && fromtag *-candidate && package systemd-boot kernel shim grub2 pesign fwupd fwupd-efi :: allow has_perm secure-boot && package systemd-boot kernel shim grub2 pesign fwupd fwupd-efi :: allow # CoreOS continuous builds, https://pagure.io/releng/issue/8165 operation tag && tag f*-coreos-continuous && has_perm coreos-continuous :: allow ```