Add compose action runner for the Atomic Desktops organization

siosm commented

2026-01-23 14:37:00 +00:00

This is an issue to request a "dedicated"/"beafier" runner for CI for the Atomic Desktops repo.

For the Atomic Desktops CI, we would like to be able to do full composes (build of the images) on PRs to validate changes.

For now, we'll limit the checks to manifest validation and dependency resolutions.

But like we had on pagure.io with the Zuul CI, we would like to be able to compose at least the smallest "base" image. Ideally we would compose all our supported images.

To avoid hogging the shared runners for everyone, it would be great if we could have a dedicated runner instance.

See initial discussion in: #356

Thanks!

This is an issue to request a "dedicated"/"beafier" runner for CI for the Atomic Desktops repo. For the Atomic Desktops CI, we would like to be able to do full composes (build of the images) on PRs to validate changes. For now, we'll limit the checks to manifest validation and dependency resolutions. But like we had on pagure.io with the Zuul CI, we would like to be able to compose at least the smallest "base" image. Ideally we would compose all our supported images. To avoid hogging the shared runners for everyone, it would be great if we could have a dedicated runner instance. See initial discussion in: https://forge.fedoraproject.org/forge/forge/issues/356 Thanks!

siosm commented

2026-01-23 15:45:05 +00:00

Author

And we'll also need this runner to let us create containers that have more privileges so all the more important that it's a distinct one.

I initially thought that we would be able to do dependency resolutions with those privileges but rpm-ostree wants the full thing: github.com/coreos/rpm-ostree@953c4e8d05

NOTE: A very important side effect of this is that we now require "privileged" containers on hosts without user namespaces, and on userns hosts, require CLONE_NEWUSER to be exported to the container host.

And we'll also need this runner to let us create containers that have more privileges so all the more important that it's a distinct one. I initially thought that we would be able to do dependency resolutions with those privileges but rpm-ostree wants the full thing: https://github.com/coreos/rpm-ostree/commit/953c4e8d0584ee3996e579056ee26dd133ef5b74 > NOTE: A *very* important side effect of this is that we now require "privileged" containers on hosts without user namespaces, and on userns hosts, require `CLONE_NEWUSER` to be exported to the container host.

siosm commented

2026-03-18 11:37:56 +00:00

Author

As pagure.io is now expected to be decommissioned around mid June (https://discussion.fedoraproject.org/t/decommissioning-of-pagure-io-anticipated-by-flock-2026/181997), please help us move forward here so that we can migrate the Atomic Desktops source repo to the new forge. Thanks

humaton commented

2026-03-25 14:36:23 +00:00

Owner

Hi @siosm,

This reply got lost in my browser tabs on multiple occasions. We are discussing with the team how to implement your request. All of our runners are unprivileged containers.

So we were talking about having a dedicated VM for each architecture for your organization with privileged containers. The work is going to land in the next forge sprint.

Hi @siosm, This reply got lost in my browser tabs on multiple occasions. We are discussing with the team how to implement your request. All of our runners are unprivileged containers. So we were talking about having a dedicated VM for each architecture for your organization with privileged containers. The work is going to land in the next forge sprint.

👍 1

humaton added the

labels

2026-03-25 14:36:49 +00:00

humaton added a new dependency

2026-03-25 14:37:20 +00:00

#359 Enable infrastructure for aarch64 runners

humaton added a new dependency

2026-03-25 14:42:55 +00:00

#474 Investigate forgejo multi-arch capapibilities

siosm commented

2026-03-25 15:09:04 +00:00

Author

Thanks a lot! Note that we only need x86_64 & aarch64 for the Atomic Desktops.

siosm commented

2026-03-25 15:18:24 +00:00

Author

And we can also start with x86_64 only for the migration, that covers 99% of our users.

lenkaseg commented

2026-04-28 13:59:43 +00:00

Member

There is an aarch64 runner added for experimenting on Forge Staging, playgroung org:
https://forge.stg.fedoraproject.org/playground
It's an unprivileged runner.

Would it be ok for you to test if this setup would work for your need on staging first?

I added you to the staging playground organization. You might need to logout and login again to forge-staging for it to take effect. You can create a repo there (or use any of those already created).
The runner has a label aarch64-1.

In case you'd prefer a different named org on staging - atomic perhaps - let me know. Or if staging environment is not suitable and you'd prefer directly production.

There is an aarch64 runner added for experimenting on Forge Staging, playgroung org: https://forge.stg.fedoraproject.org/playground It's an unprivileged runner. Would it be ok for you to test if this setup would work for your need on staging first? I added you to the staging playground organization. You might need to logout and login again to forge-staging for it to take effect. You can create a repo there (or use any of those already created). The runner has a label `aarch64-1`. In case you'd prefer a different named org on staging - `atomic` perhaps - let me know. Or if staging environment is not suitable and you'd prefer directly production.

lenkaseg commented

2026-04-28 14:01:04 +00:00

Member

About the privileged, do you need both architectures privileged as well?

hricky commented

2026-04-29 08:43:33 +00:00

Collaborator

I've done some testing on the aarch64-1 unprivileged runner in the playground org, but just like with the fedora-* runners, I was unable to build a container image.

We would like to request two organizations in the staging environment - bootc and atomic - with privileged runners. I'm not sure if a dedicated VM for each architecture with privileged runner is possible in the staging environment. If not, we would like to test just privileged runners for both organizations and architectures. What is the proper way to request these organizations? Should we open a ticket? Thanks!

cc @siosm

I've done some testing on the `aarch64-1` unprivileged runner in the `playground` org, but just like with the `fedora-*` runners, I was unable to build a container image. We would like to request two organizations in the staging environment - `bootc` and `atomic` - with privileged runners. I'm not sure if a dedicated VM for each architecture with privileged runner is possible in the staging environment. If not, we would like to test just privileged runners for both organizations and architectures. What is the proper way to request these organizations? Should we open a ticket? Thanks! cc @siosm

lenkaseg commented

2026-04-29 10:09:05 +00:00

Member

We'll discuss the option of the privileged runners in the team and let you know soon.

I opened a ticket for the staging orgs: #546

We'll discuss the option of the privileged runners in the team and let you know soon. I opened a ticket for the staging orgs: https://forge.fedoraproject.org/forge/forge/issues/546

lenkaseg commented

2026-04-29 16:22:58 +00:00

Member

Ok, there is a privileged runner on Forge Staging Playground org available to test, label aarch64-priv.
Please let me know if it works @hricky @siosm

Ok, there is a privileged runner on Forge Staging Playground org available to test, label `aarch64-priv`. Please let me know if it works @hricky @siosm

hricky commented

2026-04-29 18:44:14 +00:00

Collaborator

I did some tests and it looks good so far, but it seems that the allocated disk space for the runner is only 5G. See:
https://forge.stg.fedoraproject.org/playground/fedora-runner-experimental/actions/runs/146/jobs/0/attempt/1
https://forge.stg.fedoraproject.org/playground/fedora-runner-experimental/actions/runs/169/jobs/0/attempt/1

Can it be increased? Thanks!

I did some tests and it looks good so far, but it seems that the allocated disk space for the runner is only 5G. See: https://forge.stg.fedoraproject.org/playground/fedora-runner-experimental/actions/runs/146/jobs/0/attempt/1 https://forge.stg.fedoraproject.org/playground/fedora-runner-experimental/actions/runs/169/jobs/0/attempt/1 Can it be increased? Thanks!

siosm commented

2026-04-30 08:01:32 +00:00

Author

About the privileged, do you need both architectures privileged as well?

Yes

And yes, we need much more disk space, at least 10GB if I remember correctly but maybe even more.

> About the privileged, do you need both architectures privileged as well? Yes And yes, we need much more disk space, at least 10GB if I remember correctly but maybe even more.

👍 1

lenkaseg commented

2026-04-30 08:31:23 +00:00

Member

I'll get a larger VM in that case. Will let you know about the progress, but since I have to find a person to spin those VMs for me, might not be available until next week.

lenkaseg referenced this issue from infra/tickets

2026-05-05 09:52:28 +00:00

Request for resources - large sized aws arm64 VM with public IP for Forge runners #13319

siosm commented

2026-05-05 14:17:00 +00:00

Author

It looks like Zuul CI is not working anymore on pagure.io (example: https://pagure.io/workstation-ostree-config/pull-request/756, I don't know if it's intentional or a bug). Thus this is becoming more pressing now as we need to get the CI up and running again here.

siosm referenced this issue from releng/tickets

2026-05-05 14:22:16 +00:00

Migration of workstation-ostree-config to forge.fedoraproject.org #13222

siosm referenced this issue from ci/tickets

2026-05-05 14:23:29 +00:00

Zuul CI on pagure.io/github.com hosted projects - what should we do ? #544

lecris commented

2026-05-05 14:28:14 +00:00

@siosm wrote in #364 (comment):

About the privileged, do you need both architectures privileged as well?

Yes

And yes, we need much more disk space, at least 10GB if I remember correctly but maybe even more.

Oh, was not made aware of these roadblocks, I only considered

This project doesn't use any special features of Zuul CI so it may be able to migrate to a simpler architecture.

Could help with trying to spin up a testing-farm job for those and run on the smaller workers. Can still prioritize and rush that if it still helps right now.

@siosm wrote in https://forge.fedoraproject.org/forge/forge/issues/364#issuecomment-677252: > > About the privileged, do you need both architectures privileged as well? > > Yes > > And yes, we need much more disk space, at least 10GB if I remember correctly but maybe even more. Oh, was not made aware of these roadblocks, I only considered > This project doesn't use any special features of Zuul CI so it may be able to migrate to a simpler architecture. Could help with trying to spin up a testing-farm job for those and run on the smaller workers. Can still prioritize and rush that if it still helps right now.

siosm commented

2026-05-05 18:51:03 +00:00

Author

Sure, I don't know what would be needed / what would a testing farm job do but I will take any help I can get to have CI here. Thanks

lecris commented

2026-05-06 07:45:40 +00:00

It's basically just a different CI runner from forgejo actions with the runners managed by testing-farm instead of fedora-infra. It does require a tmt setup, but that can be configured such that it is a thin wrapper of whatever workflow you prefer, I see that in this case it is just. I will make a PR for that repo with a fork on the ci organization to show it in action

It's basically just a different CI runner from forgejo actions with the runners managed by `testing-farm` instead of `fedora-infra`. It does require a `tmt` setup, but that can be configured such that it is a thin wrapper of whatever workflow you prefer, I see that in this case it is `just`. I will make a PR for that repo with a fork on the ci organization to show it in action

lecris referenced this issue from atomic-desktops/config

2026-05-06 08:33:12 +00:00

POC: Forgejo actions with testing-farm/tmt #757

lecris commented

2026-05-06 08:33:53 +00:00

PR up, and failing, but will figure out the issues there: atomic-desktops/config#757

PR up, and failing, but will figure out the issues there: https://forge.fedoraproject.org/atomic-desktops/config/pulls/757

lenkaseg commented

2026-05-06 14:47:10 +00:00

Member

There is a new arm64 VM with 20GB of storage and I just enabled two aarch64 runners there, privileged and unprivileged, available from the playground org (the atomic-desktops and bootc orgs will be available soon on staging I suppose and when they are, I will move those runners under those orgs).

I kept the labels the same, so no need to change anything when you test it: unprivileged: aarch64-1 and privileged: aarch64-priv.

Please let me know if it works!

@siosm

There is a new arm64 VM with 20GB of storage and I just enabled two aarch64 runners there, privileged and unprivileged, available from the playground org (the `atomic-desktops` and `bootc` orgs will be available soon on staging I suppose and when they are, I will move those runners under those orgs). I kept the labels the same, so no need to change anything when you test it: unprivileged: `aarch64-1` and privileged: `aarch64-priv`. Please let me know if it works! @siosm

👍 1

lenkaseg self-assigned this

2026-05-07 08:02:28 +00:00

lenkaseg added this to the Backlog project

2026-05-07 08:03:45 +00:00

lenkaseg added

and removed

labels

2026-05-07 08:03:58 +00:00

lenkaseg added a new dependency

2026-05-07 08:06:54 +00:00

#561 Create x86_64 privileged runner on an aws VM

siosm referenced this issue from atomic-desktops/tracker

2026-05-07 16:22:46 +00:00

Setup CI on the repo in the new forge #114

lenkaseg commented

2026-05-08 14:56:23 +00:00

Member

Here link to the playground org: https://forge.stg.fedoraproject.org/playground

I see you're member there, so you should be able to create or use any repos there and run a test action on those runners.

When you confirm the runners are suitable for you, I will get them for you in production Forge.

Here link to the playground org: https://forge.stg.fedoraproject.org/playground I see you're member there, so you should be able to create or use any repos there and run a test action on those runners. When you confirm the runners are suitable for you, I will get them for you in production Forge.

lenkaseg commented

2026-05-13 11:15:03 +00:00

Member

Ok, we have atomic-desktops and bootc orgs on Forge Staging, so I created aarch64 privileged runner for those two orgs.

Please test them and let me know if they are sufficient for you purposes!

Ok, we have [atomic-desktops](https://forge.stg.fedoraproject.org/atomic-desktops) and [bootc](https://forge.stg.fedoraproject.org/bootc) orgs on Forge Staging, so I created aarch64 privileged runner for those two orgs. Please test them and let me know if they are sufficient for you purposes!

lenkaseg commented

2026-05-13 11:15:27 +00:00

Member

Labels for both is aarch64-privileged.

Labels for both is `aarch64-privileged`.

hricky commented

2026-05-14 15:11:21 +00:00

Collaborator

We tested the aarch64-privileged runner with the Fedora Atomic Desktops config repo. All validations and verifications are successful. The base image compose is also successful. For the current setup of this particular repo, the aarch64-privileged runner should be sufficient:
https://forge.stg.fedoraproject.org/atomic-desktops/config-ci-test/actions/runs/20/jobs/0/attempt/1

We will next try to compose all supported images. We will also continue to test the aarch64-privileged runner with other repos in both orgs.

Is it possible to create a similar runner for the x86_64 arch in both orgs? Thanks!

We tested the `aarch64-privileged` runner with the [Fedora Atomic Desktops config](https://forge.stg.fedoraproject.org/atomic-desktops/config-ci-test/pulls/1) repo. All validations and verifications are successful. The base image compose is also successful. For the current setup of this particular repo, the `aarch64-privileged` runner should be sufficient: https://forge.stg.fedoraproject.org/atomic-desktops/config-ci-test/actions/runs/20/jobs/0/attempt/1 We will next try to compose all supported images. We will also continue to test the `aarch64-privileged` runner with other repos in both orgs. Is it possible to create a similar runner for the x86_64 arch in both orgs? Thanks!

👍 2

siosm commented

2026-05-15 13:55:36 +00:00

Author

As @hricky said, this setup works for us, so I think we should make the runners available in the main Forgejo instance.

lenkaseg commented

2026-05-15 14:24:31 +00:00

Member

Excellent! I'll make the runners for both architectures available on prod under. To resume, that would be:

atomic-desktops org:

1 aarch64 privileged
1 x86_64 privileged

Do you want to keep the unprivileged runner for running repo events and such?

Also, I cannot see any bootc org on production. Do you want to create one?

Excellent! I'll make the runners for both architectures available on prod under. To resume, that would be: `atomic-desktops` org: - 1 aarch64 privileged - 1 x86_64 privileged Do you want to keep the unprivileged runner for running repo events and such? Also, I cannot see any `bootc` org on production. Do you want to create one?

siosm commented

2026-05-15 14:36:00 +00:00

Author

Excellent! I'll make the runners for both architectures available on prod under. To resume, that would be:

atomic-desktops org:

1 aarch64 privileged

1 x86_64 privileged

Yes

Do you want to keep the unprivileged runner for running repo events and such?

Yes, please, that will be useful to run smaller jobs in parallel and not block the rest of Fedora on our jobs (notably if we end up using testing-farm/tmt for testing as this locks a runner for the entire duration of the job: https://forge.fedoraproject.org/ci/testing-farm).

Also, I cannot see any bootc org on production. Do you want to create one?

I think we should do that independently and in a separate ticket. Discussion was in progress in https://gitlab.com/fedora/bootc/tracker/-/work_items/75.

> Excellent! I'll make the runners for both architectures available on prod under. To resume, that would be: > > `atomic-desktops` org: > > * 1 aarch64 privileged > * 1 x86_64 privileged Yes > Do you want to keep the unprivileged runner for running repo events and such? Yes, please, that will be useful to run smaller jobs in parallel and not block the rest of Fedora on our jobs (notably if we end up using testing-farm/tmt for testing as this locks a runner for the entire duration of the job: https://forge.fedoraproject.org/ci/testing-farm). > Also, I cannot see any `bootc` org on production. Do you want to create one? I think we should do that independently and in a separate ticket. Discussion was in progress in https://gitlab.com/fedora/bootc/tracker/-/work_items/75.

lecris commented

2026-05-15 14:46:32 +00:00

@lenkaseg Re the blocking resources for the rest in ☝️, I have a suggestion in #568, would appreciate some thoughts on it

@lenkaseg Re the blocking resources for the rest in ☝️, I have a suggestion in https://forge.fedoraproject.org/forge/forge/issues/568, would appreciate some thoughts on it

👍 1

hricky commented

2026-05-16 12:02:32 +00:00

Collaborator

We tested building all images sequentially in one job, but disk and system resources of the aarch64-privileged runner seem to become exhausted. Here is the size of the ./cache directory after building the images locally:

base-atomic: 4,8G
silverblue: 6,3G
kinoite: 8,8G
sway-atomic: 9,3G
budgie-atomic: 11G
cosmic-atomic: 12G

We are already cleaning between builds with a custom action, but the issue is probably that we are hitting the disk size limit during the build process itself. With 17 GB total disk space (13 GB usable) and 6 images building sequentially, we're likely run out of resources:

Running posttrans scripts...done
error: Installing packages: Running %posttrans for systemd-resolved: bwrap(/bin/sh): Child process exited with code 254
nitor.socket' → '/usr/lib/systemd/system/systemd-resolved-monitor.socket'.
qemu-guest-agent.post: Created symlink '/etc/systemd/system/dev-virtio\x2dports-org.qemu.guest_agent.0.device.wants/qemu-guest-agent.service' → '/usr/lib/systemd/system/qemu-guest-agent.service'.
qemu-guest-agent.post: Unit /usr/lib/systemd/system/qemu-guest-agent.service is added as a dependency to a non-existent unit dev-virtio\x2dports-org.qemu.guest_agent.0.device.
grub2-efi-aa64.posttrans: grub2-probe: error: failed to get canonical path of `/boot/grub2'.
wireplumber.posttrans: Created symlink '/etc/systemd/user/pipewire-session-manager.service' → '/usr/lib/systemd/user/wireplumber.service'.
wireplumber.posttrans: Created symlink '/etc/systemd/user/pipewire.service.wants/wireplumber.service' → '/usr/lib/systemd/user/wireplumber.service'.
systemd-resolved.posttrans: /proc/self/fd/5: fork: retry: Resource temporarily unavailable
systemd-resolved.posttrans: /proc/self/fd/5: fork: retry: Resource temporarily unavailable
systemd-resolved.posttrans: /proc/self/fd/5: fork: retry: Resource temporarily unavailable
systemd-resolved.posttrans: /proc/self/fd/5: fork: retry: Resource temporarily unavailable
systemd-resolved.posttrans: /proc/self/fd/5: fork: Resource temporarily unavailable
error: Failed to run compose tree: ExitStatus(unix_wait_status(256))
error: Recipe `compose-image` failed with exit code 1
⚙️ [runner]: exitcode '1': failure

See: https://forge.stg.fedoraproject.org/atomic-desktops/config-ci-test/actions/runs/61/jobs/0/attempt/2.

We switched the workflow to only run a single compose per job sequentially (with cleanup in between), but we still get the following error on some composes:

Generating container image
error: Encapsulating: Skopeo copy: skopeo failed: time="2026-05-16T01:04:37Z" level=fatal msg="committing the finished image: write /workspace/atomic-desktops/config-ci-test/budgie-atomic.ociarchive: no space left on device"
error: Failed to run compose container-encapsulate: ExitStatus(unix_wait_status(256))
error: Recipe `compose-image` failed with exit code 1
⚙️ [runner]: exitcode '1': failure

See: https://forge.stg.fedoraproject.org/atomic-desktops/config-ci-test/actions/runs/65/jobs/1/attempt/1.

We tested building all images sequentially in one job, but disk and system resources of the `aarch64-privileged` runner seem to become exhausted. Here is the size of the `./cache` directory after building the images locally: ``` base-atomic: 4,8G silverblue: 6,3G kinoite: 8,8G sway-atomic: 9,3G budgie-atomic: 11G cosmic-atomic: 12G ``` We are already cleaning between builds with a custom action, but the issue is probably that we are hitting the disk size limit during the build process itself. With 17 GB total disk space (13 GB usable) and 6 images building sequentially, we're likely run out of resources: ``` Running posttrans scripts...done error: Installing packages: Running %posttrans for systemd-resolved: bwrap(/bin/sh): Child process exited with code 254 nitor.socket' → '/usr/lib/systemd/system/systemd-resolved-monitor.socket'. qemu-guest-agent.post: Created symlink '/etc/systemd/system/dev-virtio\x2dports-org.qemu.guest_agent.0.device.wants/qemu-guest-agent.service' → '/usr/lib/systemd/system/qemu-guest-agent.service'. qemu-guest-agent.post: Unit /usr/lib/systemd/system/qemu-guest-agent.service is added as a dependency to a non-existent unit dev-virtio\x2dports-org.qemu.guest_agent.0.device. grub2-efi-aa64.posttrans: grub2-probe: error: failed to get canonical path of `/boot/grub2'. wireplumber.posttrans: Created symlink '/etc/systemd/user/pipewire-session-manager.service' → '/usr/lib/systemd/user/wireplumber.service'. wireplumber.posttrans: Created symlink '/etc/systemd/user/pipewire.service.wants/wireplumber.service' → '/usr/lib/systemd/user/wireplumber.service'. systemd-resolved.posttrans: /proc/self/fd/5: fork: retry: Resource temporarily unavailable systemd-resolved.posttrans: /proc/self/fd/5: fork: retry: Resource temporarily unavailable systemd-resolved.posttrans: /proc/self/fd/5: fork: retry: Resource temporarily unavailable systemd-resolved.posttrans: /proc/self/fd/5: fork: retry: Resource temporarily unavailable systemd-resolved.posttrans: /proc/self/fd/5: fork: Resource temporarily unavailable error: Failed to run compose tree: ExitStatus(unix_wait_status(256)) error: Recipe `compose-image` failed with exit code 1 ⚙️ [runner]: exitcode '1': failure ``` See: https://forge.stg.fedoraproject.org/atomic-desktops/config-ci-test/actions/runs/61/jobs/0/attempt/2. We switched the workflow to only run a single compose per job sequentially (with cleanup in between), but we still get the following error on some composes: ``` Generating container image error: Encapsulating: Skopeo copy: skopeo failed: time="2026-05-16T01:04:37Z" level=fatal msg="committing the finished image: write /workspace/atomic-desktops/config-ci-test/budgie-atomic.ociarchive: no space left on device" error: Failed to run compose container-encapsulate: ExitStatus(unix_wait_status(256)) error: Recipe `compose-image` failed with exit code 1 ⚙️ [runner]: exitcode '1': failure ``` See: https://forge.stg.fedoraproject.org/atomic-desktops/config-ci-test/actions/runs/65/jobs/1/attempt/1.

lenkaseg commented

2026-05-18 09:45:06 +00:00

Member

Can you give me an estimate of storage requirements? Would 30 GB be sufficient? Or 50?

ryanlerch added

and removed

labels

2026-05-18 10:22:12 +00:00

hricky commented

2026-05-20 08:12:00 +00:00

Collaborator

We'll need up to 100 GB. We realize that's a large amount of disk space. We plan to build image artifacts, and insufficient space may require cleaning between builds, as we're doing currently. Thanks!

humaton commented

2026-05-20 10:34:40 +00:00

Owner

Now we are getting into a resource requirements outside of the scope of Forge. If you require full composes for CI, we need to find a way to use the testing-farm as @lecris is proposing.

Now we are getting into a resource requirements outside of the scope of Forge. If you require full composes for CI, we need to find a way to use the `testing-farm` as @lecris is proposing.

siosm commented

2026-05-20 14:28:13 +00:00

Author

We are giving this a try in atomic-desktops/config#757 and the experience is quite bad. We would need #568 for the testing-farm support to make sense as we could do the builds in parallel and not block current forge runners.

What are your limits right now? Can we make this work for now and we'll migrate to something else once this is ready? We also have work in progress to get Konflux to do the composes on PRs so that could be another option in the future but we really need something that works today in the meantime.

We are giving this a try in https://forge.fedoraproject.org/atomic-desktops/config/pulls/757 and the experience is quite bad. We would need https://forge.fedoraproject.org/forge/forge/issues/568 for the `testing-farm` support to make sense as we could do the builds in parallel and not block current forge runners. What are your limits right now? Can we make this work for now and we'll migrate to something else once this is ready? We also have work in progress to get Konflux to do the composes on PRs so that could be another option in the future but we really need something that works today in the meantime.

siosm commented

2026-05-20 14:34:25 +00:00

Author

If we reduce the scope to only the base atomic image composes (what we were doing on Zuul CI before) we can do with the runner as is. It's not great but that will cover the basics until we have something more powerful.

mwinters referenced this issue from fdwg/datanommer-docker

2026-05-20 21:16:18 +00:00

Build this in a runner #1

humaton commented

2026-05-21 10:22:05 +00:00

Owner

@siosm wrote in #364 (comment):

We are giving this a try in atomic-desktops/config#757 and the experience is quite bad. We would need #568 for the testing-farm support to make sense as we could do the builds in parallel and not block current forge runners.

This was put on this team sprint with high priority.

What are your limits right now? Can we make this work for now and we'll migrate to something else once this is ready? We also have work in progress to get Konflux to do the composes on PRs so that could be another option in the future but we really need something that works today in the meantime.

The current storage limit is set to 20G, the capacity was set to default 1 for all runners. That will change in a moment, as @lenkaseg is working on it. But the privileged runner in question is running on very small VM in AWS so the capacity will be 2.

@siosm wrote in https://forge.fedoraproject.org/forge/forge/issues/364#issuecomment-713800: > We are giving this a try in atomic-desktops/config#757 and the experience is quite bad. We would need #568 for the `testing-farm` support to make sense as we could do the builds in parallel and not block current forge runners. This was put on this team sprint with high priority. > What are your limits right now? Can we make this work for now and we'll migrate to something else once this is ready? We also have work in progress to get Konflux to do the composes on PRs so that could be another option in the future but we really need something that works today in the meantime. The current storage limit is set to 20G, the `capacity` was set to default 1 for all runners. That will change in a moment, as @lenkaseg is working on it. But the privileged runner in question is running on very small VM in AWS so the capacity will be 2.

humaton commented

2026-05-21 10:25:16 +00:00

Owner

For the unprivileged runners, the only constrain is the cluster HW that we are running on.

lenkaseg commented

2026-05-21 11:46:54 +00:00

Member

Ok, the aarch64-priv in atomic-desktops org on staging has capacity set to 2 concurrents jobs. More capacity probably does not make sense. The VM has 4 cores 15 GB RAM and 13GB of free space.

Ok, the `aarch64-priv` in `atomic-desktops` org on staging has capacity set to 2 concurrents jobs. More capacity probably does not make sense. The VM has 4 cores 15 GB RAM and 13GB of free space.

👍 1

hricky commented

2026-05-21 14:39:46 +00:00

Collaborator

We tested the runner with smaller images, for which the 13GB of usable space is sufficient. The runner with the new setup with 2 concurrent jobs is approximately 27% faster. We tested it by increasing the zstd compression level to the max. However, I am not entirely sure it matters, but I think it is faster anyway.

Rows
Columns

Add compose action runner for the Atomic Desktops organization #364