Switching quay.io/fedora/fedora-bootc to be built by Konflux #13115

Open
opened 2025-12-02 15:55:31 +00:00 by cverna · 12 comments
Contributor

Hi 👋, we are moving fedora-bootc builds from Pungi to Fedora Konflux (for example https://quay.io/repository/bootc-devel/fedora-bootc-rawhide-standard?tab=tags&tag=latest))

The Question: Is it acceptable to push these images to quay.io/fedora signed only by the Konflux key (not the official Fedora/RelEng key)? This is our final blocker (https://gitlab.com/fedora/bootc/base-images/-/issues/51#note_2843289127)

Long term we want to have https://github.com/fedora-infra/siguldry/issues/49 but in the mean time it would be great if we could make the switch to Konflux.

Hi 👋, we are moving fedora-bootc builds from Pungi to Fedora Konflux (for example https://quay.io/repository/bootc-devel/fedora-bootc-rawhide-standard?tab=tags&tag=latest)) The Question: Is it acceptable to push these images to quay.io/fedora signed only by the Konflux key (not the official Fedora/RelEng key)? This is our final blocker (https://gitlab.com/fedora/bootc/base-images/-/issues/51#note_2843289127) Long term we want to have https://github.com/fedora-infra/siguldry/issues/49 but in the mean time it would be great if we could make the switch to Konflux.
Contributor

As far as I can tell, we have never signed quay.io/fedora/fedora - probably related to the fact that we've never used container-native tooling to build it.

So I don't think this is a blocker actually.

But it would clearly make sense to ensure we're signing it, and also we should move the app base image under this same infrastructure too.

As far as I can tell, we have never signed `quay.io/fedora/fedora` - probably related to the fact that we've never used container-native tooling to build it. So I don't think this is a blocker actually. But it would clearly make sense to ensure we're signing it, and also we should move the app base image under this same infrastructure too.
Contributor

IMO for OS updates we shouldn't ship them without some sort of verification.

Application containers are usually sandboxed in some way to prevent the blast radius from a potential exploit so I could see shipping containers there that aren't signed/verified, but I think OS updates containers should require a higher level of security.

For OS Updates we've always had one of the following for official fedora updates (I think):

  • dnf/rpm: checking individual RPM signatures
  • ostree repo: OSTree commit signatures + verification
  • container updates (CoreOS today): OOB GPG signing and signature verification
IMO for OS updates we shouldn't ship them without some sort of verification. Application containers are usually sandboxed in some way to prevent the blast radius from a potential exploit so I could see shipping containers there that aren't signed/verified, but I think OS updates containers should require a higher level of security. For OS Updates we've always had one of the following for official fedora updates (I think): - dnf/rpm: checking individual RPM signatures - ostree repo: OSTree commit signatures + verification - container updates (CoreOS today): OOB [GPG signing](https://github.com/coreos/fedora-coreos-tracker/issues/1969#issuecomment-3251151820) and [signature verification](https://github.com/coreos/fedora-coreos-config/commit/7ef45117dc26580d21fef2a8eee5095d200481be)
Author
Contributor

Just to be clear the images are signed, but using the Fedora Konflux instance key which is not available under https://fedoraproject.org/security

Just to be clear the images are signed, but using the Fedora Konflux instance key which is not available under https://fedoraproject.org/security
Contributor

👍 - interested to have someone from releng weigh in here.

👍 - interested to have someone from releng weigh in here.
Owner

yeah, thats all right from my understanding.

We have also #10624 that no one has yet done about signing container images.

So, if we pushed these images signed by the konflux instance key, would that cause any problems for users? ie, would it yell that they didn't have the key or that it was untrusted or anything? And would there be some way we could (if only manually) let people verify with that key?

But yeah, it would be great to have this working

yeah, thats all right from my understanding. We have also https://forge.fedoraproject.org/releng/tickets/issues/10624 that no one has yet done about signing container images. So, if we pushed these images signed by the konflux instance key, would that cause any problems for users? ie, would it yell that they didn't have the key or that it was untrusted or anything? And would there be some way we could (if only manually) let people verify with that key? But yeah, it would be great to have this working

AFAIK it won't yell as the /etc/containers/policy.json allows anything for the docker transport by default.

And would there be some way we could (if only manually) let people verify with that key?

Yes see below the commands to manually check:

PUBKEY=/tmp/konflux-pub.key
# logged into Fedora Konflux cluster
oc get secret -n openshift-pipelines public-key -o json | jq -r '.data["cosign.pub"] | @base64d' > $PUBKEY
# then to verify an image
cosign verify --key $PUBKEY --insecure-ignore-tlog quay.io/bootc-devel/fedora-bootc-rawhide-standard:latest

and one could update its /etc/containers/policy.json accordingly with that Konflux pubkey to verify automatically during the pull.

AFAIK it won't yell as the `/etc/containers/policy.json` allows anything for the docker transport by default. > And would there be some way we could (if only manually) let people verify with that key? Yes see below the commands to manually check: ``` PUBKEY=/tmp/konflux-pub.key # logged into Fedora Konflux cluster oc get secret -n openshift-pipelines public-key -o json | jq -r '.data["cosign.pub"] | @base64d' > $PUBKEY # then to verify an image cosign verify --key $PUBKEY --insecure-ignore-tlog quay.io/bootc-devel/fedora-bootc-rawhide-standard:latest ``` and one could update its `/etc/containers/policy.json` accordingly with that Konflux pubkey to verify automatically during the pull.
Contributor

Correct, rather than have users configure the policy I would think we (Fedora) would want to ship a policy that required signature verification for containers from repos we know are signed.

Correct, rather than have users configure the policy I would think we (Fedora) would want to ship a policy that required signature verification for containers from repos we know are signed.
Member

Something shipped in the quay.io/fedora namespace should be using the Fedora key, not some other one. Is there a reason this can't be wired up somehow?

Something shipped in the quay.io/fedora namespace should be using the Fedora key, not some other one. Is there a reason this can't be wired up somehow?

If releng offers something like what's described in https://github.com/fedora-infra/siguldry/issues/49#issuecomment-2949889535 then we'd "just" have to wrap this in a Tekton task and chain it to the bootc Konflux release pipeline.

If releng offers something like what's described in https://github.com/fedora-infra/siguldry/issues/49#issuecomment-2949889535 then we'd "just" have to wrap this in a Tekton task and chain it to the bootc Konflux release pipeline.

I don't think we should ship official release artifacts signed by the Konflux key as anything produced by Konflux is signed by this key. We should make sure that we have official release artifacts (and only those) signed by the Fedora keys.

I don't think we should ship official release artifacts signed by the Konflux key as anything produced by Konflux is signed by this key. We should make sure that we have official release artifacts (and only those) signed by the Fedora keys.

But to come back to Colin's point, fedora-bootc images currently published (https://quay.io/repository/fedora/fedora-bootc?tab=tags) are not signed so I'm not sure this should block this.

This discussion should probably happen in the context of a change request, maybe https://fedoraproject.org/wiki/Changes/konflux-atomic-change-proposal ?

Edit: Fixed the missing not in the sentence above.

But to come back to Colin's point, fedora-bootc images currently published (https://quay.io/repository/fedora/fedora-bootc?tab=tags) are not signed so I'm **not** sure this should block this. This discussion should probably happen in the context of a change request, maybe https://fedoraproject.org/wiki/Changes/konflux-atomic-change-proposal ? Edit: Fixed the missing **not** in the sentence above.
Contributor

Application containers are usually sandboxed in some way to prevent the blast radius from a potential exploit so I could see shipping containers there that aren't signed/verified, but I think OS updates containers should require a higher level of security.

I agree with the spirit of your point, but note particularly for Fedora-bootc we intend that many use cases actually derive at build time instead of updating directly, so it really is fundamentally much more like the app container image.

Now I hope everyone would agree that we should be building all fedora containers with Konflux and signing them with a Fedora key.

But in the interim I don't see any blockers to changing fedora-bootc over to being shipped via Konflux and dropping the Konflux signature again with the rationale that it matches the app base.

Then we'd cut over to signing both with the Fedora key.

> Application containers are usually sandboxed in some way to prevent the blast radius from a potential exploit so I could see shipping containers there that aren't signed/verified, but I think OS updates containers should require a higher level of security. I agree with the spirit of your point, but note particularly for Fedora-bootc we intend that many use cases actually derive at build time instead of updating directly, so it really is fundamentally much more like the app container image. Now I hope everyone would agree that we should be building all fedora containers with Konflux and signing them with a Fedora key. But in the interim I don't see any blockers to changing fedora-bootc over to being shipped via Konflux and dropping the Konflux signature again with the rationale that it matches the app base. Then we'd cut over to signing both with the Fedora key.
Sign in to join this conversation.
No milestone
No project
No assignees
7 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
releng/tickets#13115
No description provided.