Container Signing Initial Proposal #6
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
I’ve raised this as a Security ticket because it’s the SIG I’m closest to. If we go ahead with this, the work will span Cloud, Infra, Atomic, etc. I'd like to answer the questions that we can before bothering other SIGs with this.
Fedora doesn’t currently sign its container images, and it’d be neat if we did. This also opens us up to providing things like SBOMs, enforcing image verification on the atomic distros, etc. We’ve not really had the tools to do this before now, and this ticket collects some musings on how we can do it.
Red Hat’s docs for this are pretty good and worth reading https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/building_running_and_managing_containers/assembly_signing-container-images_building-running-and-managing-containers.
What do we sign?
Container signatures are generated by creating a JSON payload like the one below and signing it with an X509 certificate or GPG key.
Container registries can natively handle the X509 version, but using a GPG signature requires deploying a separate lookaside cache server and configuring the client to look at it. See
/etc/containers/registries.d/registry.access.redhat.com.yamlfor an example of that.You’re also strongly encouraged to push a certificate transparency assertion to a platform like Rekor, but that doesn’t work if you’re using GPG.
How should we sign it?
cosigncan do all the signing logic as a stand-alone tool, and whilst we don’t package it today we could start. Other tools such asskopeocan also perform the signing in flight.We should sign these payloads with a dedicated key. These keys currently live in Sigul, although at some point they’ll hopefully move to Siguldry. I recently wrote some docs on signing containers with a key stored in Siguldry (https://github.com/fedora-infra/siguldry/tree/main/siguldry-pkcs11#container-signing) which we could use as a starting point.
This opens the question of when should that signing actually occur. In an ideal world I think this would be as soon as Kiwi builds the image so there is the smallest possible window for tampering to occur, but that probably wont work. The signable blob includes the image reference that’s being pushed, and Kiwi doesn’t currently know where its images will be pushed to. We could teach it that, but I think that’s probably leaking information from later in the image pipeline to too early of a stage.
The alternative is that we attach the signature at the point we push the image to a public container registry, which is currently done by https://pagure.io/cloud-image-uploader (which under the hood wraps
skopeo copy).Signing would either need to happen as part of the
skopeo copy(which just delegates togpgorsq, so already works for GPG signing) or by splitting that job up into a multi-stage process where we pull, sign, and push. Ifskopeocan do everything we need during the copy, I think we should go that way.What key do we use?
This is potentially more of a headache than I originally expected. There seems to be disagreement whether GPG or X509 signatures are the future here. Kubernetes tools like Kyverno and
containerdonly understand X509 signatures but container-tools tooling (Podman, CRI-o) can handle either method. If possible, maybe we sign with both formats?As far as I can tell, X509 implementations tend to let you specify a key per image reference or for all references on an image path (e.g.
quay.io/fedora/fedora:*). GPG verification seems to expect the same key for the whole “namespace” (e.g.quay.io/fedora/*).This will make it difficult for us to use a different signing key per release, but the security implications of signing all contianer images with the same key does make me a bit uncomfortable.
Components
We need to answer some questions, based on which there are some tasks to complete:
cloud-image-uploadercloud-image-uploaderhas some means of publishing files tocloud-image-uploaderto set the relevant skopeo copy arguments.cloud-image-uploadercloud-image-uploaderOpenShift manifests/etc/containers/registries.dso clients start verifying these certificatesI’m very open to putting this through the Fedora change process, but I’d like answers to the questions before doing so. I’d like to present it as more of a cohesive plan, and this ticket is a stream of thought. As I’m not a member of the Infra team, I can raise pull requests but can’t independently make most of those changes.
Raised https://pagure.io/cloud-image-uploader/pull-request/73 for the
cloud-image-uploadercode changesAs somebody who is not very familiar with container tooling - does this mean podman can verify OpenPGP container signatures (via podman-sequoia?) but Fedora currently doesn't actually sign the containers we publish? Would using cosign here mean that podman can't verify the signatures, or does it support both cosign and OpenPGP signatures?
Yes
I believe it defaults to GnuPG but can use either, I'll try to confirm.
Yeah. It never got implemented in Sigul but now that Siguldry has a generic PKCS#11 interface it's much easier for us to implement.
In
/etc/containersyou can tell it whether to use Cosign or PGP signaturesFor example
/etc/containers/registries.d/registry.access.redhat.com.yamlThis tells Podman to pull PGP signatures from that URL, or you could set
use-sigstore-attachments: trueto tell it to pull Cosign signatures from the same container registry it pulled the image from (and you'll then need to set a signing key in/etc/containers/policy.json).So before we get too far along implementing this, there's a few things we should probably consider.
I don't think the cloud image uploader covers all the images Fedora produces that would be interested in signatures (I think Atomic Desktop bootc images fall into this category?). I'm also not thrilled at the idea of signing that late in the build process. I'd definitely at least like to consider what it would look like to have containers signed like RPMs, ostree commits, etc in a robosignatory-like AMQP consumer dedicated to that purpose.
The other thing is my impression (as someone not on top of what the container people are up to and like most lately) is that cosign-type signatures are what everyone supports, and OpenPGP is a podman-only thing that Red Hat started and didn't ultimately catch on (citation needed).
@jcline wrote in #6 (comment):
I believe you're correct, and I'm hoping to conduct an audit of the images we publish soon unless you're aware of some existing authority?
I think this is an option certainly worth considering, particularly as I assume Robosignatory is due a Rust rewrite soon. That tool would already have access to the signing credentials and is probably as early in the process as we can reasonably put it. I still don't love that we'd need to tell Robosignatory the URLs that the containers are going to be published to (that feels like it's leaking information between the stages of CI) but this seems like a more acceptable place than within Kiwi.
Is this something you'd be open to spiking within Robosignatory? I'm concerned that it's too far from how the other handlers work in there so might be worth waiting for a rewrite with Siguldry.
I think you're correct on that, which is disappointing as it seems like OpenPGP would actually fit our architecture better.
Just to answer some doubts/questions from the discussion above. Yes, cosign signatures are preferred and more widespread. They're natively supported by most registries and clients and don't require a lookaside. Although AIUI currently for anyone with PQC requirements, the GPG approach is the only approach for now, but presumably the sigstore ecosystem will catch up. I'll try to get an actual podman/container-libs SME (which I'm not :)) to chime in.
CoreOS doesn't use the cloud image uploader. This is an amusing side note, but FCOS currently does ship OCI images with GPG signatures using the Fedora GPG keys. It's a tremendous hack though bordering on the "not technically supported". That code lives here.
I think Atomic Workstation currently does use cloud image uploader (at least it uses the same Pungi pipeline bits as other container images), but ideally won't in the future.
The fedora-bootc images use the newest instance of Fedora Konflux for CI but the official builds still use Pungi.
Which brings me to the largest gap I see: there is no mention of Konflux here at all, but to me that should be the primary target for this work, no? On the CoreOS side we definitely want to build with Konflux soon, and ideally we align all of Fedora and move all Fedora container builds there eventually. It already handles a lot of what we're talking about here, including SBOM generation, provenance attestation, and signing. The main work item I think is to figure out how to extend Konflux/Chains to support talking to Siguldry? That'll need input from the Konflux team.
(I work on container signing for Podman + friends.)
The payload part is true-ish (and that might be relevant if Fedora wanted to implement its own signing code, e.g. to integrate with a signing server), but otherwise, the better way to think about it is: There are at least 3 different signature formats around (and probably quite a few more not relevant to Podman and other Fedora consumers):
The sigstore signatures can use an X.509 PKI, but that’s not required (and I would say not particularly useful, because there are no name/scope constraints on the CA-signed certificates, like a web PKI that issues a certificate to be valid only for "fedora.example").
Currently, PQC signatures are implemented for GPG (if/when that lands into Sequoia-PGP and rust-podman-sequoia). PQC signatures for sigstore are impossible in the cosign v1/v2 sigstore, and there is some effort (with an uncertain timeline, measured in quarters at least) to support them in sigstore as of cosign v3.
So, sadly, there’s no ideal format to recommend as of today.
It is possible for a single image to be signed using more than one of these mechanisms (and RHEL does do that), with all the extra work that would imply. Right now, it might make sense to publish sigstore non-PQC signatures (for convenience and interoperability with cosign) and GPG PQC (well, PQC hybrid) signatures (for PQC safety). Going sigstore-only for now, and planning to move to sigstore-PQC in a few years, is also an option. (I’ll defer to actual crypto experts on timing and urgency of PQC algorithms.)
The Rekor transparency is not useful, no-one can tell from a record there what was actually signed (not even the name of the image!). It is sort of relevant for “keyless” signing (where the X.509 PKI is managed in a special way to issue short-term certificates, not useful for Fedora I think) only as a timestamp authority. Otherwise the major “benefit” of uploading to a Rekor server is that
cosign verifycan be used without a scary--insecure-ignore-tlogoption; but if Fedora wanted to maintain its own Rekor server, cosign users would need to add an option to point at it, anyway, so that’s not much of a benefit either.(cosign only supports the sigstore formats.) The two do interoperate, but
cosign signneeds to be invoked with--recursiveandsign-container-identity=$fullRepoWithTag.One more option is to have the build system sign the image at creation / initial upload time, as a proof of no tampering (perhaps with an “unreleased, for testing” key; or with a production key, but a testing-only image reference), and to have a later publish step do
skopeo copy --remove-signatures --sign-by…=…to add a “production” signature with a production key.In Podman+friends,
policy.jsondoes allow specifying a policy per repo (quay.io/fedora/fedora:*) as well. It would even be possible to specify a policy for every single repo:tag individually, but that doesn’t really scale. (There is no way to specify a single policy for a set of tags, e.g.quay.io/fedora/*:45, and I’d rather not add that flexibility; I’d push for re-arranging the repo structure toquay.io/fedora/43/{fedora,python}instead.)@thebeanogamer wrote in #6 (comment):
Yes, I plan to replace Robosignatory for a number of reasons. We also can't enable any of this until we've moved to Siguldry so that's not a problem. A very-much-not-near-ready branch is here.
For RPMs, we push signatures back into Koji. However, for images (including containers), we don't; it might be interesting to do that as well, but it may require a bit of work on the Koji side. If we did we could follow a similar dance to RPMs where things are tagged as "signing-pending" until a signature is attached. That way the image upload step can defer to running when a signature is pushed.
@jlebon wrote in #6 (comment):
Thanks for chiming in 😄
Yes, Konflux is largely an unknown. I looked around their documentation but didn't find details on signing anything with HSM-backed keys or anything like that. I've inquired in their Matrix channel, and if they respond I'll certainly do what I can to support it.
@mitr wrote in #6 (comment):
Thanks for the detailed overview of things. My initial thought is to start with cosign v1/v2 signatures. If there's a huge demand specifically for OpenPGP signatures, we can consider tackling those later.
Thank you all for the feedback you've given so far, I'm really grateful for the time you've spent replying.
To try and respond to a few points:
@jlebon wrote in #6 (comment):
I think this is a nice to have at this point, I'm confident that Sigstore will support PQC ciphers by the time that's cryptographically relevant. @jcline I don't know if Siguldry is actually in a position to support those algorithms yet, if we wanted to provide them today for OpenPGP users?
Oh... oh my... I guess if it works then sure?
Speaking as an outsider, Konflux does seem very cool but I've not yet had the pleasure of using it. I don't really have any insight into if/when Fedora's build pipelines would start migrating over. In any form this work is going to be a slow burn, but I don't know whether Konflux is worth waiting for or if we should investigate interim solutions, particularly given that I'd like the flow for Container Signing to look similar to all the other Signing we do.
@mitr wrote in #6 (comment):
Thank you for this, that's a much better explanation than I did of the three-ish signing formats.
You make a good point about the lack of a proper Certificate Authority system and us still needing to distribute the keys (something I actually like about the OpenPGP approach, given we already have to do that). Are X.509 hierarchies supported at all? I.e. can we have a root CA for all container signing, configure clients to trust that, then use intermediate CAs for signing individual releases? Just trying to think of ways to limit the scope of a key compromise.
Makes sense about the questionable usefulness if it's just a hash that gets published. It does help prove that the signing happens, but as you describe it the log isn't searchable in the way https://crt.sh is.
Coming from a Web PKI background, surely us hosting our own Certificate Transparency server defeats the point? The whole idea is that a third party is attesting to the fact that the signature happened.
This is an interesting point and maybe actually one for a wider discussion. In Robosignatory's current trust model, something justifies being signed because we pulled it from a trusted source and received an AMQ message telling us to. Technically nothing verifies that the artefact was actually built by CI and wasn't just dropped into CI's output directory/registry. The defense-in-depth nature of where this happens in CI means I'm not too concerned about the risk, but if we could think of a better way to do this then we should.
Is there any way to provide drop-in style config for this? It's currently a
%config(noreplace)which I wouldn't want to change, but that means from the distro perspective it's difficult to start enforcing those signature checks outside of new installations ofcontainers-common. I guess pretty few people will have modified the file, and those who have are the kind of users to explicitly add our key anyway.@jcline wrote in #6 (comment):
Cool, glad to know that work is underway. Happy to help however I can, but not being a Rust developer I'm probably not very useful.
In terms of the images currently published onto Quay in the
fedoranamespace, this is what we're looking at:These images have been published recently by
cloud-image-uploader:elneln-toolboxfedorafedora-sway-atomicfedora-budgie-atomicfedora-bootcfedora-toolboxfedora-iotfedora-minimalfedora-silverbluefedora-kinoiteThese images have been published recently by other means:
fedora-coreos&fedora-coreos-kubevirtpython-314)kickstart-artifactsThese images haven't been published this year and so can probably be ignored.
fedora-sericeafas2discourse-operatorcommunishift-authorization-operatorchariottkojiresultsdbmbox-operatorkerberosmbsbodhifedmsg-relaybodhiI think this means that excluding CoreOS and SCL images, we can perform the signing within the Fedora infrastructure before publishing them. I'm not entirely sure what we do about those, as we can't really ask them to rework all their CI just for this.
@thebeanogamer wrote in #6 (comment):
Client-side, it is in a good position to add support, but the OpenPGP standard is still in review, and there's no Sequoia release that supports it yet.
On the server side, the only reason it can't create PQC key types is there's ongoing discussion on the rust-openssl project about what that looks like. There's another (partial) wrapper, ossl, that I could use in the short term so that's not a complete blocker, but without client support I've not bothered.
I would plan for containers to get the same treatment the other artifacts do; when the relevant standards arrive we'll add support and submit a change proposal to discuss how we'll switch over. We likely want to provide support for "legacy" clients for a while since many distributions can consume the containers, which means older signature formats.
@thebeanogamer wrote in #6 (comment):
...snip...
It's this change:
https://fedoraproject.org/wiki/Changes/KickstartOciArtifacts
Yeah, some of those seem like internal fedora applications, not sure why they would have been pushed, but I guess they were at some time.
A hierarchy is supported:
policy.jsoncan only include to the top-level CA, with intermediate certificates shipped in each individual signature. But we have no revocation mechanisms at all, so it doesn’t help for protecting against an intermediate CA compromise.For resiliency on key compromise, the policy can typically be set up with multiple public keys / CA certs / …, and in that case any of them is accepted. So a policy could ship with a “live” public key and a backup one in cold storage, and revocation on compromise would happen by shipping a new
policy.jsonthat omits the compromised key (and perhaps adds a third one).(My general position on revocation is that on compromise of that kind, removing trust in the key is the easiest part; users would need to audit / prune their running systems of anything previously pulled which is signed by the now-compromised key. So there’s not that much to be gained by automating revocation, a lot of complexity, and given very rare use, a lot of risk to it.)
No. In the Web PKI, the transparency protects the shared namespace where hundreds of CAs can sign, but there is a single domain namespace shared across all of them, against a malicious CA.
With container images (assuming no cross-vendor PKI of that kind; and assuming the registry operator is not signing on behalf of clients — something that some users might want for public registries like
quay.io/docker.io), the holder of the signing key / CA certificate for a domain / registry is also authoritative for what images are published on that domain / registry. (And if a malicious container image is found and correctly signed, the nonrepudiable signature is already a proof of malfeasance without a transparency record; alternatively, a transparency record without having access to the image is not sufficient to tell that the signing was malicious.) So the Web PKI “transparency” mechanism is not really applicable at all, and a bad match / model.The Rekor transparency is advertised to exist as a protection against key compromise, where
IMHO it’s fairly tenuous, and most organizations 1) probably wouldn’t know that they should deploy such a watch service, 2) wouldn’t want to bother, and 3) might find it hard to argue that in case of compromise of their signing service, there is some other still trusted list of images that should be signed.
No, and so far I’ve been resisting that, because with drop-ins come conflicts and lack of clarity.
@jlebon wrote in #6 (comment):
I asked the Konflux team for details since they have no documentation on this front, and was pointed to this repository with no additional context.
If folks are both interested in Konflux and in having their content signed by Fedora keys, someone (not me) will have to do a bit more digging and documentation. Until evidence is presented otherwise, I'll assume Konflux already supports PKCS #11 and it's just a matter of configuration on Konflux's side.
Hi everyone, Fedora newbie here who works with Konflux and signing 😄
So, as far as I know, for container signing in upstream Konflux, there's only one task that does not require some Red Hat internal service (and it's for "keyless" Sigstore signing).
@jcline I might need more context to be understand what this support would mean for Fedora to answer this, but overall I would say Konflux does not have PKCS#11 support.
Again, I'm a newbie to the Fedora infra so I might not be aware of all the details, but from what I'm reading in this thread I think the fastest approach would be to create a new Tekton task that Konflux can use to talk to Sigul/Siguldry clients.
@robertohueso I think the questions about Konflux and PKCS#11 come from a lack of understanding about how and when signing happens in Konflux.
If signing is just another step in the user-controlled CI job, then yes we could probably just run a Siguldry client in there and sign using that. I was under the impression that Konflux also had some kind of external signing that it did automatically using Tekton Chains. If so, that's the bit we'd need to teach how to sign using PKCS#11.
I would like for all our images to at least have detached PGP signatures. That probably requires some kind of adjustment to Koji to add a storage mechanism for it. @mikem, is there a way to store detached signatures alongside the images themselves?
(I am personally uninterested in sigstore/cosign stuff as the trust model is fundamentally broken in my eyes, but if someone wants to do that work too for OCI stuff, be my guest.)
Konflux does leverage Tekton Chains to generate provenance for all artifacts that are produced (it can also sign artifacts, but I think that it should stop doing that: https://github.com/tektoncd/chains/issues/1346). Since Tekton Chains will happily generate this provenance and produce a signature based on a simple result (https://tekton.dev/docs/chains/slsa-provenance/#type-hinting), the attestation/signature doesn't have any meaning without first verifying it. Chains will just as happily attest a malicious package as it will a legitimate package.
What Konflux does provide, however, is a means to establish trust in artifacts and then re-sign those artifacts afterwards. In all of the current pipelines, this is done by running a policy check (i.e. Conforma) to ensure that the provenance meets all of the required properties. Only after this policy check passes will the rest of the tasks in the "release" process continue.
If the Fedora community wants to establish some other mechanism to sign artifacts, that is fine in my opinion. If there is no use for the sigstore stack to be used in Fedora for now, then it can be left out. If there comes a time when you want to produce multiple signatures, then it can always be added back in the future. One of the benefits of sigstore signatures is that there is a standard for storing these signatures right next to container images.
In principle, I think a build system should either be signing its outcomes or otherwise providing a trusted record of “we built this binary out of these sources” (where “trusted record” can be “we wrote it into a database and we trust it will never be tampered with”, sure, depending on specific security analysis).
Determining whether a many-megabyte binary is “malicious” is ~infeasible; the only way around that is to have all binaries built by a trusted build system recording what sources the binary came from. That allows inspecting the sources to see whether they are malicious, either before a build or afterwards (e.g. given specific indications of an attack involving the binary).
Now, that build-system signature doesn’t need to be exposed to users; it can be just one of the requirements for signing with a production key.