Konflux FAS Service Account #13341
Labels
No labels
after freeze
automation
backlog
blocked
change-ack
change-nak
change-noreleng
changes
Closed As
Can't Fix
Closed As
Duplicate
Closed As
Fixed
Closed As
Fixed with Explanation
Closed As
Get back later
Closed As
Grooming
Closed As
Insufficient data
Closed As
Invalid
Closed As
It's all good
Closed As
taiga
Closed As
upstream
day-to-day
dev
docs
easyfix
epel
f26
f27
f28
f29
f30
f31
f32
f33
f34
f35
f36
f37
f38
f39
f40
f41
f42
f43
f44
f45
fedora
groomed
high-gain
high-trouble
in-progress
in-review
investigation
legal
low-gain
low-trouble
mass rebuild
medium-gain
medium-trouble
meeting
mini-initiative
new_artifact
ops
pdc_retirement
rawhide
RCA
review
script
sidetarget
sprint-0
sprint-1
sprint-2
sprint-3
sprint-4
sprint-5
unfrozen
waiting on external
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
release-process
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
No milestone
No project
No assignees
8 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
releng/tickets#13341
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The Konflux community recently announced the support of the forge [1].
In Atomic Dekstops SIG, we would like to start building the different variants with Konflux but we are blocked with a security limitation: the Forge/Gitea only offers the creation of application tokens tied to individual user accounts.
That token is used by Konflux to 1. create/manage the Pipeline-as-code webhooks and 2. Interact with the repo (PR creation, feedback comment on PR, etc).
In Gitlab, application tokens are tied to the repo itself which is more manageable.
That token is set as a secret within the Fedora Konflux tenant [2] i.e: a kubernetes namespace. So, anyone with admin Role could access the personal user token which is not desirable. Secondly, all the Konflux operations would appear with the name and profile picture of that user, which would be misleading.
One option is to create a Konflux FAS Service Account user managed by the releng team. One would add it as contributor to the repo that needs to be onboarded in the Fedora Konflux cluster. And then, a releng admin would create a token per tenant, or SIG, or project, the granularity needs to be defined.
In some, that process would replicate the GitHub Konflux app.
Creating a dedicated FAS username is an option, but a general Konflux service account is more desirable since others will likely need one as well.
What are your thoughts?
[1] forge/forge#232
[2] https://gitlab.com/fedora/infrastructure/konflux/tenants-config/-/tree/main
@ryanlerch any thoughts here or more clever ways to do things?
I don't think we currently have an elegant solution for this. There is this upstream feature request. We might put it on the forge roadmap, but private issues are the top priority.
I think that we should create and maintain konflux-forge-bot FAS account for now, and maintain access tokens.
Right, https://codeberg.org/forgejo/forgejo/issues/4992 would be the long term fix.
If you're ok with the short term solution, could you provide us a token from that konflux-forge-bot FAS account please ? Then we'll be able to continue the Konflux integration.
I've created the fas account, logged into forge with it, so it exists there now (password is in ansible-private).
Questions:
xref https://gitlab.com/fedora/bootc/base-images/-/work_items/51
I think this secret needs to be managed by the people managing the Fedora Konflux instance, as it's a global thing right?
I could imagine though that we have a service account per forgejo namespace (in which case, the token would need to be provided per tenant namespace AFAIU) but I don't know if the Konflux Forgjeo code supports that or not.
Reading https://konflux-ci.dev/docs/building/creating-forgejo/ again, the token needs:
So if I understand correctly this means that we need to add the Konflux user as a member of the org where we want to use Konflux and grant it access to the repos.
But if we have a single user, and tokens are not scope by repos, wouldn't that mean that once we setup this token in Konflux then I can have Konflux tasks write to any repos in the Forge where Konflux is set up? Am I missing something?
So maybe we do need a per-org Konflux account so that we don't share the token and repo access to broadly.
So this got me back to what Colin suggested: we likely need a service account per Forgejo namespace/org.
@siosm wrote in #13341 (comment):
There are 2 types of tokens in Forgejo, broad token for all the repos account has access to or token for specific repository
@humaton wrote in #13341 (comment):
Not sure if the repo scoped token can be used for Konflux needs. I was getting "token with not enough permissions" with it.
Konflux needs: issue r+w, org r, repo r+w, user r.
Repo scoped token can have max repo r+w, issue r+w
Edit: actually Konflux needed write:user as well IIRC
I'm confused now. I though that repo scoped token was still an RFE: https://codeberg.org/forgejo/forgejo/issues/4992
@siosm wrote in #13341 (comment):
That request is about Repository tokens, We are talking about User tokens with repository scope.
Thinking about this, for now we will have to create
<orgname>-konflux-botaccount for interested parties. If this account/tokens need to be under control of 3rd party we should require 2fa for each of those accounts so we prevent account hijack by password leak.Well, I don't see a big problem with having a global konflux service account for forge.fedoraproject.org, as long as it's clearly maintained by a joint forge.fedoraproject.org/Konflux setup.
IOW I think our next step here is to ask the Fedora Konflux ops team to take ownership of this service account (we file a ticket with them?)
I filed https://redhat.atlassian.net/browse/KFLUXSPRT-8326 - I couldn't find offhand a ticketing path for the Fedora instance
@humaton wrote in #13341 (comment):
OK, can we start with the
atomicandatomic-desktopsorgs and create accounts and tokens for access to use with Konflux? That should let us to get started building with Fedora's Konflux instance for those two projects for now.@walters wrote in #13341 (comment):
Doesn't that mean that anyone with access to the token would get full access to all repos on the forge? That sounds really dangerous.
Properly scoped token are apparently not ready:
Scoped token have been added in Forgejo 15.0 but apparently according to Lenka it's too limited: https://forgejo.org/2026-04-release-v15-0/#repository-specific-access-tokens
Yes, the point is that the token would be a secret managed by the Fedora Konflux admins, not by any individual tenant.
Regarding "full access" though - I haven't looked but if forgejo suports what github does ("all changes must come through a PR", "no force pushes to git main", and crucially "bot tokens can't approve PRs") it really limits potential damage from service accounts like this.
Hi, I am from Konflux infrastructure team.
We need to discuss how we will make this SA token available in the different tenant namespaces and how/who will be responsible to rotate the token.
On Konflux side, we do use the ExternalSecretOperator so one option I can think of is to setup a ClusterSecretStore, connected to some secret management system(e.g. vault) where the token would be stored. Then each tenant could create ExternalSecret in their NS that will extract the token. When token is rotated at the source, it would automatically propagate. If all tenant need this, we can also create a ClusterExternalSecret which will create the k8s secret in all namespace, we can do ns selection with label or something.
That said, this would mean that all the admins of all the namespaces where this token/secret is created would have access to all the repos.
When I looked at this it seems Konflux for Github relies on a global app which acts as an integration point.
I think since we have the source code of and control both ends here, nothing stops us from creating something similar. However right now, I think PaC for Forgejo only supports per-tenant secrets - we'd need to fix that first.
Again - I don't think we need (or want) a separate service account per tenant.
OK I finally sat down and spent some tokens, here's some LLM generated research - I didn't look at the code myself, but this sounds sane:
Generated-by: https://github.com/cgwalters/cgwalters#llms
Konflux Forgejo integration: global
konflux-bottokenProblem
Fedora's Konflux instance needs to interact with forge.fedoraproject.org
(Forgejo) — managing webhooks, setting commit status, posting PR comments.
Forgejo doesn't have GitHub Apps, so authentication uses a personal access
token tied to a user account.
The current approach requires each tenant namespace to have its own copy of
this token as a Kubernetes secret. This means every tenant admin can read the
token, and all Konflux activity appears under an individual user's name.
Solution
PaC (Pipelines-as-Code) already supports global repository settings. A
single
RepositoryCR in the PaC controller namespace can provide gitprovider credentials that are inherited by all tenant
RepositoryCRs thatdon't specify their own. The token never needs to be copied into tenant
namespaces via ExternalSecretOperator or any other mechanism — PaC resolves
it at runtime from the controller namespace.
For the current use case (public repos, trusted tenants), this is sufficient
with zero code changes.
Setup
1. Forgejo bot account (already done)
Kevin created the
konflux-botFAS account and logged into the forge withit. Password is in ansible-private. The account needs:
(e.g.
atomic,atomic-desktops)2. Create a token
Generate a token for
konflux-botathttps://forge.fedoraproject.org/user/settings/applicationswith scopes:read:issue,write:issueread:organizationread:repository,write:repositoryread:user3. Store the token as a Kubernetes secret in the PaC controller namespace
And a webhook secret (shared between PaC and the Forgejo webhooks):
These can be created manually, via Ansible, or via a single
ExternalSecretpulling from Vault — whatever Fedora infra prefers formanaging secrets in the controller namespace. The important thing is that
these secrets exist only in the controller namespace, not in tenant
namespaces.
4. Create the global Repository CR
The CR must be named
pipelines-as-code(the default) unless the controlleris configured with a different name via
PAC_CONTROLLER_GLOBAL_REPOSITORY.It must live in the PaC controller namespace.
5. Tenant setup
Tenants create
RepositoryCRs in their namespace withoutgit_provider.secret. PaC'sSetupAuthenticatedClientdetects the missingsecret, falls back to the controller namespace, and merges the global
credentials:
The
git_providerblock must be present (even without asecretfield).If it's omitted entirely (
git_provider: nullor missing), PaC skips themerge logic and the global fallback won't activate. The
typefield can beeither
forgejoorgitea— PaC treats them as equivalent.No per-tenant secret provisioning. No ExternalSecretOperator. No
ClusterExternalSecret. Tenants don't need access to the token at all.
How it works at runtime
When a webhook fires:
RepositoryCR, seesgit_provider.secretis nil, and detects the globalRepositoryCR(named
pipelines-as-code) in the controller namespace.secretNSto the controller namespace (not the tenant namespace)before merging the global CR's settings into the local CR. This ensures
the secret is read from the right place.
controller namespace via
GetSecret(namespace=secretNS, ...).signature, setting commit status, posting comments on the PR.
basic-auth secret so Tekton can clone the repo. For public repos this is
redundant (clone works without auth), but PaC does it when
SecretAutoCreationis enabled (the default).Security for the current situation
The repos on forge.fedoraproject.org are public, and the tenants are trusted
Fedora contributors. The token does leak into tenant namespaces via the
basic-auth secret, but:
public.
trusted users.
until a PipelineRun actually happens; the global secret stays in the
controller namespace.
This is acceptable for the current situation and unblocks Atomic Desktops
and other SIGs immediately.
Why not ExternalSecretOperator to distribute to tenants?
Hugares suggested using ESO with a
ClusterExternalSecretto push the tokeninto every tenant namespace. This is the opposite of what we want:
readable by every tenant admin.
propagation) to solve a problem that doesn't exist with the global
Repository CR approach.
Vault (a single
ExternalSecret), but distributing it to tenants isunnecessary and counterproductive.
Future hardening (when private repos or untrusted tenants arrive)
The global static token approach works for public repos with trusted tenants.
When private repos or untrusted tenants enter the picture, the token leaking
into tenant namespaces becomes a real concern. At that point:
PaC's Forgejo provider can be extended to mint short-lived, repo-scoped
clone tokens (using the Forgejo admin API) and revoke them after each
PipelineRun. The full-scope token stays in the controller namespace; only a
narrow
read:repositorytoken scoped to one repo enters the tenantnamespace.
Forgejo upstream could add token TTL (
expires_at) and allow token-basedauth for the token management API (currently requires basic auth for the
user endpoint, though the admin endpoint works with token auth).
These are code changes for later. The immediate path is purely operational.
this looks good, since this is a secret/token managed at Konflux level, I could put it the konflux vault, along side other konflux secret (e.g. GH app keys) and get ESO to extract this in PAC NS.
Awesome! Can you sync with the Fedora Infra folks to get that secret mentioned above?
That said, the bot user is solving the "Konflux can post to Fedora Forge" problem - but we actually need to ensure we have webhooks setup to handle edge triggering the other way ("Forge event" -> Konflux). That was mentioned above briefly but I believe needs doing too:
Here's another LLM-assisted dump:
Forgejo system webhook setup for Fedora Konflux
This covers only the webhook — how Forgejo delivers events to PaC. The
bot account and API token are a separate concern (see forgejo.md).
What the webhook does
When someone pushes or opens a PR on forge.fedoraproject.org, PaC needs
to know about it. Forgejo delivers that notification via an HTTP POST to
the PaC controller's ingress URL. PaC checks the HMAC-SHA256 signature
to verify the payload is authentic, then looks up a matching Repository
CR by repo URL. If one exists, it triggers a pipeline. If not, it
silently ignores the event.
Why a system webhook
Forgejo supports per-repo webhooks and site-wide system webhooks. A
system webhook fires for events across all repos on the instance.
Per-repo webhooks require someone (or the
build-serviceoperator) tocreate a webhook on each repo via the Forgejo API, which needs a token
with repo write access and code changes to
build-serviceto support aglobal Forgejo credential. System webhooks skip all of that — one
webhook covers the entire forge instance.
PaC handles this fine. It's the same pattern as GitHub Apps: one webhook
endpoint, events for all repos, matched by URL against Repository CRs.
Unmatched events are discarded before any API calls or secret resolution.
Setup
Fedora infra and Konflux admins need to agree on a shared webhook secret.
Generate one (e.g.
openssl rand -hex 32) and exchange it out of band.Fedora infra (Forgejo site admin)
Create a system webhook. Via the admin UI (Site Administration →
Webhooks → Add Webhook) or via API:
Fedora infra needs the PaC controller ingress URL from the Konflux
admins. This is a one-time setup.
Konflux admins (PaC controller namespace)
Store the shared secret in a Kubernetes secret:
Reference it in the global Repository CR's
webhook_secretfield. (Theglobal CR also holds the bot token reference — see forgejo.md for that
side.)
Rotation
If the secret needs to rotate, both sides update at the same time:
Fedora infra updates the system webhook config, Konflux admins update
the Kubernetes secret. There's no automatic rotation — it's a shared
secret exchanged over a secure channel.
Is it possible to have a service account per Forgejo namespace/org ? We'd limit the scope per namespace.One solution @siosm came up with earlier in a meeting is to create a mailing list for each namespace/org with the owners on it. Then, we'd create the FAS account with that email, and finally add it all the required permissions to the repos we want to onboard in Konflux.
EDIT: forget this comment. I opened that page before Colin came up with his better plan
@jcapitao I know we discussed that in the meeting, but while that was running I generated the above, which I think will avoid the service-account-per-namespace and I believe that will be better - do you agree?
@walters wrote in #13341 (comment):
@walters fully agree. I edited my previous comment explaining the reason. Let's continue with your plan 👍
https://redhat.atlassian.net/browse/KFLUXSPRT-8326 was moved to https://redhat.atlassian.net/browse/KFLUXINFRA-4162.
Once we've set up a few repos, we should write a guide / doc to enroll your repo and setup things properly:
konflux-forge-botuser to org members (wrong username in LLM output)konflux-forge-botor direct pushes at all without a reviewI've added the bot user to https://accounts.fedoraproject.org/group/forge-atomic-desktops-members/ & https://accounts.fedoraproject.org/group/forge-atomic-members/
I've generated the token and will get it to the konflux folks (hopefully tomorrow).
I got the token, uploaded it to Vault used by Konflux. Here is the PR I have to setup the global PAC repository: https://github.com/redhat-appstudio/infra-deployments/pull/13285
The default global PAC repository is done, we need to create the webhook on Forge side
I sent info to @kevin for the webhook to setup on the forge side, once this is done, we will be able to test the whole flow
@walters @jcapitao @siosm Both global repo and webhook have been setup, now time to try it out in a tenant NS.
Do one of you have time to test this? I am going on PTO for 2 weeks so I would have setup a repo/build to validate but do not have time before I leave.
FYI @maniskum will be there to if any help needed from Konflux infra side for next 2 weeks
Hi Hugo, happy to help! Maybe we can hop in a synchronous meeting?
We're in https://meet.google.com/yfv-knyq-onp?authuser=0
I made https://forge.fedoraproject.org/atomic/ci-sandbox
We tested this by creating and application and component in NS atomic-tenant but we did not go really far. The Konflux build-service, responsible to configure repository CR at NS level, base on the component, does not work if we do not have a valid forge token at the NS level.
TLDR: To support this repository inheritance, we need to backlog a feature for Konflux to support this use case.
2 options:
OK so now we need to re-discuss this (from above):
What is too limited? OK hmm I spent some processing time and tokens on this:
Generated-by: https://github.com/cgwalters/cgwalters#llms
TL;DR: Forgejo tokens vs. Konflux
Forgejo has no org-scoped tokens (only "All", "Public-only", or a per-repo
list, and the per-repo list is hard-restricted to
repository+issuescopeonly — no
organization/user). Org-scoping has been requested upstreamsince 2023 (
forgejo#1712) but never shipped; only repo-list scoping landed.Pipelines-as-Code (used by Konflux) only needs the extra
organization/userscope for two narrow things:read:organization— PaC's team-based ACL policy (allowedTeams),avoidable by using OWNERS-file/collaborator trust instead (no code change,
just a config choice).
read:user— a one-off self-ID lookup (GetMyUserInfo) so PaC knows whichPR comment is its own before editing it in place. PaC's
RepositoryCRalready has an unused
spec.git_provider.userfield that could replacethis live lookup with a configured username — a small (~10-15 line) patch
to
pkg/provider/gitea/gitea.go, not yet implemented anywhere upstream.Bottom line: with an OWNERS-based ACL config choice + a small upstream
PaC patch, a plain Forgejo repo-scoped token (
write:repository+write:issue) would be enough for Konflux — no global "All repos" bot tokenneeded, and no dependency on Forgejo's unimplemented org-scoping.
I guess there is kind of another concept, we could stand up a little custom proxy service in front of forgejo and have konflux talk to that, and basically just automatically allow those two permissions (intercept those two APIs and provide another token which allows them)...or just patch our forgejo to allow them by default - I mean, they're just read permissions and I don't think we have much in the way of private data here at all...
OK there's another variant, I spent some more (cpu, tokens, bit of human brainpower) on this:
And the more I think about things the more I think this is really going to be best, because no pipelines-as-code means no crappy comment spam...just clean forgejo CI status.
Generated-by: https://github.com/cgwalters/cgwalters#llms
Forgejo Action → Konflux direct trigger (static secret)
Basically the Forgejo equivalent of what RHEL-on-GitLab (RoG) already does in
production: a CI job holds a long-lived, namespace-scoped Kubernetes token as
a secret, renders a
PipelineRunpinned to a pipeline definition via the gitresolver,
oc applys it, streams logs back into its own job output, andreports status — no Pipelines-as-Code webhook/
RepositoryCR involved.Requires an existing Konflux tenant namespace with RBAC to create
PipelineRuns and read pod logs in it (no bootstrap possible without that).Shape of the workflow
.forgejo/workflows/build.yml:pipelinerun-template.yaml(rendered withenvsubstbeforeapply):trigger-build.sh:Optionally report final status back via Forgejo's own built-in Actions
token (no Konflux-held Forgejo credential needed for that half).
Future enhancement
Swap the
oc login --token=...step for an OIDC token exchange (ForgejoActions can mint short-lived OIDC ID tokens today) traded for a scoped,
per-run Kubernetes credential via a small broker service. Everything else
above (render, apply, tail, wait, report) stays the same. That removes the
long-lived secret and generalizes to more repos via a reviewed mapping table
instead of one hand-provisioned secret each — worth doing once this is
proven out, not a blocker to starting.
Is there anything blocking us from trying that last solution (Forgejo Action → Konflux direct trigger (static secret))? If not, I would say let's try it
I like the idea to bypass Pipeline-as-Code, at least for now.
I filed https://github.com/konflux-ci/support/issues/26 to request the Role with the right permisions, and also the authentication token.
I'll test it with https://forge.stg.fedoraproject.org/atomic-desktops/jcapitao-konflux-test/pulls/1 (pipeline template and scripts are not ready yet)
Do we need that? I don't think we need a global role, we can just add it to https://gitlab.com/fedora/infrastructure/konflux/tenants-config/-/tree/main/clusters/kflux-fedora-01/tenants/atomic-tenant?ref_type=heads right?
Last time I tried to create a custom role, ArgoCD didn't have enough permissions to apply it. IIRC there were some kyverno rule that prevented the operation. A Konflux admin had to create it manually c.f. https://gitlab.com/fedora/infrastructure/konflux/tenants-config/-/merge_requests/143
But I'll give another try, let's see
Oh hmm. But do we need a role? Can't we just create a serviceaccount using one of the stock roles?
I managed to trigger a PipelineRun from a Forgejo runner e.g https://konflux-ui.apps.kflux-fedora-01.84db.p1.openshiftapps.com/ns/atomic-tenant/applications/fedora-silverblue-jcapitao-test/pipelineruns/fedora-silverblue-jcapitao-test-on-pr-jvbqh with https://forge.stg.fedoraproject.org/atomic-desktops/jcapitao-konflux-test/pulls/1
As Colin mentioned, I reused a stock role c.f. https://gitlab.com/fedora/infrastructure/konflux/tenants-config/-/merge_requests/231/diffs?commit_id=d313d351636b98b5fd9c9bab686de0a0e91af843. Though that ClusterRole is missing permission to read logs, see https://github.com/konflux-ci/support/issues/26#issuecomment-5204911658.
I think we should move on with that solution 👍 while awaiting for a proper support of Pipeline-as-Code and Forgejo integration.