zlopez commented

Owner

Describe what you would like us to do:

As the investigation is now done, we can follow the document and do the migration. It will be best to wait after Fedora 39 will be out as this will allow us to get rid of plenty of legacy workflow.

We will need some transition period, so the users still can access the content on registry.fp.o until we move everything on or it's no longer needed.

When do you need this to be done by? (YYYY/MM/DD)

Not urgent, needs to wait till F39 will be out.

# Describe what you would like us to do: ---- As the [investigation](https://pagure.io/fedora-infrastructure/issue/10386) is now done, we can follow the [document](https://fedora-arc.readthedocs.io/en/latest/registry_to_quay/index.html) and do the migration. It will be best to wait after Fedora 39 will be out as this will allow us to get rid of plenty of legacy workflow. We will need some transition period, so the users still can access the content on registry.fp.o until we move everything on or it's no longer needed. # When do you need this to be done by? (YYYY/MM/DD) ---- Not urgent, needs to wait till F39 will be out.

rishi commented

2023-11-30 21:36:40 +00:00

Just to repeat what I mentioned a while before on the investigation ticket.

Toolbx still uses registry.fedoraproject.org for its fedora-toolbox images. From Fedora 39 onwards these images are produced by Fedora's nightly composes.

We are open to moving to quay.io but we need some migration period because the URL to the image is baked into the toolbox RPM, and we can't change it without knowing exactly what it would be.

Just to repeat what I mentioned a while before on the [investigation](https://pagure.io/fedora-infrastructure/issue/10386#comment-806812) ticket. Toolbx still uses `registry.fedoraproject.org` for its `fedora-toolbox` images. From Fedora 39 [onwards](https://fedoraproject.org/wiki/Changes/ToolbxReleaseBlocker) these images are produced by Fedora's nightly composes. We are open to moving to `quay.io` but we need some migration period because the URL to the image is baked into the `toolbox` RPM, and we can't change it without knowing exactly what it would be.

kevin commented

2023-11-30 22:54:34 +00:00

Owner

FWIW, IMHO, we should try and redirect existing registry.fedoraproject.org to quay.io if tools will understand that. Or I guess proxy them if not. At least initially. This will allow us to 'fail back' if there's problems or we need to down the road. But I am not sure the details on if thats possible.

zlopez commented

2024-01-18 15:25:52 +00:00

Author

Owner

Metadata Update from @zlopez:

Issue assigned to t0xic0der

**Metadata Update from @zlopez**: - Issue assigned to t0xic0der

kevin commented

2024-01-24 21:03:01 +00:00

Owner

So, there is some question about how we want to do things.

I personally think we should keep things pointed to the registry.fedoraproject.org, and then that redirects them to quay.io for the actual image.
The advantage of this is that we control it, and can repoint it somewhere else if we like, or run our own registry for some part of it and redirect other things.
It also means we can change it in our infra and it immediately would take effect.

If on the other hand, we change everything to say 'use quay.io', there's the advantage that we don't have to redirect and it's all on them.
But the downside is that if ever we need to change it again, we will have to change it in all the places it's currently set (documentation, podman, toolbox, etc).
Any changes will take rebuilding things and pushing out updates and be a long and slow process.

Can folks chime in with their thoughts on this so we can decide how to approach it?

So, there is some question about how we want to do things. I personally think we should keep things pointed to the registry.fedoraproject.org, and then that redirects them to quay.io for the actual image. The advantage of this is that we control it, and can repoint it somewhere else if we like, or run our own registry for some part of it and redirect other things. It also means we can change it in our infra and it immediately would take effect. If on the other hand, we change everything to say 'use quay.io', there's the advantage that we don't have to redirect and it's all on them. But the downside is that if ever we need to change it again, we will have to change it in all the places it's currently set (documentation, podman, toolbox, etc). Any changes will take rebuilding things and pushing out updates and be a long and slow process. Can folks chime in with their thoughts on this so we can decide how to approach it?

rishi commented

2024-01-25 02:33:22 +00:00

So, there is some question about how we want to do things.

I personally think we should keep things pointed to the registry.fedoraproject.org, and then that redirects them to quay.io for the actual image.
The advantage of this is that we control it, and can repoint it somewhere else if we like, or run our own registry for some part of it and redirect other things.
It also means we can change it in our infra and it immediately would take effect.

If on the other hand, we change everything to say 'use quay.io', there's the advantage that we don't have to redirect and it's all on them.
But the downside is that if ever we need to change it again, we will have to change it in all the places it's currently set (documentation, podman, toolbox, etc).
Any changes will take rebuilding things and pushing out updates and be a long and slow process.

Yes, I am with you on this one, assuming that the underlying technical realities let us use registry.fedoraproject.org as a front.

A hard switch to quay.io can be done later once things have settled and we are confident about it. Until then, the possibly of a revert and all the associated churn, makes it wise to continue pointing the clients to registry.fedoraproject.org.

> So, there is some question about how we want to do things. > > I personally think we should keep things pointed to the registry.fedoraproject.org, and then that redirects them to quay.io for the actual image. > The advantage of this is that we control it, and can repoint it somewhere else if we like, or run our own registry for some part of it and redirect other things. > It also means we can change it in our infra and it immediately would take effect. > > If on the other hand, we change everything to say 'use quay.io', there's the advantage that we don't have to redirect and it's all on them. > But the downside is that if ever we need to change it again, we will have to change it in all the places it's currently set (documentation, podman, toolbox, etc). > Any changes will take rebuilding things and pushing out updates and be a long and slow process. Yes, I am with you on this one, assuming that the underlying technical realities let us use `registry.fedoraproject.org` as a front. A hard switch to `quay.io` can be done later once things have settled and we are confident about it. Until then, the possibly of a revert and all the associated churn, makes it wise to continue pointing the clients to `registry.fedoraproject.org`.

sumantrom commented

2024-01-25 03:18:56 +00:00

So, there is some question about how we want to do things.

I personally think we should keep things pointed to the registry.fedoraproject.org, and then that redirects them to quay.io for the actual image.
The advantage of this is that we control it, and can repoint it somewhere else if we like, or run our own registry for some part of it and redirect other things.
It also means we can change it in our infra and it immediately would take effect.

If on the other hand, we change everything to say 'use quay.io', there's the advantage that we don't have to redirect and it's all on them.
But the downside is that if ever we need to change it again, we will have to change it in all the places it's currently set (documentation, podman, toolbox, etc).
Any changes will take rebuilding things and pushing out updates and be a long and slow process.

Yes, I am with you on this one, assuming that the underlying technical realities let us use registry.fedoraproject.org as a front.

A hard switch to quay.io can be done later once things have settled and we are confident about it. Until then, the possibly of a revert and all the associated churn, makes it wise to continue pointing the clients to registry.fedoraproject.org.

I am also +1 to this approach of keeping registry.fp.o as a front and then redirecting. I still would want to check out the latency situation after the redirect. Any chance you can give us a timeline for testing?

> > So, there is some question about how we want to do things. > > > > I personally think we should keep things pointed to the registry.fedoraproject.org, and then that redirects them to quay.io for the actual image. > > The advantage of this is that we control it, and can repoint it somewhere else if we like, or run our own registry for some part of it and redirect other things. > > It also means we can change it in our infra and it immediately would take effect. > > > > If on the other hand, we change everything to say 'use quay.io', there's the advantage that we don't have to redirect and it's all on them. > > But the downside is that if ever we need to change it again, we will have to change it in all the places it's currently set (documentation, podman, toolbox, etc). > > Any changes will take rebuilding things and pushing out updates and be a long and slow process. > > Yes, I am with you on this one, assuming that the underlying technical realities let us use `registry.fedoraproject.org` as a front. > > A hard switch to `quay.io` can be done later once things have settled and we are confident about it. Until then, the possibly of a revert and all the associated churn, makes it wise to continue pointing the clients to `registry.fedoraproject.org`. I am also +1 to this approach of keeping registry.fp.o as a front and then redirecting. I still would want to check out the latency situation after the redirect. Any chance you can give us a timeline for testing?

siosm commented

2024-02-12 12:06:46 +00:00

I personally think we should keep things pointed to the registry.fedoraproject.org, and then that redirects them to quay.io for the actual image.
The advantage of this is that we control it, and can repoint it somewhere else if we like, or run our own registry for some part of it and redirect other things.
It also means we can change it in our infra and it immediately would take effect.

Oh, this is indeed a good idea. We started "migrating" to quay.io images in Fedora CoreOS land but this looks like a better approach if this works well.

> I personally think we should keep things pointed to the registry.fedoraproject.org, and then that redirects them to quay.io for the actual image. > The advantage of this is that we control it, and can repoint it somewhere else if we like, or run our own registry for some part of it and redirect other things. > It also means we can change it in our infra and it immediately would take effect. Oh, this is indeed a good idea. We started "migrating" to quay.io images in Fedora CoreOS land but this looks like a better approach if this works well.

ppisar commented

2024-02-27 08:52:54 +00:00

May I ask what's the official place for Fedora containers now?

A documentation https://docs.fedoraproject.org/en-US/containers/#_where_to_find_the_containers points to registry.fedoraproject.org where the images were not updated since November 2023 https://registry.fedoraproject.org/repo/fedora/tags/.

May I ask what's the official place for Fedora containers now? A documentation <https://docs.fedoraproject.org/en-US/containers/#_where_to_find_the_containers> points to registry.fedoraproject.org where the images were not updated since November 2023 <https://registry.fedoraproject.org/repo/fedora/tags/>.

zlopez commented

2024-02-27 11:41:23 +00:00

Author

Owner

Most of the projects hosted on registry.fp.o either moved or moving to quay.io. There is a Fedora organization on quay.io, where you should find most of the images.

Most of the projects hosted on registry.fp.o either moved or moving to quay.io. There is a [Fedora organization on quay.io](https://quay.io/organization/fedora), where you should find most of the images.

kevin commented

2024-02-27 15:33:10 +00:00

Owner

Yes, but note... IMHO we should just keep using registry.fedoraproject.org and have it redirect, so in that case we shouldn't need to change any docs.

rishi commented

2024-02-27 15:34:14 +00:00

May I ask what's the official place for Fedora containers now?

Yes, just keep using registry.fedoraproject.org. :)

> May I ask what's the official place for Fedora containers now? Yes, just keep using `registry.fedoraproject.org`. :)

jlebon commented

2024-04-29 16:42:33 +00:00

To clarify, are we saying that registry.fp.o should still be the canonical repo and that Quay.io is an implementation detail, or that the redirect is more of a backcompat thing but that the expectation is that eventually we shut down registry.fp.o? Clearly, the quay.io repos have already gone around and been documented quite some time now, so I don't know how realistic it is to consider it an implementation detail at this point.

A downside I raised in https://github.com/coreos/fedora-coreos-config/pull/2988 is that using registry.fp.o means that service is now also in your dependency path. Being dependent on less things being up is very valuable in a CI context.

To clarify, are we saying that registry.fp.o should still be the canonical repo and that Quay.io is an implementation detail, or that the redirect is more of a backcompat thing but that the expectation is that eventually we shut down registry.fp.o? Clearly, the quay.io repos have already gone around and been documented quite some time now, so I don't know how realistic it is to consider it an implementation detail at this point. A downside I raised in https://github.com/coreos/fedora-coreos-config/pull/2988 is that using registry.fp.o means that service is now also in your dependency path. Being dependent on less things being up is very valuable in a CI context.

kevin commented

2024-04-30 01:27:29 +00:00

Owner

Yeah, I guess I was imagining that we would keep registry.fedoraproject.org for a while redirecting to quay.io and make sure there were not any big issues that would cause us to roll back.
If we did have to roll back we just change dns, instead of updaing a bunch of packages, etc.

You do make a good point that it adds another 'hop' in there and also makes us a dependency...

Yeah, I guess I was imagining that we would keep registry.fedoraproject.org for a while redirecting to quay.io and make sure there were not any big issues that would cause us to roll back. If we did have to roll back we just change dns, instead of updaing a bunch of packages, etc. You do make a good point that it adds another 'hop' in there and also makes us a dependency...

dustymabe commented

2024-04-30 15:52:11 +00:00

Member

Yeah, I guess I was imagining that we would keep registry.fedoraproject.org for a while redirecting to quay.io

I think it's fine to keep the redirector, but maybe we don't promote it? i.e. if people are using registry.fedoraproject.org then it would keep working.. but our docs and things all get updated to the new location?

> Yeah, I guess I was imagining that we would keep registry.fedoraproject.org for a while redirecting to quay.io I think it's fine to keep the redirector, but maybe we don't promote it? i.e. if people are using registry.fedoraproject.org then it would keep working.. but our docs and things all get updated to the new location?

rishi commented

2024-08-12 17:43:36 +00:00

What's the latest thinking on this topic? Is there still a possibility that we might roll back away from quay.io and point registry.fedoraproject.org at something else?

I am asking because I want to figure out if Toolbx should start using the new quay.io URLs or stick to registry.fedoraproject.org.

What's the latest thinking on this topic? Is there still a possibility that we might roll back away from `quay.io` and point `registry.fedoraproject.org` at something else? I am asking because I want to figure out if Toolbx should start using the new `quay.io` URLs or stick to `registry.fedoraproject.org`.

kevin commented

2024-08-12 19:10:09 +00:00

Owner

The plan is definitely still to go to quay.io, just too many fires burning to get to it.

We need to sort out flatpak stuff still.

If you want to go go quay.io it should be already in sync for toolbox, so go for it.

The plan is definitely still to go to quay.io, just too many fires burning to get to it. We need to sort out flatpak stuff still. If you want to go go quay.io it should be already in sync for toolbox, so go for it.

rishi commented

2024-08-12 19:13:01 +00:00

The plan is definitely still to go to quay.io, just too many fires burning to get to it.

We need to sort out flatpak stuff still.

If you want to go go quay.io it should be already in sync for toolbox, so go for it.

Ok, understood. Thanks, Kevin.

> The plan is definitely still to go to quay.io, just too many fires burning to get to it. > > We need to sort out flatpak stuff still. > > If you want to go go quay.io it should be already in sync for toolbox, so go for it. Ok, understood. Thanks, Kevin.

zlopez commented

2025-01-07 11:49:40 +00:00

Author

Owner

This is still waiting for the flatpak to migrate.

yselkowitz commented

2025-01-10 19:17:37 +00:00

@otaylor is much more familiar with the delivery mechanics of flatpaks than I am, but not sure how much time he has for this right now. In his absence, I guess it will be up to me, but I'll need to be brought up to speed.

zlopez commented

2025-02-12 15:44:17 +00:00

Author

Owner

I'm now trying to look at pointing the flatpak-indexer to quay.io and see if that works.

zlopez commented

2025-03-06 08:59:11 +00:00

Author

Owner

Finally got to it and found out where to obtain test data for tests. I will need to remove the ODCS part as this is causing for some reason issues on staging.

zlopez commented

2025-03-13 09:47:23 +00:00

Author

Owner

Currently trying to get the flatpak-indexer running on my own machine, but I hit this issue and waiting for some solution https://github.com/fedora-infra/fedora-messaging/issues/440

smilner commented

2025-09-22 17:47:22 +00:00

@zlopez are we still in process for this migration?

zlopez commented

2025-09-23 03:23:46 +00:00

Author

Owner

@smilner I still have this on my TODO list, but currently have other priorities.

zlopez commented

2025-10-20 13:21:35 +00:00

Author

Owner

Metadata Update from @zlopez:

Issue tagged with: sprint-0

**Metadata Update from @zlopez**: - Issue tagged with: sprint-0

zlopez commented

2025-10-20 13:21:49 +00:00

Author

Owner

Metadata Update from @zlopez:

Issue assigned to zlopez (was: t0xic0der)

**Metadata Update from @zlopez**: - Issue assigned to zlopez (was: t0xic0der)

rishi commented

2025-12-17 23:37:50 +00:00

@pingou submitted a pull request to move Toolbx away from registry.fedoraproject.org URLs to their quay.io counterparts. I am curious if the registry.fedoraproject.org URLs are about to stop working soon. If so, we need to migrate toolbox(1) away from them on older distributions.

@pingou submitted a [pull request](https://github.com/containers/containertoolbx.org/pull/55) to move Toolbx away from `registry.fedoraproject.org` URLs to their `quay.io` counterparts. I am curious if the `registry.fedoraproject.org` URLs are about to stop working soon. If so, we need to migrate `toolbox(1)` away from them on older distributions.

zlopez commented

2025-12-18 08:26:05 +00:00

Author

Owner

The last thing remaining is to point flatpak-indexer to quay.io, but not sure how long does that take.

So the registry.fedoraproject.org will not go away anytime soon.

The last thing remaining is to point `flatpak-indexer` to `quay.io`, but not sure how long does that take. So the `registry.fedoraproject.org` will not go away anytime soon.

rishi commented

2026-01-20 01:18:21 +00:00

My apologies. I should have been more specific in my question.

Does it mean that registry.fedoraproject.org will be going away after flatpak-indexer is pointed at quay.io? Any rough idea of what soon looks like?

I am asking because we will have to migrate away from registry.fedoraproject.org on distributions as old as RHEL 8.10, where it's impossible to justify casual changes. So, we need some lead time to prepare for it.

I am wondering if we should already start at least migrating the code and the documentation in newer releases because this is a matter of time, or we can let the status quo continue for the foreseeable future.

My apologies. I should have been more specific in my question. Does it mean that `registry.fedoraproject.org` will be going away after `flatpak-indexer` is pointed at `quay.io`? Any rough idea of what *soon* looks like? I am asking because we will have to migrate away from `registry.fedoraproject.org` on distributions as old as RHEL 8.10, where it's impossible to justify casual changes. So, we need some lead time to prepare for it. I am wondering if we should already start at least migrating the code and the documentation in newer releases because this is a matter of time, or we can let the status quo continue for the foreseeable future.

zlopez commented

2026-01-20 10:14:55 +00:00

Author

Owner

@kevin What is the plan for registry.fedoraproject.org after we migrate everything from it. Will it stay or do we want to decommission it?

kevin commented

2026-01-20 23:33:41 +00:00

Owner

My orig plan was to keep it around, but after talking with folks and thinking about it more, I think we want to retire it.
That said, there's no date currently, we need to sort out flatpaks and then anything else still left and only then we can set a retirement date.

So, I wouldn't stress over it. We also are happy to work with stakeholders on setting that date...

My orig plan was to keep it around, but after talking with folks and thinking about it more, I think we want to retire it. That said, there's no date currently, we need to sort out flatpaks and then anything else still left and only _then_ we can set a retirement date. So, I wouldn't stress over it. We also are happy to work with stakeholders on setting that date...

rishi commented

2026-01-22 13:36:32 +00:00

Okay, thanks for clarifying that @zlopez & @kevin !

I will get started with migrating Toolbx away from registry.fedoraproject.org, and we'll see how the situation with old distributions shakes out.

Okay, thanks for clarifying that @zlopez & @kevin ! I will get started with migrating Toolbx away from `registry.fedoraproject.org`, and we'll see how the situation with old distributions shakes out.

siosm commented

2026-01-22 14:36:12 +00:00

Hum, what changed here? Is keeping the redirect that much load on the infra?

If we really want to do this, we would have to update all the references to registry.fedoraproject.org in all the Containerfiles everywhere and make sure that the container stack redirects to the right place. That would be a massive change.

Hum, what changed here? Is keeping the redirect that much load on the infra? If we really want to do this, we would have to update all the references to `registry.fedoraproject.org` in all the Containerfiles everywhere and make sure that the container stack redirects to the right place. That would be a massive change.

zlopez added this to the sprint ( 2026-01-26 to 2026-02-09 ) project

2026-01-22 17:54:03 +00:00

kevin commented

2026-01-22 23:21:25 +00:00

Owner

Sorry, perhaps I misunderstood the context here or did not explain myself well. :(

From my view nothing has changed.

I'm 100% fine keeping a redirect, but what I want is to no longer push to two places as we do now, and no longer maintain our own registry running docker-distribution, and no longer have...

/oci_registry   24T   20T  4.7T  81% /srv/registry

20TB and growing space used.

So, yes, the redirect/name will stay, but the backend/content I hope we can retire in favor of just having that once at quay.io.

Does that clarify?

Sorry, perhaps I misunderstood the context here or did not explain myself well. :( From my view nothing has changed. I'm 100% fine keeping a redirect, but what I want is to no longer push to two places as we do now, and no longer maintain our own registry running docker-distribution, and no longer have... ``` /oci_registry 24T 20T 4.7T 81% /srv/registry ``` 20TB and growing space used. So, yes, the redirect/name will stay, but the backend/content I hope we can retire in favor of just having that once at quay.io. Does that clarify?

siosm commented

2026-01-23 09:58:24 +00:00

OK, thanks for the clarification. I'm on board with keeping just a redirect from registry.fedoraproject.org to quay.io and retiring our own hosted registry.

OK, thanks for the clarification. I'm on board with keeping just a redirect from `registry.fedoraproject.org` to `quay.io` and retiring our own hosted registry.

zlopez added the

points

13

label

2026-01-26 09:17:07 +00:00

zlopez self-assigned this

2026-01-26 09:17:16 +00:00

kevin modified the project from sprint ( 2026-01-26 to 2026-02-09 ) to Sprint ( 2026-01-26 to 2026-02-08 )

2026-01-26 17:29:29 +00:00

zlopez commented

2026-01-29 12:55:45 +00:00

Author

Owner

I did all the necessary changes to make flatpak-indexer work with latest upstream and quay.io. The problem is that the flatpak-indexer is just generating deltas and metadata for flatpaks. It doesn't do any uploading. So currently trying to discuss this with Fedora Flatpaks folks where this is being done as there isn't anything at all on quay.io.

kevin commented

2026-01-29 21:21:18 +00:00

Owner

Yeah, currently I think the flow is:

An update is submitted and goes to testing in bodhi and it syncs it from koji to candidate-registry.
When the update goes to stable it copies it from candidate-registry to registry.

So, I guess we need to teach bodhi to upload to quay.io (in addition to local registries)

Yeah, currently I think the flow is: An update is submitted and goes to testing in bodhi and it syncs it from koji to candidate-registry. When the update goes to stable it copies it from candidate-registry to registry. So, I guess we need to teach bodhi to upload to quay.io (in addition to local registries)

yselkowitz commented

2026-01-29 21:40:27 +00:00

IIUC it is koji(-flatpak) that uploads to candidate-registry as part of the build. Bodhi then copies that image to registry as part of the update flow. registry.fp.o is the hard-coded default in bodhi but it is configurable per https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/base/templates/production.ini.j2#_209

zlopez commented

2026-01-30 08:14:05 +00:00

Author

Owner

I will start looking into it probably next week, as I understand we want it to upload to both registry.fedoraproject.org and quay.io for some transition period.

Do we want to move also candidate-registry to quay.io?

I will start looking into it probably next week, as I understand we want it to upload to both registry.fedoraproject.org and quay.io for some transition period. Do we want to move also candidate-registry to quay.io?

kevin commented

2026-01-30 23:03:08 +00:00

Owner

Yes, we will need a place for bodhi to put things that are in updates-testing before moving to updates.
I think this should just be a slightly differently named repo in the fedora quay.io org.

Yes, we will need a place for bodhi to put things that are in updates-testing before moving to updates. I think this should just be a slightly differently named repo in the fedora quay.io org.

zlopez commented

2026-02-02 10:09:58 +00:00

Author

Owner

I understand that, does flatpak-updates-testing sounds right?

I understand that, does `flatpak-updates-testing` sounds right?

yselkowitz commented

2026-02-02 15:10:50 +00:00

All flatpaks go through bodhi so all tags have -updates. I don't think we need to include that in quay.io namespaces though.

All flatpaks go through bodhi so all tags have `-updates`. I don't think we need to include that in quay.io namespaces though.

kevin commented

2026-02-02 17:34:54 +00:00

Owner

Yeah, I'd think 'flatpak-testing' would be fine.

zlopez commented

2026-02-03 08:46:42 +00:00

Author

Owner

Just to make it clear, do we need any change in https://pagure.io/koji-flatpak? From what I understand that just uploads to candidate-registry before bodhi uploads it to updates-testing . Or is updates-testing and candidate-registry the same place?

Just to make it clear, do we need any change in https://pagure.io/koji-flatpak? From what I understand that just uploads to `candidate-registry` before bodhi uploads it to `updates-testing` . Or is `updates-testing` and `candidate-registry` the same place?

yselkowitz commented

2026-02-03 13:20:54 +00:00

IIUC koji-flatpak handles only the initial upload to candidate-registry during the build. Everything after that is handled by bodhi. So unless/until candidate-registry is changing, no changes to koji-flatpak should be necessary.

kevin commented

2026-02-04 00:22:12 +00:00

Owner

Well, thats something we should decide I guess.

We could:

Just keep using/hosting our own candidate registry
Setup a quay.io candidate registry and switch to using that

The advantage of 1 is that we are not dependent on another external service...
The advantage of 2 is that we are not required to keep our own registry around anymore.

Well, thats something we should decide I guess. We could: 1. Just keep using/hosting our own candidate registry 2. Setup a quay.io candidate registry and switch to using that The advantage of 1 is that we are not dependent on another external service... The advantage of 2 is that we are not required to keep our own registry around anymore.

yselkowitz commented

2026-02-04 01:19:01 +00:00

Maybe switching candidate-registry should be considered (and tracked) separately, as it is not relevant to decommissioning registry?

zlopez commented

2026-02-04 08:29:36 +00:00

Author

Owner

In that case, let's make candidate-registry separate issue and we can resolve it later.

I will start looking at Bodhi for the registry move.

In that case, let's make candidate-registry separate issue and we can resolve it later. I will start looking at Bodhi for the registry move.

kevin modified the project from Sprint ( 2026-01-26 to 2026-02-08 ) to Sprint ( 2026-02-09 to 2026-02-22 )

2026-02-09 00:27:40 +00:00

zlopez commented

2026-02-10 14:36:46 +00:00

Author

Owner

So I have PR for Bodhi in place, but I wonder how is the best way to test it. If it's better to just build bodhi package in infra-stg tag with the patch and test it on staging or just try to test it out in local development environment.

I will probably try to start with local environment and if that doesn't work, try the staging approach.

So I have [PR for Bodhi](https://github.com/fedora-infra/bodhi/pull/6077) in place, but I wonder how is the best way to test it. If it's better to just build bodhi package in infra-stg tag with the patch and test it on staging or just try to test it out in local development environment. I will probably try to start with local environment and if that doesn't work, try the staging approach.

zlopez commented

2026-02-12 08:54:26 +00:00

Author

Owner

So I tried to test the changes in local dev environment, but to do that it needs to start a new compose which seems to not work in local dev environment.
I asked in #bodhi channel what is the best way to test the changes out as if they already have some process in place I don't want to mess with that.

So I tried to test the changes in local dev environment, but to do that it needs to start a new compose which seems to not work in local dev environment. I asked in #bodhi channel what is the best way to test the changes out as if they already have some process in place I don't want to mess with that.

zlopez commented

2026-02-16 09:57:43 +00:00

Author

Owner

@mattia promised he will do deployment of my changes in staging to test it out. I will wait till that is in place to test it out.

mattia commented

2026-02-16 17:02:50 +00:00

Member

@zlopez wrote in #11543 (comment):

@mattia promised he will do deployment of my changes in staging to test it out. I will wait till that is in place to test it out.

I was looking at deploy the PR yesterday, then I just left a comment on it. Shall we use 'quay.io/fedora/flatpak' for staging tests or is there a different namespace?
Since the setting is changing from string to a list, I need to fix the config at the same time I push the commit into staging, otherwise bodhi will crash.

@zlopez wrote in https://forge.fedoraproject.org/infra/tickets/issues/11543#issuecomment-392754: > @mattia promised he will do deployment of my changes in staging to test it out. I will wait till that is in place to test it out. I was looking at deploy the PR yesterday, then I just left a comment on it. Shall we use 'quay.io/fedora/flatpak' for staging tests or is there a different namespace? Since the setting is changing from string to a list, I need to fix the config at the same time I push the commit into staging, otherwise bodhi will crash.

zlopez commented

2026-02-17 08:31:59 +00:00

Author

Owner

@mattia For staging we have quay.io/fedora-testingflatpak.

@mattia For staging we have `quay.io/fedora-testingflatpak`.

mattia commented

2026-02-17 17:15:30 +00:00

Member

@zlopez I have built the bodhi-server rpm with your changes in f43-infra-stg.
Now we need this config change to be pushed to ansible and rebuild the base image and deploy staging bodhi, but ansible is now being moved to the new forge, so I can't do that now. I'll look when the move is over.

@zlopez I have built the bodhi-server rpm with your changes in f43-infra-stg. Now we need [this](https://pagure.io/fork/mattia/fedora-infra/ansible/c/3489dcd36d6cf58d5fc76aae9c4e273db0a33285?branch=bodhi-stg-flatpak) config change to be pushed to ansible and rebuild the base image and deploy staging bodhi, but ansible is now being moved to the new forge, so I can't do that now. I'll look when the move is over.

👍 1

kevin commented

2026-02-17 22:51:37 +00:00

Owner

Sorry about that.

Everything should be moved now, so you should be able to fork the forge repo and push your changes there and make a pr.

Sorry about that. Everything should be moved now, so you should be able to fork the forge repo and push your changes there and make a pr.

mattia commented

2026-02-18 15:11:52 +00:00

Member

The PR is ready, waiting for someone to merge it.

The [PR](https://forge.fedoraproject.org/infra/ansible/pulls/3121) is ready, waiting for someone to merge it.

zlopez commented

2026-02-18 15:55:50 +00:00

Author

Owner

Merged.

mattia commented

2026-02-18 18:28:34 +00:00

Member

I've pushed the patched bodhi into stg, happy testing! ;-)
Let me know if you need to synch the running db to a newer version, stg still has rawhide as F43...

I've pushed the patched bodhi into stg, happy testing! ;-) Let me know if you need to synch the running db to a newer version, stg still has rawhide as F43...

zlopez commented

2026-02-19 12:33:58 +00:00

Author

Owner

@mattia You are right, the db needs to be updated. I installed the version you built in f43-infra-stg and updated the config on staging. But when I try to push F42F compose I just get that the packages are not in correct tag and looking at koji on staging, that is right.

Could you update the db on staging? I assume I will hit some issues with permissions for the user on quay.io, but that is what this testing is about.

@mattia You are right, the db needs to be updated. I installed the version you built in `f43-infra-stg` and updated the config on staging. But when I try to push `F42F` compose I just get that the packages are not in correct tag and looking at koji on staging, that is right. Could you update the db on staging? I assume I will hit some issues with permissions for the user on quay.io, but that is what this testing is about.

mattia commented

2026-02-19 17:42:22 +00:00

Member

@zlopez done, the staging db is now in synch with prod db as of today.

ngompa commented

2026-02-19 17:43:35 +00:00

Does this mean that registry.fp.o content will no longer be available to Syrians? We just unblocked them, but my understanding is that it's still blocked for Red Hat commercial properties (of which quay.io is one of them).

Does this mean that registry.fp.o content will no longer be available to Syrians? We _just_ unblocked them, but my understanding is that it's still blocked for Red Hat commercial properties (of which quay.io is one of them).

patrikp referenced this issue

2026-02-19 17:57:45 +00:00

container registry pruning #11512

kevin commented

2026-02-19 20:06:32 +00:00

Owner

@ngompa wrote in #11543 (comment):

Does this mean that registry.fp.o content will no longer be available to Syrians? We just unblocked them, but my understanding is that it's still blocked for Red Hat commercial properties (of which quay.io is one of them).

Perhaps @jflory7 could look into this ?

@ngompa wrote in https://forge.fedoraproject.org/infra/tickets/issues/11543#issuecomment-537203: > Does this mean that registry.fp.o content will no longer be available to Syrians? We _just_ unblocked them, but my understanding is that it's still blocked for Red Hat commercial properties (of which quay.io is one of them). Perhaps @jflory7 could look into this ?

zlopez commented

2026-02-23 09:02:44 +00:00

Author

Owner

@mattia wrote in #11543 (comment):

@zlopez done, the staging db is now in synch with prod db as of today.

Thanks, will try it again today.

@mattia wrote in https://forge.fedoraproject.org/infra/tickets/issues/11543#issuecomment-537201: > @zlopez done, the staging db is now in synch with prod db as of today. Thanks, will try it again today.

zlopez modified the project from Sprint ( 2026-02-09 to 2026-02-22 ) to Sprint ( 2026-02-23 to 2026-03-08 )

2026-02-23 09:27:29 +00:00

zlopez commented

2026-02-23 13:24:40 +00:00

Author

Owner

Encountered a problem when testing the bodhi on staging and wondering how to fix it. Each time I try to start a compose the builds are missing on koji.stg.fedoraproject.org (which is understandable as the bodhi was synced from prod) and the compose fails on that. @mattia How are you testing composes on staging?

zlopez added

and removed

labels

2026-02-23 15:41:55 +00:00

mattia commented

2026-02-23 17:49:07 +00:00

Member

@zlopez wrote in #11543 (comment):

How are you testing composes on staging?

I never did ;-)
The truth is that staging is mostly broken because it bases on the prod db which differs from what we have in stg koji. I ever only used stg to test some basic UI functionality.

@zlopez wrote in https://forge.fedoraproject.org/infra/tickets/issues/11543#issuecomment-541191: > How are you testing composes on staging? I never did ;-) The truth is that staging is mostly broken because it bases on the prod db which differs from what we have in stg koji. I ever only used stg to test some basic UI functionality.

zlopez commented

2026-02-24 10:43:40 +00:00

Author

Owner

I wonder how to test the change than.

@mattia Any ideas?

Could it be possible to just try running the skopeo command with some testing flatpak?

I wonder how to test the change than. @mattia Any ideas? Could it be possible to just try running the skopeo command with some testing flatpak?

zlopez commented

2026-02-26 12:41:39 +00:00

Author

Owner

Currently trying to run the skope command locally and it seems that this could be enough to test it out. Although I'm having some issues with authentication to quay.io, but that was expected.

zlopez commented

2026-02-26 14:05:13 +00:00

Author

Owner

So I was able to test out the push to quay.io, but found few problems along the way:

The quay.io doesn't allow nested organizations, so I can't really use https://quay.io/fedora-testing/flatpak, as that is only repository. I had to create a new organization for testing https://quay.io/organization/fedora-flatpak-testing?tab=repos. I already tested out uploading 0ad flatpak to this organization using skopeo copy -a --retry-times=10 docker://candidate-registry.fedoraproject.org/0ad:0.27.1-2 docker://quay.io/fedora-flatpak-testing/0ad:0.27.1-2 using bot account
The repository have to be created separately for each flatpak, as the bot account has permissions per repository (which needs to be added for each newly created repository). This could be probably automated to run each time when new flatpak is added.
The authentication credentials for quay.io bot needs to be saved in a location skopeo can find it, either adding --authfile parameter to skopeo or using default location, which I did. In my case I just used podman login, but that isn't persistent from what I was able to find about that.

So I was able to test out the push to quay.io, but found few problems along the way: 1. The quay.io doesn't allow nested organizations, so I can't really use https://quay.io/fedora-testing/flatpak, as that is only repository. I had to create a new organization for testing https://quay.io/organization/fedora-flatpak-testing?tab=repos. I already tested out uploading 0ad flatpak to this organization using `skopeo copy -a --retry-times=10 docker://candidate-registry.fedoraproject.org/0ad:0.27.1-2 docker://quay.io/fedora-flatpak-testing/0ad:0.27.1-2` using bot account 2. The repository have to be created separately for each flatpak, as the bot account has permissions per repository (which needs to be added for each newly created repository). This could be probably automated to run each time when new flatpak is added. 3. The authentication credentials for quay.io bot needs to be saved in a location skopeo can find it, either adding `--authfile` parameter to skopeo or using default location, which I did. In my case I just used `podman login`, but that isn't persistent from what I was able to find about that.

zlopez commented

2026-03-03 14:39:40 +00:00

Author

Owner

Notes from today meeting with flatpak folks:

Name of the organization on quay.io should be fedora-flatpaks and fedora-flatpaks-testing (for staging)
There will be automation added for creating repositories in the organization and adding permissions to that repository to upload bot, when new repository will be requested and validated. This change will be done in scm_request_processor toddler
We also discussed that we should probably move candidate-registry.fedoraproject.org as well to quay.io, but that will be future target and needs changes in koji-flatpak

Notes from today meeting with flatpak folks: 1. Name of the organization on quay.io should be `fedora-flatpaks` and `fedora-flatpaks-testing` (for staging) 2. There will be automation added for creating repositories in the organization and adding permissions to that repository to upload bot, when new repository will be requested and validated. This change will be done in `scm_request_processor` toddler 3. We also discussed that we should probably move `candidate-registry.fedoraproject.org` as well to quay.io, but that will be future target and needs changes in `koji-flatpak`

kevin commented

2026-03-03 19:55:35 +00:00

Owner

Sounds reasonable to me. We should make sure we have admin/ownership of those in case we need to manage things later...

👍 1

zlopez commented

2026-03-04 13:22:07 +00:00

Author

Owner

So today I was able to create the fedora-flatpaks-testing organization and found out how to overcome the initial problems with the bot permissions and the inability to create repositories automatically. That means that we don't need to do any changes in scm_request_processor and just use the bot itself to create repositories.
I just need to create a script that will move everything currently hosted on registry.fedoraproject.org to new organization on quay.io. Will test that out on staging first. It will have to be ran only once, after that the repository will be automatically created when first build of the flatpak will be copied over from candidate-registry.

So today I was able to create the [fedora-flatpaks-testing organization](https://quay.io/organization/fedora-flatpaks-testing) and found out how to overcome the initial problems with the bot permissions and the inability to create repositories automatically. That means that we don't need to do any changes in `scm_request_processor` and just use the bot itself to create repositories. I just need to create a script that will move everything currently hosted on `registry.fedoraproject.org` to new organization on `quay.io`. Will test that out on staging first. It will have to be ran only once, after that the repository will be automatically created when first build of the flatpak will be copied over from `candidate-registry`.

yselkowitz commented

2026-03-04 16:22:53 +00:00

There are certain containers in which we have been using namespaces, mostly runtimes, runtime extensions, and EOL-rebases. How would this work on quay.io?

kevin commented

2026-03-04 20:55:18 +00:00

Owner

Also, I wonder... could we figure out a cleanup policy? ie have some way for quay.io to know whats no longer live content and can be removed? I guess that could be a more complicated sidetask.

zlopez commented

2026-03-05 08:09:23 +00:00

Author

Owner

@yselkowitz wrote in #11543 (comment):

There are certain containers in which we have been using namespaces, mostly runtimes, runtime extensions, and EOL-rebases. How would this work on quay.io?

As namespaces are managed as organizations on quay.io and I didn't found the way to create nested organization I think this could be a problem. Could we change that to simple quay.io/<organization>/<repository> structure?

@yselkowitz wrote in https://forge.fedoraproject.org/infra/tickets/issues/11543#issuecomment-570422: > There are certain containers in which we have been using namespaces, mostly runtimes, runtime extensions, and EOL-rebases. How would this work on quay.io? As namespaces are managed as organizations on quay.io and I didn't found the way to create nested organization I think this could be a problem. Could we change that to simple `quay.io/<organization>/<repository>` structure?

zlopez commented

2026-03-05 08:10:25 +00:00

Author

Owner

@kevin wrote in #11543 (comment):

Also, I wonder... could we figure out a cleanup policy? ie have some way for quay.io to know whats no longer live content and can be removed? I guess that could be a more complicated sidetask.

I assume this could be resolved as separate issue. But it would be nice to have something like that in place.

@kevin wrote in https://forge.fedoraproject.org/infra/tickets/issues/11543#issuecomment-570459: > Also, I wonder... could we figure out a cleanup policy? ie have some way for quay.io to know whats no longer live content and can be removed? I guess that could be a more complicated sidetask. I assume this could be resolved as separate issue. But it would be nice to have something like that in place.

kevin added

and removed

labels

2026-03-09 14:33:45 +00:00

zlopez modified the project from Sprint ( 2026-02-23 to 2026-03-08 ) to Sprint ( 2026-03-09 to 2026-03-23 )

2026-03-09 14:49:44 +00:00

ngompa commented

2026-03-17 14:08:51 +00:00

@zlopez wrote in #11543 (comment):

@yselkowitz wrote in #11543 (comment):

There are certain containers in which we have been using namespaces, mostly runtimes, runtime extensions, and EOL-rebases. How would this work on quay.io?

As namespaces are managed as organizations on quay.io and I didn't found the way to create nested organization I think this could be a problem. Could we change that to simple quay.io/<organization>/<repository> structure?

Nested repositories exist in Quay.

@zlopez wrote in https://forge.fedoraproject.org/infra/tickets/issues/11543#issuecomment-573697: > @yselkowitz wrote in #11543 (comment): > > > There are certain containers in which we have been using namespaces, mostly runtimes, runtime extensions, and EOL-rebases. How would this work on quay.io? > > As namespaces are managed as organizations on quay.io and I didn't found the way to create nested organization I think this could be a problem. Could we change that to simple `quay.io/<organization>/<repository>` structure? [Nested repositories exist in Quay](https://redhat.atlassian.net/browse/PROJQUAY-543).

zlopez commented

2026-03-18 08:43:00 +00:00

Author

Owner

@ngompa I didn't found a way to create a repository inside repository, couldn't even find anything about that when searching the web. If there is a way it must be really not intuitive.

But I see that the ticket you are pointing to only implemented allowing slashes in repository names. I tried this with skopeo when I first tried to copy a repository to quay.io/fedora/flatpaks and it failed, but maybe it would work if the nested structure is just one level. Will need to try this in the https://quay.io/organization/fedora-flatpaks-testing with some repository that has nested structure in flatpaks.

@ngompa I didn't found a way to create a repository inside repository, couldn't even find anything about that when searching the web. If there is a way it must be really not intuitive. But I see that the ticket you are pointing to only implemented allowing slashes in repository names. I tried this with `skopeo` when I first tried to copy a repository to `quay.io/fedora/flatpaks` and it failed, but maybe it would work if the nested structure is just one level. Will need to try this in the https://quay.io/organization/fedora-flatpaks-testing with some repository that has nested structure in flatpaks.

zlopez commented

2026-03-18 17:38:36 +00:00

Author

Owner

Today I had meeting with @yselkowitz and here is the plan we decided on (we also tested out migrating runtime, to see if the structure would be any issue, see the result https://quay.io/repository/fedora-flatpaks-testing/f43/flatpak-runtime):

Contact @mattia and get https://github.com/fedora-infra/bodhi/pull/6077 to bodhi (I was able to test out that this is working). We also need to resolve the automated login of quay.io bot on bodhi
If we manage to do first step before final freeze there would be rebuilt of all the flatpaks needed for F44 and with the bodhi change all of them should be copied to quay.io as well
Switch the flatpak-indexer to quay.io - we can do this whenever we want after bodhi change as it will generate metadata only for new builds

Nice to have:

Figure out a way to do pruning of flatpaks on quay.io

Today I had meeting with @yselkowitz and here is the plan we decided on (we also tested out migrating runtime, to see if the structure would be any issue, see the result https://quay.io/repository/fedora-flatpaks-testing/f43/flatpak-runtime): 1. Contact @mattia and get https://github.com/fedora-infra/bodhi/pull/6077 to bodhi (I was able to test out that this is working). We also need to resolve the automated login of quay.io bot on bodhi 2. If we manage to do first step before final freeze there would be rebuilt of all the flatpaks needed for F44 and with the bodhi change all of them should be copied to quay.io as well 3. Switch the flatpak-indexer to quay.io - we can do this whenever we want after bodhi change as it will generate metadata only for new builds Nice to have: * Figure out a way to do pruning of flatpaks on quay.io

kevin commented

2026-03-18 18:45:57 +00:00

Owner

Thanks again for working on this.
One suggestion/idea: I would very much like to keep the registry.fedoraproject.org name (so we don't need to change all the clients, and also so we can manage it/switch it if we want later transparently). So, can we keep that and just add a redirect when we want to move requests to quay.io?

Of course I guess we still need a flag day due to the indexer. Or.... could we run two instances? One for each registry, then switch, then drop the old one after we are sure we are moved?

Thanks again for working on this. One suggestion/idea: I would very much like to keep the registry.fedoraproject.org name (so we don't need to change all the clients, and also so we can manage it/switch it if we want later transparently). So, can we keep that and just add a redirect when we want to move requests to quay.io? Of course I guess we still need a flag day due to the indexer. Or.... could we run two instances? One for each registry, then switch, then drop the old one after we are sure we are moved?

zlopez commented

2026-03-19 10:35:48 +00:00

Author

Owner

The indexer is tricky one in this, as I tried originally to run both registry.fp.o and quay.io simultaneously, but they were touching the same files and rewriting them under each other nose. So it's not possible to have indexer to point to both at once, at least my knowledge of that tool is limited to make that work.

But the bodhi change will let it copy to both registry.fp.o and quay.io, so the flatpaks will be in both places till we will not remove registry from the loop.

It shouldn't be hard to just keep the registry.fp.o as redirect. We just need to redirect it for docker:// urls (at least I hope that will be enough) and change it to point to the new flatpak organization on quay.io. We can even keep indexer to point to registry.fp.o/fedora-flatpaks/<flatpak>, which with the correct redirect will point to quay.io/fedora-flatpaks/<flatpak>.

The indexer is tricky one in this, as I tried originally to run both registry.fp.o and quay.io simultaneously, but they were touching the same files and rewriting them under each other nose. So it's not possible to have indexer to point to both at once, at least my knowledge of that tool is limited to make that work. But the bodhi change will let it copy to both registry.fp.o and quay.io, so the flatpaks will be in both places till we will not remove registry from the loop. It shouldn't be hard to just keep the registry.fp.o as redirect. We just need to redirect it for `docker://` urls (at least I hope that will be enough) and change it to point to the new flatpak organization on quay.io. We can even keep indexer to point to `registry.fp.o/fedora-flatpaks/<flatpak>`, which with the correct redirect will point to `quay.io/fedora-flatpaks/<flatpak>`.

kevin commented

2026-03-19 18:24:29 +00:00

Owner

Yeah, so if we had to back out the move we could just repoint the indexer and have it reindex things, so thats probibly fine.

ngompa commented

2026-03-19 18:26:48 +00:00

@zlopez wrote in #11543 (comment):

@ngompa I didn't found a way to create a repository inside repository, couldn't even find anything about that when searching the web. If there is a way it must be really not intuitive.

It might not be enabled on Quay.io yet? Might be worth asking the Quay.io folks.

@zlopez wrote in https://forge.fedoraproject.org/infra/tickets/issues/11543#issuecomment-579312: > @ngompa I didn't found a way to create a repository inside repository, couldn't even find anything about that when searching the web. If there is a way it must be really not intuitive. > It might not be enabled on Quay.io yet? Might be worth asking the Quay.io folks.

zlopez commented

2026-03-20 08:55:52 +00:00

Author

Owner

@ngompa It's actually working, see https://quay.io/repository/fedora-flatpaks-testing/f43/flatpak-runtime

kevin modified the project from Sprint ( 2026-03-09 to 2026-03-23 ) to Sprint ( 2026-03-23 to 2026-04-06 )

2026-03-23 14:23:15 +00:00

zlopez commented

2026-03-24 14:57:13 +00:00

Author

Owner

I created a https://quay.io/organization/fedora-flatpaks organization today and setup the bot for that, the contact for this organization is admin+flatpaks@fedoraproject.org.

I also prepared the changes for bodhi deployment for quay.io infra/ansible#3236.

I also talked with @mattia and https://github.com/fedora-infra/bodhi/pull/6077 will probably be in bodhi after final freeze, but everything should be prepared when this change lands.

I created a https://quay.io/organization/fedora-flatpaks organization today and setup the bot for that, the contact for this organization is admin+flatpaks@fedoraproject.org. I also prepared the changes for bodhi deployment for quay.io https://forge.fedoraproject.org/infra/ansible/pulls/3236. I also talked with @mattia and https://github.com/fedora-infra/bodhi/pull/6077 will probably be in bodhi after final freeze, but everything should be prepared when this change lands.

kevin removed this from the Sprint ( 2026-03-23 to 2026-04-06 ) project

2026-04-07 20:07:26 +00:00

zlopez added this to the Sprint ( 2026-04-06 to 2026-04-20 ) project

2026-04-20 12:23:00 +00:00

zlopez modified the project from Sprint ( 2026-04-06 to 2026-04-20 ) to Sprint ( 2026-04-20 to 2026-05-04 )

2026-04-20 12:23:37 +00:00

james referenced this issue from a commit

2026-05-01 16:44:57 +00:00

[flatpak-indexer] Make the deployment simpler

zlopez modified the project from Sprint ( 2026-04-20 to 2026-05-04 ) to Sprint ( 2026-05-04 to 2026-05-18 )

2026-05-04 14:20:59 +00:00

zlopez commented

2026-05-06 10:56:59 +00:00

Author

Owner

The changes for bodhi were deployed to production I will watch the bodhi for any failures related to this change.

zlopez commented

2026-05-06 11:24:52 +00:00

Author

Owner

Next step after Bodhi will be tested out is to deploy flatpak-indexer with quay.io changes on production.

mattia commented

2026-05-07 05:29:26 +00:00

Member

@zlopez wrote in #11543 (comment):

Next step after Bodhi will be tested out is to deploy flatpak-indexer with quay.io changes on production.

Note that Bodhi must be redeployed to reflect the version update and config change. I'm not trying to do that now as I'm about to leave for my day at work.

@zlopez wrote in https://forge.fedoraproject.org/infra/tickets/issues/11543#issuecomment-697494: > Next step after Bodhi will be tested out is to deploy flatpak-indexer with quay.io changes on production. Note that Bodhi must be redeployed to reflect the version update and config change. I'm not trying to do that now as I'm about to leave for my day at work.

zlopez commented

2026-05-07 07:38:21 +00:00

Author

Owner

So we have issue with the bodhi change

May 06 17:23:09 bodhi-backend01.rdu3.fedoraproject.org celery-3[155038]: b'time="2026-05-06T
17:23:09Z" level=fatal msg="initializing destination docker://registry.fedoraproject.org/kre
name:5.0.60_20260502.101040.4e03e61-1: getting username and password: reading JSON file \"/
etc/bodhi/flatpak-bot-auth.json\": unmarshaling JSON at \"/etc/bodhi/flatpak-bot-auth.json
\": invalid character 'Z' looking for beginning of value"\n'

So we have issue with the bodhi change ``` May 06 17:23:09 bodhi-backend01.rdu3.fedoraproject.org celery-3[155038]: b'time="2026-05-06T 17:23:09Z" level=fatal msg="initializing destination docker://registry.fedoraproject.org/kre name:5.0.60_20260502.101040.4e03e61-1: getting username and password: reading JSON file \"/ etc/bodhi/flatpak-bot-auth.json\": unmarshaling JSON at \"/etc/bodhi/flatpak-bot-auth.json \": invalid character 'Z' looking for beginning of value"\n' ```

zlopez commented

2026-05-07 08:00:41 +00:00

Author

Owner

This was actually fixed by @kevin already. And we have a first bunch of flatpaks in https://quay.io/organization/fedora-flatpaks?tab=repos :-)

zlopez commented

2026-05-11 06:42:11 +00:00

Author

Owner

Found out that all the repositories created directly by CLI push are created as private by default. So currently investigating how to set them to be public by default when pushed by Bodhi.

zlopez commented

2026-05-12 13:55:50 +00:00

Author

Owner

So it's not possible to set it as this by default, but it should be possible to create an script with api calls that will do that for every new repository automatically like this:

curl -X PUT \
    -H "Authorization: Bearer $QUAY_TOKEN" \
    -H "Content-Type: application/json" \
    https://quay.io/api/v1/repository/$NAMESPACE/$REPO/changevisibility \
    -d '{"visibility": "public"}'

So it's not possible to set it as this by default, but it should be possible to create an script with api calls that will do that for every new repository automatically like this: ``` curl -X PUT \ -H "Authorization: Bearer $QUAY_TOKEN" \ -H "Content-Type: application/json" \ https://quay.io/api/v1/repository/$NAMESPACE/$REPO/changevisibility \ -d '{"visibility": "public"}' ```

kevin modified the project from Sprint ( 2026-05-04 to 2026-05-18 ) to Sprint ( 2026-05-18 to 2026-06-01)

2026-05-18 14:35:52 +00:00

zlopez modified the project from Sprint ( 2026-05-18 to 2026-06-01) to Current Sprint ( 2026-06-01 to 2026-06-14)

2026-06-01 14:22:35 +00:00

zlopez referenced this issue from a commit

2026-06-12 13:27:11 +00:00

[bodhi2] Add script to make flatpak repos public

zlopez referenced this issue from infra/ansible

2026-06-12 13:27:38 +00:00

[bodhi2] Add script to make flatpak repos public #3412

zlopez commented

2026-06-12 13:28:51 +00:00

Author

Owner

I created a script that should set the repositories public once created (PR).

I created a script that should set the repositories public once created ([PR](https://forge.fedoraproject.org/infra/ansible/pulls/3412)).

zlopez referenced this issue from a commit

2026-06-12 13:31:34 +00:00

[bodhi2] Add script to make flatpak repos public

zlopez referenced this issue from a commit

2026-06-12 13:36:59 +00:00

[bodhi2] Add script to make flatpak repos public

james modified the project from Current Sprint ( 2026-06-01 to 2026-06-14) to Sprint ( 2026-06-15 to 2026-06-28)

2026-06-15 14:25:58 +00:00

zlopez referenced this issue from a commit

2026-06-24 07:56:50 +00:00

[bodhi2] Add script to make flatpak repos public

zlopez commented

2026-06-24 09:17:25 +00:00

Author

Owner

The script is now deployed and tested out in bodhi. It will run as cron job every hour (~3s run time). All the repositories are now public.

So only thing that remains is to deploy flatpak-indexer on production to point to quay.io.

The script is now deployed and tested out in bodhi. It will run as cron job every hour (~3s run time). All the repositories are now public. So only thing that remains is to deploy flatpak-indexer on production to point to quay.io.

kevin commented

2026-06-24 18:57:04 +00:00

Owner

I'm trying to think of how we can do this and not require a flag day where everything is broken.

Can we just make a duplicate flatpak-indexer-quay application and point it to quay and wait for it to catch up/index everything (or does it only index new things?)

I'm trying to think of how we can do this and not require a flag day where everything is broken. Can we just make a duplicate flatpak-indexer-quay application and point it to quay and wait for it to catch up/index everything (or does it only index new things?)

zlopez commented

2026-06-25 07:28:32 +00:00

Author

Owner

@kevin I tried that in the past and unfortunately the flatpak-indexer is saving metadata to specific location, which is being accessed by the users, and they just replaced each other. But we can leave it pointing to registry.fedoraproject.org for now and switch it at some time in future.

Probably with some changes in the app itself it should be possible, but it's not possible to do this just in ansible.

@kevin I tried that in the past and unfortunately the flatpak-indexer is saving metadata to specific location, which is being accessed by the users, and they just replaced each other. But we can leave it pointing to registry.fedoraproject.org for now and switch it at some time in future. Probably with some changes in the app itself it should be possible, but it's not possible to do this just in ansible.

kevin commented

2026-06-25 16:59:48 +00:00

Owner

Well, I meant:

Keep the current flatpak-indexer exactly as it is now, pointed to registry.fedoraproject.org (our registry)
Make a new flatpak-indexer-quay that operates only on quay.io repos.

They shouldn't conflict, as they are pointing to different repos?

But then I am not sure how the indexer works... is it only operating on new things? Or does it catch up and index everything that doesn't have one? I can't recall.

Well, I meant: * Keep the current flatpak-indexer exactly as it is now, pointed to registry.fedoraproject.org (our registry) * Make a new flatpak-indexer-quay that operates only on quay.io repos. They shouldn't conflict, as they are pointing to different repos? But then I am not sure how the indexer works... is it only operating on new things? Or does it catch up and index everything that doesn't have one? I can't recall.

zlopez commented

2026-06-26 08:16:29 +00:00

Author

Owner

It's not problem to work with the different upstream repos, but the metadata are saved to same location ${OUTPUT_DIR}/fedora and it's single json metadata file that is either filled with quay.io source or registry.fedoraproject.org. You can see it in the config.yaml.

From what I understand it's similar to metadata file for RPM repositories. So single file listing where to download the actual file.

It's not problem to work with the different upstream repos, but the metadata are saved to same location `${OUTPUT_DIR}/fedora` and it's single json metadata file that is either filled with quay.io source or registry.fedoraproject.org. You can see it in the [config.yaml](https://forge.fedoraproject.org/infra/ansible/src/commit/d6f46658ec83be02f1362aacf1c2f6907a88943d/roles/openshift-apps/flatpak-indexer/templates/config.yaml#L31). From what I understand it's similar to metadata file for RPM repositories. So single file listing where to download the actual file.

kevin commented

2026-06-26 17:02:37 +00:00

Owner

Yeah, we would still have to have a flag day for that I guess... but we could still generate the flatpak-latest.json for quay.io and only switch the one we publish on the proxies when we are ready to switch over?

Not sure here, but I'd like to avoid us switching and then having to wait for indexing and things to be broken for users while that happens. If we can avoid it.

Yeah, we would still have to have a flag day for that I guess... but we could still generate the flatpak-latest.json for quay.io and only switch the one we publish on the proxies when we are ready to switch over? Not sure here, but I'd like to avoid us switching and then having to wait for indexing and things to be broken for users while that happens. If we can avoid it.

yselkowitz commented

2026-06-29 00:28:49 +00:00

Sounds like I'll need to rebuild all current flatpaks in order to assure that all content is pushed to quay.io as a prerequisite of this switch?

zlopez commented

2026-06-29 09:31:28 +00:00

Author

Owner

@kevin I can save the files to different location for quay.io, that is possible to do in config, it will just not be used for now.

@yselkowitz Everything should be on quay.io already, as the upload is working to https://quay.io/organization/fedora-flatpaks?tab=repos. I will need to play with staging and try to deploy quay.io separate flatpak-indexer. That will probably take some time till it's ready.

@kevin I can save the files to different location for quay.io, that is possible to do in config, it will just not be used for now. @yselkowitz Everything should be on quay.io already, as the upload is working to https://quay.io/organization/fedora-flatpaks?tab=repos. I will need to play with staging and try to deploy quay.io separate flatpak-indexer. That will probably take some time till it's ready.

zlopez modified the project from Sprint ( 2026-06-15 to 2026-06-28) to Sprint ( 2026-06-29 to 2026-07-12)

2026-06-29 14:23:52 +00:00

zlopez referenced this issue from a commit

2026-06-29 15:11:47 +00:00

[flatpak-indexer] Separate quay.io instance

zlopez referenced this issue from infra/ansible

2026-06-29 15:12:15 +00:00

[flatpak-indexer] Separate quay.io instance #3453

zlopez referenced this issue from a commit

2026-06-29 15:31:00 +00:00

[flatpak-indexer] Separate quay.io instance

yselkowitz commented

2026-06-29 16:33:13 +00:00

IIUC only those flatpaks that have been updated since the uploader was updated is on quay.io, but those that haven't been rebuilt since the F44 flatpak mass rebuild are not. That quay.io org shows 364 containers, but 567 flatpaks have at least one F44 release.

zlopez commented

2026-06-29 16:34:52 +00:00

Author

Owner

We can definitely wait with the switch to quay.io till everything is there.

kevin commented

2026-06-29 16:38:34 +00:00

Owner

Is there no way to tell the indexer to just reindex everything? ;(

zlopez commented

2026-06-30 08:49:42 +00:00

Author

Owner

If there is, I don't know about it.

zlopez referenced this issue from a commit

2026-06-30 11:20:41 +00:00

[flatpak-indexer] Separate quay.io instance

zlopez commented

2026-06-30 12:40:01 +00:00

Author

Owner

I'm trying to play with the quay.io instance of flatpak-indexer and I wonder if we actually need to generate deltas anymore or does quay.io does that for us?

Also there is no support for saving deltas or app icons on quay.io or at least I didn't found any. Currently we do it like this

{% if env == 'staging' %}
deltas_uri: https://registry{{ env_suffix }}.fedoraproject.org/deltas/
icons_uri: https://registry{{ env_suffix }}.fedoraproject.org/app-icons/
{% else %}
deltas_uri: https://cdn.registry.fedoraproject.org/deltas/
icons_uri: https://cdn.registry.fedoraproject.org/app-icons/
{% endif %}

I'm not sure what to do with this when we switch to quay.io. If we want to keep this or not?

I'm trying to play with the quay.io instance of flatpak-indexer and I wonder if we actually need to generate deltas anymore or does quay.io does that for us? Also there is no support for saving deltas or app icons on quay.io or at least I didn't found any. Currently we do it like this ``` {% if env == 'staging' %} deltas_uri: https://registry{{ env_suffix }}.fedoraproject.org/deltas/ icons_uri: https://registry{{ env_suffix }}.fedoraproject.org/app-icons/ {% else %} deltas_uri: https://cdn.registry.fedoraproject.org/deltas/ icons_uri: https://cdn.registry.fedoraproject.org/app-icons/ {% endif %} ``` I'm not sure what to do with this when we switch to quay.io. If we want to keep this or not?

kevin commented

2026-06-30 16:10:19 +00:00

Owner

Yeah, my understanding/recollection of how the indexer works is that it makes the deltas and indexes and we sync those to our proxy servers, where users fetch them.

We could make the quay.io indexer sync to a new place... say 'quay/deltas' and 'quay/app-icons'. That would require some work on sundries in ansible to sync those things, but it should be just copying the way it syncs the other existing stuff.

Then, we still need a flag day, where we just swap in the quay directories for existing ones?

Yeah, my understanding/recollection of how the indexer works is that it makes the deltas and indexes and we sync those to our proxy servers, where users fetch them. We could make the quay.io indexer sync to a new place... say 'quay/deltas' and 'quay/app-icons'. That would require some work on sundries in ansible to sync those things, but it should be just copying the way it syncs the other existing stuff. Then, we still need a flag day, where we just swap in the quay directories for existing ones?

zlopez commented

2026-07-01 12:06:23 +00:00

Author

Owner

That makes sense. I will need to check what changes will need to be done on flatpak-indexer to work like that.

For now I noticed that the quay.io returns 401 for the differ container pull requests (which shouldn't be as it's public). And the differ container, which generates the delta is processing tasks for both quay.io and registry.fp.o, so maybe we will need some code changes in flatpak-indexer itself to get it working with both at once. Some way of filtering tasks from quay.io and registry.fp.o.

Right now this seems to be more complex than the changes in Bodhi.

That makes sense. I will need to check what changes will need to be done on flatpak-indexer to work like that. For now I noticed that the quay.io returns 401 for the differ container pull requests (which shouldn't be as it's public). And the differ container, which generates the delta is processing tasks for both quay.io and registry.fp.o, so maybe we will need some code changes in flatpak-indexer itself to get it working with both at once. Some way of filtering tasks from quay.io and registry.fp.o. Right now this seems to be more complex than the changes in Bodhi.

zlopez commented

2026-07-08 14:58:10 +00:00

Author

Owner

It seems I found out a simpler way to do it. It will mean we will have everything duplicated in flatpak-indexer project, but it will allow us to have two completely separate instances that are sharing just the volume that the output is saved to.

I'm testing the approach on staging currently.

It seems I found out a simpler way to do it. It will mean we will have everything duplicated in flatpak-indexer project, but it will allow us to have two completely separate instances that are sharing just the volume that the output is saved to. I'm testing the approach on staging currently.

zlopez modified the project from Sprint ( 2026-06-29 to 2026-07-12) to Current Sprint ( 2026-07-13 to 2026-07-26)

2026-07-13 14:55:00 +00:00

Rows
Columns

Migration of registry.fedoraproject.org to quay.io #11543

Describe what you would like us to do:

When do you need this to be done by? (YYYY/MM/DD)