Migration of registry.fedoraproject.org to quay.io #11543

Open
opened 2023-09-25 11:10:45 +00:00 by zlopez · 75 comments
Owner

Describe what you would like us to do:


As the investigation is now done, we can follow the document and do the migration. It will be best to wait after Fedora 39 will be out as this will allow us to get rid of plenty of legacy workflow.

We will need some transition period, so the users still can access the content on registry.fp.o until we move everything on or it's no longer needed.

When do you need this to be done by? (YYYY/MM/DD)


Not urgent, needs to wait till F39 will be out.

# Describe what you would like us to do: ---- As the [investigation](https://pagure.io/fedora-infrastructure/issue/10386) is now done, we can follow the [document](https://fedora-arc.readthedocs.io/en/latest/registry_to_quay/index.html) and do the migration. It will be best to wait after Fedora 39 will be out as this will allow us to get rid of plenty of legacy workflow. We will need some transition period, so the users still can access the content on registry.fp.o until we move everything on or it's no longer needed. # When do you need this to be done by? (YYYY/MM/DD) ---- Not urgent, needs to wait till F39 will be out.

Just to repeat what I mentioned a while before on the investigation ticket.

Toolbx still uses registry.fedoraproject.org for its fedora-toolbox images. From Fedora 39 onwards these images are produced by Fedora's nightly composes.

We are open to moving to quay.io but we need some migration period because the URL to the image is baked into the toolbox RPM, and we can't change it without knowing exactly what it would be.

Just to repeat what I mentioned a while before on the [investigation](https://pagure.io/fedora-infrastructure/issue/10386#comment-806812) ticket. Toolbx still uses `registry.fedoraproject.org` for its `fedora-toolbox` images. From Fedora 39 [onwards](https://fedoraproject.org/wiki/Changes/ToolbxReleaseBlocker) these images are produced by Fedora's nightly composes. We are open to moving to `quay.io` but we need some migration period because the URL to the image is baked into the `toolbox` RPM, and we can't change it without knowing exactly what it would be.
Owner

FWIW, IMHO, we should try and redirect existing registry.fedoraproject.org to quay.io if tools will understand that. Or I guess proxy them if not. At least initially. This will allow us to 'fail back' if there's problems or we need to down the road. But I am not sure the details on if thats possible.

FWIW, IMHO, we should try and redirect existing registry.fedoraproject.org to quay.io if tools will understand that. Or I guess proxy them if not. At least initially. This will allow us to 'fail back' if there's problems or we need to down the road. But I am not sure the details on if thats possible.
Author
Owner

Metadata Update from @zlopez:

  • Issue assigned to t0xic0der
**Metadata Update from @zlopez**: - Issue assigned to t0xic0der
Owner

So, there is some question about how we want to do things.

I personally think we should keep things pointed to the registry.fedoraproject.org, and then that redirects them to quay.io for the actual image.
The advantage of this is that we control it, and can repoint it somewhere else if we like, or run our own registry for some part of it and redirect other things.
It also means we can change it in our infra and it immediately would take effect.

If on the other hand, we change everything to say 'use quay.io', there's the advantage that we don't have to redirect and it's all on them.
But the downside is that if ever we need to change it again, we will have to change it in all the places it's currently set (documentation, podman, toolbox, etc).
Any changes will take rebuilding things and pushing out updates and be a long and slow process.

Can folks chime in with their thoughts on this so we can decide how to approach it?

So, there is some question about how we want to do things. I personally think we should keep things pointed to the registry.fedoraproject.org, and then that redirects them to quay.io for the actual image. The advantage of this is that we control it, and can repoint it somewhere else if we like, or run our own registry for some part of it and redirect other things. It also means we can change it in our infra and it immediately would take effect. If on the other hand, we change everything to say 'use quay.io', there's the advantage that we don't have to redirect and it's all on them. But the downside is that if ever we need to change it again, we will have to change it in all the places it's currently set (documentation, podman, toolbox, etc). Any changes will take rebuilding things and pushing out updates and be a long and slow process. Can folks chime in with their thoughts on this so we can decide how to approach it?

So, there is some question about how we want to do things.

I personally think we should keep things pointed to the registry.fedoraproject.org, and then that redirects them to quay.io for the actual image.
The advantage of this is that we control it, and can repoint it somewhere else if we like, or run our own registry for some part of it and redirect other things.
It also means we can change it in our infra and it immediately would take effect.

If on the other hand, we change everything to say 'use quay.io', there's the advantage that we don't have to redirect and it's all on them.
But the downside is that if ever we need to change it again, we will have to change it in all the places it's currently set (documentation, podman, toolbox, etc).
Any changes will take rebuilding things and pushing out updates and be a long and slow process.

Yes, I am with you on this one, assuming that the underlying technical realities let us use registry.fedoraproject.org as a front.

A hard switch to quay.io can be done later once things have settled and we are confident about it. Until then, the possibly of a revert and all the associated churn, makes it wise to continue pointing the clients to registry.fedoraproject.org.

> So, there is some question about how we want to do things. > > I personally think we should keep things pointed to the registry.fedoraproject.org, and then that redirects them to quay.io for the actual image. > The advantage of this is that we control it, and can repoint it somewhere else if we like, or run our own registry for some part of it and redirect other things. > It also means we can change it in our infra and it immediately would take effect. > > If on the other hand, we change everything to say 'use quay.io', there's the advantage that we don't have to redirect and it's all on them. > But the downside is that if ever we need to change it again, we will have to change it in all the places it's currently set (documentation, podman, toolbox, etc). > Any changes will take rebuilding things and pushing out updates and be a long and slow process. Yes, I am with you on this one, assuming that the underlying technical realities let us use `registry.fedoraproject.org` as a front. A hard switch to `quay.io` can be done later once things have settled and we are confident about it. Until then, the possibly of a revert and all the associated churn, makes it wise to continue pointing the clients to `registry.fedoraproject.org`.

So, there is some question about how we want to do things.

I personally think we should keep things pointed to the registry.fedoraproject.org, and then that redirects them to quay.io for the actual image.
The advantage of this is that we control it, and can repoint it somewhere else if we like, or run our own registry for some part of it and redirect other things.
It also means we can change it in our infra and it immediately would take effect.

If on the other hand, we change everything to say 'use quay.io', there's the advantage that we don't have to redirect and it's all on them.
But the downside is that if ever we need to change it again, we will have to change it in all the places it's currently set (documentation, podman, toolbox, etc).
Any changes will take rebuilding things and pushing out updates and be a long and slow process.

Yes, I am with you on this one, assuming that the underlying technical realities let us use registry.fedoraproject.org as a front.

A hard switch to quay.io can be done later once things have settled and we are confident about it. Until then, the possibly of a revert and all the associated churn, makes it wise to continue pointing the clients to registry.fedoraproject.org.

I am also +1 to this approach of keeping registry.fp.o as a front and then redirecting. I still would want to check out the latency situation after the redirect. Any chance you can give us a timeline for testing?

> > So, there is some question about how we want to do things. > > > > I personally think we should keep things pointed to the registry.fedoraproject.org, and then that redirects them to quay.io for the actual image. > > The advantage of this is that we control it, and can repoint it somewhere else if we like, or run our own registry for some part of it and redirect other things. > > It also means we can change it in our infra and it immediately would take effect. > > > > If on the other hand, we change everything to say 'use quay.io', there's the advantage that we don't have to redirect and it's all on them. > > But the downside is that if ever we need to change it again, we will have to change it in all the places it's currently set (documentation, podman, toolbox, etc). > > Any changes will take rebuilding things and pushing out updates and be a long and slow process. > > Yes, I am with you on this one, assuming that the underlying technical realities let us use `registry.fedoraproject.org` as a front. > > A hard switch to `quay.io` can be done later once things have settled and we are confident about it. Until then, the possibly of a revert and all the associated churn, makes it wise to continue pointing the clients to `registry.fedoraproject.org`. I am also +1 to this approach of keeping registry.fp.o as a front and then redirecting. I still would want to check out the latency situation after the redirect. Any chance you can give us a timeline for testing?

I personally think we should keep things pointed to the registry.fedoraproject.org, and then that redirects them to quay.io for the actual image.
The advantage of this is that we control it, and can repoint it somewhere else if we like, or run our own registry for some part of it and redirect other things.
It also means we can change it in our infra and it immediately would take effect.

Oh, this is indeed a good idea. We started "migrating" to quay.io images in Fedora CoreOS land but this looks like a better approach if this works well.

> I personally think we should keep things pointed to the registry.fedoraproject.org, and then that redirects them to quay.io for the actual image. > The advantage of this is that we control it, and can repoint it somewhere else if we like, or run our own registry for some part of it and redirect other things. > It also means we can change it in our infra and it immediately would take effect. Oh, this is indeed a good idea. We started "migrating" to quay.io images in Fedora CoreOS land but this looks like a better approach if this works well.

May I ask what's the official place for Fedora containers now?

A documentation https://docs.fedoraproject.org/en-US/containers/#_where_to_find_the_containers points to registry.fedoraproject.org where the images were not updated since November 2023 https://registry.fedoraproject.org/repo/fedora/tags/.

May I ask what's the official place for Fedora containers now? A documentation <https://docs.fedoraproject.org/en-US/containers/#_where_to_find_the_containers> points to registry.fedoraproject.org where the images were not updated since November 2023 <https://registry.fedoraproject.org/repo/fedora/tags/>.
Author
Owner

Most of the projects hosted on registry.fp.o either moved or moving to quay.io. There is a Fedora organization on quay.io, where you should find most of the images.

Most of the projects hosted on registry.fp.o either moved or moving to quay.io. There is a [Fedora organization on quay.io](https://quay.io/organization/fedora), where you should find most of the images.
Owner

Yes, but note... IMHO we should just keep using registry.fedoraproject.org and have it redirect, so in that case we shouldn't need to change any docs.

Yes, but note... IMHO we should just keep using registry.fedoraproject.org and have it redirect, so in that case we shouldn't need to change any docs.

May I ask what's the official place for Fedora containers now?

Yes, just keep using registry.fedoraproject.org. :)

> May I ask what's the official place for Fedora containers now? Yes, just keep using `registry.fedoraproject.org`. :)

To clarify, are we saying that registry.fp.o should still be the canonical repo and that Quay.io is an implementation detail, or that the redirect is more of a backcompat thing but that the expectation is that eventually we shut down registry.fp.o? Clearly, the quay.io repos have already gone around and been documented quite some time now, so I don't know how realistic it is to consider it an implementation detail at this point.

A downside I raised in https://github.com/coreos/fedora-coreos-config/pull/2988 is that using registry.fp.o means that service is now also in your dependency path. Being dependent on less things being up is very valuable in a CI context.

To clarify, are we saying that registry.fp.o should still be the canonical repo and that Quay.io is an implementation detail, or that the redirect is more of a backcompat thing but that the expectation is that eventually we shut down registry.fp.o? Clearly, the quay.io repos have already gone around and been documented quite some time now, so I don't know how realistic it is to consider it an implementation detail at this point. A downside I raised in https://github.com/coreos/fedora-coreos-config/pull/2988 is that using registry.fp.o means that service is now also in your dependency path. Being dependent on less things being up is very valuable in a CI context.
Owner

Yeah, I guess I was imagining that we would keep registry.fedoraproject.org for a while redirecting to quay.io and make sure there were not any big issues that would cause us to roll back.
If we did have to roll back we just change dns, instead of updaing a bunch of packages, etc.

You do make a good point that it adds another 'hop' in there and also makes us a dependency...

Yeah, I guess I was imagining that we would keep registry.fedoraproject.org for a while redirecting to quay.io and make sure there were not any big issues that would cause us to roll back. If we did have to roll back we just change dns, instead of updaing a bunch of packages, etc. You do make a good point that it adds another 'hop' in there and also makes us a dependency...

Yeah, I guess I was imagining that we would keep registry.fedoraproject.org for a while redirecting to quay.io

I think it's fine to keep the redirector, but maybe we don't promote it? i.e. if people are using registry.fedoraproject.org then it would keep working.. but our docs and things all get updated to the new location?

> Yeah, I guess I was imagining that we would keep registry.fedoraproject.org for a while redirecting to quay.io I think it's fine to keep the redirector, but maybe we don't promote it? i.e. if people are using registry.fedoraproject.org then it would keep working.. but our docs and things all get updated to the new location?

What's the latest thinking on this topic? Is there still a possibility that we might roll back away from quay.io and point registry.fedoraproject.org at something else?

I am asking because I want to figure out if Toolbx should start using the new quay.io URLs or stick to registry.fedoraproject.org.

What's the latest thinking on this topic? Is there still a possibility that we might roll back away from `quay.io` and point `registry.fedoraproject.org` at something else? I am asking because I want to figure out if Toolbx should start using the new `quay.io` URLs or stick to `registry.fedoraproject.org`.
Owner

The plan is definitely still to go to quay.io, just too many fires burning to get to it.

We need to sort out flatpak stuff still.

If you want to go go quay.io it should be already in sync for toolbox, so go for it.

The plan is definitely still to go to quay.io, just too many fires burning to get to it. We need to sort out flatpak stuff still. If you want to go go quay.io it should be already in sync for toolbox, so go for it.

The plan is definitely still to go to quay.io, just too many fires burning to get to it.

We need to sort out flatpak stuff still.

If you want to go go quay.io it should be already in sync for toolbox, so go for it.

Ok, understood. Thanks, Kevin.

> The plan is definitely still to go to quay.io, just too many fires burning to get to it. > > We need to sort out flatpak stuff still. > > If you want to go go quay.io it should be already in sync for toolbox, so go for it. Ok, understood. Thanks, Kevin.
Author
Owner

This is still waiting for the flatpak to migrate.

This is still waiting for the flatpak to migrate.

@otaylor is much more familiar with the delivery mechanics of flatpaks than I am, but not sure how much time he has for this right now. In his absence, I guess it will be up to me, but I'll need to be brought up to speed.

@otaylor is much more familiar with the delivery mechanics of flatpaks than I am, but not sure how much time he has for this right now. In his absence, I guess it will be up to me, but I'll need to be brought up to speed.
Author
Owner

I'm now trying to look at pointing the flatpak-indexer to quay.io and see if that works.

I'm now trying to look at pointing the flatpak-indexer to quay.io and see if that works.
Author
Owner

Finally got to it and found out where to obtain test data for tests. I will need to remove the ODCS part as this is causing for some reason issues on staging.

Finally got to it and found out where to obtain test data for tests. I will need to remove the ODCS part as this is causing for some reason issues on staging.
Author
Owner

Currently trying to get the flatpak-indexer running on my own machine, but I hit this issue and waiting for some solution https://github.com/fedora-infra/fedora-messaging/issues/440

Currently trying to get the flatpak-indexer running on my own machine, but I hit this issue and waiting for some solution https://github.com/fedora-infra/fedora-messaging/issues/440

@zlopez are we still in process for this migration?

@zlopez are we still in process for this migration?
Author
Owner

@smilner I still have this on my TODO list, but currently have other priorities.

@smilner I still have this on my TODO list, but currently have other priorities.
Author
Owner

Metadata Update from @zlopez:

  • Issue tagged with: sprint-0
**Metadata Update from @zlopez**: - Issue tagged with: sprint-0
Author
Owner

Metadata Update from @zlopez:

  • Issue assigned to zlopez (was: t0xic0der)
**Metadata Update from @zlopez**: - Issue assigned to zlopez (was: t0xic0der)

@pingou submitted a pull request to move Toolbx away from registry.fedoraproject.org URLs to their quay.io counterparts. I am curious if the registry.fedoraproject.org URLs are about to stop working soon. If so, we need to migrate toolbox(1) away from them on older distributions.

@pingou submitted a [pull request](https://github.com/containers/containertoolbx.org/pull/55) to move Toolbx away from `registry.fedoraproject.org` URLs to their `quay.io` counterparts. I am curious if the `registry.fedoraproject.org` URLs are about to stop working soon. If so, we need to migrate `toolbox(1)` away from them on older distributions.
Author
Owner

The last thing remaining is to point flatpak-indexer to quay.io, but not sure how long does that take.

So the registry.fedoraproject.org will not go away anytime soon.

The last thing remaining is to point `flatpak-indexer` to `quay.io`, but not sure how long does that take. So the `registry.fedoraproject.org` will not go away anytime soon.

My apologies. I should have been more specific in my question.

Does it mean that registry.fedoraproject.org will be going away after flatpak-indexer is pointed at quay.io? Any rough idea of what soon looks like?

I am asking because we will have to migrate away from registry.fedoraproject.org on distributions as old as RHEL 8.10, where it's impossible to justify casual changes. So, we need some lead time to prepare for it.

I am wondering if we should already start at least migrating the code and the documentation in newer releases because this is a matter of time, or we can let the status quo continue for the foreseeable future.

My apologies. I should have been more specific in my question. Does it mean that `registry.fedoraproject.org` will be going away after `flatpak-indexer` is pointed at `quay.io`? Any rough idea of what *soon* looks like? I am asking because we will have to migrate away from `registry.fedoraproject.org` on distributions as old as RHEL 8.10, where it's impossible to justify casual changes. So, we need some lead time to prepare for it. I am wondering if we should already start at least migrating the code and the documentation in newer releases because this is a matter of time, or we can let the status quo continue for the foreseeable future.
Author
Owner

@kevin What is the plan for registry.fedoraproject.org after we migrate everything from it. Will it stay or do we want to decommission it?

@kevin What is the plan for registry.fedoraproject.org after we migrate everything from it. Will it stay or do we want to decommission it?
Owner

My orig plan was to keep it around, but after talking with folks and thinking about it more, I think we want to retire it.
That said, there's no date currently, we need to sort out flatpaks and then anything else still left and only then we can set a retirement date.

So, I wouldn't stress over it. We also are happy to work with stakeholders on setting that date...

My orig plan was to keep it around, but after talking with folks and thinking about it more, I think we want to retire it. That said, there's no date currently, we need to sort out flatpaks and then anything else still left and only _then_ we can set a retirement date. So, I wouldn't stress over it. We also are happy to work with stakeholders on setting that date...

Okay, thanks for clarifying that @zlopez & @kevin !

I will get started with migrating Toolbx away from registry.fedoraproject.org, and we'll see how the situation with old distributions shakes out.

Okay, thanks for clarifying that @zlopez & @kevin ! I will get started with migrating Toolbx away from `registry.fedoraproject.org`, and we'll see how the situation with old distributions shakes out.

Hum, what changed here? Is keeping the redirect that much load on the infra?

If we really want to do this, we would have to update all the references to registry.fedoraproject.org in all the Containerfiles everywhere and make sure that the container stack redirects to the right place. That would be a massive change.

Hum, what changed here? Is keeping the redirect that much load on the infra? If we really want to do this, we would have to update all the references to `registry.fedoraproject.org` in all the Containerfiles everywhere and make sure that the container stack redirects to the right place. That would be a massive change.
Owner

Sorry, perhaps I misunderstood the context here or did not explain myself well. :(

From my view nothing has changed.

I'm 100% fine keeping a redirect, but what I want is to no longer push to two places as we do now, and no longer maintain our own registry running docker-distribution, and no longer have...

/oci_registry   24T   20T  4.7T  81% /srv/registry

20TB and growing space used.

So, yes, the redirect/name will stay, but the backend/content I hope we can retire in favor of just having that once at quay.io.

Does that clarify?

Sorry, perhaps I misunderstood the context here or did not explain myself well. :( From my view nothing has changed. I'm 100% fine keeping a redirect, but what I want is to no longer push to two places as we do now, and no longer maintain our own registry running docker-distribution, and no longer have... ``` /oci_registry 24T 20T 4.7T 81% /srv/registry ``` 20TB and growing space used. So, yes, the redirect/name will stay, but the backend/content I hope we can retire in favor of just having that once at quay.io. Does that clarify?

OK, thanks for the clarification. I'm on board with keeping just a redirect from registry.fedoraproject.org to quay.io and retiring our own hosted registry.

OK, thanks for the clarification. I'm on board with keeping just a redirect from `registry.fedoraproject.org` to `quay.io` and retiring our own hosted registry.
zlopez self-assigned this 2026-01-26 09:17:16 +00:00
Author
Owner

I did all the necessary changes to make flatpak-indexer work with latest upstream and quay.io. The problem is that the flatpak-indexer is just generating deltas and metadata for flatpaks. It doesn't do any uploading. So currently trying to discuss this with Fedora Flatpaks folks where this is being done as there isn't anything at all on quay.io.

I did all the necessary changes to make flatpak-indexer work with latest upstream and quay.io. The problem is that the flatpak-indexer is just generating deltas and metadata for flatpaks. It doesn't do any uploading. So currently trying to discuss this with Fedora Flatpaks folks where this is being done as there isn't anything at all on quay.io.
Owner

Yeah, currently I think the flow is:

An update is submitted and goes to testing in bodhi and it syncs it from koji to candidate-registry.
When the update goes to stable it copies it from candidate-registry to registry.

So, I guess we need to teach bodhi to upload to quay.io (in addition to local registries)

Yeah, currently I think the flow is: An update is submitted and goes to testing in bodhi and it syncs it from koji to candidate-registry. When the update goes to stable it copies it from candidate-registry to registry. So, I guess we need to teach bodhi to upload to quay.io (in addition to local registries)

IIUC it is koji(-flatpak) that uploads to candidate-registry as part of the build. Bodhi then copies that image to registry as part of the update flow. registry.fp.o is the hard-coded default in bodhi but it is configurable per https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/base/templates/production.ini.j2#_209

IIUC it is koji(-flatpak) that uploads to candidate-registry as part of the build. Bodhi then copies that image to registry as part of the update flow. registry.fp.o is the hard-coded default in bodhi but it is configurable per https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/base/templates/production.ini.j2#_209
Author
Owner

I will start looking into it probably next week, as I understand we want it to upload to both registry.fedoraproject.org and quay.io for some transition period.

Do we want to move also candidate-registry to quay.io?

I will start looking into it probably next week, as I understand we want it to upload to both registry.fedoraproject.org and quay.io for some transition period. Do we want to move also candidate-registry to quay.io?
Owner

Yes, we will need a place for bodhi to put things that are in updates-testing before moving to updates.
I think this should just be a slightly differently named repo in the fedora quay.io org.

Yes, we will need a place for bodhi to put things that are in updates-testing before moving to updates. I think this should just be a slightly differently named repo in the fedora quay.io org.
Author
Owner

I understand that, does flatpak-updates-testing sounds right?

I understand that, does `flatpak-updates-testing` sounds right?

All flatpaks go through bodhi so all tags have -updates. I don't think we need to include that in quay.io namespaces though.

All flatpaks go through bodhi so all tags have `-updates`. I don't think we need to include that in quay.io namespaces though.
Owner

Yeah, I'd think 'flatpak-testing' would be fine.

Yeah, I'd think 'flatpak-testing' would be fine.
Author
Owner

Just to make it clear, do we need any change in https://pagure.io/koji-flatpak? From what I understand that just uploads to candidate-registry before bodhi uploads it to updates-testing . Or is updates-testing and candidate-registry the same place?

Just to make it clear, do we need any change in https://pagure.io/koji-flatpak? From what I understand that just uploads to `candidate-registry` before bodhi uploads it to `updates-testing` . Or is `updates-testing` and `candidate-registry` the same place?

IIUC koji-flatpak handles only the initial upload to candidate-registry during the build. Everything after that is handled by bodhi. So unless/until candidate-registry is changing, no changes to koji-flatpak should be necessary.

IIUC koji-flatpak handles only the initial upload to candidate-registry during the build. Everything after that is handled by bodhi. So unless/until candidate-registry is changing, no changes to koji-flatpak should be necessary.
Owner

Well, thats something we should decide I guess.

We could:

  1. Just keep using/hosting our own candidate registry
  2. Setup a quay.io candidate registry and switch to using that

The advantage of 1 is that we are not dependent on another external service...
The advantage of 2 is that we are not required to keep our own registry around anymore.

Well, thats something we should decide I guess. We could: 1. Just keep using/hosting our own candidate registry 2. Setup a quay.io candidate registry and switch to using that The advantage of 1 is that we are not dependent on another external service... The advantage of 2 is that we are not required to keep our own registry around anymore.

Maybe switching candidate-registry should be considered (and tracked) separately, as it is not relevant to decommissioning registry?

Maybe switching candidate-registry should be considered (and tracked) separately, as it is not relevant to decommissioning registry?
Author
Owner

In that case, let's make candidate-registry separate issue and we can resolve it later.

I will start looking at Bodhi for the registry move.

In that case, let's make candidate-registry separate issue and we can resolve it later. I will start looking at Bodhi for the registry move.
Author
Owner

So I have PR for Bodhi in place, but I wonder how is the best way to test it. If it's better to just build bodhi package in infra-stg tag with the patch and test it on staging or just try to test it out in local development environment.

I will probably try to start with local environment and if that doesn't work, try the staging approach.

So I have [PR for Bodhi](https://github.com/fedora-infra/bodhi/pull/6077) in place, but I wonder how is the best way to test it. If it's better to just build bodhi package in infra-stg tag with the patch and test it on staging or just try to test it out in local development environment. I will probably try to start with local environment and if that doesn't work, try the staging approach.
Author
Owner

So I tried to test the changes in local dev environment, but to do that it needs to start a new compose which seems to not work in local dev environment.
I asked in #bodhi channel what is the best way to test the changes out as if they already have some process in place I don't want to mess with that.

So I tried to test the changes in local dev environment, but to do that it needs to start a new compose which seems to not work in local dev environment. I asked in #bodhi channel what is the best way to test the changes out as if they already have some process in place I don't want to mess with that.
Author
Owner

@mattia promised he will do deployment of my changes in staging to test it out. I will wait till that is in place to test it out.

@mattia promised he will do deployment of my changes in staging to test it out. I will wait till that is in place to test it out.

@zlopez wrote in #11543 (comment):

@mattia promised he will do deployment of my changes in staging to test it out. I will wait till that is in place to test it out.

I was looking at deploy the PR yesterday, then I just left a comment on it. Shall we use 'quay.io/fedora/flatpak' for staging tests or is there a different namespace?
Since the setting is changing from string to a list, I need to fix the config at the same time I push the commit into staging, otherwise bodhi will crash.

@zlopez wrote in https://forge.fedoraproject.org/infra/tickets/issues/11543#issuecomment-392754: > @mattia promised he will do deployment of my changes in staging to test it out. I will wait till that is in place to test it out. I was looking at deploy the PR yesterday, then I just left a comment on it. Shall we use 'quay.io/fedora/flatpak' for staging tests or is there a different namespace? Since the setting is changing from string to a list, I need to fix the config at the same time I push the commit into staging, otherwise bodhi will crash.
Author
Owner

@mattia For staging we have quay.io/fedora-testingflatpak.

@mattia For staging we have `quay.io/fedora-testingflatpak`.

@zlopez I have built the bodhi-server rpm with your changes in f43-infra-stg.
Now we need this config change to be pushed to ansible and rebuild the base image and deploy staging bodhi, but ansible is now being moved to the new forge, so I can't do that now. I'll look when the move is over.

@zlopez I have built the bodhi-server rpm with your changes in f43-infra-stg. Now we need [this](https://pagure.io/fork/mattia/fedora-infra/ansible/c/3489dcd36d6cf58d5fc76aae9c4e273db0a33285?branch=bodhi-stg-flatpak) config change to be pushed to ansible and rebuild the base image and deploy staging bodhi, but ansible is now being moved to the new forge, so I can't do that now. I'll look when the move is over.
Owner

Sorry about that.

Everything should be moved now, so you should be able to fork the forge repo and push your changes there and make a pr.

Sorry about that. Everything should be moved now, so you should be able to fork the forge repo and push your changes there and make a pr.

The PR is ready, waiting for someone to merge it.

The [PR](https://forge.fedoraproject.org/infra/ansible/pulls/3121) is ready, waiting for someone to merge it.
Author
Owner

Merged.

Merged.

I've pushed the patched bodhi into stg, happy testing! ;-)
Let me know if you need to synch the running db to a newer version, stg still has rawhide as F43...

I've pushed the patched bodhi into stg, happy testing! ;-) Let me know if you need to synch the running db to a newer version, stg still has rawhide as F43...
Author
Owner

@mattia You are right, the db needs to be updated. I installed the version you built in f43-infra-stg and updated the config on staging. But when I try to push F42F compose I just get that the packages are not in correct tag and looking at koji on staging, that is right.

Could you update the db on staging? I assume I will hit some issues with permissions for the user on quay.io, but that is what this testing is about.

@mattia You are right, the db needs to be updated. I installed the version you built in `f43-infra-stg` and updated the config on staging. But when I try to push `F42F` compose I just get that the packages are not in correct tag and looking at koji on staging, that is right. Could you update the db on staging? I assume I will hit some issues with permissions for the user on quay.io, but that is what this testing is about.

@zlopez done, the staging db is now in synch with prod db as of today.

@zlopez done, the staging db is now in synch with prod db as of today.

Does this mean that registry.fp.o content will no longer be available to Syrians? We just unblocked them, but my understanding is that it's still blocked for Red Hat commercial properties (of which quay.io is one of them).

Does this mean that registry.fp.o content will no longer be available to Syrians? We _just_ unblocked them, but my understanding is that it's still blocked for Red Hat commercial properties (of which quay.io is one of them).
Owner

@ngompa wrote in #11543 (comment):

Does this mean that registry.fp.o content will no longer be available to Syrians? We just unblocked them, but my understanding is that it's still blocked for Red Hat commercial properties (of which quay.io is one of them).

Perhaps @jflory7 could look into this ?

@ngompa wrote in https://forge.fedoraproject.org/infra/tickets/issues/11543#issuecomment-537203: > Does this mean that registry.fp.o content will no longer be available to Syrians? We _just_ unblocked them, but my understanding is that it's still blocked for Red Hat commercial properties (of which quay.io is one of them). Perhaps @jflory7 could look into this ?
Author
Owner

@mattia wrote in #11543 (comment):

@zlopez done, the staging db is now in synch with prod db as of today.

Thanks, will try it again today.

@mattia wrote in https://forge.fedoraproject.org/infra/tickets/issues/11543#issuecomment-537201: > @zlopez done, the staging db is now in synch with prod db as of today. Thanks, will try it again today.
Author
Owner

Encountered a problem when testing the bodhi on staging and wondering how to fix it. Each time I try to start a compose the builds are missing on koji.stg.fedoraproject.org (which is understandable as the bodhi was synced from prod) and the compose fails on that. @mattia How are you testing composes on staging?

Encountered a problem when testing the bodhi on staging and wondering how to fix it. Each time I try to start a compose the builds are missing on koji.stg.fedoraproject.org (which is understandable as the bodhi was synced from prod) and the compose fails on that. @mattia How are you testing composes on staging?

@zlopez wrote in #11543 (comment):

How are you testing composes on staging?

I never did ;-)
The truth is that staging is mostly broken because it bases on the prod db which differs from what we have in stg koji. I ever only used stg to test some basic UI functionality.

@zlopez wrote in https://forge.fedoraproject.org/infra/tickets/issues/11543#issuecomment-541191: > How are you testing composes on staging? I never did ;-) The truth is that staging is mostly broken because it bases on the prod db which differs from what we have in stg koji. I ever only used stg to test some basic UI functionality.
Author
Owner

I wonder how to test the change than.

@mattia Any ideas?

Could it be possible to just try running the skopeo command with some testing flatpak?

I wonder how to test the change than. @mattia Any ideas? Could it be possible to just try running the skopeo command with some testing flatpak?
Author
Owner

Currently trying to run the skope command locally and it seems that this could be enough to test it out. Although I'm having some issues with authentication to quay.io, but that was expected.

Currently trying to run the skope command locally and it seems that this could be enough to test it out. Although I'm having some issues with authentication to quay.io, but that was expected.
Author
Owner

So I was able to test out the push to quay.io, but found few problems along the way:

  1. The quay.io doesn't allow nested organizations, so I can't really use https://quay.io/fedora-testing/flatpak, as that is only repository. I had to create a new organization for testing https://quay.io/organization/fedora-flatpak-testing?tab=repos. I already tested out uploading 0ad flatpak to this organization using skopeo copy -a --retry-times=10 docker://candidate-registry.fedoraproject.org/0ad:0.27.1-2 docker://quay.io/fedora-flatpak-testing/0ad:0.27.1-2 using bot account
  2. The repository have to be created separately for each flatpak, as the bot account has permissions per repository (which needs to be added for each newly created repository). This could be probably automated to run each time when new flatpak is added.
  3. The authentication credentials for quay.io bot needs to be saved in a location skopeo can find it, either adding --authfile parameter to skopeo or using default location, which I did. In my case I just used podman login, but that isn't persistent from what I was able to find about that.
So I was able to test out the push to quay.io, but found few problems along the way: 1. The quay.io doesn't allow nested organizations, so I can't really use https://quay.io/fedora-testing/flatpak, as that is only repository. I had to create a new organization for testing https://quay.io/organization/fedora-flatpak-testing?tab=repos. I already tested out uploading 0ad flatpak to this organization using `skopeo copy -a --retry-times=10 docker://candidate-registry.fedoraproject.org/0ad:0.27.1-2 docker://quay.io/fedora-flatpak-testing/0ad:0.27.1-2` using bot account 2. The repository have to be created separately for each flatpak, as the bot account has permissions per repository (which needs to be added for each newly created repository). This could be probably automated to run each time when new flatpak is added. 3. The authentication credentials for quay.io bot needs to be saved in a location skopeo can find it, either adding `--authfile` parameter to skopeo or using default location, which I did. In my case I just used `podman login`, but that isn't persistent from what I was able to find about that.
Author
Owner

Notes from today meeting with flatpak folks:

  1. Name of the organization on quay.io should be fedora-flatpaks and fedora-flatpaks-testing (for staging)
  2. There will be automation added for creating repositories in the organization and adding permissions to that repository to upload bot, when new repository will be requested and validated. This change will be done in scm_request_processor toddler
  3. We also discussed that we should probably move candidate-registry.fedoraproject.org as well to quay.io, but that will be future target and needs changes in koji-flatpak
Notes from today meeting with flatpak folks: 1. Name of the organization on quay.io should be `fedora-flatpaks` and `fedora-flatpaks-testing` (for staging) 2. There will be automation added for creating repositories in the organization and adding permissions to that repository to upload bot, when new repository will be requested and validated. This change will be done in `scm_request_processor` toddler 3. We also discussed that we should probably move `candidate-registry.fedoraproject.org` as well to quay.io, but that will be future target and needs changes in `koji-flatpak`
Owner

Sounds reasonable to me. We should make sure we have admin/ownership of those in case we need to manage things later...

Sounds reasonable to me. We should make sure we have admin/ownership of those in case we need to manage things later...
Author
Owner

So today I was able to create the fedora-flatpaks-testing organization and found out how to overcome the initial problems with the bot permissions and the inability to create repositories automatically. That means that we don't need to do any changes in scm_request_processor and just use the bot itself to create repositories.
I just need to create a script that will move everything currently hosted on registry.fedoraproject.org to new organization on quay.io. Will test that out on staging first. It will have to be ran only once, after that the repository will be automatically created when first build of the flatpak will be copied over from candidate-registry.

So today I was able to create the [fedora-flatpaks-testing organization](https://quay.io/organization/fedora-flatpaks-testing) and found out how to overcome the initial problems with the bot permissions and the inability to create repositories automatically. That means that we don't need to do any changes in `scm_request_processor` and just use the bot itself to create repositories. I just need to create a script that will move everything currently hosted on `registry.fedoraproject.org` to new organization on `quay.io`. Will test that out on staging first. It will have to be ran only once, after that the repository will be automatically created when first build of the flatpak will be copied over from `candidate-registry`.

There are certain containers in which we have been using namespaces, mostly runtimes, runtime extensions, and EOL-rebases. How would this work on quay.io?

There are certain containers in which we have been using namespaces, mostly runtimes, runtime extensions, and EOL-rebases. How would this work on quay.io?
Owner

Also, I wonder... could we figure out a cleanup policy? ie have some way for quay.io to know whats no longer live content and can be removed? I guess that could be a more complicated sidetask.

Also, I wonder... could we figure out a cleanup policy? ie have some way for quay.io to know whats no longer live content and can be removed? I guess that could be a more complicated sidetask.
Author
Owner

@yselkowitz wrote in #11543 (comment):

There are certain containers in which we have been using namespaces, mostly runtimes, runtime extensions, and EOL-rebases. How would this work on quay.io?

As namespaces are managed as organizations on quay.io and I didn't found the way to create nested organization I think this could be a problem. Could we change that to simple quay.io/<organization>/<repository> structure?

@yselkowitz wrote in https://forge.fedoraproject.org/infra/tickets/issues/11543#issuecomment-570422: > There are certain containers in which we have been using namespaces, mostly runtimes, runtime extensions, and EOL-rebases. How would this work on quay.io? As namespaces are managed as organizations on quay.io and I didn't found the way to create nested organization I think this could be a problem. Could we change that to simple `quay.io/<organization>/<repository>` structure?
Author
Owner

@kevin wrote in #11543 (comment):

Also, I wonder... could we figure out a cleanup policy? ie have some way for quay.io to know whats no longer live content and can be removed? I guess that could be a more complicated sidetask.

I assume this could be resolved as separate issue. But it would be nice to have something like that in place.

@kevin wrote in https://forge.fedoraproject.org/infra/tickets/issues/11543#issuecomment-570459: > Also, I wonder... could we figure out a cleanup policy? ie have some way for quay.io to know whats no longer live content and can be removed? I guess that could be a more complicated sidetask. I assume this could be resolved as separate issue. But it would be nice to have something like that in place.
Sign in to join this conversation.
No milestone
No assignees
12 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
infra/tickets#11543
No description provided.