Migration of registry.fedoraproject.org to quay.io #11543
Labels
No labels
announcement
anubis
authentication
aws
backlog
blocked
bodhi
ci
cloud
communishift
copr
database
day-to-day
dc-move
deprecated
dev
discourse
dns
downloads
easyfix
epel
firmitas
forgejo_migration
Gain
High
Gain
Low
Gain
Medium
gitlab
greenwave
hardware
help wanted
high-trouble
koji
koschei
lists
low-trouble
medium-trouble
mirrorlists
monitoring
Needs investigation
odcs
OpenShift
ops
outage
packager_workflow_blocker
pagure
permissions
Priority
Needs Review
Priority
Next Meeting
Priority
🔥 URGENT 🔥
Priority
Waiting on Assignee
Priority
Waiting on External
Priority
Waiting on Reporter
rabbitmq
release-monitoring
releng
request-for-resources
s390x
security
SMTP
sprint-0
sprint-1
src.fp.o
staging
unfreeze
waiverdb
websites-general
wiki
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
No milestone
No project
No assignees
12 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
infra/tickets#11543
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Describe what you would like us to do:
As the investigation is now done, we can follow the document and do the migration. It will be best to wait after Fedora 39 will be out as this will allow us to get rid of plenty of legacy workflow.
We will need some transition period, so the users still can access the content on registry.fp.o until we move everything on or it's no longer needed.
When do you need this to be done by? (YYYY/MM/DD)
Not urgent, needs to wait till F39 will be out.
Just to repeat what I mentioned a while before on the investigation ticket.
Toolbx still uses
registry.fedoraproject.orgfor itsfedora-toolboximages. From Fedora 39 onwards these images are produced by Fedora's nightly composes.We are open to moving to
quay.iobut we need some migration period because the URL to the image is baked into thetoolboxRPM, and we can't change it without knowing exactly what it would be.FWIW, IMHO, we should try and redirect existing registry.fedoraproject.org to quay.io if tools will understand that. Or I guess proxy them if not. At least initially. This will allow us to 'fail back' if there's problems or we need to down the road. But I am not sure the details on if thats possible.
Metadata Update from @zlopez:
So, there is some question about how we want to do things.
I personally think we should keep things pointed to the registry.fedoraproject.org, and then that redirects them to quay.io for the actual image.
The advantage of this is that we control it, and can repoint it somewhere else if we like, or run our own registry for some part of it and redirect other things.
It also means we can change it in our infra and it immediately would take effect.
If on the other hand, we change everything to say 'use quay.io', there's the advantage that we don't have to redirect and it's all on them.
But the downside is that if ever we need to change it again, we will have to change it in all the places it's currently set (documentation, podman, toolbox, etc).
Any changes will take rebuilding things and pushing out updates and be a long and slow process.
Can folks chime in with their thoughts on this so we can decide how to approach it?
Yes, I am with you on this one, assuming that the underlying technical realities let us use
registry.fedoraproject.orgas a front.A hard switch to
quay.iocan be done later once things have settled and we are confident about it. Until then, the possibly of a revert and all the associated churn, makes it wise to continue pointing the clients toregistry.fedoraproject.org.I am also +1 to this approach of keeping registry.fp.o as a front and then redirecting. I still would want to check out the latency situation after the redirect. Any chance you can give us a timeline for testing?
Oh, this is indeed a good idea. We started "migrating" to quay.io images in Fedora CoreOS land but this looks like a better approach if this works well.
May I ask what's the official place for Fedora containers now?
A documentation https://docs.fedoraproject.org/en-US/containers/#_where_to_find_the_containers points to registry.fedoraproject.org where the images were not updated since November 2023 https://registry.fedoraproject.org/repo/fedora/tags/.
Most of the projects hosted on registry.fp.o either moved or moving to quay.io. There is a Fedora organization on quay.io, where you should find most of the images.
Yes, but note... IMHO we should just keep using registry.fedoraproject.org and have it redirect, so in that case we shouldn't need to change any docs.
Yes, just keep using
registry.fedoraproject.org. :)To clarify, are we saying that registry.fp.o should still be the canonical repo and that Quay.io is an implementation detail, or that the redirect is more of a backcompat thing but that the expectation is that eventually we shut down registry.fp.o? Clearly, the quay.io repos have already gone around and been documented quite some time now, so I don't know how realistic it is to consider it an implementation detail at this point.
A downside I raised in https://github.com/coreos/fedora-coreos-config/pull/2988 is that using registry.fp.o means that service is now also in your dependency path. Being dependent on less things being up is very valuable in a CI context.
Yeah, I guess I was imagining that we would keep registry.fedoraproject.org for a while redirecting to quay.io and make sure there were not any big issues that would cause us to roll back.
If we did have to roll back we just change dns, instead of updaing a bunch of packages, etc.
You do make a good point that it adds another 'hop' in there and also makes us a dependency...
I think it's fine to keep the redirector, but maybe we don't promote it? i.e. if people are using registry.fedoraproject.org then it would keep working.. but our docs and things all get updated to the new location?
What's the latest thinking on this topic? Is there still a possibility that we might roll back away from
quay.ioand pointregistry.fedoraproject.orgat something else?I am asking because I want to figure out if Toolbx should start using the new
quay.ioURLs or stick toregistry.fedoraproject.org.The plan is definitely still to go to quay.io, just too many fires burning to get to it.
We need to sort out flatpak stuff still.
If you want to go go quay.io it should be already in sync for toolbox, so go for it.
Ok, understood. Thanks, Kevin.
This is still waiting for the flatpak to migrate.
@otaylor is much more familiar with the delivery mechanics of flatpaks than I am, but not sure how much time he has for this right now. In his absence, I guess it will be up to me, but I'll need to be brought up to speed.
I'm now trying to look at pointing the flatpak-indexer to quay.io and see if that works.
Finally got to it and found out where to obtain test data for tests. I will need to remove the ODCS part as this is causing for some reason issues on staging.
Currently trying to get the flatpak-indexer running on my own machine, but I hit this issue and waiting for some solution https://github.com/fedora-infra/fedora-messaging/issues/440
@zlopez are we still in process for this migration?
@smilner I still have this on my TODO list, but currently have other priorities.
Metadata Update from @zlopez:
Metadata Update from @zlopez:
@pingou submitted a pull request to move Toolbx away from
registry.fedoraproject.orgURLs to theirquay.iocounterparts. I am curious if theregistry.fedoraproject.orgURLs are about to stop working soon. If so, we need to migratetoolbox(1)away from them on older distributions.The last thing remaining is to point
flatpak-indexertoquay.io, but not sure how long does that take.So the
registry.fedoraproject.orgwill not go away anytime soon.My apologies. I should have been more specific in my question.
Does it mean that
registry.fedoraproject.orgwill be going away afterflatpak-indexeris pointed atquay.io? Any rough idea of what soon looks like?I am asking because we will have to migrate away from
registry.fedoraproject.orgon distributions as old as RHEL 8.10, where it's impossible to justify casual changes. So, we need some lead time to prepare for it.I am wondering if we should already start at least migrating the code and the documentation in newer releases because this is a matter of time, or we can let the status quo continue for the foreseeable future.
@kevin What is the plan for registry.fedoraproject.org after we migrate everything from it. Will it stay or do we want to decommission it?
My orig plan was to keep it around, but after talking with folks and thinking about it more, I think we want to retire it.
That said, there's no date currently, we need to sort out flatpaks and then anything else still left and only then we can set a retirement date.
So, I wouldn't stress over it. We also are happy to work with stakeholders on setting that date...
Okay, thanks for clarifying that @zlopez & @kevin !
I will get started with migrating Toolbx away from
registry.fedoraproject.org, and we'll see how the situation with old distributions shakes out.Hum, what changed here? Is keeping the redirect that much load on the infra?
If we really want to do this, we would have to update all the references to
registry.fedoraproject.orgin all the Containerfiles everywhere and make sure that the container stack redirects to the right place. That would be a massive change.Sorry, perhaps I misunderstood the context here or did not explain myself well. :(
From my view nothing has changed.
I'm 100% fine keeping a redirect, but what I want is to no longer push to two places as we do now, and no longer maintain our own registry running docker-distribution, and no longer have...
20TB and growing space used.
So, yes, the redirect/name will stay, but the backend/content I hope we can retire in favor of just having that once at quay.io.
Does that clarify?
OK, thanks for the clarification. I'm on board with keeping just a redirect from
registry.fedoraproject.orgtoquay.ioand retiring our own hosted registry.I did all the necessary changes to make flatpak-indexer work with latest upstream and quay.io. The problem is that the flatpak-indexer is just generating deltas and metadata for flatpaks. It doesn't do any uploading. So currently trying to discuss this with Fedora Flatpaks folks where this is being done as there isn't anything at all on quay.io.
Yeah, currently I think the flow is:
An update is submitted and goes to testing in bodhi and it syncs it from koji to candidate-registry.
When the update goes to stable it copies it from candidate-registry to registry.
So, I guess we need to teach bodhi to upload to quay.io (in addition to local registries)
IIUC it is koji(-flatpak) that uploads to candidate-registry as part of the build. Bodhi then copies that image to registry as part of the update flow. registry.fp.o is the hard-coded default in bodhi but it is configurable per https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/base/templates/production.ini.j2#_209
I will start looking into it probably next week, as I understand we want it to upload to both registry.fedoraproject.org and quay.io for some transition period.
Do we want to move also candidate-registry to quay.io?
Yes, we will need a place for bodhi to put things that are in updates-testing before moving to updates.
I think this should just be a slightly differently named repo in the fedora quay.io org.
I understand that, does
flatpak-updates-testingsounds right?All flatpaks go through bodhi so all tags have
-updates. I don't think we need to include that in quay.io namespaces though.Yeah, I'd think 'flatpak-testing' would be fine.
Just to make it clear, do we need any change in https://pagure.io/koji-flatpak? From what I understand that just uploads to
candidate-registrybefore bodhi uploads it toupdates-testing. Or isupdates-testingandcandidate-registrythe same place?IIUC koji-flatpak handles only the initial upload to candidate-registry during the build. Everything after that is handled by bodhi. So unless/until candidate-registry is changing, no changes to koji-flatpak should be necessary.
Well, thats something we should decide I guess.
We could:
The advantage of 1 is that we are not dependent on another external service...
The advantage of 2 is that we are not required to keep our own registry around anymore.
Maybe switching candidate-registry should be considered (and tracked) separately, as it is not relevant to decommissioning registry?
In that case, let's make candidate-registry separate issue and we can resolve it later.
I will start looking at Bodhi for the registry move.
So I have PR for Bodhi in place, but I wonder how is the best way to test it. If it's better to just build bodhi package in infra-stg tag with the patch and test it on staging or just try to test it out in local development environment.
I will probably try to start with local environment and if that doesn't work, try the staging approach.
So I tried to test the changes in local dev environment, but to do that it needs to start a new compose which seems to not work in local dev environment.
I asked in #bodhi channel what is the best way to test the changes out as if they already have some process in place I don't want to mess with that.
@mattia promised he will do deployment of my changes in staging to test it out. I will wait till that is in place to test it out.
@zlopez wrote in #11543 (comment):
I was looking at deploy the PR yesterday, then I just left a comment on it. Shall we use 'quay.io/fedora/flatpak' for staging tests or is there a different namespace?
Since the setting is changing from string to a list, I need to fix the config at the same time I push the commit into staging, otherwise bodhi will crash.
@mattia For staging we have
quay.io/fedora-testingflatpak.@zlopez I have built the bodhi-server rpm with your changes in f43-infra-stg.
Now we need this config change to be pushed to ansible and rebuild the base image and deploy staging bodhi, but ansible is now being moved to the new forge, so I can't do that now. I'll look when the move is over.
Sorry about that.
Everything should be moved now, so you should be able to fork the forge repo and push your changes there and make a pr.
The PR is ready, waiting for someone to merge it.
Merged.
I've pushed the patched bodhi into stg, happy testing! ;-)
Let me know if you need to synch the running db to a newer version, stg still has rawhide as F43...
@mattia You are right, the db needs to be updated. I installed the version you built in
f43-infra-stgand updated the config on staging. But when I try to pushF42Fcompose I just get that the packages are not in correct tag and looking at koji on staging, that is right.Could you update the db on staging? I assume I will hit some issues with permissions for the user on quay.io, but that is what this testing is about.
@zlopez done, the staging db is now in synch with prod db as of today.
Does this mean that registry.fp.o content will no longer be available to Syrians? We just unblocked them, but my understanding is that it's still blocked for Red Hat commercial properties (of which quay.io is one of them).
@ngompa wrote in #11543 (comment):
Perhaps @jflory7 could look into this ?
@mattia wrote in #11543 (comment):
Thanks, will try it again today.
Encountered a problem when testing the bodhi on staging and wondering how to fix it. Each time I try to start a compose the builds are missing on koji.stg.fedoraproject.org (which is understandable as the bodhi was synced from prod) and the compose fails on that. @mattia How are you testing composes on staging?
@zlopez wrote in #11543 (comment):
I never did ;-)
The truth is that staging is mostly broken because it bases on the prod db which differs from what we have in stg koji. I ever only used stg to test some basic UI functionality.
I wonder how to test the change than.
@mattia Any ideas?
Could it be possible to just try running the skopeo command with some testing flatpak?
Currently trying to run the skope command locally and it seems that this could be enough to test it out. Although I'm having some issues with authentication to quay.io, but that was expected.
So I was able to test out the push to quay.io, but found few problems along the way:
skopeo copy -a --retry-times=10 docker://candidate-registry.fedoraproject.org/0ad:0.27.1-2 docker://quay.io/fedora-flatpak-testing/0ad:0.27.1-2using bot account--authfileparameter to skopeo or using default location, which I did. In my case I just usedpodman login, but that isn't persistent from what I was able to find about that.Notes from today meeting with flatpak folks:
fedora-flatpaksandfedora-flatpaks-testing(for staging)scm_request_processortoddlercandidate-registry.fedoraproject.orgas well to quay.io, but that will be future target and needs changes inkoji-flatpakSounds reasonable to me. We should make sure we have admin/ownership of those in case we need to manage things later...
So today I was able to create the fedora-flatpaks-testing organization and found out how to overcome the initial problems with the bot permissions and the inability to create repositories automatically. That means that we don't need to do any changes in
scm_request_processorand just use the bot itself to create repositories.I just need to create a script that will move everything currently hosted on
registry.fedoraproject.orgto new organization onquay.io. Will test that out on staging first. It will have to be ran only once, after that the repository will be automatically created when first build of the flatpak will be copied over fromcandidate-registry.There are certain containers in which we have been using namespaces, mostly runtimes, runtime extensions, and EOL-rebases. How would this work on quay.io?
Also, I wonder... could we figure out a cleanup policy? ie have some way for quay.io to know whats no longer live content and can be removed? I guess that could be a more complicated sidetask.
@yselkowitz wrote in #11543 (comment):
As namespaces are managed as organizations on quay.io and I didn't found the way to create nested organization I think this could be a problem. Could we change that to simple
quay.io/<organization>/<repository>structure?@kevin wrote in #11543 (comment):
I assume this could be resolved as separate issue. But it would be nice to have something like that in place.