Cleaning script for communishift #11825
Labels
No labels
announcement
anubis
authentication
aws
backlog
blocked
bodhi
ci
cloud
communishift
copr
database
day-to-day
dc-move
deprecated
dev
discourse
dns
downloads
easyfix
epel
firmitas
forgejo_migration
Gain
High
Gain
Low
Gain
Medium
gitlab
greenwave
hardware
help wanted
high-trouble
koji
koschei
lists
low-trouble
medium-trouble
mirrorlists
monitoring
Needs investigation
odcs
OpenShift
ops
outage
packager_workflow_blocker
pagure
permissions
Priority
Needs Review
Priority
Next Meeting
Priority
🔥 URGENT 🔥
Priority
Waiting on Assignee
Priority
Waiting on External
Priority
Waiting on Reporter
rabbitmq
release-monitoring
releng
request-for-resources
s390x
security
SMTP
sprint-0
sprint-1
src.fp.o
staging
unfreeze
waiverdb
websites-general
wiki
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
No milestone
No project
No assignees
6 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
infra/tickets#11825
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Describe what you would like us to do:
As communishift is now up and announced we should create an ansible playbook to do the cleaning and update the post release docs, so it's not forgotten.
When do you need this to be done by? (YYYY/MM/DD)
Not urgent, but sooner the better
@zlopez do you have an example of cleaning the post release doc ?
Anyway, I can work on it , but need clarification from someone who worked on it .
@kevin Do we have some docs for post release steps that needs to be done by Fedora Infra or is this part of release engineering guide?
Really the 'cleaning' here should just be 'oc delete project/$i' for all projects (that are not openshift-*/system ones)
Perhaps we could enhance the existing playbook for communishift to remove the projects instead of adding them, then you run that and remove them from the list in git.
We do have https://docs.fedoraproject.org/en-US/infra/release_guide/final_release/ but perhaps we could add a 'post final release' one?
@kevin I agree that adding
post final releaseguide makes sense as this shouldn't be bothering people during release.After DevConf.CZ and discussion happening there, I would like to add that part of this should be also some announcements sent to contacts for communishift projects that will let owners know the post release cleaning is coming.
This could happen a month before the release and then again 14 days, so owners have time to react.
Metadata Update from @dkirwan:
I saw this get built by someone at my university some years back, so maybe its not too hard to adapt to the communishift cluster and infra: https://github.com/WillNilges/ShelfLife
looks like it was also built for an openshift cluster and i believe it was also set up to notify people via email when their stuff was about to expire.
As far as scheduling when things get deleted, i think it may be nice to reuse the same date as the EOL-ing of F[N-2] (so 2024-11-12 for F41, see https://fedorapeople.org/groups/schedule/f-41/f-41-key-tasks.html) to give users an extra heads up well in advance (i.e. the user docs could document that the deletion date is around the time of the EOLing of the fedora release).
That said, an arbitrary date like that that may change every cycle may be a moving target thats hard to pin down and automate things against.
Metadata Update from @zlopez:
We will have this as part of Fedora Infra hackfest to discuss when and how to delete the project. The delete with EOL date makes sense, but how much before it we should notify owners of the project. And all of this should be documented as well.
For now we have two requests for not deleting the projects:
As Fedora 39 EOL is coming near, we should move this forward.
@dkirwan Did you have time to work on this?
@zlopez when this is needed can you assign this task of clean the cluster? I'll go through what ever is involved manually, add an ansible task and gather all the moving parts into a SOP.
@dkirwan What task do you mean?
The SOP would be great, we also need to let owners of the projects know about that each cycle of cleaning. So they can react in time.
Looking at adding a playbook to retrieve project admin users emails from each project then send them a notification email about incoming project deletions.
Considering the following template:
Might make it more discoverable when the date is?
I guess we decided to do this when EOLing the old fedora release. So, in this case it's 2024-11-26 that F39 goes eol. We could put in the date, or just try and say something like 'at the same time the oldest fedora is retired (1 month after a new fedora release)" or something?
I also wonder, perhaps we should make a ticket in advance and just tell people to add to that one ticket instead of filing a bunch of them?
("To request an exemption for project deletion please add your request to https://pagure.io/fedora-infrastructure/issue/12345678")
Otherwise looks good.
I like the idea of having one ticket created in advance, we can close it once the cleanup is done.
Updated the communishfit role, and added a new playbook. Can retrieve the communishift projects plus the email addresses of the admins from fasjson. Also have task that can then send the email template in ansible.
Final piece of work to update the To field to point at these admin email addresses.
Awesome. I think at this point we should give them more time (since next week is a holiday in the us and not much notice), but perhaps sometime in mid december?
Did anything happen here?
I guess we should re-notify people again and see what would be cleaned up?
From memory only a single tenant responded in anyway to request the projects be exempted, might need to rethink the auto deleting part entirely. Would feel happier if that was done manually, fairly easy to just oc delete project etc.. especially as it will require extra steps such as disabling/removing FAS groups as not sure disabling them will prevent fasjson returning the data, must test that.
Can do another run of the notifications steps, just let me know when we want to do the actual deleting steps, I can make sure everything is properly documented in an SOP.
The F42 release is already out and F40 EOL, so we can do the delete anytime.
Do I understand it correctly that only one tenant didn't want the project to be deleted?
Yeah only one tenant responded, that might mean the rest marked the reminder email read and never actually read it/saw it.
Perhaps before deleting, shut any services contained within down, and see if anyone complains?
Doing a scream test is not a bad idea. Let's disable the project for a week and see if there will be any response first and then delete it.
We can make that part of the official process, so there is still a time for folks to reach out.
This is a really good idea. Please make this a part of the official process.
Not sure if there is a public list of exceptions somewhere in playbooks and not sure if the one tenant was me. So please, just don't remove these two:
At this point I think re-notifying people might be in order... but yeah, if we have a way to just 'suspend' a project for a while where we could easily just re-enable it if someone comes asking could be a better way to do things.
https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/all#_42
@frostyx Yes we mark the fact projects should not be deleted here. I believe both your projects are marked correctly.
Finally had a minute to revisit this, since there was little to no engagement from the communishift admins the last time the emails were sent out.. i figure most are filtering emails from the admin address, so I want to send an email to the infra list and BCC the administrators. Hopefully they will pay attention to mails on the infra list.
Looks like the emails get filtered though.. can we white list them or should I look at a different way to send these notifications?
@dkirwan Do you mean that they have filters in place that filter out the e-mails? Or what do you want to whitelist here?
So I need to be able to send emails to the infra list from the batcave.. ill look this back up and get the address tehy are coming from.
Seems our users are ignoring the ones I am sending directly as I had only one person respond/engage, and I dont want to be in the situation where I am tracking people down manually otherwise we should just skip this entirely and do that instead.
Hm, I thought that sending e-mails from batcave to infrastructure@lists.fedoraproject.org should be possible, but it could be problem that the sender is not subscribed to the list.
From which e-mail address are the mails sent?
I think its this address.. admin at fedoraproject.org
I'm not sure if we want to add
admin@fedoraproject.orgtoinfrastructure@lists.fedoraproject.orgas it will add a lot of e-mails to admins inbox. @kevin What do you think about this?We can subscribe it and set it to 'no mail' so it doesn't get any of the emails.
Or set as a 'non member' address thats allowed to post.
Let me do that then, I didn't know that there is this option.
During that I found out that the

admin@fedoraproject.orgis already a nonmember on the list.@dkirwan Maybe it wasn't there last time you tried to sent the e-mail. Could you try it now?
It seems that the
admin@fedoraproject.orgneeds to be a member, otherwise it couldn't send e-mails. So I changed the settings for the user to accept any e-mail sent from that user. @dkirwan Could you test it now?Seems like that fixed it @zlopez ! Can you check the emails coming in the infra list and see what we think..
Spoke with @zlopez we will give another week for tenants to respond. Then in one week will start disabling projects (2026-04-22), and 2 weeks after that date start to delete them (2026-05-06).
Sent out another reminder email.
And disabled a number of projects.
Will remove these next week if no one screams.
Moving ahead with deleting these remaining projects:
Post cleanup projects:
SOP updated https://docs.fedoraproject.org/en-US/infra/sysadmin_sops/communishift_cleanup_script/