Cleaning script for communishift #11825

Closed
opened 2024-03-07 15:25:49 +00:00 by zlopez · 41 comments
Owner

Describe what you would like us to do:


As communishift is now up and announced we should create an ansible playbook to do the cleaning and update the post release docs, so it's not forgotten.

When do you need this to be done by? (YYYY/MM/DD)


Not urgent, but sooner the better

# Describe what you would like us to do: ---- As communishift is now up and announced we should create an ansible playbook to do the cleaning and update the post release docs, so it's not forgotten. # When do you need this to be done by? (YYYY/MM/DD) ---- Not urgent, but sooner the better

@zlopez do you have an example of cleaning the post release doc ?
Anyway, I can work on it , but need clarification from someone who worked on it .

@zlopez do you have an example of cleaning the post release doc ? Anyway, I can work on it , but need clarification from someone who worked on it .
Author
Owner

@kevin Do we have some docs for post release steps that needs to be done by Fedora Infra or is this part of release engineering guide?

@kevin Do we have some docs for post release steps that needs to be done by Fedora Infra or is this part of release engineering guide?
Owner

Really the 'cleaning' here should just be 'oc delete project/$i' for all projects (that are not openshift-*/system ones)

Perhaps we could enhance the existing playbook for communishift to remove the projects instead of adding them, then you run that and remove them from the list in git.

We do have https://docs.fedoraproject.org/en-US/infra/release_guide/final_release/ but perhaps we could add a 'post final release' one?

Really the 'cleaning' here should just be 'oc delete project/$i' for all projects (that are not openshift-*/system ones) Perhaps we could enhance the existing playbook for communishift to remove the projects instead of adding them, then you run that and remove them from the list in git. We do have https://docs.fedoraproject.org/en-US/infra/release_guide/final_release/ but perhaps we could add a 'post final release' one?
Author
Owner

@kevin I agree that adding post final release guide makes sense as this shouldn't be bothering people during release.

@kevin I agree that adding `post final release` guide makes sense as this shouldn't be bothering people during release.
Author
Owner

After DevConf.CZ and discussion happening there, I would like to add that part of this should be also some announcements sent to contacts for communishift projects that will let owners know the post release cleaning is coming.
This could happen a month before the release and then again 14 days, so owners have time to react.

After DevConf.CZ and discussion happening there, I would like to add that part of this should be also some announcements sent to contacts for communishift projects that will let owners know the post release cleaning is coming. This could happen a month before the release and then again 14 days, so owners have time to react.
Member

Metadata Update from @dkirwan:

  • Issue assigned to dkirwan
**Metadata Update from @dkirwan**: - Issue assigned to dkirwan

I saw this get built by someone at my university some years back, so maybe its not too hard to adapt to the communishift cluster and infra: https://github.com/WillNilges/ShelfLife

looks like it was also built for an openshift cluster and i believe it was also set up to notify people via email when their stuff was about to expire.

I saw this get built by someone at my university some years back, so maybe its not too hard to adapt to the communishift cluster and infra: https://github.com/WillNilges/ShelfLife looks like it was also built for an openshift cluster and i believe it was also set up to notify people via email when their stuff was about to expire.

As far as scheduling when things get deleted, i think it may be nice to reuse the same date as the EOL-ing of F[N-2] (so 2024-11-12 for F41, see https://fedorapeople.org/groups/schedule/f-41/f-41-key-tasks.html) to give users an extra heads up well in advance (i.e. the user docs could document that the deletion date is around the time of the EOLing of the fedora release).

That said, an arbitrary date like that that may change every cycle may be a moving target thats hard to pin down and automate things against.

As far as scheduling when things get deleted, i think it may be nice to reuse the same date as the EOL-ing of F[N-2] (so 2024-11-12 for F41, see https://fedorapeople.org/groups/schedule/f-41/f-41-key-tasks.html) to give users an extra heads up well in advance (i.e. the user docs could document that the deletion date is around the time of the EOLing of the fedora release). That said, an arbitrary date like that that may change every cycle may be a moving target thats hard to pin down and automate things against.
Author
Owner

Metadata Update from @zlopez:

  • Issue tagged with: communishift
**Metadata Update from @zlopez**: - Issue tagged with: communishift
Author
Owner

We will have this as part of Fedora Infra hackfest to discuss when and how to delete the project. The delete with EOL date makes sense, but how much before it we should notify owners of the project. And all of this should be documented as well.

For now we have two requests for not deleting the projects:

We will have this as part of Fedora Infra hackfest to discuss when and how to delete the project. The delete with EOL date makes sense, but how much before it we should notify owners of the project. And all of this should be documented as well. For now we have two requests for not deleting the projects: * [communishift-log-detective](https://pagure.io/fedora-infrastructure/issue/11868) * [communishift-fedora-review-service](https://pagure.io/fedora-infrastructure/issue/11814)
Author
Owner

As Fedora 39 EOL is coming near, we should move this forward.

@dkirwan Did you have time to work on this?

As Fedora 39 EOL is coming near, we should move this forward. @dkirwan Did you have time to work on this?
Member

@zlopez when this is needed can you assign this task of clean the cluster? I'll go through what ever is involved manually, add an ansible task and gather all the moving parts into a SOP.

@zlopez when this is needed can you assign this task of clean the cluster? I'll go through what ever is involved manually, add an ansible task and gather all the moving parts into a SOP.
Author
Owner

@dkirwan What task do you mean?

The SOP would be great, we also need to let owners of the projects know about that each cycle of cleaning. So they can react in time.

@dkirwan What task do you mean? The SOP would be great, we also need to let owners of the projects know about that each cycle of cleaning. So they can react in time.
Member

Looking at adding a playbook to retrieve project admin users emails from each project then send them a notification email about incoming project deletions.

Considering the following template:

to: "{{ item.value.comma_separated_admin_email_list}}"
from: "admin@fedoraproject.org"
subject: "Fedora Communishift Notification: {{ item.value.name }}"
body: "Dear Administrator,

This is a reminder that the Communishift project {{ item.value.name }} will be deleted during the Fedora post release process. Please ensure you have a backup of any important configuration or data from your project.

To request an exemption for project deletion please open a ticket here: https://pagure.io/fedora-infrastructure/issues"
Looking at adding a playbook to retrieve project admin users emails from each project then send them a notification email about incoming project deletions. Considering the following template: ``` to: "{{ item.value.comma_separated_admin_email_list}}" from: "admin@fedoraproject.org" subject: "Fedora Communishift Notification: {{ item.value.name }}" body: "Dear Administrator, This is a reminder that the Communishift project {{ item.value.name }} will be deleted during the Fedora post release process. Please ensure you have a backup of any important configuration or data from your project. To request an exemption for project deletion please open a ticket here: https://pagure.io/fedora-infrastructure/issues" ```
Owner

Might make it more discoverable when the date is?

I guess we decided to do this when EOLing the old fedora release. So, in this case it's 2024-11-26 that F39 goes eol. We could put in the date, or just try and say something like 'at the same time the oldest fedora is retired (1 month after a new fedora release)" or something?

I also wonder, perhaps we should make a ticket in advance and just tell people to add to that one ticket instead of filing a bunch of them?
("To request an exemption for project deletion please add your request to https://pagure.io/fedora-infrastructure/issue/12345678")

Otherwise looks good.

Might make it more discoverable when the date is? I guess we decided to do this when EOLing the old fedora release. So, in this case it's 2024-11-26 that F39 goes eol. We could put in the date, or just try and say something like 'at the same time the oldest fedora is retired (1 month after a new fedora release)" or something? I also wonder, perhaps we should make a ticket in advance and just tell people to add to that one ticket instead of filing a bunch of them? ("To request an exemption for project deletion please add your request to https://pagure.io/fedora-infrastructure/issue/12345678") Otherwise looks good.
Author
Owner

I like the idea of having one ticket created in advance, we can close it once the cleanup is done.

I like the idea of having one ticket created in advance, we can close it once the cleanup is done.
Member

Screenshot_from_2024-11-21_16-09-45.png

Updated the communishfit role, and added a new playbook. Can retrieve the communishift projects plus the email addresses of the admins from fasjson. Also have task that can then send the email template in ansible.

Final piece of work to update the To field to point at these admin email addresses.

[![Screenshot_from_2024-11-21_16-09-45.png](/fedora-infrastructure/issue/raw/files/bbbb7cb2d9a361549622783b40601d1ae073bf23d1bd398636dd2e950f621127-Screenshot_from_2024-11-21_16-09-45.png)](/fedora-infrastructure/issue/raw/files/bbbb7cb2d9a361549622783b40601d1ae073bf23d1bd398636dd2e950f621127-Screenshot_from_2024-11-21_16-09-45.png) Updated the communishfit role, and added a new playbook. Can retrieve the communishift projects plus the email addresses of the admins from fasjson. Also have task that can then send the email template in ansible. Final piece of work to update the To field to point at these admin email addresses.
Owner

Awesome. I think at this point we should give them more time (since next week is a holiday in the us and not much notice), but perhaps sometime in mid december?

Awesome. I think at this point we should give them more time (since next week is a holiday in the us and not much notice), but perhaps sometime in mid december?
Owner

Did anything happen here?

I guess we should re-notify people again and see what would be cleaned up?

Did anything happen here? I guess we should re-notify people again and see what would be cleaned up?
Member

From memory only a single tenant responded in anyway to request the projects be exempted, might need to rethink the auto deleting part entirely. Would feel happier if that was done manually, fairly easy to just oc delete project etc.. especially as it will require extra steps such as disabling/removing FAS groups as not sure disabling them will prevent fasjson returning the data, must test that.

Can do another run of the notifications steps, just let me know when we want to do the actual deleting steps, I can make sure everything is properly documented in an SOP.

From memory only a single tenant responded in anyway to request the projects be exempted, might need to rethink the auto deleting part entirely. Would feel happier if that was done manually, fairly easy to just oc delete project etc.. especially as it will require extra steps such as disabling/removing FAS groups as not sure disabling them will prevent fasjson returning the data, must test that. Can do another run of the notifications steps, just let me know when we want to do the actual deleting steps, I can make sure everything is properly documented in an SOP.
Author
Owner

The F42 release is already out and F40 EOL, so we can do the delete anytime.

Do I understand it correctly that only one tenant didn't want the project to be deleted?

The F42 release is already out and F40 EOL, so we can do the delete anytime. Do I understand it correctly that only one tenant didn't want the project to be deleted?
Member

Yeah only one tenant responded, that might mean the rest marked the reminder email read and never actually read it/saw it.

Perhaps before deleting, shut any services contained within down, and see if anyone complains?

Yeah only one tenant responded, that might mean the rest marked the reminder email read and never actually read it/saw it. Perhaps before deleting, shut any services contained within down, and see if anyone complains?
Author
Owner

Doing a scream test is not a bad idea. Let's disable the project for a week and see if there will be any response first and then delete it.

We can make that part of the official process, so there is still a time for folks to reach out.

Doing a scream test is not a bad idea. Let's disable the project for a week and see if there will be any response first and then delete it. We can make that part of the official process, so there is still a time for folks to reach out.

Perhaps before deleting, shut any services contained within down, and see if anyone complains?

This is a really good idea. Please make this a part of the official process.

Do I understand it correctly that only one tenant didn't want the project to be deleted?

Not sure if there is a public list of exceptions somewhere in playbooks and not sure if the one tenant was me. So please, just don't remove these two:

> Perhaps before deleting, shut any services contained within down, and see if anyone complains? This is a really good idea. Please make this a part of the official process. > Do I understand it correctly that only one tenant didn't want the project to be deleted? Not sure if there is a public list of exceptions somewhere in playbooks and not sure if the one tenant was me. So please, just don't remove these two: - https://pagure.io/fedora-infrastructure/issue/11868 - https://pagure.io/fedora-infrastructure/issue/11814
Owner

At this point I think re-notifying people might be in order... but yeah, if we have a way to just 'suspend' a project for a while where we could easily just re-enable it if someone comes asking could be a better way to do things.

At this point I think re-notifying people might be in order... but yeah, if we have a way to just 'suspend' a project for a while where we could easily just re-enable it if someone comes asking could be a better way to do things.
Member

https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/all#_42

@frostyx Yes we mark the fact projects should not be deleted here. I believe both your projects are marked correctly.

Not sure if there is a public list of exceptions somewhere in playbooks and not sure if the one tenant was me. So please, just don't remove these two:

https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/all#_42 @frostyx Yes we mark the fact projects should not be deleted here. I believe both your projects are marked correctly. > Not sure if there is a public list of exceptions somewhere in playbooks and not sure if the one tenant was me. So please, just don't remove these two: > > - https://pagure.io/fedora-infrastructure/issue/11868 > - https://pagure.io/fedora-infrastructure/issue/11814
Member

Finally had a minute to revisit this, since there was little to no engagement from the communishift admins the last time the emails were sent out.. i figure most are filtering emails from the admin address, so I want to send an email to the infra list and BCC the administrators. Hopefully they will pay attention to mails on the infra list.

Looks like the emails get filtered though.. can we white list them or should I look at a different way to send these notifications?

Finally had a minute to revisit this, since there was little to no engagement from the communishift admins the last time the emails were sent out.. i figure most are filtering emails from the admin address, so I want to send an email to the infra list and BCC the administrators. Hopefully they will pay attention to mails on the infra list. Looks like the emails get filtered though.. can we white list them or should I look at a different way to send these notifications?
Author
Owner

@dkirwan Do you mean that they have filters in place that filter out the e-mails? Or what do you want to whitelist here?

@dkirwan Do you mean that they have filters in place that filter out the e-mails? Or what do you want to whitelist here?
Member

So I need to be able to send emails to the infra list from the batcave.. ill look this back up and get the address tehy are coming from.

Seems our users are ignoring the ones I am sending directly as I had only one person respond/engage, and I dont want to be in the situation where I am tracking people down manually otherwise we should just skip this entirely and do that instead.

So I need to be able to send emails to the infra list from the batcave.. ill look this back up and get the address tehy are coming from. Seems our users are ignoring the ones I am sending directly as I had only one person respond/engage, and I dont want to be in the situation where I am tracking people down manually otherwise we should just skip this entirely and do that instead.
Author
Owner

Hm, I thought that sending e-mails from batcave to infrastructure@lists.fedoraproject.org should be possible, but it could be problem that the sender is not subscribed to the list.

From which e-mail address are the mails sent?

Hm, I thought that sending e-mails from batcave to infrastructure@lists.fedoraproject.org should be possible, but it could be problem that the sender is not subscribed to the list. From which e-mail address are the mails sent?
Member

I think its this address.. admin at fedoraproject.org

I think its this address.. admin at fedoraproject.org
Author
Owner

I'm not sure if we want to add admin@fedoraproject.org to infrastructure@lists.fedoraproject.org as it will add a lot of e-mails to admins inbox. @kevin What do you think about this?

I'm not sure if we want to add `admin@fedoraproject.org` to `infrastructure@lists.fedoraproject.org` as it will add a lot of e-mails to admins inbox. @kevin What do you think about this?
Owner

We can subscribe it and set it to 'no mail' so it doesn't get any of the emails.
Or set as a 'non member' address thats allowed to post.

We can subscribe it and set it to 'no mail' so it doesn't get any of the emails. Or set as a 'non member' address thats allowed to post.
Author
Owner

Let me do that then, I didn't know that there is this option.

Let me do that then, I didn't know that there is this option.
Author
Owner

During that I found out that the admin@fedoraproject.org is already a nonmember on the list.
image

@dkirwan Maybe it wasn't there last time you tried to sent the e-mail. Could you try it now?

During that I found out that the `admin@fedoraproject.org` is already a nonmember on the list. ![image](/attachments/301c1dca-0077-47aa-9bab-9781ad0b7f71) @dkirwan Maybe it wasn't there last time you tried to sent the e-mail. Could you try it now?
Author
Owner

It seems that the admin@fedoraproject.org needs to be a member, otherwise it couldn't send e-mails. So I changed the settings for the user to accept any e-mail sent from that user. @dkirwan Could you test it now?

It seems that the `admin@fedoraproject.org` needs to be a member, otherwise it couldn't send e-mails. So I changed the settings for the user to accept any e-mail sent from that user. @dkirwan Could you test it now?
Member

Seems like that fixed it @zlopez ! Can you check the emails coming in the infra list and see what we think..

Seems like that fixed it @zlopez ! Can you check the emails coming in the infra list and see what we think..
Member

Spoke with @zlopez we will give another week for tenants to respond. Then in one week will start disabling projects (2026-04-22), and 2 weeks after that date start to delete them (2026-05-06).

Spoke with @zlopez we will give another week for tenants to respond. Then in one week will start disabling projects (2026-04-22), and 2 weeks after that date start to delete them (2026-05-06).
Member

Sent out another reminder email.

And disabled a number of projects.

✅ OK to delete: communishift-avant
✅ OK to delete: communishift-commops-analytics
✅ OK to delete: communishift-commops-datanom
✅ OK to delete: communishift-discoursepolls
✅ OK to delete: communishift-eventbot
✅ OK to delete: communishift-forgejo
✅ OK to delete: communishift-fossology
✅ OK to delete: communishift-gitlabce
✅ OK to delete: communishift-jitsi
✅ OK to delete: communishift-lightspeed-build
✅ OK to delete: communishift-mattdm
✅ OK to delete: communishift-metrics
✅ OK to delete: communishift-ocm
✅ OK to delete: communishift-openscanhub
✅ OK to delete: communishift-planet
✅ OK to delete: communishift-standupbot

Will remove these next week if no one screams.

Sent out another reminder email. And disabled a number of projects. ``` ✅ OK to delete: communishift-avant ✅ OK to delete: communishift-commops-analytics ✅ OK to delete: communishift-commops-datanom ✅ OK to delete: communishift-discoursepolls ✅ OK to delete: communishift-eventbot ✅ OK to delete: communishift-forgejo ✅ OK to delete: communishift-fossology ✅ OK to delete: communishift-gitlabce ✅ OK to delete: communishift-jitsi ✅ OK to delete: communishift-lightspeed-build ✅ OK to delete: communishift-mattdm ✅ OK to delete: communishift-metrics ✅ OK to delete: communishift-ocm ✅ OK to delete: communishift-openscanhub ✅ OK to delete: communishift-planet ✅ OK to delete: communishift-standupbot ``` Will remove these next week if no one screams.
Member
Exemption added for: communishift-standupbot

Moving ahead with deleting these remaining projects:

✅ OK to delete: communishift-avant
✅ OK to delete: communishift-commops-analytics
✅ OK to delete: communishift-commops-datanom
✅ OK to delete: communishift-discoursepolls
✅ OK to delete: communishift-eventbot
✅ OK to delete: communishift-forgejo
✅ OK to delete: communishift-fossology
✅ OK to delete: communishift-gitlabce
✅ OK to delete: communishift-jitsi
✅ OK to delete: communishift-lightspeed-build
✅ OK to delete: communishift-mattdm
✅ OK to delete: communishift-metrics
✅ OK to delete: communishift-ocm
✅ OK to delete: communishift-openscanhub
✅ OK to delete: communishift-planet
``` Exemption added for: communishift-standupbot ``` Moving ahead with deleting these remaining projects: ``` ✅ OK to delete: communishift-avant ✅ OK to delete: communishift-commops-analytics ✅ OK to delete: communishift-commops-datanom ✅ OK to delete: communishift-discoursepolls ✅ OK to delete: communishift-eventbot ✅ OK to delete: communishift-forgejo ✅ OK to delete: communishift-fossology ✅ OK to delete: communishift-gitlabce ✅ OK to delete: communishift-jitsi ✅ OK to delete: communishift-lightspeed-build ✅ OK to delete: communishift-mattdm ✅ OK to delete: communishift-metrics ✅ OK to delete: communishift-ocm ✅ OK to delete: communishift-openscanhub ✅ OK to delete: communishift-planet ```
Member

Post cleanup projects:

communishift-admins                                               Active
communishift-authorization-operator                               Active
communishift-dev-test                                             Active
communishift-fedora-coreos-ai-helpers                             Active
communishift-fedora-review-service                                Active
communishift-log-detective                                        Active
communishift-standupbot                                           Active
communishift-weekly-bootc                                         Active

SOP updated https://docs.fedoraproject.org/en-US/infra/sysadmin_sops/communishift_cleanup_script/

Post cleanup projects: ``` communishift-admins Active communishift-authorization-operator Active communishift-dev-test Active communishift-fedora-coreos-ai-helpers Active communishift-fedora-review-service Active communishift-log-detective Active communishift-standupbot Active communishift-weekly-bootc Active ``` SOP updated https://docs.fedoraproject.org/en-US/infra/sysadmin_sops/communishift_cleanup_script/
Sign in to join this conversation.
No milestone
No project
No assignees
6 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
infra/tickets#11825
No description provided.