package ACL cleanup for packages that are retired on all branches currently doesn't happen #13155
Labels
No labels
announcement
anubis
authentication
aws
backlog
blocked
bodhi
ci
cloud
communishift
copr
database
day-to-day
dc-move
deprecated
dev
discourse
dns
downloads
easyfix
epel
firmitas
forgejo_migration
Gain
High
Gain
Low
Gain
Medium
gitlab
greenwave
hardware
help wanted
high-trouble
koji
koschei
lists
low-trouble
medium-trouble
mirrorlists
monitoring
Needs investigation
odcs
OpenShift
ops
outage
packager_workflow_blocker
pagure
permissions
Priority
Needs Review
Priority
Next Meeting
Priority
🔥 URGENT 🔥
Priority
Waiting on Assignee
Priority
Waiting on External
Priority
Waiting on Reporter
rabbitmq
release-monitoring
releng
request-for-resources
s390x
security
SMTP
sprint-0
sprint-1
src.fp.o
staging
unfreeze
waiverdb
websites-general
wiki
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
infra/tickets#13155
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description of request
It appears that the process / cron job / toddler / poddler which should handle cleaning up ACLs (i.e. removing all users / groups and assigning to "orphan") for packages that are "retired on all active branches" isn't running.
Example: https://src.fedoraproject.org/rpms/deepin-kwin
So...
The pod was in a weird state (because of reboots I think). So, I restarted it and fired off the cronjob...
It didn't find that package, so I think something is wrong with it or the toddler that processes these.
I do see it in retired-packages...
@lenkaseg could you take a look?
The cron job seems to be running as it should, once a month at 8AM:
I see the package is assigned to
orphanin both Fedora and EPEL and has no maintainers.@decathorpe Could you give me a hint what to look for on the src page? The members? I see 4 members currently there, should those be removed by the toddler? I'm not too familiar with this toddler.
As far as I know, the three non-orphan maintainers should get removed. Or maybe I'm mis-remembering something here?
I found another example where it might more clear: https://src.fedoraproject.org/rpms/rust-env_logger0.4
This package is retired in all current Fedora branches, and this has been the case since the Fedora 41 EOL, so at least more than a month ago. But the cleanup hasn't touched it yet at all.
Comparing
deepin-kwinpackage with processedrust-glycin2, I see:For rust-glycin2:
For deepin-kwin:
So I suppose those fields
commitandadminshould be empty and they are not.I'll go step by step to check where the problem is.
I think the problem of
deepin-kwinpackage is, that it's retired only onrawhide. So the toddler did not identify it as fully retired back then (last branch there is for the package is f41). Now f41 is not even active, so the toddler will not pick it up even if it would be retired on all it's branches now. The same applies forrust-env_logger0.4and it's likely there is more such packages.The script of the toddler clean_retired_packages checks if the package is retired on all active branches.
The toddler never identifies
deepin-kwinandrust-env_logger0.4as fully retired.What is the definition of fully retired package?
That sounds like a logic flaw in the checks to me then? :(
For example, deepin-kwin was retired in rawhide before the f42 branch point, so it doesn't even have f42, f43, or f44 branches, so it is, for all intents and purposes, retired on all active branches.
It is blocked from the f45, f44, f43, f42 tags in koji (and was never in EPEL, so the package isn't in any of the epel* tags), so maybe that would be an easier way to check? i.e. get list of koji tags corresponding to active releases, and check if package is blocked in all of them.
I think in the end it's comparing two lists:
['rawhide'], which is the only active branch on which
deepin-kwinis retired with['f30',
'f31',
'f32',
'f33',
'f34',
'f35',
'f36',
'f37',
'f38',
'f39',
'f40',
'f41',
'main',
'rawhide']
which are all
deepin-kwins branches. Since these two lists don't match, the package is never identified as fully retired and is not processed.Now I'd need to know what's the best approach here.
Is being retired on rawhide and at the same time being orphaned qualifies the package for the ACL cleanup?
@decathorpe wrote in #13155 (comment):
Sounds good, will do!
Lol, interesting behavior of the forge here.
When I want to assign this issue to me - by clicking "Assign to me" in the assign section, it gets assigned to abompard (I suppose because he is the first on the list of allowed assignees and I'm not in that list).
I suppose I'd have to be added to the forge infra members group to be able to assign tickets to me here.
I've added you.
So, I made a script to identify all packages that are:
example:
active releases are ['eln', 'epel10.1', 'epel10.2', 'epel10', 'epel8', 'epel9', 'epel9-next', 'f42', 'f43', 'f44', 'rawhide']
deepin-kwin's branches are ['f30', 'f31', 'f32', 'f33', 'f34', 'f35', 'f36', 'f37', 'f38', 'f39', 'f40','f41','main','rawhide']
The intersection would be ['rawhide'].
=> when package is blocked in koji to each of it's active branch, then it is considered fully retired and sent for cleanup.
Could someone check this logic please? Because it identified a WHOLE LOT of such packages, does not feel right. I checked some of them randomly, would all these qualify for cleanup?
Only a sample, there were hundreds:
ant-antlr3, antic, deepin-kwin, rust-env_logger0.4, deepin-metacity, deepin-network-utils, deepin-qt5dxcb-plugin, deepin-topbar, contour, ...
More of the log: https://gist.github.com/LenkaSeg/8d944d56d31d6437141a25c046e7bb5a
I opened this pull request:
apps/toddlers#394
Does the logic of identifying a fully retired packages make sense? Not too sure here.
Here's the script I used to look for the fully retired packages: https://gist.github.com/LenkaSeg/aec473156b05b88d8b33ae7c09913837
Getting the koji tags of retired's package's branches that correspond with active releases and checking if the package is blocked on all of them.
(The script is a bit cumbersome I know, but actually handwritten without AI :) )
I think the logic makes sense to me. I spot checked a few packages and they all do indeed seem retired for a while, etc.
I think before we roll this out we might ask more widely if anyone sees any problems with this list.
In that case I'll optimize the script and run it once more to get the full list of packages.
Improved the script a bit, but it still crashed in mid-rawhide loop.
Anyways, it collected more packages that are fully retired according to this new logic: https://gist.githubusercontent.com/LenkaSeg/11f06ad02b1e2f2745d38c02fa2bae04/raw/68885355a00ddf45c4123284f47cf4ddc7def876/gistfile1.txt
sheesh, that seems like a pretty long list indeed... but again checking some seems they are all really fully retired.
Talking about this issue with @abompard and I will add a filter to the script, that the output would show only packages that were not processed yet (they are blocked on all branches, that are active and at the same time have at least one access_user that is not orphan - so that would mean the package it is not fully processed.) That should filter out all of the fully processed retired packages and give us a smaller list of only those that would need processing. I'd ignore the access_groups for now.
Improved the script to filter out the already processed packages: https://gist.github.com/LenkaSeg/b844ea74f0ef3f890c6cb27db5253f9e
Looking at the output, checked some packages and they seem to be fully retired and unprocessed.
Sample:
Full list coming soon.
More complete log:
https://gist.githubusercontent.com/LenkaSeg/24f92264dcd6176bd9744a6fb74252b2/raw/990065a89c846b49d2a432c40d75dfadf3c8440a/gistfile1.txt