releng/tickets

Fork 2

some package retirements are not getting processed #13263

New issue

Open

opened 2026-03-12 12:38:24 +00:00 by decathorpe · 13 comments

decathorpe commented

2026-03-12 12:38:24 +00:00

Contributor

Describe the issue

I have retired some packages during / shortly after the F44 Beta Freeze, and they are not getting blocked in koji as they should be. I have waited a few days since the F44 Beta Freeze ended to make sure it's not just a delay in processing, but they're still not blocked and still available from repositories:

rust-indicatif0.16 on f44
rust-quick-xml0.17 on f44
rust-argfile0.2 on rawhide/f45 and f44
rust-linux-raw-sys0.11 on rawhide/f45 and f44

Packages that I have retired today were blocked correctly, so it looks like there's a gap between F44 Beta Freeze and now where some packages were not processed correctly, i.e. neither by the async message queue watching thing, nor by the daily cron job thing.

When do you need this? (YYYY/MM/DD)

Before the start of the F44 Final Freeze: 2026-03-31

When is this no longer needed or useful? (YYYY/MM/DD)

N/A

If we cannot complete this, what is the impact? [Dependencies/Blocker]

Retired packages remain available in repositories.

Checklist

I have checked existing issues for duplicates
I have filled out all the fields above
I have provided relevant links or references (if applicable)

## Describe the issue I have retired some packages during / shortly after the F44 Beta Freeze, and they are not getting blocked in koji as they should be. I have waited a few days since the F44 Beta Freeze ended to make sure it's not just a delay in processing, but they're still not blocked and still available from repositories: - rust-indicatif0.16 on f44 - rust-quick-xml0.17 on f44 - rust-argfile0.2 on rawhide/f45 and f44 - rust-linux-raw-sys0.11 on rawhide/f45 and f44 Packages that I have retired today were blocked correctly, so it looks like there's a gap between F44 Beta Freeze and now where some packages were not processed correctly, i.e. neither by the async message queue watching thing, nor by the daily cron job thing. ## When do you need this? (YYYY/MM/DD) Before the start of the F44 Final Freeze: 2026-03-31 ## When is this no longer needed or useful? (YYYY/MM/DD) N/A ## If we cannot complete this, what is the impact? [Dependencies/Blocker] Retired packages remain available in repositories. --- ## Checklist - [x] I have checked existing issues for duplicates - [x] I have filled out all the fields above - [x] I have provided relevant links or references (if applicable)

patrikp added the

investigation

ops

labels

2026-03-13 08:40:53 +00:00

patrikp commented

2026-03-13 08:48:29 +00:00

Member

@lenkaseg
Hello, is there any chance you could take a look at this when you get the time? Thanks a lot.

@lenkaseg Hello, is there any chance you could take a look at this when you get the time? Thanks a lot.

decathorpe commented

2026-03-13 11:32:22 +00:00

Author

Contributor

Additional cases from today:

rust-argfile0.2 in epel10.3 and epel9
rust-linux-raw-sys0.11 in epel10.3 and epel9
rust-atty in epel10.3
rust-ordered-float4 in epel10.3 and epel9

Additional cases from today: - rust-argfile0.2 in epel10.3 and epel9 - rust-linux-raw-sys0.11 in epel10.3 and epel9 - rust-atty in epel10.3 - rust-ordered-float4 in epel10.3 and epel9

decathorpe commented

2026-03-15 17:20:02 +00:00

Author

Contributor

More cases:

rust-toml0.7 in f45, epel9, epel9-next
rust-toml_edit0.19 in f45, epel9, epel9-next
rust-winnow0.5 in f45, epel9

More cases: - rust-toml0.7 in f45, epel9, epel9-next - rust-toml_edit0.19 in f45, epel9, epel9-next - rust-winnow0.5 in f45, epel9

decathorpe commented

2026-03-21 16:33:24 +00:00

Author

Contributor

It looks like processing of retired packages is still very unreliable. Either the message-based one or the cron job (or both) seem to be missing things.

I checked all my recently retired packages again, and these are still not processed correctly:

package	retired branches	missing blocks
rust-indicatif0.16	f45, f44	f44
rust-quick-xml0.17	f45, f44	f44
rust-argfile0.2	f45, f44, epel10, epel9	f45, f44, epel9
rust-linux-raw-sys0.11	f45, f44, epel10, epel9	f45, f44, epel9

Other missing blocks from previous comments have been (partially) processed in the meantime:

rust-argfile0.2: only epel10.3 tag blocked, others missing (see above)
rust-linux-raw-sys0.11: only epel10.3 tag blocked, others missing (see above)
rust-atty: correctly processed now
rust-ordered-float4: correctly processed now
rust-toml0.7, rust-toml_edit0.19, rust-winnow0.5: correctly processed now

I really don't understand why some packages don't seem to get processed at all, while others are only getting processed partially or only with a few days' delay.

It looks like processing of retired packages is still very unreliable. Either the message-based one or the cron job (or both) seem to be missing things. I checked all my recently retired packages again, and these are still not processed correctly: | package | retired branches | missing blocks | | --- | --- | --- | | rust-indicatif0.16 | f45, f44 | f44 | | rust-quick-xml0.17 | f45, f44 | f44 | | rust-argfile0.2 | f45, f44, epel10, epel9 | f45, f44, epel9 | | rust-linux-raw-sys0.11 | f45, f44, epel10, epel9 | f45, f44, epel9 | Other missing blocks from previous comments have been (partially) processed in the meantime: - rust-argfile0.2: only epel10.3 tag blocked, others missing (see above) - rust-linux-raw-sys0.11: only epel10.3 tag blocked, others missing (see above) - rust-atty: correctly processed now - rust-ordered-float4: correctly processed now - rust-toml0.7, rust-toml_edit0.19, rust-winnow0.5: correctly processed now I really don't understand why some packages don't seem to get processed at all, while others are only getting processed partially or only with a few days' delay.

jnsamyak added this to the Sprint ( 2026-03-23 to 2026-04-05 ) project

2026-03-23 13:56:49 +00:00

jnsamyak self-assigned this

2026-03-23 13:56:52 +00:00

jnsamyak added

and removed

labels

2026-03-23 13:56:56 +00:00

lenkaseg commented

2026-03-24 09:22:08 +00:00

Contributor

Looking at it.
I see the problem. Fetching the epel10.2 retired packages fails because there is no such a branch, it's epel10 I suppose. It crashes and never continues to process the fedora branches.
Fixing.

Looking at it. I see the problem. Fetching the epel10.2 retired packages fails because there is no such a branch, it's epel10 I suppose. It crashes and never continues to process the fedora branches. Fixing.

lenkaseg commented

2026-03-24 09:44:01 +00:00

Contributor

Is this assumption correct?
epel8 => koji tag epel8
epel9 => koji tag epel9
epel10.1 => koji tag epel10.1
epel10.2 => koji tag epel10.2 (here the script is failing)
epel10(rawhide) => koji tag epel10.3

Ok, the quickfix is to make the toddler not crash when it cannot find the list of retired packages. Seems the problem is not the toddler, but the script that identifies retired packages. Did something change recently with epel tags? Is this assumption correct? epel8 => koji tag epel8 epel9 => koji tag epel9 epel10.1 => koji tag epel10.1 epel10.2 => koji tag epel10.2 (here the script is failing) epel10(rawhide) => koji tag epel10.3

lenkaseg commented

2026-03-24 09:48:47 +00:00

Contributor

I don't see any logs of the get_retired_packages here: https://lists.fedoraproject.org/archives/list/releng-cron@lists.fedoraproject.org/

Going to try the script locally.

I don't see any logs of the get_retired_packages here: https://lists.fedoraproject.org/archives/list/releng-cron@lists.fedoraproject.org/ Going to try the script locally.

lenkaseg referenced this issue from apps/toddlers

2026-03-24 09:52:21 +00:00

koji_block_retired: do not fail when lookaside json not found #396

lenkaseg commented

2026-03-24 09:52:45 +00:00

Contributor

In the meantime, here is the quickfix: apps/toddlers#396

In the meantime, here is the quickfix: https://forge.fedoraproject.org/apps/toddlers/pulls/396

lenkaseg commented

2026-03-24 13:22:59 +00:00

Contributor

Ok, so the script is not the problem either.
The problem is missing branch epel10.2, for example: https://src.fedoraproject.org/rpms/gdal/branches?branchname=rawhide
If the packages do not have the epel10.2 branch => they cannot be retired on it => there is no retired_in_epel10.2.json file: https://src.fedoraproject.org/lookaside/

So the fix above should take into consideration situations like this.

Ok, so the script is not the problem either. The problem is missing branch epel10.2, for example: https://src.fedoraproject.org/rpms/gdal/branches?branchname=rawhide If the packages do not have the epel10.2 branch => they cannot be retired on it => there is no retired_in_epel10.2.json file: https://src.fedoraproject.org/lookaside/ So the fix above should take into consideration situations like this.

lenkaseg commented

2026-03-24 15:19:07 +00:00

Contributor

To make it more accurate, there is no missing branch epe10.2, it just does not have any retired packages, so obviously no json file on lookaside.

jnsamyak closed this issue

2026-04-07 09:09:47 +00:00

decathorpe commented

2026-04-07 10:04:26 +00:00

Author

Contributor

Why was this closed? This hasn't been addressed yet.