infra/tickets

Fork 1

automatic bodhi update creation for rawhide and ELN not working since July 13 ~23:00 UTC #13461

New issue

Closed

opened 2026-07-14 13:33:31 +00:00 by decathorpe · 12 comments

decathorpe commented

2026-07-14 13:33:31 +00:00

Description of request

It looks like the automatic update creation for rawhide and ELN builds isn't working. The last build that got tagged into f45 by bodhi was:

Mon Jul 13 23:10:09 2026 xev-1.2.7-4.fc45 tagged into f45 by bodhi [still active]

(from koji list-history --tag f45 --after="2026-07-13 23:00" | grep bodhi)

There is a growing backlog of builds tagged f45-updates-candidate and eln-updates-candidate that aren't getting picked up.

### Description of request It looks like the automatic update creation for rawhide and ELN builds isn't working. The last build that got tagged into f45 by bodhi was: > Mon Jul 13 23:10:09 2026 xev-1.2.7-4.fc45 tagged into f45 by bodhi [still active] (from `koji list-history --tag f45 --after="2026-07-13 23:00" | grep bodhi`) There is a growing backlog of builds tagged f45-updates-candidate and eln-updates-candidate that aren't getting picked up.

kevin commented

2026-07-14 14:06:16 +00:00

Owner

I think i see the problem and have fixed it, but will see/watch.

kevin added the

labels

2026-07-14 15:36:57 +00:00

kevin added this to the Sprint ( 2026-07-13 to 2026-07-26) project

2026-07-14 15:37:01 +00:00

kevin self-assigned this

2026-07-14 15:37:03 +00:00

kevin commented

2026-07-14 15:39:38 +00:00

Owner

So, the issue was that the bodhi consumer pod had stopped... consuming.

Likely due to the reinstall of the vmhost yesterday that had rabbitmq01 on it. We need to disentangle things so rabbitmq01 being down doesn't cause all the consumers to have problems. ;(

In any case a:

oc rollout restart deploy bodhi-consumer -n bodhi

restarted the pod and it consumed all the back messages.
I guess CI is churning through the backlog now.

So, the issue was that the bodhi consumer pod had stopped... consuming. Likely due to the reinstall of the vmhost yesterday that had rabbitmq01 on it. We need to disentangle things so rabbitmq01 being down doesn't cause all the consumers to have problems. ;( In any case a: oc rollout restart deploy bodhi-consumer -n bodhi restarted the pod and it consumed all the back messages. I guess CI is churning through the backlog now.

kevin commented

2026-07-14 16:24:52 +00:00

Owner

I think it's mostly processed everything, there's a few more still in ci...

Please re-open if you see any further issues.

I think it's mostly processed everything, there's a few more still in ci... Please re-open if you see any further issues.

kevin closed this issue

2026-07-14 16:24:52 +00:00

petersen commented

2026-07-15 03:25:52 +00:00

Seems stuck again:

⬢F45~$ koji-tool tagged f45-updates-candidate | wc -l
43
⬢F45~$ koji-tool tagged eln-updates-candidate | wc -l
22

Seems stuck again: ``` ⬢F45~$ koji-tool tagged f45-updates-candidate | wc -l 43 ⬢F45~$ koji-tool tagged eln-updates-candidate | wc -l 22 ```

jnsamyak commented

2026-07-15 04:54:58 +00:00

Member

Looking into it

jnsamyak reopened this issue

2026-07-15 04:54:58 +00:00

jnsamyak commented

2026-07-15 05:13:27 +00:00

Member

it is not the bodhi pod, checking bodhi logs

jnsamyak commented

2026-07-15 05:14:37 +00:00

Member

caught it, should be fixing now

(edit: it didnt work)

caught it, should be fixing now (edit: it didnt work)

petersen commented

2026-07-15 12:49:51 +00:00

Actually sorry - I only looked at the number of builds earlier: it seems all those 43 builds were from late May...

decathorpe commented

2026-07-15 14:47:30 +00:00

Author

It looks like this is fixed now.

jnsamyak commented

2026-07-15 14:58:06 +00:00

Member

Correct!

jnsamyak closed this issue

2026-07-15 14:58:06 +00:00

petersen commented

2026-07-15 16:34:34 +00:00

But some strange untags seemed to happen to some of those packages in early June by oscar.

I might still be good to cleanup the rest of f45-updates-testing-pending perhaps.

But some strange untags seemed to happen to some of those packages in early June by `oscar`. I might still be good to cleanup the rest of `f45-updates-testing-pending` perhaps.

kevin commented

2026-07-15 16:53:00 +00:00

Owner

oscar is the user that does koji-gc: https://fedoraproject.org/wiki/Koji/GarbageCollection

so its normal that old builds that aren't in releases are untagged by it.

oscar is the user that does koji-gc: https://fedoraproject.org/wiki/Koji/GarbageCollection so its normal that old builds that aren't in releases are untagged by it.