Test massive side-tag update in bodhi staging

yselkowitz commented

2025-08-08 00:22:46 +00:00

Member

Describe the issue
We are looking at changing how distrobuildsync-eln creates updates for its builds, by using bodhi side-tag updates. The question is if bodhi can handle a mass rebuild batch (which could be 3k+ and possibly much larger in the future) or if we need to bypass bodhi in such a case. (Right now it just tags builds into eln-updates candidate, so one update per build, which is suboptimal in any case and overwhelms the system during an ELN mass rebuild.)

To that end, we want to test bodhi's capacity by creating a side-tag, populating it with a few thousand builds (even better if somehow we don't actually have to create the builds), submitting it, and monitoring bodhi and koji performance, all in staging.

More background: https://github.com/fedora-eln/eln/issues/200 (in comments)

When do you need this? (YYYY/MM/DD)
At your convenience.
When is this no longer needed or useful? (YYYY/MM/DD)
N/A
If we cannot complete your request, what is the impact?
We either don't proceed with the desired improvements, or we do so without proper testing.

* Describe the issue We are looking at changing how distrobuildsync-eln creates updates for its builds, by using bodhi side-tag updates. The question is if bodhi can handle a mass rebuild batch (which could be 3k+ and possibly much larger in the future) or if we need to bypass bodhi in such a case. (Right now it just tags builds into eln-updates candidate, so one update per build, which is suboptimal in any case and overwhelms the system during an ELN mass rebuild.) To that end, we want to test bodhi's capacity by creating a side-tag, populating it with a few thousand builds (even better if somehow we don't actually have to create the builds), submitting it, and monitoring bodhi and koji performance, all in staging. More background: https://github.com/fedora-eln/eln/issues/200 (in comments) * When do you need this? (YYYY/MM/DD) At your convenience. * When is this no longer needed or useful? (YYYY/MM/DD) N/A * If we cannot complete your request, what is the impact? We either don't proceed with the desired improvements, or we do so without proper testing.

jnsamyak commented

2025-08-11 15:14:07 +00:00

Owner

Metadata Update from @jnsamyak:

Issue tagged with: investigation, medium-gain, medium-trouble, review

**Metadata Update from @jnsamyak**: - Issue tagged with: investigation, medium-gain, medium-trouble, review

kevin commented

2025-08-25 15:12:17 +00:00

Owner

@mattia can you see any easy way to test this in staging?

mattia commented

2025-08-26 16:05:11 +00:00

I don't know if the Koji prod db to the stg db can be synchronized, so that we could already have a lot of "new" builds that bodhi stg is not aware of...
Those builds could then be tagged in a side-tag and a new side-tag update submitted in bodhi stg. At least the initial tagging of all builds in pending-signing should work, after that I don't know if bodhi stg correctly lists to the message queue to process later steps.

I don't know if the Koji prod db to the stg db can be synchronized, so that we could already have a lot of "new" builds that bodhi stg is not aware of... Those builds could then be tagged in a side-tag and a new side-tag update submitted in bodhi stg. At least the initial tagging of all builds in pending-signing should work, after that I don't know if bodhi stg correctly lists to the message queue to process later steps.

kevin commented

2025-08-26 22:08:43 +00:00

Owner

We could do a sync of course, but not sure there would be sufficent builds that it doesn't know about? well, I guess it depends on when the last time we synced the bodhi database?

mattia commented

2025-08-27 06:38:13 +00:00

Bodhi stg db was synced last time two months ago, there F43 is still Rawhide. There are currently ~8.900 updates in prod which shouldn't be known by bodhi stg. The count surely has multiple updates for the same package, but I think we can reach a couple of thousand of unique packages.

kevin commented

2025-08-27 20:46:26 +00:00

Owner

ok. I can start a sync. It usually takes a day or so.

kevin commented

2025-08-29 17:28:58 +00:00

Owner

The sync is all done.

Can you look at making the update? or who would like to work on that? :)

The sync is all done. Can you look at making the update? or who would like to work on that? :)

yselkowitz commented

2025-08-29 19:03:42 +00:00

Author

Member

Looks like I have access to koji and bodhi stg, but where can I find the builds that were just synced?

mattia commented

2025-08-30 06:47:17 +00:00

I think I don't quite understand the question, anyway we'll need to:

create a side-tag in koji.stg
for each package, get a list of the latest build for F43 on koji.stg (F43 because it is still "Rawhide" in bodhi.stg)
if the build is not present in bodhi.stg, we'll need to make sure to remove all tags and add it to the side-tag
create the side-tag update in bodhi-stg

Does anyone see any simpler way?

I think I don't quite understand the question, anyway we'll need to: - create a side-tag in koji.stg - for each package, get a list of the latest build for F43 on koji.stg (F43 because it is still "Rawhide" in bodhi.stg) - if the build is not present in bodhi.stg, we'll need to make sure to remove all tags and add it to the side-tag - create the side-tag update in bodhi-stg Does anyone see any simpler way?

yselkowitz commented

2025-09-08 22:52:13 +00:00

Author

Member

f43-build-side-117736 created on koji.stg

mattia commented

2025-09-09 16:32:54 +00:00

I've not been really much around lately... I will try to write a script to extract/compare/tag builds in the side-tag as soon as I get some time (a week? perhaps two...)

mattia commented

2025-09-20 07:43:19 +00:00

Ok, I've tagged ~23k builds in f43-build-side-117736 (koji is still running tag tasks ATM).
For reference, here it's what I ran (in staging bodhi-web pod terminal to overcome koji authentication and authorization):

import rpm
koji_client = request.koji
# get all F43 builds
koji_client.multicall = True
koji_client.listTagged('f43-updates-candidate', latest=True)
koji_client.listTagged('f43-updates-testing', latest=True)
koji_client.listTagged('f43-signing-pending', latest=True)
koji_client.listTagged('f43-updates-testing-pending', latest=True)
koji_client.listTagged('f43-updates-pending', latest=True)
koji_client.listTagged('f43', latest=True)
koji_client.listTagged('f43-updates', latest=True)
response = koji_client.multiCall()

# get only the latest builds and that aren't already in bodhi stg
result = dict()
pkgset = set()
for taglist in response:
    if isinstance(taglist, dict):
        continue
    else:
        for build in taglist[0]:
            item = {'nvr': build['nvr'],'id': build['id'],'package_name': build['package_\
name'],'owner_name': build['owner_name'],}
            if m.Build.get(item['nvr']) is not None:
                continue
            if not item['package_name'] in pkgset:
                pkgset.add(item['package_name'])
                result[item['package_name']] = item['nvr']
            else:
                if rpm.labelCompare(item['nvr'], result[item['package_name']]):
                    result[item['package_name']] = item['nvr']

# filter out builds from older releases (maybe pkg was retired?)
nvrs = []
for pkg, nvr in result.items():
    if nvr.endswith('fc43'):
        nvrs.append(nvr)

# tag in side-tag
koji_client.multicall = True
for build in nvrs:
    koji_client.tagBuild('f43-build-side-117736', build, force=True)
koji_client.multiCall()

Ok, I've tagged ~23k builds in `f43-build-side-117736` (koji is still running tag tasks ATM). For reference, here it's what I ran (in staging bodhi-web pod terminal to overcome koji authentication and authorization): ``` import rpm koji_client = request.koji # get all F43 builds koji_client.multicall = True koji_client.listTagged('f43-updates-candidate', latest=True) koji_client.listTagged('f43-updates-testing', latest=True) koji_client.listTagged('f43-signing-pending', latest=True) koji_client.listTagged('f43-updates-testing-pending', latest=True) koji_client.listTagged('f43-updates-pending', latest=True) koji_client.listTagged('f43', latest=True) koji_client.listTagged('f43-updates', latest=True) response = koji_client.multiCall() # get only the latest builds and that aren't already in bodhi stg result = dict() pkgset = set() for taglist in response: if isinstance(taglist, dict): continue else: for build in taglist[0]: item = {'nvr': build['nvr'],'id': build['id'],'package_name': build['package_\ name'],'owner_name': build['owner_name'],} if m.Build.get(item['nvr']) is not None: continue if not item['package_name'] in pkgset: pkgset.add(item['package_name']) result[item['package_name']] = item['nvr'] else: if rpm.labelCompare(item['nvr'], result[item['package_name']]): result[item['package_name']] = item['nvr'] # filter out builds from older releases (maybe pkg was retired?) nvrs = [] for pkg, nvr in result.items(): if nvr.endswith('fc43'): nvrs.append(nvr) # tag in side-tag koji_client.multicall = True for build in nvrs: koji_client.tagBuild('f43-build-side-117736', build, force=True) koji_client.multiCall() ```

mattia commented

2025-09-21 06:12:46 +00:00

... I think I might have clogged koji stg... it still running tagging tasks (ca ~1.8k left)!
This might happen in prod as well when bodhi will start to move the builds of such large updates. Perhaps there are limited resources in stg in comparison to prod, so the impact on prod will be lesser.

... I think I might have clogged koji stg... it still running tagging tasks (ca ~1.8k left)! This might happen in prod as well when bodhi will start to move the builds of such large updates. Perhaps there are limited resources in stg in comparison to prod, so the impact on prod will be lesser.

kevin commented

2025-09-22 18:10:36 +00:00

Owner

Yeah, stg koji has vastly smaller amount of builders... and in fact I need to rebuild some since the dc move, so it even has less than normal. :)

It seems like it's done?

Yeah, stg koji has vastly smaller amount of builders... and in fact I need to rebuild some since the dc move, so it even has less than normal. :) It seems like it's done?

yselkowitz commented

2025-09-22 21:31:57 +00:00

Author

Member

https://bodhi.stg.fedoraproject.org/updates/FEDORA-2025-3d83cfd7e4

kevin commented

2025-11-07 00:20:33 +00:00

Owner

So, BTW, it wasn't done I don't think... the staging builders got in a weird state and were not checking in.

I fixed that (and moved them all to f43) eariler this week.

It had a backlog of about 13,000 tag tasks.

it's got about 2300 left. Should finish later tonight.

So, BTW, it wasn't done I don't think... the staging builders got in a weird state and were not checking in. I fixed that (and moved them all to f43) eariler this week. It had a backlog of about 13,000 tag tasks. it's got about 2300 left. Should finish later tonight.

yselkowitz commented

2025-11-07 00:25:52 +00:00

Author

Member

The question I suppose is if they got in a weird state because of this update?

yselkowitz commented

2025-11-28 19:31:01 +00:00

Author

Member

Reviving this on the new tracker. It takes a long while to open the page for the update, but eventually it does load. The update is still in pending.

kevin commented

2025-11-30 20:44:12 +00:00

Owner

Likely in stg it will always sit in pending because we don't have signing there, so bodhi is likely waiting for them to be signed.

I think probibly we should just try it in prod...

Likely in stg it will always sit in pending because we don't have signing there, so bodhi is likely waiting for them to be signed. I think probibly we should just try it in prod...

zlopez commented

2025-12-11 09:31:49 +00:00

Member

It seems that this test caused issue https://pagure.io/fedora-infrastructure/issue/12932, probably some optimization could be done.

yselkowitz referenced this issue

2025-12-12 01:57:03 +00:00

Create and configure a new user for ELNBuildSync #13132

kevin commented

2026-02-19 18:17:07 +00:00

Owner

So, it seems it's looping oddly...

for example:

Tue Feb 17 01:37:26 2026 perl-ExtUtils-AutoInstall-0.64-28.fc43 tagged into f43-build-side-117736-signing-pending by bodhi
Tue Feb 17 18:50:11 2026 perl-ExtUtils-AutoInstall-0.64-28.fc43 untagged from f43-build-side-117736-signing-pending by bodhi
Wed Feb 18 06:19:15 2026 perl-ExtUtils-AutoInstall-0.64-28.fc43 tagged into f43-build-side-117736-signing-pending by bodhi
Wed Feb 18 18:50:24 2026 perl-ExtUtils-AutoInstall-0.64-28.fc43 untagged from f43-build-side-117736-signing-pending by bodhi

I see the update itself was deleted? (at least it 404s for me now).

So, it seems it's looping oddly... for example: ``` Tue Feb 17 01:37:26 2026 perl-ExtUtils-AutoInstall-0.64-28.fc43 tagged into f43-build-side-117736-signing-pending by bodhi Tue Feb 17 18:50:11 2026 perl-ExtUtils-AutoInstall-0.64-28.fc43 untagged from f43-build-side-117736-signing-pending by bodhi Wed Feb 18 06:19:15 2026 perl-ExtUtils-AutoInstall-0.64-28.fc43 tagged into f43-build-side-117736-signing-pending by bodhi Wed Feb 18 18:50:24 2026 perl-ExtUtils-AutoInstall-0.64-28.fc43 untagged from f43-build-side-117736-signing-pending by bodhi ``` I see the update itself was deleted? (at least it 404s for me now).

kevin commented

2026-02-19 18:18:40 +00:00

Owner

I deleted the side tag.

I am canceling all the 376167 tagging tasks (but it's slow going)

I deleted the side tag. I am canceling all the 376167 tagging tasks (but it's slow going)

kevin referenced this issue

2026-02-19 18:38:36 +00:00

Staging Koji has hundreds of thousands of queued tagBuild tasks #13234

mattia commented

2026-02-19 21:45:38 +00:00

@kevin wrote in #12869 (comment):

I see the update itself was deleted? (at least it 404s for me now).

Yes, I did a synch of bodhi's prod db into stg as It was needed to test the multiple destination registries for flatpak. I thought that the massive update was not needed anymore.

@kevin wrote in https://forge.fedoraproject.org/releng/tickets/issues/12869#issuecomment-537214: > I see the update itself was deleted? (at least it 404s for me now). Yes, I did a synch of bodhi's prod db into stg as It was needed to test the multiple destination registries for flatpak. I thought that the massive update was not needed anymore.

kevin commented

2026-02-19 22:15:31 +00:00

Owner

yeah, we don't need it anymore, but do you have any idea why bodhi was trying to still tag/untag from the signing-pending sidetag tag?

kevin commented

2026-02-19 23:36:17 +00:00

Owner

Only down to 361228 now... ;(

Might have to resort to just doing a prod->staging koji db sync to clear them out. ;)

Only down to 361228 now... ;( Might have to resort to just doing a prod->staging koji db sync to clear them out. ;)

mattia commented

2026-02-20 17:37:10 +00:00

@kevin wrote in #12869 (comment):

yeah, we don't need it anymore, but do you have any idea why bodhi was trying to still tag/untag from the signing-pending sidetag tag?

My only thought is that the signing-pending sidetag is tagged by celery tasks. Maybe those tasks were still living somewhere even after they were supposed to be completed?

@kevin wrote in https://forge.fedoraproject.org/releng/tickets/issues/12869#issuecomment-537249: > yeah, we don't need it anymore, but do you have any idea why bodhi was trying to still tag/untag from the signing-pending sidetag tag? My only thought is that the signing-pending sidetag is tagged by celery tasks. Maybe those tasks were still living somewhere even after they were supposed to be completed?

kevin commented

2026-02-20 18:56:28 +00:00

Owner

Hum... does it use a rabbitmq queue for those? If it was, they are all empty now...

down to 307784... gonna take a while. ;(

Hum... does it use a rabbitmq queue for those? If it was, they are all empty now... down to 307784... gonna take a while. ;(

Rows
Columns

Test massive side-tag update in bodhi staging #12869