[WIP] Update Mass Branching SOP #489

Draft
patrikp wants to merge 1 commit from patrikp/docs:sop_mass_branching into main
Member
No description provided.
Signed-off-by: Patrik Polakovič <patrik@alphamail.org>
Signed-off-by: Patrik Polakovič <patrik@alphamail.org>
patrikp force-pushed sop_mass_branching from 9955edb238 to 707027bf24 2026-02-06 09:54:50 +00:00 Compare
Owner

Is this ready for review? Or do you have more changes to push first?

Is this ready for review? Or do you have more changes to push first?
Owner

For the openh264 section at the bottom. It's a mish mash currently. Remove all that and replace with:

  • A week before/day before step that makes a set of empty repos in /srv/web/codecs.fedoraproject.org/openh264/45
    ( where 45 is the new rawhide version).
  • In the section where we adjust mirrormanager for various rawhide things, add to that:
    UPDATE repository_redirect SET to_repo = regexp_replace(to_repo, '-44$', '-45') WHERE from_repo ILIKE '%openh264%';
  • Add a note to get a new rawhide (45) build and send that to cisco and update the empty repo with that one once we have it.
For the openh264 section at the bottom. It's a mish mash currently. Remove all that and replace with: * A week before/day before step that makes a set of empty repos in /srv/web/codecs.fedoraproject.org/openh264/45 ( where 45 is the new rawhide version). * In the section where we adjust mirrormanager for various rawhide things, add to that: UPDATE repository_redirect SET to_repo = regexp_replace(to_repo, '-44$', '-45') WHERE from_repo ILIKE '%openh264%'; * Add a note to get a new rawhide (45) build and send that to cisco and update the empty repo with that one once we have it.
Author
Member

Is this ready for review? Or do you have more changes to push first?

It's not ready for review yet, I would like to format it better, the information is there but some of it is pretty chaotic. I'll be sure to tag you once it's ready for review/feedback.

> Is this ready for review? Or do you have more changes to push first? It's not ready for review yet, I would like to format it better, the information is there but some of it is pretty chaotic. I'll be sure to tag you once it's ready for review/feedback.
Member

Some notes from my perspective:

The fedfind release metadata should be updated just before we fire the new composes. I'm pretty sure now that's the least-bad time to do it.

We should ensure Bodhi has a critpath file for the new branched release BEFORE any new updates can be created for the newly-branched release. You can see what critpath files it currently has by rsh'ing into any running Bodhi container and looking in /etc/bodhi/critpath. The files for current Rawhide are named rawhide*, they are not named by release number (I should try and remember why, actually). The files for all other releases are named by release number. We need the files for the new Branched number to exist, or else Bodhi won't consider any updates to be critpath so nothing will be gated.

There are a few details here. We can wait till we can do a real critpath generation - which means the new Branched compose has to run and be synced out first, then you can run an oc command to manually trigger the generation. Or we can, in some way, just copy the previous Rawhide data to the newly-branched release number at some point - we could do this as a one-off special action at branch time somehow, or we could adjust the script to just always copy the 'rawhide' files to the current Rawhide release number as well, that should be safe.

Also, do we keep Koji blocked until this is done, or do we unblock Koji but try to block Bodhi somehow until it's done?

Some notes from my perspective: The [fedfind release metadata](https://fedorapeople.org/groups/qa/metadata/release.json) should be updated *just before* we fire the new composes. I'm pretty sure now that's the least-bad time to do it. We should ensure Bodhi has a critpath file for the new branched release *BEFORE* any new updates can be created for the newly-branched release. You can see what critpath files it currently has by rsh'ing into any running Bodhi container and looking in `/etc/bodhi/critpath`. The files for current Rawhide are named `rawhide*`, they are not named by release number (I should try and remember why, actually). The files for all other releases are named by release number. We need the files for the new Branched number to exist, or else Bodhi won't consider any updates to be critpath so nothing will be gated. There are a few details here. We can wait till we can do a *real* critpath generation - which means the new Branched compose has to run and be synced out first, then you can run an `oc` command to manually trigger the generation. *Or* we can, in some way, just copy the previous Rawhide data to the newly-branched release number at some point - we could do this as a one-off special action at branch time somehow, or we could adjust the script to just always copy the 'rawhide' files to the current Rawhide release number as well, that *should* be safe. Also, do we keep Koji blocked until this is done, or do we unblock Koji but try to block Bodhi somehow until it's done?
Member

OK, based on a suggestion by @kevin , releng/tooling#13022 should solve the critpath issue - with that, we'll just keep using the most recent Rawhide critpath data for the new Branched release until such time as we manage to do a successful new critpath generation run after the branching.

OK, based on a suggestion by @kevin , https://forge.fedoraproject.org/releng/tooling/pulls/13022 should solve the critpath issue - with that, we'll just keep using the most recent Rawhide critpath data for the new Branched release until such time as we manage to do a successful new critpath generation run after the branching.
Member

Oh, also, we should improve the mirrormanager instructions. Linking to a random pagure issue with old release numbers isn't great, and it's missing info on how to get into the database in the first place. We didn't quite do it right this time around so clearly the instructions/procedure aren't good enough.

Ideally it would be ansiblized somehow, if not, there should be better instructions in-line in the doc, with the release numbers correctly templated and details on how to get into the correct db.

Oh, also, we should improve the mirrormanager instructions. Linking to a random pagure issue with old release numbers isn't great, and it's missing info on how to get into the database in the first place. We didn't quite do it right this time around so clearly the instructions/procedure aren't good enough. Ideally it would be ansiblized somehow, if not, there should be better instructions in-line in the doc, with the release numbers correctly templated and details on how to get into the correct db.
Member

Couple of repo issues that showed up: fedora-openh264 repo for Rawhide was 404 for a while (this seems fixed now, I guess @kevin fixed it?) and fedora-updates-archive repo for F44 is 404 currently. These both should be sorted at branch time as they break openQA ostree tests (rpm-ostree seems to treat all repos as critical and fail if any is unreachable).

Couple of repo issues that showed up: fedora-openh264 repo for Rawhide was 404 for a while (this seems fixed now, I guess @kevin fixed it?) and [fedora-updates-archive repo for F44 is 404 currently](https://forge.fedoraproject.org/infra/tickets/issues/13125). These both should be sorted at branch time as they break openQA ostree tests (rpm-ostree seems to treat all repos as critical and fail if any is unreachable).
Owner

Yeah, I did fix the openh264 rawhide repo (well, I made an empty one).

For updates-archive, thats @dustymabe

Yeah, I did fix the openh264 rawhide repo (well, I made an empty one). For updates-archive, thats @dustymabe
Member

From @dustymabe for updates-archive:

the steps for adding a new RELEASE stub is in the README under Rough notes for initializing the archive repo from scratch. The keys to the bucket were given to us by releng.

From @dustymabe for updates-archive: the steps for adding a new RELEASE stub is in the [README](https://pagure.io/releng/archive-repo-manager/blob/main/f/README.md) under `Rough notes for initializing the archive repo from scratch`. The keys to the bucket were given to us by releng.
@ -464,3 +485,3 @@
[source,bash,subs="attributes"]
----
$ koji list-builds --state=0 --type=rpm | grep fc{branched} | awk '{print $1}'
$ koji list-builds --state=0 --type=rpm | grep fc{rawhide} | awk '{print $1}'
Member

This really depends, as I mentioned this before, we always update the attributes before the branching happens, and we don't need to go through the hassle of changing variables; instead, this adds in T-1 day about updating the attributes; it should be a branched tag only that we need to cancel.

This really depends, as I mentioned this before, we always update the attributes before the branching happens, and we don't need to go through the hassle of changing variables; instead, this adds in T-1 day about updating the attributes; it should be a branched tag only that we need to cancel.
@ -590,1 +621,3 @@
$ bodhi releases edit --name F{branched}F --eol YYYY-MM-DD
$ bodhi releases edit --name F{rawhide} --eol YYYY-MM-DD
$ bodhi releases edit --name F{rawhide}F --eol YYYY-MM-DD
$ bodhi releases edit --name F{rawhide} --branch f{rawhide}
Member

I think again, with the above confusion,n we need to set the date of branched release, not the rawhide. +1 on removing the unwated releases

I think again, with the above confusion,n we need to set the date of branched release, not the rawhide. +1 on removing the unwated releases
@ -706,0 +698,4 @@
- `pungi-fedora`
- `fedora-kiwi-descriptions`
- `fedora-comps`
- `workstation-ostree-config`
Member

I recommend merging this as an inital step for The T day of branching

I recommend merging this as an inital step for The T day of branching
patrikp force-pushed sop_mass_branching from 707027bf24 to fee25ac50d 2026-02-14 17:58:23 +00:00 Compare
Member

Another thing I found: fedora-appstream-metadata should be updated at branch time, see https://bugzilla.redhat.com/show_bug.cgi?id=2440511 . Tagging @siosm

Another thing I found: fedora-appstream-metadata should be updated at branch time, see https://bugzilla.redhat.com/show_bug.cgi?id=2440511 . Tagging @siosm
Member

Ooh, another thing: we should add a run of the openqa.yml playbook with -t testcase_stats to the steps once the FedoraBranched variable is set true.

Ooh, another thing: we should add a run of the `openqa.yml` playbook with `-t testcase_stats` to the steps once the `FedoraBranched` variable is set true.
patrikp force-pushed sop_mass_branching from fee25ac50d to 6e6e163969 2026-02-22 15:31:58 +00:00 Compare
Contributor

Hey @patrikp, just a heads up that I've opened PR #496 to remove dead PDC references across the docs (Issue #344). I specifically avoided touching modules/release_guide/pages/sop_mass_branching.adoc to ensure I didn't cause any merge conflicts with your rewrite here. Good luck with the chaos ; )

Hey @patrikp, just a heads up that I've opened PR #496 to remove dead PDC references across the docs (Issue #344). I specifically avoided touching `modules/release_guide/pages/sop_mass_branching.adoc` to ensure I didn't cause any merge conflicts with your rewrite here. Good luck with the chaos ; )
This pull request has changes conflicting with the target branch.
  • modules/release_guide/pages/sop_mass_branching.adoc
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u sop_mass_branching:patrikp-sop_mass_branching
git switch patrikp-sop_mass_branching

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff patrikp-sop_mass_branching
git switch patrikp-sop_mass_branching
git rebase main
git switch main
git merge --ff-only patrikp-sop_mass_branching
git switch patrikp-sop_mass_branching
git rebase main
git switch main
git merge --no-ff patrikp-sop_mass_branching
git switch main
git merge --squash patrikp-sop_mass_branching
git switch main
git merge --ff-only patrikp-sop_mass_branching
git switch main
git merge patrikp-sop_mass_branching
git push origin main
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
5 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
infra/docs!489
No description provided.