RHEL10 adoption #12712

Open
opened 2025-08-07 21:03:56 +00:00 by kevin · 12 comments
Owner

Now that we have moved almost all of our rhel instances to 9, it's time to move them to 10. ;)

There's currently 107 rhel9 instances.

The virthosts should be pretty easy to move first. I think we even have everything available needed for them.

Thats 30 of the total.

I'll start in on those soon.

Now that we have moved almost all of our rhel instances to 9, it's time to move them to 10. ;) There's currently 107 rhel9 instances. The virthosts should be pretty easy to move first. I think we even have everything available needed for them. Thats 30 of the total. I'll start in on those soon.
Member

This is obv. blocked on freeze atm.

But had a quick look at our EL hosts, and (everything followed by just staging)...

OS                 Hosts
  RedHat        :    115     
    RedHat/8.10 :     13     
    RedHat/9.3  :      1     
    RedHat/9.6  :    101

RedHat        :     28   
    RedHat/8.10 :      2
    RedHat/9.6  :     26

9 of those 8.10 hosts are '*.rdu-cc.*' (which I think are moving "soon", and we could test el10 at that point?)

The last 4 are:

  • pagure-stg01
  • pagure02
  • pkgs01
  • pkgs01.stg

...so we could experiment/test on the non-prod ones.

This is obv. blocked on freeze atm. But had a quick look at our EL hosts, and (everything followed by just staging)... ``` OS Hosts RedHat : 115 RedHat/8.10 : 13 RedHat/9.3 : 1 RedHat/9.6 : 101 RedHat : 28 RedHat/8.10 : 2 RedHat/9.6 : 26 ``` 9 of those 8.10 hosts are '\*.rdu-cc.\*' (which I think are moving "soon", and we could test el10 at that point?) The last 4 are: * pagure-stg01 * pagure02 * pkgs01 * pkgs01.stg ...so we could experiment/test on the non-prod ones.
Author
Owner

The staging part of it isn't blocked. ;)

I'd like to setup a meeting with you and @gwmngilfen and go through the upgrade process. (and anyone else who wants to watch!)
Probibly my plan is to do vmhosts/bvmhosts first/soon and then slowly expand from there.
This is on my list after signing stuff, but if you all could start working on them that would be just great for me.

pagure and pkgs are blocked / stuck on el8 because pagure (the package is not available on 9, much less 10), so they are probibly just out of this entirely and will go away still el8.

The staging part of it isn't blocked. ;) I'd like to setup a meeting with you and @gwmngilfen and go through the upgrade process. (and anyone else who wants to watch!) Probibly my plan is to do vmhosts/bvmhosts first/soon and then slowly expand from there. This is on my list after signing stuff, but if you all could start working on them that would be just great for me. pagure and pkgs are blocked / stuck on el8 because pagure (the package is not available on 9, much less 10), so they are probibly just out of this entirely and will go away still el8.
Author
Owner

We did have that meeting a while ago. :)

Since then vmhost-x86-04.stg and vmhost-x86-05 in prod have been moved. ;)

There's 105 rhel9 instances now.
Will start with the stg vmhosts/bvmhosts and move out from there.

We did have that meeting a while ago. :) Since then vmhost-x86-04.stg and vmhost-x86-05 in prod have been moved. ;) There's 105 rhel9 instances now. Will start with the stg vmhosts/bvmhosts and move out from there.
Author
Owner

Here's a first rough cut of hosts. Of course we are going into freeze tomorrow, but we can work on the staging ones and work on investigating the ones that probibly need migrations or packages built.

We need to fix #13035 before we do anymore tho at least.

easyish (we might have what we need to do these,
will still require downtime and coordination):

  • backup01.rdu3.fedoraproject.org
  • bastion01.rdu3.fedoraproject.org
  • bastion02.rdu3.fedoraproject.org
  • bvmhost-a64-01.rdu3.fedoraproject.org
  • bvmhost-a64-01.stg.rdu3.fedoraproject.org
  • bvmhost-a64-02.rdu3.fedoraproject.org
  • bvmhost-a64-03.rdu3.fedoraproject.org
  • bvmhost-a64-04.rdu3.fedoraproject.org
  • bvmhost-a64-05.rdu3.fedoraproject.org
  • bvmhost-x86-01.rdu3.fedoraproject.org
  • bvmhost-x86-01.stg.rdu3.fedoraproject.org
  • bvmhost-x86-02.rdu3.fedoraproject.org
  • bvmhost-x86-02.stg.rdu3.fedoraproject.org
  • bvmhost-x86-03.rdu3.fedoraproject.org
  • bvmhost-x86-03.stg.rdu3.fedoraproject.org
  • bvmhost-x86-04.rdu3.fedoraproject.org
  • bvmhost-x86-05.rdu3.fedoraproject.org
  • bvmhost-x86-06.rdu3.fedoraproject.org
  • bvmhost-x86-riscv01.rdu3.fedoraproject.org
  • dl01.rdu3.fedoraproject.org
  • dl02.rdu3.fedoraproject.org
  • dl03.rdu3.fedoraproject.org
  • dl04.rdu3.fedoraproject.org
  • dl05.rdu3.fedoraproject.org
  • download-cc-rdu01.fedoraproject.org
  • download-ib01.fedoraproject.org
  • download-iso01.rdu3.fedoraproject.org
  • memcached02.rdu3.fedoraproject.org
  • memcached02.stg.rdu3.fedoraproject.org
  • noc01.rdu3.fedoraproject.org
  • noc02.fedoraproject.org
  • ns01.rdu3.fedoraproject.org
  • ns02.fedoraproject.org
  • ns03.rdu3.fedoraproject.org
  • ns05.fedoraproject.org
  • os-control01.rdu3.fedoraproject.org
  • os-control01.stg.rdu3.fedoraproject.org
  • qvmhost-x86-01.rdu3.fedoraproject.org
  • secondary01.rdu3.fedoraproject.org
  • sign-bridge01.rdu3.fedoraproject.org
  • smtp-mm-cc-rdu01.fedoraproject.org
  • smtp-mm-ib01.fedoraproject.org
  • smtp-mm-iso01.rdu3.fedoraproject.org
  • smtp-mm-osuosl01.fedoraproject.org
  • sundries01.rdu3.fedoraproject.org
  • sundries01.stg.rdu3.fedoraproject.org
  • sundries02.rdu3.fedoraproject.org
  • sundries02.stg.rdu3.fedoraproject.org
  • tang01.rdu3.fedoraproject.org
  • tang02.rdu3.fedoraproject.org
  • vmhost-x86-01.rdu3.fedoraproject.org
  • vmhost-x86-01.stg.rdu3.fedoraproject.org
  • vmhost-x86-02.rdu3.fedoraproject.org
  • vmhost-x86-02.stg.rdu3.fedoraproject.org
  • vmhost-x86-03.rdu3.fedoraproject.org
  • vmhost-x86-03.stg.rdu3.fedoraproject.org
  • vmhost-x86-04.rdu3.fedoraproject.org
  • vmhost-x86-05.stg.rdu3.fedoraproject.org
  • vmhost-x86-iso02.rdu3.fedoraproject.org
  • vmhost-x86-iso03.rdu3.fedoraproject.org
  • vmhost-x86-iso04.rdu3.fedoraproject.org
  • zabbix01.rdu3.fedoraproject.org
  • zabbix01.stg.rdu3.fedoraproject.org

medium - needs more extenstive testing and/or packages built
still will need scheduling, etc.

  • autosign01.rdu3.fedoraproject.org
  • flatpak-cache01.rdu3.fedoraproject.org
  • ibiblio02.fedoraproject.org
  • ibiblio05.fedoraproject.org
  • internetx02.fedoraproject.org
  • log01.rdu3.fedoraproject.org
  • mailman01.rdu3.fedoraproject.org
  • mailman01.stg.rdu3.fedoraproject.org
  • osuosl02.fedoraproject.org
  • people01.fedoraproject.org

hard (needs downtime, migrations, longer outage, etc):

  • batcave01.rdu3.fedoraproject.org
  • db01.rdu3.fedoraproject.org
  • db01.stg.rdu3.fedoraproject.org
  • db03.rdu3.fedoraproject.org
  • db03.stg.rdu3.fedoraproject.org
  • db-datanommer01.stg.rdu3.fedoraproject.org
  • db-datanommer02.rdu3.fedoraproject.org
  • db-fas01.rdu3.fedoraproject.org
  • db-fas01.stg.rdu3.fedoraproject.org
  • db-koji01.rdu3.fedoraproject.org
  • db-koji01.stg.rdu3.fedoraproject.org
  • db-openqa01.rdu3.fedoraproject.org
  • db-riscv-koji01.rdu3.fedoraproject.org
  • dedicatedsolutions01.fedoraproject.org
  • ipa01.rdu3.fedoraproject.org
  • ipa01.stg.rdu3.fedoraproject.org
  • ipa02.rdu3.fedoraproject.org
  • ipa02.stg.rdu3.fedoraproject.org
  • ipa03.rdu3.fedoraproject.org
  • ipa03.stg.rdu3.fedoraproject.org
  • rabbitmq01.rdu3.fedoraproject.org
  • rabbitmq01.stg.rdu3.fedoraproject.org
  • rabbitmq02.rdu3.fedoraproject.org
  • rabbitmq02.stg.rdu3.fedoraproject.org
  • rabbitmq03.rdu3.fedoraproject.org
  • rabbitmq03.stg.rdu3.fedoraproject.org
  • storinator01.rdu3.fedoraproject.org
  • torrent01.fedoraproject.org
  • value01.rdu3.fedoraproject.org
  • value01.stg.rdu3.fedoraproject.org
Here's a first rough cut of hosts. Of course we are going into freeze tomorrow, but we can work on the staging ones and work on investigating the ones that probibly need migrations or packages built. We need to fix #13035 before we do anymore tho at least. easyish (we might have what we need to do these, will still require downtime and coordination): - [ ] backup01.rdu3.fedoraproject.org - [ ] bastion01.rdu3.fedoraproject.org - [ ] bastion02.rdu3.fedoraproject.org - [ ] bvmhost-a64-01.rdu3.fedoraproject.org - [x] bvmhost-a64-01.stg.rdu3.fedoraproject.org - [ ] bvmhost-a64-02.rdu3.fedoraproject.org - [ ] bvmhost-a64-03.rdu3.fedoraproject.org - [ ] bvmhost-a64-04.rdu3.fedoraproject.org - [ ] bvmhost-a64-05.rdu3.fedoraproject.org - [ ] bvmhost-x86-01.rdu3.fedoraproject.org - [x] bvmhost-x86-01.stg.rdu3.fedoraproject.org - [ ] bvmhost-x86-02.rdu3.fedoraproject.org - [x] bvmhost-x86-02.stg.rdu3.fedoraproject.org - [ ] bvmhost-x86-03.rdu3.fedoraproject.org - [x] bvmhost-x86-03.stg.rdu3.fedoraproject.org - [ ] bvmhost-x86-04.rdu3.fedoraproject.org - [ ] bvmhost-x86-05.rdu3.fedoraproject.org - [ ] bvmhost-x86-06.rdu3.fedoraproject.org - [ ] bvmhost-x86-riscv01.rdu3.fedoraproject.org - [x] dl01.rdu3.fedoraproject.org - [x] dl02.rdu3.fedoraproject.org - [x] dl03.rdu3.fedoraproject.org - [x] dl04.rdu3.fedoraproject.org - [x] dl05.rdu3.fedoraproject.org - [x] download-cc-rdu01.fedoraproject.org - [x] download-ib01.fedoraproject.org - [x] download-iso01.rdu3.fedoraproject.org - [x] memcached02.rdu3.fedoraproject.org - [x] memcached02.stg.rdu3.fedoraproject.org - [ ] noc01.rdu3.fedoraproject.org - [ ] noc02.fedoraproject.org - [x] ns01.rdu3.fedoraproject.org - [x] ns02.fedoraproject.org - [x] ns03.rdu3.fedoraproject.org - [x] ns05.fedoraproject.org - [ ] os-control01.rdu3.fedoraproject.org - [ ] os-control01.stg.rdu3.fedoraproject.org - [ ] qvmhost-x86-01.rdu3.fedoraproject.org - [ ] secondary01.rdu3.fedoraproject.org - [ ] sign-bridge01.rdu3.fedoraproject.org - [x] smtp-mm-cc-rdu01.fedoraproject.org - [x] smtp-mm-ib01.fedoraproject.org - [x] smtp-mm-iso01.rdu3.fedoraproject.org - [x] smtp-mm-osuosl01.fedoraproject.org - [ ] sundries01.rdu3.fedoraproject.org - [ ] sundries01.stg.rdu3.fedoraproject.org - [ ] sundries02.rdu3.fedoraproject.org - [ ] sundries02.stg.rdu3.fedoraproject.org - [x] tang01.rdu3.fedoraproject.org - [x] tang02.rdu3.fedoraproject.org - [ ] vmhost-x86-01.rdu3.fedoraproject.org - [x] vmhost-x86-01.stg.rdu3.fedoraproject.org - [ ] vmhost-x86-02.rdu3.fedoraproject.org - [x] vmhost-x86-02.stg.rdu3.fedoraproject.org - [ ] vmhost-x86-03.rdu3.fedoraproject.org - [x] vmhost-x86-03.stg.rdu3.fedoraproject.org - [ ] vmhost-x86-04.rdu3.fedoraproject.org - [x] vmhost-x86-05.stg.rdu3.fedoraproject.org - [ ] vmhost-x86-iso02.rdu3.fedoraproject.org - [ ] vmhost-x86-iso03.rdu3.fedoraproject.org - [ ] vmhost-x86-iso04.rdu3.fedoraproject.org - [ ] zabbix01.rdu3.fedoraproject.org - [ ] zabbix01.stg.rdu3.fedoraproject.org medium - needs more extenstive testing and/or packages built still will need scheduling, etc. - [ ] autosign01.rdu3.fedoraproject.org - [ ] flatpak-cache01.rdu3.fedoraproject.org - [ ] ibiblio02.fedoraproject.org - [x] ibiblio05.fedoraproject.org - [x] internetx02.fedoraproject.org - [ ] log01.rdu3.fedoraproject.org - [ ] mailman01.rdu3.fedoraproject.org - [ ] mailman01.stg.rdu3.fedoraproject.org - [x] osuosl02.fedoraproject.org - [ ] people01.fedoraproject.org hard (needs downtime, migrations, longer outage, etc): - [ ] batcave01.rdu3.fedoraproject.org - [ ] db01.rdu3.fedoraproject.org - [ ] db01.stg.rdu3.fedoraproject.org - [ ] db03.rdu3.fedoraproject.org - [ ] db03.stg.rdu3.fedoraproject.org - [ ] db-datanommer01.stg.rdu3.fedoraproject.org - [ ] db-datanommer02.rdu3.fedoraproject.org - [ ] db-fas01.rdu3.fedoraproject.org - [ ] db-fas01.stg.rdu3.fedoraproject.org - [ ] db-koji01.rdu3.fedoraproject.org - [ ] db-koji01.stg.rdu3.fedoraproject.org - [ ] db-openqa01.rdu3.fedoraproject.org - [ ] db-riscv-koji01.rdu3.fedoraproject.org - [x] dedicatedsolutions01.fedoraproject.org - [ ] ipa01.rdu3.fedoraproject.org - [ ] ipa01.stg.rdu3.fedoraproject.org - [ ] ipa02.rdu3.fedoraproject.org - [ ] ipa02.stg.rdu3.fedoraproject.org - [ ] ipa03.rdu3.fedoraproject.org - [ ] ipa03.stg.rdu3.fedoraproject.org - [ ] rabbitmq01.rdu3.fedoraproject.org - [ ] rabbitmq01.stg.rdu3.fedoraproject.org - [ ] rabbitmq02.rdu3.fedoraproject.org - [ ] rabbitmq02.stg.rdu3.fedoraproject.org - [ ] rabbitmq03.rdu3.fedoraproject.org - [ ] rabbitmq03.stg.rdu3.fedoraproject.org - [ ] storinator01.rdu3.fedoraproject.org - [ ] torrent01.fedoraproject.org - [ ] value01.rdu3.fedoraproject.org - [ ] value01.stg.rdu3.fedoraproject.org
kevin self-assigned this 2026-03-04 23:56:19 +00:00
Author
Owner

I haven't been able to work on this too much this sprint due to other work.

We did do some prep work however (fixing the lmdb change).

I do plan to work on it in this next sprint.

I haven't been able to work on this too much this sprint due to other work. We did do some prep work however (fixing the lmdb change). I do plan to work on it in this next sprint.
Member

Managed to do vmhost-x86-03.stg.rdu3.fedoraproject.org today. Was easier than I was worried about, but there were complicated/missing bits in the hackmd docs (docs. were updated). As always doing > watching.

Hopefully can do a bunch of staging over the next two weeks, and then can plan on how many prod. hosts we can try to do during each of the following mass updates+reboots.

Managed to do `vmhost-x86-03.stg.rdu3.fedoraproject.org` today. Was easier than I was worried about, but there were complicated/missing bits in the hackmd docs (docs. were updated). As always doing > watching. Hopefully can do a bunch of staging over the next two weeks, and then can plan on how many prod. hosts we can try to do during each of the following mass updates+reboots.
Member

Note that almost all of the vmhosts in staging were done. The lone holdout is bvmhost-a64-01.stg.rdu3 which can't PXE boot, so we need to find a workaround.

No prod during the next sprint, but we can look at maybe doing some of the app VMs:

  • ipa0x.stg
  • mailman01.stg
  • memcached02.stg
  • rabbitmq0x.stg
  • zabbix01.stg
Note that almost all of the vmhosts in staging were done. The lone holdout is `bvmhost-a64-01.stg.rdu3` which can't PXE boot, so we need to find a workaround. No prod during the next sprint, but we can look at maybe doing some of the app VMs: - ipa0x.stg - mailman01.stg - memcached02.stg - rabbitmq0x.stg - zabbix01.stg
Author
Owner

FYI, I did work this last week to try and get aarch64's able to pxe boot... I think bvmhost-a64-01.stg should be ready to reinstall via pxe now, but needs to be added to dhcpd.conf first.

FYI, I did work this last week to try and get aarch64's able to pxe boot... I think bvmhost-a64-01.stg should be ready to reinstall via pxe now, but needs to be added to dhcpd.conf first.
Author
Owner

Done nameservers, smtp-mm's, almost all colovirt.

Done nameservers, smtp-mm's, almost all colovirt.
Author
Owner

bvmhost-a64-01.stg was done a while back.

We plan to do more virthosts this week in the outage time permitting.

bvmhost-a64-01.stg was done a while back. We plan to do more virthosts this week in the outage time permitting.
Author
Owner

#13382 ( ipa cluster ) and #13383 ( rabbitmq ) filed.

https://forge.fedoraproject.org/infra/tickets/issues/13382 ( ipa cluster ) and https://forge.fedoraproject.org/infra/tickets/issues/13383 ( rabbitmq ) filed.
Author
Owner

Down to 72 hosts.

6 ipa servers are tracked in seperate ticket
6 rabbitmq servers are tracked in seperate ticket
ibiblio02 should be done tomorrow (closing the colo_virt ticket)

I'm going to look at doing staging database servers next.

We should probibly look at the schedule and setup an outage soon to do a bunch more bvmhost/vmhosts and databases in prod.

mailman will need epel10 mailman3 stack arriving.
sign-vaults and autosign will need siguldry landing (there is a pr for staging)
zabbix could be done anytime.
batcave I could also look at doing somewhat soon.

Down to 72 hosts. 6 ipa servers are tracked in seperate ticket 6 rabbitmq servers are tracked in seperate ticket ibiblio02 should be done tomorrow (closing the colo_virt ticket) I'm going to look at doing staging database servers next. We should probibly look at the schedule and setup an outage soon to do a bunch more bvmhost/vmhosts and databases in prod. mailman will need epel10 mailman3 stack arriving. sign-vaults and autosign will need siguldry landing (there is a pr for staging) zabbix could be done anytime. batcave I could also look at doing somewhat soon.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
infra/tickets#12712
No description provided.