docs/modules/release_guide/pages/sop_epel_minor_eol.adoc

_partials/attributes.adoc

EPEL Minor Version End of Life

Description

Each minor release of EPEL is maintained as laid out in the EPEL policy.

At the conclusion of the maintenance period, an EPEL release enters "End of Life" status, also known as retirement. This procedure describes the tasks necessary to move a release to that status.

Preparatory steps

Send announcement

Before the actual retirement takes place, send a reminder to the epel-devel@ list that it will be happening soon. You can use this email as a reference.

Note	We’re not able to declare a specific date for the retirement because it is dependent on the RHEL schedule. As a rough guideline, try to send the reminder announcement at the beginning of the month in which the next RHEL minor version is expected to occur.

Prepare ansible changes

Some of the EOL process is controlled by ansible playbooks. Before these are run some changes will be needed in the ansible repo. Set these up ahead of time in a pull request, which will be merged on EOL day. These changes should include:

• vars/all/EPEL.yaml

  • change the epel_branched_minor variable to false

  • change the epel_z_minor variable to {epel_z_minor}

• roles/bodhi2/backend/files/new-updates-sync

  • remove the epel{epel_major}.{epel_eol_minor} section of the RELEASES dictionary

• roles/batcave/files/rhel{epel_major}-sync

  • change {epel_major}.{epel_eol_minor} to {epel_major}.{epel_z_minor} in RELEASEVERS

You can use this pull request as a reference.

RHEL buildroot pre-seeding

To speed up the sync of RHEL infra repos, pre-seed the RHEL {epel_major}.{epel_z_minor} directory on batcave01 by copying the {epel_major}.{epel_eol_minor} directory.

$ cd /mnt/fedora/app/fi-repo/rhel/rhel{epel_major}
$ mkdir {epel_major}.{epel_z_minor}
$ rsync -avH --link-dest=$PWD/{epel_major}.{epel_eol_minor}/repos {epel_major}.{epel_eol_minor}/repos/ {epel_major}.{epel_z_minor}/repos/

EOL day

Send announcement

Reply to the previous announcement to state that the mass branching is starting. Also send the corresponding mail archive link to the #epel:fedoraproject.org matrix channel.

Koji tasks

Disable builds by removing targets.

$ koji remove-target epel{epel_major}.{epel_eol_minor}
$ koji remove-target epel{epel_major}.{epel_eol_minor}-candidate

Remove any existing sidetags from the EOL release.

$ koji remove-sidetag $(koji list-sidetags --basetag epel{epel_major}.{epel_eol_minor}-build)

This must be done with koji remove-sidetag and not koji remove-target because just removing the target of a sidetag will leave it in a weird state where it cannot be removed.

Final stable push

Do a final stable push for the EPEL-{epel_major}.{epel_eol_minor} release in Bodhi, following the Pushing Updates SOP.

Bodhi tasks

Run the following bodhi commands to set the EPEL-{epel_major}.{epel_eol_minor} release to the archived state.

$ bodhi releases edit --name EPEL-{epel_major}.{epel_eol_minor} --state archived

Merge ansible changes

Merge the pending ansible pull request.

Run the playbooks

Run the following playbooks to apply the changes made in ansible.

$ sudo rbac-playbook groups/batcave.yml
$ sudo rbac-playbook groups/bodhi-backend.yml
$ sudo rbac-playbook groups/koji-hub.yml
$ sudo rbac-playbook -t all,rollout openshift-apps/bodhi.yml

You will need to have someone enable sshd on autosign02 and then run:

$ sudo rbac-playbook manual/autosign.yml

And then someone with the robosignatory passphrase will need to restart it.

Manually sync RHEL repos

Normally RHEL repos are synced via a cron job, but run them manually the first time to ensure completion. Run the sync script on batcave01:

$ sudo /mnt/fedora/app/fi-repo/rhel/rhel{epel_major}/rhel{epel_major}-sync

Koji tasks again

Remove the CentOS snapshot external repos.

$ koji remove-external-repo c{epel_major}-snapshot-baseos    epel{epel_major}.{epel_z_minor}-base
$ koji remove-external-repo c{epel_major}-snapshot-appstream epel{epel_major}.{epel_z_minor}-base
$ koji remove-external-repo c{epel_major}-snapshot-crb       epel{epel_major}.{epel_z_minor}-base

Add the RHEL {epel_major}.{epel_z_minor} external repos.

$ koji add-external-repo \
      --mode bare \
      --priority 10 \
      --tag epel{epel_major}.{epel_z_minor}-base \
      rhel{epel_major}.{epel_z_minor}-baseos \
      'https://infrastructure.fedoraproject.org/repo/rhel/rhel{epel_major}/{epel_major}.{epel_z_minor}/repos/$arch/baseos/'
$ koji add-external-repo \
      --mode bare \
      --priority 20 \
      --tag epel{epel_major}.{epel_z_minor}-base \
      rhel{epel_major}.{epel_z_minor}-appstream \
      'https://infrastructure.fedoraproject.org/repo/rhel/rhel{epel_major}/{epel_major}.{epel_z_minor}/repos/$arch/appstream/'
$ koji add-external-repo \
      --mode bare \
      --priority 30 \
      --tag epel{epel_major}.{epel_z_minor}-base \
      rhel{epel_major}.{epel_z_minor}-crb \
      'https://infrastructure.fedoraproject.org/repo/rhel/rhel{epel_major}/{epel_major}.{epel_z_minor}/repos/$arch/crb/'

Move epel{epel_major}z floating tag to next minor version.

$ koji remove-tag-inheritance epel{epel_major}z epel{epel_major}.{epel_eol_minor}
$ koji add-tag-inheritance epel{epel_major}z epel{epel_major}.{epel_z_minor}

Move epel{epel_major}-infra floating tag to next minor version.

$ koji remove-tag-inheritance epel{epel_major}-infra epel{epel_major}.{epel_eol_minor}-build
$ koji add-tag-inheritance epel{epel_major}-infra epel{epel_major}.{epel_z_minor}-build

Mirror tasks

Log into the MirrorManager Admin Panel and reconfigure the repository redirects as follows.

• epel-z-{epel_major} → epel-z-{epel_major}.{epel_z_minor}

• epel-z-debug-{epel_major} → epel-z-debug-{epel_major}.{epel_z_minor}

• epel-z-source-{epel_major} → epel-z-source-{epel_major}.{epel_z_minor}

• epel-z-testing-{epel_major} → epel-z-testing-{epel_major}.{epel_z_minor}

• epel-z-testing-debug-{epel_major} → epel-z-testing-debug-{epel_major}.{epel_z_minor}

• epel-z-testing-source-{epel_major} → epel-z-testing-source-{epel_major}.{epel_z_minor}

Run the following commands on batcave01 to switch symlinks on the mirrors.

$ sudo ln -sfn {epel_major}.{epel_z_minor} /pub/epel/{epel_major}z
$ sudo ln -sfn {epel_major}.{epel_z_minor} /pub/epel/testing/{epel_major}z

Koschei

Remove the EPEL {epel_major}.{epel_eol_minor} collection in koschei.

To be able do this, you will first need to log into os-control01 and run the following commands to get into the console.

$ oc project koschei
$ oc rsh deploy/admin

After that, you will be able to delete the EPEL {epel_major}.{epel_eol_minor} collection using the koschei-admin tool.

$ koschei-admin delete-collection --force epel{epel_major}.{epel_eol_minor}

Send announcement

Announce the completion of the retirement. Post this to Fedora Discussion in the Project Discussion category with the #epel-sig tag. Also send an email to the epel-devel@ and devel@ lists.

Consider Before Running

• Resource contention in infrastructure, such as outages

• Extenuating circumstances for specific planned updates, if any

• Send the reminder announcement, if it isn’t sent already