releng/tickets
15
3
Fork 2
Code Issues 79 Pull requests Projects Releases Packages Wiki Activity Actions

Fedora Releng Tracker Init #13095

Merged
jnsamyak merged 1 commit from jnsamyak/tickets:main into main 2025-11-18 16:40:21 +00:00
Conversation 0 Commits 1 Files changed 122 +159 -9342
122 changed files with 159 additions and 9342 deletions
Download patch file Download diff file Expand all files Collapse all files

17
.github/ISSUE_TEMPLATE.md vendored
View file

@ -1,17 +0,0 @@
⚠️ **ISSUE TRACKER HAS MOVED**
This issue tracker is **closed**.
Please file your issue at the new location:
**https://forge.fedoraproject.org/releng/tickets**
---
All existing issues have been migrated to Fedora Forge.
Search for existing issues before filing a new one:
https://forge.fedoraproject.org/releng/tickets
Thank you!

28
.github/PULL_REQUEST_TEMPLATE.md vendored
View file

@ -1,28 +0,0 @@
⚠️ **THIS REPOSITORY HAS MOVED**
This repository no longer accepts pull requests.
Please submit your contribution to the new repository:
**https://forge.fedoraproject.org/releng/releng**
---
## How to Submit Your Changes
1. **Clone the new repository:**
```bash
git clone https://forge.fedoraproject.org/releng/tooling.git
```
2. **Make your changes** and commit them
3. **Push to your fork** on Fedora Forge
4. **Open a Pull Request** at:
https://forge.fedoraproject.org/releng/releng/pulls
---
Thank you for contributing to Fedora Release Engineering!

33
CONTRIBUTING.md
View file

@ -1,33 +0,0 @@
# ⚠️ THIS REPOSITORY HAS MOVED
**This repository is no longer accepting contributions here.**
## 🔗 New Contribution Location
Please submit all contributions to the new repository on Fedora Forge:
**https://forge.fedoraproject.org/releng/tooling**
---
## 📝 How to Contribute at the New Location
1. **Visit:** https://forge.fedoraproject.org/releng/
2. **Read the Contributing Guide:** Available in the new repository
3. **Open Pull Requests:** Submit to the Fedora Forge repository
4. **File Issues:** Use https://forge.fedoraproject.org/releng/tickets
---
## 🔄 Update Your Git Remote
If you have a local clone of this repository, update it to point to the new location:
```bash
git remote set-url origin https://forge.fedoraproject.org/releng/tooling.git
```
---
**All active development is now at:** https://forge.fedoraproject.org/releng/tooling

151
MIGRATION.md
View file

@ -1,151 +0,0 @@
# Repository Migration to Fedora Forge
## Overview
The Fedora Release Engineering repository has been migrated from its previous location to the new Fedora Forge infrastructure.
## New Locations
| Component | Previous Location | New Location |
|-----------|------------------|--------------|
| **Repository** | https://pagure.io/releng | https://forge.fedoraproject.org/releng/tooling |
| **Issue Tracker** | https://pagure.io/releng/issues | https://forge.fedoraproject.org/releng/tickets |
| **Pull Requests** | https://pagure.io/releng/pull-requests | https://forge.fedoraproject.org/releng/releng/pulls |
## Migration Details
### What Was Migrated
✅ **Source Code**
- Complete Git history preserved
- All branches migrated
- Tags and releases maintained
✅ **Issues**
- All open issues migrated to new tracker
- Issue numbers and metadata preserved
- Comments and attachments transferred
✅ **Documentation**
- README files updated
- Contributing guidelines migrated
- Conventions documentation preserved
### Migration Timeline
- **Announcement Date:** [Date]
- **Migration Date:** [Date]
- **Old Repository Status:** Read-only archive
- **Issue Tracker Status:** Closed
## For Repository Users
### Update Your Git Remote
If you have a local clone of this repository:
```bash
# Check your current remote
git remote -v
# Update to new location
git remote set-url origin https://forge.fedoraproject.org/releng/releng.git
# Verify the change
git remote -v
```
### Fresh Clone
For new clones, use the new repository location:
```bash
git clone https://forge.fedoraproject.org/releng/releng.git
cd releng
```
## For Contributors
### Submitting Changes
1. **Fork** the repository at https://forge.fedoraproject.org/releng/releng
2. **Clone** your fork locally
3. **Create** a feature branch for your changes
4. **Push** your changes to your fork
5. **Open** a Pull Request at the new repository
### Filing Issues
1. **Search** existing issues: https://forge.fedoraproject.org/releng/tickets
2. **File new issues** at the new tracker
3. **Reference** related issues using the new issue numbers
## For Automation & CI/CD
### Update References
If your automation or CI/CD pipelines reference this repository:
**URLs to Update:**
- Repository URLs → `https://forge.fedoraproject.org/releng/tooling`
- Issue tracker URLs → `https://forge.fedoraproject.org/releng/tickets`
- Documentation URLs → `https://docs.fedoraproject.org/en-US/infra/release_guide/`
**API Endpoints:**
- Check Fedora Forge API documentation for new endpoints
- Update authentication tokens if needed
## Benefits of Fedora Forge
The migration to Fedora Forge provides:
🚀 **Modern Infrastructure**
- Improved performance and reliability
- Better integration with Fedora ecosystem
- Enhanced security features
🤝 **Better Collaboration**
- Unified platform for all Fedora projects
- Improved code review tools
- Better issue tracking and project management
🔧 **Enhanced Workflows**
- Native CI/CD integration
- Better Git workflow support
- Improved notification system
## Support
### Getting Help
If you encounter issues with the migration:
1. **Check Documentation:** https://docs.fedoraproject.org/en-US/infra/release_guide/
2. **Ask on IRC/Matrix:** #fedora-releng on Libera.Chat
3. **Mailing List:** devel@lists.fedoraproject.org
4. **File an Issue:** https://forge.fedoraproject.org/releng/tickets
### Common Issues
**Q: My local clone stopped working**
A: Update your remote URL as shown above
**Q: Where did my open issue go?**
A: Check https://forge.fedoraproject.org/releng/tickets - all issues were migrated
**Q: Can I still access the old repository?**
A: Yes, it remains available as a read-only archive for reference
**Q: What about pull requests that were open?**
A: Contact the release engineering team if you had open PRs that need attention
## Historical Reference
The original repository information and licensing details are preserved in `README.old`.
---
**For the latest information, always refer to:** https://forge.fedoraproject.org/releng
**Last Updated:** [Date]

59
Makefile
View file

@ -1,59 +0,0 @@
PYTHON=python3
PEP8=$(PYTHON)-pep8
COVERAGE=coverage
ifeq ($(PYTHON),python3)
COVERAGE=coverage3
endif
TEST_DEPENDENCIES = python3-pep8 python3-pocketlint
TEST_DEPENDENCIES += python3-koji fedora-cert packagedb-cli
TEST_DEPENDENCIES += python3-fedmsg-core python3-configparser
TEST_DEPENDENCIES := $(shell echo $(sort $(TEST_DEPENDENCIES)) | uniq)
check-requires:
@echo "*** Checking if the dependencies required for testing and analysis are available ***"
@status=0 ; \
for pkg in $(TEST_DEPENDENCIES) ; do \
test_output="$$(rpm -q --whatprovides "$$pkg")" ; \
if [ $$? != 0 ]; then \
echo "$$test_output" ; \
status=1 ; \
fi ; \
done ; \
exit $$status
install-requires:
@echo "*** Installing the dependencies required for testing and analysis ***"
dnf install -y $(TEST_DEPENDENCIES)
test: check-requires
@echo "*** Running unittests with $(PYTHON) ***"
PYTHONPATH=.:scripts/:$(PYTHONPATH) $(PYTHON) -m unittest discover -v -s tests/ -p '*_test.py'
coverage: check-requires
@echo "*** Running unittests with $(COVERAGE) for $(PYTHON) ***"
PYTHONPATH=.:tests/ $(COVERAGE) run --branch -m unittest discover -v -s tests/ -p '*_test.py'
$(COVERAGE) report --include="scripts/*" --show-missing
$(COVERAGE) report --include="scripts/*" > coverage-report.log
pylint: check-requires
@echo "*** Running pylint ***"
PYTHONPATH=.:tests/:$(PYTHONPATH) tests/pylint/runpylint.py
pep8: check-requires
@echo "*** Running pep8 compliance check ***"
$(PEP8) --ignore=E501,E402,E731 tests/ scripts/ > pep8.log
check:
@status=0; \
$(MAKE) pylint || status=1; \
$(MAKE) pep8 || status=1; \
exit $$status
flake8:
@echo "*** Running flake8 against push-two-week-atomic only ***"
flake8-2 --ignore=W503,E131,E501,E226,E302,E265,E303 scripts/push-two-week-atomic.py
ci: check coverage
.PHONY: check pylint pep8 test

269
README.md
View file

@ -1,98 +1,191 @@
# ⚠️ THIS REPOSITORY HAS MOVED
# Fedora Release Engineering - Issue Tracker
## 🔗 New Location
Welcome to the Fedora Release Engineering (RelEng) issue tracker!
This repository has been migrated to Fedora Forge and is **no longer maintained here**.
## 🎯 What is Fedora Release Engineering?
**New Repository:** https://forge.fedoraproject.org/releng/releng
Fedora Release Engineering is the team responsible for creating, building, and delivering official Fedora releases. We manage the infrastructure and processes that transform thousands of packages into cohesive Fedora products.
**Issue Tracker:** https://forge.fedoraproject.org/releng/tickets
### Our Responsibilities
**Release Management**
- Create official Fedora releases (Workstation, Server, Cloud, Spins, etc.)
- Manage release schedules and milestones
- Coordinate freeze periods and go/no-go decisions
**Build Infrastructure**
- Administer the Koji build system
- Manage compose generation and distribution
- Handle package signing and repository creation
**Package Lifecycle**
- Process package updates and new submissions
- Manage package retirement and orphan cleanup
- Handle mass rebuilds for toolchain updates
**Quality Assurance**
- Monitor build health and critical path packages
- Track Failure to Build From Source (FTBFS) issues
- Validate upgrade paths and compose integrity
**Process & Policy**
- Set and enforce freeze management policies
- Report progress to FESCo (Fedora Engineering Steering Committee)
- Maintain and develop release engineering tools
## 📋 What This Issue Tracker Is For
This tracker is for reporting issues related to Fedora Release Engineering processes, tools, and infrastructure.
### ✅ Appropriate Issues
**Infrastructure & Tools**
- Koji build system issues
- Compose generation problems
- Repository synchronization failures
- Signing infrastructure issues
**Release Process**
- Release milestone concerns
- Branching issues
- Mass rebuild coordination
- Update push problems
**Package Management**
- Retirement/unretirement requests
- Critical path package questions
- Tag management issues
- Override requests
**Automation & Scripts**
- Release engineering script bugs
- Automation failures
- Tool enhancement requests
- CI/CD pipeline issues
**Documentation**
- Process documentation improvements
- Missing or unclear procedures
- Release guide updates
### ❌ Issues That Belong Elsewhere
**Individual Package Builds**
- File at the package's issue tracker on [Fedora Packages](https://src.fedoraproject.org/)
- For build failures, contact the package maintainer first
**Bodhi Update Problems**
- File at [Bodhi issue tracker](https://github.com/fedora-infra/bodhi/issues)
**Infrastructure (Non-RelEng)**
- General Fedora infrastructure: [Fedora Infrastructure](https://pagure.io/fedora-infrastructure/issues)
**Package Reviews**
- Use [Bugzilla](https://bugzilla.redhat.com/) for package reviews
**General Support Questions**
- Ask on mailing lists: devel@lists.fedoraproject.org
- IRC/Matrix: [#releng:fedoraproject.org](https://app.element.io/#/room/#releng:fedoraproject.org)
## 🚀 How to Use This Tracker
### Before Filing an Issue
1. **Search existing issues** to avoid duplicates
2. **Check documentation**: https://docs.fedoraproject.org/en-US/infra/release_guide/
3. **Verify it's a RelEng issue** (see appropriate issues above)
4. **Gather relevant information** (logs, error messages, reproduction steps)
## 👥 Who We Are
The Release Engineering team consists of:
- Full-time Fedora Release Engineers
- Part-time contributors and proven packagers
- Community volunteers
We work to ensure Fedora releases are:
- **Reliable** - Consistent quality and stability
- **Timely** - On schedule with predictable cadence
- **Secure** - Properly signed and verified
- **Accessible** - Available through multiple channels
## 📚 Resources
### Documentation
- **Release Guide**: https://docs.fedoraproject.org/en-US/infra/release_guide/
- **Fedora Infrastructure Docs**: https://docs.fedoraproject.org/en-US/infra/
### Code & Tools
- **RelEng Tooling Repository**: https://forge.fedoraproject.org/releng/tooling
- **Scripts & Automation**: Check the tooling repo for current scripts
### Communication Channels
**Issue Tracker** (you are here!)
- https://forge.fedoraproject.org/releng/tickets
**Mailing List**
- devel@lists.fedoraproject.org (general discussion)
- [Subscribe/Archives](https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/)
**Chat (IRC/Matrix)**
- Matrix: [#releng:fedoraproject.org](https://app.element.io/#/room/#releng:fedoraproject.org)
- IRC: #releng:fedoraproject.org
## 🤝 Contributing
Want to help improve Fedora Release Engineering?
1. **Participate in discussions** on issues
2. **Submit pull requests** to the [tooling repository](https://forge.fedoraproject.org/releng/tooling)
3. **Improve documentation** with your knowledge
4. **Join RelEng meetings** to stay informed
5. **Report issues** you encounter
Check our [Contributing Guide](https://forge.fedoraproject.org/releng/tooling/src/branch/main/CONTRIBUTING.md) for more details.
## ❓ Frequently Asked Questions
**Q: How do I request a package unretirement?**
A: File an issue here with the package name and justification for unretirement.
**Q: My package build failed in Koji, what do I do?**
A: If it's a package-specific issue, contact the maintainer. If it's a broader infrastructure issue affecting multiple packages, file an issue here.
**Q: How do I request a buildroot override?**
A: Use the [Bodhi web interface](https://bodhi.fedoraproject.org/) for override requests.
**Q: When is the next release?**
A: Check the [Fedora Release Schedule](https://fedoraproject.org/wiki/Releases/Schedule).
**Q: How do I become a Release Engineer?**
A: Start by contributing to RelEng tasks, attending meetings, and getting involved in the community. Contact the team through our communication channels.
## 📅 Release Schedule
Fedora follows a time-based release schedule with releases approximately every 6 months. Each release goes through several phases:
1. **Branching** - New release branch created
2. **Development** - Active development and testing
3. **Freeze Periods** - Various freezes (Change Checkpoint, Beta, Final)
4. **Beta Release** - First public release for testing
5. **Final Release** - Official stable release
6. **Maintenance** - Updates and security fixes
Current and upcoming schedules: https://fedoraproject.org/wiki/Releases/Schedule
## 📞 Emergency Contact
For urgent, release-blocking issues:
1. **File a high-priority issue** here with `[URGENT]` in the title
2. **Notify on IRC/Matrix** in #fedora-releng
3. **Email the mailing list** if immediate response needed
---
## 📍 What This Means
**Thank you for helping make Fedora releases better!**
### For Contributors
- **All new contributions** should be submitted to the [new repository on Fedora Forge](https://forge.fedoraproject.org/releng/releng)
- **Pull requests** opened here will not be reviewed or merged
- Please redirect any work to the new location
The Fedora Release Engineering Team
### For Issue Reporters
- **Issue tracker is closed** here
- All existing open issues have been migrated to [Fedora Forge Tickets](https://forge.fedoraproject.org/releng/tickets)
- **New issues** must be filed at the new tracker
- Search for existing issues at the new location before filing
### For Users
- **Scripts and tools** are maintained at the new location
- **Documentation** has been updated at the new repository
- Bookmark the new repository for future reference
---
## 🚀 Quick Links
| Resource | New Location |
|----------|-------------|
| **Repository** | https://forge.fedoraproject.org/releng/tooling |
| **Issue Tracker** | https://forge.fedoraproject.org/releng/tickets |
| **Documentation** | https://docs.fedoraproject.org/en-US/infra/release_guide/ |
---
## 📋 Migration Information
### Timeline
- **Migration Date:** [Date when migration completed]
- **Repository Status:** Read-only / Archived
- **Issue Tracker Status:** Closed
### What Was Migrated
- ✅ All source code and scripts
- ✅ Git history preserved
- ✅ Open issues and discussions
- ✅ Documentation and conventions
- ✅ Release engineering tools and workflows
### What To Do Next
1. **Update your remotes:**
```bash
git remote set-url origin https://forge.fedoraproject.org/releng/releng.git
```
2. **Clone from new location:**
```bash
git clone https://forge.fedoraproject.org/releng/releng.git
```
3. **Update bookmarks** to point to Fedora Forge
4. **File new issues** at the new tracker
---
## 📞 Contact
For questions about the migration or the new infrastructure:
- **Mailing List:** [devel@lists.fedoraproject.org](mailto:devel@lists.fedoraproject.org)
- **IRC/Matrix:** https://app.element.io/#/room/#releng:fedoraproject.org
- **Issue Tracker:** https://forge.fedoraproject.org/releng/tickets
---
## 🏛️ About Fedora Forge
Fedora Forge is the new unified infrastructure for Fedora project development, providing:
- Modern Git-based workflows
- Integrated issue tracking
- Better collaboration tools
- Improved CI/CD integration
Learn more: https://forge.fedoraproject.org/
---
**This repository will remain available in read-only mode for historical reference.**
**All active development has moved to Fedora Forge: https://forge.fedoraproject.org/releng/releng**
*Last Updated: $(date +%Y-%m-%d)*

47
README.old
View file

@ -1,47 +0,0 @@
This is the Fedora Release Engineering GIT repo. Random stuff is tossed
here.
#########################################################################
# #
# For more information please see: #
# https://docs.fedoraproject.org/en-US/infra/ #
# #
#########################################################################
Everything is copyrighted by the respective authors. You can use and
redistribute the code under the terms of version 2 or later of the
GNU Public License as published by the Free Software Foundation.
To make licensing easier, license headers in the source files will be
a single line reference to Unique License Identifiers as defined by
the Linux Foundation's SPDX project [1]. For example,
in a source file the full "GPL v2.0 or later" header text will be
replaced by a single line:
SPDX-License-Identifier: GPL-2.0+
the license terms of all files in the source tree should be defined
by such License Identifiers; in no case a file can contain more than
one such License Identifier list.
If a "SPDX-License-Identifier:" line references more than one Unique
License Identifier, then this means that the respective file can be
used under the terms of either of these licenses, i. e. with
SPDX-License-Identifier: GPL-2.0+ LGPL-2.1+
you can chose between GPL-2.0+ and LGPL-2.1+ licensing.
We use the SPDX Unique License Identifiers here; these are available
at [2].
[1] http://spdx.org/
[2] http://spdx.org/licenses/
Full name SPDX Identifier OSI Approved File name URI
============================================================================================================================================================
GNU General Public License v2.0 or later GPL-2.0+ Y gpl-2.0.txt http://www.gnu.org/licenses/gpl-2.0.txt
GNU Lesser General Public License v2.1 or later LGPL-2.1+ Y lgpl-2.1.txt http://www.gnu.org/licenses/old-licenses/lgpl-2.1.txt
Creative Commons Attribution Share Alike 3.0 CC-BY-SA-3.0 CC-BY-SA-3.0.html http://spdx.org/licenses/CC-BY-SA-3.0.html

18
TESTING
View file

@ -1,18 +0,0 @@
Code testing helps shake out issues before they get into the wider world.
A very basic vagrant file is provided to allow contributors to work on issues
without installing all of the python3 libaries to their local workstation.
Tests may be added via the tests directory in the format:
tests/<class|script>_test.py
'make test' - runs current tests
'make pylint' - runs the python3 linter
'make pep8' - runs pep8 checks against the code
'make install-requires' - installs required python libraries
'make check' - runs pylint and pep8 against all code
'make flake8' - runs fake8 test against scripts/push-two-week-atomic.py
The testing framework is written for Python3 to help fix issues during the
conversion to Python3 scheduled for Fedora 27.

14
Vagrantfile vendored
View file

@ -1,14 +0,0 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :
# SPDX-License-Identifier: GPL-2.0+
# Authors:
# Robert Marshall <rmarshall@redhat.com>
Vagrant.configure("2") do |config|
config.vm.box = "fedora/29-cloud-base"
config.vm.provision "shell", inline: <<-SHELL
dnf update -y
dnf install -y vim python3 fpaste
SHELL
end

21
configs/23-atomic-updates.mash
View file

@ -1,21 +0,0 @@
# mash config file
[23-atomic-updates]
rpm_path = %(arch)s/
repodata_path = %(arch)s/
source_path = source/
debuginfo = True
multilib = True
multilib_method = devel
tag = f23-atomic-updates
inherit = False
strict_keys = True
keys = 34EC9CBA
arches = i386 x86_64 armhfp
delta = True
max_delta_rpm_size = 800000000
max_delta_rpm_age = 604800
delta_workers = 1
distro_tags = cpe:/o:fedoraproject:fedora:23 Twenty Three
hash_packages = True

22
configs/24-openh264.mash
View file

@ -1,22 +0,0 @@
# mash config file
[24-openh264]
rpm_path = %(arch)s/
repodata_path = %(arch)s/
source_path = source/
debuginfo = True
multilib = True
multilib_method = devel
tag = f24-openh264
inherit = False
strict_keys = True
keys = 81b46521
arches = i386 x86_64 armhfp
delta = True
max_delta_rpm_size = 800000000
max_delta_rpm_age = 604800
delta_workers = 8
# Change distro_tags as fedora-release version gets bumped
distro_tags = cpe:/o:fedoraproject:fedora:24 Twenty Four
hash_packages = True

22
configs/26-openh264.mash
View file

@ -1,22 +0,0 @@
# mash config file
[26-openh264]
rpm_path = %(arch)s/
repodata_path = %(arch)s/
source_path = source/
debuginfo = True
multilib = True
multilib_method = devel
tag = f26-openh264
inherit = False
strict_keys = True
keys = 81b46521
arches = i386 x86_64 armhfp
delta = True
max_delta_rpm_size = 800000000
max_delta_rpm_age = 604800
delta_workers = 8
# Change distro_tags as fedora-release version gets bumped
distro_tags = cpe:/o:fedoraproject:fedora:26 Twenty Six
hash_packages = True

22
configs/27-openh264.mash
View file

@ -1,22 +0,0 @@
# mash config file
[27-openh264]
rpm_path = %(arch)s/
repodata_path = %(arch)s/
source_path = source/
debuginfo = True
multilib = True
multilib_method = devel
tag = f27-openh264
inherit = False
strict_keys = True
keys = 81b46521
arches = i386 x86_64 armhfp
delta = True
max_delta_rpm_size = 800000000
max_delta_rpm_age = 604800
delta_workers = 8
# Change distro_tags as fedora-release version gets bumped
distro_tags = cpe:/o:fedoraproject:fedora:27 Twenty Seven
hash_packages = True

22
configs/28-openh264.mash
View file

@ -1,22 +0,0 @@
# mash config file
[28-openh264]
rpm_path = %(arch)s/
repodata_path = %(arch)s/
source_path = source/
debuginfo = True
multilib = True
multilib_method = devel
tag = f28-openh264
inherit = False
strict_keys = True
keys = 9DB62FB1
arches = i386 x86_64 armhfp aarch64 ppc64 ppc64le s390x
delta = True
max_delta_rpm_size = 800000000
max_delta_rpm_age = 604800
delta_workers = 8
# Change distro_tags as fedora-release version gets bumped
distro_tags = cpe:/o:fedoraproject:fedora:28 Twenty Eight
hash_packages = True

22
configs/29-openh264.mash
View file

@ -1,22 +0,0 @@
# mash config file
[29-openh264]
rpm_path = %(arch)s/
repodata_path = %(arch)s/
source_path = source/
debuginfo = True
multilib = True
multilib_method = devel
tag = f29-openh264
inherit = False
strict_keys = True
keys = 429476b4
arches = i386 x86_64 armhfp aarch64 ppc64 ppc64le s390x
delta = True
max_delta_rpm_size = 800000000
max_delta_rpm_age = 604800
delta_workers = 8
# Change distro_tags as fedora-release version gets bumped
distro_tags = cpe:/o:fedoraproject:fedora:29 Twenty Nine
hash_packages = True

8
configs/mash.conf
View file

@ -1,8 +0,0 @@
[defaults]
configdir = ./configs/
buildhost = https://koji.fedoraproject.org/kojihub
repodir = file:///mnt/koji
use_sqlite = True
use_repoview = False
workdir = /var/tmp/releng-mash/

115
koji-archive-policy
View file

@ -1,115 +0,0 @@
# Don't mess with things already moved
volume fedora_koji_archive00 :: skip
hastag do-not-archive-yet :: skip
# Don't move modular builds yet
hashtag f*modular* :: skip
#match release *.el[5678] *.el[5678]_[0-9] *.el[5678]_1[0-9] :: skip
# stuff we're not ready to move yet
#buildtag RHEL-[5678]* rhel-[5678]* *-rhel-[5678]* dist-[567]E* :: skip
#buildtag *-[567]E *-[567]E-* :: skip
#hastag RHEL-[5678]* rhel-[5678]* *-rhel-[5678]* dist-[567]E* :: skip
#hastag *-[567]E *-[567]E-* :: skip
buildtag dist-fc6* :: fedora_koji_archive00
imported && hastag dist-fc6* :: fedora_koji_archive00
buildtag dist-fc7* :: fedora_koji_archive00
hastag f7-final :: fedora_koji_archive00
hastag f7-test4 :: fedora_koji_archive00
imported && hastag dist-fc7* :: fedora_koji_archive00
imported && hastag fe7-merge :: fedora_koji_archive00
buildtag dist-f8* :: fedora_koji_archive00
hastag dist-f8* :: fedora_koji_archive00
hastag f8-final :: fedora_koji_archive00
buildtag dist-f9* :: fedora_koji_archive00
hastag dist-f9* :: fedora_koji_archive00
hastag f9-alpha :: fedora_koji_archive00
hastag f9-betal :: fedora_koji_archive00
hastag f9-final :: fedora_koji_archive00
hastag f9-build-cutoff :: fedora_koji_archive00
hastag f9-cutoff :: fedora_koji_archive00
buildtag dist-f10* :: fedora_koji_archive00
hastag dist-f10* :: fedora_koji_archive00
hastag f10-alpha :: fedora_koji_archive00
hastag f10-betal :: fedora_koji_archive00
hastag f10-final :: fedora_koji_archive00
hastag f10-kernel :: fedora_koji_archive00
buildtag dist-f11* :: fedora_koji_archive00
hastag dist-f11* :: fedora_koji_archive00
hastag f11* :: fedora_koji_archive00
buildtag dist-f12* :: fedora_koji_archive00
hastag dist-f12* :: fedora_koji_archive00
hastag f12* :: fedora_koji_archive00
buildtag dist-f13* :: fedora_koji_archive00
hastag dist-f13* :: fedora_koji_archive00
hastag f13* :: fedora_koji_archive00
buildtag dist-f14* :: fedora_koji_archive00
hastag dist-f14* :: fedora_koji_archive00
hastag f14* :: fedora_koji_archive00
buildtag dist-f15* :: fedora_koji_archive00
hastag dist-f15* :: fedora_koji_archive00
hastag f15* :: fedora_koji_archive00
buildtag dist-f16* :: fedora_koji_archive00
hastag dist-f16* :: fedora_koji_archive00
hastag f16* :: fedora_koji_archive00
buildtag dist-f17* :: fedora_koji_archive00
hastag dist-f17* :: fedora_koji_archive00
hastag f17* :: fedora_koji_archive00
buildtag dist-f18* :: fedora_koji_archive00
hastag dist-f18* :: fedora_koji_archive00
hastag f18* :: fedora_koji_archive00
buildtag dist-f19* :: fedora_koji_archive00
hastag dist-f19* :: fedora_koji_archive00
hastag f19* :: fedora_koji_archive00
buildtag dist-f20* :: fedora_koji_archive00
hastag dist-f20* :: fedora_koji_archive00
hastag f20* :: fedora_koji_archive00
buildtag dist-f21* :: fedora_koji_archive01
hastag dist-f21* :: fedora_koji_archive01
hastag f21* :: fedora_koji_archive01
buildtag dist-f22* :: fedora_koji_archive01
hastag dist-f22* :: fedora_koji_archive01
hastag f22* :: fedora_koji_archive01
buildtag dist-f23* :: fedora_koji_archive01
hastag dist-f23* :: fedora_koji_archive01
hastag f23* :: fedora_koji_archive01
buildtag dist-f24* :: fedora_koji_archive01
hastag dist-f24* :: fedora_koji_archive01
hastag f24* :: fedora_koji_archive01
buildtag dist-f25* :: fedora_koji_archive01
hastag dist-f25* :: fedora_koji_archive01
hastag f25* :: fedora_koji_archive01
buildtag dist-f26* :: fedora_koji_archive01
hastag dist-f26* :: fedora_koji_archive01
hastag f26* :: fedora_koji_archive01
buildtag dist-f27* :: fedora_koji_archive02
hastag dist-f27* :: fedora_koji_archive02
hastag f27* :: fedora_koji_archive02
buildtag dist-f28* :: fedora_koji_archive02
hastag dist-f28* :: fedora_koji_archive02
hastag f28* :: fedora_koji_archive02

35
mail-templates/01-mass-rebuild-start.txt
View file

@ -1,35 +0,0 @@
Hi all,
Per the Fedora Linux fN schedule [1] we have started a mass rebuild on
YYYY-MM-DD for Fedora fN. We are running this mass rebuild for the
changes listed in:
https://pagure.io/releng/issues?status=Open&tags=mass+rebuild
This mass rebuild will be done in a side tag (fN-rebuild) and merged
when completed.
Failures can be seen
https://kojipkgs.fedoraproject.org/mass-rebuild/fN-failures.html
<https://kojipkgs.fedoraproject.org/mass-rebuild/fN-failures.html>
Things still needing rebuilding
https://kojipkgs.fedoraproject.org/mass-rebuild/fN-need-rebuild.html
<https://kojipkgs.fedoraproject.org/mass-rebuild/fN-need-rebuild.html>
FTBFS (Fails To Build From Source) bugs will be filed shortly after the
mass rebuild is complete.
Please be sure to let releng know if you see any bugs in the reporting.
You can contact releng in the #releng:fedoraproject.org room on Matrix,
or by dropping an email to our list [2] or filing an issue in pagure [3].
This email template is also in https://pagure.io/releng if you wish to
propose improvements or changes to it.
Regards,
Fedora Release Engineering
[1] https://fedorapeople.org/groups/schedule/f-fN/f-fN-key-tasks.html
[2] https://lists.fedoraproject.org/admin/lists/rel-eng.lists.fedoraproject.org/
[3] https://pagure.io/releng/

29
mail-templates/02-mass-rebuild-finished.txt
View file

@ -1,29 +0,0 @@
Hi all,
Per the Fedora Linux fN schedule [1] we started a mass rebuild for
Fedora Linux fN on YYYY-MM-DD. We did a mass rebuild for Fedora Linux fN
for:
<list changes that need mass rebuild here>
https://fedoraproject.org/wiki/Changes/NAME
The mass rebuild was done in a side tag (fN-rebuild) and moved over to
fN. Failures can be seen
https://kojipkgs.fedoraproject.org/mass-rebuild/fN-failures.html
Things still needing rebuilding
https://kojipkgs.fedoraproject.org/mass-rebuild/fN-need-rebuild.html
X builds have been tagged into fN, there is currently Y failed builds
that need to be addressed by the package maintainers. FTBFS bugs will be
filed shortly.
Please be sure to let releng know if you see any bugs in the reporting.
You can contact releng in the #releng:fedoraproject.org room on Matrix,
or by dropping an email to our list [2] or filing an issue in pagure [3].
Regards,
Fedora Release Engineering
[1] https://fedorapeople.org/groups/schedule/f-fN/f-fN-key-tasks.html
[2] https://lists.fedoraproject.org/admin/lists/rel-eng.lists.fedoraproject.org/
[3] https://pagure.io/releng/

24
mail-templates/03-mass-branch.txt
View file

@ -1,24 +0,0 @@
Hi All,
Fedora Linux fN has now been branched, please be sure to do a
'git fetch -v' to pick up the new branch. As an additional reminder,
rawhide/fN has been completely isolated from previous releases, which
means that anything you do for fN you also have to do in the rawhide
branch and do a build there. There will be a Fedora Linux fN compose and
it will appear in [1] once complete.
Bodhi is currently enabled in the fN branch like it is for rawhide, with
automatic update creation. At the hit Beta change freeze point in the
Fedora Linux fN schedule [2] updates-testing will be enabled and manual
bodhi updates will be required as in all stable releases.
fN/branched release is frozen right now until we get a successful
compose, expect that your fN builds won't be available immediately.
Thanks for understanding.
Regards,
Fedora Release Engineering
[1] https://dl.fedoraproject.org/pub/fedora/linux/development/fN/
[2] https://fedorapeople.org/groups/schedule/f-fN/f-fN-key-tasks.html

40
mail-templates/05-beta-release.txt
View file

@ -1,40 +0,0 @@
Fedora Linux fN Beta Released
------------------------------------------
The Fedora Project is pleased to announce the immediate availability of
Fedora Linux fN Beta, the next step towards our planned Fedora Linux fN
release at the end of MONTH.
Download the prerelease from our Get Fedora site:
* Get Fedora Linux fN Beta Workstation: https://getfedora.org/workstation/download/
* Get Fedora Linux fN Beta Server: https://getfedora.org/server/download/
* Get Fedora Linux fN Beta IoT: https://getfedora.org/iot/download/
* Get Fedora Linux fN Beta CoreOS: <LINK NEEDED>
* Get Fedora Linux fN Beta Cloud: <LINK NEEDED>
Or, check out one of our popular variants, including KDE Plasma, Xfce,
and other desktop environments:
* Get Fedora Linux fN Beta Spins: https://spins.fedoraproject.org/prerelease
* Get Fedora Linux fN Beta Labs: https://labs.fedoraproject.org/prerelease
## Beta Release Highlights
<insert talking points here>
For more details about the release, read the full announcement at
* https://fedoramagazine.org/announcing-fedora-fN-beta/
or look for the prerelease pages in the download sections at
* https://getfedora.org/
Since this is a Beta release, we expect that you may encounter bugs or
missing features. To report issues encountered during testing, contact
the Fedora QA team via the test@lists.fedoraproject.org mailing list or
in #fedora-qa on Libera Chat or the #qa:fedoraproject.org Matrix room.
Regards,
Fedora Release Engineering

21
mail-templates/07-inactive-packagers-removal.txt
View file

@ -1,21 +0,0 @@
Hi all,
Today, YYYY-MM-DD, we will be removing inactive packagers
from the packager group.
This is in accordance with the FESCo policy on inactive packagers:
https://docs.fedoraproject.org/en-US/fesco/Policy_for_inactive_packagers/
If the removed user is 'main admin' for a package, this package
will be orphaned. If there are co-maintainers for the package,
one of them should take the role of 'main admin',
by clicking "✋ Take" on
`https://src.fedoraproject.org/rpms/<package>`".
Otherwise any packager may take the package while it's orphaned.
After 6 weeks, the package will be retired.
After another 8 weeks, a new review is needed to unretire it.
see https://docs.fedoraproject.org/en-US/fesco/Policy_for_orphan_and_retired_packages/
for more details.
Packages that have been orphaned are:

19
mail-templates/08-end-of-life.txt
View file

@ -1,19 +0,0 @@
Hello all,
Fedora Linux NN will go end of life for updates and support on
YYYY-MM-DD.
No more updates of any kind, including security updates or security
announcements, will be available for Fedora Linux NN after this
date. No pending updates for Fedora Linux NN will be pushed to stable.
Fedora Linux NN+1 will continue to receive updates until approximately
one month after the release of Fedora Linux NN+3. The maintenance
schedule of Fedora Linux releases is documented here[1]. The docs also
contain instructions[2] on how to upgrade from a previous release of
Fedora Linux to a version receiving updates.
Regards,
Fedora Release Engineering
[1] https://docs.fedoraproject.org/en-US/releases/lifecycle/#_maintenance_schedule
[2] https://docs.fedoraproject.org/en-US/quick-docs/upgrading-fedora-new-release/

272
scripts/LICENSE
View file

@ -1,272 +0,0 @@
The GNU General Public License (GPL)
Version 2, June 1991
Copyright (C) 1989, 1991 Free Software Foundation, Inc.
59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The licenses for most software are designed to take away your freedom to
share and change it. By contrast, the GNU General Public License is
intended to guarantee your freedom to share and change free software--to
make sure the software is free for all its users. This General Public
License applies to most of the Free Software Foundation's software and to
any other program whose authors commit to using it. (Some other Free
Software Foundation software is covered by the GNU Library General Public
License instead.) You can apply it to your programs, too.
When we speak of free software, we are referring to freedom, not price.
Our General Public Licenses are designed to make sure that you have the
freedom to distribute copies of free software (and charge for this service
if you wish), that you receive source code or can get it if you want it,
that you can change the software or use pieces of it in new free programs;
and that you know you can do these things.
To protect your rights, we need to make restrictions that forbid anyone to
deny you these rights or to ask you to surrender the rights. These
restrictions translate to certain responsibilities for you if you
distribute copies of the software, or if you modify it.
For example, if you distribute copies of such a program, whether gratis or
for a fee, you must give the recipients all the rights that you have. You
must make sure that they, too, receive or can get the source code. And you
must show them these terms so they know their rights.
We protect your rights with two steps: (1) copyright the software, and (2)
offer you this license which gives you legal permission to copy,
distribute and/or modify the software.
Also, for each author's protection and ours, we want to make certain that
everyone understands that there is no warranty for this free software. If
the software is modified by someone else and passed on, we want its
recipients to know that what they have is not the original, so that any
problems introduced by others will not reflect on the original authors'
reputations.
Finally, any free program is threatened constantly by software patents. We
wish to avoid the danger that redistributors of a free program will
individually obtain patent licenses, in effect making the program
proprietary. To prevent this, we have made it clear that any patent must
be licensed for everyone's free use or not licensed at all.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. This License applies to any program or other work which contains a
notice placed by the copyright holder saying it may be distributed under
the terms of this General Public License. The "Program", below, refers to
any such program or work, and a "work based on the Program" means either
the Program or any derivative work under copyright law: that is to say, a
work containing the Program or a portion of it, either verbatim or with
modifications and/or translated into another language. (Hereinafter,
translation is included without limitation in the term "modification".)
Each licensee is addressed as "you".
Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope. The act of running
the Program is not restricted, and the output from the Program is covered
only if its contents constitute a work based on the Program (independent
of having been made by running the Program). Whether that is true depends
on what the Program does.
1. You may copy and distribute verbatim copies of the Program's source
code as you receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice and
disclaimer of warranty; keep intact all the notices that refer to this
License and to the absence of any warranty; and give any other recipients
of the Program a copy of this License along with the Program.
You may charge a fee for the physical act of transferring a copy, and you
may at your option offer warranty protection in exchange for a fee.
2. You may modify your copy or copies of the Program or any portion of it,
thus forming a work based on the Program, and copy and distribute such
modifications or work under the terms of Section 1 above, provided that
you also meet all of these conditions:
a) You must cause the modified files to carry prominent notices stating
that you changed the files and the date of any change.
b) You must cause any work that you distribute or publish, that in whole
or in part contains or is derived from the Program or any part thereof,
to be licensed as a whole at no charge to all third parties under the
terms of this License.
c) If the modified program normally reads commands interactively when
run, you must cause it, when started running for such interactive use in
the most ordinary way, to print or display an announcement including an
appropriate copyright notice and a notice that there is no warranty (or
else, saying that you provide a warranty) and that users may
redistribute the program under these conditions, and telling the user
how to view a copy of this License. (Exception: if the Program itself is
interactive but does not normally print such an announcement, your work
based on the Program is not required to print an announcement.)
These requirements apply to the modified work as a whole. If identifiable
sections of that work are not derived from the Program, and can be
reasonably considered independent and separate works in themselves, then
this License, and its terms, do not apply to those sections when you
distribute them as separate works. But when you distribute the same
sections as part of a whole which is a work based on the Program, the
distribution of the whole must be on the terms of this License, whose
permissions for other licensees extend to the entire whole, and thus to
each and every part regardless of who wrote it.
Thus, it is not the intent of this section to claim rights or contest your
rights to work written entirely by you; rather, the intent is to exercise
the right to control the distribution of derivative or collective works
based on the Program.
In addition, mere aggregation of another work not based on the Program
with the Program (or with a work based on the Program) on a volume of a
storage or distribution medium does not bring the other work under the
scope of this License.
3. You may copy and distribute the Program (or a work based on it, under
Section 2) in object code or executable form under the terms of Sections 1
and 2 above provided that you also do one of the following:
a) Accompany it with the complete corresponding machine-readable source
code, which must be distributed under the terms of Sections 1 and 2
above on a medium customarily used for software interchange; or,
b) Accompany it with a written offer, valid for at least three years, to
give any third party, for a charge no more than your cost of physically
performing source distribution, a complete machine-readable copy of the
corresponding source code, to be distributed under the terms of Sections
1 and 2 above on a medium customarily used for software interchange; or,
c) Accompany it with the information you received as to the offer to
distribute corresponding source code. (This alternative is allowed only
for noncommercial distribution and only if you received the program in
object code or executable form with such an offer, in accord with
Subsection b above.)
The source code for a work means the preferred form of the work for making
modifications to it. For an executable work, complete source code means
all the source code for all modules it contains, plus any associated
interface definition files, plus the scripts used to control compilation
and installation of the executable. However, as a special exception, the
source code distributed need not include anything that is normally
distributed (in either source or binary form) with the major components
(compiler, kernel, and so on) of the operating system on which the
executable runs, unless that component itself accompanies the executable.
If distribution of executable or object code is made by offering access to
copy from a designated place, then offering equivalent access to copy the
source code from the same place counts as distribution of the source code,
even though third parties are not compelled to copy the source along with
the object code.
4. You may not copy, modify, sublicense, or distribute the Program except
as expressly provided under this License. Any attempt otherwise to copy,
modify, sublicense or distribute the Program is void, and will
automatically terminate your rights under this License. However, parties
who have received copies, or rights, from you under this License will not
have their licenses terminated so long as such parties remain in full
compliance.
5. You are not required to accept this License, since you have not signed
it. However, nothing else grants you permission to modify or distribute
the Program or its derivative works. These actions are prohibited by law
if you do not accept this License. Therefore, by modifying or distributing
the Program (or any work based on the Program), you indicate your
acceptance of this License to do so, and all its terms and conditions for
copying, distributing or modifying the Program or works based on it.
6. Each time you redistribute the Program (or any work based on the
Program), the recipient automatically receives a license from the original
licensor to copy, distribute or modify the Program subject to these terms
and conditions. You may not impose any further restrictions on the
recipients' exercise of the rights granted herein. You are not responsible
for enforcing compliance by third parties to this License.
7. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot distribute
so as to satisfy simultaneously your obligations under this License and
any other pertinent obligations, then as a consequence you may not
distribute the Program at all. For example, if a patent license would not
permit royalty-free redistribution of the Program by all those who receive
copies directly or indirectly through you, then the only way you could
satisfy both it and this License would be to refrain entirely from
distribution of the Program.
If any portion of this section is held invalid or unenforceable under any
particular circumstance, the balance of the section is intended to apply
and the section as a whole is intended to apply in other circumstances.
It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any such
claims; this section has the sole purpose of protecting the integrity of
the free software distribution system, which is implemented by public
license practices. Many people have made generous contributions to the
wide range of software distributed through that system in reliance on
consistent application of that system; it is up to the author/donor to
decide if he or she is willing to distribute software through any other
system and a licensee cannot impose that choice.
This section is intended to make thoroughly clear what is believed to be a
consequence of the rest of this License.
8. If the distribution and/or use of the Program is restricted in certain
countries either by patents or by copyrighted interfaces, the original
copyright holder who places the Program under this License may add an
explicit geographical distribution limitation excluding those countries,
so that distribution is permitted only in or among countries not thus
excluded. In such case, this License incorporates the limitation as if
written in the body of this License.
9. The Free Software Foundation may publish revised and/or new versions of
the General Public License from time to time. Such new versions will be
similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the Program
specifies a version number of this License which applies to it and "any
later version", you have the option of following the terms and conditions
either of that version or of any later version published by the Free
Software Foundation. If the Program does not specify a version number of
this License, you may choose any version ever published by the Free
Software Foundation.
10. If you wish to incorporate parts of the Program into other free
programs whose distribution conditions are different, write to the author
to ask for permission. For software which is copyrighted by the Free
Software Foundation, write to the Free Software Foundation; we sometimes
make exceptions for this. Our decision will be guided by the two goals of
preserving the free status of all derivatives of our free software and of
promoting the sharing and reuse of software generally.
NO WARRANTY
11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
REPAIR OR CORRECTION.
12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES
ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT
LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES
SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE
WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN
ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
END OF TERMS AND CONDITIONS

24
scripts/fesco/sig-policy/UNLICENSE
View file

@ -1,24 +0,0 @@
This is free and unencumbered software released into the public domain.
Anyone is free to copy, modify, publish, use, compile, sell, or
distribute this software, either in source code form or as a compiled
binary, for any purpose, commercial or non-commercial, and by any
means.
In jurisdictions that recognize copyright laws, the author or authors
of this software dedicate any and all copyright interest in the
software to the public domain. We make this dedication for the benefit
of the public at large and to the detriment of our heirs and
successors. We intend this dedication to be an overt act of
relinquishment in perpetuity of all present and future rights to this
software under copyright law.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.
For more information, please refer to <https://unlicense.org>

2
scripts/pyproject.toml
View file

@ -1,2 +0,0 @@
[tool.black]
line-length = 100

129
scripts_new/CONVENTIONS.md
View file

@ -1,129 +0,0 @@
# Fedora Release Engineering Scripts - Naming Conventions
This document establishes standardized naming conventions for all scripts in the reorganized `scripts/` directory structure. These conventions improve consistency, readability, and maintainability.
## 🎯 Core Principles
1. **Consistency**: All scripts follow the same naming patterns
2. **Clarity**: Names clearly indicate script purpose and type
3. **Maintainability**: Predictable naming makes navigation easier
4. **Future-proof**: Standards that scale as the repository grows
## 📝 Naming Rules
### File Names
#### ✅ Use snake_case (underscores) for all script names
- **Correct**: `mass_rebuild.py`, `find_failures.py`, `stage_release.sh`
- **Incorrect**: `mass-rebuild.py`, `find-failures.py`, `stage-release.sh`
**Rationale**: Snake_case is more consistent with Python conventions and easier to work with programmatically.
#### ✅ Always include appropriate file extensions
- **Python scripts**: `.py`
- **Shell scripts**: `.sh`
- **Configuration files**: `.conf`, `.ini`, `.yaml`, etc.
- **Templates**: `.j2` (Jinja2), `.tmpl`, etc.
- **Executables**: No extension only if truly meant to be CLI tools
**Examples**:
- `mass_rebuild.py` (Python script)
- `create_repos.sh` (Shell script)
- `compose_config.conf` (Configuration)
- `email_template.j2` (Jinja2 template)
#### ✅ Use lowercase for all file and directory names
- **Correct**: `mass_rebuilds/`, `koji_import.py`
- **Incorrect**: `Mass_Rebuilds/`, `Koji_Import.py`
#### ✅ Use descriptive, action-oriented names
- **Good**: `find_failed_builds.py`, `retire_orphaned_packages.py`
- **Poor**: `script.py`, `util.py`, `helper.sh`
### Directory Names
#### ✅ Use lowercase with hyphens for multi-word directory names
- **Correct**: `mass-rebuilds/`, `quality-assurance/`, `release-process/`
- **Incorrect**: `mass_rebuilds/`, `qualityAssurance/`, `release_process/`
**Rationale**: Hyphens in directory names are more readable in URLs and file paths, while underscores in filenames work better with import statements and variable names.
## 📋 Specific Patterns
### Script Categories
#### Build & Compose Scripts
```
build_current.py # Build current compose
build_previous.py # Build previous release
koji_import.py # Import builds to Koji
koji_compare.py # Compare Koji builds
compose_info_builder.py # Build compose metadata
```
#### Package Management Scripts
```
find_failures.py # Find failed builds
block_retired.py # Block retired packages
retire_packages.py # Retire package list
orphan_packages.py # Orphan package list
```
#### Release Process Scripts
```
mass_rebuild.py # Mass rebuild orchestration
stage_release.sh # Stage release for publication
sign_packages.py # Sign package batches
create_torrents.py # Generate release torrents
```
#### Quality Assurance Scripts
```
check_builds.py # Check build health
test_compose.py # Test compose quality
validate_paths.py # Validate upgrade paths
```
## 🔄 Migration Guidelines
### Renaming Existing Scripts
When migrating scripts to the new structure, rename according to these patterns:
#### Before → After
```bash
# Hyphens to underscores
mass-rebuild.py → mass_rebuild.py
check-latest-build.py → check_latest_build.py
find-bad-builds.py → find_bad_builds.py
# Add missing extensions
mashcompose → mash_compose.py
buildepelbeta → build_epel_beta.sh
torrent-generator → generate_torrents.py
# Improve clarity
build-current.py → build_current_compose.py
clean-amis.py → clean_ami_images.py
get_retired_packages.sh → find_retired_packages.sh
```
## ❓ FAQ
**Q: Should configuration files follow the same naming?**
A: Yes, but use appropriate extensions (.conf, .yaml, .ini) instead of .py/.sh.
**Q: How do we handle template files?**
A: Use clear extensions like `.j2` for Jinja2, `.tmpl` for generic templates.
**Q: Do you have any helper where we can check about renaming?**
A: Yes, please consider looking into rename_script_helper, and validate_conventions scripts from the misc directory.
## 📚 References
- [Fedora Packaging Guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/)
---
**Maintainer**: Fedora Release Engineering Team
**Questions**: File issues in [fedora-releng repository](https://pagure.io/releng)

32
scripts_new/README.md
View file

@ -1,32 +0,0 @@
# Fedora Release Engineering Scripts (New Structure)
This directory contains the reorganized structure for Fedora Release Engineering scripts, designed for better maintainability and clarity.
## 📁 Directory Structure
- **[builds/](builds/)** - Build system tools and compose generation
- **[release-process/](release-process/)** - Release workflow orchestration and management
- **[packages/](packages/)** - Package lifecycle and maintenance tools
- **[quality-assurance/](quality-assurance/)** - Testing, validation, and quality tools
- **[sync/](sync/)** - Repository synchronization utilities
- **[infrastructure/](infrastructure/)** - Infrastructure management and governance
- **[utilities/](utilities/)** - General-purpose helper scripts
- **[misc/](misc/)** - Uncategorized or legacy scripts
## 🔧 Migration Status
**⚠️ This is the new directory structure under development.**
Scripts are being migrated from the flat `scripts/` directory to this organized structure. See [CONVENTIONS.md](CONVENTIONS.md) for naming standards and migration guidelines.
## 📖 Getting Started
1. Review [CONVENTIONS.md](CONVENTIONS.md) for naming and organization standards
2. Check individual directory READMEs for specific tool documentation
3. Use the directory structure to quickly locate relevant scripts
## 🔗 Related
- Original scripts: `../scripts/` (legacy structure)
- Conventions: [CONVENTIONS.md](CONVENTIONS.md)
- Issues: [fedora-releng repository](https://pagure.io/releng)

14
scripts_new/builds/README.md
View file

@ -1,14 +0,0 @@
# Build System Tools
Scripts for managing Fedora builds, composes, and Koji operations.
## Subdirectories
- **[compose/](compose/)** - Compose generation and validation tools
- **[koji/](koji/)** - Koji build system utilities and maintenance
## Purpose
This directory contains tools that directly interact with the build infrastructure to create, manage, and validate Fedora package builds and composes.
**Status**: 🚧 Migration in progress

14
scripts_new/builds/compose/README.md
View file

@ -1,14 +0,0 @@
# Compose Tools
Scripts for generating, validating, and managing Fedora composes.
## Purpose
Tools that handle the creation and validation of Fedora release composes, including metadata generation and quality checks.
**Examples of scripts that will be migrated here:**
- Compose building and validation
- Metadata generation
- Compose health checking
**Status**: 🚧 Awaiting migration

170
scripts_new/builds/compose/build_composeinfo.py
View file

@ -1,170 +0,0 @@
#!/usr/bin/python3
# Copyright (C) 2013 Red Hat Inc.
# SPDX-License-Identifier: GPL-2.0+
import os, os.path
import sys
from argparse import ArgumentParser
import logging
import configparser
import pprint
#.composeinfo
#[product]
#family = Fedora
#name = Fedora-17-Alpha-RC1
#version = 17
#variants = Fedora
#
#[variant-Fedora]
#variants =
#arches = x86_64,i386
#id = Fedora
#name = Fedora
#type = variant
#
#[variant-Fedora.x86_64]
#arch = x86_64
#debuginfo = x86_64/debug
#os_dir = x86_64/os
#sources = source/SRPMS
def filelists(d, fname=None):
files = []
for file in os.listdir(d):
fulldir = os.path.join(d, file)
#"and 'Modular' not in fulldir" is for fixing https://pagure.io/releng/issue/7488
#Once the tools that use .composeinfo gets updated to support repos with no images
#we can remove the conditional.
if os.path.isdir(fulldir) and 'Modular' not in fulldir:
flist = [os.path.join(fulldir, x) for x in os.listdir(fulldir) \
if os.path.isfile(os.path.join(fulldir, x)) and x == fname]
files.extend(flist)
files.extend(filelists(fulldir, fname=fname))
return files
def buildCompose(composePath, treeinfos, name):
composeFile = os.path.join(composePath, '.composeinfo')
if os.path.exists(composeFile):
logging.critical(".composeinfo already exists under %s", composeFile)
return 5
variants = dict()
composeInfoParser = configparser.ConfigParser()
treeInfoParser = configparser.ConfigParser()
for treeinfo in treeinfos:
logging.debug("treeinfo: %s", treeinfo)
# treepath is relative from the composepath.
treepath = treeinfo.replace('/.treeinfo','').replace(composePath,'./')
try:
treeInfoParser.read(treeinfo)
except configparser.MissingSectionHeaderError as e:
logging.critical('%s is not parsable: %s', treeinfo, e)
return 10
variant = treeInfoParser.get('general','variant') or 'Fedora'
version = treeInfoParser.get('general','version')
family = treeInfoParser.get('general','family')
arch = treeInfoParser.get('general','arch')
if not name:
name = '%s-%s' % (family, version)
archSection = 'variant-%s.%s' % (variant, arch)
composeInfoParser.add_section(archSection)
composeInfoParser.set(archSection, 'arch', arch)
composeInfoParser.set(archSection, 'os_dir', os.path.normpath(treepath))
# Check for debuginfo repo
debuginfo = os.path.normpath(os.path.join(treepath,'../debug'))
if os.path.exists(os.path.join(composePath, debuginfo, 'repodata')):
composeInfoParser.set(archSection, 'debuginfo', debuginfo)
# Check for sources repo
sources = os.path.normpath(os.path.join(treepath,'../../source/SRPMS'))
if os.path.exists(os.path.join(composePath, sources, 'repodata')):
composeInfoParser.set(archSection, 'sources', sources)
# Fedora only has one variant, but in case that ever changes we build
# the dictionary. Maybe spins would be considered variants?
if variant not in variants:
variants[variant] = dict(arches = [],
id = variant,
name = variant,
type = 'variant')
# We do have mutliple arches in Fedora.
variants[variant]['arches'].append(arch)
logging.debug('\n%s', pprint.pformat(variants))
# Write out our main product description
composeInfoParser.add_section('product')
composeInfoParser.set('product', 'family', family)
composeInfoParser.set('product', 'name', name)
composeInfoParser.set('product', 'version', version)
composeInfoParser.set('product', 'variants', ','.join(variants.keys()))
# Write out each variant
for variant in variants.values():
section = 'variant-%s' % variant['id']
composeInfoParser.add_section(section)
composeInfoParser.set(section, 'arches', ','.join(variant['arches']))
composeInfoParser.set(section, 'id', variant['id'])
composeInfoParser.set(section, 'name', variant['name'])
composeInfoParser.set(section, 'type', variant['type'])
# Fedora doesn't have sub-variants.
composeInfoParser.set(section, 'variants', '')
composeInfoParser.write(open(composeFile,'w'))
def main():
parser = ArgumentParser(usage = '%(prog)s [options] Directory')
parser.add_argument("-n", "--name",
default=None,
help="Alternate name to use, otherwise we use family+version")
parser.add_argument("-v", "--debug",
action='store_true',
default=False,
help="show debug messages")
parser.add_argument("-q", "--quiet",
action='store_true',
default=False,
help="less messages")
args, compose = parser.parse_known_args()
LOG_FORMAT = '%(asctime)s - %(levelname)s - %(filename)s - ' \
'%(funcName)s:%(lineno)s - %(message)s'
if args.debug:
LOG_LEVEL = logging.DEBUG
elif args.quiet:
LOG_LEVEL = logging.CRITICAL
else:
LOG_LEVEL = logging.INFO
LOG_FORMAT = '%(message)s'
formatter = logging.Formatter(LOG_FORMAT)
stdout_handler = logging.StreamHandler(sys.stdout)
stdout_handler.setFormatter(formatter)
logger = logging.getLogger('')
logger.addHandler(stdout_handler)
logger.setLevel(LOG_LEVEL)
if len(compose) == 0:
logging.critical("No directory specified")
parser.print_help()
return 1
elif len(compose) > 1:
logging.critical("Only specify one directory")
parser.print_help()
return 2
if not os.path.isdir(compose[0]):
logging.critical("%s is not a directory", compose[0])
parser.print_help()
return 3
treeinfos = filelists(compose[0], '.treeinfo')
if not treeinfos:
logging.critical("No .treeinfo(s) found under %s", compose[0])
return 4
return buildCompose(compose[0], treeinfos, args.name)
if __name__ == '__main__':
sys.exit(main())

14
scripts_new/builds/koji/README.md
View file

@ -1,14 +0,0 @@
# Koji Utilities
Scripts for interacting with and managing the Koji build system.
## Purpose
Tools for Koji build operations, import/export, monitoring, and maintenance tasks.
**Examples of scripts that will be migrated here:**
- Koji build import/export
- Build monitoring and reporting
- Koji maintenance and cleanup
**Status**: 🚧 Awaiting migration

115
scripts_new/builds/koji/koji_build_srpm.py
View file

@ -1,115 +0,0 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
#
# Copyright (C) 2011-2013 Red Hat, Inc.
# SPDX-License-Identifier: GPL-2.0
# Author: Dan Horák <dhorak@redhat.com>
#
# Grab source rpm for a build from primary koji and build it in secondary koji
#
import os
import sys
import koji
import logging
import urlgrabber.grabber as grabber
import urlgrabber.progress as progress
import time
import random
import string
import argparse
# get architecture, tag/target and build from command line
parser = argparse.ArgumentParser(description='Build srpm from primary koji in secondary koji.')
parser.add_argument("--keytab", help="specify a Kerberos keytab to use")
parser.add_argument("--principal", help="specify a Kerberos principal to use")
parser.add_argument('--scratch', action='store_true', help='scratch build')
parser.add_argument('--verbose', action='store_true', help='enables additional output, overrides --quiet')
parser.add_argument('--quiet', action='store_true', help='suppresses non error related output')
parser.add_argument('arch', help='secondary arch koji where to build srpms')
parser.add_argument('tag', help='build to this tag')
parser.add_argument('build', nargs='+', help='NVR')
args = parser.parse_args()
LOCALKOJIHUB = args.arch
REMOTEKOJIHUB = 'fedora'
PACKAGEURL = 'http://kojipkgs.fedoraproject.org/'
# Should probably set these from a koji config file
# Should only be used for ssl login
SERVERCA = os.path.expanduser('~/.fedora-server-ca.cert')
CLIENTCA = os.path.expanduser('~/.fedora-upload-ca.cert')
CLIENTCERT = os.path.expanduser('~/.fedora.cert')
session_opts = {}
session_opts['krbservice'] = 'host'
session_opts['krb_rdns'] = False
if args.verbose:
loglevel = logging.DEBUG
elif args.quiet:
loglevel = logging.ERROR
else:
loglevel = logging.INFO
logging.basicConfig(format='%(levelname)s: %(message)s',
level=loglevel)
def _unique_path(prefix):
"""Create a unique path fragment by appending a path component
to prefix. The path component will consist of a string of letter and numbers
that is unlikely to be a duplicate, but is not guaranteed to be unique."""
# Use time() in the dirname to provide a little more information when
# browsing the filesystem.
# For some reason repr(time.time()) includes 4 or 5
# more digits of precision than str(time.time())
return '%s/%r.%s' % (prefix, time.time(),
''.join([random.choice(string.ascii_letters) for i in range(8)]))
# setup the koji session
logging.info('Setting up koji session')
local_koji_module = koji.get_profile_module(LOCALKOJIHUB)
remote_koji_module = koji.get_profile_module(REMOTEKOJIHUB)
localkojisession = local_koji_module.ClientSession(local_koji_module.config.server, session_opts)
remotekojisession = remote_koji_module.ClientSession(remote_koji_module.config.server)
if os.path.isfile(CLIENTCERT):
localkojisession.ssl_login(CLIENTCERT, CLIENTCA, SERVERCA)
else:
if args.keytab and args.principal:
localkojisession.gssapi_login(principal=args.principal, keytab=args.keytab)
else:
localkojisession.gssapi_login()
pg = progress.TextMeter()
for build in args.build:
buildinfo = remotekojisession.getBuild(build)
logging.debug("build=%s" % (buildinfo))
if buildinfo == None:
logging.critical("build %s doesn't exist" % (build))
break
fname = "%s.src.rpm" % buildinfo['nvr']
url = "%s/packages/%s/%s/%s/src/%s" % (PACKAGEURL, buildinfo['package_name'], buildinfo['version'], buildinfo['release'], fname)
if not os.path.isfile(fname):
file = grabber.urlgrab(url, progress_obj = pg, text = "%s" % (fname))
serverdir = _unique_path('cli-build')
logging.info("uploading %s ..." % (build))
localkojisession.uploadWrapper(fname, serverdir, blocksize=65536)
source = "%s/%s" % (serverdir, fname)
if args.scratch:
opts = {}
opts['scratch'] = True
else:
opts = None
localkojisession.build(source, args.tag, opts=opts, priority=2)
logging.info("submitted build: %s" % buildinfo['nvr'])

103
scripts_new/builds/koji/koji_prune_old_builds.py
View file

@ -1,103 +0,0 @@
#!/usr/bin/python
#
# prune-tag.py - A utility to prune all but the latest build in a given tag.
#
# Copyright (C) 2009-2013 Red Hat, Inc.
# SPDX-License-Identifier: GPL-2.0
#
# Authors:
# Jesse Keating <jkeating@redhat.com>
#
# This program requires koji installed, as well as configured.
import os
import argparse
import sys
import koji
import logging
status = 0
builds = {}
untag = []
loglevel = ''
# Setup a dict of our key names as sigul knows them to the actual key ID
# that koji would use. We should get this from sigul somehow.
# Create a parser to parse our arguments
parser = argparse.ArgumentParser(usage = '%(prog)s [options] tag')
parser.add_argument('-v', '--verbose', action='count', default=0,
help='Be verbose, specify twice for debug')
parser.add_argument('-n', '--dry-run', action='store_true', default=False,
help='Perform a dry run without untagging')
parser.add_argument('-p', '--koji-profile', default="fedora",
help='Select a koji profile to use')
KOJIHUB = args.koji_profile
# Get our options and arguments
args, extras = parser.parse_known_args()
if args.verbose <= 0:
loglevel = logging.WARNING
elif args.verbose == 1:
loglevel = logging.INFO
else: # options.verbose >= 2
loglevel = logging.DEBUG
logging.basicConfig(format='%(levelname)s: %(message)s',
level=loglevel)
# Check to see if we got any arguments
if not extras:
parser.print_help()
sys.exit(1)
tag = extras[0]
# setup the koji session
logging.info('Setting up koji session')
koji_module = koji.get_profile_module(KOJIHUB)
kojisession = koji_module.ClientSession(koji_module.config.server)
if not kojisession.gssapi_login():
logging.error('Unable to log into koji')
sys.exit(1)
# Get a list of tagged packages
logging.info('Getting builds from %s' % tag)
tagged = kojisession.listTagged(tag)
logging.debug('Got %s builds' % len(builds))
# Sort builds by package
for b in tagged:
builds.setdefault(b['package_name'], []).append(b)
# Find the packages with multiple builds
for pkg in sorted(builds.keys()):
if len(builds[pkg]) > 1:
logging.debug('Leaving newest build %s' % builds[pkg][0]['nvr'])
for build in builds[pkg][1:]:
logging.debug('Adding %s to untag list' % build['nvr'])
untag.append(build['nvr'])
# Now untag all the builds
logging.info('Untagging %s builds' % len(untag))
if not args.dry_run:
kojisession.multicall = True
for build in untag:
if not args.dry_run:
kojisession.untagBuildBypass(tag, build, force=True)
logging.debug('Untagging %s' % build)
if not args.dry_run:
results = kojisession.multiCall()
for build, result in zip(untag, results):
if isinstance(result, dict):
logging.error('Error tagging %s' % build)
if result['traceback']:
logging.error(' ' + result['traceback'][-1])
status = 1
logging.info('All done, pruned %s builds.' % len(untag))
sys.exit(status)

81
scripts_new/builds/koji/koji_task_report.py
View file

@ -1,81 +0,0 @@
#!/usr/bin/python3
# -*- coding: utf-8 -*-
#
# Copyright 2017, 2021 Red Hat, Inc.
# License: GPLv2
# Author: Dan Horák <dhorak@redhat.com>
#
# Get statistics about waiting for builders from koji
#
# usage: koji-task-report.py [-h] [--channel CHANNEL] [--profile PROFILE] [--method METHOD] arch datefrom [dateto]
# ./koji-task-report.py s390x yesterday
# ./koji-task-report.py --channel compose --method runroot s390x today
#
import argparse
import koji
from datetime import datetime, timedelta
# get the date for tasks check from command line
parser = argparse.ArgumentParser()
parser.add_argument("arch", help="specific architecture to check for")
parser.add_argument("--channel", help="specific channel to check for", default="default")
parser.add_argument("--profile", help="koji profile (fedora, brew, stream)", default="fedora")
parser.add_argument("--method", help="method (BuildArch, runroot, ...)", default="buildArch")
parser.add_argument("datefrom", help="select tasks started since")
parser.add_argument("dateto", help="select tasks started till", nargs="?", default="now")
args = parser.parse_args()
koji_module = koji.get_profile_module(args.profile)
session = koji_module.ClientSession(koji_module.config.server)
channel = args.channel
channelinfo = session.getChannel(channel)
arches = []
if args.arch.find(',') > 0:
arches = args.arch.split(',')
else:
arches.append(args.arch)
opts = {}
opts['channel_id'] = channelinfo['id']
opts['createdAfter'] = args.datefrom
opts['createdBefore'] = args.dateto
opts['method'] = args.method
opts['arch'] = arches
# we want finished tasks
opts['state'] = [koji.TASK_STATES['CLOSED'], koji.TASK_STATES['FAILED']]
print(("\nReading completed '%s' Koji tasks in channel '%s' between %s and %s ...") % (args.profile, channel, opts['createdAfter'], opts['createdBefore']))
tasks = session.listTasks(opts)
total_waited = timedelta()
max_waited = timedelta()
for task in tasks:
created = datetime.fromisoformat(task['create_time'])
started = datetime.fromisoformat(task['start_time'])
waited = started - created
total_waited += waited
if waited > max_waited:
max_waited = waited
print(("%s,%s") % (task['create_time'], waited))
# print(("task id=%s\tcreated=%s\tstarted=%s") % (task['id'], task['create_time'], task['start_time']))
if len(tasks) > 0:
print(("%s tasks, average waiting %s, maximum waiting %s") % (len(tasks), (total_waited/len(tasks)), max_waited))
else:
print("no tasks found")
opts['state'] = [koji.TASK_STATES['OPEN']]
print(("\nReading running '%s' Koji tasks in channel '%s' between %s and %s ...") % (args.profile, channel, opts['createdAfter'], opts['createdBefore']))
tasks = session.listTasks(opts)
print(("%s tasks running") % (len(tasks)))
opts['state'] = [koji.TASK_STATES['FREE']]
print(("\nReading waiting '%s' Koji tasks in channel '%s' between %s and %s ...") % (args.profile, channel, opts['createdAfter'], opts['createdBefore']))
tasks = session.listTasks(opts)
print(("%s tasks waiting") % (len(tasks)))

56
scripts_new/builds/koji/utils/find_bad_builds.py
View file

@ -1,56 +0,0 @@
#!/usr/bin/python
# Copyright (C) 2013 Red Hat Inc,
# SPDX-License-Identifier: GPL-2.0+
#
# template for finding builds that meet some time/buildroot component critera.
# Edit to suit.
from __future__ import print_function
import koji
kojisession = koji.ClientSession('http://koji.fedoraproject.org/kojihub')
kojisession.gssapi_login()
potentials = []
tocheck = []
needbuild = []
reallyneedbuild = []
f8builds = kojisession.listTagged('dist-f8', inherit=True, latest=True)
for build in f8builds:
if build['creation_time'] > '2007-06-12 04:01:15.000000':
potentials.append(build)
for build in potentials:
if build['creation_time'] < '2007-07-31 02:10:19.000000':
tocheck.append(build)
for build in tocheck:
for task in kojisession.getTaskChildren(build['task_id']):
if build in needbuild:
continue
if task['method'] == 'buildArch':
for rootid in kojisession.listBuildroots(taskID=task['id']):
for pkg in kojisession.listRPMs(componentBuildrootID=rootid['id']):
if pkg['name'] == 'binutils':
if pkg['version'] == '2.17.50.0.16':
if not build in needbuild:
needbuild.append(build)
elif pkg['version'] == '2.17.50.0.17' and pkg['release'] < '7':
if not build in needbuild:
needbuild.append(build)
else:
print("%s had binutils, but it was %s" % (build['nvr'], pkg['nvr']))
rebuildnames = []
for build in needbuild:
for rpm in kojisession.listBuildRPMs(build['nvr']):
if rpm['arch'] == 'ppc':
if not build in reallyneedbuild:
reallyneedbuild.append(build)
rebuildnames.append(build['name'])
rebuildnames.sort()
for build in rebuildnames:
print(build)

14
scripts_new/infrastructure/README.md
View file

@ -1,14 +0,0 @@
# Infrastructure Management
Scripts for managing Fedora infrastructure and governance processes.
## Subdirectories
- **[messaging/](messaging/)** - Fedora messaging and notification systems
- **[fesco/](fesco/)** - FESCo governance and policy tools
## Purpose
Tools that support Fedora infrastructure operations and governance processes.
**Status**: 🚧 Migration in progress

0
scripts_new/infrastructure/fesco/.gitignore vendored
View file

1
scripts_new/infrastructure/fesco/bug-filing
View file

@ -1 +0,0 @@
release-process/bug-filing/

1
scripts_new/infrastructure/fesco/follow_policy.py
View file

@ -1 +0,0 @@
scripts_new/release-process/bug-filing/ftbfs/follow_policy.py

1
scripts_new/infrastructure/fesco/orphaned
View file

@ -1 +0,0 @@
packages/orphaned

224
scripts_new/infrastructure/fesco/sig-policy/sig_policy.py
View file

@ -1,224 +0,0 @@
#!/usr/bin/python3
# sig_policy.py
# =============
#
# This script enacts the FESCo SIG Policy as documented here:
# https://docs.fedoraproject.org/en-US/fesco/SIG_policy/
#
# Author: Fabio Valentini <decathorpe@gmail.org>
# SPDX-License-Identifier: Unlicense
import argparse
import functools
import os
import sys
import fedrq.config
import requests
from fedrq.backends.base import RepoqueryBase
# (namespace, name of SIG group, ACL, package name filter)
POLICY = [
# Flatpak SIG: https://pagure.io/fesco/fesco-docs/pull-request/72
("flatpaks", "flatpak-sig", "commit", lambda x: x),
# Go SIG: https://pagure.io/fesco/fesco-docs/pull-request/68
("rpms", "go-sig", "commit", lambda x: x in go_packages()),
# Haskell SIG: https://pagure.io/fesco/fesco-docs/pull-request/89
("rpms", "haskell-lang-sig", "commit", lambda x: x.startswith("ghc-")),
# R SIG: https://pagure.io/fesco/fesco-docs/pull-request/69
("rpms", "r-maint-sig", "commit", lambda x: x.startswith("R-") or x in r_packages()),
# Rust SIG: https://pagure.io/fesco/fesco-docs/pull-request/66
("rpms", "rust-sig", "commit", lambda x: x.startswith("rust-")),
]
PAGURE_DIST_GIT_DATA_URL = "https://src.fedoraproject.org/extras/pagure_bz.json"
VALID_ACLS = ["ticket", "commit", "admin"]
@functools.cache
def get_rq() -> RepoqueryBase:
"""
Return a RepoqueryBase object with the rawhide buildroot repositories
"""
return fedrq.config.get_config().get_rq("rawhide", "@buildroot")
@functools.cache
def go_packages() -> set[str]:
rq = get_rq()
query = rq.query(
requires=rq.query(name=["golang", "golang-bin", "go-rpm-macros"]),
# Only BuildRequires are covered by the policy
arch="src",
)
packages = {package.name for package in query}
return packages
@functools.cache
def r_packages() -> set[str]:
rq = get_rq()
query = rq.query(
requires__glob="libR.so*",
# Only Requires are covered by the policy
arch__neq="src",
)
packages = {package.source_name for package in query}
# add "R" which does not link with "libR.so" itself
packages.add("R")
return packages
def get_package_data() -> dict[str, list[str]]:
"""
Download the latest cached mapping from source package name -> list of
(co)maintainers from pagure-dist-git.
Raises an exception if the HTTP GET request failed, or if the data is not
valid JSON in the expected format.
"""
ret = requests.get(PAGURE_DIST_GIT_DATA_URL)
ret.raise_for_status()
data = ret.json()
return data
def add_package_acl(namespace: str, package: str, group: str, acl: str, token: str):
"""
Send an HTTP POST request to the pagure API endpoint for modifying ACLs on
a project.
Raises an exception if an HTTP error status was returned, or if the network
request failed for other reasons.
"""
if acl not in VALID_ACLS:
raise ValueError(f"Not a valid ACL: {acl}")
url = f"https://src.fedoraproject.org/api/0/{namespace}/{package}/git/modifyacls"
payload = {
"user_type": "group",
"name": group,
"acl": acl,
}
headers = {
"Authorization": f"token {token}",
"Content-Type": "application/x-www-form-urlencoded",
}
response = requests.post(url, data=payload, headers=headers)
response.raise_for_status()
def main() -> int:
cli = argparse.ArgumentParser()
cli.add_argument(
"--dry-run",
"-n",
dest="dry",
action="store_true",
help="print results but do not modify any data",
)
cli.add_argument(
"--api-token",
dest="token",
action="store",
default=None,
help="API token for src.fedoraproject.org (overrides PAGURE_API_TOKEN)",
)
args = cli.parse_args()
token = args.token or os.environ.get("PAGURE_API_TOKEN")
if not token:
print("PAGURE_API_TOKEN environment variable not set.", file=sys.stderr)
return 1
try:
package_data = get_package_data()
except IOError as ex:
print("Failed to fetch data from pagure-dist-git:", file=sys.stderr)
print(ex, file=sys.stderr)
return 1
# keep track of failed requests
failures = dict()
for (namespace, group, acl, filtr) in POLICY:
print(f"Processing group: {group}")
packages = package_data[namespace]
# keep track of candidate packages
candidates = []
for (package, maintainers) in packages.items():
# check if the package matches the filter set by the policy
if not filtr(package):
continue
# check if the package is already retired on all branches
if maintainers == ["orphan"]:
continue
# check if the package already has the group as co-maintainer
# FIXME: this cannot check whether the ACL is present but too low
if f"@{group}" not in maintainers:
candidates.append(package)
if not candidates:
print(f"No pending actions for group {group!r}.")
print()
continue
# keep track of failed requests
failed = []
for candidate in candidates:
print(f"- add {group!r} with {acl!r} ACL to '{namespace}/{candidate}'")
if not args.dry:
try:
add_package_acl(namespace, candidate, group, acl, token)
except Exception as ex:
print(ex, file=sys.stderr)
failed.append(candidate)
if failed:
failures[group] = failed
print()
if not failures:
print("Finished successfully.")
return 0
print("Finished with errors:")
for (group, failed) in failures:
for package in failed:
print(f"- failed to add {group!r} group to package {package!r}")
return 1
if __name__ == "__main__":
try:
exit(main())
except KeyboardInterrupt:
print("Cancelled.")
exit(0)
except Exception as e:
print(e, file=sys.stderr)
exit(1)

40
scripts_new/infrastructure/messaging/fedora_messaging.sh
View file

@ -1,40 +0,0 @@
# fedora_messaging.sh
#
# This is a bash shell script that is meant to be sourced by other scripts
# and aims to deliver functions common to sending messages to
# fedora-messaging[0] in rel-eng shell scripts.
#
# [0] - https://github.com/fedora-infra/fedora-messaging
# NOTES:
# FIXME: This might be obsolete:
# - Scripts that source this should define at least the following:
# FEDMSG_MODNAME
# FEDMSG_CERTPREFIX
#
# Example:
#
# FEDMSG_MODNAME="compose"
# FEDMSG_CERTPREFIX="bodhi"
# source ./scripts_new/infrastructure/messaging/fedora_messaging.sh
#
# fedmsg_json_start=$(printf '{"log": "start", "branch": "f24", "arch": "x86_64"}')
# send_fedora_message "${fedmsg_json_start}" f24 start
# This uses the new fedora-messaging bus:
LOGGER=releng/scripts_new/infrastructure/messaging/fedora_messaging_logger.py
function send_fedora_message()
{
jsoninput="${1}"
dist="${2}"
topic="${3}"
echo ${jsoninput} | $LOGGER \
--cert-prefix ${FEDMSG_CERTPREFIX} \
--modname ${FEDMSG_MODNAME} \
--topic ".${dist}.${topic}" \
--json-input
}

108
scripts_new/infrastructure/messaging/fedora_messaging_logger.py
View file

@ -1,108 +0,0 @@
#!/usr/bin/python3
# fedora_messaging replacement for fedmsg-logger
# implemented are only features required by releng/scripts_new/infrastructure/messaging/fedora_messaging.sh
# Copyright (c) 2019 Red Hat, Inc.
#
# Authors:
# Karsten Hopp <karsten@redhat.com>
import argparse
import json
from fedora_messaging import api, message
parser = argparse.ArgumentParser(description="Process commandline parameters.")
parser.add_argument(
"--topic-prefix",
dest="topicprefix",
default="",
help="Prefix for the topic of each message sent.",
)
parser.add_argument(
"--modname",
dest="modname",
default="",
help="More control over the topic. Think org.fp.MODNAME.TOPIC.",
)
parser.add_argument(
"--message", dest="message", default="", help="The message to send."
)
parser.add_argument(
"--topic", dest="topic", default="", help="Think org.fedoraproject.dev.logger.TOPIC"
)
parser.add_argument(
"--json-input",
dest="jsoninput",
action="store_true",
help="Take each line of input as JSON.",
)
# unused options for backwards compatibility with fedmsg-logger scripts:
parser.add_argument(
"--cert-prefix",
dest="certprefix",
default="",
help="Specify a different cert from /etc/fedora-messaging/ (unused)",
)
parser.add_argument(
"--io-threads",
dest="io-threads",
type=int,
default=1,
help="Number of io threads for 0mq to use (unused)",
)
parser.add_argument(
"--config-filename", dest="config-filename", default="", help="Config file to use."
)
parser.add_argument(
"--print-config",
dest="print-config",
action="store_true",
help="Simply print out the configuration and exit. No action taken. (unused)",
)
parser.add_argument(
"--timeout",
dest="timeout",
type=int,
default=0,
help="Timeout in seconds for any blocking zmq operations. (unused)",
)
parser.add_argument(
"--high-water-mark",
dest="high-water-mark",
type=int,
default=0,
help="Limit on the number of messages in the queue before blocking. (unused)",
)
parser.add_argument(
"--linger",
dest="linger",
type=int,
default=0,
help="Number of milliseconds to wait before timing out connections. (unused)",
)
args = parser.parse_args()
print(args)
print(args.topic)
if args.jsoninput:
import sys
msgstring=""
for line in sys.stdin:
msgstring += line
body = json.loads(msgstring)
elif args.message:
msgstring='{"log":"start","msg":"' +args.message+ '"}'
body = json.loads(msgstring)
wholetopic=args.topicprefix+args.modname+args.topic
msg = message.Message(
topic=wholetopic,
headers={u"niceness": u"very"},
body=body,
)
api.publish(msg)

0
scripts_new/infrastructure/messaging/governance/.gitignore vendored
View file

9
scripts_new/misc/README.md
View file

@ -1,9 +0,0 @@
# Miscellaneous Scripts
Uncategorized scripts and legacy tools that don't fit into other categories.
## Purpose
Temporary location for scripts that need further analysis for proper categorization, or legacy scripts with unclear current status.
**Status**: 🚧 Triage in progress

24
scripts_new/misc/close_old_ftbfs_bugz.sh
View file

@ -1,24 +0,0 @@
#!/bin/bash
set -e
set -u
set -o pipefail
tracker=${1:-1803234}
dist=${1:-fc33}
for line in $(bugzilla query --blocked $tracker --status NEW --outputformat "%{id}@%{component}@%{creation_time}"); do
line=( ${line/@/ } )
bug=${line[0]}
line=( ${line[1]/@/ } )
component=${line[0]}
echo $component >&2
creation_time=${line[1]}
builds=$(koji list-builds --package $component --after "$creation_time" --state=COMPLETE --quiet) || continue
# XXX any better way to filter builds by release?
builds=$(echo "$builds" | grep "$dist " | cut -f1 -d' ' | tr '\n' ' ' || true)
if ! [ -z "$builds" ]; then
echo "$builds"
bugzilla modify --status CLOSED --close NEXTRELEASE --comment "The following builds were made after this report was opened: $builds" $bug
fi
done

13
scripts_new/misc/rename_script_helper.sh
View file

@ -1,13 +0,0 @@
#!/bin/bash
# rename_script.sh - Helper for batch renaming during migration
# Convert hyphens to underscores in filename
rename_file() {
local old_name="$1"
local new_name=$(echo "$old_name" | sed 's/-/_/g')
if [[ "$old_name" != "$new_name" ]]; then
echo "Renaming: $old_name → $new_name"
git mv "$old_name" "$new_name"
fi
}

22
scripts_new/misc/validate_conventions.py
View file

@ -1,22 +0,0 @@
#!/usr/bin/env python3
"""validate_conventions.py - Check adherence to naming conventions"""
import os
import re
from pathlib import Path
def validate_filename(filepath):
"""Validate a single file against conventions"""
name = filepath.name
# Check for hyphens in filenames (should use underscores)
if '-' in name and not filepath.is_dir():
return f"❌ {filepath}: Use underscores not hyphens in filenames"
# Check for missing extensions on scripts
if filepath.suffix == '' and not filepath.is_dir():
first_line = filepath.read_text(errors='ignore').split('\n')[0]
if first_line.startswith('#!'):
return f"⚠️ {filepath}: Script missing file extension"
return None

16
scripts_new/packages/README.md
View file

@ -1,16 +0,0 @@
# Package Management Tools
Scripts for managing package lifecycles and dist-git operations.
## Subdirectories
- **[lifecycle/](lifecycle/)** - Package retirement, blocking, and lifecycle management
- **[maintenance/](maintenance/)** - Package health monitoring and failure detection
- **[orphaned/](orphaned/)** - Orphaned package processing and retirement
- **[distgit/](distgit/)** - Dist-git repository operations
## Purpose
Tools that manage packages throughout their lifecycle in Fedora, from creation through retirement.
**Status**: 🚧 Migration in progress

0
scripts_new/packages/distgit/.gitignore vendored
View file

0
scripts_new/packages/distgit/__init__.py
View file

272
scripts_new/packages/distgit/distgit_branch_unused.py
View file

@ -1,272 +0,0 @@
#!/usr/bin/python3
"""This script checks if a branch may be deleted.
1. A branch may be removed safely when, for all commits in that branch
not reachable from other branches, there are no complete koji builds.
Examples:
-\------A---\
\----------\----- rawhide
'A' has been merged into 'rawhide', so it can be trivially deletected
without checking any builds.
/---/-------\-B"-B'-B
-/---/---\-----\----rawhide
\--C
'B' has commits that are not found anywhere else (B, B', and B"), and
we need to check in koji if it knows about any builds from those
commits.
2. Release branches are protected: as an additional constraint,
release branches (fNN, elN, epelN, …) cannot be deleted if any builds
were done for this release. This means we preserve the branch
identification, even if we don't need this to preserve commits.
For branches older than f21, bodhi information is not available and we
cannot check if builds have been performed, so this script always
refuses removal.
Removal of the 'rawhide' branch is always refused.
3. Removal is refused in some additional corner cases:
- the spec file cannot be parsed
- multiple spec files are found
- the package has special characteristics that require manual review
Note: when *branch* is specified as a remote branch (e.g. "origin/f33"),
remote branches are checked. This mode is useful when run in a clone
of the canonical origin repository. When *branch* is specified as a
local branch, local branches are checked. This mode is useful when run
in the original repo.
"""
import argparse
import pathlib
import re
import subprocess
import pygit2
import requests
import koji as _koji
BODHI_RELEASES = 'https://bodhi.fedoraproject.org/releases/?rows_per_page=1000'
NORMAL_BRANCHES = r'^(f\d{1,2}|el\d|epel\d|epel1\d)$'
MACRO_DEF_RE = re.compile(rb'^%(global|define)\s+(?P<macro>\S+)\s+(?P<value>.+)$')
_KOJI_SESSION = None
def koji_session(opts):
global _KOJI_SESSION
if not _KOJI_SESSION:
koji = _koji.get_profile_module(opts.koji_profile)
session_opts = koji.grab_session_options(koji.config)
session = koji.ClientSession(koji.config.server, session_opts)
_KOJI_SESSION = (session, koji)
return _KOJI_SESSION
def koji_builds_exist(tag, package, opts):
session, _ = koji_session(opts)
print(f'Checking for {package} in tag {tag}...', end=' ')
tagged = session.listTagged(tag, latest=True, inherit=False, package=package)
print(tagged[0]['nvr'] if tagged else '(no)')
return bool(tagged)
def bodhi_builds_exist(branch, package, opts):
releases = requests.get(BODHI_RELEASES).json()['releases']
for entry in releases:
if entry['branch'] == branch:
tags = [v for k,v in entry.items() if k.endswith('_tag') and v]
print(f'Found branch {branch} in bodhi with tags:', ', '.join(tags))
for tag in tags:
if koji_builds_exist(tag, package, opts):
return True
print(f'No builds found in koji for branch {branch}')
return False
print(f'Branch {branch} not found in bodhi, checking if branch matches pattern...')
m = re.match(NORMAL_BRANCHES, branch)
if m:
print('...it does, do not delete')
return True
print('...no match, seems OK to remove')
return False
def find_hash(build):
return build['source'].rsplit('#', 1)[1]
def list_builds(package, opts):
session, koji = koji_session(opts)
try:
pkg = session.getPackageID(package, strict=True)
except koji.GenericError as e:
if 'Invalid package name' in str(e):
return {}
else:
raise
builds = session.listBuilds(packageID=pkg, state=koji.BUILD_STATES['COMPLETE'])
with session.multicall(strict=True) as msession:
for build in builds:
if build['source'] is None:
build['source'] = msession.getTaskInfo(build['task_id'], request=True)
for build in builds:
if isinstance(build['source'], koji.VirtualCall):
r = build['source'].result
if r is None:
# This seems to happen for very old builds, e.g. buildbot-0.7.5-1.fc7.
build['source'] = None
nvr, time = build['nvr'], build['creation_time']
print(f'Warning: build {nvr} from {time} has no source, ignoring.')
else:
build['source'] = r['request'][0]
by_hash = {find_hash(b):b for b in builds if b['source']}
return by_hash
def containing_branches(repo, commit, *, local, ignore_branch=None):
if local:
containing = repo.branches.local.with_commit(commit)
else:
containing = repo.branches.remote.with_commit(commit)
for b in containing:
branch = repo.branches[b]
if branch != ignore_branch:
yield branch
def rpm_eval(expression, macros):
cmd = ['rpm']
for macro, value in macros.items():
cmd.append('--define')
cmd.append(f'{macro} {value}')
cmd.append('--eval')
cmd.append(expression)
return subprocess.check_output(cmd, text=True).strip()
def name_in_spec_file(commit, package):
try:
spec = (commit.tree / f'{package}.spec').data
except KeyError:
print(f"Commit {commit.hex} doesn't have '{package}.spec', looking for other specs.")
specs = set()
for candidate in commit.tree:
if candidate.name.endswith(".spec"):
specs.add(candidate)
print(f"Found '{candidate.name}'.")
if not specs:
print(f"Commit {commit.hex} doesn't have '*.spec', assuming package is unbuildable.")
return None
if len(specs) > 1:
msg = f"Commit {commit.hex} has multiple '*.spec' files, aborting."
raise NotImplementedError(msg)
spec = specs.pop().data
# We don't try to decode the whole spec file here, to reduce the chances of trouble.
# Just any interesting lines.
macros = {}
for line in spec.splitlines():
try:
if line.startswith(b'Name:'):
name = line[5:].decode().strip()
return rpm_eval(name, macros)
macro_def = MACRO_DEF_RE.match(line)
if macro_def:
macros[macro_def.group('macro').decode()] = macro_def.group('value').decode()
except UnicodeDecodeError:
print(f"Something is wrong: commit {commit.hex} has busted encoding'.")
raise
def do_opts():
parser = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawTextHelpFormatter)
parser.add_argument('--koji-profile', default='koji')
parser.add_argument('--package')
parser.add_argument('--repository', default='.', type=pathlib.Path)
parser.add_argument('branch')
opts = parser.parse_args()
if opts.package is None:
opts.package = opts.repository.absolute().name
return opts
def branch_is_reachable(opts):
repo = pygit2.Repository(opts.repository)
try:
branch = repo.branches.local[opts.branch]
branch_name = branch.branch_name
local = True
except KeyError:
branch = repo.branches.remote[opts.branch]
l = len(branch.remote_name)
branch_name = branch.branch_name[l+1:]
local = False
if branch_name == 'rawhide':
print("Branch 'rawhide' cannot be deleted.")
return 1
if bodhi_builds_exist(branch_name, opts.package, opts):
print('Branch was used to build packages, cannot delete.')
return 1
print('Querying koji for builds...')
builds = {opts.package: list_builds(opts.package, opts)}
other = list(containing_branches(repo, branch.target, local=local, ignore_branch=branch))
if other:
names = ', '.join(o.name for o in other)
print(f'Branch merged into {names}. Safe to delete.')
return 0
print('Branch has commits not found anywhere else, checking builds...')
for n, commit in enumerate(repo.walk(branch.target, pygit2.GIT_SORT_TOPOLOGICAL)):
subj = commit.message.splitlines()[0][:60]
print(f'{n}: {commit.hex[:7]} {subj}')
other = list(containing_branches(repo, commit, local=local, ignore_branch=branch))
if other:
names = ', '.join(o.name for o in other)
print(f'Commit {commit.hex} referenced from {names}. Stopping iteration.')
break
# Figure out the name used in the spec file in that commit.
# This is for the following case:
# * Repo 'foo' exists and is active
# * Repo 'foo2' (like a compat version of foo) exists and is active
# * People have 'Name: foo' in 'foo2.spec' and make a build
# * Koji will record this package as 'foo', even though it was built from 'foo2' repo
try:
real_name = name_in_spec_file(commit, opts.package)
except UnicodeDecodeError:
return 1
if real_name is not None and real_name not in builds:
print(f"{commit.hex} has Name: {real_name}. Looking for builds.")
builds[real_name] = list_builds(real_name, opts)
for name in builds:
built = builds[name].get(commit.hex, None)
if built:
print(f"Sorry, {commit.hex} built as {built['nvr']}.")
koji_link = f"https://koji.fedoraproject.org/koji/taskinfo?taskID={built['task_id']}"
print(f"See {koji_link}.")
return 1
print('No builds found, seems OK to delete.')
return 0
if __name__ == '__main__':
opts = do_opts()
print(f'Checking package {opts.package} in {opts.repository.absolute()}')
exit(branch_is_reachable(opts))

51
scripts_new/packages/distgit/get_retired_packages.sh
View file

@ -1,51 +0,0 @@
#!/usr/bin/bash
set -eu
declare -a active_branches=( rawhide )
declare -A retired
declare -a git_branches
declare -r checkout_path="${1:-/srv/git/rpms}"
# Get active releases from bodhi and store them in an array
active_branches=("$active_branches $(curl -X GET -s -H 'Accept: application/json' 'https://bodhi.fedoraproject.org/releases/?exclude_archived=true' | jq -r '[ .releases[].branch | select(test("\\d+$"))] | unique | .[]')")
for git_repo in ${checkout_path}/*.git; do
# Declare these helper arrays inside the loop
declare -a branch_intersection=()
declare -a unicorn=()
# Silence pushd to avoid unnecessary output
pushd $git_repo >/dev/null
# Get the name of the package
package=$(basename $git_repo .git)
# Get the branches of the git repo, to check only in thse that exist, to avoid git fatal errors
git_branches=($(git branch | cut -c 3-))
# Create intersection of the active releases and the git branches to avoid git fatal errors
for release in ${active_branches[@]}; do
for branch in "${git_branches[@]}"; do
if [[ $release == $branch ]]; then
branch_intersection+=("$release")
fi
done
done
# Deduplicate the branch array
unicorn=($(printf "%s\n" "${branch_intersection[@]}" | sort -u))
# Check for presence of a dead.package (indicates retired package)
for release in ${unicorn[@]}; do
if [[ -n "$(git ls-tree ${release} --name-only -- dead.package)" ]]; then
# Add retired package to an associative array as a value under a key that represents release
retired["$release"]+="\"$package\","
fi
done
# Silence pushd to avoid unnecessary output
popd >/dev/null
done
# Store the retired packages in separate json files by release
for release in "${!retired[@]}"; do
printf '{"%s": [%s]}\n' "$release" "${retired[$release]:0:-1}" > /srv/cache/lookaside/retired_in_${release}.json
done

49
scripts_new/packages/distgit/give_package.py
View file

@ -1,49 +0,0 @@
#! /usr/bin/python3 -tt
""" Give a package in pagure-on-dist-git from one user to another.
This can also be used to give the package to the 'orphan' user.
You need a privileged pagure token in /etc/fedrepo_req/config.ini
[admin]
pagure_api_token = something secret
You can generate such a token on pkgs02 with:
$ PAGURE_CONFIG=/etc/pagure/pagure.cfg pagure-admin admin-token --help
"""
# Copyright (c) 2017 Red Hat
# SPDX-License-Identifier: GPL-2.0
#
# Authors:
# Ralph Bean <rbean@redhat.com>
import argparse
import sys
try:
import utilities
except ImportError:
print("Try setting PYTHONPATH to find the utilities.py file.")
raise
PAGURE_URL = 'https://src.fedoraproject.org/api/0/'
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("package", help="The package that should be given.")
parser.add_argument("custodian", help="The user taking over the package.")
args = parser.parse_args()
session = utilities.retry_session()
try:
namespace, package = args.package.split('/')
except:
print("Package must be like <namespace>/<name>, not %r" % args.package)
sys.exit(1)
utilities.give_package(session, namespace, package, args.custodian)
if __name__ == "__main__":
main()

95
scripts_new/packages/distgit/orphan_all_packages.py
View file

@ -1,95 +0,0 @@
#! /usr/bin/python3 -tt
""" Orphan all packages of a given set of users.
If there are other committers on a package, the first one is promoted to be the
new owner.
If there are no other committers, then the package is given to the `orphan`
user.
You need a privileged pagure token in /etc/fedrepo_req/config.ini
[admin]
pagure_api_token = something secret
You can generate such a token on pkgs02 with:
$ PAGURE_CONFIG=/etc/pagure/pagure.cfg pagure-admin admin-token --help
"""
# Copyright (c) 2017 Red Hat
# SPDX-License-Identifier: GPL-2.0
#
# Authors:
# Ralph Bean <rbean@redhat.com>
import argparse
try:
import utilities
except ImportError:
print("Try setting PYTHONPATH to find the utilities.py file.")
raise
PAGURE_URL = 'https://src.fedoraproject.org/api/0/'
def get_all_packages_for_user(session, user):
url = PAGURE_URL + 'projects'
params = dict(owner=user, fork=False)
response = session.get(url, params=params, timeout=400)
if not bool(response):
raise IOError("Failed GET %r %r" % (response.request.url, response))
for project in response.json()['projects']:
yield project
def triage_packages(packages, user):
for package in packages:
for kind in ('admin', 'commit'):
others = package['access_users'][kind]
try:
others.remove(user)
except ValueError:
# Owner doesn't have commit. Weird, but ok.
pass
if others:
# Select the first one to become the new owner.
yield package, others[0]
break
else:
yield package, 'orphan'
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("users", nargs="*",
help="Users to remove.")
args = parser.parse_args()
session = utilities.retry_session()
for user in args.users:
print("Investigating packages for user %r" % user)
packages = get_all_packages_for_user(session, user)
transfers = triage_packages(packages, user)
# Exhaust the generator
transfers = list(transfers)
for package, custodian in transfers:
print("%s/%s will be given to %s" % (
package['namespace'], package['name'], custodian))
response = input("Is this okay? [y/N]")
if response.lower() not in ('y', 'yes'):
print("!! OK. Bailing out for %r" % user)
continue
print("Starting transfers")
for package, custodian in transfers:
namespace, name = package['namespace'], package['name']
utilities.give_package(session, namespace, name, custodian)
if __name__ == "__main__":
main()

111
scripts_new/packages/distgit/orphan_listed_packages.py
View file

@ -1,111 +0,0 @@
"""
This script is useful to bulk orphan listed packages.
E.g. when they fail to install or fail to build.
"""
import logging
import os
import sys
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
REASON = "Important bug not fixed"
# REASON = "Fails to build from source"
# REASON = "Orphaned by releng"
PACKAGES = {
# "pkg_name": "https://bugzilla.redhat.com/XXX or a different info",
}
PAGURE_TOKEN = os.getenv("PAGURE_TOKEN")
LOG = logging.getLogger(__name__)
BASE_URL = "https://src.fedoraproject.org"
def retry_session():
session = requests.Session()
retry = Retry(
total=5,
read=5,
connect=5,
backoff_factor=0.3,
status_forcelist=(500, 502, 504),
)
adapter = HTTPAdapter(max_retries=retry)
session.mount("http://", adapter)
session.mount("https://", adapter)
return session
def orphan_package(name, namespace="rpms", reason=REASON, reason_info=None):
"""Give the specified project on dist_git to the ``orphan`` user."""
LOG.debug("Going to orphan: %s/%s", namespace, name)
session = retry_session()
# Orphan the package
url = f"{BASE_URL}/_dg/orphan/{namespace}/{name}"
headers = {"Authorization": f"token {PAGURE_TOKEN}"}
data = {
"orphan_reason": reason,
}
if reason:
data["orphan_reason_info"] = reason_info
req = session.post(url, data=data, headers=headers)
if not req.ok:
print("**** REQUEST FAILED")
print(" - Orphan package")
print(req.url)
print(data)
print(headers)
print(req.text)
else:
print(f"{namespace}/{name} is orphaned")
session.close()
def get_bugzilla_overrides(name, namespace="rpms"):
"""Returns bugzilla overrides of the specified package.. """
LOG.debug("Checking for bugzilla overrides on %s/%s", namespace, name)
session = retry_session()
req = session.get(f"{BASE_URL}/_dg/bzoverrides/{namespace}/{name}")
return req.json()
def reset_bugzilla_overrides(name, namespace="rpms"):
""" Reset the Fedora bugzilla overrides of the specified package."""
overrides = get_bugzilla_overrides(name)
if overrides["fedora_assignee"] is None:
LOG.debug("No bugzilla overrides on %s/%s", namespace, name)
return
LOG.debug("Resetting bugzilla overrides on %s/%s", namespace, name)
url = f"{BASE_URL}/_dg/bzoverrides/{namespace}/{name}"
headers = {"Authorization": f"token {PAGURE_TOKEN}"}
username = overrides["fedora_assignee"]
overrides["fedora_assignee"] = None
session = retry_session()
req = session.post(url, headers=headers, data=overrides)
if not req.ok:
print("**** REQUEST FAILED")
print(" - Remove bugzilla overrides")
print(req.url)
print(req.text)
else:
print(f" {username} has no longer a bugzilla overrides on {namespace}/{name}")
session.close()
if __name__ == "__main__":
if not PACKAGES:
sys.exit("Define PACKAGES first")
for package in PACKAGES:
orphan_package(package, reason_info=PACKAGES[package])
reset_bugzilla_overrides(package)

390
scripts_new/packages/distgit/retire_packagers.py
View file

@ -1,390 +0,0 @@
#!/usr/bin/python3
"""
This script queries dist-git for all the packages a given packager maintains,
has commit or watches.
Package that the packager is the main admin are then orphaned. The packager is
then removed from all packages that they have commit for and their watch status
is reset on every packages that they are watching.
"""
import argparse
import collections
import logging
import os
import sys
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
_log = logging.getLogger(__name__)
dist_git_base = "https://src.fedoraproject.org"
pagure_token = None
def retry_session():
session = requests.Session()
retry = Retry(
total=5,
read=5,
connect=5,
backoff_factor=0.3,
status_forcelist=(500, 502, 504),
)
adapter = HTTPAdapter(max_retries=retry)
session.mount("http://", adapter)
session.mount("https://", adapter)
return session
def setup_logging(log_level: int):
handlers = []
_log.setLevel(log_level)
# We want all messages logged at level INFO or lower to be printed to stdout
info_handler = logging.StreamHandler(stream=sys.stdout)
handlers.append(info_handler)
if log_level == logging.INFO:
# In normal operation, don't decorate messages
for handler in handlers:
handler.setFormatter(logging.Formatter("%(message)s"))
logging.basicConfig(level=log_level, handlers=handlers)
def get_arguments(args):
""" Load and parse the CLI arguments."""
parser = argparse.ArgumentParser(
description="Looks for the specified list of users what they "
"maintain or watch in dist-git.\nIf --retire is specified, all the ACL "
"the packager(s) have in dist-git will be removed. If they are main admins "
"of some packages, these packages will be orphaned. If they have commit "
"access on some packages, they will no longer have these access. If they "
"watch a package, their watch status will be reset. Note: the source of "
"information is refreshed hourly, so if you run the script twice with "
"`--retire` you may not see a difference here."
)
parser.add_argument(
dest="usernames", nargs="*", help="Names of the users to retire.",
)
parser.add_argument(
"--from-file",
dest="users_file",
help="Path to a file containing the users to check (one per line).",
)
parser.add_argument(
"--retire",
action="store_true",
default=False,
help="Retire the user(s) (ie: orphan, remove from ACL, reset watch)",
)
parser.add_argument(
"--api-token",
dest="pagure_token",
default=os.environ.get("PAGURE_TOKEN"),
help="Pagure token to use to interact with dist-git. It can also be set "
"via the PAGURE_TOKEN environment variable. (This script requires the "
"`modifyproject` ACL to work)",
)
report_group = parser.add_mutually_exclusive_group()
report_group.add_argument(
"--watch",
action="store_const",
dest="report",
const="watch",
default="all",
help="Only report/act on watched projects",
)
report_group.add_argument(
"--maintain",
action="store_const",
dest="report",
const="maintain",
help="Only report/act projects the packagers have commit access to",
)
log_level_group = parser.add_mutually_exclusive_group()
log_level_group.add_argument(
"--debug",
action="store_const",
dest="log_level",
const=logging.DEBUG,
default=logging.INFO,
help="Enable debugging output",
)
return parser.parse_args(args)
def user_access(session, username, namespace_name):
""" Returns whether the specified username is listed in the maintainers
list of the specified package and a set of all maintainers therein. """
req = session.get(f"{dist_git_base}/api/0/{namespace_name}")
project = req.json()
maintainers = set()
for acl in project["access_users"]:
maintainers.update(set(project["access_users"][acl]))
if username == project["user"]["name"]:
level = "main admin"
elif username in maintainers:
level = "maintainer"
else:
level = None
return level, maintainers
def get_bugzilla_overrides(username, namespace, name):
""" Returns whether the specified username is set in the bugzilla overrides
of the specified package.. """
_log.debug(
"Checking for bugzilla overrides on %s/%s for %s", namespace, name, username
)
base_url = dist_git_base.rstrip("/")
session = retry_session()
req = session.get(f"{dist_git_base}/_dg/bzoverrides/{namespace}/{name}")
return req.json()
def unwatch_package(namespace, name, username):
""" Reset the watch status of the given user on the specified project. """
_log.debug("Going to reset watch status of %s on %s/%s", username, namespace, name)
base_url = dist_git_base.rstrip("/")
session = retry_session()
# Reset the watching status
url = f"{base_url}/api/0/{namespace}/{name}/watchers/update"
headers = {"Authorization": f"token {pagure_token}"}
data = {"status": -1, "watcher": username}
req = session.post(url, data=data, headers=headers)
if not req.ok:
print("**** REQUEST FAILED")
print(" - Unwatch package")
print(req.url)
print(data)
print(headers)
print(req.text)
else:
print(f" {username} is no longer watching {namespace}/{name}")
session.close()
def orphan_package(session, namespace, name, username):
""" Give the specified project on dist_git to the ``orphan`` user.
"""
_log.debug("Going to orphan: %s/%s from %s", namespace, name, username)
base_url = dist_git_base.rstrip("/")
session = retry_session()
# Orphan the package
url = f"{base_url}/_dg/orphan/{namespace}/{name}"
headers = {"Authorization": f"token {pagure_token}"}
data = {
"orphan_reason": "Orphaned by releng",
}
req = session.post(url, data=data, headers=headers)
if not req.ok:
print("**** REQUEST FAILED")
print(" - Orphan package")
print(req.url)
print(data)
print(headers)
print(req.text)
else:
print(f" {username} is no longer the main admin of {namespace}/{name}")
session.close()
def remove_access(namespace, name, username, usertype):
""" Remove the ACL of the specified user/group on the specified project. """
_log.debug("Going to remove %s from %s/%s", username, namespace, name)
base_url = dist_git_base.rstrip("/")
session = retry_session()
# Remove ACL on the package
url = f"{base_url}/api/0/{namespace}/{name}/git/modifyacls"
headers = {"Authorization": f"token {pagure_token}"}
data = {
"user_type": usertype,
"name": username,
}
req = session.post(url, data=data, headers=headers)
if not req.ok:
print("**** REQUEST FAILED")
print(" - Remove ACL")
print(req.url)
print(data)
print(req.text)
else:
print(f" {username} is no longer maintaining {namespace}/{name}")
session.close()
if usertype == "user":
# Reset the watching status
unwatch_package(namespace, name, username)
def reset_bugzilla_overrides(username, namespace, name, overrides):
""" Reset the the bugzilla overrides of the specified package so that the
specified user no longer has one. """
_log.debug(
"Resetting bugzilla overrides on %s/%s for %s", namespace, name, username
)
base_url = dist_git_base.rstrip("/")
url = f"{base_url}/_dg/bzoverrides/{namespace}/{name}"
headers = {"Authorization": f"token {pagure_token}"}
for key in overrides:
if overrides[key] == username:
overrides[key] = None
session = retry_session()
req = session.post(url, headers=headers, data=overrides)
if not req.ok:
print("**** REQUEST FAILED")
print(" - Remove bugzilla overrides")
print(req.url)
print(data)
print(req.text)
else:
print(f" {username} has no longer a bugzilla overrides on {namespace}/{name}")
session.close()
def main(args):
""" For the specified list of users, retrieve what they are maintaining
or watching in dist-git."""
args = get_arguments(args)
setup_logging(log_level=args.log_level)
_log.debug("Log level set to: %s", args.log_level)
if args.pagure_token:
global pagure_token
pagure_token = args.pagure_token
if not pagure_token and args.retire:
_log.debug(
"Trying to retrieve pagure_api_token from the fedscm configuration file"
)
try:
import fedscm_admin.config
from fedscm_admin import CONFIG
api_token = fedscm_admin.config.get_config_item(CONFIG, "pagure_api_token")
except:
pass
if not pagure_token and args.retire:
print(
"No pagure token set in the CLI argument or via the PAGURE_TOKEN "
"environment variable or found in the fedscm configuration file. "
"Going to ignore --retire"
)
args.retire = False
usernames = []
if args.users_file:
_log.debug("Loading usernames for file: %s", args.users_file)
if not os.path.exists(args.users_file):
_log.info("No such file found: %s", args.users_file)
try:
with open(args.users_file) as stream:
usernames = [
l.strip() for l in stream.readlines() if l.strip()
]
except Exception as err:
_log.debug(
"Failed to load/read the file: %s, error is: %s", args.users_file, err
)
else:
_log.debug("Loading usernames for the CLI arguments")
usernames = args.usernames
# We load the info from the pagure_bz file which will tell us everything
# that would be synced to bugzilla (POC and CC)
_log.debug("Loading info from dist-git's pagure_bz.json file")
session = retry_session()
req = session.get(f"{dist_git_base}/extras/pagure_bz.json")
pagure_bz = req.json()
session.close()
packages_per_user = collections.defaultdict(set)
for namespace in pagure_bz:
for package in pagure_bz[namespace]:
_log.debug("Processing %s/%s", namespace, package)
for user in pagure_bz[namespace][package]:
if user in usernames:
packages_per_user[user].add(f"{namespace}/{package}")
# On the top of this, we'll also query the list from dist-git directly as
# the previous source of info while quicker to query will not include
# the packages that the packagers have access to but set their watch status
# to "unwatch".
# However, we only need to run this if we want to know about packages someone
# maintains (ie: we can bypass this section if ``--watch`` is passed to the
# CLI).
if args.report in ["all", "maintain"]:
for username in sorted(usernames):
_log.debug("Loading info from dist-git's %s's page", username)
url = f"{dist_git_base}/api/0/user/{username}?per_page=50"
while url:
req = session.get(url)
data = req.json()
for repo in data.get("repos", []):
maintainers = set(repo["user"]["name"])
for acl in repo["access_users"]:
maintainers.update(set(repo["access_users"][acl]))
if username in maintainers:
namespace = repo["namespace"]
package = repo["name"]
packages_per_user[username].add(f"{namespace}/{package}")
url = data.get("repos_pagination", {}).get("next")
if not url:
break
for username in sorted(usernames):
_log.debug("Processing user: %s", username)
for pkg in sorted(packages_per_user[username]):
level, maintainers = user_access(session, username, pkg)
namespace, name = pkg.split("/", 1)
if level:
if args.report in ["all", "maintain"]:
print(f"{username} is {level} of {namespace}/{name}")
if level == "main admin" and len(maintainers) > 1:
maintainers_strs = (f"@{m}" for m in sorted(maintainers - {username}))
maintainers_str = ", ".join(maintainers_strs)
print(f" {namespace}/{name} co-maintainers: {maintainers_str}")
if args.retire:
if level == "main admin":
orphan_package(session, namespace, name, username)
elif level == "maintainer":
remove_access(namespace, name, username, "user")
else:
if args.report in ["all", "watch"]:
print(f"{username} is watching {namespace}/{name}")
if args.retire:
unwatch_package(namespace, name, username)
overrides = get_bugzilla_overrides(username, namespace, name)
if username in overrides.values():
print(f"{username} has a bugzilla override on {namespace}/{name}")
if args.retire:
reset_bugzilla_overrides(username, namespace, name, overrides)
print()
if __name__ == "__main__":
try:
sys.exit(main(sys.argv[1:]))
except KeyboardInterrupt:
pass

62
scripts_new/packages/distgit/utilities.py
View file

@ -1,62 +0,0 @@
#! /usr/bin/python -tt
""" Utilities for manipulating dist-git (pagure). """
# Copyright (c) 2017 Red Hat
# SPDX-License-Identifier: GPL-2.0
#
# Authors:
# Ralph Bean <rbean@redhat.com>
import json
import pprint
import sys
import traceback
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
try:
from fedscm_admin.pagure import get_pagure_auth_header
admin_headers = get_pagure_auth_header('admin')
except:
traceback.print_exc()
print("Failed to load admin tokens from fedrepo-req-admin")
sys.exit(1)
PAGURE_URL = 'https://src.fedoraproject.org/api/0/'
def retry_session():
session = requests.Session()
retry = Retry(
total=5,
read=5,
connect=5,
backoff_factor=0.3,
status_forcelist=(500, 502, 504),
)
adapter = HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://', adapter)
return session
def give_package(session, namespace, package, custodian):
print("Giving %s/%s to %s" % (
namespace, package, custodian))
url = PAGURE_URL + namespace + '/' + package
payload = json.dumps({'main_admin': custodian})
response = session.patch(
url,
data=payload,
headers=admin_headers,
timeout=60,
)
if not bool(response):
try:
pprint.pprint(response.json())
except:
pass
raise IOError("Failed PATCH %r %r" % (response.request.url, response))

0
scripts_new/packages/lifecycle/.gitignore vendored
View file

0
scripts_new/packages/maintenance/.gitignore vendored
View file

0
scripts_new/packages/orphaned/.gitignore vendored
View file

19
scripts_new/packages/orphaned/Dockerfile
View file

@ -1,19 +0,0 @@
FROM registry.fedoraproject.org/fedora:latest
RUN set -xeuo pipefail ;\
dnf install --setopt=install_weak_deps=False -y \
python3 \
python3-dnf \
python3-dogpile-cache \
python3-koji \
python3-requests \
python3-texttable \
wget \
;\
dnf clean all
COPY find_unblocked_orphans.py /usr/local/bin/find_unblocked_orphans.py
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
STOPSIGNAL SIGINT

18
scripts_new/packages/orphaned/entrypoint.sh
View file

@ -1,18 +0,0 @@
#!/bin/bash
set -euo pipefail
# If UPDATE is passed, this script will download the latest version of
# find_unblocked_orphans from Pagure
UPDATE="${UPDATE-}"
script="/usr/local/bin/find_unblocked_orphans.py"
raw_url="https://www.pagure.io/releng/raw/main/f/scripts_new/packages/orphaned/find_unblocked_orphans.py"
if [ -n "${UPDATE}" ]; then
dl_dir="$(mktemp -d)"
script="${dl_dir}/find_unblocked_orphans.py"
wget "${raw_url}" -O "${script}"
fi
exec python3 "${script}" "$@"

996
scripts_new/packages/orphaned/find_unblocked_orphans.py
View file

@ -1,996 +0,0 @@
#! /usr/bin/python3
#
# find_unblocked_orphans.py - A utility to find orphaned packages in pagure
# that are unblocked in koji and to show what
# may require those orphans
#
# Copyright (c) 2009-2013 Red Hat
# SPDX-License-Identifier: GPL-2.0
#
# Authors:
# Jesse Keating <jkeating@redhat.com>
# Till Maas <opensource@till.name>
import argparse
import datetime
import email.mime.text
import hashlib
import json
import os
import smtplib
import sys
import textwrap
import time
import traceback
from collections import OrderedDict, defaultdict
from functools import lru_cache
from pathlib import Path
from queue import Queue
from threading import Thread
from typing import IO, Any, NotRequired, TypedDict, cast
import dnf
import dogpile.cache
import koji
import requests
try:
import texttable
with_table = True
except ImportError:
with_table = False
@lru_cache(maxsize=20480)
def SRPM(query, package):
# This function was stolen from pungi
"""Given a package object, get a package object for the
corresponding source rpm. Requires dnf still configured
and a valid package object."""
srpm, *_ = package.sourcerpm.split(".src.rpm")
name, version, release = srpm.rsplit("-", 2)
try:
srpmpo = query.filter(
name=name, version=version, release=release, arch="src"
).run()[0]
return srpmpo
except IndexError:
eprint(f"Error: Cannot find a source rpm for {name}-{version}-{release}")
sys.exit(1)
cache_dir = os.environ.get("XDG_CACHE_HOME", os.path.expanduser("~/.cache"))
os.makedirs(cache_dir, exist_ok=True)
cache = dogpile.cache.make_region().configure(
"dogpile.cache.dbm",
expiration_time=86400,
arguments=dict(filename=os.path.join(cache_dir, "dist-git-orphans-cache.dbm")),
)
PAGURE_URL = "https://src.fedoraproject.org"
PAGURE_MAX_ENTRIES_PER_PAGE = 100
EPEL7_RELEASE = dict(
repo="https://kojipkgs.fedoraproject.org/compose/updates/epel7/"
"compose/Everything/x86_64/os/",
source_repo="https://kojipkgs.fedoraproject.org/compose/updates/epel7/"
"compose/Everything/source/tree/",
koji_tag="epel7",
koji_hub="https://koji.fedoraproject.org/kojihub",
pagure_branch="epel7",
mailto="epel-announce@lists.fedoraproject.org",
bcc=[],
)
EPEL8_RELEASE = dict(
repo="https://kojipkgs.fedoraproject.org/compose/updates/epel8/"
"compose/Everything/x86_64/os/",
source_repo="https://kojipkgs.fedoraproject.org/compose/updates/epel8/"
"compose/Everything/source/tree/",
koji_tag="epel8",
koji_hub="https://koji.fedoraproject.org/kojihub",
pagure_branch="epel8",
mailto="epel-announce@lists.fedoraproject.org",
bcc=[],
)
EPEL9_RELEASE = dict(
repo="https://kojipkgs.fedoraproject.org/compose/updates/epel9/"
"compose/Everything/x86_64/os/",
source_repo="https://kojipkgs.fedoraproject.org/compose/updates/epel9/"
"compose/Everything/source/tree/",
koji_tag="epel9",
koji_hub="https://koji.fedoraproject.org/kojihub",
pagure_branch="epel9",
mailto="epel-announce@lists.fedoraproject.org",
bcc=[],
)
RAWHIDE_RELEASE = dict(
repo="https://kojipkgs.fedoraproject.org/compose/rawhide/"
"latest-Fedora-Rawhide/compose/Everything/x86_64/os",
source_repo="https://kojipkgs.fedoraproject.org/compose/rawhide/"
"latest-Fedora-Rawhide/compose/Everything/source/tree/",
koji_tag="f44",
koji_hub="https://koji.fedoraproject.org/kojihub",
pagure_branch="rawhide",
mailto="devel@lists.fedoraproject.org",
bcc=[],
)
BRANCHED_RELEASE = dict(
repo="https://kojipkgs.fedoraproject.org/compose/branched/"
"latest-Fedora-43/compose/Everything/x86_64/os",
source_repo="https://kojipkgs.fedoraproject.org/compose/branched/"
"latest-Fedora-43/compose/Everything/source/tree/",
koji_tag="f43",
pagure_branch="f43",
koji_hub="https://koji.fedoraproject.org/kojihub",
mailto="devel@lists.fedoraproject.org",
bcc=[],
)
RELEASES = {
"rawhide": RAWHIDE_RELEASE,
"branched": BRANCHED_RELEASE,
"epel9": EPEL9_RELEASE,
"epel8": EPEL8_RELEASE,
"epel7": EPEL7_RELEASE,
}
# pagure uid for orphan
ORPHAN_UID = "orphan"
HEADER = """\
SPECIAL NOTE: As of https://pagure.io/fesco/issue/3447,
all packages containing Golang libraries that are not leaves are exempted from
automatic retirement until the new Golang Packaging Guidelines are approved by
the Packaging Committee and published.
These packages are still listed in the report, but there is an additional list
of packages with exemptions included at the end.
Applications written in Go that do not include libraries used by other Go packages
are NOT subject to this exemption and will be retired as usual.
It is recommended not to unorphan any Golang libraries in the interim.
Instead, the Go SIG suggests waiting until the new guidelines are published and
then porting your packages to the new tooling that uses vendored dependencies,
obsoleting the current approach involving golang-*-devel packages.
Packagers interested in being early adopters or testers of the new tooling are
welcome to join #golang:fedoraproject.org on Matrix.
Once the new guidelines are fully implemented, automatic retirements of
orphaned Golang packages will resume after a six-week grace period.
The following packages are orphaned and will be retired when they
are orphaned for six weeks, unless someone adopts them. If you know for sure
that the package should be retired, please do so now with a proper reason:
https://fedoraproject.org/wiki/How_to_remove_a_package_at_end_of_life
Note: If you received this mail directly you (co)maintain one of the affected
packages or a package that depends on one. Please adopt the affected package or
retire your depending package to avoid broken dependencies, otherwise your
package will be retired when the affected package gets retired.
Request package ownership via the *Take* button in the left column on
https://src.fedoraproject.org/rpms/<pkgname>
Full report available at:
https://a.gtmx.me/orphans/orphans.txt
grep it for your FAS username and follow the dependency chain.
For human readable dependency chains,
see https://packager-dashboard.fedoraproject.org/
For all orphaned packages,
see https://packager-dashboard.fedoraproject.org/orphan
"""
FOOTER = """-- \nThe script creating this output is run and developed by Fedora
Release Engineering. Please report issues at its pagure instance:
https://pagure.io/releng/
The sources of this script can be found at:
https://pagure.io/releng/blob/main/f/scripts/find_unblocked_orphans.py
"""
def eprint(*args, **kwargs):
kwargs.setdefault("file", sys.stderr)
kwargs.setdefault("flush", True)
print(*args, **kwargs)
def send_mail(from_, to, subject, text, bcc=None):
if bcc is None:
bcc = []
msg = email.mime.text.MIMEText(text)
msg["Subject"] = subject
msg["From"] = from_
msg["To"] = to
if isinstance(to, str):
to = [to]
smtp = smtplib.SMTP("127.0.0.1")
errors = smtp.sendmail(from_, to + bcc, msg.as_string())
smtp.quit()
return errors
class PagureInfo:
def __init__(self, package, branch=RELEASES["rawhide"]["pagure_branch"], ns="rpms"):
self.package = package
self.branch = branch
try:
response = requests.get(f"{PAGURE_URL}/api/0/{ns}/{package}")
self.pkginfo = response.json()
if "error" in self.pkginfo:
# This is likely a "project not found" 404 error.
raise ValueError(self.pkginfo["error"])
except Exception:
eprint(f"Error getting pagure info for {ns}/{package} on {branch}")
traceback.print_exc(file=sys.stderr)
self.pkginfo = None
return
def get_people_and_emails(self) -> tuple[list[str], list[str]]:
if self.pkginfo is None:
return [], []
people = set()
emails = set()
for kind in ["access_users", "access_groups"]:
for persons in self.pkginfo[kind].values():
for person in persons:
if kind == "access_groups":
people.add("@" + person)
emails.add(f"{person}-members@fedoraproject.org")
else:
people.add(person)
emails.add(f"{person}@fedoraproject.org")
return sorted(people), sorted(emails)
@property
def age(self):
then = self.status_change
now = datetime.datetime.now(datetime.timezone.utc)
return now - then
@property
def status_change(self):
if self.pkginfo is None:
return datetime.datetime.now(datetime.timezone.utc)
# See https://pagure.io/pagure/issue/2412
if "date_modified" in self.pkginfo:
status_change = float(self.pkginfo["date_modified"])
else:
status_change = float(self.pkginfo["date_created"])
status_change_dt = datetime.datetime.fromtimestamp(
status_change, tz=datetime.timezone.utc
)
return status_change_dt
def __getitem__(self, *args, **kwargs):
return self.pkginfo.__getitem__(*args, **kwargs)
def setup_dnf(
repo=RELEASES["rawhide"]["repo"],
source_repo=RELEASES["rawhide"]["source_repo"],
):
"""Setup dnf query with two repos"""
base = dnf.Base()
# use digest to make repo id unique for each URL
for baseurl, name in (repo, "repo"), (source_repo, "repo-source"):
r = base.repos.add_new_repo(
name + "-" + hashlib.sha256(baseurl.encode()).hexdigest(),
base.conf,
baseurl=[baseurl],
skip_if_unavailable=False,
)
r.enable()
r.load()
base.fill_sack(load_system_repo=False, load_available_repos=True)
return base.sack.query()
@cache.cache_on_arguments()
def orphan_packages(namespace="rpms"):
pkgs, pages = get_pagure_orphans(namespace)
eprint(f"({pages} pages)", end=" ")
for page in range(2, pages + 1):
if page % 10:
eprint(".", end="")
else:
eprint(page, end="")
new_pkgs, _ = get_pagure_orphans(namespace, page)
pkgs.update(new_pkgs)
return pkgs
@cache.cache_on_arguments()
def get_pagure_orphans(namespace, page=1):
url = PAGURE_URL + "/api/0/projects"
params = dict(
owner=ORPHAN_UID,
namespace=namespace,
page=page,
per_page=PAGURE_MAX_ENTRIES_PER_PAGE,
)
tries = 0
response = requests.get(url, params=params)
while not bool(response):
msg = f"{response.request.url!r} gave {response!r}"
if tries > 20:
raise IOError(msg)
print(msg, file=sys.stderr)
time.sleep(tries)
tries += 1
response = requests.get(url, params=params)
pkgs = response.json()["projects"]
pages = response.json()["pagination"]["pages"]
return {p["name"]: p for p in pkgs}, pages
def unblocked_packages(
packages,
tagID=RELEASES["rawhide"]["koji_tag"],
kojihub=RELEASES["rawhide"]["koji_hub"],
):
unblocked = []
kojisession = koji.ClientSession(kojihub)
kojisession.multicall = True
for p in packages:
kojisession.listPackages(tagID=tagID, pkgID=p, inherited=True)
listings = kojisession.multiCall()
# Check the listings for unblocked packages.
for pkgname, result in zip(packages, listings):
if isinstance(result, list):
[pkg] = result
if pkg:
if not pkg[0]["blocked"]:
package_name = pkg[0]["package_name"]
unblocked.append(package_name)
else:
# TODO - what state does this condition represent?
pass
else:
print(f"ERROR: {pkgname}: {result}")
return unblocked
class DepChecker:
def __init__(self, release, repo=None, source_repo=None, namespace="rpms"):
self.release = release
repo = repo or RELEASES[release]["repo"]
source_repo = source_repo or RELEASES[release]["source_repo"]
dnfquery = setup_dnf(repo=repo, source_repo=source_repo)
self.dnfquery = dnfquery
self.pagureinfo_queue = Queue()
self.pagure_dict = {}
self.not_in_repo = []
self.dep_chain = defaultdict(set)
# create_mapping()
src_by_bin = {} # Dict of source pkg objects by binary package objects
bin_by_src = {} # Dict of binary pkgobjects by srpm name
# Populate the dicts
for rpm_package in self.dnfquery:
if rpm_package.arch == "src":
continue
srpm = SRPM(self.dnfquery, rpm_package)
src_by_bin[rpm_package] = srpm
if srpm.name in bin_by_src:
bin_by_src[srpm.name].append(rpm_package)
else:
bin_by_src[srpm.name] = [rpm_package]
self._src_by_bin = src_by_bin
self._bin_by_src = bin_by_src
@property
def by_src(self):
return self._bin_by_src
@property
def by_bin(self):
return self._src_by_bin
def find_dependent_packages(self, srpmname, ignore):
"""Return packages depending on packages built from SRPM ``srpmname``
that are built from different SRPMS not specified in ``ignore``.
:param ignore: list of binary package names that will not be
returned as dependent packages or considered as alternate
providers
:type ignore: list() of str()
:returns: OrderedDict dependent_package: list of requires only
provided by package ``srpmname`` {dep_pkg: [prov, ...]}
"""
# Some of this code was stolen from repoquery
dependent_packages = {}
# Handle packags not found in the repo
try:
rpms = self.by_src[srpmname]
except KeyError:
# If we don't have a package in the repo, there is nothing to do
eprint(f"Package {srpmname} not found in repo")
self.not_in_repo.append(srpmname)
rpms = []
# provides of all packages built from ``srpmname``
provides = []
for pkg in rpms:
# add all the provides from the package as strings
string_provides = [str(prov) for prov in pkg.provides]
provides.extend(string_provides)
# add all files as provides
# pkg.files is a list of paths
# sometimes paths start with "//" instead of "/"
# normalise "//" to "/":
# os.path.normpath("//") == "//", but
# os.path.normpath("///") == "/"
file_provides = [os.path.normpath(f"//{fn}") for fn in pkg.files]
provides.extend(file_provides)
# Zip through the provides and find what's needed
for prov in provides:
# check only base provide, ignore specific versions
# "foo = 1.fc20" -> "foo"
base_provide, *_ = prov.split()
# FIXME: Workaround for:
# https://bugzilla.redhat.com/show_bug.cgi?id=1191178
if base_provide[0] == "/":
base_provide = base_provide.replace("[", "?")
base_provide = base_provide.replace("]", "?")
# Elide provide if also provided by another package
for pkg in self.dnfquery.filter(provides=base_provide):
# FIXME: might miss broken dependencies in case the other
# provider depends on a to-be-removed package as well
if pkg.name in ignore:
# eprint(f"Ignoring provider package {pkg.name}")
pass
elif pkg not in rpms:
break
else:
for dependent_pkg in self.dnfquery.filter(requires=base_provide):
# skip if the dependent rpm package belongs to the
# to-be-removed Fedora package
if dependent_pkg in self.by_src[srpmname]:
continue
# use setdefault to either create an entry for the
# dependent package or add the required prov
dependent_packages.setdefault(dependent_pkg, set()).add(prov)
return OrderedDict(sorted(dependent_packages.items()))
def pagure_worker(self):
branch = RELEASES[self.release]["pagure_branch"]
while True:
package = self.pagureinfo_queue.get()
if package not in self.pagure_dict:
pkginfo = PagureInfo(package, branch)
qsize = self.pagureinfo_queue.qsize()
eprint(f"Got info for {package} on {branch}, todo: {qsize}")
self.pagure_dict[package] = pkginfo
self.pagureinfo_queue.task_done()
def recursive_deps(self, packages, max_deps=20):
incomplete = []
# Start threads to get information about (co)maintainers for packages
for _ in range(0, 2):
people_thread = Thread(target=self.pagure_worker)
people_thread.daemon = True
people_thread.start()
# get a list of all rpm_pkgs that are to be removed
rpm_pkg_names = []
for name in packages:
self.pagureinfo_queue.put(name)
# Empty list if pkg is only for a different arch
bin_pkgs = self.by_src.get(name, [])
rpm_pkg_names.extend([p.name for p in bin_pkgs])
# dict for all dependent packages for each to-be-removed package
dep_map = OrderedDict()
for name in sorted(packages):
self.dep_chain[name] = (
set()
) # explicitly initialize the set for the orphaned
eprint(f"Getting packages depending on: {name}")
ignore = rpm_pkg_names
dep_map[name] = OrderedDict()
to_check = [name]
allow_more = True
seen = []
while True:
eprint(f"to_check ({len(to_check)}): {to_check}")
check_next = to_check.pop(0)
seen.append(check_next)
dependent_packages = self.find_dependent_packages(check_next, ignore)
if dependent_packages:
new_names = []
new_srpm_names = set()
for pkg, dependencies in dependent_packages.items():
if pkg.arch != "src":
srpm_name = self.by_bin[pkg].name
else:
srpm_name = pkg.name
if (
srpm_name not in to_check
and srpm_name not in new_names
and srpm_name not in seen
):
new_names.append(srpm_name)
new_srpm_names.add(srpm_name)
for dep in dependencies:
dep_map[name].setdefault(
srpm_name, OrderedDict()
).setdefault(pkg, set()).add(dep)
for new_srpm_name in new_srpm_names:
self.dep_chain[new_srpm_name].add(check_next)
self.pagureinfo_queue.put(new_srpm_name)
ignore.extend(new_names)
if allow_more:
to_check.extend(new_names)
found_deps = dep_map[name].keys()
dep_count = len(set(found_deps) | set(to_check))
if dep_count > max_deps:
todo_deps = max_deps - len(found_deps)
if todo_deps < 0:
todo_deps = 0
incomplete.append(name)
eprint(f"Dep count is {dep_count}")
eprint(f"incomplete is {incomplete}")
allow_more = False
to_check = to_check[0:todo_deps]
if not to_check:
break
if not allow_more:
eprint(
f"More than {max_deps} broken deps for package "
f"'{name}', dependency check not completed"
)
eprint("Waiting for (co)maintainer information...", end=" ")
self.pagureinfo_queue.join()
eprint("done")
return dep_map, incomplete
def maintainer_table(
packages, pagure_dict
) -> tuple[Any, dict[str, set[str]], list[str]]:
affected_people: dict[str, set[str]] = {}
all_addresses: set[str] = set()
if with_table:
table = texttable.Texttable(max_width=80)
table.header(["Package", "(co)maintainers", "Status Change"])
table.set_cols_align(["l", "l", "l"])
table.set_deco(table.HEADER)
else:
table = ""
for package_name in packages:
pkginfo = pagure_dict[package_name]
people, addresses = pkginfo.get_people_and_emails()
all_addresses.update(addresses)
for p in people:
affected_people.setdefault(p, set()).add(package_name)
p = ", ".join(people)
age = pkginfo.age
agestr = f"{age.days // 7} weeks ago"
if with_table:
table.add_row([package_name, p, agestr])
else:
table += f"{package_name} {p} {agestr}\n"
all_addresses.discard(f"{ORPHAN_UID}@fedoraproject.org")
if with_table:
table = table.draw()
return table, affected_people, sorted(all_addresses)
def dependency_info(dep_map, affected_people, pagure_dict, incomplete):
info = ""
for package_name, subdict in dep_map.items():
if subdict:
pkginfo = pagure_dict[package_name]
status_change = pkginfo.status_change.strftime("%Y-%m-%d")
age = pkginfo.age.days // 7
fmt = "Depending on: {} ({}), status change: {} ({} weeks ago)\n"
info += fmt.format(package_name, len(subdict.keys()), status_change, age)
for fedora_package, dependent_packages in subdict.items():
people, _ = pagure_dict[fedora_package].get_people_and_emails()
for p in people:
affected_people.setdefault(p, set()).add(package_name)
p = ", ".join(people)
info += f"\t{fedora_package} (maintained by: {p})\n"
for dep in dependent_packages:
provides = ", ".join(sorted(dependent_packages[dep]))
info += f"\t\t{dep} requires {provides}\n"
info += "\n"
if package_name in incomplete:
info += f"\tToo many dependencies for {package_name}, "
info += "not all listed here\n\n"
return info
def maintainer_info(affected_people):
info = ""
for person in sorted(affected_people):
packages = affected_people[person]
if person == ORPHAN_UID:
continue
info += f"{person}: {', '.join(packages)}\n"
return info
def stream_to_set(stream: IO[str]) -> set[str]:
with stream:
result: set[str] = set()
for line in stream:
result.add(line.strip())
return result
def get_golang_exemptions(package_data_path: Path, packages: list[str]) -> list[str]:
result: list[str] = []
all_golang = stream_to_set(package_data_path.joinpath("all_packages").open())
not_exempt = stream_to_set(
package_data_path.joinpath("fesco_3447_not_exempt").open()
)
for package in packages:
if package in all_golang and package not in not_exempt:
result.append(package)
return result
class PackageInfoDict(TypedDict):
"""
Represents a JSON blob containing a info about orphaned packages
"""
affected_people: dict[str, list[str]]
addresses: list[str]
orphans: list[str]
orphans_breaking_deps: list[str]
orphans_breaking_deps_stale: list[str]
orphans_not_breaking_deps: list[str]
orphans_not_breaking_deps_stale: list[str]
ftbfs_breaking_deps: list[str]
ftbfs_not_breaking_deps: list[str]
# Superset of affected_people and co-maintainers of dependencies
all_affected_people: dict[str, list[str]]
# Only included when the script is run with a path to the Go package data
golang_exemptions: NotRequired[list[str]]
def package_info(
unblocked,
dep_map,
depchecker,
orphans=None,
failed=None,
week_limit=6,
release="",
incomplete=[],
go_package_data: Path | None = None,
) -> tuple[str, PackageInfoDict]:
info = ""
info_dict: dict[str, Any] = {}
pagure_dict = depchecker.pagure_dict
table, affected_people, addresses = maintainer_table(unblocked, pagure_dict)
info_dict["affected_people"] = {
key: list(value) for key, value in affected_people.items()
}
info_dict["addresses"] = addresses
info += table
info += "\n\nThe following packages require above mentioned packages:\n"
info += dependency_info(dep_map, affected_people, pagure_dict, incomplete)
info_dict["all_affected_people"] = {
key: list(value) for key, value in affected_people.items()
}
info += "Affected (co)maintainers\n"
info += maintainer_info(affected_people)
if release:
release_text = f" ({release})"
branch = RELEASES[release]["pagure_branch"]
else:
release_text = ""
wrapper = textwrap.TextWrapper(
break_long_words=False, subsequent_indent=" ", break_on_hyphens=False
)
def wrap_and_format(label, pkgs):
count = len(pkgs)
text = f"{label} ({count}): {' '.join(pkgs)}"
wrappedtext = "\n" + wrapper.fill(text) + "\n\n"
return wrappedtext
if orphans:
orphans = [o for o in orphans if o in unblocked]
info_dict["orphans"] = orphans
info += wrap_and_format("Orphans", orphans)
orphans_breaking_deps = [o for o in orphans if dep_map.get(o)]
info_dict["orphans_breaking_deps"] = orphans_breaking_deps
info += wrap_and_format("Orphans (dependend on)", orphans_breaking_deps)
orphans_breaking_deps_stale = [
o
for o in orphans_breaking_deps
if (pagure_dict[o].age.days // 7) >= week_limit
]
info_dict["orphans_breaking_deps_stale"] = orphans_breaking_deps_stale
info += wrap_and_format(
f"Orphans{release_text} for at least {week_limit} " "weeks (dependend on)",
orphans_breaking_deps_stale,
)
orphans_not_breaking_deps = [o for o in orphans if not dep_map.get(o)]
info_dict["orphans_not_breaking_deps"] = orphans_not_breaking_deps
info += wrap_and_format(
f"Orphans{release_text} (not depended on)", orphans_not_breaking_deps
)
orphans_not_breaking_deps_stale = [
o
for o in orphans_not_breaking_deps
if (pagure_dict[o].age.days // 7) >= week_limit
]
info_dict["orphans_not_breaking_deps_stale"] = orphans_not_breaking_deps_stale
if orphans_not_breaking_deps_stale:
eprint(
f"fedretire --orphan --branch {branch} -- "
+ " ".join(orphans_not_breaking_deps_stale)
)
info += wrap_and_format(
f"Orphans{release_text} for at least {week_limit} "
"weeks (not dependend on)",
orphans_not_breaking_deps_stale,
)
breaking: set[str] = set()
for package, deps in dep_map.items():
breaking = breaking.union(set(deps))
if breaking:
info += wrap_and_format("Depending packages" + release_text, sorted(breaking))
if orphans:
reverse_deps: dict[str, list[str]] = OrderedDict()
stale_breaking: set[str] = set()
for package in orphans_breaking_deps_stale:
for depender in dep_map[package]:
reverse_deps.setdefault(depender, []).append(package)
stale_breaking = stale_breaking.union(set(dep_map[package].keys()))
for depender, providers in reverse_deps.items():
eprint(
f"fedretire --orphan-dependent {' '.join(providers)} "
f"--branch {branch} -- {depender}"
)
for providingpkg in providers:
eprint("fedretire --orphan --branch " f"{branch} -- {providingpkg}")
info += wrap_and_format(
f"Packages depending on packages orphaned{release_text} "
f"for more than {week_limit} weeks",
sorted(stale_breaking),
)
if failed:
ftbfs_label = "FTBFS" + release_text
info += wrap_and_format(ftbfs_label, failed)
ftbfs_breaking_deps = [o for o in failed if o in dep_map and dep_map[o]]
info_dict["ftbfs_breaking_deps"] = ftbfs_breaking_deps
info += wrap_and_format(ftbfs_label + " (depended on)", ftbfs_breaking_deps)
ftbfs_not_breaking_deps = [
o for o in failed if o not in dep_map or not dep_map[o]
]
info_dict["ftbfs_not_breaking_deps"] = ftbfs_not_breaking_deps
info += wrap_and_format(
ftbfs_label + " (not depended on)", ftbfs_not_breaking_deps
)
if depchecker.not_in_repo:
info += wrap_and_format(
"Not found in repo" + release_text, sorted(depchecker.not_in_repo)
)
if go_package_data:
info_dict["golang_exemptions"] = get_golang_exemptions(
go_package_data, unblocked
)
info += wrap_and_format(
f"Golang orphans{release_text} that are exempted from retirement",
info_dict["golang_exemptions"],
)
return info, cast(PackageInfoDict, info_dict)
def main():
parser = argparse.ArgumentParser()
parser.add_argument(
"--skip-orphans",
dest="skip_orphans",
help="Do not look for orphans",
default=False,
action="store_true",
)
parser.add_argument(
"--max_deps",
dest="max_deps",
type=int,
help="set max_deps on recursive find deps",
default=20,
)
parser.add_argument("--release", choices=RELEASES.keys(), default="rawhide")
parser.add_argument(
"--mailto", default=None, help="Send mail to this address (for testing)"
)
parser.add_argument(
"--send",
default=False,
action="store_true",
help="Actually send mail including Bcc addresses to mailing list",
)
parser.add_argument(
"--source-repo", default=None, help="Source repo URL to use for depcheck"
)
parser.add_argument("--repo", default=None, help="Repo URL to use for depcheck")
parser.add_argument(
"--json",
default=None,
help="Export info about orphaned " "packages to a specified JSON file",
)
parser.add_argument(
"--no-skip-blocked",
default=True,
dest="skipblocked",
action="store_false",
help="Do not skip blocked pkgs",
)
parser.add_argument("--mailfrom", default="nobody@fedoraproject.org")
filetype = argparse.FileType("w", encoding="utf-8")
parser.add_argument(
"-o",
"--output",
help="Output report to a text file",
type=filetype,
default=filetype("-"),
)
parser.add_argument(
"failed", nargs="*", help="Additional packages, e.g. FTBFS packages"
)
go_package_data_env = os.environ.get("GO_PACKAGE_DATA")
parser.add_argument(
"--go-package-data",
help="Path to https://gitlab.com/fedora/sigs/go/package-data checkout",
default=Path(go_package_data_env) if go_package_data_env else None,
type=Path,
)
args = parser.parse_args()
failed = args.failed
if args.source_repo is not None:
RELEASES[args.release]["source_repo"] = args.source_repo
if args.repo is not None:
RELEASES[args.release]["repo"] = args.repo
started_at = datetime.datetime.now(datetime.timezone.utc)
text = "Report started at %s\n\n" % (started_at.strftime("%Y-%m-%d %H:%M:%S UTC"))
if args.skip_orphans:
orphans = []
unblocked = failed
else:
# list of orphans from pagure
eprint("Contacting pagure for list of orphans...", end=" ")
orphans = sorted(orphan_packages())
eprint("done")
allpkgs = sorted(list(set(list(orphans) + failed)))
if args.skipblocked:
eprint("Getting builds from koji...", end=" ")
koji_tag = RELEASES[args.release]["koji_tag"]
koji_hub = RELEASES[args.release]["koji_hub"]
unblocked = unblocked_packages(allpkgs, tagID=koji_tag, kojihub=koji_hub)
eprint("done")
text += HEADER.format(RELEASES[args.release]["koji_tag"].upper())
eprint("Setting up dependency checker...", end=" ")
depchecker = DepChecker(args.release)
eprint("done")
eprint("Calculating dependencies...", end=" ")
# Create dnf object and depsolve out if requested.
# TODO: add app args to either depsolve or not
dep_map, incomplete = depchecker.recursive_deps(unblocked, args.max_deps)
eprint("done")
info, package_info_dict = package_info(
unblocked,
dep_map,
depchecker,
orphans=orphans,
failed=failed,
release=args.release,
incomplete=incomplete,
go_package_data=args.go_package_data,
)
addresses = package_info_dict["addresses"]
text += "\n"
text += info
text += FOOTER
finished_at = datetime.datetime.now(datetime.timezone.utc)
text += "\nReport finished at %s" % (finished_at.strftime("%Y-%m-%d %H:%M:%S UTC"))
args.output.write(text + "\n")
if args.json is not None:
eprint(f"Saving {args.json} with machine readable info")
sc = {
pkg: depchecker.pagure_dict[pkg].status_change.isoformat()
for pkg in orphans
if pkg in depchecker.pagure_dict
}
ap = {pkg: sorted(reasons) for pkg, reasons in depchecker.dep_chain.items()}
json_data = {
"status_change": sc,
"affected_packages": ap,
"started_at": started_at.isoformat(),
"finished_at": finished_at.isoformat(),
**dict(package_info_dict),
}
try:
with open(args.json, "w") as f:
json.dump(json_data, f, indent=4, sort_keys=True)
except OSError as e:
eprint(f"Cannot save {args.json}:", end=" ")
eprint(f"{type(e).__name__}: e")
if args.mailto or args.send:
now = datetime.datetime.now(datetime.timezone.utc)
today = now.strftime("%Y-%m-%d")
subject = f"Orphaned Packages in {args.release} ({today})"
if args.mailto:
mailto = args.mailto
else:
mailto = RELEASES[args.release]["mailto"]
if args.send:
bcc = addresses + RELEASES[args.release]["bcc"]
else:
bcc = None
mail_errors = send_mail(args.mailfrom, mailto, subject, text, bcc)
if mail_errors:
eprint("mail errors: " + repr(mail_errors))
eprint(f"Addresses ({len(addresses)}):", ", ".join(addresses))
if __name__ == "__main__":
main()

14
scripts_new/packages/orphaned/pyproject.toml
View file

@ -1,14 +0,0 @@
[tool.black]
line-length = 89
[tool.isort]
profile = "black"
[[tool.mypy.overrides]]
module = [
"bugzilla",
"dnf.*",
"koji.*",
"texttable.*",
]
ignore_missing_imports = true

8
scripts_new/packages/orphaned/requirements.txt
View file

@ -1,8 +0,0 @@
# retire.py
python-bugzilla
click
# find_unblocked_orphans.py
dogpile.cache
koji
requests
texttable

249
scripts_new/packages/orphaned/retire.py
View file

@ -1,249 +0,0 @@
#!/usr/bin/env python3
# Copyright (C) 2024 Maxwell G <maxwell@gtmx.me>
# SPDX-License-Identifier: GPL-2.0-or-later
from __future__ import annotations
import dataclasses
import datetime
import json
import os.path
import subprocess
from collections.abc import Iterator, Sequence
from contextlib import AbstractContextManager, nullcontext
from functools import partial
from tempfile import TemporaryDirectory
from typing import IO, TYPE_CHECKING
from urllib.parse import urljoin
import click
import requests
import requests.adapters
from bugzilla import Bugzilla
if TYPE_CHECKING:
from _typeshed import StrPath
from find_unblocked_orphans import PackageInfoDict
JSON_DOWNLOAD = "https://a.gtmx.me/orphans/orphans.json"
ORPHAN_UID = "orphan"
PACKAGE_API_URL = "https://src.fedoraproject.org/api/0/rpms/"
BUGZILLA_API_URL = "https://bugzilla.redhat.com"
DEFAULT_DISTGIT_MESSAGE = "Orphaned for 6+ weeks"
TEMPLATE = """Automation has figured out the package is retired in Fedora {}.
If you like it to be unretired, please open a ticket at
https://pagure.io/releng/new_issue?template=package_unretirement
"""
def get_requests_session() -> requests.Session:
session = requests.Session()
retry = requests.adapters.Retry()
for protocol in "http://", "https://":
session.mount(protocol, requests.adapters.HTTPAdapter(max_retries=retry))
return session
bz_session = Bugzilla(BUGZILLA_API_URL)
session = get_requests_session()
def run(
cmd: Sequence[StrPath],
*,
capture_text: bool = False,
dry_run=False,
log: bool = True,
**kwargs,
) -> subprocess.CompletedProcess | None:
kwargs.setdefault("check", True)
if capture_text:
kwargs["text"] = True
kwargs["capture_output"] = True
if log:
start = "Would run" if dry_run else "Running"
click.secho(
f"* {start}: {tuple(map(str, cmd))}",
err=True,
fg="yellow" if dry_run else "blue",
)
if dry_run:
return None
return subprocess.run(cmd, **kwargs) # noqa: PLW1510
@dataclasses.dataclass()
class CLIContext:
json_data: PackageInfoDict
package_list: Sequence[str] | None = None
@property
def orphans_stale(self) -> list[str]:
if self.package_list is not None:
return sorted(self.package_list)
return sorted(
{
*self.json_data["orphans_breaking_deps_stale"],
*self.json_data["orphans_not_breaking_deps_stale"],
}
)
def package_iter(self, check: bool = True) -> Iterator[str]:
for package in self.orphans_stale:
if check and (owner := ensure_orphaned(package)):
click.secho(f"! {package} is owned by {owner}", fg="red", err=True)
continue
yield package
def ensure_orphaned(package: str) -> str | None:
req = session.get(urljoin(PACKAGE_API_URL, package))
req.raise_for_status()
owner = req.json()["user"]["name"]
return owner if owner != ORPHAN_UID else None
pass_obj = click.make_pass_decorator(CLIContext, True)
@click.group(
context_settings={"help_option_names": ["-h", "--help"], "show_default": True}
)
@click.option("--json", "json_file", default=JSON_DOWNLOAD)
@click.option("--list", "package_list_file", type=click.File())
@click.pass_context
def main(ctx: click.Context, json_file: str, package_list_file: IO[str] | None) -> None:
if json_file.startswith(("http://", "https://")):
data = session.get(json_file).json()
else:
with open(json_file, "r", encoding="utf-8") as fp:
data = json.load(fp)
package_list: list[str] | None = None
if package_list_file:
package_list = [p.strip() for p in package_list_file]
package_list_file.close()
ctx.obj = CLIContext(json_data=data, package_list=package_list)
CHECK_OPT = partial(
click.option,
"--check / --no-check",
default=False,
help="Whether to check that packages are actually orphaned",
)
@main.command(name="list")
@CHECK_OPT(default=False)
@pass_obj
def list_command(args: CLIContext, check: bool) -> None:
"""
List files that have been orphaned for over 6 weeks
"""
for package in args.package_iter(check):
click.echo(package)
def retire_distgit(
package: str,
directory: str,
dry_run: bool,
branches: Sequence[str] = (),
message: str | None = None,
) -> None:
dir = os.path.join(directory, package)
in_dir = partial(run, dry_run=dry_run, cwd=dir)
run(["fedpkg", "clone", package, dir])
in_dir(["fedpkg", "retire", message or DEFAULT_DISTGIT_MESSAGE])
for branch in branches:
run(["git", "switch", branch], cwd=dir)
in_dir(["git", "merge", "--gpg-sign", "rawhide"])
in_dir(["git", "push"])
def retire_bugs(package: str, dry_run: bool, branches: Sequence[str] = ()) -> None:
branches = ("rawhide", *branches)
for branch in branches:
query = bz_session.build_query(
product="Fedora",
version=branch.lstrip("f"),
component=package,
status=["__open__"],
)
bugs = bz_session.query(query)
bug_ids: list[str] = []
for bug in bugs:
begin = "Would close" if dry_run else "Closing"
fg = "yellow" if dry_run else "blue"
click.secho(f"* {begin}: {bug.id} --- {bug.short_desc}", err=True, fg=fg)
bug_ids.append(bug.id)
if not dry_run:
update = bz_session.build_update(
comment=TEMPLATE.format(branch.title()),
status="CLOSED",
resolution="WONTFIX",
)
bz_session.update_bugs(bug_ids, update)
def retire(
package: str,
directory: str,
dry_run: bool,
branches: Sequence[str] = (),
message: str | None = None,
) -> None:
click.secho("distgit", bold=True)
retire_distgit(package, directory, dry_run, branches, message)
click.secho("bugzilla", bold=True)
retire_bugs(package, dry_run, branches)
@main.command(name="retire")
@click.option("-n", "--dry-run", default=False, is_flag=True)
@click.option("--workdir")
@click.option("-b", "--branch", "branches", multiple=True)
@click.option(
"--lf",
"--log-file",
"log_file",
help="Append a list of retired packages to a file",
type=click.File("a"),
)
@click.option("--message", help="Message to use for distgit retirement commit")
@CHECK_OPT(default=True)
@pass_obj
def retire_command(
args: CLIContext,
dry_run: bool,
workdir: str,
branches: Sequence[str],
check: bool,
log_file: IO[str] | None,
message: str | None,
) -> None:
"""
Retire packages that have been orphaned for over 6 weeks
"""
if workdir:
os.makedirs(workdir)
cm: AbstractContextManager[str] = (
nullcontext(workdir) if workdir else TemporaryDirectory() # type: ignore[assignment]
)
now = format(datetime.datetime.now(datetime.timezone.utc), "%Y-%m-%d")
with cm as workdir:
for package in args.package_iter(check):
click.secho(f"{package}", underline=True, fg="green")
retire(package, workdir, dry_run, branches, message)
if log_file:
log_file.write(f"{package} {now}\n")
click.echo()
if __name__ == "__main__":
main()

35
scripts_new/packages/orphaned/tox.ini
View file

@ -1,35 +0,0 @@
[tox]
env_list =
formatters
lint
typing
[testenv:formatters]
description = Run formatters
skip_install = true
deps =
isort
black
commands =
black {posargs} find_unblocked_orphans.py retire.py
isort {posargs} find_unblocked_orphans.py retire.py
[testenv:lint]
description = Run linters
skip_install = true
deps =
ruff
commands =
ruff check {posargs} find_unblocked_orphans.py retire.py
[testenv:typing]
description = Run type checkers
skip_install = true
deps =
-r requirements.txt
mypy
types-requests
commands =
mypy {posargs} find_unblocked_orphans.py retire.py
set_env =
PYTHONPATH=${PWD}

15
scripts_new/quality-assurance/README.md
View file

@ -1,15 +0,0 @@
# Quality Assurance Tools
Scripts for testing, validation, and quality control in Fedora releases.
## Subdirectories
- **[testing/](testing/)** - Build and release testing tools
- **[critpath/](critpath/)** - Critical path package management
- **[ftbfs/](ftbfs/)** - Failure to Build From Source (FTBFS) handling
## Purpose
Ensures quality and reliability of Fedora releases through automated testing and validation.
**Status**: 🚧 Migration in progress

0
scripts_new/quality-assurance/critpath/.gitignore vendored
View file

427
scripts_new/quality-assurance/critpath/critpath.py
View file

@ -1,427 +0,0 @@
#!/usr/bin/python3
#
# Copyright (C) 2013 Red Hat Inc,
# SPDX-License-Identifier: GPL-2.0+
#
# Authors: Will Woods <wwoods@redhat.com>
# Seth Vidal <skvidal@fedoraproject.org>
# Robert Marshall <rmarshall@redhat.com>
# Adam Williamson <awilliam@redhat.com>
# this is a script, not a public module, we don't need docstrings
# pylint: disable=missing-module-docstring,missing-class-docstring,missing-function-docstring
import sys
import argparse
from collections import defaultdict
import json
import yaml
import shutil
from tempfile import mkdtemp
from urllib.request import urlopen
import dnf
import dnf.exceptions
class SackError(Exception):
pass
# Set some constants
# **IMPORTANT**: before adding any group to this list, ensure the
# corresponding decision context is in at least one policy in the
# Greenwave configuration:
# https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/greenwave/templates/fedora.yaml
# bad things will happen if any update is in a critical path group
# with no corresponding greenwave policy. If no gating is required
# for packages in the group, the context only needs to be added to
# the "null" policies at the top of the file
CRITPATH_GROUPS = [
"@core",
"@critical-path-anaconda",
"@critical-path-apps",
"@critical-path-base",
"@critical-path-build",
"@critical-path-cloud",
"@critical-path-compose",
"@critical-path-gnome",
"@critical-path-kde",
"@critical-path-lxde",
"@critical-path-lxqt",
"@critical-path-server",
"@critical-path-standard",
"@critical-path-xfce",
]
# these are the only groups we currently want to mark ELN updates as
# critpath for
ELN_CRITPATH_GROUPS = [
"@core",
"@critical-path-anaconda",
"@critical-path-base",
"@critical-path-compose"
]
PRIMARY_ARCHES = ("aarch64", "x86_64")
ALTERNATE_ARCHES = ("ppc64le", "s390x")
BODHI_RELEASEURL = "https://bodhi.fedoraproject.org/releases/?rows_per_page=500"
FEDORA_BASEURL = "http://dl.fedoraproject.org/pub/fedora/linux/"
FEDORA_ALTERNATEURL = "http://dl.fedoraproject.org/pub/fedora-secondary/"
# note this name is synthetic; it does not correspond to an actual comps group
COREOS_CRITPATH_GROUP = "critical-path-coreos"
COREOS_CONFIG_URL = "https://raw.githubusercontent.com/coreos/coreos-ci/main/bodhi-testing.yaml"
# used as a cache by get_coreos_gating_data
COREOS_GATED_SRPMS = None
# used as a cache by get_bodhi_releases
BODHIRELEASES = {}
def get_bodhi_releases():
global BODHIRELEASES
if not BODHIRELEASES:
bodhijson = json.loads(urlopen(BODHI_RELEASEURL).read().decode("utf8"))["releases"]
devrels = {
int(rel['version']) for rel in bodhijson if rel['state'] in ('pending', 'frozen') and
rel['id_prefix'] == 'FEDORA' and rel["version"].isdigit()
}
if devrels:
BODHIRELEASES[str(max(devrels))] = "rawhide"
if len(devrels) > 1:
BODHIRELEASES[str(min(devrels))] = "branched"
stabrels = {
int(rel['version']) for rel in bodhijson if rel['state'] == 'current' and
rel['id_prefix'] == 'FEDORA' and rel["version"].isdigit()
}
for relnum in stabrels:
BODHIRELEASES[str(relnum)] = "stable"
return BODHIRELEASES
def get_coreos_gated_srpms():
global COREOS_GATED_SRPMS
if COREOS_GATED_SRPMS is None:
config_data = yaml.safe_load(urlopen(COREOS_CONFIG_URL).read().decode("utf8"))
COREOS_GATED_SRPMS = config_data.get("gated-srpms", [])
return COREOS_GATED_SRPMS
def get_paths(release, forcebranched=False):
"""This does a certain amount of fudging so we can refer to
Branched by its release number or "branched", and Rawhide by its
release number or "rawhide" or "devel".
"""
relnums = get_bodhi_releases()
if relnums.get(release) == "stable" and not forcebranched:
return (
f"releases/{release}/Everything/$basearch/os",
f"updates/{release}/Everything/$basearch"
)
elif release in ("rawhide", "devel") or relnums.get(release) == "rawhide":
return ("development/rawhide/Everything/$basearch/os", "")
elif release == "branched" or relnums.get(release) == "branched" or forcebranched:
if release == "branched":
try:
release = [relnum for relnum in relnums if relnums[relnum] == "branched"][0]
except IndexError:
raise ValueError("Cannot find a branched release.")
return (f"development/{release}/Everything/$basearch/os", "")
raise ValueError(f"Unrecognized release {release}.")
def get_source(pkg):
return pkg.rsplit("-", 2)[0]
def nvr(pkg):
return "-".join([pkg.name, pkg.ver, pkg.rel])
def expand_dnf_critpath(urls, arch):
print(f"Resolving {arch} dependencies with DNF")
base = dnf.Base()
temp_cache_dir = mkdtemp(suffix="-critpath")
temp_install_root = mkdtemp(suffix="-critpath-installroot")
conf = base.conf
# cache download data somewhere else
conf.cachedir = temp_cache_dir
# do not use the data from the previous runs of system dnf or groups will
# be marked incorrectly
conf.persistdir = temp_cache_dir
conf.installroot = temp_install_root
conf.arch = arch
packages = dict()
try:
# add a new repo requires an id, a conf object, and a baseurl
# make sure we don't load the system repo and get local data
print(f"Basearch: {conf.basearch}")
print(f"Arch: {conf.arch}")
for url in urls:
print(f"Repo: {url}")
# mark all critpath groups in base object
for group in CRITPATH_GROUPS:
base.reset(repos=True, goal=True, sack=True)
repoids = []
for (num, url) in enumerate(urls):
repoid = arch + str(num)
repoids.append(repoid)
base.repos.add_new_repo(repoid, conf, baseurl=[url])
base.fill_sack(load_system_repo=False)
for repoid in repoids:
if base.repos[repoid].enabled is False:
raise SackError
# load up the comps data from configured repositories
base.read_comps()
group = group.replace("@", "")
try:
base.group_install(group, ["mandatory", "default", "optional"], strict=False)
except dnf.exceptions.CompsError as err:
if str(err).startswith("Group id") and str(err).endswith("does not exist."):
print(f"Warning: group {group} does not exist for arch {conf.arch}")
continue
# resolve the groups marked in base object
base.resolve()
packages[group] = base.transaction.install_set
return packages
finally:
base.close()
del base
del conf
shutil.rmtree(temp_cache_dir)
shutil.rmtree(temp_install_root)
def parse_args():
releases = sorted(get_bodhi_releases().keys())
releases.extend(["branched", "devel", "rawhide", "all"])
parser = argparse.ArgumentParser()
mexcgroup = parser.add_mutually_exclusive_group()
parser.add_argument(
"release",
choices=releases,
help="The release to work on (a release number, 'branched', 'rawhide', or 'all'). In "
"'all' mode, work on all current and pending releases, naming files in the format "
"expected by Bodhi).",
)
mexcgroup.add_argument(
"--nvr",
action="store_true",
default=False,
help="output full NVR instead of just package name",
)
mexcgroup.add_argument(
"--srpm",
action="store_true",
default=False,
help="Output source RPMS instead of binary RPMS (for uploading to PDC)",
)
parser.add_argument(
"-a",
"--arches",
default=",".join(PRIMARY_ARCHES),
help="Primary arches to evaluate (%(default)s)",
)
parser.add_argument(
"-s",
"--altarches",
default=",".join(ALTERNATE_ARCHES),
help="Alternate arches to evaluate (%(default)s)",
)
parser.add_argument(
"-o",
"--output",
default="critpath.txt",
help="name of file to write flat plaintext critpath list (ignored for 'all') (%(default)s)",
)
parser.add_argument(
"-j",
"--jsonout",
default="critpath.json",
help="name of file to write grouped JSON critpath list (ignored for 'all') (%(default)s)",
)
parser.add_argument(
"-u",
"--url",
default=FEDORA_BASEURL,
help="URL to fedora/linux directory for primary arches",
)
parser.add_argument(
"-r",
"--alturl",
default=FEDORA_ALTERNATEURL,
help="URL to fedora-secondary directory for alternate arches",
)
parser.add_argument(
"-c",
"--composeurl",
required=False,
help="URL to a complete (not arch split) compose, overrides -u and -r",
)
parser.add_argument(
"--noaltarch",
action="store_true",
default=False,
help="Not to run for alternate architectures",
)
parser.add_argument(
"--with-coreos",
action="store_true",
default=False,
help="Add packages configured as gating in CoreOS CI config",
)
return parser.parse_args()
# Things that aren't as easy to express via argparse
def validate_args(args):
if args.with_coreos and not args.srpm:
raise Exception("--with-coreos requires --srpm")
def write_files(critpath, outpath, jsonout):
wrapped = {"rpm": critpath}
with open(jsonout, mode="w", encoding="utf-8") as jsonoutfh:
json.dump(wrapped, jsonoutfh, sort_keys=True, indent=4)
print(f"Wrote grouped critpath data to {jsonout}")
if jsonout == "rawhide.json":
# kinda ugly hack so we get critpath data for ELN
# https://pagure.io/releng/issue/12303
elnfiltered = {
group: critpath[group] for group in critpath if f"@{group}" in ELN_CRITPATH_GROUPS
}
elnwrapped = {"rpm": elnfiltered}
with open("eln.json", mode="w", encoding="utf-8") as elnoutfh:
json.dump(elnwrapped, elnoutfh, sort_keys=True, indent=4)
print(f"Also wrote ELN-filtered grouped critpath data to eln.json")
pkgs = set()
for grppkgs in critpath.values():
pkgs = pkgs.union(set(grppkgs))
with open(outpath, mode="w", encoding="utf-8") as outfh:
for packagename in sorted(pkgs):
outfh.write(packagename + "\n")
package_count = len(pkgs)
outtext = f"Wrote {package_count} items to {outpath}"
if outpath == "rawhide.txt":
# kinda ugly hack so we get critpath data for ELN
# https://pagure.io/releng/issue/12303
elnpkgs = set()
for grppkgs in elnfiltered.values():
elnpkgs = elnpkgs.union(set(grppkgs))
with open("eln.txt", mode="w", encoding="utf-8") as elnoutfh:
for packagename in sorted(elnpkgs):
elnoutfh.write(packagename + "\n")
elnpkg_count = len(elnpkgs)
outtext += f" and {elnpkg_count} items to eln.txt"
print(outtext)
def generate_critpath(release, args, output, jsonout, forcebranched=False):
check_arches = args.arches.split(",")
alternate_check_arches = args.altarches.split(",")
package_count = 0
updateurl = None
updatealturl = None
if args.composeurl:
baseurl = args.composeurl + "/Everything/$basearch/os"
alturl = args.composeurl + "/Everything/$basearch/os"
else:
paths = get_paths(release, forcebranched=forcebranched)
baseurl = args.url + paths[0]
alturl = args.alturl + paths[0]
if paths[1]:
updateurl = args.url + paths[1]
updatealturl = args.alturl + paths[1]
print(f"Using Base URL {baseurl}")
print(f"Using alternate arch base URL {alturl}")
if updateurl:
print(f"Using update URL {updateurl}")
if updatealturl:
print(f"Using alternate arch update URL {updatealturl}")
# Do the critpath expansion for each arch
critpath = defaultdict(set)
for arch in check_arches + alternate_check_arches:
urls = [baseurl, updateurl]
if arch in alternate_check_arches:
if args.noaltarch:
continue
urls = [alturl, updatealturl]
# strip None update URLs when we're not using them
urls = [url for url in urls if url]
print(f"Expanding critical path for {arch}")
try:
pkgdict = expand_dnf_critpath(urls, arch)
except dnf.exceptions.RepoError:
# this is a dumb workaround for the 'interregnum problem'
# where for a few days each cycle a new release is marked
# stable in bodhi, but isn't actually in the stable path
# on the mirror yet. this should never recurse because
# releases/ isn't in the branched base URL, but just in
# case, check forcebranched too
if "releases/" in baseurl and not forcebranched:
print(f"Failed to find release {release} at stable path {baseurl}!")
print("Trying branched path instead...")
return generate_critpath(release, args, output, jsonout, forcebranched=True)
raise
for (group, pkgs) in pkgdict.items():
package_count = len(pkgs)
print(f"{package_count} packages in {group} for {arch}")
if args.nvr:
critpath[group].update([nvr(pkg) for pkg in pkgs])
elif args.srpm:
critpath[group].update([get_source(pkg.sourcerpm) for pkg in pkgs])
else:
critpath[group].update([pkg.name for pkg in pkgs])
# note for these, we don't do any depsolving
if args.with_coreos:
coreos_gated_srpms = get_coreos_gated_srpms()
critpath[COREOS_CRITPATH_GROUP].update([srpm['name'] for srpm in coreos_gated_srpms])
del pkgdict
print()
# Turn sets back into lists (so we can JSON-dump them)
for group in critpath:
critpath[group] = sorted(critpath[group])
write_files(critpath, output, jsonout)
def main():
args = parse_args()
validate_args(args)
release = args.release
if release == "all":
relnums = get_bodhi_releases()
for release in relnums:
print(f"Working on release {release}")
# the name expected by Bodhi is '[gitbranchname].json';
# for most releases this is 'f[relnum].json' but for
# Rawhide it is 'rawhide.json'
if relnums[release] == "rawhide":
generate_critpath(release, args, "rawhide.txt", "rawhide.json")
else:
generate_critpath(release, args, f"f{release}.txt", f"f{release}.json")
else:
generate_critpath(release, args, args.output, args.jsonout)
if __name__ == "__main__":
try:
main()
except KeyboardInterrupt:
sys.stderr.write("Interrupted, exiting...\n")
sys.exit(1)
# vim: set textwidth=100 ts=8 et sw=4:

0
scripts_new/quality-assurance/testing/.gitignore vendored
View file

18
scripts_new/release-process/README.md
View file

@ -1,18 +0,0 @@
# Release Process Management
Scripts that orchestrate and manage Fedora release workflows.
## Subdirectories
- **[mass-rebuilds/](mass-rebuilds/)** - Mass rebuild orchestration and coordination
- **[staging/](staging/)** - Release staging and preparation tools
- **[atomic/](atomic/)** - Atomic/OSTree release management
- **[mass-branching/](mass-branching/)** - Branch creation for new releases
- **[signing/](signing/)** - Package and release signing tools
- **[torrents/](torrents/)** - Release torrent generation
## Purpose
Contains scripts that manage the end-to-end release process, from mass rebuilds through final release publication.
**Status**: 🚧 Migration in progress

0
scripts_new/release-process/atomic/.gitignore vendored
View file

40
scripts_new/release-process/atomic/update_ostree_refs.sh
View file

@ -1,40 +0,0 @@
#!/usr/bin/bash
N=$1
# Define the artifacts and the architectures they support
declare -A artifacts
artifacts["silverblue"]="x86_64 aarch64 ppc64le"
artifacts["kinoite"]="x86_64 aarch64 ppc64le"
artifacts["onyx"]="x86_64"
artifacts["sericea"]="x86_64 aarch64"
OSTREE_COMPOSE_BASEDIR='/mnt/koji/compose/ostree/repo'
OSTREE_REPO_BASEDIR='/mnt/koji/ostree/repo/'
function do_things() {
for variant in "${!artifacts[@]}"; do
for arch in ${artifacts["$variant"]}; do
# Delete the existing update ref if it exists
sudo ostree refs --delete "fedora/${N}/${arch}/updates/${variant}" || true
# Create the new updates ref based on the main ref
sudo -u ftpsync ostree refs --create="fedora/${N}/${arch}/updates/${variant}" "fedora/${N}/${arch}/${variant}"
# Then delete the main ref
sudo ostree refs --delete "fedora/${N}/${arch}/${variant}"
# Then create a new main ref that is an alias to the updates ref
sudo -u ftpsync ostree refs --alias --create="fedora/${N}/${arch}/${variant}" "fedora/${N}/${arch}/updates/${variant}"
done
done
}
pushd $OSTREE_COMPOSE_BASEDIR
do_things
popd
pushd $OSTREE_REPO_BASEDIR
do_things
popd
pushd $OSTREE_REPO_BASEDIR
sudo ostree summary -u
popd

5
scripts_new/release-process/bug-filing/follow-policy/close-fti.j2
View file

@ -1,5 +0,0 @@
{% include "header.j2" %}
All subpackages of a package against which this bug was filled are now installable or removed from Fedora {{ release }}.
Thanks for taking care of it!

14
scripts_new/release-process/bug-filing/follow-policy/create-footer.j2
View file

@ -1,14 +0,0 @@
If you know about this problem and are planning on fixing it, please acknowledge so by setting the bug status to ASSIGNED. If you don't have time to maintain this package, consider orphaning it, so maintainers of dependent packages realize the problem.
If you don't react accordingly to the policy for FTBFS/FTI bugs (https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_to_install/), your package may be orphaned in 8+ weeks.
P.S. The data was generated solely from koji buildroot, so it might be newer than the latest compose or the content on mirrors. To reproduce, use the koji/local repo only, e.g. in mock:
$ mock -r fedora-{{ release }}-x86_64 --config-opts mirrored=False install{% for pkg in pkg_problems %} {{ pkg }}{% endfor %}
P.P.S. If this bug has been reported in the middle of upgrading multiple dependent packages, please consider using side tags: https://docs.fedoraproject.org/en-US/fesco/Updates_Policy/#updating-inter-dependent-packages
Thanks!

12
scripts_new/release-process/bug-filing/follow-policy/create-fti.j2
View file

@ -1,12 +0,0 @@
{% include "header.j2" %}
Your package ({{ src }}) Fails To Install in Fedora {{ release }}:
{% for pkg, problems in pkg_problems.items() -%}
can't install {{ pkg }}:
{% for problem in problems -%}
- {{ problem }}
{% endfor %}
{% endfor -%}
{% include "create-footer.j2" %}

440
scripts_new/release-process/bug-filing/follow-policy/follow_policy.py
View file

@ -1,440 +0,0 @@
#!/usr/bin/python3
import collections
import datetime
import os
import re
import sys
import bugzilla
import click
import jinja2
import requests
import solv
TEMPLATE_DIR = os.path.dirname(os.path.realpath(__file__))
# If this file exists, it will be used for authentication.
# If it does not exist, the default config file will be used.
# This allows to easily run this script as a dedicated Bugzilla user.
# See `man bugzilla` for what is supposed to be in that file.
BUGZILLA_CONFIG = os.path.expanduser('~/.config/python-bugzilla/bugzillarc-fti')
NOW = datetime.datetime.now(datetime.timezone.utc)
TRACKERS = {
"F32FailsToInstall": 1750909,
"F33FailsToInstall": 1803235,
"F34FailsToInstall": 1868279,
"F35FailsToInstall": 1927313,
"F36FailsToInstall": 1992487,
"F37FailsToInstall": 2045109,
"F38FailsToInstall": 2117177,
"F39FailsToInstall": 2168845,
"F40FailsToInstall": 2231790,
"F41FailsToInstall": 2260877,
"F42FailsToInstall": 2300529,
"F43FailsToInstall": 2339435,
"F44FailsToInstall": 2384425,
}
RAWHIDE = "44"
def _bzdate_to_python(date):
return datetime.datetime.strptime(str(date), "%Y%m%dT%H:%M:%S").replace(
tzinfo=datetime.timezone.utc
)
def handle_orphaning(bug, tracker, reminder_template):
bz = bug.bugzilla
history = bug.get_history_raw()["bugs"][0]["history"]
try:
start_time = _bzdate_to_python(
next(
u["when"]
for u in history
for c in u["changes"]
if c["field_name"] == "blocks"
and str(tracker.id) in {b.strip() for b in c["added"].split(",")}
)
)
except StopIteration:
start_time = _bzdate_to_python(bug.creation_time)
diff = NOW - start_time
if diff < datetime.timedelta(weeks=1):
print(
f"→ Week did not pass since bug started to block tracker ({start_time}), skipping…",
)
return
# Only reliable way to get whether needinfos were set is go through history
needinfos = [
u
for u in history
for c in u["changes"]
if f"needinfo?({bug.assigned_to})" in c["added"]
]
bzupdate = None
flag = {"name": "needinfo", "status": "?", "requestee": bug.assigned_to}
if not needinfos:
print("Asking for the first needinfo")
bzupdate = bz.build_update(
comment=reminder_template.render(nth="first", step=3, orphan_weeks=7),
flags=[flag],
)
else:
try:
needinfo_after_week = next(
_bzdate_to_python(n["when"])
for n in needinfos
if NOW - _bzdate_to_python(n["when"]) >= datetime.timedelta(weeks=1)
)
except StopIteration:
print(
f"→ Week did not pass since first needinfo ({_bzdate_to_python(needinfos[0]['when'])}), skipping…",
)
return
try:
needinfo_after_four_weeks = next(
_bzdate_to_python(n["when"])
for n in needinfos
if _bzdate_to_python(n["when"]) - needinfo_after_week
>= datetime.timedelta(weeks=3)
)
if NOW - needinfo_after_four_weeks >= datetime.timedelta(weeks=4):
print("Opening releng ticket")
print(f' * `{bug.component}` ([bug](https://bugzilla.redhat.com/show_bug.cgi?id={bug.id}))', file=sys.stderr)
else:
print(
f"→ 4 weeks did not pass since second needinfo ({needinfo_after_four_weeks}), skipping…",
)
return
except StopIteration:
if NOW - needinfo_after_week >= datetime.timedelta(weeks=3):
print("Asking for another needinfo")
bzupdate = bz.build_update(
comment=reminder_template.render(nth="second", step=4, orphan_weeks=4),
flags=[flag],
)
else:
print(
f"→ 3 weeks did not pass since first needinfo ({needinfo_after_week}), skipping…",
)
return
if bzupdate is not None:
result = bz.update_bugs([bug.id], bzupdate)
if "flags" in bzupdate and not result["bugs"][0]["changes"]:
# FIXME: Probably bug(s) in bugzilla and should be reported there
# 1. Accounts which change email do not force needinfo change
# 2. RHBZ can have multiple flags of the same type, but python-bugzilla does not like it much
# https://github.com/python-bugzilla/python-bugzilla/issues/118
flags = bzupdate["flags"]
flags_to_unset = [
f for f in bug.flags if f["name"] in set(f["name"] for f in flags)
]
flags = [f for f in bug.flags if f["name"] == "needinfo"]
if not flags_to_unset:
raise AssertionError(
"Flags update did not happen, neither there are flags to remove"
)
# If there are any needinfos, we will drop all of them and then create a new one
bz.update_bugs(
[bug.id],
bz.build_update(
flags=[
{"name": "needinfo", "id": f["id"], "status": "X"}
for f in flags
]
),
)
# Retry setting a flag
bz.update_bugs([bug.id], bz.build_update(flags=bzupdate["flags"]))
def find_broken_packages(pool):
solver = pool.Solver()
solver.set_flag(solv.Solver.SOLVER_FLAG_IGNORE_RECOMMENDED, True)
# Check for packages installability
candq = set(pool.solvables)
while candq:
jobs = [
pool.Job(
solv.Job.SOLVER_SOLVABLE
| solv.Job.SOLVER_INSTALL
| solv.Job.SOLVER_WEAK,
p.id,
)
for p in candq
]
solver.solve(jobs)
candq_n = candq - set(pool.id2solvable(s) for s in solver.raw_decisions(1))
if candq == candq_n:
# No more packages is possible to resolve
break
candq = candq_n
ftbfs = {}
fti = collections.defaultdict(dict)
if not candq:
return ftbfs, fti
for s in candq:
problems = solver.solve(
[pool.Job(solv.Job.SOLVER_SOLVABLE | solv.Job.SOLVER_INSTALL, s.id)]
)
if not problems:
continue
elif len(problems) > 1:
raise AssertionError
problem = problems[0]
if s.arch in {"src", "nosrc"}:
srcname = s.name
tmp = ftbfs[s.name] = []
else:
srcname = s.lookup_sourcepkg().rsplit("-", 2)[0]
tmp = fti[srcname][s.name] = []
for rule in problem.findallproblemrules():
if rule.type != solv.Solver.SOLVER_RULE_PKG:
raise NotImplementedError(f"Unsupported rule type: {rule.type}")
tmp.append(
[
{
"type": info.type,
"dep": info.dep,
"solvable": info.solvable,
"othersolvable": info.othersolvable,
"str": info.problemstr(),
}
for info in rule.allinfos()
]
)
return ftbfs, fti
@click.command()
@click.option(
"--release",
type=click.Choice(sorted(set(t[1:3] for t in TRACKERS.keys()))),
default=RAWHIDE,
show_default=True,
help="Fedora release",
)
def follow_policy(release):
pool = solv.Pool()
pool.setarch()
reponame = f"koji{release}"
for r in (reponame,): # f"{reponame}-source"):
repo = pool.add_repo(r)
f = solv.xfopen(f"/var/cache/dnf/{r}.solv")
repo.add_solv(f)
f.close()
pool.addfileprovides()
pool.createwhatprovides()
ftbfs, fti = find_broken_packages(pool)
bz_kwargs = {"configpaths": [BUGZILLA_CONFIG]} if os.path.exists(BUGZILLA_CONFIG) else {}
bz = bugzilla.Bugzilla("https://bugzilla.redhat.com", **bz_kwargs)
# ftbfsbug = bz.getbug(f"F{release}FTBFS")
ftibug = bz.getbug(f"F{release}FailsToInstall")
fti_report = {}
# TODO: report obsoleted packages
for src, pkg_rules in fti.items():
problems = collections.defaultdict(list)
for pkg, rules in pkg_rules.items():
if re.search(r"^rust-.*-devel$", pkg) and int(release) < 34:
continue
if pkg.startswith("dummy-test-package"):
continue
# All direct problems will be in the first rule
for info in rules[0]:
# TODO: add support for reporting conflicts and such
if (
info["solvable"].name != pkg
or info["type"] != solv.Solver.SOLVER_RULE_PKG_NOTHING_PROVIDES_DEP
):
# Only interested in missing providers of a package itself
continue
# Skip hacky multilib packages
if (
"(x86-32)" in str(info["dep"])
or "dssi-vst-wine" in str(info["dep"])
or "lmms-vst" in str(info["dep"])
):
continue
problems[pkg].append(info["str"])
if problems:
fti_report[src] = problems
pkg_owners = requests.get(
"https://src.fedoraproject.org/extras/pagure_poc.json"
).json()
ftibug = bz.getbug(f"F{release}FailsToInstall")
query_fti = bz.build_query(
product="Fedora",
status="__open__",
include_fields=[
"id",
"status",
"component",
"assigned_to",
"flags",
"blocks",
"creation_time",
],
)
query_fti["blocks"] = ftibug.id
query_fti["limit"] = 1000
query_results = bz.query(query_fti)
if len(query_results) == 1000:
raise NotImplementedError('Bugzilla pagination not yet implemented')
current_ftis = {b.component: b for b in query_results
if b.component != 'distribution'}
env = jinja2.Environment(loader=jinja2.FileSystemLoader(TEMPLATE_DIR))
env.globals["release"] = release
fti_template = env.get_template("create-fti.j2")
for src, pkgs in sorted(fti_report.items()):
if src in current_ftis:
print(
f"Skipping {src} because bug already exists: {current_ftis[src].id}",
)
continue
if pkg_owners["rpms"][src]["fedora"] == "orphan":
# Skip reporting bugs for orphaned packages
continue
description = fti_template.render(src=src, pkg_problems=pkgs)
summary = f"F{release}FailsToInstall: {', '.join(pkgs)}"
if len(summary) > 255:
summary = f"F{release}FailsToInstall: Multiple packages built from {src}"
bz_version = release if release != RAWHIDE else "rawhide"
create_fti_info = bz.build_createbug(
product="Fedora",
version=bz_version,
component=src,
summary=summary,
description=description,
blocks=ftibug.id,
)
print(description)
bz.createbug(create_fti_info)
fixed_ftis = {src: b for src, b in current_ftis.items() if src not in fti_report}
# Ignore bugs which have pending updates in Bodhi
for src, b in list(fixed_ftis.items()):
if b.status not in {"MODIFIED", "ON_QA", "VERIFIED"}:
continue
print(
f"Checking {b.id} if it was submitted as an update to appropriate release",
)
comments = b.getcomments()
try:
next(
c
for c in comments
if c["creator"] == "updates@fedoraproject.org"
and f"Fedora {release}" in c["text"]
)
print(f"Bug for {src} ({b.id}) has a pending update, ignoring")
del fixed_ftis[src]
except StopIteration:
pass
if fixed_ftis:
comment = env.get_template("close-fti.j2").render()
close = bz.build_update(
comment=comment, status="CLOSED", resolution="WORKSFORME"
)
unblock = bz.build_update(comment=comment, blocks_remove=ftibug.id)
unblock["minor_update"] = True
to_close = [
b.id
for b in fixed_ftis.values()
if not (set(b.blocks) - {ftibug.id}) & set(TRACKERS.values())
]
to_unblock = [b.id for b in fixed_ftis.values() if b.id not in to_close]
if to_close:
print(f"Closing FTI bugs for fixed components: {to_close}")
bz.update_bugs(to_close, close)
if to_unblock:
print(f"Unblocking FTI tracker for fixed components: {to_unblock}")
bz.update_bugs(to_unblock, unblock)
current_ftis = {
src: b for src, b in current_ftis.items() if src not in fixed_ftis
}
# we clear all needinfos on closed bugzillas
for bug_id in to_close:
bug = bz.getbug(bug_id)
flags = [f for f in bug.flags if f["name"] == "needinfo"]
if flags:
print(f"Clearing {len(flags)} needinfo flags from closed {bug.id}")
bz.update_bugs(
[bug.id],
bz.build_update(
flags=[
{"name": "needinfo", "id": f["id"], "status": "X"}
for f in flags
]
),
)
else:
print("No FTI bugs to close, everything is still broken")
# Update bugs for orphaned packages
orphaned = {
src: b
for src, b in current_ftis.items()
if pkg_owners["rpms"][src]["fedora"] == "orphan"
}
for src, b in orphaned.items():
click.echo(f"Checking if need to send notice to the orphaned package: {src} ({b.id})")
comments = b.getcomments()
update = False
try:
next(c for c in comments if "This package has been orphaned." in c["text"])
continue
except StopIteration:
pass
bz.update_bugs(
[b.id],
bz.build_update(
comment=f"""This package has been orphaned.
You can pick it up at https://src.fedoraproject.org/rpms/{src} by clicking button "Take". If nobody picks it up, it will be retired and removed from a distribution.""",
status="NEW",
),
)
# Now we care only about bugs in NEW state
current_ftis = {
src: b
for src, b in current_ftis.items()
if b.status == "NEW" and src not in orphaned
}
reminder_template = env.get_template("reminder-fti.j2")
for src, b in current_ftis.items():
print(f"Checking {b.id} ({src})…")
handle_orphaning(b, ftibug, reminder_template)
if __name__ == "__main__":
follow_policy()

4
scripts_new/release-process/bug-filing/follow-policy/header.j2
View file

@ -1,4 +0,0 @@
Hello,
Please note that this comment was generated automatically by https://pagure.io/releng/blob/main/f/scripts/ftbfs-fti/follow-policy.py
If you feel that this output has mistakes, please open an issue at https://pagure.io/releng/

22
scripts_new/release-process/bug-filing/follow-policy/reminder-fti.j2
View file

@ -1,22 +0,0 @@
{% include "header.j2" %}
This package fails to install and maintainers are advised to take one of the following actions:
- Fix this bug and close this bugzilla once the update makes it to the repository.
(The same script that posted this comment will eventually close this bugzilla
when the fixed package reaches the repository, so you don't have to worry about it.)
or
- Move this bug to ASSIGNED if you plan on fixing this, but simply haven't done so yet.
or
- Orphan the package if you no longer plan to maintain it.
If you do not take one of these actions, the process at https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_to_install/#_package_removal_for_long_standing_ftbfs_and_fti_bugs will continue.
This package may be orphaned in {{ orphan_weeks }}+ weeks.
This is the {{ nth }} reminder (step {{ step }}) from the policy.
Don't hesitate to ask for help on https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/ if you are unsure how to fix this bug.

0
scripts_new/release-process/bug-filing/ftbfs/.gitignore vendored
View file

161
scripts_new/release-process/bug-filing/ftbfs/mass_rebuild_close_bugs.py
View file

@ -1,161 +0,0 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: MIT
import argparse
from datetime import datetime, timezone
import logging
import queue
import threading
import bugzilla
import koji
from mass_rebuilds_info import MASSREBUILDS
LOGGER = logging.getLogger(__name__)
def koji2datetime(d):
return datetime.fromisoformat(d)
def bug2str(bug):
return f"{bug.id} ({bug.summary})"
def release2disttag(release):
assert release.startswith('f')
return 'fc' + release[1:]
def parse_bool(s):
if s.lower() in {'0', 'false', 'no'}:
return False
if s.lower() in {'1', 'true', 'yes'}:
return True
raise ValueError
def bug_closer(to_close, bz, dry_run):
while True:
item = to_close.get()
if not item:
break
bug, build = item
update = bz.build_update(status="CLOSED",
resolution="NEXTRELEASE",
comment=f"""\
There has been at least one successfull build after mass rebuild.
{build["nvr"]}: https://koji.fedoraproject.org/koji/buildinfo?buildID={build["id"]}
""")
LOGGER.info(f"{bug2str(bug)}\n"
" → Closing")
if not dry_run:
bz.update_bugs([bug.id], update)
to_close.task_done()
def main():
rebuilds_info = {k: v for k, v in MASSREBUILDS.items() if "buildtag" in v}
parser = argparse.ArgumentParser(description="Close FTBFS bugs which have been fixed")
verbose_opt = parser.add_mutually_exclusive_group()
verbose_opt.add_argument("-d", "--debug", action="store_true")
verbose_opt.add_argument("-v", "--verbose", action="store_true")
parser.add_argument("-n", "--dry-run", action="store_true")
parser.add_argument("-t", "--threads", type=int, default=0)
parser.add_argument("--strict-disttag", type=parse_bool,
help='Only close bugs where the build tag matches the release '
'(ignore builds for different releases)',
nargs='?', const=True, metavar='BOOL')
parser.add_argument("--strict-title", type=parse_bool,
help='Only close bugs where the title matches the expected pattern '
'(<component>: FTBFS in ...)',
nargs='?', const=True, metavar='BOOL',
default=True)
parser.add_argument("release", choices=rebuilds_info.keys())
args = parser.parse_args()
# Setup logging
handler = logging.StreamHandler()
LOGGER.addHandler(handler)
if args.debug:
logging.basicConfig(level=logging.DEBUG)
if args.verbose:
LOGGER.setLevel(level=logging.DEBUG)
handler.setLevel(logging.DEBUG)
else:
LOGGER.setLevel(level=logging.INFO)
handler.setLevel(logging.INFO)
if args.threads <= 0:
import multiprocessing
args.threads = multiprocessing.cpu_count()
massrebuild = rebuilds_info[args.release]
rebuild_time = koji2datetime(massrebuild["epoch"]).replace(tzinfo=timezone.utc)
bz = bugzilla.Bugzilla("https://bugzilla.redhat.com")
ks = koji.ClientSession("https://koji.fedoraproject.org/kojihub")
query = bz.build_query(product=massrebuild["product"],
version=massrebuild["version"],
blocked=str(massrebuild["tracking_bug"]),
include_fields=["id",
"is_open",
"summary",
"component",
"creation_time"])
ftbfs = bz.query(query)
tag = ks.getTagID(massrebuild["buildtag"], strict=True)
bugs = []
ks.multicall = True
for bug in ftbfs:
if not bug.is_open:
# DO NOT TOUCH CLOSED BUGZ!
LOGGER.debug(f"{bug2str(bug)}\n"
" → Skipping closed bug")
continue
if (args.strict_title and
not bug.summary.startswith(f"{bug.component}: FTBFS in F")):
# They might need special care
LOGGER.debug(f"{bug2str(bug)}\n"
" → Skipping bug with non-standard name")
continue
bugs.append(bug)
ks.getLatestBuilds(tag, package=bug.component)
builds = [ret[0] for ret in ks.multiCall(strict=True)]
# Spawn workers
to_close = queue.Queue()
threads = []
for _ in range(args.threads):
t = threading.Thread(target=bug_closer, args=(to_close, bz, args.dry_run))
t.start()
threads.append(t)
for bug, builds in zip(bugs, builds):
builds = [build for build in builds if koji2datetime(build["creation_time"]) >= rebuild_time]
if not builds:
LOGGER.debug(f"{bug2str(bug)}\n"
" → No successful builds")
continue
for build in builds:
if args.strict_disttag:
nvr = build['nvr']
if not nvr.endswith('.' + release2disttag(args.release)):
LOGGER.debug(f"{bug2str(bug)}\n"
" → Ignoring build with wrong nvr ({nvr})")
continue
to_close.put((bug, build))
# Wait untill all bugs closed
to_close.join()
# Stop workers
for _ in range(args.threads):
to_close.put(None)
for t in threads:
t.join()
if __name__ == "__main__":
main()

261
scripts_new/release-process/bug-filing/ftbfs/mass_rebuild_file_bugs.py
View file

@ -1,261 +0,0 @@
#!/usr/bin/python3
#
# mass_rebuild_file_bugs.py - A utility to discover failed builds in a
# given tag and file bugs in bugzilla for these failed builds
#
# Copyright (C) 2013 Red Hat, Inc.
# SPDX-License-Identifier: GPL-2.0+
#
# Authors:
# Stanislav Ochotnicky <sochotnicky@redhat.com>
#
from __future__ import print_function
import koji
import getpass
import tempfile
import urllib
from datetime import datetime
from bugzilla.rhbugzilla import RHBugzilla
from xmlrpc.client import Fault
import sys
import os
# Dir containing find_failures and mass_rebuild_info
sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'mass-rebuilds'))
from find_failures import get_failed_builds
# contains info about all rebuilds, add new rebuilds there and update rebuildid
# here
from mass_rebuilds_info import MASSREBUILDS
rebuildid = 'f43'
failures = {} # dict of owners to lists of packages that failed.
failed = [] # raw list of failed packages
BZ_PAGE_SIZE = 1000
bzurl = 'https://bugzilla.redhat.com'
BZCLIENT = RHBugzilla(url="%s/xmlrpc.cgi" % bzurl,
user="releng@fedoraproject.org")
DEFAULT_COMMENT = \
"""{component} failed to build from source in {product} {version}/f{rawhide_version}
https://koji.fedoraproject.org/koji/taskinfo?taskID={task_id}
{extrainfo}
For details on the mass rebuild see:
{wikipage}
Please fix {component} at your earliest convenience and set the bug's status to
ASSIGNED when you start fixing it. If the bug remains in NEW state for 8 weeks,
{component} will be orphaned. Before branching of {product} {nextversion},
{component} will be retired, if it still fails to build.
For more details on the FTBFS policy, please visit:
https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_to_install/
"""
def report_failure(massrebuild, component, task_id, logs,
summary="{component}: FTBFS in {product} {version}/f{rawhide_version}",
comment=DEFAULT_COMMENT, extrainfo=""):
"""This function files a new bugzilla bug for component with given
arguments
Keyword arguments:
massrebuild -- generic info about mass rebuild such as tracking_bug,
Bugzilla product, version, wikipage
component -- component (package) to file bug against
task_id -- task_id of failed build
logs -- list of URLs to the log file to attach to the bug report
summary -- short bug summary (if not default)
comment -- first comment describing the bug in more detail (if not default)
"""
format_values = dict(**massrebuild)
format_values["task_id"] = task_id
format_values["component"] = component
format_values["nextversion"] = str(int(massrebuild["rawhide_version"]) + 1)
format_values["extrainfo"] = extrainfo
summary = summary.format(**format_values)
comment = comment.format(**format_values)
data = {'product': massrebuild["product"],
'component': component,
'version': massrebuild["version"],
'short_desc': summary,
'comment': comment,
'blocks': massrebuild["tracking_bug"],
'rep_platform': 'Unspecified',
'bug_severity': 'unspecified',
'op_sys': 'Unspecified',
'bug_file_loc': '',
'priority': 'unspecified',
}
try:
print('Creating the bug report')
bug = BZCLIENT.createbug(**data)
bug.refresh()
print(bug)
attach_logs(bug, logs)
except Fault as ex:
print(ex)
#Because of having image build requirement of having the image name in koji
#as a package name, they are missing in the components of koji and we need
#to skip them.
#if ex.faultCode == -32000:
# print(component)
if "There is no component" in ex.faultString:
print(ex.faultString)
return None
else:
username = input('Bugzilla username: ')
BZCLIENT.login(user=username,
password=getpass.getpass())
return report_failure(massrebuild, component, task_id, logs, summary,
comment)
return bug
def attach_logs(bug, logs):
if isinstance(bug, int):
bug = BZCLIENT.getbug(bug)
for log in logs:
name = log.rsplit('/', 1)[-1]
try:
response = urllib.request.urlopen(log)
except urllib.error.HTTPError as e:
#sometimes there wont be any logs attached to the task.
#skip attaching logs for those tasks
if e.code == 404:
print("Failed to attach {} log".format(name))
continue
else:
break
fp = tempfile.TemporaryFile()
CHUNK = 2 ** 20
while True:
chunk = response.read(CHUNK)
if not chunk:
break
fp.write(chunk)
filesize = fp.tell()
# Bugzilla file limit, still possibly too much
# FILELIMIT = 32 * 1024
# Just use 32 KiB:
FILELIMIT = 2 ** 15
if filesize > FILELIMIT:
fp.seek(filesize - FILELIMIT)
comment = "file {} too big, will only attach last {} bytes".format(
name, FILELIMIT)
else:
comment = ""
fp.seek(0)
try:
print('Attaching file %s to the ticket' % name)
# arguments are: idlist, attachfile, description, ...
attid = BZCLIENT.attachfile(
bug.id, fp, name, content_type='text/plain', file_name=name,
comment=comment
)
except Fault as ex:
print(ex)
raise
finally:
fp.close()
def get_filed_bugs(tracking_bug):
"""Query bugzilla if given bug has already been filed
arguments:
tracking_bug -- bug used to track failures
Keyword arguments:
product -- bugzilla product (usually Fedora)
component -- component (package) to file bug against
version -- component version to file bug for (usually rawhide for Fedora)
summary -- short bug summary
"""
query_data = {'blocks': tracking_bug, 'offset': 0, 'limit': 1000 }
bzurl = 'https://bugzilla.redhat.com'
bzclient = RHBugzilla(url="%s/xmlrpc.cgi" % bzurl)
results = _iterate_query(query_data,bzclient)
return results
def get_task_failed(kojisession, task_id):
''' For a given task_id, use the provided kojisession to return the
task_id of the first children that failed to build.
'''
for child in kojisession.getTaskChildren(task_id):
if child['state'] == koji.TASK_STATES["FAILED"]: # 5 == Failed
return child['id']
def _iterate_query(querydata, bzclient):
"""Iterate Bugzilla query until all results are fetched."""
results = bzclient.query(querydata)
if len(results) == BZ_PAGE_SIZE:
last_result_id = results[-1].id
querydata['f1'] = 'bug_id'
querydata['o1'] = 'greaterthan'
querydata['v1'] = last_result_id
results += _iterate_query(querydata,bzclient)
return results
def _is_relevant_bugzilla(bug, epoch_datetime):
"""We only care for bugs that are not yet closed or that were modified after
the given epoch"""
if bug.status != 'CLOSED':
return True
last_change_datetime = datetime.strptime(str(bug.last_change_time),
'%Y%m%dT%H:%M:%S')
return last_change_datetime > epoch_datetime
if __name__ == '__main__':
massrebuild = MASSREBUILDS[rebuildid]
kojisession = koji.ClientSession('https://koji.fedoraproject.org/kojihub')
print('Getting the list of failed builds...')
failbuilds = get_failed_builds(kojisession, massrebuild['epoch'],
massrebuild['buildtag'],
massrebuild['desttag'])
print('Getting the list of filed bugs...')
filed_bugs = get_filed_bugs(massrebuild['tracking_bug'])
epoch_datetime = datetime.fromisoformat(massrebuild['epoch'])
filed_bugs_components = [bug.component for bug in filed_bugs
if _is_relevant_bugzilla(bug, epoch_datetime)]
for build in failbuilds:
task_id = build['task_id']
component = build['package_name']
work_url = 'https://kojipkgs.fedoraproject.org/work'
child_id = get_task_failed(kojisession, task_id)
if not child_id:
print('No children failed for task: %s (%s)' % (
task_id, component))
logs = []
else:
base_path = koji.pathinfo.taskrelpath(child_id)
log_url = "%s/%s/" % (work_url, base_path)
build_log = log_url + "build.log"
root_log = log_url + "root.log"
state_log = log_url + "state.log"
logs = [build_log, root_log, state_log]
if component not in filed_bugs_components:
print("Filing bug for %s" % component)
report_failure(massrebuild, component, task_id, logs)
filed_bugs_components.append(component)
else:
print("Skipping %s, bug already filed" % component)

159
scripts_new/release-process/bug-filing/ftbfs_weekly_reminder.py
View file

@ -1,159 +0,0 @@
#!/usr/bin/python3
import configparser
from datetime import datetime, date, timedelta
import bugzilla
import pathlib
import sys
from click import progressbar
import logging
import massrebuildsinfo
assert sys.version_info[0] > 2, 'Needs Python 3'
# Modify logging configuration
LOGGER = logging.getLogger()
LOGGER.setLevel(logging.ERROR) # Only log errors
fh = logging.FileHandler('bugzilla.log')
fh.setLevel(logging.DEBUG)
LOGGER.addHandler(fh)
# get the latest Fedora version and tracking bug
for key, value in massrebuildsinfo.MASSREBUILDS.items():
fedora = key
tracking = value['tracking_bug']
break
# Get credentials from config
config_path = pathlib.Path('./ftbfs.cfg')
if not config_path.exists():
config_path = pathlib.Path('/etc/ftbfs.cfg')
if not config_path.exists():
raise RuntimeError('Create a config file as ./ftbfs.cfg or /etc/ftbfs.cfg')
config = configparser.ConfigParser()
config.read(config_path)
URL = config['bugzilla'].get('url', 'https://bugzilla.redhat.com')
API_KEY = config['bugzilla'].get('api_key', None)
FEDORA = config['bugzilla'].get('fedora', fedora).upper().replace('F', '')
TRACKING = config['bugzilla'].get('tracking', tracking)
TEMPLATE = f"""Dear Maintainer,
your package has an open Fails To Build From Source bug for Fedora {FEDORA}.
Action is required from you.
If you can fix your package to build, perform a build in koji, and either create
an update in bodhi, or close this bug without creating an update, if updating is
not appropriate [1]. If you are working on a fix, set the status to ASSIGNED to
acknowledge this. If you have already fixed this issue, please close this Bugzilla report.
Following the policy for such packages [2], your package will be orphaned if
this bug remains in NEW state more than 8 weeks (not sooner than {{orphanon}}).
A week before the mass branching of Fedora {int(FEDORA)+1} according to the schedule [3],
any packages not successfully rebuilt at least on Fedora {int(FEDORA)-1} will be
retired regardless of the status of this bug.
[1] https://docs.fedoraproject.org/en-US/fesco/Updates_Policy/
[2] https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_to_install/
[3] https://fedorapeople.org/groups/schedule/f-{int(FEDORA)+1}/f-{int(FEDORA)+1}-key-tasks.html
""" # noqa
cache_dir = pathlib.Path('~/.cache/FTBFS_weekly_reminder/').expanduser()
cache_dir.mkdir(exist_ok=True)
ALREADY_FILED = cache_dir / 'ALREADY_FILED'
bzapi = bugzilla.Bugzilla(URL, api_key=API_KEY)
failed = []
updated = []
def new_ftbfs_bugz(tracker=TRACKING):
query = bzapi.build_query(product='Fedora', status='NEW')
query['blocks'] = tracker
return bzapi.query(query)
def needinfo(requestee):
return {
'name': 'needinfo',
'requestee': requestee,
'status': '?',
}
def send_reminder(bug, comment=TEMPLATE, set_needinfo=True):
created = date(*bug.creation_time.timetuple()[:3])
orphanon = created + timedelta(days=7*8)
comment = comment.format(orphanon=orphanon.isoformat())
flags = [needinfo(bug.assigned_to)] if set_needinfo else []
update = bzapi.build_update(comment=comment, flags=flags)
try:
bzapi.update_bugs([bug.id], update)
except Exception as e:
LOGGER.exception(bug.weburl)
if "You can't ask" in getattr(e, 'faultString', ''):
print(e.faultString, file=sys.stderr)
return send_reminder(bug, comment=comment, set_needinfo=False)
if 'set multiple times' in getattr(e, 'faultString', ''):
return send_reminder(bug, comment=comment, set_needinfo=False)
failed.append(bug)
else:
updated.append(bug)
with open(ALREADY_FILED, 'a') as f:
print(bug.id, file=f)
ignore = []
today = datetime.today()
if ALREADY_FILED.exists():
age = today - datetime.fromtimestamp(ALREADY_FILED.stat().st_mtime)
# we gracefully approximate a "less than a week" age here
# the file is intended for immediate repeated runs, not forever
if age.days < 6:
print(f'Loading bug IDs from {ALREADY_FILED}. Will not file those. '
f'Remove {ALREADY_FILED} to stop this from happening.')
ignore = [
int(l.rstrip()) for l in ALREADY_FILED.read_text().splitlines()
]
else:
target = ALREADY_FILED.parent / f'~{ALREADY_FILED.name}'
print(f'Moving too old {ALREADY_FILED} to {target}')
ALREADY_FILED.rename(target)
print('Gathering bugz, this can take a while...')
bugz = new_ftbfs_bugz()
print(f'There are {len(bugz)} NEW bugz, will send a reminder')
if ignore:
print(f'Will ignore {len(ignore)} bugz from {ALREADY_FILED}')
print(f'Will update {len(set(b.id for b in bugz) - set(ignore))} bugz')
def _item_show_func(bug):
if bug is None:
return 'Finished!'
return bug.weburl
with progressbar(bugz, item_show_func=_item_show_func) as bugbar:
for bug in bugbar:
if bug.id not in ignore:
send_reminder(bug)
print(f'Updated {len(updated)} bugz')
if failed:
print(f'Failed to update {len(failed)} bugz', file=sys.stderr)
for bug in failed:
print(bug.weburl, file=sys.stderr)
sys.exit(1)
elif ALREADY_FILED.exists():
target = ALREADY_FILED.parent / f'~{ALREADY_FILED.name}'
print(f'Moving {ALREADY_FILED} to {target}, all bugz filed')
ALREADY_FILED.rename(target)

60
scripts_new/release-process/end-of-life/koji_cleanup_signed_rpms.py
View file

@ -1,60 +0,0 @@
#!/usr/bin/python
# clean up eol signed rpm
# Copyright (C) 2013 Red Hat, Inc.
# SPDX-License-Identifier: GPL-2.0
"""Remove signed rpms after a release is end-of-life
"""
from __future__ import print_function
import os
KEYS = [
'069C8460', '30C9ECF8',
'4F2A6FD2', '897DA07A',
'1AC70CE6', '6DF2196F',
'DF9B0AE9', '0B86274E',
'4EBFC273', 'D22E77F2',
'57BBCCBA', 'E8E40FDE',
'97A1071F', '069C8460',
'10d90a9e', 'a82ba4b7',
'f8df67e6', '1aca3465',
'de7f38bd', 'a4d647e9',
'fb4b18e6', 'ba094068',
'246110c1', 'efe550f5',
'95a43f54', 'a0a7badb',
'8e1431d5', 'a29cb19c',
'873529b8', '34ec9cba',
'030d5aed', '81b46521',
'e372e838', 'fdb19c98',
'3b921d09', '64dab85d',
'f5282ee4', '9db62fb1',
'429476b4', 'cfc659b9',
'3c3359c4', '12c944d0',
'9570ff31', '45719a39',
'38ab71f4', '5323552a',
'eb10b464', '18b8e74c',
'a15B79cc',
]
PREFIX = 'data/signed'
ROOT_PATH = '/mnt/koji/packages'
for root, dirs, files in os.walk(ROOT_PATH, topdown=False):
for name in files:
filepath = os.path.join(root, name)
for key in KEYS:
if os.path.join(PREFIX, str.lower(key)) in filepath:
print(filepath)
if os.path.exists(filepath):
os.remove(filepath)
continue
for name in dirs:
filepath = os.path.join(root, name)
for key in KEYS:
if os.path.join(PREFIX, str.lower(key)) in filepath:
print(filepath)
if os.path.exists(filepath):
os.rmdir(filepath)
continue

14
scripts_new/release-process/mass-branching/README.md
View file

@ -1,14 +0,0 @@
# Branch Creation Tools
Scripts for creating new release branches in Fedora and EPEL.
## Subdirectories
- **[fedora/](fedora/)** - Fedora release branching tools
- **[epel/](epel/)** - EPEL release branching tools
## Purpose
Manages the creation of new branches when starting development for new Fedora releases or EPEL versions.
**Status**: 🚧 Awaiting migration

0
scripts_new/release-process/mass-branching/fedora/.gitignore vendored
View file

121
scripts_new/release-process/mass-branching/fedora/create_empty_repositories.sh
View file

@ -1,121 +0,0 @@
#!/bin/bash
#This whole script needs improvement, it is just a quick fix.
release=$1
mkdir -p /pub/fedora/linux/updates/$release/Everything/{aarch64,armhfp,x86_64}/{Packages,debug,drpms}
mkdir -p /pub/fedora/linux/updates/$release/Modular/{aarch64,armhfp,x86_64}/{Packages,debug,drpms}
mkdir -p /pub/fedora-secondary/updates/$release/Everything/{i386,ppc64le,s390x}/{Packages,debug,drpms}
mkdir -p /pub/fedora-secondary/updates/$release/Modular/{i386,ppc64le,s390x}/{Packages,debug,drpms}
for dir in /pub/fedora/linux/updates/$release/Everything/*
do
createrepo_c $dir
done
for dir in /pub/fedora/linux/updates/$release/Modular/*
do
createrepo_c $dir
done
for dir in /pub/fedora-secondary/updates/$release/Everything/*
do
createrepo_c $dir
done
for dir in /pub/fedora-secondary/updates/$release/Modular/*
do
createrepo_c $dir
done
mkdir -p /pub/fedora/linux/updates/$release/Everything/source/tree/Packages
mkdir -p /pub/fedora/linux/updates/$release/Modular/source/tree/Packages
createrepo_c /pub/fedora/linux/updates/$release/Everything/source/tree
createrepo_c /pub/fedora/linux/updates/$release/Modular/source/tree
for dir in /pub/fedora/linux/updates/$release/Everything/*/debug/
do
mkdir -p $dir/Packages
createrepo_c $dir
done
for dir in /pub/fedora/linux/updates/$release/Modular/*/debug/
do
mkdir -p $dir/Packages
createrepo_c $dir
done
for dir in /pub/fedora-secondary/updates/$release/Everything/*/debug/
do
mkdir -p $dir/Packages
createrepo_c $dir
done
for dir in /pub/fedora-secondary/updates/$release/Modular/*/debug/
do
mkdir -p $dir/Packages
createrepo_c $dir
done
#Testing repos
mkdir -p /pub/fedora/linux/updates/testing/$release/Everything/{aarch64,armhfp,x86_64}/{Packages,debug,drpms}
mkdir -p /pub/fedora/linux/updates/testing/$release/Modular/{aarch64,armhfp,x86_64}/{Packages,debug,drpms}
mkdir -p /pub/fedora-secondary/updates/testing/$release/Everything/{i386,ppc64le,s390x}/{Packages,debug,drpms}
mkdir -p /pub/fedora-secondary/updates/testing/$release/Modular/{i386,ppc64le,s390x}/{Packages,debug,drpms}
for dir in /pub/fedora/linux/updates/testing/$release/Everything/*
do
createrepo_c $dir
done
for dir in /pub/fedora/linux/updates/testing/$release/Modular/*
do
createrepo_c $dir
done
for dir in /pub/fedora-secondary/updates/testing/$release/Everything/*
do
createrepo_c $dir
done
for dir in /pub/fedora-secondary/updates/testing/$release/Modular/*
do
createrepo_c $dir
done
mkdir -p /pub/fedora/linux/updates/testing/$release/Everything/source/tree/Packages
mkdir -p /pub/fedora/linux/updates/testing/$release/Modular/source/tree/Packages
createrepo_c /pub/fedora/linux/updates/testing/$release/Everything/source/tree
createrepo_c /pub/fedora/linux/updates/testing/$release/Modular/source/tree
for dir in /pub/fedora/linux/updates/testing/$release/Everything/*/debug/
do
mkdir -p $dir/Packages
createrepo_c $dir
done
for dir in /pub/fedora/linux/updates/testing/$release/Modular/*/debug/
do
mkdir -p $dir/Packages
createrepo_c $dir
done
for dir in /pub/fedora-secondary/updates/testing/$release/Everything/*/debug/
do
mkdir -p $dir/Packages
createrepo_c $dir
done
for dir in /pub/fedora-secondary/updates/testing/$release/Modular/*/debug/
do
mkdir -p $dir/Packages
createrepo_c $dir
done

216
scripts_new/release-process/mass-branching/fedora/create_koji_release_tags.sh
View file

@ -1,216 +0,0 @@
#!/bin/bash
release=$1
case "${release}" in
''|*[!0-9]*)
is_number="false"
;;
*)
is_number="true"
;;
esac
if [ "${is_number}" = "false" ]; then
echo "The release value may only be a number."
exit 1
fi
old_release=$(bc -l <<< "${release}-1")
PRIMARY_ARCHES="i686 x86_64 aarch64 ppc64le s390x"
FLATPAK_ARCHES="x86_64 aarch64 ppc64le"
KOJICLI=koji
"${KOJICLI}" clone-tag --all --latest-only "f${old_release}" "f${release}"
"${KOJICLI}" edit-tag -x mock.package_manager=dnf5 "f${release}"
"${KOJICLI}" add-tag "f${release}-container"
"${KOJICLI}" add-tag --parent "f${release}-container" --arches="x86_64 aarch64" "f${release}-container-build"
"${KOJICLI}" add-tag --parent "f${release}" "f${release}-updates"
"${KOJICLI}" add-tag --parent "f${release}" "f${release}-compose"
"${KOJICLI}" add-tag --parent "f${release}" "f${release}-updates-candidate"
"${KOJICLI}" add-tag --parent "f${release}" "f${release}-updates-testing"
"${KOJICLI}" add-tag --parent "f${release}-updates-testing" "f${release}-updates-testing-pending"
"${KOJICLI}" add-tag --parent "f${release}" "f${release}-updates-pending"
"${KOJICLI}" add-tag --parent "f${release}" "f${release}-override"
"${KOJICLI}" add-tag --parent "f${release}-override" --arches="${PRIMARY_ARCHES}" "f${release}-build"
"${KOJICLI}" add-tag --parent "f${release}-updates-testing-pending" "f${release}-signing-pending"
"${KOJICLI}" add-tag --parent "f${release}-updates" "f${release}-pending"
"${KOJICLI}" add-tag --parent "f${release}-build" --arches="${PRIMARY_ARCHES}" "f${release}-infra"
"${KOJICLI}" add-tag --parent "f${release}-infra" --arches="${PRIMARY_ARCHES}" "f${release}-infra-stg"
"${KOJICLI}" add-tag --parent "f${release}-infra-stg" "f${release}-infra-candidate"
"${KOJICLI}" add-tag --parent "f${release}-infra-stg" --arches="${PRIMARY_ARCHES}" "f${release}-infra-build"
"${KOJICLI}" add-tag --parent "f${release}" "f${release}-openh264"
"${KOJICLI}" add-tag --parent "f${release}" "f${release}-atomic"
"${KOJICLI}" add-tag --parent "f${release}" "f${release}-atomic-host-installer"
"${KOJICLI}" add-tag --parent "f${release}" "f${release}-iot"
"${KOJICLI}" add-tag "f${release}-atomic-host-overrides"
"${KOJICLI}" edit-tag -x tag2distrepo.enabled=True -x tag2distrepo.keys=47dd8ef9 "f${release}-infra"
"${KOJICLI}" edit-tag -x tag2distrepo.enabled=True -x tag2distrepo.keys=47dd8ef9 "f${release}-infra-stg"
"${KOJICLI}" edit-tag -x sidetag_rpm_macros_allowed='_with_bootstrap' "f${release}-build"
"${KOJICLI}" edit-tag --perm=fedora-override "f${release}-override"
"${KOJICLI}" edit-tag --perm=admin "f${release}-updates"
"${KOJICLI}" edit-tag --perm=admin "f${release}-updates-testing"
"${KOJICLI}" edit-tag --perm=autosign "f${release}-updates-testing-pending"
"${KOJICLI}" edit-tag --perm=admin "f${release}-updates-pending"
"${KOJICLI}" edit-tag --perm=admin "f${release}-atomic"
"${KOJICLI}" edit-tag --perm=autosign "f${release}-signing-pending"
"${KOJICLI}" edit-tag --perm=infra "f${release}-infra"
"${KOJICLI}" edit-tag --perm=infra "f${release}-infra-build"
"${KOJICLI}" edit-tag --perm=infra "f${release}-infra-stg"
"${KOJICLI}" edit-tag --perm=infra "f${release}-infra-candidate"
"${KOJICLI}" edit-tag --perm=atomic "f${release}-atomic-host-installer"
"${KOJICLI}" edit-tag --perm=atomic "f${release}-atomic-host-overrides"
# FCOS continuous builds: https://pagure.io/releng/issue/8165
"${KOJICLI}" add-tag "f${release}-coreos-continuous" --arches="x86_64 aarch64 ppc64le s390x"
"${KOJICLI}" add-target "f${release}-coreos-continuous" "f${release}-build" "f${release}-coreos-continuous"
"${KOJICLI}" edit-tag -x tag2distrepo.enabled=True "f${release}-coreos-continuous"
# FCOS signing tags that feed the coreos-pool tag: https://pagure.io/releng/issue/8294
"${KOJICLI}" add-tag "f${release}-coreos-signing-pending" --parent coreos-pool
"${KOJICLI}" add-tag --parent module-package-list "module-f${release}-build"
"${KOJICLI}" add-external-repo -t "module-f${release}-build" "f${release}-build" "https://kojipkgs.fedoraproject.org/repos/f${release}-build/latest/\$arch/"
"${KOJICLI}" tag-pkg "f${release}-build" "$(${KOJICLI} latest-build "f${old_release}-build" glibc64 glibc32 --quiet|sed -e "s| .*||g" )"
# Set up a corresponding set of tags for containers.
container_release=${release}-container
"${KOJICLI}" add-tag --parent "f${container_release}" "f${container_release}-updates"
"${KOJICLI}" add-tag --parent "f${container_release}-updates" "f${container_release}-updates-candidate"
"${KOJICLI}" add-tag --parent "f${container_release}-updates" "f${container_release}-updates-testing"
"${KOJICLI}" add-tag --parent "f${container_release}-updates-testing" "f${container_release}-updates-testing-pending"
"${KOJICLI}" add-tag --parent "f${container_release}-updates" "f${container_release}-updates-pending"
"${KOJICLI}" add-tag --parent "f${container_release}-updates" "f${container_release}-override"
"${KOJICLI}" add-target "f${release}" "f${release}-build" "f${release}-updates-candidate"
"${KOJICLI}" add-target "f${release}-candidate" "f${release}-build" "f${release}-updates-candidate"
"${KOJICLI}" add-target "f${release}-infra" "f${release}-infra-build" "f${release}-infra-candidate"
"${KOJICLI}" add-target "f${release}-container-candidate" "f${release}-container-build" "f${release}-container-updates-candidate"
"${KOJICLI}" edit-target rawhide --dest-tag="f${release}-updates-candidate" --build-tag="f${release}-build"
"${KOJICLI}" edit-target rawhide-container-candidate --dest-tag="f${release}-container-updates-candidate" --build-tag="f${release}-container-build"
"${KOJICLI}" remove-tag-inheritance rawhide "f${old_release}"
"${KOJICLI}" add-tag-inheritance rawhide "f${release}"
#### The pre-F39 tags structure for Flatpak containers, that needs to be created ####
# Set up a corresponding set of tags for flatpaks.
# They are only setup for branched release.
"${KOJICLI}" clone-tag --config --groups --pkgs "f${old_release}-flatpak" "f${release}-flatpak"
"${KOJICLI}" add-tag --parent "f${release}-flatpak" "f${release}-flatpak-updates"
"${KOJICLI}" add-tag --parent "f${release}-flatpak-updates-testing-pending" "f${release}-flatpak-updates-candidate"
"${KOJICLI}" add-tag --parent "f${release}-flatpak-updates" "f${release}-flatpak-updates-testing"
"${KOJICLI}" add-tag --parent "f${release}-flatpak-updates-testing" "f${release}-flatpak-updates-testing-pending"
"${KOJICLI}" add-tag --parent "f${release}-flatpak-updates" "f${release}-flatpak-updates-pending"
"${KOJICLI}" add-tag --parent "f${release}-flatpak-updates" "f${release}-flatpak-override"
# Add packages from the stable release
flatpak_pkgs=$(koji list-pkgs --quiet --tag="f${old_release}-flatpak" | awk '{ print $1}')
"${KOJICLI}" add-pkg --owner=releng "f${release}-flatpak" $flatpak_pkgs
#### Destination tags ####
# These inherit from f${old_release} so that pkglist information doesn't need to be individually
# maintained for them, but are --intransitive so that we can fine-tune things when
# creating build tags
# Packages that are rebuilt for inclusion in runtimes (perhaps with reduced dependencies)
"${KOJICLI}" add-tag "f${release}-flatpak-runtime"
"${KOJICLI}" add-tag-inheritance "f${release}-flatpak-runtime" "f${release}" --intransitive
# this package is blocked from the main tag
"${KOJICLI}" unblock-pkg "f${release}-flatpak-runtime" flatpak-runtime-config
# Packages that are rebuilt for inclusion in applications (with prefix=/app)
# This has an accompanying dist-repo created using the tag2distrepo plugin, which we use
# when building application containers.
"${KOJICLI}" add-tag "f${release}-flatpak-app" \
--arches="${FLATPAK_ARCHES}" \
-x tag2distrepo.enabled=True \
-x tag2distrepo.inherit=False \
-x tag2distrepo.latest=True
"${KOJICLI}" add-tag-inheritance "f${release}-flatpak-app" "f${release}" --intransitive
"${KOJICLI}" add-group-pkg "f${release}-flatpak-app" build flatpak-rpm-macros flatpak-runtime-config
"${KOJICLI}" add-group-pkg "f${release}-flatpak-app" srpm-build flatpak-rpm-macros flatpak-runtime-config
#### Build tags ####
# The repository that tools (flatpak-module-tools, flatpak, ostree, etc) are installed from
# when building a flatpak. This is also used as the build tag when building Flatpaks
"${KOJICLI}" add-tag "f${release}-flatpak-container-build" --parent="f${release}-build" --arches="${FLATPAK_ARCHES}"
"${KOJICLI}" add-group "f${release}-flatpak-container-build" flatpak-build
"${KOJICLI}" add-group-pkg "f${release}-flatpak-container-build" flatpak-build flatpak-module-tools dnf5 tar
# The repository for build dependencies when building runtime RPMs
"${KOJICLI}" add-tag "f${release}-flatpak-runtime-build" \
--arches="${FLATPAK_ARCHES}" \
-x rpm.macro.distcore='.fc%{fedora}runtime1' \
-x rpm.macro.flatpak_runtime=1
"${KOJICLI}" add-tag-inheritance "f${release}-flatpak-runtime-build" "f${release}-flatpak-runtime" --priority=0
"${KOJICLI}" add-tag-inheritance "f${release}-flatpak-runtime-build" "f${release}-build" --priority=10
# The repository for build dependencies when building application RPMs
"${KOJICLI}" add-tag "f${release}-flatpak-app-build" \
--arches="${FLATPAK_ARCHES}" \
-x rpm.macro.distcore='.fc%{fedora}app1' \
-x rpm.macro._without_mingw=1
"${KOJICLI}" add-tag-inheritance "f${release}-flatpak-app-build" "f${release}-flatpak-app" --priority=0
"${KOJICLI}" add-tag-inheritance "f${release}-flatpak-app-build" "f${release}-flatpak-runtime" --priority=10
"${KOJICLI}" add-tag-inheritance "f${release}-flatpak-app-build" "f${release}-build" --priority=20
#### Tags defining repositories for container installation ####
# An override tag that we can tag packages that have not gone stable yet for runtime inclusion
"${KOJICLI}" add-tag "f${release}-flatpak-runtime-override" --parent=f${release}-updates
# The repository for getting runtime packages from when creating a container
"${KOJICLI}" add-tag "f${release}-flatpak-runtime-packages" --arches="${FLATPAK_ARCHES}"
"${KOJICLI}" add-tag-inheritance "f${release}-flatpak-runtime-packages" "f${release}-flatpak-runtime" --priority=0
"${KOJICLI}" add-tag-inheritance "f${release}-flatpak-runtime-packages" "f${release}-flatpak-runtime-override" --priority=10
#### Extra Data ####
# Add extra data to the Flatpak build tag pointing to a) the tag where to find runtimes
# b) the tag to take runtime packages from c) the tag to take applications from
# and say that we're using a dist-repo for app_package_tag.
"${KOJICLI}" edit-tag "f${release}-flatpak-container-build" \
-x flatpak.runtime_tag="f${release}-flatpak-updates-candidate" \
-x flatpak.runtime_package_tag="f${release}-flatpak-runtime-packages" \
-x flatpak.app_package_tag="f${release}-flatpak-app" \
-x flatpak.app_package_dist_repo=true
#### Targets ####
# Targets for building Flatpak containers, runtime RPMs, and application RPMs
"${KOJICLI}" add-target "f${release}-flatpak-candidate" "f${release}-flatpak-container-build" "f${release}-flatpak-updates-candidate"
"${KOJICLI}" add-target "f${release}-flatpak-runtime" "f${release}-flatpak-runtime-build" "f${release}-flatpak-runtime"
"${KOJICLI}" add-target "f${release}-flatpak-app" "f${release}-flatpak-app-build" "f${release}-flatpak-app"
# And a lovely target just to make kojira rebuild the installation repository for runtimes
"${KOJICLI}" add-target "f${release}-flatpak-runtime-kojira" "f${release}-flatpak-runtime-packages" "f${release}-flatpak-app"
# Point eln to new rawhide release
"${KOJICLI}" remove-tag-inheritance eln "f${old_release}"
"${KOJICLI}" add-tag-inheritance eln "f${release}"
"${KOJICLI}" remove-tag-inheritance eln-build "f${old_release}-build"
"${KOJICLI}" add-tag-inheritance eln-build "f${release}-build" --priority 5
# Targets for kiwi image builds to use their own tag/target in order to set mock to use old chroot
koji add-tag --parent "f${release}-compose" --arches="x86_64 aarch64 ppc64le s390x" "f${release}-kiwi-build"
koji edit-tag --perm=admin "f${release}-kiwi-build"
koji edit-tag -x mock.new_chroot=0 f${release}-kiwi-build
koji add-target "f${release}-kiwi" "f${release}-kiwi-build" "f${release}"
# We don't need group as it should be cloned from future release tags
# koji add-group f${release} kiwi-build
# koji add-group-pkg f${release} kiwi-build kiwi-cli kiwi-systemdeps
# Create `image-builder-build` koji build group
koji add-tag --parent "f${release}-compose" --arches="x86_64 aarch64 ppc64le s390x" "f${release}-image-builder-build"
koji edit-tag --perm=admin "f${release}-image-builder-build"
koji edit-tag -x mock.new_chroot=0 f${release}-image-builder-build
koji add-target "f${release}-image-builder" "f${release}-image-builder-build" "f${release}"
koji add-group f${release}-image-builder-build image-builder-build
koji add-group-pkg f${release}-image-builder-build image-builder-build image-builder distribution-gpg-keys

47
scripts_new/release-process/mass-branching/fedora/get_active_packages.sh
View file

@ -1,47 +0,0 @@
#!/usr/bin/bash
set -eu
# Define variables
CHECKOUT_PATH="${1:-/srv/git/rpms}"
OUTPUT_TEMP_PACKAGES="package_list.txt"
OUTPUT_TEMP_RETIRED="retired_packages.txt"
OUTPUT_FINAL="components_f42.txt"
# Step 1: Get all package names from the rpms directory
echo "Fetching all package names..."
: > "$OUTPUT_TEMP_PACKAGES" # Clear or create file
for git_repo in ${CHECKOUT_PATH}/*.git; do
git_repo_name="$(basename "${git_repo}" .git)"
echo "$git_repo_name" >> "$OUTPUT_TEMP_PACKAGES"
done
echo "Stored all package names in $OUTPUT_TEMP_PACKAGES."
# Step 1.5: Check if the package has a .spec file in rawhide, filtering out never-imported packages
echo "Filtering packages that do not have a rawhide .spec file..."
: > "$OUTPUT_TEMP_PACKAGES.filtered" # Create a new temp file
while IFS= read -r pkg; do
# Escape '+' characters in package name for regex
escaped_pkg="${pkg//+/\\+}"
if git -C "${CHECKOUT_PATH}/${pkg}.git" ls-tree -r rawhide --name-only | grep -qE "^${escaped_pkg}\\.spec$"; then
echo "$pkg" >> "$OUTPUT_TEMP_PACKAGES.filtered"
fi
done < "$OUTPUT_TEMP_PACKAGES"
mv "$OUTPUT_TEMP_PACKAGES.filtered" "$OUTPUT_TEMP_PACKAGES"
echo "Filtered package list is now stored in $OUTPUT_TEMP_PACKAGES."
# Step 2: Fetch retired packages from Fedora JSON source
echo "Fetching retired packages..."
curl -s https://src.fedoraproject.org/lookaside/retired_in_rawhide.json | jq -r '.rawhide[]' > "$OUTPUT_TEMP_RETIRED"
echo "Stored retired packages in $OUTPUT_TEMP_RETIRED."
# Step 3: Remove retired packages from the full package list to get active packages
echo "Filtering active packages..."
grep -Fxvf "$OUTPUT_TEMP_RETIRED" "$OUTPUT_TEMP_PACKAGES" | awk '{print "rpm/" $0}' > "$OUTPUT_FINAL"
echo "Stored active package list in $OUTPUT_FINAL."
# Step 4: Remove temporary files
rm -f "$OUTPUT_TEMP_PACKAGES" "$OUTPUT_TEMP_RETIRED"
echo "Removed temporary files."
echo "Process completed successfully! 🚀"

165
scripts_new/release-process/mass-branching/fedora/koji_http_toggle.sh
View file

@ -1,165 +0,0 @@
#!/usr/bin/env bash
# =============================================================================
# Koji HTTP Toggle (nftables)
# -----------------------------------------------------------------------------
# Purpose:
# Quickly BLOCK or UNBLOCK external HTTP submissions to Koji by dropping
# TCP/80 from specified proxies (proxy01/proxy10 by default) using nftables.
#
# What it does:
# - Resolves hostnames to IPv4 addresses (via `getent ahostsv4`)
# - Ensures `table inet filter` and `chain input` exist
# - Creates a named set `koji_http_block_src` and a rule that drops
# tcp dport 80 if ip saddr is in that set
# - Adds/removes IPs to/from the set (idempotent)
#
# Usage:
# sudo /usr/local/sbin/koji-http-toggle.sh block # block proxy01/proxy10
# sudo /usr/local/sbin/koji-http-toggle.sh status # show current state
# sudo /usr/local/sbin/koji-http-toggle.sh unblock # remove block
#
# Install:
# sudo install -m 0755 koji-http-toggle.sh /usr/local/sbin/koji-http-toggle.sh
#
# Env overrides (optional):
# PORT=<port> default: 80
# PROXY01=<hostname> default: proxy01.rdu3.fedoraproject.org
# PROXY10=<hostname> default: proxy10.rdu3.fedoraproject.org
#
# Examples:
# sudo PROXY01=proxy01.example.org PROXY10=proxy10.example.org \
# /usr/local/sbin/koji-http-toggle.sh block
#
# Notes:
# - IPv4 only (adjust type to ip6_addr + `getent ahostsv6` if you need IPv6).
# - If firewalld manages your host, prefer firewalld rich rules instead.
# - To persist direct-nft config across reboot:
# sudo sh -c 'nft list ruleset > /etc/nftables.conf'
# sudo systemctl enable --now nftables
#
# Troubleshooting:
# - "syntax error near `}'" → quote the chain creation command (we do).
# - "No such file or directory" → table/chain likely missing; the script
# creates them automatically.
# =============================================================================
set -euo pipefail
# Defaults (override with env vars if needed)
PORT="${PORT:-80}"
HOSTS=(
"${PROXY01:-proxy01.rdu3.fedoraproject.org}"
"${PROXY10:-proxy10.rdu3.fedoraproject.org}"
)
TABLE="inet filter" # nft table and family
CHAIN="input" # nft chain to hook into
SET="koji_http_block_src" # named set of source IPs to block
COMMENT="koji-http-block" # helpful tag for future auditing
# Resolve each hostname to a single IPv4 address (first match)
resolve_ips() {
local ip
for h in "${HOSTS[@]}"; do
ip="$(getent ahostsv4 "$h" | awk '{print $1; exit}')" || true
if [[ -z "${ip:-}" ]]; then
echo "ERROR: could not resolve IPv4 for $h" >&2
exit 1
fi
echo "$ip"
done
}
# Ensure table and chain exist; chain hooks into input with default accept
ensure_table_chain() {
if ! nft list table $TABLE >/dev/null 2>&1; then
nft add table $TABLE
fi
if ! nft list chain $TABLE $CHAIN >/dev/null 2>&1; then
nft "add chain $TABLE $CHAIN { type filter hook input priority 0; policy accept; }"
fi
}
# Ensure the named set exists to hold source IPs
ensure_set() {
if ! nft list set $TABLE $SET >/dev/null 2>&1; then
nft "add set $TABLE $SET { type ipv4_addr; comment \"$COMMENT\"; }"
fi
}
# Ensure a single drop rule exists that references the named set
ensure_rule() {
if ! nft list chain $TABLE $CHAIN | grep -q "tcp dport $PORT .* @$SET .* drop"; then
nft add rule $TABLE $CHAIN tcp dport $PORT ip saddr @$SET drop comment \"$COMMENT\"
fi
}
# Add the proxies' IPs into the set (safe to run multiple times)
block() {
ensure_table_chain
ensure_set
ensure_rule
local ips=()
mapfile -t ips < <(resolve_ips)
# Build "{ ip1,ip2,... }" for nft add element
local elems
elems="$(printf "{ %s }" "$(IFS=,; echo "${ips[*]}")")"
# Add elements; ignore if already present
nft add element $TABLE $SET "$elems" 2>/dev/null || true
echo "Blocked HTTP (tcp/$PORT) from: ${ips[*]}"
}
# Remove all IPs from the set and delete the drop rule
unblock() {
# Flush the set if it exists
if nft list set $TABLE $SET >/dev/null 2>&1; then
nft flush set $TABLE $SET
fi
# Remove the rule by handle (safer than pattern-delete)
if nft list chain $TABLE $CHAIN | grep -q "tcp dport $PORT .* @$SET .* drop"; then
local handle
handle="$(nft -a list chain $TABLE $CHAIN | awk '/tcp dport '"$PORT"'.* @'"$SET"'.* drop/ {print $NF}')"
if [[ -n "${handle:-}" ]]; then
nft delete rule $TABLE $CHAIN handle "$handle"
fi
fi
echo "Unblocked. (Set cleared and rule removed if present.)"
}
# Show current table, chain, and set contents
status() {
echo "=== nft: $TABLE ==="
nft list table $TABLE 2>/dev/null || { echo "(no $TABLE)"; return; }
echo
echo "=== Elements in @$SET ==="
nft list set $TABLE $SET 2>/dev/null || echo "(no set $SET)"
}
usage() {
cat <<EOF
Usage: $0 {block|unblock|status}
Env overrides:
PORT=<port> (default: 80)
PROXY01=<hostname> (default: proxy01.rdu3.fedoraproject.org)
PROXY10=<hostname> (default: proxy10.rdu3.fedoraproject.org)
Examples:
sudo $0 block
sudo $0 status
sudo $0 unblock
EOF
}
case "${1:-}" in
block) block ;;
unblock) unblock ;;
status) status ;;
*) usage; exit 1 ;;
esac

14
scripts_new/release-process/mass-rebuilds/README.md
View file

@ -1,14 +0,0 @@
# Mass Rebuild Tools
Scripts for orchestrating mass rebuilds during Fedora release cycles.
## Purpose
Tools that coordinate large-scale package rebuilds, typically performed twice per release cycle for toolchain updates.
**Examples of scripts that will be migrated here:**
- Mass rebuild orchestration
- Rebuild status tracking
- Bug filing for failed rebuilds
**Status**: 🚧 Awaiting migration

63
scripts_new/release-process/mass-rebuilds/find_FTBFS.py
View file

@ -1,63 +0,0 @@
#!/usr/bin/python -tt
# vim: fileencoding=utf8
#
# find_FTBFS.py - Find FTBFS packages
#
# SPDX-License-Identifier: GPL-2.0
#
# Authors:
# Till Maas <opensource@till.name>
#
from __future__ import print_function
import argparse
import operator
import koji
from massrebuildsinfo import MASSREBUILDS
kojihub = 'https://koji.fedoraproject.org/kojihub'
kojisession = koji.ClientSession(kojihub)
parser = argparse.ArgumentParser()
parser.add_argument("--check-tag", default="f29", help="Tag to check")
parser.add_argument("--since-rebuild", default="f27",
help="Mass-rebuild to use as reference for cut-off date")
parser.add_argument("packages", nargs="*", metavar="package",
help="if specified, only check whether the specified "
"packages were not rebuild")
args = parser.parse_args()
massrebuild = MASSREBUILDS[args.since_rebuild]
if args.packages:
all_koji_pkgs = args.packages
else:
all_koji_pkgs = kojisession.listPackages(args.check_tag, inherited=True)
unblocked = sorted([pkg for pkg in all_koji_pkgs if not pkg['blocked']],
key=operator.itemgetter('package_name'))
kojisession.multicall = True
for pkg in unblocked:
kojisession.listBuilds(pkg['package_id'],
state=koji.BUILD_STATES["COMPLETE"],
createdAfter=massrebuild['epoch'])
builds = kojisession.multiCall()
package_map = zip(unblocked, builds)
name_map = [(x['package_name'], b) for (x, b) in package_map]
# packages with no builds since epoch
unbuilt = [x for (x, b) in name_map if b == [[]]]
# remove packages that have never build, e.g. EPEL-only packages
kojisession.multicall = True
for pkg_name in unbuilt:
kojisession.getLatestRPMS(args.check_tag, pkg_name)
last_builds = kojisession.multiCall()
last_builds_map = zip(unbuilt, last_builds)
ftbfs = [p for p, b in last_builds_map if b != [[[], []]]]
print("\n".join(ftbfs))

144
scripts_new/release-process/mass-rebuilds/find_failures.py
View file

@ -1,144 +0,0 @@
#!/usr/bin/python
#
# find-failures.py - A utility to discover failed builds in a given tag
# Output is currently rough html
#
# Copyright (C) 2013 Red Hat Inc,
# SPDX-License-Identifier: GPL-2.0+
#
#
# Authors:
# Jesse Keating <jkeating@redhat.com>
# Ralph Bean <rbean@redhat.com>
#
from __future__ import print_function
import koji
import operator
import datetime