- Python 98.7%
- Shell 1.3%
Adam Williamson
f209553299
The main goal here is to also download and modify filelists and modules metadata to fix some problems I saw in EPEL 9 update tests. We need the filelists metadata to get correct results if a package has a dependency on a file that is not present in the primary metadata; if we don't also download filelists, we'll get an incorrect "new" broken dependency because repoclosure on the modified repository will not be able to find the package that contains the file. Similarly, we need modules metadata (if present) to ensure that dnf knows module packages are module packages. Without the modules metadata it treats all module packages as non-module packages, so repoclosure on the modified repo might incorrectly say a dep of a non-module package is fixed because it can be satisfied by a module package. However, there was a big trap lurking here: the filelists metadata is even larger than the primary metadata, and both are kinda huge. Previously we were using ElementTree.parse(), which loads the entire XML tree into memory; this was already using >6GB of RAM for a typical primary metadata file. If you tried to load both primary and filelists into RAM at once (as my initial attempt did) it uses a huge amount of RAM and probably gets OOM killed. So, we'll parse the XML as a chunked bytestring. As we go along, we split out package elements, one by one. Everything that is not part of a package element gets passed straight through to the output file. When we encounter a package element, we parse it with lxml (which gives us a nice ~2x speedup over ElementTree). We decide whether to drop it. We do this the same way as before for the primary metadata, but record the pkgid (which is usually the checksum). When parsing the filelists metadata, we take the list of pkgids removed from the primary metadata as input, and remove all package elements with the same pkgid. If we decide to drop the package, we move to the end of it in the current input chunk. Otherwise, we write the chunk through to the output file, then move ahead and continue. We use the `open()` methods of various compression libraries to achieve transparent decompression and recompression, and track uncompressed checksums and sizes along the way. This also adds support for various other compression formats; previously we assumed zstd. Supporting at least gzip is important as there are still extant RHEL releases with only gzip-compressed metadata. We switch the other uses of ElementTree to lxml for consistency. Signed-off-by: Adam Williamson <awilliam@redhat.com> |
||
|---|---|---|
| .forgejo/workflows | ||
| tests | ||
| .gitignore | ||
| CHANGELOG.md | ||
| COPYING | ||
| install.requires | ||
| pyproject.toml | ||
| README.md | ||
| release.sh | ||
| rmdepcheck.py | ||
| tests.requires | ||
| tox.ini | ||
| tox.requires | ||
rmdepcheck
rmdepcheck is an RPM dependency check tool based on a repository metadata modification approach. It works by comparing a checked repository to one or more base repositories. First, checks are run on the base repositories as-is. Next, modified copies of the base repositories' metadata is created, with all packages from the same source RPM(s) as the package(s) in the checked repositories removed. Finally, checks are run again on the modified base repositories, with the checked repositories available to the dependency solver. The results of the two runs are compared. New failures should indicate problems introduced by the checked repositories. Also, some relevant checks are run on the checked repositories with reference to the modified base repositories.
Optionally, additional base repositories can be specified which will not be modified, and additional new repositories can be specified which will not be checked directly. The former is intended for testing scenarios like stable Fedora releases, which have a frozen release repository which is never modified, and an updates repository which is updated. The latter is intended for multilib scenarios; it may be desirable to use such an additional repository for packages for the multilib arch(es), if e.g. installability of these should not be tested directly.
An alternative mode allows simply testing the consequences of removing a list of source packages entirely; in this mode, in the second step, the base repository's metadata is modified to entirely remove all binary packages built from the specified source packages. The installability check is skipped in this context.
Requirements
rmdepcheck has no run-time Python dependencies outside the standard library. However, it requires several command-line utilities:
- dnf
- zstd
- curl
It checks for these, and will exit early with an error if any of them is not found. rmdepcheck is written primarily for Red Hat-family distributions, but should in theory be usable anywhere these utilities can be installed (and forward slashes act as directory separators).
If you use a version of dnf older than 5.2.15.0, you may see false failures for 'rich' dependencies, as older dnf versions did not handle these correctly. Use 5.4.0.0 or newer for the best handling of 'rich' dependencies (5.2.15.0 through 5.3.0.0 ignored them entirely; 5.4.0.0 checks them correctly).
Installation
Installation of rmdepcheck is entirely optional, it can be run just as well directly from the
repository. Otherwise, rmdepcheck uses setuptools for installation and is PEP 518-compliant. You
can build and install with e.g. the build module and pip. rmdepcheck can also be installed
directly from PyPI with pip and other tools.
Usage
Simple usage looks like this:
rmdepcheck https://a.base.repo.example/repo,file:///another/baserepo file:///the/testedrepo
The to-be-modified base repositories are specified as a comma-separated list. Repositories are always specified as URLs. Only file:// , http:// and https:// URLs are accepted.
For the alternative 'removal' mode, usage looks like:
rmdepcheck --removes https://a.base.repo.example/repo,file:///another/baserepo sourcepkg1,sourcepkg2
This tests removing sourcepkg1 and sourcepkg2, and all binary packages built from them, from the base repositories.
If you want to test repositories containing packages of an arch that does not match the system on
which you are testing, pass --arch <arch>, where <arch> is the arch you wish to test.
For more complex usage, see rmdepcheck --help.
Note rmdepcheck is really only intended for use as a script, not as an importable library. If you want to use it as a library go ahead, but this isn't a supported use case and bugs in it may not be addressed.
Testing build dependencies
You can include source package repositories in the base repository set. This has the effect of
testing for build dependency breakages, as the dependencies of a source package are its build
requirements. You can identify build dependency breakages in the output by the package name ending
in .src.
When doing this, you must also include at least one matching binary repository, or all requirements of all source packages will be unresolvable and the tool will run slowly and produce useless output.
Example:
rmdepcheck https://a.base.repo.example/binaryrepo,https://a.base.repo.example/sourcerepo file:///the/testedrepo
License
rmdepcheck is released under the GPL, version 3 or later.
See COPYING and the header of rmdepcheck.py itself.
Contributing
Issues and pull requests can be filed in Fedora Forge.
ANY USE OF AI/LLM IN THE PRODUCTION OF A PULL REQUEST MUST BE CLEARLY DISCLOSED. This is for legal reasons, as the copyrightability of LLM-generated code is (as of April 2026) disputed and unclear. This is also for technical reasons, as reviewers may need to look out for different problems when reviewing LLM-generated code, compared to human-generated code.
Please include an Assisted-by line in the commit message, specifying the model, tool and/or
service used in creating the pull request, and a more detailed explanation of how LLM technologies
were used in the pull request description. This can be copied/pasted from PR to PR if the workflow
remains the same.
Here is a sample commit message:
Make the frobnosticator reticulate splines better
By rejigging the frobnosticator, we can reticulate splines twice as fast!
Signed-off-by: Bob Roberts <bob@example.com>
Assisted-by: OpenCode 1.3.15 | claude-4.6-opus-high
Pull requests must be signed off (use the -s git argument). By signing off
your pull request you are agreeing to the
Developer's Certificate of Origin:
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.