Packaging two Python scripts, one of which is a wrapper for the
other, is pretty awkward. But more importantly, we can make the
EL "wrapper" work better with a refactor, too. It's now a stand
alone script called `rdc-compose`. The original script is renamed
to `rdc-repos`.
The CLI scripts now share a lot of bits via `shared.py`. We put
handy run-it-locally wrappers for each CLI script at the top
level; you *can* install the package properly and you'll get
`rdc-repos` and `rdc-compose` executables but we'll likely rarely
do that.
`rdc-compose` will now run faster because it will reuse the same
metadata between variants (previously it got re-downloaded with
every execution of `rmdepcheck.py`) and it can now nicely collect
up all the repoclosure strings before using the same parse-and-
exit code as rmdepcheck, which solves a lot of awkward issues
around output.
We also add tests covering `rdc-compose`, as part of which it made
sense to factor out the Pungi config URL discovery (which I meant
to do anyway). We fix a bug that was exposed by writing the tests:
we have to check variants that depend on variants we have packages
for, even if we don't have packages for those variants.
This isn't great git hygiene, sorry. Should have made the refactor,
the addition of tests and the bug fix into separate commits. But
it's awkward and time-consuming to unpick now and I don't think
it's a good use of time.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
After much discussion with @yselkowitz, this should be the best
option for ELN testing, I hope.
ELN (like other ELs) has multiple variants with repositories.
There is a mapping in Pungi config of relationships between the
variants. Each variant is expected to be repoclosure-complete,
possibly in itself, possibly with reference to some or all of
the other variants. Each variant contains only a subset of the
packages in the buildroot.
We have been asked to make it so that, given an arbitrary set of
"packages under test" for a given EL, the packages it contains
from each variant should be tested separately against that
variant, so we fail if replacing the packages in the variant with
the matching packages from the set under test would cause new
repoclosure issues in that variant, or the packages under test
for that variant have installability issues when tested against
the variant and any others it "depends" on.
Packages from the set under test that are not in any variant
should be ignored, as repoclosure of and installability within
the buildroot is not currently desired to be tested.
This wrapper script takes as inputs a compose that we should test
against, and a filesystem path containing the packages to be
tested. The compose is usually expected to be the latest compose
of the EL under test. Using the metadata, we discover all variants
in the compose that have repositories (except Buildroot, which is
ignored). We then list out the binary packages by name in each
variant repository.
We then split the set of packages-under-test into several temporary
variant repositories: each package is placed in the repo for all
variants of which the identically-named package in the compose is a
part, if any.
We then get and parse a copy of the variant mapping from the
Pungi config. Then, for each "populated" variant (that is, a
variant for which the packages-under-test set contains at least
one package), we run rmdepcheck, with the compose repos for the
variant plus any variants it "depends" on as the baserepos, the
local temporary variant repository as the newrepo, and the local
temp repos for all "depended-on" variants as addrepos. All
rmdepcheck output is passed through.
We track the return codes of each rmdepcheck run and return the
sum of each unique return code.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Instead of painfully retrieving, decompressing, parsing, editing
and reconstructing the repodata, let's use dnf's excludepkgs
option to achieve the same thing. This is much simpler, faster,
and more robust.
To 'modify' the modified base repos, we use dnf repoquery to
list the packages they contain and their source RPM names, then
construct a table with the NEVRs to be removed for each repo.
Then when we do the repoclosure command, we pass
--setopt <repo>.excludepkgs=<nevrlist> for each modified repo.
This tells dnf to act as if the specified NEVRs simply do not
exist in the specified repo, which is exactly what we want to
achieve.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The main goal here is to also download and modify filelists and
modules metadata to fix some problems I saw in EPEL 9 update
tests. We need the filelists metadata to get correct results if
a package has a dependency on a file that is not present in the
primary metadata; if we don't also download filelists, we'll get
an incorrect "new" broken dependency because repoclosure on the
modified repository will not be able to find the package that
contains the file.
Similarly, we need modules metadata (if present) to ensure that
dnf knows module packages are module packages. Without the modules
metadata it treats all module packages as non-module packages,
so repoclosure on the modified repo might incorrectly say a dep
of a non-module package is fixed because it can be satisfied by a
module package.
However, there was a big trap lurking here: the filelists metadata
is even larger than the primary metadata, and both are kinda huge.
Previously we were using ElementTree.parse(), which loads the
entire XML tree into memory; this was already using >6GB of RAM
for a typical primary metadata file. If you tried to load both
primary and filelists into RAM at once (as my initial attempt did)
it uses a huge amount of RAM and probably gets OOM killed.
So, we'll parse the XML as a chunked bytestring. As we go along,
we split out package elements, one by one. Everything that is
not part of a package element gets passed straight through to the
output file.
When we encounter a package element, we parse it with lxml (which
gives us a nice ~2x speedup over ElementTree). We decide whether
to drop it. We do this the same way as before for the primary
metadata, but record the pkgid (which is usually the checksum).
When parsing the filelists metadata, we take the list of pkgids
removed from the primary metadata as input, and remove all
package elements with the same pkgid.
If we decide to drop the package, we move to the end of it in
the current input chunk. Otherwise, we write the chunk through
to the output file, then move ahead and continue.
We use the `open()` methods of various compression libraries to
achieve transparent decompression and recompression, and track
uncompressed checksums and sizes along the way. This also adds
support for various other compression formats; previously we
assumed zstd. Supporting at least gzip is important as there are
still extant RHEL releases with only gzip-compressed metadata.
We switch the other uses of ElementTree to lxml for consistency.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
