Add rdc-el-wrapper, an opinionated wrapper for ELN testing (#37)

After much discussion with @yselkowitz, this should be the best
option for ELN testing, I hope.

ELN (like other ELs) has multiple variants with repositories.
There is a mapping in Pungi config of relationships between the
variants. Each variant is expected to be repoclosure-complete,
possibly in itself, possibly with reference to some or all of
the other variants. Each variant contains only a subset of the
packages in the buildroot.

We have been asked to make it so that, given an arbitrary set of
"packages under test" for a given EL, the packages it contains
from each variant should be tested separately against that
variant, so we fail if replacing the packages in the variant with
the matching packages from the set under test would cause new
repoclosure issues in that variant, or the packages under test
for that variant have installability issues when tested against
the variant and any others it "depends" on.

Packages from the set under test that are not in any variant
should be ignored, as repoclosure of and installability within
the buildroot is not currently desired to be tested.

This wrapper script takes as inputs a compose that we should test
against, and a filesystem path containing the packages to be
tested. The compose is usually expected to be the latest compose
of the EL under test. Using the metadata, we discover all variants
in the compose that have repositories (except Buildroot, which is
ignored). We then list out the binary packages by name in each
variant repository.

We then split the set of packages-under-test into several temporary
variant repositories: each package is placed in the repo for all
variants of which the identically-named package in the compose is a
part, if any.

We then get and parse a copy of the variant mapping from the
Pungi config. Then, for each "populated" variant (that is, a
variant for which the packages-under-test set contains at least
one package), we run rmdepcheck, with the compose repos for the
variant plus any variants it "depends" on as the baserepos, the
local temporary variant repository as the newrepo, and the local
temp repos for all "depended-on" variants as addrepos. All
rmdepcheck output is passed through.

We track the return codes of each rmdepcheck run and return the
sum of each unique return code.

Signed-off-by: Adam Williamson <awilliam@redhat.com>

README.md

@@ -20,6 +20,22 @@ An alternative mode allows simply testing the consequences of *removing* a list
 entirely; in this mode, in the second step, we exclude all binary packages built from the specified
 source packages. The installability check is skipped in this context.
 
+## rdc-el-wrapper
+
+rdc-el-wrapper is an opinionated wrapper for running rmdepcheck on Enterprise Linux. The EL case
+is complicated because EL composes contain multiple variants with package repositories. These
+contain a subset of all packages in the buildroot. Each variant is expected to be repoclosure-
+complete with regard to itself and possibly some or all of the other variants; the mapping for
+this is kept in Pungi configuration.
+
+Because of this we cannot just test the packages-under-test against the buildroot, as we do for
+Fedora. We must split the packages-under-test set into per-variant repositories, with each package
+included in each repository it should be part of (we assume this mapping is not changed from the
+compose we're testing against, and discover it from that). We then test each of the split repos
+against the same variant repo from the compose, with other variant repos in scope on both sides,
+as per the Pungi mapping. This is a lot of work but should provide the most accurate and useful
+results.
+
 ## Requirements
 
 rmdepcheck has no run-time Python dependencies outside the standard library. Its only external
@@ -46,8 +62,8 @@ Simple usage looks like this:
 rmdepcheck https://a.base.repo.example/repo,file:///another/baserepo file:///the/testedrepo
 ```
 
-The to-be-modified base repositories are specified as a comma-separated list. Repositories are
-always specified as URLs. Only file:// , http:// and https:// URLs are accepted.
+The repositories are specified as a comma-separated list. Repositories should be specified as URLs,
+but for convenience, file:// can be omitted. Only file:// , http:// and https:// URLs are accepted.
 
 For the alternative 'removal' mode, usage looks like:
 ```
@@ -66,6 +82,15 @@ Note rmdepcheck is really only intended for use as a script, not as an importabl
 want to use it as a library go ahead, but this isn't a supported use case and bugs in it may not
 be addressed.
 
+rdc-el-wrapper usage looks like this:
+```
+rdc-el-wrapper https://kojipkgs.fedoraproject.org/compose/eln/latest-Fedora-eln/compose /the/testedrepo
+```
+
+An optional `--arch` arg can be passed as for rmdepcheck. Note that you **MUST NOT** include
+file:// in the tested repo arg, and it **MUST** be a local filesystem path. rdc-el-wrapper does
+not support remote tested repos.
+
 ### Testing build dependencies
 
 You can include source package repositories in the base repository set. This has the effect of

install.requires

@@ -0,0 +1 @@
+requests

rdc-el-wrapper.py

@@ -0,0 +1,255 @@
+#!/usr/bin/python3
+
+# Copyright Red Hat
+#
+# This file is part of rmdepcheck.
+#
+# rmdepcheck is free software; you can redistribute it
+# and/or modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation, either version 3 of
+# the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# Author(s): Adam Williamson <awilliam@redhat.com>
+
+
+"""Opinionated rmdepcheck wrapper that handles the awkward Enterprise
+Linux case, where we have to test against multiple shipped variants
+with complex lookaside relationships, all filtered from a shared build
+root.
+"""
+
+# Standard libraries
+
+import argparse
+import ast
+import os
+import pathlib
+import platform
+import shutil
+import subprocess
+import sys
+import tempfile
+from functools import partial
+from urllib.parse import urlparse
+
+from requests import Session
+from requests.adapters import HTTPAdapter
+from requests.packages.urllib3.util.retry import Retry
+
+VrDict = dict[str, str]
+
+DNFTEMP = tempfile.TemporaryDirectory(prefix="rdcelndnf", dir="/var/tmp")
+DNFARGS = ["dnf", "--setopt", f"cachedir={DNFTEMP.name}", "-q", "--disablerepo=*"]
+HERE = os.path.abspath(os.path.dirname(__file__))
+SESSION = Session()
+SUBPCAPTURE = partial(subprocess.run, capture_output=True, text=True, check=False)
+
+
+def split_put(put: pathlib.Path, repotemp: pathlib.Path, vrs: VrDict) -> set[str]:
+    """Splits a directory containing the packages to be tested into
+    multiple repositories per variant. put ('packages under test')
+    is the filesystem path containing packages under test (it cannot
+    be an HTTP URL, in this opinionated wrapper). repotemp is a Path
+    to a temporary directory to store the split repos. vrs is a dict
+    with variant IDs as keys and full compose repo URLs as values.
+    For each package under test, we get its name, look for a package
+    of the same name in each of the variant repositories, and if
+    there is one, we put the package in a subdirectory of repotemp,
+    named for the variant. At the end, we run createrepo in each of
+    the subdirectories we created, and return their names.
+    """
+    if not (put.exists() and put.is_dir()):
+        raise ValueError(f"split_put: invalid put value {put}")
+    put = put.absolute()
+    # generate per-variant package name lists
+    pkgs = {}
+    for variant, repo in vrs.items():
+        args = DNFARGS + [
+            "--repofrompath", f"{variant},{repo}", "repoquery", "--queryformat", "%{name}\n"
+        ]
+        pkgs[variant] = SUBPCAPTURE(args).stdout.splitlines()
+    # track which variants we 'populate'
+    populated = set()
+    for _file in os.listdir(put):
+        if _file.endswith(".rpm"):
+            pkgname = _file.rsplit("-", 2)[0]
+            for (variant, pkgnames) in pkgs.items():
+                if pkgname in pkgnames:
+                    populated.add(variant)
+                    os.makedirs(repotemp / variant, exist_ok=True)
+                    try:
+                        # hardlink
+                        os.link(put / _file, repotemp / variant / _file)
+                    except OSError:
+                        # copy
+                        shutil.copy2(put / _file, repotemp / variant / _file)
+    for variant in populated:
+        subprocess.run(("createrepo", str(repotemp / variant)), capture_output=True, check=True)
+    return populated
+
+
+def get_variant_repos(compose: str, arch: str) -> VrDict:
+    """Discover variants with repositories in a compose, return a
+    dict with variant IDs as keys and repo URLs as values.
+    """
+    resp = SESSION.get(f"{compose}/metadata/composeinfo.json")
+    resp.raise_for_status()
+    ci = resp.json()
+    variants = ci["payload"]["variants"]
+    ret = {}
+    for variant in variants:
+        if variants[variant]["id"].lower() == "buildroot":
+            continue
+        repopath = variants[variant]["paths"].get("repository", {}).get(arch, "")
+        if not repopath:
+            continue
+        ret[variants[variant]["id"]] = f"{compose}/{repopath}"
+    return ret
+
+
+def get_val():
+    """Get the variants_as_lookaside list which tells us which
+    variants pull in which other variants. FIXME: this is currently
+    ELN specific.
+    """
+    resp = SESSION.get(
+        "https://forge.fedoraproject.org/releng/pungi-fedora/raw/branch/eln/fedora/override.conf"
+    )
+    resp.raise_for_status()
+    override = resp.text
+    # 23 is the length of the search string. we really want to search
+    # for this exact string as it's significant
+    split = override[override.index("variant_as_lookaside = ")+23:]
+    val = split[:split.index("]") + 1]
+    # eek
+    return ast.literal_eval(val)
+
+
+def url_check(arg: str) -> str:
+    """Check arg is an http or https URL."""
+    parsed = urlparse(arg)
+    if parsed.scheme in ("http", "https"):
+        return arg
+    if parsed.scheme:
+        raise ValueError(f"Unsupported URL scheme {parsed.scheme} in {arg}")
+    raise ValueError(f"No URL scheme in {arg}")
+
+
+def check_arch(arg: str) -> str:
+    """Use host arch if no value passed."""
+    if arg:
+        return arg
+    return platform.machine()
+
+
+def check_utils() -> None:
+    """Check required utilities are installed."""
+    missing = []
+    for prog in (("dnf", "--version"), ("createrepo", "--version")):
+        try:
+            subprocess.run(prog, stdout=subprocess.DEVNULL, check=True)
+        except FileNotFoundError:
+            missing.append(prog[0])
+    if missing:
+        sys.exit("Please install missing required utilities: " + " ".join(missing))
+
+
+def parse_args() -> argparse.Namespace:
+    """Parse arguments with argparse."""
+    parser = argparse.ArgumentParser(
+        description=("Opinionated rmdepcheck wrapper for EL use cases.")
+    )
+    parser.add_argument(
+        "compose",
+        type=url_check,
+        help="The URL(s) of a compose to discover variant repos from and check against",
+    )
+    parser.add_argument(
+        "repo",
+        type=pathlib.Path,
+        help="Local filesystem path containing packages to test",
+    )
+    parser.add_argument(
+        "--arch",
+        type=check_arch,
+        default="",
+        help="Arch to operate on. Must match arch of passed repository",
+    )
+    return parser.parse_args()
+
+
+def main() -> None:
+    """Main loop."""
+    try:
+        # set up requests session with retries
+        retries = Retry(
+            total=5,
+            backoff_factor=0.5,
+            status_forcelist=[502, 503, 504],
+            allowed_methods={'GET'},
+        )
+        SESSION.mount('https://', HTTPAdapter(max_retries=retries))
+
+        # check DNF is installed
+        check_utils()
+
+        # set up args
+        args = parse_args()
+        if f"--forcearch={args.arch}" not in DNFARGS:
+            DNFARGS.append(f"--forcearch={args.arch}")
+
+        # find variants, do the repo split
+        vrs = get_variant_repos(args.compose, args.arch)
+        val = get_val()
+        with tempfile.TemporaryDirectory(prefix="rdcelnrepo", dir="/var/tmp") as repotd:
+            repotemp = pathlib.Path(repotd)
+            variants = split_put(args.repo, repotemp, vrs)
+
+            rets = set()
+            for variant in variants:
+                # figure out what other variants to pull in per val
+                depvars = set()
+                for pair in val:
+                    if pair[0] == variant:
+                        depvars.add(pair[1])
+                # put the compose repos for all variants in basearg, even
+                # ones we didn't 'populate' locally
+                basearg = ",".join(vrs[var] for var in set([variant]).union(depvars))
+                newarg = str(repotemp / variant)
+                # put depended-on variant-split PUT repos that actually
+                # exist in addrepos
+                # variants contains only variants we 'populated'
+                addarg = ",".join(str(repotemp / var) for var in depvars if var in variants)
+                # run rmdepcheck, track return code
+                # FIXME: what to do about JSON mode? capture output, combine?
+                rargs = (
+                    f"{HERE}/rmdepcheck.py",
+                    "--arch",
+                    args.arch,
+                    "--addrepos",
+                    addarg,
+                    basearg,
+                    newarg
+                )
+                rets.add(subprocess.run(rargs, check=False).returncode)
+
+        # we exit the sum of all *unique* return codes
+        sys.exit(sum(rets))
+
+    except KeyboardInterrupt:
+        sys.stderr.write("Interrupted, exiting...\n")
+        sys.exit(1)
+
+
+if __name__ == "__main__":  # pragma: no cover
+    main()
+
+# vim: set textwidth=100 ts=8 et sw=4: