ELN want rmdepcheck to run against composed repos, not buildroot #555
Labels
No labels
area/installability
area/jenkins
area/rpmdeplint
area/rpminspect
area/STI
area/STI-compat
area/Testing Farm
area/tmt
area/Zuul CI
blocked
bug
discussion
feature
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Type
Help
Type
Incident
UX
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Status/Blocked
Status/Discuss
Type
Bug
Type
Feature
Type
Task
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Blocks
Reference
ci/tickets#555
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Currently we run rmdepcheck against the buildroot repo for all distros it tests. However, @yselkowitz asked that for ELN it should be run against the variant repos in the latest compose, not against the buildroot. This is because the buildroot contains packages that are filtered out of the actual composes, and he doesn't consider it a goal for such packages to always be installable on all arches. He's only interested in maintaining repoclosure for the compose repos.
Doing this has the risk that we may get failures if an update requires a package from another update that went stable since the most recent compose. But this is less likely for ELN than Rawhide for two reasons: ELN is composed 3x per day, and ELN updates are created and tested in batches.
I already changed this in the openQA pipeline and sent a PR to do it in rmdepcheck-pipeline. @lecris requested a ticket here for tracking, so here it is.
Ugh, actually, I think this is more complicated. Um. So, if we just pass in only the composed repos as the 'modified base repos' in rmdepcheck terms, that can give us false errors for unsatisfied dependencies in the packages under test - the 'installability' check rmdepcheck does. The packages under test might need something that's only in the buildroot (because they're buildroot-only packages themselves, or whatever).
So I think we need to do something like...
Both of these require work on rmdepcheck itself, it's not possible to do either right now.
Yes, this is far more complicated than it seems. We only care about repoclosure for RHEL shipped repos. While we would like Extras to repoclosure as well, it's not perfect atm and is difficult to get right (at least the way we are currently handling Extras).
FWIW there is a Buildroot "variant" which contains any RPMs not in any other, minus certain RPMs which we intentionally do not ship in RHEL. That may be somewhat better than using eln-build (which also includes rawhide content for any packages not in ELN+Extras), but again we do not care about repoclosure there.
The only way to make this test reliable is to essentially act as pungi would do, by only testing those RPMs which end up in the compose for each arch, and ignoring those that end up in Buildroot. That does start by using shipped repos, but you also have to figure out what to test and what not.
I'm not sure I get what you're saying here. Pungi just composes the variants and runs repoclosure on each one, that's all it does. It's not complicated. It seems like you're implying that we have to do something beyond just identifying the variants, but I don't understand what that is, and Pungi does not seem to do anything beyond that. When I say "shipped repos" what I mean specifically is using the repos under https://kojipkgs.fedoraproject.org/compose/eln/latest-Fedora-eln/compose/ , except for Buildroot. That's what we implemented already. I think that's fine for the "repoclosure" check; the problem is we may get false failures for the "installability" check.
Will keep an eye out on the investigation here. The main purpose of this ticket is to port the final design to the Fedora-CI testing-farm profile https://gitlab.com/testing-farm/profiles/-/merge_requests/82
My latest proposed fix is an opinionated wrapper that runs rmdepcheck exactly the way yaakov asked for. If I understood him right. WIP, need to tweak it a bit tomorrow. If I stick with that plan, it will make life quite easy for the pipelines.
@lecris asked me a question which prompted me to think, hmm, are we already doing something like this in the existing pipelines for installability and rpmdeplint (etc)?
So I went looking, and the answer is more or less "kinda". This gets long, so I put a tl;dr section at the bottom; skip to that if you're bored.
Details
We have (at least) two sets of pipelines for running installability, rpmdeplint etc: osci - that is an RH-only link, sorry for non-RH folks - and fedora-ci. Both of them ultimately spin up a VM in testing-farm and then run the tests through mini-tps, passing in some env vars that affect its behavior and
archandossettings that I think tell testing farm what kind of VM to spin up. I've focused on the installability pipelines in the below write-up, I didn't look at rpmdeplint or others yet.Both pipelines run mtps-prepare-system on the VM, passing
-p <profile>to specify a profile; this is read out ofconfig.jsonin the pipeline repo by Jenkins and injected into the VM as an environment variable calledPROFILE_NAME. The fedora-ci pipeline also passes--fixrepo --enablebuildroot. The osci pipeline also passes--fixrepoand may pass--enablebuildroot- it's configurable at the Jenkins level via theENABLE_BUILDROOTvar (for CentOS Stream at least, Jenkins is called out of GitLab CI, presumably it can get set either way up there).mtps-prepare-system basically reads in args and turns them into environment vars - fun fact, at this point the environment has
PROFILEandPROFILE_NAMEset to the same value, I think - then looks for a repository file in<minitpsdir>/profiles/*/repos/which is named exactly<profile>.repo. If it doesn't find one, it bails. If it does find one, it sets it as env varPFILE, then figures out what the directory abovereposwas called and sets it as env varOS_ID- this will be "centos-stream", "fedora", or "rhel". Then it looks for a file<minitpsdir>/profiles/<OS_ID>/prepare-systemand sources it. Whew.For rhel and centos-stream, the
prepare-systemfile wipes/yum/repos.d/*.confand replaces it with the repo file it discovered earlier and stashed asPFILE; it installs this as/etc/yum.repos.d/profile.repoin, I think, all cases. For fedora, it only does this for ELN; otherwise it just uses the default repo config of the VM. For rhel, it can additionally install extra repository files; this seems to be used to add some kind of flatpak repository config. This is not used on CentOS Stream or Fedora.Here are sample repo config files for each profile:
As you can see, each one enables pretty much all the variant repos except Buildroot.
If
--enablebuildrootwas passed, the RHEL and CentOS profiles and the ELN branch of the Fedora profile enable the Buildroot variant repository. The non-ELN branch of the Fedora profile writes out a config file pointing to the 'latest' buildroot repo for the release.The
--fixrepoarg is ignored on fedora and centos-stream. On rhel, it does, oh god, horrible bash / sed things. AFAICT it seems to be trying to replace the/latest-RHEL-<ver>/component of the baseurl in each installed repo config file with the compose ID of the current "latest" compose. I've no idea why it does this; perhaps it's an attempt to guard against the "latest" compose changing during the execution of the test?Anyhoo! Point is: on RHEL, CentOS Stream and ELN paths, we're now executing in a VM with a tailored repository config that probably points to the "latest" compose, and has most of the variant repos enabled.
But wait, there's more!
The OSCI pipeline's
prepare.sh(which is executing in the VM) reads aTEST_COMPOSE_URLenv var. If that's set, it copies a template repo config to/etc/yum.repos.d/test-compose.repo, replacing placeholder strings in the template with the URL. The template expects the URL to be the top level of a compose, and enables most of the variant repos within that compose. So ifTEST_COMPOSE_URLis set, we now have two sets of compose repositories, one for the "latest" compose, one for whatever was passed in asTEST_COMPOSE_URL. I would kinda expect this path to wipe or disable/etc/yum.repos.d/profile.confbut it doesn't seem to.I cannot figure out exactly where
TEST_COMPOSE_URLgets set (or not) - I cannot find the bit in the GitLab CI stuff that triggers Jenkins; it's somehow osci-pipelines but the string 'installability' does not appear in that repo so it must be reading the info on what Jenkins pipelines to trigger from somewhere else, but I can't figure out where. There seems to be a thing called 'ci-mediator' involved but I can't find it.Anyhow at least on the specific job I'm using as a base for my investigation, which was triggered from a CentOS Stream PR, once we get through Jenkins to Testing Farm, we can see that it is set, and so is another important var:
As far as I can figure,
TEST_COMPOSE_URLgets set when we're in some kind of flow which also includes a test compose - i.e. before we hit installability, we've done a test compose with the packages under test. It gets set to the URL of that compose.The
TEST_SHIPPED_RPMS_ONLYvar is also significant. That gets read by the OSCI pipeline'sprepare.shwhich turns it into the--onlyinrepoparam to mtps-get-task, which it calls after all the prep stuff above is done. In mtps-get-task, it gets turned into two calls to a filter_available function, which figures out the package name for each of the RPMs under test, then callsdnf infoon it and filters it out if there's no result. i.e., it filters out packages which aren't present in any repository enabled on the system at the timemtps-get-taskis run - which should be the repositories that result from all the above shenanigans.There's a note in the Jenkinsfile that
TEST_SHIPPED_RPMS_ONLY"requires TEST_COMPOSE_URL", which I don't entirely understand. I think maybe it really means that it will only be entirely reliable if we're on a test compose flow? Otherwise we'll determine whether the packages under test are "in the compose" based on the 'latest' compose, which I guess may not be considered 100% accurate? @msrb if you can explain in more detail that'd be great. I also thinkTEST_SHIPPED_RPMS_ONLYplusENABLE_BUILDROOTwon't filter anything out, but probably you'd never intentionally set those together.ENABLE_BUILDROOTis not respected for theTEST_COMPOSE_URLrepo config - the template file that's used does not contain the Buildroot variant at all - but the Buildroot repo inprofile.confwould be enabled, and presumably all the packages would be found in that if they weren't found anywhere else.tl;dr
Anyhow: if I've understood all that correctly, for EL cases, the installability pipeline and mini-tps always test in an environment where most variant repos are enabled. If
ENABLE_BUILDROOTis set, the Buildroot variant repo is enabled; if not, it isn't. IfTEST_SHIPPED_RPMS_ONLYis set, mini-tps will attempt to filter out packages that are not in any of the enabled variant repos when it is downloading the set of packages to test.So...it does at least avoid testing buildroot-only packages. But it doesn't do the precise variant mapping we do in this wrapper. If a package under test is intended for variant X, but depends on something from variant Y which is not in variant X's lookaside definition, the test will pass when it should fail, I believe.
I initially gave a quick glance over mtps and concluded it is horrible, the details are even more :/. Let's not try to replicate that. Enabling the buildroot and (appropriate?) eln repo should be handled on Fedora-CI side using testing-farm profile/policy (effectively injecting a specified tmt
preparestep) for all tests (installability,rpminspect,rmdepchecketc.) and we can then tap those tests to use the appropriate repo/artifacts.So on the wrapper side, what would be the input for it to figure out what repos/buildroot to enable? The most common input I have are the
.rpmfiles to be tested, and one layer above that thekoji_build/bodhi_updatethat provided them. Anything more than that would need some custom mapping or sorts, or is there some other idea on how to get these?Also, how do we handle bodhi updates that span multiple repos? That is a specific issue for
rmdepcheck, I think for all the other cases we just enable all repos under it.I already figured all that stuff out for rmdepcheck, basically - it's in quality/rmdepcheck#40 . We figure out the variant of each package in the tested set by what repo it's in in the base repos (this doesn't handle a package moving, of course, but then a package moving between variants is an operation independent of the update system anyway so there's no really reliable way to handle that), and we get the mapping of what other variants each variant depends on from pungi config (it's rather ugly, though, I wish we could make that file be...not Python). The wrapper approach should make things very easy for the rmdepcheck pipeline in Fedora CI - it just needs to know the thing it's testing is for ELN, and if so, call the wrapper. This is the openQA pipeline change - as you can see it's very easy.
I wanted to make sure I wasn't duplicating work that was already done, and switch track if I was. Given that the jenkins/mtps path is pretty gnarly, and it doesn't handle the per-repo split, and the fact that rmdepcheck's execution context is somewhat different, I think it's OK for me to keep going on the rmdepcheck-specific-wrapper approach at least for now.
But yeah, if we want to downstream rmdepcheck or figure this stuff out for the other generic tests, we might need to reconcile things somehow. I think diverging from the mtps setup does come with a bit of a risk of making it harder to share stuff between Fedora, RHEL and CentOS, although the fact we already have separate pipelines I guess implies that's already not working perfectly?
The execution context does make it a bit awkward to share things between rmdepcheck and the other tests: rmdepcheck doesn't care at all what repositories are configured in the environment it's running in, it always ignores them. It needs the appropriate repository setup to be passed in on the command line. I suppose I could add a mode where we treat all enabled system repositories as modified base repositories...but that still wouldn't let us get things right easily because of the variant split issue.
I do think the variant repo split would actually be necessary for
installabilityto give correct results for ELN. I don't think it's necessary forrpminspect.Consider the case where packages in variant B are only meant to depend on other packages from variant B, or packages from variant A. If an update includes a package from variant B, and changes its dependencies so it now depends on something from variant C, it should fail
installability, but I think at present it would not.@adamwill wrote in #555 (comment):
I do not think it would be possible for
installabilityto account for that, not without significant work onmini-tps. I am fine withrmdepcheckcovering the issues from that instead. The coverage forinstallabilityis mainly on the scriplets and upgrade (/downgrade for unknown reason) paths, and I don't think we would have interesting cases for those?Ah, fair point, yeah. I could imagine perhaps a very edge case where a scriptlet requires something from an out of scope repo, but mehh.
I agree the current downstream setup is labyrinthine and difficult to understand. Actually, this is probably the most complete write up that exists for it, at least that I have seen.
The
Jenkinsfiles for this are stored in an internal Red Hat system, but TL;DR theTEST_COMPOSE_URLis determined by finding thetest-composejob corresponding to the build artifact. Thetest-composejob actually runs in GitLab now, not in Jenkins, so we are going to start taking theTEST_COMPOSE_URLvia GitLabenvmetadata rather than trying to work it out on the fly.The idea was to create a generic service to mediate between the billion different test environments. From one system (GitLab for us), we
POSTa "workitem" to CI Mediator - a description of he job we want to run. Then a CI system - which could be anything at all - listens/watches/polls/whatever CI Mediator, detects that "workitem", runs a job, then posts back a "result" object to CI Mediator. Our system detects the result, parses it and makes it pretty for the end user, and exits.There were lots of potential advantages to using CI Mediator; it provided a nice generic interface with retries to any CI system with minimal work, it decoupled the test-triggering infra from GitLab (which would have made it easier to hook it into Fedora), to name a few. It didn't take off as expected though, and now Konflux has arrived, and we will probably discontinue it and run
installabilityand all the other tests either in Konflux directly, or via Testing Farm directly from GitLab, or some similar plan that involves the certain demise of Jenkins.Collaboration on this front would be welcome, and any points where we could converge I would view with great interest.
Thanks for the info!
Just out of interest - in Fedora this is kinda done at the message system level. When a Bodhi update is created or edited, a specific message is published. Test systems are expected to listen out for that message and run appropriate tests, then report the results into resultsdb in a specific expected format. If you do that, your results show up in the UI and can be gated on.
We don't exactly do this for pull requests at present though (only "Fedora CI" is running tests on pull requests ATM, I think). And we have a much smaller multiplicity of test systems to worry about.
I am all in favor of the demise of Jenkins! I think lecris has similar plans for "Fedora CI", we'll probably get together and figure out some details at Flock. Are you going to be there by any chance?
For the older generic tests I think / hope we should try to align things as much as reasonably possible as we go forward, without forcing too much weirdness on Fedora for things it happily doesn't have to care about (like modularity and mysterious Flatpak side repos).
For rmdepcheck specifically, now I've dug through everything, I think the execution context primarily makes it hard to share the setup with the older generic tests. rmdepcheck doesn't expect to run in an environment that matches the packages being tested; you can run it out of literally any environment with a new-enough dnf in it. It expects you to pass appropriate arguments for the packages being tested. That's sufficiently different that sharing the setup steps doesn't really make sense, I don't think. I'm going to continue work on my approach of basically providing an alternative interface for running it in EL contexts, where you just tell it the compose you want to test against and it figures out the rest. I'm probably not going to try and handle modularity/flatpak repos/whatever for now, if there's interest in adopting rmdepcheck in downstream contexts where that would be needed, we can work it out later.
The current system is message based, so that's basically how it works for everyone except OSCI going via CI Mediator. Tests are triggering off draft build completion messages (except the jobs that trigger off test compose completion messages).
One big problem we have is that, In the "old" gating, greenwave was listening out for messages for tests continuously. So if a QE team retried a test and it passed, or it got waived, greenwave picks up on that and you're done - no manual intervention required. We don't have that in GitLab as once the test completes, the job closes and that's that. Then we get into inconsistent behaviour territory. Depending on who wrote the test, either:
One of the reasons CI Mediator didn't catch on was people didn't want to/didn't have capacity to provide the on-demand trigger necessary for the system to work nicely, controlled from GitLab. The existing message-based system is a compromise, to get the tests running in the merge requests.
We could of course write a service to listen for these retry messages and do the job greenwave used to do for us, but that is antithetical to the "turn off as much as possible" crusade. Most of the QE teams do want to migrate to use TF/Konflux, so we will probably one day get to the stage where we do have the triggers necessary to sidestep UMB/Kafka completely and run the show from GitLab/Konflux, but we're not there yet and it is not imminent.
Unfortunately not able to secure that budget this year, which is a shame. Going forward it will be easier, as I will have my own travel and accommodation sorted. My wife and I home school, and we are having our midlife crises. We went down to Manchester on Monday and bought an ex-police riot van - a 7m long Mercedes Sprinter - and we're going to convert it into a mobile school/office. It's called Mercules, and I'm going to put my desk in the cell that's still in the back. I'll definitely be at the next European Flock.
Agreed 100%.
Makes sense 👍
Heh, funny story on that: we never found that behavior reliable in Fedora and I made an argument that it is actually contrary to the microservices design. It requires greenwave to have some knowledge of the decisions it will be asked for (in order to know what messages to act on how) and that's not how it's supposed to work.
So in Fedora we stopped using the greenwave "decision update" mechanism at all. Instead, Bodhi listens for 'new result' messages from resultsdb and 'new waiver' messages from waiverdb, and decides when to ask Greenwave for a new decision. We've found this much more accurate and reliable.
Of course, it helps that we only have one system querying Greenwave. If you have several, implementing this in all of them probably adds up...
So, the practical requirement here should be fixed with https://github.com/fedora-ci/rmdepcheck-pipeline/pull/23 I believe. @lecris any reason not to close this?