Drop/migrate and consolidate jenkins jobs #553
Labels
No labels
area/installability
area/jenkins
area/rpmdeplint
area/rpminspect
area/STI
area/STI-compat
area/Testing Farm
area/tmt
area/Zuul CI
blocked
bug
discussion
feature
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Type
Help
Type
Incident
UX
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Status/Blocked
Status/Discuss
Type
Bug
Type
Feature
Type
Task
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
ci/tickets#553
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Breaking down the state of https://osci-jenkins-1.ci.fedoraproject.org/:
*/copr-rebuildbranch)dist-git-pipelinedoes not support bodhi-updates yet*-pipelinescratch-build-test-elnbranch)scratch-build-test is that odd one with a hand-curated mapping of 'if the package under test is A, do scratch builds of X, Y and Z against it' mappings, right?
I'm not sure specifically, no. I'm not up on the details of who owns this test and who wants it to exist / values the results from it.
My sort of default answer to any "how can we move this off Jenkins?" question would be something like "can we just have a fedora-messaging consumer which runs it as a Testing Farm job?" but that's a fairly naive take as I don't really have any detailed knowledge on what "running things as Testing Farm jobs" looks like in practice.
I'm glad you tagged me in this ticket, though, because weirdly I was doing exactly the same exercise over the weekend: thinking "why do we have/need a Jenkins instance at all?" and looking through the list of stuff in it to see what to do with it...
Yes that is indeed the case, and here is the said map. The challenge of converting the test into a tmt one is the managing of the koji token
The answer to this is no, because testing-farm is not in charge of communicating results back, that's packit's job. But otherwise yes, a testing-farm+packit would be possible for this, and that's why I am making them in line with packit's usage for PRs.
Plan looks solid to me, but we need OSCI input, I am not the architect of these, and we do not want to break anything. I believe some of the pieces might be shared with RHEL CI.
@mvadkert wrote in #553 (comment):
Could you ping them to take a look. I have pinged @msrb, but I do not believe he will get the notification
Asked OSCI to provide feedback:
https://redhat-internal.slack.com/archives/C04L304KVH9/p1775642351118089
I don't know about any of the ELN pipelines, but I think they are just some leftovers from many years ago.
I don't think the service which ingests results to the ResultsDB supports bodhi-updates, or at least it never worked for me: https://pagure.io/fedora-ci/general/issue/145
What would be the benefit? It's easy to test changes in tmt and surrounding scripts if those files live next the pipeline files. But I don't mind if those files are moved.
The fedora-ci/*-pipeline(s) are only used by Fedora CI.
This is where container images are created. Those container images are then referenced in tmt files in the pipeline repositories. I am not sure what's the goal here? tmt files are moved away from the pipelines repos, and this code is merged to the pipeline repos? Note these two repos don't have any jobs in Jenkins.
I don't know what this is, no idea who created it 🤷
Contact person for this test would be Martin Cermak (rh-mcermak at GitHub).
@msrb wrote in #553 (comment):
See
rmdepcheck-pipeline. The plan for now is to duplicate the results. Will work on better ways to extract relevant tests/checks from testing-farm later on.@msrb wrote in #553 (comment):Better contribution workflow from the Fedora users and aligns with the general migration to forge.fp.o. I am not against having a mirror on github where additional contributors can come.@msrb wrote in #553 (comment):
To avoid having to maintain a container, instead moving those steps into the
preparestep of the tmt file.rpmdeplint-pipelinealready underwent that change,rpminspectone is more involved.I see, the results are not reported for the Bodhi update, but for the individual builds. rpmdelint/rpmdepcheck is probably one test where it would make more sense to report results on the update as a whole, because that's what being tested.
dist-git-pipeline runs separate tests for individual builds in the update, so reporting for each build makes sense (as it is today).
I don't mind either way, but then those -runner files should move there as well? Because the tmt files are just metadata.
Fair enough. Although I'd argue that none of these tests actually need tmt 😅
I believe https://forge.fedoraproject.org/apps/ci-resultsdb-listener is still what we're using to forward 'ci' result messages to resultsdb. It's not particularly complicated; it shouldn't be more than a day or two's work to make it support update messages. But we'd probably want it to somehow filter out the ones published by systems which report directly to resultsdb as well as publishing
cimessages - at least openQA, not sure about others.@adamwill I think it would be better to move away from the messaging+listener solution completely. If people think that rpmdeplint/rpmdepcheck pipelines should be reporting on Bodhi updates (instead of individual builds), then my suggestion would be to simply use the ResultsDB API.
However... since "fedora_update" is basically a different "subject_type", will Greenwave continue working correctly in Bodhi? Or is Bodhi already querying Greenwave separately for build decisions and the overall update decision? Or maybe I am looking for a problem that is not even there...
I'm a bit confused here. We need a message consumer of some kind, at minimum, to know when to run tests; the standard way to do this (used by both openQA and the Jenkins triggers, AIUI) is to listen for the
org.fedoraproject.prod.bodhi.update.status.testing.koji-build-group.build.completemessage, which (since a few years ago) is carefully set up to be published at exactly the points we want testing systems to test updates (i.e. when they're created or edited). For openQA we use a Python fedora-messaging consumer for this; for Fedora CI we have the triggers which use a thing calledrabbitMQSubscriberthat they get from...somewhere (I do not understand how that works).When I wrote "can we just have a fedora-messaging consumer which runs it as a Testing Farm job?" I was talking about triggering, not reporting. The bit I've never entirely understood is why we are, AIUI, essentially using Jenkins to do the job fedora_openqa does for openQA - listen out for a message, do some prep steps, then trigger a test run in a test system (openQA in the fedora_openqa case; Testing Farm in the Fedora CI case). A whole Jenkins instance feels like overkill for the job. It also introduces the confusion for end users (packagers) trying to look at test results - they never understand why sometimes they wind up looking at a Jenkins job where they have to dig out the URL to the Testing Farm job where the actual tests ran, and sometimes they wind up seeing the Testing Farm job. fedora_openqa is entirely obscured from end users' view, they never see any artifact from it; if something goes wrong in it, the sysadmins (i.e. me) get a bunch of failure emails and it's our (my) job to deal with it.
Yes, because openQA reports results against updates.
Ok, I misunderstood -- I was talking about reporting 😅
Testing Farm would need to know what to run for each release/component, store configuration for each workflow, and know where to report those results afterwards? Isn't Packit in the same boat with Jenkins when it comes to Testing Farm integration?
Here's what I think could work: don't trigger on builds. Have Bodhi (or something else) basically submit requests to Testing Farm -- this is the update, with these builds inside. Run these tests for me, and use these webhooks when you're done. However, this would be a Bodhi-specific solution (not a bad thing).
I admit that I don't know much about fedora_openqa+openQA. Jenkins is a CI system, although quite outdated. Before Testing Farm, the testing was actually happening on the Jenkins instance, not outside. So it was basically fedora_openqa+openQA together? With Testing Farm came some significant benefits: like easy access to composes+a dedicated team that would make latest composes available. And challenges - it's basically a one way street: you tell us what tmt tests to run, and on what compose, and we will let you VIEW the results. Which is great for individual developers/QEs ("run my tests and show me the results" -- super easy, exactly what they need), but less great for integration.
And last but not least: Jenkins is what many teams are running internally.
However, I do agree that the whole architecture feels off.
If Jenkins manages to call Testing Farm, then the URL propagated back to the user should be the TF results page, not Jenkins.
If Jenkins manages to submit a scratch build to Koji (well, not anymore, Packit is doing this now), then the URL propagated back to user should be the Koji build page.
If Jenkins fails to call those external services, it falls back to sending its own URL.
Or at least that's what should be happening. If not, then it's a bug. How would fedora_openqa handle the situation when openQA doesn't respond?
This is already pretty close to what we're doing, for at least some of the triggers/pipelines. We don't trigger on Koji builds any more. We trigger on Bodhi update-ready-for-testing-messages, which look like this. The messages contain the update ID, the list of builds in it at the time the message was published, and the sheer existence of the message communicates "test this now" - that is the sole purpose of these messages.
Yes, this is Bodhi-specific, as you mentioned. That's why (AIUI) @lecris is trying to set things up so we can also trigger the same tests for e.g. dist-git pull requests, with as much shared code and as sensible a design as possible.
I'm pretty sure I still wind up staring at Jenkins sometimes. I'll try and remember to file an issue next time it happens.
It retries until openQA damn well does respond, and floods my inbox with complaints until it's fixed. :D This behavior is rather configurable, though, for fedora-messaging consumers; you have a lot of control over what the consumer should do if there's an exception. See here.
@adamwill wrote in #553 (comment):
From what I remember it always had an exit code is 2 not 1. I cannot find an example case anymore (and we would need to be quick to store them since we have a 3 day retention of the logs)