Enabling some monitoring for Fedora-CI

lecris commented

2026-04-22 14:13:16 +00:00

Description of request

We would like to ask some help in setting up the monitoring for Fedora-CI. Currently the other monitoring infrastructures that we have available is packit's and testing-farm's which are not publicly visible, and that is primarily why I think getting something worked together with Fedora-infra would be better.

The main things that the monitoring should have are:

Be publicly visible
Catch when more than n tests are failing or are taking more time than normal
Report when something is going wrong on matrix or slack or anything else
Allow for other interesting parties to get relevant notifications, e.g. when many rpminspect tests fail have the rpminspect team CC-ed

The current infrastructure in Fedora-CI:

Jenkins layer that executes testing-farm jobs on bodhi update. Should have the most information
Packit-service layer that does koji scratch-build and testing-farm jobs on PRs. Also should have the most information
Testing-farm webhooks. Currently only reports when a whole test starts running and finished, all other parsing has to be done manually 🙁
tmt tests that run underneath everything. Can be extended with an arbitrary report plugin, but is only run at the end of a job (should probably avoid this if possible)

CC: @gwmngilfen

### Description of request We would like to ask some help in setting up the monitoring for Fedora-CI. Currently the other monitoring infrastructures that we have available is packit's and testing-farm's which are not publicly visible, and that is primarily why I think getting something worked together with Fedora-infra would be better. The main things that the monitoring should have are: - Be publicly visible - Catch when more than `n` tests are failing or are taking more time than normal - Report when something is going wrong on matrix or slack or anything else - Allow for other interesting parties to get relevant notifications, e.g. when many rpminspect tests fail have the rpminspect team CC-ed The current infrastructure in Fedora-CI: - Jenkins layer that executes testing-farm jobs on bodhi update. Should have the most information - Packit-service layer that does koji scratch-build and testing-farm jobs on PRs. Also should have the most information - Testing-farm webhooks. Currently only reports when a whole test starts running and finished, all other parsing has to be done manually 🙁 - tmt tests that run underneath everything. Can be extended with an arbitrary `report` plugin, but is only run at the end of a job (should probably avoid this if possible) CC: @gwmngilfen

james added the

labels

2026-04-23 19:06:35 +00:00

james commented

2026-04-23 19:07:11 +00:00

Member

Really not sure how easy this is to integrate into zabbix ... Doesn't seem like the normal things we monitor, so 5 might not be enough.

lecris commented

2026-04-27 10:07:58 +00:00

Author

Totally, ideas of how break down this would be good. Maybe we can start with the dashboard and hooking the jenkins jobs into it?

gwmngilfen commented

2026-04-30 13:21:30 +00:00

Member

Sorry for the lag, I lost track of this after @lecris asked me in chat.

I think this is generally achievable. What I know is:

Zabbix can notify arbitrary users in arbitrary ways. Today that looks like "all alerts of Average or above > Matrix" but it can be much more flexible. We can run some trials of email or Matrix-to-a-different room on staging whenever you like.
In terms of consuming data, we can use a variety of methinds. Here it sounds like a query to the Jenkins API might be the best way to go, but we also have other options.

@lecris we should sync up on this and toss some ideas around of a more concrete plan. I'll ping you in chat.

Sorry for the lag, I lost track of this after @lecris asked me in chat. I think this is generally achievable. What I know is: - Zabbix can notify arbitrary users in arbitrary ways. Today that looks like "all alerts of Average or above > Matrix" but it can be much more flexible. We can run some trials of email or Matrix-to-a-different room on staging whenever you like. - In terms of consuming data, we can use a variety of methinds. Here it *sounds* like a query to the Jenkins API might be the best way to go, but we also have other options. @lecris we should sync up on this and toss some ideas around of a more concrete plan. I'll ping you in chat.

👍 1

lecris commented

2026-04-30 15:01:28 +00:00

Author

Ok, so there are 2 parts of this, the metrics from Jenkins and the metrics from packit. The latter probably needs more fine-tuning of the metrics, will continue the discussion in https://github.com/packit/packit-service/issues/3117.

For the jenkins part, the metrics are available under the /prometheus/ path (probably best not to make the actual url easy to scrape 😉). The things that should be monitored in there right now are:

default_jenkins_builds_duration_milliseconds_summary_count
default_jenkins_builds_duration_milliseconds_summary_sum

Those are separated by parameters jenkins_job, status, DIST_GIT_BRANCH being the main ones. We can start with status="UNSTABLE" and status="FAILURE" for the jenkins_job="fedora-ci/rmdepcheck-pipeline/main".

Ok, so there are 2 parts of this, the metrics from Jenkins and the metrics from packit. The latter probably needs more fine-tuning of the metrics, will continue the discussion in https://github.com/packit/packit-service/issues/3117. For the jenkins part, the metrics are available under the `/prometheus/` path (probably best not to make the actual url easy to scrape 😉). The things that should be monitored in there right now are: - `default_jenkins_builds_duration_milliseconds_summary_count` - `default_jenkins_builds_duration_milliseconds_summary_sum` Those are separated by parameters `jenkins_job`, `status`, `DIST_GIT_BRANCH` being the main ones. We can start with `status="UNSTABLE"` and `status="FAILURE"` for the `jenkins_job="fedora-ci/rmdepcheck-pipeline/main"`.

gwmngilfen self-assigned this

2026-05-05 14:25:22 +00:00

gwmngilfen added this to the Sprint ( 2026-05-04 to 2026-05-18 ) project

2026-05-05 14:25:27 +00:00

kevin modified the project from Sprint ( 2026-05-04 to 2026-05-18 ) to Sprint ( 2026-05-18 to 2026-06-01)

2026-05-18 14:34:19 +00:00

gwmngilfen commented

2026-05-20 08:39:50 +00:00

Member

@lecris so the Jenkins part is ready to hook up then? I can maybe look at that next week, if so.

lecris commented

2026-05-20 09:00:21 +00:00

Author

Yes it should be ready for first iteration. Let me know if you need me during that.

gwmngilfen modified the project from Sprint ( 2026-05-18 to 2026-06-01) to Current Sprint ( 2026-06-01 to 2026-06-14)

2026-06-01 14:20:42 +00:00

rcallwoo modified the project from Current Sprint ( 2026-06-01 to 2026-06-14) to Sprint ( 2026-06-15 to 2026-06-28)

2026-06-15 14:24:17 +00:00

gwmngilfen commented

2026-06-22 11:58:24 +00:00

Member

I've started testing this out. We have a template (Fedora/CI/Jenkins by HTTP) that is pulling the Prometheus data and parsing it for the rmdepcheck/UNSTABLE combo on each branch. It has thrown up some questions:

We have no CI hosts in Fedora Zabbix, so we'll have to decide where this belongs. Right now it's attached to zabbix01 for testing.
There's a potentially large matrix of job/status/branch to look at. We may need to move to LLDiscovery rather than hand-coding it
It's not clear exactly what the metrics represent (what is a count of a job with "milliseconds" in the title?" nor when/if they reset their counters

Right now we've set up these test items (no triggers) and are letting it run for a few days to gather data to look at. Thanks to @lecris for the help!

I've started testing this out. We have a template (Fedora/CI/Jenkins by HTTP) that is pulling the Prometheus data and parsing it for the rmdepcheck/UNSTABLE combo on each branch. It has thrown up some questions: - We have no CI hosts in Fedora Zabbix, so we'll have to decide where this belongs. Right now it's attached to zabbix01 for testing. - There's a potentially large matrix of job/status/branch to look at. We may need to move to LLDiscovery rather than hand-coding it - It's not clear exactly what the metrics represent (what is a count of a job with "milliseconds" in the title?" nor when/if they reset their counters Right now we've set up these test items (no triggers) and are letting it run for a few days to gather data to look at. Thanks to @lecris for the help!

gwmngilfen modified the project from Sprint ( 2026-06-15 to 2026-06-28) to Sprint ( 2026-06-29 to 2026-07-12)

2026-06-29 14:33:47 +00:00

gwmngilfen modified the project from Sprint ( 2026-06-29 to 2026-07-12) to Sprint ( 2026-07-13 to 2026-07-26)

2026-07-13 14:55:08 +00:00

kevin removed this from the Sprint ( 2026-07-13 to 2026-07-26) project

2026-07-27 14:26:03 +00:00

Rows
Columns

Enabling some monitoring for Fedora-CI #13285

Description of request