Monitoring in Fedora-CI #567

New issue

Open

opened 2026-06-24 11:50:05 +00:00 by lecris · 0 comments

lecris commented

2026-06-24 11:50:05 +00:00

Owner

Goal

Be able to get notified of failing tests, CC relevant test owners, detect test duration drifts.

Separations

The monitoring should be monitored separately by:

trigger: bodhi update or PR
test job: scratch-build, rmdepcheck, installability
dist-git branch: rawhide, epel10
status: error, fail, pass
subcheck: rpminspect/license

Desired trackers

Some of the things that should trigger a notification:

a test failed on a stable test, e.g. rmdepcheck
multiple tests in a row are failing
jobs are taking unusually long time to run
a test failed with error instead of fail
a gating subcheck failed

Graphs would also be useful to visualize the state:

job duration distribution
trend of job states

Dashboard functionality is also important to be able to:

see chronologically the job states
be able to aggregate and filter by state, branch, etc.
have a list of affected updates or builds
allow users to filter for their own jobs

How can we handle this?

After some experimentation and looking at the states of Jenkins and packit

Monitoring on jenkins is not feasible

The prometheus approach seems close, but it only shows the rolling statistics of the job state count. Other metrics are not useful.

The direct Jenkins query is slightly more useful, but it suffers from 403 issue lockouts. It is also hard to figure out the appropriate query in order to get useful notifications.

Monitoring on packit would be costly

Packit dashboard is a no-go since all of the processing is done on the client side.

Packit-service might be a good home, but it would make it compete with the various jobs it is trying to do. The data is very local though.

Some stretch goals here seem beyond packit-service's use like authentication, filtering. Packit service might benefit from delegating those tasks to a standalone service.

Maybe a standalone app?

The issues with the monitoring here is mostly that we need to aggregate the state of the jobs and reexport or push it in a form that zabbix (or dashboard frontends) can consume. Having a dedicated app that does this would be the most flexible. The process would be

listen for a fedora messaging bus to get updates on new job states
process the job, squeezing the data and save it in a running queue
update zabbix with new statistics of the job
expose specific metric endpoints that are relevant to trigger or visualization
instruct zabbix on what are the significant metrics so it can construct subitems

## Goal Be able to get notified of failing tests, CC relevant test owners, detect test duration drifts. ## Separations The monitoring should be monitored separately by: - trigger: bodhi update or PR - test job: scratch-build, rmdepcheck, installability - dist-git branch: rawhide, epel10 - status: error, fail, pass - subcheck: rpminspect/license ## Desired trackers Some of the things that should trigger a notification: - a test failed on a stable test, e.g. rmdepcheck - multiple tests in a row are failing - jobs are taking unusually long time to run - a test failed with error instead of fail - a gating subcheck failed Graphs would also be useful to visualize the state: - job duration distribution - trend of job states Dashboard functionality is also important to be able to: - see chronologically the job states - be able to aggregate and filter by state, branch, etc. - have a list of affected updates or builds - allow users to filter for their own jobs ## How can we handle this? After some experimentation and looking at the states of Jenkins and packit ### Monitoring on jenkins is not feasible The prometheus approach seems close, but it only shows the rolling statistics of the job state count. Other metrics are not useful. The direct Jenkins query is slightly more useful, but it suffers from 403 issue lockouts. It is also hard to figure out the appropriate query in order to get useful notifications. ### Monitoring on packit would be costly Packit dashboard is a no-go since all of the processing is done on the client side. Packit-service might be a good home, but it would make it compete with the various jobs it is trying to do. The data is very local though. Some stretch goals here seem beyond packit-service's use like authentication, filtering. Packit service might benefit from delegating those tasks to a standalone service. ### Maybe a standalone app? The issues with the monitoring here is mostly that we need to aggregate the state of the jobs and reexport or push it in a form that zabbix (or dashboard frontends) can consume. Having a dedicated app that does this would be the most flexible. The process would be - listen for a fedora messaging bus to get updates on new job states - process the job, squeezing the data and save it in a running queue - update zabbix with new statistics of the job - expose specific metric endpoints that are relevant to trigger or visualization - instruct zabbix on what are the significant metrics so it can construct subitems