infra/tickets

Fork 1

Zabbix monitoring for Copr pools of builders #13389

New issue

Closed

opened 2026-06-02 09:51:42 +00:00 by frostyx · 5 comments

frostyx commented

2026-06-02 09:51:42 +00:00

Description of request

It was already mentioned in ci/tickets#559, but it was off-topic there. My long-term goal was to have a monitoring of Copr pools of builders, and we are really close to finishing it. @gwmngilfen said Copr monitoring is on his radar too, so I wanted to ask, if you could take look at the Zabbix configuration, please? It'd help a lot.

I created this API endpoint
https://download.copr-dev.fedorainfracloud.org/resalloc/api/stats

There is no OpenAPI documentation for it, but it's quite self-explanatory, e.g.:

"aws_x86_64_normal_prod": {
  "last_attempt_to_start": 1780391418.6749854,
  "last_successful_start": 1780391418.6749854,
  "startup_success_rate": 0.999145507725002,
  "startup_time_avg": 101.43275752654108
},

The aws_x86_64_normal_prod is a pool name, don't try to parse it, or anything like that
If last_successful_start is too long ago, it doesn't mean there is a problem. But if it is too long ago while at the same time last_attempt_to_start is seconds/minutes ago, there definitely is a problem
Sometimes spawning a VM fails, we just discard it and try again. So startup_success_rate doesn't have to be 100% ... but if it goes below 60%, somebody should really take a look
The startup_time_avg is a bit tricky. It takes vastly different times to start builders in different pools. For aws_aarch64_spot_dev it is 79.5 seconds, for copr_ic_s390x_br_sao_2_dev it is 340.1 seconds. Would it be possible to have a default warning value at 200 seconds with the possibility to override it for specific pools?
There are some pools that have null values, I'd simply ignore those
If the endpoint itself is not available, we definitely need a notification

I'd be much appreciated if you could take a look, but if you are not interested in working on this, please let me know, I'd try to implement it myself.

### Description of request It was already mentioned in https://forge.fedoraproject.org/ci/tickets/issues/559, but it was off-topic there. My long-term goal was to have a monitoring of Copr pools of builders, and we are really close to finishing it. @gwmngilfen said Copr monitoring is on his radar too, so I wanted to ask, if you could take look at the Zabbix configuration, please? It'd help a lot. I created this API endpoint https://download.copr-dev.fedorainfracloud.org/resalloc/api/stats There is no OpenAPI documentation for it, but it's quite self-explanatory, e.g.: ```json "aws_x86_64_normal_prod": { "last_attempt_to_start": 1780391418.6749854, "last_successful_start": 1780391418.6749854, "startup_success_rate": 0.999145507725002, "startup_time_avg": 101.43275752654108 }, ``` - The `aws_x86_64_normal_prod` is a pool name, don't try to parse it, or anything like that - If `last_successful_start` is too long ago, it doesn't mean there is a problem. But if it is too long ago while at the same time `last_attempt_to_start` is seconds/minutes ago, there definitely is a problem - Sometimes spawning a VM fails, we just discard it and try again. So `startup_success_rate` doesn't have to be 100% ... but if it goes below 60%, somebody should really take a look - The `startup_time_avg` is a bit tricky. It takes vastly different times to start builders in different pools. For `aws_aarch64_spot_dev` it is 79.5 seconds, for `copr_ic_s390x_br_sao_2_dev` it is `340.1` seconds. Would it be possible to have a default warning value at 200 seconds with the possibility to override it for specific pools? - There are some pools that have `null` values, I'd simply ignore those - If the endpoint itself is not available, we definitely need a notification I'd be much appreciated if you could take a look, but if you are not interested in working on this, please let me know, I'd try to implement it myself.

frostyx referenced this issue from ci/tickets

2026-06-02 09:54:29 +00:00

Monitor failed copr builds #559

gwmngilfen commented

2026-06-02 10:59:59 +00:00

Member

This is great, and I can help - probably next week though. Thanks for working on it!

So, a few questions/thoughts:

Zabbix can use a discovery-based system, I assume we'd like to use that here? That way we can curl the URL, and use the non-null pools as discovered things to monitor?
Getting the data from each item is clearly fine and we should do that
Comparing two items (attempt vs success) is fine for triggers
Overriding specific pools is possible, and can be done from Ansible, no probs there
Notification for the endpoint is fine too

I presume most, if not all, of these would notify to #buildsys:fedoraproject.org ?

This is great, and I can help - probably next week though. Thanks for working on it! So, a few questions/thoughts: - Zabbix can use a discovery-based system, I assume we'd like to use that here? That way we can curl the URL, and use the non-null pools as discovered things to monitor? - Getting the data from each item is clearly fine and we should do that - Comparing two items (attempt vs success) is fine for triggers - Overriding specific pools is possible, and can be done from Ansible, no probs there - Notification for the endpoint is fine too I presume most, if not all, of these would notify to #buildsys:fedoraproject.org ?

frostyx commented

2026-06-02 11:44:27 +00:00

Author

This is great, and I can help - probably next week though. Thanks for working on it!

Next week is fine, thank you very much

Zabbix can use a discovery-based system, I assume we'd like to use that here? That way we can curl the URL, and use the non-null pools as discovered things to monitor?

I am a total noob when it comes to Zabbix, so if you don't mind, I'd leave that up to you, whatever solution you think is the best. For sure curling the URL has to be some part of the process, but if it is "dynamic discovery" or if we simply have one check for copr-backend which fetches the URL and then loops over all the pools, does all the checks and raises some exception or something if it finds a problem ...

I presume most, if not all, of these would notify to #buildsys:fedoraproject.org ?

Yep, that would be the best.

> This is great, and I can help - probably next week though. Thanks for working on it! Next week is fine, thank you very much > Zabbix can use a discovery-based system, I assume we'd like to use that here? That way we can curl the URL, and use the non-null pools as discovered things to monitor? I am a total noob when it comes to Zabbix, so if you don't mind, I'd leave that up to you, whatever solution you think is the best. For sure curling the URL has to be some part of the process, but if it is "dynamic discovery" or if we simply have one check for copr-backend which fetches the URL and then loops over all the pools, does all the checks and raises some exception or something if it finds a problem ... > I presume most, if not all, of these would notify to #buildsys:fedoraproject.org ? Yep, that would be the best.

👍 1

gwmngilfen added the

points

monitoring

copr

labels

2026-06-02 11:58:12 +00:00

gwmngilfen self-assigned this

2026-06-02 11:58:22 +00:00

gwmngilfen added this to the Sprint ( 2026-06-15 to 2026-06-28) project

2026-06-15 14:35:15 +00:00

gwmngilfen modified the project from Sprint ( 2026-06-15 to 2026-06-28) to Sprint ( 2026-06-29 to 2026-07-12)

2026-06-29 14:34:25 +00:00

gwmngilfen commented

2026-07-07 15:37:59 +00:00

Member

@frostyx this is all set up on STG, and we have alerts in the channel, so seems to be working. For each pool we have:

high-level trigger if a pool is not parseable
(last_attempted - last_success) > 3600
success_rate < 60
avg_startup > 200 (configurable per pool)

Lets give that a day or two and then copy to prod...

@frostyx this is all set up on STG, and we have alerts in the channel, so seems to be working. For each pool we have: - high-level trigger if a pool is not parseable - (last_attempted - last_success) > 3600 - success_rate < 60 - avg_startup > 200 (configurable per pool) Lets give that a day or two and then copy to prod...

frostyx commented

2026-07-09 05:02:12 +00:00

Author

Thank you very much @gwmngilfen,
this is going to be such a big improvement for the overall Copr stability/availability.

Thank you very much @gwmngilfen, this is going to be such a big improvement for the overall Copr stability/availability.

gwmngilfen commented

2026-07-09 13:25:08 +00:00

Member

This has been rolled out to prod now, with some further tweaks. We've also learned a bit about how to to RBAC and dashboard access for other teams, which is valuable ;)

THanks for your patience @frostyx, ping me if you need more monitoring or help ;)

This has been rolled out to prod now, with some further tweaks. We've also learned a bit about how to to RBAC and dashboard access for other teams, which is valuable ;) THanks for your patience @frostyx, ping me if you need more monitoring or help ;)

❤️ 1

gwmngilfen closed this issue

2026-07-09 13:25:09 +00:00