infra/tickets
24
9
Fork 1
Code Issues 81 Projects Wiki Activity Actions

Filter out some Copr related Zabbix warnings #13450

New issue
Closed
opened 2026-07-02 13:17:31 +00:00 by frostyx · 4 comments
frostyx commented 2026-07-02 13:17:31 +00:00
Copy link

Description of request

@gwmngilfen configured a Matrix room for the Copr team where we receive Zabbix warnings. This is much appreciated. We have one problem which is that we receive too many notifications and the important ones get lost in the noise. So I went through all the unique messages that we got and categorized them as either relevant or irrelevant for our team.

We definitely want to get these:

❌ copr-ping check older than 50mins (3754929)
Severity: Average started at 2026.06.10 13:39:23 on copr-be.aws.fedoraproject.org

⚠ Copr: copr-be.aws.fedoraproject.org return code is not 200 (5 checks) (3755533)
Severity: Warning started at 2026.06.10 13:58:54 on copr-fe.aws.fedoraproject.org

⚠ Copr: copr.fedorainfracloud.org does not have correct text (3756438)
Severity: Warning started at 2026.06.10 14:18:52 on copr-be.aws.fedoraproject.org

⚠ Linux: FS [/var/lib/dist-git]: Space is low (used > 80%, total 8857.7GB) (3801957)
Severity: Warning started at 2026.06.12 08:41:49 on copr-dist-git.aws.fedoraproject.org

⚠ COPR CDN Check failed (3869307)
Severity: Warning started at 2026.06.15 09:58:24 on copr-fe.aws.fedoraproject.org

⚠ Apache: Service is down (4136727)
Severity: Warning started at 2026.06.28 07:01:56 on copr-fe.aws.fedoraproject.org

ℹ MD Raid array md2 is in recovery mode on vmhost-p09-copr03.rdu3.fedoraproject.org (3870258)
Severity: Information started at 2026.06.15 11:19:50 on vmhost-p09-copr03.rdu3.fedoraproject.org

🔥 MD Raid array md0 is degraded on vmhost-p09-copr03.rdu3.fedoraproject.org (3818561)
Severity: High started at 2026.06.13 02:13:23 on vmhost-p09-copr03.rdu3.fedoraproject.org

❌ Linux: High memory utilization (>90% for 5m) (3756263)
Severity: Average started at 2026.06.10 14:17:22 on copr-be.aws.fedoraproject.org

These are not helpful for us, and should probably go to #fedora-noc. I don't know how to fix either of these:

🔥 MGMT: Chassis unreachable by HTTP (3754791)
Severity: High started at 2026.06.10 13:35:59 on vmhost-x86-copr01.rdu3.fedoraproject.org

✅ Linux: Interface eth5: Link down (4d 20h 56m 54s)
(kevin) acknowledged, commented and closed problem at 2026.06.10 16:52:17
expected.

⚠ Linux: Number of installed packages has been changed (3931379)
Severity: Warning started at 2026.06.18 13:24:32 on vmhost-p09-copr03.rdu3.fedoraproject.org

ℹ Linux: /etc/passwd has been changed (3931380)
Severity: Information started at 2026.06.18 13:24:35 on vmhost-p09-copr03.rdu3.fedoraproject.org

ℹ Linux: Operating system description has changed (4087316)
Severity: Information started at 2026.06.25 23:01:54 on vmhost-x86-copr03.rdu3.fedoraproject.org

🔥 Linux: Active checks are not available (3930081)
Severity: High started at 2026.06.18 11:28:40 on vmhost-p09-copr03.rdu3.fedoraproject.org

❌ Linux: Zabbix agent is not available (or nodata for 30m) (3930308)
Severity: Average started at 2026.06.18 11:51:30 on vmhost-p09-copr03.rdu3.fedoraproject.org

🔥 Host DNS: Unavailable by ICMP ping (3930508)
Severity: High started at 2026.06.18 12:03:27 on vmhost-p09-copr03.rdu3.fedoraproject.org

The load and high CPU utilization ones, we don't know

⚠ Linux: Load average is too high (per CPU load over 6 for 5m) (3755230)
Severity: Warning started at 2026.06.10 13:46:34 on copr-be.aws.fedoraproject.org

⚠ Linux: High CPU utilization (over 90% for 5m) (3772263)
Severity: Warning started at 2026.06.11 04:44:29 on copr-dist-git.aws.fedoraproject.org

Is there some severity above warning? Like critical? If yes, we would like to only get that one. If not, please continue sending the warnings.

### Description of request @gwmngilfen configured a Matrix room for the Copr team where we receive Zabbix warnings. This is much appreciated. We have one problem which is that we receive too many notifications and the important ones get lost in the noise. So I went through all the unique messages that we got and categorized them as either relevant or irrelevant for our team. We definitely want to get these: ``` ❌ copr-ping check older than 50mins (3754929) Severity: Average started at 2026.06.10 13:39:23 on copr-be.aws.fedoraproject.org ⚠ Copr: copr-be.aws.fedoraproject.org return code is not 200 (5 checks) (3755533) Severity: Warning started at 2026.06.10 13:58:54 on copr-fe.aws.fedoraproject.org ⚠ Copr: copr.fedorainfracloud.org does not have correct text (3756438) Severity: Warning started at 2026.06.10 14:18:52 on copr-be.aws.fedoraproject.org ⚠ Linux: FS [/var/lib/dist-git]: Space is low (used > 80%, total 8857.7GB) (3801957) Severity: Warning started at 2026.06.12 08:41:49 on copr-dist-git.aws.fedoraproject.org ⚠ COPR CDN Check failed (3869307) Severity: Warning started at 2026.06.15 09:58:24 on copr-fe.aws.fedoraproject.org ⚠ Apache: Service is down (4136727) Severity: Warning started at 2026.06.28 07:01:56 on copr-fe.aws.fedoraproject.org ℹ MD Raid array md2 is in recovery mode on vmhost-p09-copr03.rdu3.fedoraproject.org (3870258) Severity: Information started at 2026.06.15 11:19:50 on vmhost-p09-copr03.rdu3.fedoraproject.org 🔥 MD Raid array md0 is degraded on vmhost-p09-copr03.rdu3.fedoraproject.org (3818561) Severity: High started at 2026.06.13 02:13:23 on vmhost-p09-copr03.rdu3.fedoraproject.org ❌ Linux: High memory utilization (>90% for 5m) (3756263) Severity: Average started at 2026.06.10 14:17:22 on copr-be.aws.fedoraproject.org ``` These are not helpful for us, and should probably go to #fedora-noc. I don't know how to fix either of these: ``` 🔥 MGMT: Chassis unreachable by HTTP (3754791) Severity: High started at 2026.06.10 13:35:59 on vmhost-x86-copr01.rdu3.fedoraproject.org ✅ Linux: Interface eth5: Link down (4d 20h 56m 54s) (kevin) acknowledged, commented and closed problem at 2026.06.10 16:52:17 expected. ⚠ Linux: Number of installed packages has been changed (3931379) Severity: Warning started at 2026.06.18 13:24:32 on vmhost-p09-copr03.rdu3.fedoraproject.org ℹ Linux: /etc/passwd has been changed (3931380) Severity: Information started at 2026.06.18 13:24:35 on vmhost-p09-copr03.rdu3.fedoraproject.org ℹ Linux: Operating system description has changed (4087316) Severity: Information started at 2026.06.25 23:01:54 on vmhost-x86-copr03.rdu3.fedoraproject.org 🔥 Linux: Active checks are not available (3930081) Severity: High started at 2026.06.18 11:28:40 on vmhost-p09-copr03.rdu3.fedoraproject.org ❌ Linux: Zabbix agent is not available (or nodata for 30m) (3930308) Severity: Average started at 2026.06.18 11:51:30 on vmhost-p09-copr03.rdu3.fedoraproject.org 🔥 Host DNS: Unavailable by ICMP ping (3930508) Severity: High started at 2026.06.18 12:03:27 on vmhost-p09-copr03.rdu3.fedoraproject.org ``` The load and high CPU utilization ones, we don't know ``` ⚠ Linux: Load average is too high (per CPU load over 6 for 5m) (3755230) Severity: Warning started at 2026.06.10 13:46:34 on copr-be.aws.fedoraproject.org ⚠ Linux: High CPU utilization (over 90% for 5m) (3772263) Severity: Warning started at 2026.06.11 04:44:29 on copr-dist-git.aws.fedoraproject.org ``` Is there some severity above warning? Like critical? If yes, we would like to only get that one. If not, please continue sending the warnings.
gwmngilfen commented 2026-07-02 13:41:04 +00:00
Member
Copy link

Thanks for this. In #noc we use a filter of "above Warning" which still leaves us "Average", "High", and "Disaster" levels, while "Warning" and "Informational" stay in the Zabbix UI. So for COPR checks like the CDN, copr-ping, etc, we can simply set the status to "Average" or higher, and apply the same filter.

The issue is with the generic checks like MD raid, Apache, etc. These come from more generic templates which apply to many hosts, so simply raising the alert level isn't what we want to do. We can match on specific trigger names, but that might mean you miss out on other things you care about later.

My gut feeling is that we need some negative-tagging like "not-copr" or such that we can add to infra-wide templates, but I'm not super happy with that. I'll think on it.

Thanks for this. In #noc we use a filter of "above Warning" which still leaves us "Average", "High", and "Disaster" levels, while "Warning" and "Informational" stay in the Zabbix UI. So for COPR checks like the CDN, copr-ping, etc, we can simply set the status to "Average" or higher, and apply the same filter. The issue is with the generic checks like MD raid, Apache, etc. These come from more generic templates which apply to many hosts, so simply raising the alert level isn't what we want to do. We can match on *specific* trigger names, but that might mean you miss out on other things you care about later. My gut feeling is that we need some negative-tagging like "not-copr" or such that we can add to infra-wide templates, but I'm not super happy with that. I'll think on it.
❤️ 1
frostyx commented 2026-07-02 14:11:20 +00:00
Author
Copy link

So for COPR checks like the CDN, copr-ping, etc, we can simply set the status to "Average" or higher, and apply the same filter.

Sounds good to me

> So for COPR checks like the CDN, copr-ping, etc, we can simply set the status to "Average" or higher, and apply the same filter. Sounds good to me
gwmngilfen commented 2026-07-06 14:27:38 +00:00
Member
Copy link

So I don't think a simple severity-level check works, you have some items you want that are low-prio, and some you don't want that are high-prio.

Here's what I've set up on prod just now, which I think covers what you need:

image

Specifically this is either a check marked as copr-team, or one of the named checks above. I used a match on tags, so other problems with those services will be reported, rather than using explicit trigger names.

Let's see how that goes - I haven't made the change yet on staging, so we can perhaps compare.

So I don't think a simple severity-level check works, you have some items you want that are low-prio, and some you don't want that are high-prio. Here's what I've set up on prod just now, which I think covers what you need: ![image](/attachments/d4b5531c-bb7e-41b5-9315-769970e1980a) Specifically this is *either* a check marked as copr-team, *or* one of the named checks above. I used a match on tags, so other problems with those services will be reported, rather than using explicit trigger names. Let's see how that goes - I haven't made the change yet on staging, so we can perhaps compare.
image.png
54 KiB
gwmngilfen commented 2026-07-07 15:16:36 +00:00
Member
Copy link

I've set up the above on both prod and stg, so I'll close this for now. We can always tweak it more.

I've set up the above on both prod and stg, so I'll close this for now. We can always tweak it more.
❤️ 1
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
announcement

Archived

  
anubis
Issues related to anubis

  
authentication
Authentication related issues

  
aws
Issues related to Amazon Web Services

  
backlog
Tasks that will take longer then ~4 hours to do

Archived

  
blocked
Issue is blocked

Archived

  
bodhi
Issues with Bodhi

  
ci
Continuous Integration

  
cloud
Items in FedoraInfraCloud or COmmunishift

  
communishift
issues/questions about os.edorainfracloud.org ( Community OpenShift)

  
copr
COPR system (run by other team)

  
database
Issues related to databases

  
day-to-day
These are small self contained tasks that are part of normal operations, but are async from sprint planning.

  
dc-move
Items related to datacenter move

Archived

  
deprecated
Tools which Infrastructure may not be able to fix

Archived

  
dev
Tasks requiring some dev work

  
discourse
outsourced tool

  
dns
Changes or problems with DNS

  
downloads
rsync,dl.fp.o,AWS,torrents

  
easyfix

Archived

  
epel
Issues relating to epel

Archived

  
firmitas

   
forgejo_migration
Request migration to the new forge

  
Gain
High
 task that gets us a lot / is important / makes many people happy

Archived

  
Gain
Low
 tasks that gets us little / isn't very important / makes only 1 person or few happy

Archived

  
Gain
Medium
 task that gets us some / is somewhat important / makes some people happy

Archived

  
gitlab

   
greenwave
Issues with greenwave

  
hardware
Problem is with Fedora Infrastructure Owned/Leased Hardware

  
help wanted

   
high-trouble
task that is complex/long/difficult

Archived

  
koji
Issues with the build system

  
koschei
Items needing Koschei team

  
lists
issues with lists

  
low-trouble
task that is easy/short

Archived

  
medium-trouble
task that is medium complex / difficult

Archived

  
mirrorlists
Way to tell systems where downloads are

  
monitoring

   
Needs investigation

   
odcs
On Demand Compose Service issues

Archived

  
OpenShift
Issues with OpenShift

  
ops
ops problem now...

Archived

  
outage

   
packager_workflow_blocker
Blocks the packager workflow

Archived

  
pagure
Issues with pagure.io

  
permissions
Items which need permissions set in outside tools

  
Priority
Needs Review
 Priority of Needs Review

Archived

  
Priority
Next Meeting
 Priority of Next Meeting

Archived

  
Priority
🔥 URGENT 🔥
 Priority of 🔥 URGENT 🔥

Archived

  
Priority
Waiting on Assignee
 Priority of Waiting on Assignee

Archived

  
Priority
Waiting on External
 Priority of Waiting on External

Archived

  
Priority
Waiting on Reporter
 Priority of Waiting on Reporter

Archived

  
rabbitmq
issues with the message broker

  
release-monitoring
release-monitoring.org (Anitya)

  
releng
For problems which should be in the releng ticket system

Archived

  
request-for-resources

   
s390x
The secondary architecture hardware and problems related with it

  
security

   
SMTP
Email issues outside of lists (DMARC, SPF, aliases)

  
sprint-0

Archived

  
sprint-1

Archived

  
src.fp.o
Issues related to src.fedoraproject.org

  
staging
Issues in staging

  
unfreeze
Patches to be applied after freeze is lifted

  
waiverdb
Issues with waiverdb

  
websites-general
General items with a website focus

  
wiki
Issues with the Wiki

  
Backlog Status
Needs Review
 Work items that have been created or updated but need review, refinement, or approval before they can be considered ready for sprint planning.

  
Backlog Status
Ready
 Work items that have been reviewed, refined, and are ready to be committed to a sprint. These items have clear requirements and can be worked on immediately when capacity is available.

  
chore
Work that needs to be done but doesn't add new functionality or fix bugs. Maintenance tasks that keep the system running smoothly and the codebase clean.

  
documentation
Work focused on creating, updating, or improving documentation for users, developers, or system administrators.

  
points
01
 A task that is simple, small, and has very little uncertainty. It should be easy enough for any team member to pick up and complete quickly, without a lot of discussion or research.

  
points
02
 This is a task that is slightly more complex or larger than a "1." It might involve two or three simple steps, but the outcome is still very clear and there's little uncertainty.

  
points
03
 Medium-sized task with some noticeable complexity. It might require a bit more thought or collaboration. The steps are well-defined, but it's not a trivial change.

  
points
05
 A significant piece of work. It has more complexity and a higher degree of uncertainty than smaller tasks. It might require some research, a design discussion, or collaboration across multiple teams.

  
points
08
 Complex task with a lot of uncertainty. It's often the largest size that a team will consider for a single sprint. It likely requires a detailed breakdown and multiple team members to complete.

  
points
13
 Any task that feels like it's a "13" or more is considered an Epic. These are too large and uncertain to be worked on in a single sprint and should be broken down.

  
Priority
High
 This is for work that is critical and needs to be addressed in the next sprint. It's often for urgent bugs, important features, or a major piece of technical debt that is blocking other work.

  
Priority
Low
 This is for work that is good to do but not urgent. It's a candidate for being worked on when there's extra time or when it rises in importance. It can stay in the backlog for a longer period.

  
Priority
Medium
 This is for important work that should be done soon, but isn't as urgent as a high-priority item. It will likely be considered for the next 1-2 sprints.

  
Sprint Status
Blocked
 Work items that cannot progress due to external dependencies, blockers, or issues that need resolution.

  
Sprint Status
Done
 Work items that have been completed, reviewed, and integrated. No further work is needed within the current sprint.

  
Sprint Status
In Progress
 Work items currently being worked on by a team member. Active development, research, or implementation is happening.

  
Sprint Status
Review
 Work items that are complete but need review, testing, or approval before being marked as done.

  
Sprint Status
To Do
 Work items that are committed to the sprint but haven't been started yet. These items are ready to be picked up by team members.

  
Technical Debt
Work that addresses shortcuts, workarounds, or suboptimal implementations that were done previously to meet deadlines. These items improve code quality, maintainability, or performance without adding new features.

  
Work Item
Bug
 Issues where the system is not working as intended or designed. Bugs represent deviations from expected behavior.

  
Work Item
Epic
 Large initiatives that are broken down into multiple smaller work items (user stories, tasks, bugs). Epics represent significant features or projects that span multiple sprints.

  
Work Item
Spike
 Time-boxed research, investigation, or proof-of-concept work to reduce uncertainty or explore solutions. Spikes help inform future development decisions.

  
Work Item
Task
 Used to break down larger user stories into smaller, manageable implementation pieces. They focus on the specific technical steps needed to deliver the user story's value.

  
Work Item
User Story
 User-facing features or functionality that can be framed from a user perspective, describing what someone wants to accomplish and why.

No labels
announcement
anubis
authentication
aws
backlog
blocked
bodhi
ci
cloud
communishift
copr
database
day-to-day
dc-move
deprecated
dev
discourse
dns
downloads
easyfix
epel
firmitas
forgejo_migration
Gain
High
Gain
Low
Gain
Medium
gitlab
greenwave
hardware
help wanted
high-trouble
koji
koschei
lists
low-trouble
medium-trouble
mirrorlists
monitoring
Needs investigation
odcs
OpenShift
ops
outage
packager_workflow_blocker
pagure
permissions
Priority
Needs Review
Priority
Next Meeting
Priority
🔥 URGENT 🔥
Priority
Waiting on Assignee
Priority
Waiting on External
Priority
Waiting on Reporter
rabbitmq
release-monitoring
releng
request-for-resources
s390x
security
SMTP
sprint-0
sprint-1
src.fp.o
staging
unfreeze
waiverdb
websites-general
wiki
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Sprint ( 2026-06-29 to 2026-07-12)
Assignees
Clear assignees
No assignees
gwmngilfen
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
infra/tickets#13450
No description provided.