infra/tickets
24
9
Fork 1
Code Issues 81 Projects Wiki Activity Actions

/etc/cron.daily/sync-http-logs-and-merge.sh on log01 errors #12833

New issue
Closed
opened 2025-10-06 19:24:57 +00:00 by kevin · 12 comments
kevin commented 2025-10-06 19:24:57 +00:00
Owner
Copy link

This daily cron job is outputting errors:

/usr/local/bin/sync-http-logs.py:244: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full
details.
  config = yaml.load(config_file)
rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]
rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]
rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]
rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]
rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]
rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]
rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]
rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]
rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]
/usr/local/bin/combineHttpLogs.sh: line 27: simple_message_to_bus: command not found
/usr/local/bin/combineHttpLogs.sh: line 89: simple_message_to_bus: command not found
/usr/local/bin/combineHttpLogs.sh: line 91: simple_message_to_bus: command not found
/usr/local/bin/combineHttpLogs.sh: line 89: simple_message_to_bus: command not found
/usr/local/bin/combineHttpLogs.sh: line 91: simple_message_to_bus: command not found
...
/usr/local/bin/combineHttpLogs.sh: line 91: simple_message_to_bus: command not found
/usr/local/bin/combineHttpLogs.sh: line 89: simple_message_to_bus: command not found
/usr/local/bin/combineHttpLogs.sh: line 91: simple_message_to_bus: command not found
/usr/local/bin/combineHttpLogs.sh: line 89: simple_message_to_bus: command not found
/usr/local/bin/combineHttpLogs.sh: line 91: simple_message_to_bus: command not found
/usr/local/bin/combineHttpLogs.sh: line 89: simple_message_to_bus: command not found
/usr/local/bin/combineHttpLogs.sh: line 91: simple_message_to_bus: command not found
/usr/local/bin/combineHttpLogs.sh: line 89: simple_message_to_bus: command not found
/usr/local/bin/combineHttpLogs.sh: line 91: simple_message_to_bus: command not found
/usr/local/bin/combineHttpLogs.sh: line 89: simple_message_to_bus: command not found
/usr/local/bin/combineHttpLogs.sh: line 91: simple_message_to_bus: command not found
/usr/local/bin/combineHttpLogs.sh: line 89: simple_message_to_bus: command not found
/usr/local/bin/combineHttpLogs.sh: line 91: simple_message_to_bus: command not found
/usr/local/bin/combineHttpLogs.sh: line 100: simple_message_to_bus: command not found
/usr/local/bin/combineHttpLogs.sh: line 102: simple_message_to_bus: command not found
/usr/local/bin/combineHttpLogs.sh: line 113: simple_message_to_bus: command not found
/usr/local/bin/combineHttpLogs.sh: line 115: simple_message_to_bus: command not found
/usr/local/bin/combineHttpLogs.sh: line 132: simple_message_to_bus: command not found

The kojipkgs errors are likely because they don't have the rsyncd role setup to allow collecting from them.
No idea on the other one. ;)

This daily cron job is outputting errors: ``` /usr/local/bin/sync-http-logs.py:244: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. config = yaml.load(config_file) rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] /usr/local/bin/combineHttpLogs.sh: line 27: simple_message_to_bus: command not found /usr/local/bin/combineHttpLogs.sh: line 89: simple_message_to_bus: command not found /usr/local/bin/combineHttpLogs.sh: line 91: simple_message_to_bus: command not found /usr/local/bin/combineHttpLogs.sh: line 89: simple_message_to_bus: command not found /usr/local/bin/combineHttpLogs.sh: line 91: simple_message_to_bus: command not found ... /usr/local/bin/combineHttpLogs.sh: line 91: simple_message_to_bus: command not found /usr/local/bin/combineHttpLogs.sh: line 89: simple_message_to_bus: command not found /usr/local/bin/combineHttpLogs.sh: line 91: simple_message_to_bus: command not found /usr/local/bin/combineHttpLogs.sh: line 89: simple_message_to_bus: command not found /usr/local/bin/combineHttpLogs.sh: line 91: simple_message_to_bus: command not found /usr/local/bin/combineHttpLogs.sh: line 89: simple_message_to_bus: command not found /usr/local/bin/combineHttpLogs.sh: line 91: simple_message_to_bus: command not found /usr/local/bin/combineHttpLogs.sh: line 89: simple_message_to_bus: command not found /usr/local/bin/combineHttpLogs.sh: line 91: simple_message_to_bus: command not found /usr/local/bin/combineHttpLogs.sh: line 89: simple_message_to_bus: command not found /usr/local/bin/combineHttpLogs.sh: line 91: simple_message_to_bus: command not found /usr/local/bin/combineHttpLogs.sh: line 89: simple_message_to_bus: command not found /usr/local/bin/combineHttpLogs.sh: line 91: simple_message_to_bus: command not found /usr/local/bin/combineHttpLogs.sh: line 100: simple_message_to_bus: command not found /usr/local/bin/combineHttpLogs.sh: line 102: simple_message_to_bus: command not found /usr/local/bin/combineHttpLogs.sh: line 113: simple_message_to_bus: command not found /usr/local/bin/combineHttpLogs.sh: line 115: simple_message_to_bus: command not found /usr/local/bin/combineHttpLogs.sh: line 132: simple_message_to_bus: command not found ``` The kojipkgs errors are likely because they don't have the rsyncd role setup to allow collecting from them. No idea on the other one. ;)
james commented 2025-10-06 19:27:54 +00:00
Member
Copy link

Metadata Update from @james:

  • Issue assigned to james
**Metadata Update from @james**: - Issue assigned to james
james commented 2025-10-06 19:28:25 +00:00
Member
Copy link

simple_message_to_bus is a path problem.

simple_message_to_bus is a path problem.
james commented 2025-10-06 19:28:26 +00:00
Member
Copy link

Metadata Update from @james:

  • Issue priority set to: Waiting on Assignee (was: Needs Review)
  • Issue tagged with: low-gain, low-trouble
**Metadata Update from @james**: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: low-gain, low-trouble
james commented 2025-10-22 16:44:07 +00:00
Member
Copy link

Issue tagged with: sprint-0

Issue tagged with: sprint-0
kevin commented 2025-10-22 21:52:19 +00:00
Author
Owner
Copy link

FYI, the current logs have:

/usr/local/bin/condense-mirrorlogs.sh: line 31: simple_message_to_bus: command not found
/usr/local/bin/condense-mirrorlogs.sh: line 78: simple_message_to_bus: command not found
/usr/local/bin/condense-mirrorlogs.sh: line 80: simple_message_to_bus: command not found
/usr/local/bin/condense-mirrorlogs.sh: line 89: simple_message_to_bus: command not found
/usr/local/bin/condense-mirrorlogs.sh: line 91: simple_message_to_bus: command not found

plot ["2007-05-17":"2024-12-31"]      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:2  title 'epel4' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv'
using 1:3  title 'epel5' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:4  title 'epel6' with lines lw 3,
'/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:5  title 'epel7' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:72  title 'epel8' with lines
lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:73  title 'epel9' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:6  title 'fed03'
with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:7  title 'fed04' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:8  title
'fed05' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirro
... (bunch more of these).
^
"/usr/local/share/web-data-analysis/mirrors-data.gp" line 107: all points y value undefined!

/usr/local/bin/condense-mirrorlogs.sh: line 119: simple_message_to_bus: command not found
FYI, the current logs have: ``` /usr/local/bin/condense-mirrorlogs.sh: line 31: simple_message_to_bus: command not found /usr/local/bin/condense-mirrorlogs.sh: line 78: simple_message_to_bus: command not found /usr/local/bin/condense-mirrorlogs.sh: line 80: simple_message_to_bus: command not found /usr/local/bin/condense-mirrorlogs.sh: line 89: simple_message_to_bus: command not found /usr/local/bin/condense-mirrorlogs.sh: line 91: simple_message_to_bus: command not found plot ["2007-05-17":"2024-12-31"] '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:2 title 'epel4' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:3 title 'epel5' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:4 title 'epel6' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:5 title 'epel7' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:72 title 'epel8' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:73 title 'epel9' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:6 title 'fed03' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:7 title 'fed04' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:8 title 'fed05' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirro ... (bunch more of these). ^ "/usr/local/share/web-data-analysis/mirrors-data.gp" line 107: all points y value undefined! /usr/local/bin/condense-mirrorlogs.sh: line 119: simple_message_to_bus: command not found ```
zlopez commented 2026-01-16 14:03:30 +00:00
Owner
Copy link

I wonder how the cron job itself is executed. As I tried run-parts /etc/cron.daily and didn't saw the error. The PATH is correctly configured in /etc/crontab

[root@log01 ~][PROD-RDU3]# run-parts /etc/cron.daily/                                                                                                                                                                                       
/etc/cron.daily/sync-http-logs-and-merge.sh:                                                                                                                                                                                                
                                                                                                                      
/usr/local/bin/sync-http-logs.py:244: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.                                     
  config = yaml.load(config_file)                                                                                     
rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                              
rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110)                                                                                                                             
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]
rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110)                                                                                                                            
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110)                                                                                                                        
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]
rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110)       
rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]    
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110)                                                                                                                            
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110)                                                                                                                        
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]     
rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110)                                                                                                                              
rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                             
rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]
rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                    
rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110)
rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110)                                                                                                                            
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110)                                                                                                                        
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110)                                                                                                                             
rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110)      
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                     
rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110)  
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                     
rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110)       
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                  
rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110)      
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                     
rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110)                                                                                                                        
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                     
rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110)        
rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110)      
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110)  
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                             
rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110)
rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110)                                                                                                                              
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110)                                                                                                                            
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                     
rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110)
rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110)                                                                                                                            
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                
rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                               
rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                   
rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) 
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                              
rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110)
rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110)                                                                                                                            
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110)                                                                                                                        
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110)
rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                     
rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110)      
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                     
rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110)  
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                     
rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110)                                                                                                                             
rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                  
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                     
rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110)      
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]                                                                                                                                                           
rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110)  
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5]
I wonder how the cron job itself is executed. As I tried `run-parts /etc/cron.daily` and didn't saw the error. The PATH is correctly configured in `/etc/crontab` ``` [root@log01 ~][PROD-RDU3]# run-parts /etc/cron.daily/ /etc/cron.daily/sync-http-logs-and-merge.sh: /usr/local/bin/sync-http-logs.py:244: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. config = yaml.load(config_file) rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110) rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110) rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110) rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110) rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110) rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110) rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110) rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110) rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110) rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy05.vpn.fedoraproject.org (192.168.1.25): Connection timed out (110) rsync: [Receiver] failed to connect to proxy03.vpn.fedoraproject.org (192.168.1.7): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to proxy14.vpn.fedoraproject.org (192.168.1.159): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] rsync: [Receiver] failed to connect to kojipkgs01.rdu3.fedoraproject.org (10.16.169.106): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.5] ```
kevin commented 2026-01-19 22:05:51 +00:00
Author
Owner
Copy link

proxy03/14 were because the vpn ip wasn't getting staticly assigned. I fixed that.

proxy05 is down.

kojipkgs01 looks like a rhit firewall issue. We will need to file a internal ticket to get that allowed.
Not sure why kojipkgs02 isn't listed, it should be (and also doesn't work)

proxy03/14 were because the vpn ip wasn't getting staticly assigned. I fixed that. proxy05 is down. kojipkgs01 looks like a rhit firewall issue. We will need to file a internal ticket to get that allowed. Not sure why kojipkgs02 isn't listed, it should be (and also doesn't work)
victorkoycheff commented 2026-02-24 16:46:09 +00:00
Copy link

Hi @kevin,

I've submitted a PR to address the simple_message_to_bus: command not found errors seen in the logs.

In essence, t he issue was that the cron jobs were running with a restricted PATH, and the countme scripts specifically were attempting to call the message bus before the local path was updated. I've moved the path export to the top of the logic for all affected scripts in the web-data-analysis role.

PR: infra/ansible!3165

I verified the fix locally by simulating the cron environment (env PATH=/usr/bin:/bin). This should clear up the command errors while the rsync/firewall issues are being addressed.

PS: Also removed some funny comments ; )

Hi @kevin, I've submitted a PR to address the `simple_message_to_bus: command not found` errors seen in the logs. In essence, t he issue was that the cron jobs were running with a restricted `PATH`, and the `countme` scripts specifically were attempting to call the message bus before the local path was updated. I've moved the path export to the top of the logic for all affected scripts in the `web-data-analysis` role. **PR:** [infra/ansible!3165](https://forge.fedoraproject.org/infra/ansible/pulls/3165) I verified the fix locally by simulating the cron environment (`env PATH=/usr/bin:/bin`). This should clear up the command errors while the rsync/firewall issues are being addressed. PS: Also removed some funny comments ; )
kevin commented 2026-02-25 19:53:34 +00:00
Author
Owner
Copy link

Merged that. We will see tomorrow if it worked as expected?

Merged that. We will see tomorrow if it worked as expected?
👍 1
kevin commented 2026-03-03 20:17:25 +00:00
Author
Owner
Copy link

All the simple_message_to_bus messages are gone. Great!

There's some other messages tho:

plot ["2007-05-17":"2024-12-31"]      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:2  title 'epel4' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:3  title 'epel5' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:4  title 'epel6' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:5  title 'epel7' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:72  title 'epel8' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:73  title 'epel9' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:6  title 'fed03' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:7  title 'fed04' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:8  title 'fed05' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirro
 rsdata-all.csv' using 1:9  title 'fed06' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:10 title 'fed07' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:11 title 'fed08' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:12 title 'fed09' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:13 title 'fed10' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:14 title 'fed11' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:15 title 'fed12' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:16 title 'fed13' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:17 title 'fed14' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:18 title 'fed15' with lines lw 3,     '/var/www/html/cs
 v-reports/mirrors/mirrorsdata-all.csv' using 1:19 title 'fed16' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:20 title 'fed17' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:21 title 'fed18' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:22 title 'fed19' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:23 title 'fed20' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:24 title 'fed21' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:25 title 'fed22' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:26 title 'fed23' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:27 title 'fed24' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:28 title 'fed25' with lines lw 3
 ,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:29 title 'fed26' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:30 title 'fed27' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:31 title 'fed28' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:32 title 'fed29' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:33 title 'rawhide' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:34 title 'unk_rel' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:35 title 'EPEL' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:36 title 'Fedora' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:37 title 'alpha' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:38 ti
 tle 'ARM' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:39 title 'ARM64' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:40 title 'ia64' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:41 title 'mips' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:42 title 'ppc' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:43 title 's390' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:44 title 'sparc' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:45 title 'tilegx' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:46 title 'x86_32' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:47 title 'x86_64' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all
 .csv' using 1:48 title 'x86_32_e' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:49 title 'x86_32_f' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:50 title 'x86_64_e' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:51 title 'x86_64_f' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:52 title 'ppc_e' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:53 title 'ppc_f' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:54 title 'unknown' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:55 title 'centos' with lines lw 3,     '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:56 title 'rhel' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:57 title 'ppc64' with lines lw 3,      '/var/www/h
 tml/csv-reports/mirrors/mirrorsdata-all.csv' using 1:58 title 'ppc64le' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:59 title 'modular' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:60 title 'modular_rawhide' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:61 title 'modular_f27' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:62 title 'modular_f28' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:63 title 'modular_f29' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:64 title 'modular_f30' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:65 title 'fed30' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:66 title 'fed31' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsd
 ata-all.csv' using 1:67 title 'fed32' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:68 title 'fed33' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:69 title 'modular_f31' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:70 title 'modular_f32' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:71 title 'modular_f33' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:72 title 'epel8' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:73 title 'epel9' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:74 title 'f34' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:75 title 'f35' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:76 title 'f36' with lines lw 3,    
   '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:77 title 'f37' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:78 title 'f38' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:79 title 'f39' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:80 title 'modular_f34' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:81 title 'modular_f35' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:82 title 'modular_f36' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:83 title 'modular_f37' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:84 title 'modular_f38' with lines lw 3,      '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:85 title 'modular_f39' with lines lw 3
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ^
"/usr/local/share/web-data-analysis/mirrors-data.gp" line 107: all points y value undefined!
All the simple_message_to_bus messages are gone. Great! There's some other messages tho: ``` plot ["2007-05-17":"2024-12-31"] '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:2 title 'epel4' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:3 title 'epel5' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:4 title 'epel6' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:5 title 'epel7' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:72 title 'epel8' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:73 title 'epel9' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:6 title 'fed03' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:7 title 'fed04' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:8 title 'fed05' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirro rsdata-all.csv' using 1:9 title 'fed06' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:10 title 'fed07' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:11 title 'fed08' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:12 title 'fed09' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:13 title 'fed10' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:14 title 'fed11' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:15 title 'fed12' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:16 title 'fed13' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:17 title 'fed14' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:18 title 'fed15' with lines lw 3, '/var/www/html/cs v-reports/mirrors/mirrorsdata-all.csv' using 1:19 title 'fed16' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:20 title 'fed17' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:21 title 'fed18' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:22 title 'fed19' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:23 title 'fed20' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:24 title 'fed21' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:25 title 'fed22' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:26 title 'fed23' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:27 title 'fed24' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:28 title 'fed25' with lines lw 3 , '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:29 title 'fed26' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:30 title 'fed27' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:31 title 'fed28' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:32 title 'fed29' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:33 title 'rawhide' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:34 title 'unk_rel' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:35 title 'EPEL' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:36 title 'Fedora' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:37 title 'alpha' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:38 ti tle 'ARM' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:39 title 'ARM64' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:40 title 'ia64' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:41 title 'mips' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:42 title 'ppc' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:43 title 's390' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:44 title 'sparc' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:45 title 'tilegx' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:46 title 'x86_32' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:47 title 'x86_64' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all .csv' using 1:48 title 'x86_32_e' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:49 title 'x86_32_f' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:50 title 'x86_64_e' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:51 title 'x86_64_f' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:52 title 'ppc_e' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:53 title 'ppc_f' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:54 title 'unknown' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:55 title 'centos' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:56 title 'rhel' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:57 title 'ppc64' with lines lw 3, '/var/www/h tml/csv-reports/mirrors/mirrorsdata-all.csv' using 1:58 title 'ppc64le' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:59 title 'modular' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:60 title 'modular_rawhide' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:61 title 'modular_f27' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:62 title 'modular_f28' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:63 title 'modular_f29' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:64 title 'modular_f30' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:65 title 'fed30' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:66 title 'fed31' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsd ata-all.csv' using 1:67 title 'fed32' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:68 title 'fed33' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:69 title 'modular_f31' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:70 title 'modular_f32' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:71 title 'modular_f33' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:72 title 'epel8' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:73 title 'epel9' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:74 title 'f34' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:75 title 'f35' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:76 title 'f36' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:77 title 'f37' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:78 title 'f38' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:79 title 'f39' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:80 title 'modular_f34' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:81 title 'modular_f35' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:82 title 'modular_f36' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:83 title 'modular_f37' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:84 title 'modular_f38' with lines lw 3, '/var/www/html/csv-reports/mirrors/mirrorsdata-all.csv' using 1:85 title 'modular_f39' with lines lw 3 ^ "/usr/local/share/web-data-analysis/mirrors-data.gp" line 107: all points y value undefined! ```
victorkoycheff commented 2026-03-04 11:08:54 +00:00
Copy link

Awesome, glad the path fix worked!
Regarding the new gnuplot errors, the culprit was just a hardcoded date. The mirrors-data.gp script was strictly limiting the X-axis for all graphs to end on "2024-12-31". Since the logs are now parsing 2025/2026 data, gnuplot threw the undefined y value error because the new data fell completely outside the plotting window.
I've submitted a follow-up PR to change these to open-ended date ranges (e.g., ["2007-05-17":]) so it will automatically autoscale to whatever the newest data point is moving forward:
PR: [Insert your PR number/link here, e.g., infra/ansible!3191]
I verified the fix locally by feeding the script some mock 2026 data, and the graphs generated perfectly.

This should clear things up

Awesome, glad the path fix worked! Regarding the new `gnuplot` errors, the culprit was just a hardcoded date. The `mirrors-data.gp` script was strictly limiting the X-axis for all graphs to end on `"2024-12-31"`. Since the logs are now parsing 2025/2026 data, gnuplot threw the `undefined y value` error because the new data fell completely outside the plotting window. I've submitted a follow-up PR to change these to open-ended date ranges (e.g., `["2007-05-17":]`) so it will automatically autoscale to whatever the newest data point is moving forward: PR: [Insert your PR number/link here, e.g., infra/ansible!3191] I verified the fix locally by feeding the script some mock 2026 data, and the graphs generated perfectly. This should clear things up
kevin commented 2026-03-11 23:57:09 +00:00
Author
Owner
Copy link

No log problems since merging that pr.

The graphs look a bit weird... but that might be the old vs new data...

Lets trace that in #12426 and close this.

Thanks for the pr!

No log problems since merging that pr. The graphs look a bit weird... but that might be the old vs new data... Lets trace that in https://forge.fedoraproject.org/infra/tickets/issues/12426 and close this. Thanks for the pr!
kevin closed this issue 2026-03-11 23:57:09 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
announcement

Archived

  
anubis
Issues related to anubis

  
authentication
Authentication related issues

  
aws
Issues related to Amazon Web Services

  
backlog
Tasks that will take longer then ~4 hours to do

Archived

  
blocked
Issue is blocked

Archived

  
bodhi
Issues with Bodhi

  
ci
Continuous Integration

  
cloud
Items in FedoraInfraCloud or COmmunishift

  
communishift
issues/questions about os.edorainfracloud.org ( Community OpenShift)

  
copr
COPR system (run by other team)

  
database
Issues related to databases

  
day-to-day
These are small self contained tasks that are part of normal operations, but are async from sprint planning.

  
dc-move
Items related to datacenter move

Archived

  
deprecated
Tools which Infrastructure may not be able to fix

Archived

  
dev
Tasks requiring some dev work

  
discourse
outsourced tool

  
dns
Changes or problems with DNS

  
downloads
rsync,dl.fp.o,AWS,torrents

  
easyfix

Archived

  
epel
Issues relating to epel

Archived

  
firmitas

   
forgejo_migration
Request migration to the new forge

  
Gain
High
 task that gets us a lot / is important / makes many people happy

Archived

  
Gain
Low
 tasks that gets us little / isn't very important / makes only 1 person or few happy

Archived

  
Gain
Medium
 task that gets us some / is somewhat important / makes some people happy

Archived

  
gitlab

   
greenwave
Issues with greenwave

  
hardware
Problem is with Fedora Infrastructure Owned/Leased Hardware

  
help wanted

   
high-trouble
task that is complex/long/difficult

Archived

  
koji
Issues with the build system

  
koschei
Items needing Koschei team

  
lists
issues with lists

  
low-trouble
task that is easy/short

Archived

  
medium-trouble
task that is medium complex / difficult

Archived

  
mirrorlists
Way to tell systems where downloads are

  
monitoring

   
Needs investigation

   
odcs
On Demand Compose Service issues

Archived

  
OpenShift
Issues with OpenShift

  
ops
ops problem now...

Archived

  
outage

   
packager_workflow_blocker
Blocks the packager workflow

Archived

  
pagure
Issues with pagure.io

  
permissions
Items which need permissions set in outside tools

  
Priority
Needs Review
 Priority of Needs Review

Archived

  
Priority
Next Meeting
 Priority of Next Meeting

Archived

  
Priority
🔥 URGENT 🔥
 Priority of 🔥 URGENT 🔥

Archived

  
Priority
Waiting on Assignee
 Priority of Waiting on Assignee

Archived

  
Priority
Waiting on External
 Priority of Waiting on External

Archived

  
Priority
Waiting on Reporter
 Priority of Waiting on Reporter

Archived

  
rabbitmq
issues with the message broker

  
release-monitoring
release-monitoring.org (Anitya)

  
releng
For problems which should be in the releng ticket system

Archived

  
request-for-resources

   
s390x
The secondary architecture hardware and problems related with it

  
security

   
SMTP
Email issues outside of lists (DMARC, SPF, aliases)

  
sprint-0

Archived

  
sprint-1

Archived

  
src.fp.o
Issues related to src.fedoraproject.org

  
staging
Issues in staging

  
unfreeze
Patches to be applied after freeze is lifted

  
waiverdb
Issues with waiverdb

  
websites-general
General items with a website focus

  
wiki
Issues with the Wiki

  
Backlog Status
Needs Review
 Work items that have been created or updated but need review, refinement, or approval before they can be considered ready for sprint planning.

  
Backlog Status
Ready
 Work items that have been reviewed, refined, and are ready to be committed to a sprint. These items have clear requirements and can be worked on immediately when capacity is available.

  
chore
Work that needs to be done but doesn't add new functionality or fix bugs. Maintenance tasks that keep the system running smoothly and the codebase clean.

  
documentation
Work focused on creating, updating, or improving documentation for users, developers, or system administrators.

  
points
01
 A task that is simple, small, and has very little uncertainty. It should be easy enough for any team member to pick up and complete quickly, without a lot of discussion or research.

  
points
02
 This is a task that is slightly more complex or larger than a "1." It might involve two or three simple steps, but the outcome is still very clear and there's little uncertainty.

  
points
03
 Medium-sized task with some noticeable complexity. It might require a bit more thought or collaboration. The steps are well-defined, but it's not a trivial change.

  
points
05
 A significant piece of work. It has more complexity and a higher degree of uncertainty than smaller tasks. It might require some research, a design discussion, or collaboration across multiple teams.

  
points
08
 Complex task with a lot of uncertainty. It's often the largest size that a team will consider for a single sprint. It likely requires a detailed breakdown and multiple team members to complete.

  
points
13
 Any task that feels like it's a "13" or more is considered an Epic. These are too large and uncertain to be worked on in a single sprint and should be broken down.

  
Priority
High
 This is for work that is critical and needs to be addressed in the next sprint. It's often for urgent bugs, important features, or a major piece of technical debt that is blocking other work.

  
Priority
Low
 This is for work that is good to do but not urgent. It's a candidate for being worked on when there's extra time or when it rises in importance. It can stay in the backlog for a longer period.

  
Priority
Medium
 This is for important work that should be done soon, but isn't as urgent as a high-priority item. It will likely be considered for the next 1-2 sprints.

  
Sprint Status
Blocked
 Work items that cannot progress due to external dependencies, blockers, or issues that need resolution.

  
Sprint Status
Done
 Work items that have been completed, reviewed, and integrated. No further work is needed within the current sprint.

  
Sprint Status
In Progress
 Work items currently being worked on by a team member. Active development, research, or implementation is happening.

  
Sprint Status
Review
 Work items that are complete but need review, testing, or approval before being marked as done.

  
Sprint Status
To Do
 Work items that are committed to the sprint but haven't been started yet. These items are ready to be picked up by team members.

  
Technical Debt
Work that addresses shortcuts, workarounds, or suboptimal implementations that were done previously to meet deadlines. These items improve code quality, maintainability, or performance without adding new features.

  
Work Item
Bug
 Issues where the system is not working as intended or designed. Bugs represent deviations from expected behavior.

  
Work Item
Epic
 Large initiatives that are broken down into multiple smaller work items (user stories, tasks, bugs). Epics represent significant features or projects that span multiple sprints.

  
Work Item
Spike
 Time-boxed research, investigation, or proof-of-concept work to reduce uncertainty or explore solutions. Spikes help inform future development decisions.

  
Work Item
Task
 Used to break down larger user stories into smaller, manageable implementation pieces. They focus on the specific technical steps needed to deliver the user story's value.

  
Work Item
User Story
 User-facing features or functionality that can be framed from a user perspective, describing what someone wants to accomplish and why.

No labels
announcement
anubis
authentication
aws
backlog
blocked
bodhi
ci
cloud
communishift
copr
database
day-to-day
dc-move
deprecated
dev
discourse
dns
downloads
easyfix
epel
firmitas
forgejo_migration
Gain
High
Gain
Low
Gain
Medium
gitlab
greenwave
hardware
help wanted
high-trouble
koji
koschei
lists
low-trouble
medium-trouble
mirrorlists
monitoring
Needs investigation
odcs
OpenShift
ops
outage
packager_workflow_blocker
pagure
permissions
Priority
Needs Review
Priority
Next Meeting
Priority
🔥 URGENT 🔥
Priority
Waiting on Assignee
Priority
Waiting on External
Priority
Waiting on Reporter
rabbitmq
release-monitoring
releng
request-for-resources
s390x
security
SMTP
sprint-0
sprint-1
src.fp.o
staging
unfreeze
waiverdb
websites-general
wiki
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
infra/tickets#12833
No description provided.