infra/tickets
23
8
Fork 1
Code Issues 85 Projects Wiki Activity Actions

httpd on pkgs01 regularly dumps core #12670

New issue
Closed
opened 2025-07-17 14:50:27 +00:00 by nphilipp · 11 comments
nphilipp commented 2025-07-17 14:50:27 +00:00
Member
Copy link

We discussed it in this morning’s (08:15 UTC) Infra/RelEng standup call, I think @zlopez mentioned it first: the httpd process on pkgs01.rdu3 regularly dumps core (every odd minute or three):

[root@pkgs01 pagure][PROD-RDU3]# coredumpctl list --reverse | head
TIME                            PID   UID   GID SIG COREFILE  EXE
Thu 2025-07-17 14:16:30 GMT  1267788  1000 1494358698  11 present   /usr/sbin/httpd
Thu 2025-07-17 14:14:46 GMT  1267162  1000 1494358698  11 present   /usr/sbin/httpd
Thu 2025-07-17 14:13:19 GMT  1266272  1000 1494358698   6 present   /usr/sbin/httpd
Thu 2025-07-17 14:10:02 GMT  1265106  1000 1494358698  11 present   /usr/sbin/httpd
Thu 2025-07-17 14:09:00 GMT  1264831  1000 1494358698   6 present   /usr/sbin/httpd
Thu 2025-07-17 14:08:43 GMT  1264737  1000 1494358698  11 present   /usr/sbin/httpd
Thu 2025-07-17 14:07:36 GMT  1264147  1000 1494358698  11 present   /usr/sbin/httpd
Thu 2025-07-17 14:06:31 GMT  1263119  1000 1494358698  11 present   /usr/sbin/httpd
Thu 2025-07-17 14:06:19 GMT  1263060  1000 1494358698  11 present   /usr/sbin/httpd

Here’s an exemplary one:

[root@pkgs01 pagure][PROD-RDU3]# coredumpctl info 1267788
           PID: 1267788 (httpd)
           UID: 1000 (pagure)
           GID: 1494358698 (xpackager)
        Signal: 11 (SEGV)
     Timestamp: Thu 2025-07-17 14:16:29 GMT (15min ago)
  Command Line: pagure          -DFOREGROUND
    Executable: /usr/sbin/httpd
 Control Group: /
         Slice: -.slice
       Boot ID: d6921eff650c4d358b02b8be1d34b576
    Machine ID: 2cb1691b08bd4a8fa022e3bf3a140ff3
      Hostname: pkgs01.rdu3.fedoraproject.org
       Storage: /var/lib/systemd/coredump/core.httpd.1000.d6921eff650c4d358b02b8be1d34b576.1267788.17>
       Message: Process 1267788 (httpd) of user 1000 dumped core.
                
                Stack trace of thread 1269789:
                #0  0x00007fe279df7b53 PyErr_SetObject (libpython3.6m.so.1.0)
                #1  0x00007fe279e02c21 PyErr_SetString (libpython3.6m.so.1.0)
                #2  0x00007fe279e10be2 PyException_SetTraceback (libpython3.6m.so.1.0)
                #3  0x00007fe279d61dc1 _PyErr_FormatFromCause (libpython3.6m.so.1.0)
                #4  0x00007fe279dc8679 call_function.cold.3302 (libpython3.6m.so.1.0)
                #5  0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #6  0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0)
                #7  0x00007fe279e8f657 call_function (libpython3.6m.so.1.0)
                #8  0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #9  0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0)
                #10 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0)
                #11 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #12 0x00007fe279dec154 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0)
                #13 0x00007fe279e6c370 fast_function (libpython3.6m.so.1.0)
                #14 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0)
                #15 0x00007fe279e90ef5 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #16 0x00007fe279decac5 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0)
                #17 0x00007fe279e8e42f function_call (libpython3.6m.so.1.0)
                #18 0x00007fe279df522b PyObject_Call (libpython3.6m.so.1.0)
                #19 0x00007fe279e92120 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #20 0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0)
                #21 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0)
                #22 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #23 0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0)
                #24 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0)
                #25 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #26 0x00007fe279ded682 _PyFunction_FastCallDict (libpython3.6m.so.1.0)
                #27 0x00007fe279dee45e _PyObject_FastCallDict (libpython3.6m.so.1.0)
                #28 0x00007fe279e00290 method_call (libpython3.6m.so.1.0)
                #29 0x00007fe279df522b PyObject_Call (libpython3.6m.so.1.0)
                #30 0x00007fe279f00872 t_bootstrap (libpython3.6m.so.1.0)
                #31 0x00007fe279ea6a14 pythread_wrapper (libpython3.6m.so.1.0)
                #32 0x00007fe28c4d51ca start_thread (libpthread.so.0)
                #33 0x00007fe28bf2b8d3 __clone (libc.so.6)
                
                Stack trace of thread 1267788:
                #0  0x00007fe28c4db6e5 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007fe279deb68c take_gil (libpython3.6m.so.1.0)
                #2  0x00007fe279e9665e _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #3  0x00007fe279dec154 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0)
                #4  0x00007fe279ded87a _PyFunction_FastCallDict (libpython3.6m.so.1.0)
                #5  0x00007fe279dee45e _PyObject_FastCallDict (libpython3.6m.so.1.0)
                #6  0x00007fe279e0f588 PyObject_CallFunctionObjArgs (libpython3.6m.so.1.0)
                #7  0x00007fe279e6aa47 collect (libpython3.6m.so.1.0)
                #8  0x00007fe279ee27fb _PyGC_CollectNoFail (libpython3.6m.so.1.0)
                #9  0x00007fe279efc839 PyImport_Cleanup (libpython3.6m.so.1.0)
                #10 0x00007fe279dc402c Py_EndInterpreter (libpython3.6m.so.1.0)
                #11 0x00007fe27a2441e7 Interpreter_dealloc (mod_wsgi_python3.so)
                #12 0x00007fe279dd926e free_keys_object (libpython3.6m.so.1.0)
                #13 0x00007fe27a24d9ba wsgi_python_child_cleanup (mod_wsgi_python3.so)
                #14 0x00007fe28c70dd6e apr_pool_destroy (libapr-1.so.0)
                #15 0x00007fe27a258f95 wsgi_start_process (mod_wsgi_python3.so)
                #16 0x00007fe27a25b924 wsgi_manage_process (mod_wsgi_python3.so)
                #17 0x00007fe28c71025f apr_proc_other_child_alert (libapr-1.so.0)
                #18 0x00007fe2811e17aa event_run (mod_mpm_event.so)
                #19 0x0000562dc4d112ee ap_run_mpm (httpd)
                #20 0x0000562dc4d097a3 main (httpd)
                #21 0x00007fe28bf2c7e5 __libc_start_main (libc.so.6)
                #22 0x0000562dc4d098ee _start (httpd)
                
                Stack trace of thread 1269790:
                #0  0x00007fe28c0272df __select (libc.so.6)
                #1  0x00007fe279ee276a time_sleep (libpython3.6m.so.1.0)
                #2  0x00007fe279e8f52d call_function (libpython3.6m.so.1.0)
                #3  0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #4  0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0)
                #5  0x00007fe279e8f657 call_function (libpython3.6m.so.1.0)
                #6  0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #7  0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0)
                #8  0x00007fe279e8f657 call_function (libpython3.6m.so.1.0)
                #9  0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #10 0x00007fe279ded682 _PyFunction_FastCallDict (libpython3.6m.so.1.0)
                #11 0x00007fe279dee45e _PyObject_FastCallDict (libpython3.6m.so.1.0)
                #12 0x00007fe279e00290 method_call (libpython3.6m.so.1.0)
                #13 0x00007fe279df522b PyObject_Call (libpython3.6m.so.1.0)
                #14 0x00007fe279f00872 t_bootstrap (libpython3.6m.so.1.0)
                #15 0x00007fe279ea6a14 pythread_wrapper (libpython3.6m.so.1.0)
                #16 0x00007fe28c4d51ca start_thread (libpthread.so.0)
                #17 0x00007fe28bf2b8d3 __clone (libc.so.6)
                
                Stack trace of thread 1269788:
                #0  0x00007fe28c031477 epoll_wait (libc.so.6)
                #1  0x00007fe271fa95d8 pyepoll_poll (select.cpython-36m-x86_64-linux-gnu.so)
                #2  0x00007fe279e8f8d1 call_function (libpython3.6m.so.1.0)
                #3  0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #4  0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0)
                #5  0x00007fe279e8f657 call_function (libpython3.6m.so.1.0)
                #6  0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #7  0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0)
                #8  0x00007fe279e8f657 call_function (libpython3.6m.so.1.0)
                #9  0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #10 0x00007fe279dec154 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0)
                #11 0x00007fe279e6c370 fast_function (libpython3.6m.so.1.0)
                #12 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0)
                #13 0x00007fe279e90ef5 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #14 0x00007fe279ded682 _PyFunction_FastCallDict (libpython3.6m.so.1.0)
                #15 0x00007fe279dee45e _PyObject_FastCallDict (libpython3.6m.so.1.0)
                #16 0x00007fe279e00290 method_call (libpython3.6m.so.1.0)
                #17 0x00007fe279df522b PyObject_Call (libpython3.6m.so.1.0)
                #18 0x00007fe279e92120 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #19 0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0)
                #20 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0)
                #21 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #22 0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0)
                #23 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0)
                #24 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #25 0x00007fe279ded682 _PyFunction_FastCallDict (libpython3.6m.so.1.0)
                #26 0x00007fe279dee45e _PyObject_FastCallDict (libpython3.6m.so.1.0)
                #27 0x00007fe279e00290 method_call (libpython3.6m.so.1.0)
                #28 0x00007fe279df522b PyObject_Call (libpython3.6m.so.1.0)
                #29 0x00007fe279f00872 t_bootstrap (libpython3.6m.so.1.0)
                #30 0x00007fe279ea6a14 pythread_wrapper (libpython3.6m.so.1.0)
                #31 0x00007fe28c4d51ca start_thread (libpthread.so.0)
                #32 0x00007fe28bf2b8d3 __clone (libc.so.6)
                
                Stack trace of thread 1270890:
                #0  0x00007fe28bffa178 __nanosleep (libc.so.6)
                #1  0x00007fe28bffa07e sleep (libc.so.6)
                #2  0x00007fe27a24daf9 wsgi_reaper_thread (mod_wsgi_python3.so)
                #3  0x00007fe28c4d51ca start_thread (libpthread.so.0)
                #4  0x00007fe28bf2b8d3 __clone (libc.so.6)
                
                Stack trace of thread 1267789:
                #0  0x00007fe28c0272df __select (libc.so.6)
                #1  0x00007fe28c719919 apr_sleep (libapr-1.so.0)
                #2  0x00007fe27a24a75b wsgi_monitor_thread (mod_wsgi_python3.so)
                #3  0x00007fe28c4d51ca start_thread (libpthread.so.0)
                #4  0x00007fe28bf2b8d3 __clone (libc.so.6)
                
                Stack trace of thread 1267790:
                #0  0x00007fe28c0272df __select (libc.so.6)
                #1  0x00007fe28c719919 apr_sleep (libapr-1.so.0)
                #2  0x00007fe27a24d43a wsgi_deadlock_thread (mod_wsgi_python3.so)
                #3  0x00007fe28c4d51ca start_thread (libpthread.so.0)
                #4  0x00007fe28bf2b8d3 __clone (libc.so.6)
                
                Stack trace of thread 1269791:
                #0  0x00007fe28c4dda46 do_futex_wait.constprop.1 (libpthread.so.0)
                #1  0x00007fe28c4ddb38 __new_sem_wait_slow.constprop.0 (libpthread.so.0)
                #2  0x00007fe279deadf2 PyThread_acquire_lock_timed (libpython3.6m.so.1.0)
                #3  0x00007fe279e698f4 lock_PyThread_acquire_lock (libpython3.6m.so.1.0)
                #4  0x00007fe279e8f8d1 call_function (libpython3.6m.so.1.0)
                #5  0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #6  0x00007fe279dec154 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0)
                #7  0x00007fe279e6c370 fast_function (libpython3.6m.so.1.0)
                #8  0x00007fe279e8f657 call_function (libpython3.6m.so.1.0)
                #9  0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #10 0x00007fe279dec154 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0)
                #11 0x00007fe279ded8df _PyFunction_FastCallDict (libpython3.6m.so.1.0)
                #12 0x00007fe279dee45e _PyObject_FastCallDict (libpython3.6m.so.1.0)
                #13 0x00007fe279e00290 method_call (libpython3.6m.so.1.0)
                #14 0x00007fe279dee43c _PyObject_FastCallDict (libpython3.6m.so.1.0)
                #15 0x00007fe279ec13f0 calliter_iternext (libpython3.6m.so.1.0)
                #16 0x00007fe279e90646 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #17 0x00007fe279decac5 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0)
                #18 0x00007fe279e8e42f function_call (libpython3.6m.so.1.0)
                #19 0x00007fe279df522b PyObject_Call (libpython3.6m.so.1.0)
                #20 0x00007fe279e92120 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #21 0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0)
                #22 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0)
                #23 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #24 0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0)
                #25 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0)
                #26 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)
                #27 0x00007fe279ded682 _PyFunction_FastCallDict (libpython3.6m.so.1.0)
                #28 0x00007fe279dee45e _PyObject_FastCallDict (libpython3.6m.so.1.0)
                #29 0x00007fe279e00290 method_call (libpython3.6m.so.1.0)
                #30 0x00007fe279df522b PyObject_Call (libpython3.6m.so.1.0)
                #31 0x00007fe279f00872 t_bootstrap (libpython3.6m.so.1.0)
                #32 0x00007fe279ea6a14 pythread_wrapper (libpython3.6m.so.1.0)
                #33 0x00007fe28c4d51ca start_thread (libpthread.so.0)
                #34 0x00007fe28bf2b8d3 __clone (libc.so.6)

The SIGSEGV is delivered to the offending thread, whose backtrace always seems to have this structure:

pthread_cond_timedwait@@GLIBC_2.3.2 => take_gil => … => [Python Garbage Collector related] => … => PyImport_Cleanup => Py_EndInterpreter => … => wsgi_python_child_cleanup => apr_pool_destroy

I take it as: httpd winds down a worker, mod_wsgi cleans up the Python (sub-)interpreter, and garbage collection trips over something that doesn’t exist any longer.

I don’t know why this didn’t happen before the DC move, so I can only guess that maybe we had this problem before, a configuration hot-fix was applied which didn’t end up in Ansible, so the fix was lost in the move. Or it happened and we didn’t notice?

Anyway, let’s collect more info here.

We discussed it in this morning’s (08:15 UTC) Infra/RelEng standup call, I think @zlopez mentioned it first: the `httpd` process on `pkgs01.rdu3` regularly dumps core (every odd minute or three): ``` [root@pkgs01 pagure][PROD-RDU3]# coredumpctl list --reverse | head TIME PID UID GID SIG COREFILE EXE Thu 2025-07-17 14:16:30 GMT 1267788 1000 1494358698 11 present /usr/sbin/httpd Thu 2025-07-17 14:14:46 GMT 1267162 1000 1494358698 11 present /usr/sbin/httpd Thu 2025-07-17 14:13:19 GMT 1266272 1000 1494358698 6 present /usr/sbin/httpd Thu 2025-07-17 14:10:02 GMT 1265106 1000 1494358698 11 present /usr/sbin/httpd Thu 2025-07-17 14:09:00 GMT 1264831 1000 1494358698 6 present /usr/sbin/httpd Thu 2025-07-17 14:08:43 GMT 1264737 1000 1494358698 11 present /usr/sbin/httpd Thu 2025-07-17 14:07:36 GMT 1264147 1000 1494358698 11 present /usr/sbin/httpd Thu 2025-07-17 14:06:31 GMT 1263119 1000 1494358698 11 present /usr/sbin/httpd Thu 2025-07-17 14:06:19 GMT 1263060 1000 1494358698 11 present /usr/sbin/httpd ``` Here’s an exemplary one: ``` [root@pkgs01 pagure][PROD-RDU3]# coredumpctl info 1267788 PID: 1267788 (httpd) UID: 1000 (pagure) GID: 1494358698 (xpackager) Signal: 11 (SEGV) Timestamp: Thu 2025-07-17 14:16:29 GMT (15min ago) Command Line: pagure -DFOREGROUND Executable: /usr/sbin/httpd Control Group: / Slice: -.slice Boot ID: d6921eff650c4d358b02b8be1d34b576 Machine ID: 2cb1691b08bd4a8fa022e3bf3a140ff3 Hostname: pkgs01.rdu3.fedoraproject.org Storage: /var/lib/systemd/coredump/core.httpd.1000.d6921eff650c4d358b02b8be1d34b576.1267788.17> Message: Process 1267788 (httpd) of user 1000 dumped core. Stack trace of thread 1269789: #0 0x00007fe279df7b53 PyErr_SetObject (libpython3.6m.so.1.0) #1 0x00007fe279e02c21 PyErr_SetString (libpython3.6m.so.1.0) #2 0x00007fe279e10be2 PyException_SetTraceback (libpython3.6m.so.1.0) #3 0x00007fe279d61dc1 _PyErr_FormatFromCause (libpython3.6m.so.1.0) #4 0x00007fe279dc8679 call_function.cold.3302 (libpython3.6m.so.1.0) #5 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #6 0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0) #7 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0) #8 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #9 0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0) #10 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0) #11 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #12 0x00007fe279dec154 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0) #13 0x00007fe279e6c370 fast_function (libpython3.6m.so.1.0) #14 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0) #15 0x00007fe279e90ef5 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #16 0x00007fe279decac5 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0) #17 0x00007fe279e8e42f function_call (libpython3.6m.so.1.0) #18 0x00007fe279df522b PyObject_Call (libpython3.6m.so.1.0) #19 0x00007fe279e92120 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #20 0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0) #21 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0) #22 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #23 0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0) #24 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0) #25 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #26 0x00007fe279ded682 _PyFunction_FastCallDict (libpython3.6m.so.1.0) #27 0x00007fe279dee45e _PyObject_FastCallDict (libpython3.6m.so.1.0) #28 0x00007fe279e00290 method_call (libpython3.6m.so.1.0) #29 0x00007fe279df522b PyObject_Call (libpython3.6m.so.1.0) #30 0x00007fe279f00872 t_bootstrap (libpython3.6m.so.1.0) #31 0x00007fe279ea6a14 pythread_wrapper (libpython3.6m.so.1.0) #32 0x00007fe28c4d51ca start_thread (libpthread.so.0) #33 0x00007fe28bf2b8d3 __clone (libc.so.6) Stack trace of thread 1267788: #0 0x00007fe28c4db6e5 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0) #1 0x00007fe279deb68c take_gil (libpython3.6m.so.1.0) #2 0x00007fe279e9665e _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #3 0x00007fe279dec154 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0) #4 0x00007fe279ded87a _PyFunction_FastCallDict (libpython3.6m.so.1.0) #5 0x00007fe279dee45e _PyObject_FastCallDict (libpython3.6m.so.1.0) #6 0x00007fe279e0f588 PyObject_CallFunctionObjArgs (libpython3.6m.so.1.0) #7 0x00007fe279e6aa47 collect (libpython3.6m.so.1.0) #8 0x00007fe279ee27fb _PyGC_CollectNoFail (libpython3.6m.so.1.0) #9 0x00007fe279efc839 PyImport_Cleanup (libpython3.6m.so.1.0) #10 0x00007fe279dc402c Py_EndInterpreter (libpython3.6m.so.1.0) #11 0x00007fe27a2441e7 Interpreter_dealloc (mod_wsgi_python3.so) #12 0x00007fe279dd926e free_keys_object (libpython3.6m.so.1.0) #13 0x00007fe27a24d9ba wsgi_python_child_cleanup (mod_wsgi_python3.so) #14 0x00007fe28c70dd6e apr_pool_destroy (libapr-1.so.0) #15 0x00007fe27a258f95 wsgi_start_process (mod_wsgi_python3.so) #16 0x00007fe27a25b924 wsgi_manage_process (mod_wsgi_python3.so) #17 0x00007fe28c71025f apr_proc_other_child_alert (libapr-1.so.0) #18 0x00007fe2811e17aa event_run (mod_mpm_event.so) #19 0x0000562dc4d112ee ap_run_mpm (httpd) #20 0x0000562dc4d097a3 main (httpd) #21 0x00007fe28bf2c7e5 __libc_start_main (libc.so.6) #22 0x0000562dc4d098ee _start (httpd) Stack trace of thread 1269790: #0 0x00007fe28c0272df __select (libc.so.6) #1 0x00007fe279ee276a time_sleep (libpython3.6m.so.1.0) #2 0x00007fe279e8f52d call_function (libpython3.6m.so.1.0) #3 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #4 0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0) #5 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0) #6 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #7 0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0) #8 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0) #9 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #10 0x00007fe279ded682 _PyFunction_FastCallDict (libpython3.6m.so.1.0) #11 0x00007fe279dee45e _PyObject_FastCallDict (libpython3.6m.so.1.0) #12 0x00007fe279e00290 method_call (libpython3.6m.so.1.0) #13 0x00007fe279df522b PyObject_Call (libpython3.6m.so.1.0) #14 0x00007fe279f00872 t_bootstrap (libpython3.6m.so.1.0) #15 0x00007fe279ea6a14 pythread_wrapper (libpython3.6m.so.1.0) #16 0x00007fe28c4d51ca start_thread (libpthread.so.0) #17 0x00007fe28bf2b8d3 __clone (libc.so.6) Stack trace of thread 1269788: #0 0x00007fe28c031477 epoll_wait (libc.so.6) #1 0x00007fe271fa95d8 pyepoll_poll (select.cpython-36m-x86_64-linux-gnu.so) #2 0x00007fe279e8f8d1 call_function (libpython3.6m.so.1.0) #3 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #4 0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0) #5 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0) #6 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #7 0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0) #8 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0) #9 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #10 0x00007fe279dec154 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0) #11 0x00007fe279e6c370 fast_function (libpython3.6m.so.1.0) #12 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0) #13 0x00007fe279e90ef5 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #14 0x00007fe279ded682 _PyFunction_FastCallDict (libpython3.6m.so.1.0) #15 0x00007fe279dee45e _PyObject_FastCallDict (libpython3.6m.so.1.0) #16 0x00007fe279e00290 method_call (libpython3.6m.so.1.0) #17 0x00007fe279df522b PyObject_Call (libpython3.6m.so.1.0) #18 0x00007fe279e92120 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #19 0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0) #20 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0) #21 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #22 0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0) #23 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0) #24 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #25 0x00007fe279ded682 _PyFunction_FastCallDict (libpython3.6m.so.1.0) #26 0x00007fe279dee45e _PyObject_FastCallDict (libpython3.6m.so.1.0) #27 0x00007fe279e00290 method_call (libpython3.6m.so.1.0) #28 0x00007fe279df522b PyObject_Call (libpython3.6m.so.1.0) #29 0x00007fe279f00872 t_bootstrap (libpython3.6m.so.1.0) #30 0x00007fe279ea6a14 pythread_wrapper (libpython3.6m.so.1.0) #31 0x00007fe28c4d51ca start_thread (libpthread.so.0) #32 0x00007fe28bf2b8d3 __clone (libc.so.6) Stack trace of thread 1270890: #0 0x00007fe28bffa178 __nanosleep (libc.so.6) #1 0x00007fe28bffa07e sleep (libc.so.6) #2 0x00007fe27a24daf9 wsgi_reaper_thread (mod_wsgi_python3.so) #3 0x00007fe28c4d51ca start_thread (libpthread.so.0) #4 0x00007fe28bf2b8d3 __clone (libc.so.6) Stack trace of thread 1267789: #0 0x00007fe28c0272df __select (libc.so.6) #1 0x00007fe28c719919 apr_sleep (libapr-1.so.0) #2 0x00007fe27a24a75b wsgi_monitor_thread (mod_wsgi_python3.so) #3 0x00007fe28c4d51ca start_thread (libpthread.so.0) #4 0x00007fe28bf2b8d3 __clone (libc.so.6) Stack trace of thread 1267790: #0 0x00007fe28c0272df __select (libc.so.6) #1 0x00007fe28c719919 apr_sleep (libapr-1.so.0) #2 0x00007fe27a24d43a wsgi_deadlock_thread (mod_wsgi_python3.so) #3 0x00007fe28c4d51ca start_thread (libpthread.so.0) #4 0x00007fe28bf2b8d3 __clone (libc.so.6) Stack trace of thread 1269791: #0 0x00007fe28c4dda46 do_futex_wait.constprop.1 (libpthread.so.0) #1 0x00007fe28c4ddb38 __new_sem_wait_slow.constprop.0 (libpthread.so.0) #2 0x00007fe279deadf2 PyThread_acquire_lock_timed (libpython3.6m.so.1.0) #3 0x00007fe279e698f4 lock_PyThread_acquire_lock (libpython3.6m.so.1.0) #4 0x00007fe279e8f8d1 call_function (libpython3.6m.so.1.0) #5 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #6 0x00007fe279dec154 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0) #7 0x00007fe279e6c370 fast_function (libpython3.6m.so.1.0) #8 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0) #9 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #10 0x00007fe279dec154 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0) #11 0x00007fe279ded8df _PyFunction_FastCallDict (libpython3.6m.so.1.0) #12 0x00007fe279dee45e _PyObject_FastCallDict (libpython3.6m.so.1.0) #13 0x00007fe279e00290 method_call (libpython3.6m.so.1.0) #14 0x00007fe279dee43c _PyObject_FastCallDict (libpython3.6m.so.1.0) #15 0x00007fe279ec13f0 calliter_iternext (libpython3.6m.so.1.0) #16 0x00007fe279e90646 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #17 0x00007fe279decac5 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0) #18 0x00007fe279e8e42f function_call (libpython3.6m.so.1.0) #19 0x00007fe279df522b PyObject_Call (libpython3.6m.so.1.0) #20 0x00007fe279e92120 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #21 0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0) #22 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0) #23 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #24 0x00007fe279e6c188 fast_function (libpython3.6m.so.1.0) #25 0x00007fe279e8f657 call_function (libpython3.6m.so.1.0) #26 0x00007fe279e902b8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0) #27 0x00007fe279ded682 _PyFunction_FastCallDict (libpython3.6m.so.1.0) #28 0x00007fe279dee45e _PyObject_FastCallDict (libpython3.6m.so.1.0) #29 0x00007fe279e00290 method_call (libpython3.6m.so.1.0) #30 0x00007fe279df522b PyObject_Call (libpython3.6m.so.1.0) #31 0x00007fe279f00872 t_bootstrap (libpython3.6m.so.1.0) #32 0x00007fe279ea6a14 pythread_wrapper (libpython3.6m.so.1.0) #33 0x00007fe28c4d51ca start_thread (libpthread.so.0) #34 0x00007fe28bf2b8d3 __clone (libc.so.6) ``` The SIGSEGV is delivered to the offending thread, whose backtrace always seems to have this structure: pthread_cond_timedwait@@GLIBC_2.3.2 => take_gil => … => [Python Garbage Collector related] => … => PyImport_Cleanup => Py_EndInterpreter => … => wsgi_python_child_cleanup => apr_pool_destroy I take it as: `httpd` winds down a worker, `mod_wsgi` cleans up the Python (sub-)interpreter, and garbage collection trips over something that doesn’t exist any longer. I don’t know why this didn’t happen before the DC move, so I can only guess that maybe we had this problem before, a configuration hot-fix was applied which didn’t end up in Ansible, so the fix was lost in the move. Or it happened and we didn’t notice? Anyway, let’s collect more info here.
nphilipp commented 2025-07-17 14:59:54 +00:00
Author
Member
Copy link

I was curious and checked on pagure02:

[root@pagure02 ~][PROD]# coredumpctl --reverse | head
No coredumps found.

Perhaps comparing differences between the src.fp.o and pagure.io instances can give some insight in how to fix it.

I was curious and checked on `pagure02`: ``` [root@pagure02 ~][PROD]# coredumpctl --reverse | head No coredumps found. ``` Perhaps comparing differences between the `src.fp.o` and `pagure.io` instances can give some insight in how to fix it.
kevin commented 2025-07-17 19:13:37 +00:00
Owner
Copy link

Metadata Update from @kevin:

  • Issue priority set to: Waiting on Assignee (was: Needs Review)
  • Issue tagged with: medium-gain, medium-trouble, ops
**Metadata Update from @kevin**: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: medium-gain, medium-trouble, ops
zlopez commented 2025-07-18 06:52:42 +00:00
Owner
Copy link

@nphilipp The src.fp.o didn't produced coredumps till I added CoreDumpDirectory /tmp to httpd dist-git conf. But you can still see the errors being reported in httpd error log.

@nphilipp The src.fp.o didn't produced coredumps till I added `CoreDumpDirectory /tmp` to httpd dist-git conf. But you can still see the errors being reported in httpd error log.
nphilipp commented 2025-07-18 16:55:14 +00:00
Author
Member
Copy link

It seems that pkgs01.iad2 dumped core before the DC move, we just didn’t notice:

[root@log01 30][PROD-RDU3]# pwd
/var/log/hosts/pkgs01.iad2.fedoraproject.org/2025/06/30
[root@log01 30][PROD-RDU3]# xzgrep systemd-coredump messages.log.xz | tail -n 5
Jun 30 14:21:13 pkgs01 systemd-coredump[1782251]: Removed old coredump core.httpd.1000.f6c697a505ae4311887ed864512cdd0a.1649482.1751282904000000.lz4.
Jun 30 14:21:14 pkgs01 systemd-coredump[1782251]: Process 1780105 (httpd) of user 1000 dumped core.#012#012Stack trace of thread 1781734:#012#0  0x00007fc9ae3afa12 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0)#012#1  0x00007fc9ae42f2f0 fast_function (libpython3.6m.so.1.0)#012#2  0x00007fc9ae452537 call_function (libpython3.6m.so.1.0)#012#3  0x00007fc9ae453198 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)#012#4  0x00007fc9ae42f108 fast_function (libpython3.6m.so.1.0)#012#5  0x00007fc9ae452537 call_function (libpython3.6m.so.1.0)#012#6  0x00007fc9ae453198 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)#012#7  0x00007fc9ae3af0a4 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0)#012#8  0x00007fc9ae42f2f0 fast_function (libpython3.6m.so.1.0)#012#9  0x00007fc9ae452537 call_function (libpython3.6m.so.1.0)#012#10 0x00007fc9ae453dd8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)#012#11 0x00007fc9ae3afa15 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0)#012#12 0x00007fc9ae45130f function_call (libpython3.6m.so.1.0)#012#13 0x00007fc9ae3b822b PyObject_Call (libpython3.6m.so.1.0)#012#14 0x00007fc9ae455005 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)#012#15 0x00007fc9ae42f108 fast_function (libpython3.6m.so.1.0)#012#16 0x00007fc9ae452537 call_function (libpython3.6m.so.1.0)#012#17 0x00007fc9ae453198 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)#012#18 0x00007fc9ae42f108 fast_function (libpython3.6m.so.1.0)#012#19 0x00007fc9ae452537 call_function (libpython3.6m.so.1.0)#012#20 0x00007fc9ae453198 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)#012#21 0x00007fc9ae3b05d2 _PyFunction_FastCallDict (libpython3.6m.so.1.0)#012#22 0x00007fc9ae3b13ae _PyObject_FastCallDict (libpython3.6m.so.1.0)#012#23 0x00007fc9ae3c32c0 method_call (libpython3.6m.so.1.0)#012#24 0x00007fc9ae3b822b PyObject_Call (libpython3.6m.so.1.0)#012#25 0x00007fc9ae4c3672 t_bootstrap (libpython3.6m.so.1.0)#012#26 0x00007fc9ae469914 pythread_wrapper (libpython3.6m.so.1.0)#012#27 0x00007fc9c0a971ca n/a (/usr/lib64/libpthread-2.28.so (deleted))
Jun 30 14:21:14 pkgs01 systemd[1]: systemd-coredump@19564-1782250-0.service: Succeeded.
Jun 30 14:23:35 pkgs01 systemd-coredump[1783019]: Process 1781599 (httpd) of user 1000 dumped core.#012#012Stack trace of thread 1781766:#012#0  0x00007fc9c050352f n/a (/usr/lib64/libc-2.28.so (deleted))#012#1  0x00007fc9c054ba2c n/a (/usr/lib64/libc-2.28.so (deleted))
Jun 30 14:23:35 pkgs01 systemd[1]: systemd-coredump@19565-1783018-0.service: Succeeded.
[root@log01 30][PROD-RDU3]#

I don’t find anything in the pagure02 logs, but notice that there are only pagure0[12].vpn.* directories (neither .iad2. nor .rdu3.).

It seems that `pkgs01.iad2` dumped core before the DC move, we just didn’t notice: ``` [root@log01 30][PROD-RDU3]# pwd /var/log/hosts/pkgs01.iad2.fedoraproject.org/2025/06/30 [root@log01 30][PROD-RDU3]# xzgrep systemd-coredump messages.log.xz | tail -n 5 Jun 30 14:21:13 pkgs01 systemd-coredump[1782251]: Removed old coredump core.httpd.1000.f6c697a505ae4311887ed864512cdd0a.1649482.1751282904000000.lz4. Jun 30 14:21:14 pkgs01 systemd-coredump[1782251]: Process 1780105 (httpd) of user 1000 dumped core.#012#012Stack trace of thread 1781734:#012#0 0x00007fc9ae3afa12 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0)#012#1 0x00007fc9ae42f2f0 fast_function (libpython3.6m.so.1.0)#012#2 0x00007fc9ae452537 call_function (libpython3.6m.so.1.0)#012#3 0x00007fc9ae453198 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)#012#4 0x00007fc9ae42f108 fast_function (libpython3.6m.so.1.0)#012#5 0x00007fc9ae452537 call_function (libpython3.6m.so.1.0)#012#6 0x00007fc9ae453198 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)#012#7 0x00007fc9ae3af0a4 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0)#012#8 0x00007fc9ae42f2f0 fast_function (libpython3.6m.so.1.0)#012#9 0x00007fc9ae452537 call_function (libpython3.6m.so.1.0)#012#10 0x00007fc9ae453dd8 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)#012#11 0x00007fc9ae3afa15 _PyEval_EvalCodeWithName (libpython3.6m.so.1.0)#012#12 0x00007fc9ae45130f function_call (libpython3.6m.so.1.0)#012#13 0x00007fc9ae3b822b PyObject_Call (libpython3.6m.so.1.0)#012#14 0x00007fc9ae455005 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)#012#15 0x00007fc9ae42f108 fast_function (libpython3.6m.so.1.0)#012#16 0x00007fc9ae452537 call_function (libpython3.6m.so.1.0)#012#17 0x00007fc9ae453198 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)#012#18 0x00007fc9ae42f108 fast_function (libpython3.6m.so.1.0)#012#19 0x00007fc9ae452537 call_function (libpython3.6m.so.1.0)#012#20 0x00007fc9ae453198 _PyEval_EvalFrameDefault (libpython3.6m.so.1.0)#012#21 0x00007fc9ae3b05d2 _PyFunction_FastCallDict (libpython3.6m.so.1.0)#012#22 0x00007fc9ae3b13ae _PyObject_FastCallDict (libpython3.6m.so.1.0)#012#23 0x00007fc9ae3c32c0 method_call (libpython3.6m.so.1.0)#012#24 0x00007fc9ae3b822b PyObject_Call (libpython3.6m.so.1.0)#012#25 0x00007fc9ae4c3672 t_bootstrap (libpython3.6m.so.1.0)#012#26 0x00007fc9ae469914 pythread_wrapper (libpython3.6m.so.1.0)#012#27 0x00007fc9c0a971ca n/a (/usr/lib64/libpthread-2.28.so (deleted)) Jun 30 14:21:14 pkgs01 systemd[1]: systemd-coredump@19564-1782250-0.service: Succeeded. Jun 30 14:23:35 pkgs01 systemd-coredump[1783019]: Process 1781599 (httpd) of user 1000 dumped core.#012#012Stack trace of thread 1781766:#012#0 0x00007fc9c050352f n/a (/usr/lib64/libc-2.28.so (deleted))#012#1 0x00007fc9c054ba2c n/a (/usr/lib64/libc-2.28.so (deleted)) Jun 30 14:23:35 pkgs01 systemd[1]: systemd-coredump@19565-1783018-0.service: Succeeded. [root@log01 30][PROD-RDU3]# ``` I don’t find anything in the pagure02 logs, but notice that there are only `pagure0[12].vpn.*` directories (neither `.iad2.` nor `.rdu3.`).
nphilipp commented 2025-07-18 17:31:54 +00:00
Author
Member
Copy link

I don’t find anything in the pagure02 logs, …

Hmm, probably also because it doesn’t have CoreDumpDirectory /tmp.

> I don’t find anything in the pagure02 logs, … Hmm, probably also because it doesn’t have `CoreDumpDirectory /tmp`.
zlopez commented 2025-08-07 12:33:54 +00:00
Owner
Copy link

@nphilipp Is this still something we need to resolve?

@nphilipp Is this still something we need to resolve?
patrikp commented 2026-02-19 17:49:50 +00:00
Member
Copy link

We ran across this as part of backlog refinement during today's infra weekly and apparently it's still happening. Does anybody know what could be the cause? Have you had a chance to look deeper @nphilipp?

We ran across this as part of backlog refinement during today's infra weekly and apparently it's still happening. Does anybody know what could be the cause? Have you had a chance to look deeper @nphilipp?
victorkoycheff commented 2026-02-25 08:45:42 +00:00
Copy link

Hi @nphilipp, @patrikp and @zlopez. I looked into the stack traces provided and they point to a mod_wsgi subinterpreter teardown crash.

I've submitted a PR to force Pagure into the global application group, which is the recommended fix to stop these specific core dumps: infra/ansible#3171

Also a link to the documentation from which I got the solution:
https://modwsgi.readthedocs.io/en/master/user-guides/application-issues.html#python-simplified-gil-state-api

Hi @nphilipp, @patrikp and @zlopez. I looked into the stack traces provided and they point to a mod_wsgi subinterpreter teardown crash. I've submitted a PR to force Pagure into the global application group, which is the recommended fix to stop these specific core dumps: https://forge.fedoraproject.org/infra/ansible/pulls/3171 Also a link to the documentation from which I got the solution: https://modwsgi.readthedocs.io/en/master/user-guides/application-issues.html#python-simplified-gil-state-api
kevin commented 2026-03-11 19:07:16 +00:00
Owner
Copy link

I merged the pr and deployed it.

I will keep an eye on it today and see how it does.

Thanks for the pr @victorkoycheff !

I merged the pr and deployed it. I will keep an eye on it today and see how it does. Thanks for the pr @victorkoycheff !
victorkoycheff commented 2026-03-11 19:08:46 +00:00
Copy link

Always a pleasure 🫡

Let's see

Always a pleasure 🫡 Let's see
kevin commented 2026-03-11 23:53:38 +00:00
Owner
Copy link

I've seen 0 coredumps so far, so I think this is solved. Thanks again!

I've seen 0 coredumps so far, so I think this is solved. Thanks again!
kevin closed this issue 2026-03-11 23:53:38 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
announcement

   
anubis

Issues related to anubis

  
authentication

Authentication related issues

  
aws

Issues related to Amazon Web Services

  
backlog

Tasks that will take longer then ~4 hours to do

  
blocked

Issue is blocked

  
bodhi

Issues with Bodhi

  
ci

Continuous Integration

  
cloud

Items in FedoraInfraCloud or COmmunishift

  
communishift

issues/questions about os.edorainfracloud.org ( Community OpenShift)

  
copr

COPR system (run by other team)

  
database

Issues related to databases

  
day-to-day

These are small self contained tasks that are part of normal operations, but are async from sprint planning.

  
dc-move

Items related to datacenter move

  
deprecated

Tools which Infrastructure may not be able to fix

  
dev

Tasks requiring some dev work

  
discourse

outsourced tool

  
dns

Changes or problems with DNS

  
downloads

rsync,dl.fp.o,AWS,torrents

  
easyfix

   
epel

Issues relating to epel

  
firmitas

   
forgejo_migration

Request migration to the new forge

  
Gain
High
task that gets us a lot / is important / makes many people happy

  
Gain
Low
tasks that gets us little / isn't very important / makes only 1 person or few happy

  
Gain
Medium
task that gets us some / is somewhat important / makes some people happy

  
gitlab

   
greenwave

Issues with greenwave

  
hardware

Problem is with Fedora Infrastructure Owned/Leased Hardware

  
help wanted

   
high-trouble

task that is complex/long/difficult

  
koji

Issues with the build system

  
koschei

Items needing Koschei team

  
lists

issues with lists

  
low-trouble

task that is easy/short

  
medium-trouble

task that is medium complex / difficult

  
mirrorlists

Way to tell systems where downloads are

  
monitoring

   
Needs investigation

   
odcs

On Demand Compose Service issues

  
OpenShift

Issues with OpenShift

  
ops

ops problem now...

  
outage

   
packager_workflow_blocker

Blocks the packager workflow

  
pagure

Issues with pagure.io

  
permissions

Items which need permissions set in outside tools

  
Priority
Needs Review
Priority of Needs Review

  
Priority
Next Meeting
Priority of Next Meeting

  
Priority
🔥 URGENT 🔥
Priority of 🔥 URGENT 🔥

  
Priority
Waiting on Assignee
Priority of Waiting on Assignee

  
Priority
Waiting on External
Priority of Waiting on External

  
Priority
Waiting on Reporter
Priority of Waiting on Reporter

  
rabbitmq

issues with the message broker

  
release-monitoring

release-monitoring.org (Anitya)

  
releng

For problems which should be in the releng ticket system

  
request-for-resources

   
s390x

The secondary architecture hardware and problems related with it

  
security

   
SMTP

Email issues outside of lists (DMARC, SPF, aliases)

  
sprint-0

   
sprint-1

   
src.fp.o

Issues related to src.fedoraproject.org

  
staging

Issues in staging

  
unfreeze

Patches to be applied after freeze is lifted

  
waiverdb

Issues with waiverdb

  
websites-general

General items with a website focus

  
wiki

Issues with the Wiki

  
Backlog Status
Needs Review
Work items that have been created or updated but need review, refinement, or approval before they can be considered ready for sprint planning.

  
Backlog Status
Ready
Work items that have been reviewed, refined, and are ready to be committed to a sprint. These items have clear requirements and can be worked on immediately when capacity is available.

  
chore

Work that needs to be done but doesn't add new functionality or fix bugs. Maintenance tasks that keep the system running smoothly and the codebase clean.

  
documentation

Work focused on creating, updating, or improving documentation for users, developers, or system administrators.

  
points
01
A task that is simple, small, and has very little uncertainty. It should be easy enough for any team member to pick up and complete quickly, without a lot of discussion or research.

  
points
02
This is a task that is slightly more complex or larger than a "1." It might involve two or three simple steps, but the outcome is still very clear and there's little uncertainty.

  
points
03
Medium-sized task with some noticeable complexity. It might require a bit more thought or collaboration. The steps are well-defined, but it's not a trivial change.

  
points
05
A significant piece of work. It has more complexity and a higher degree of uncertainty than smaller tasks. It might require some research, a design discussion, or collaboration across multiple teams.

  
points
08
Complex task with a lot of uncertainty. It's often the largest size that a team will consider for a single sprint. It likely requires a detailed breakdown and multiple team members to complete.

  
points
13
Any task that feels like it's a "13" or more is considered an Epic. These are too large and uncertain to be worked on in a single sprint and should be broken down.

  
Priority
High
This is for work that is critical and needs to be addressed in the next sprint. It's often for urgent bugs, important features, or a major piece of technical debt that is blocking other work.

  
Priority
Low
This is for work that is good to do but not urgent. It's a candidate for being worked on when there's extra time or when it rises in importance. It can stay in the backlog for a longer period.

  
Priority
Medium
This is for important work that should be done soon, but isn't as urgent as a high-priority item. It will likely be considered for the next 1-2 sprints.

  
Sprint Status
Blocked
Work items that cannot progress due to external dependencies, blockers, or issues that need resolution.

  
Sprint Status
Done
Work items that have been completed, reviewed, and integrated. No further work is needed within the current sprint.

  
Sprint Status
In Progress
Work items currently being worked on by a team member. Active development, research, or implementation is happening.

  
Sprint Status
Review
Work items that are complete but need review, testing, or approval before being marked as done.

  
Sprint Status
To Do
Work items that are committed to the sprint but haven't been started yet. These items are ready to be picked up by team members.

  
Technical Debt

Work that addresses shortcuts, workarounds, or suboptimal implementations that were done previously to meet deadlines. These items improve code quality, maintainability, or performance without adding new features.

  
Work Item
Bug
Issues where the system is not working as intended or designed. Bugs represent deviations from expected behavior.

  
Work Item
Epic
Large initiatives that are broken down into multiple smaller work items (user stories, tasks, bugs). Epics represent significant features or projects that span multiple sprints.

  
Work Item
Spike
Time-boxed research, investigation, or proof-of-concept work to reduce uncertainty or explore solutions. Spikes help inform future development decisions.

  
Work Item
Task
Used to break down larger user stories into smaller, manageable implementation pieces. They focus on the specific technical steps needed to deliver the user story's value.

  
Work Item
User Story
User-facing features or functionality that can be framed from a user perspective, describing what someone wants to accomplish and why.

No labels
announcement
anubis
authentication
aws
backlog
blocked
bodhi
ci
cloud
communishift
copr
database
day-to-day
dc-move
deprecated
dev
discourse
dns
downloads
easyfix
epel
firmitas
forgejo_migration
Gain
High
Gain
Low
Gain
Medium
gitlab
greenwave
hardware
help wanted
high-trouble
koji
koschei
lists
low-trouble
medium-trouble
mirrorlists
monitoring
Needs investigation
odcs
OpenShift
ops
outage
packager_workflow_blocker
pagure
permissions
Priority
Needs Review
Priority
Next Meeting
Priority
🔥 URGENT 🔥
Priority
Waiting on Assignee
Priority
Waiting on External
Priority
Waiting on Reporter
rabbitmq
release-monitoring
releng
request-for-resources
s390x
security
SMTP
sprint-0
sprint-1
src.fp.o
staging
unfreeze
waiverdb
websites-general
wiki
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
5 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
infra/tickets#12670
No description provided.