infra/tickets

Fork 1

New connections to staging rabbitmq fail if /etc/fedora-messaging/stg-cacert.pem is used as the CA cert as it seems to only use the first cert in the file #13422

New issue

Closed

opened 2026-06-20 10:17:45 +00:00 by adamwill · 2 comments

adamwill commented

2026-06-20 10:17:45 +00:00

Member

Description of request

I woke up today and found all my consumers that listen on the staging bus were failing to connect to the server after a cron job that ran at 04:00 restarted them.

On further investigation, it seems like the /etc/fedora-messaging/stg-cacert.pem file, which I use as the ca_cert file in the messaging config (it's used for this in other configs as well, not just mine) contains both the "old" CA cert and the "new" CA cert (in relation to some kinda cert refresh @abompard is doing). rabbitmq seems to have been flipped over to using certs signed by the new CA cert yesterday(?). /etc/fedora-messaging/stg-cacert.pem comes from the fedora-messaging package.

If I create a /etc/fedora-messaging/stg-cacert-newonly.pem with only the new CA cert in it, change my consumer config to specify that file as the ca_cert, and restart the consumer, it works. So it seems like possibly fedora-messaging (or whatever exactly is implementing the connection from the client to the rabbitmq server) is only considering the first CA cert in the file? This kinda defeats the purpose of the 'combined' file unfortunately :(

### Description of request I woke up today and found all my consumers that listen on the staging bus were failing to connect to the server after a cron job that ran at 04:00 restarted them. On further investigation, it *seems* like the `/etc/fedora-messaging/stg-cacert.pem` file, which I use as the `ca_cert` file in the messaging config (it's used for this in other configs as well, not just mine) contains both the "old" CA cert and the "new" CA cert (in relation to some kinda cert refresh @abompard is doing). rabbitmq seems to have been flipped over to using certs signed by the new CA cert yesterday(?). `/etc/fedora-messaging/stg-cacert.pem` comes from the fedora-messaging package. If I create a `/etc/fedora-messaging/stg-cacert-newonly.pem` with *only* the new CA cert in it, change my consumer config to specify *that* file as the `ca_cert`, and restart the consumer, it works. So it seems like possibly fedora-messaging (or whatever exactly is implementing the connection from the client to the rabbitmq server) is only considering the *first* CA cert in the file? This kinda defeats the purpose of the 'combined' file unfortunately :(

adamwill referenced this issue from a commit

2026-06-20 12:40:58 +00:00

openQA etc: use a single-cert CA cert file for staging rabbitmq (#13422)

adamwill referenced this issue from infra/ansible

2026-06-20 12:41:21 +00:00

openQA etc: use a single-cert CA cert file for staging rabbitmq (#13422) #3432

abompard self-assigned this

2026-06-22 08:23:53 +00:00

adamwill commented

2026-06-22 08:52:58 +00:00

Author

Member

It looks like resultsdb staging is running into this too now:

[Mon Jun 22 08:32:55.319564 2026] [wsgi:error] [pid 4:tid 50] [2026-06-22 08:32:55] [4] [ERROR] pika.adapters.twisted_connection: connection_lost: Error([('SSL routines', '', 'certificate verify failed')])

and it's constantly failing to publish messages:

[Mon Jun 22 08:45:55.552606 2026] [wsgi:error] [pid 4:tid 36] [remote 10.128.2.2:52016] [2026-06-22 08:45:55,552] DEBUG in common: Preparing to publish message for result id 7190812
[Mon Jun 22 08:46:25.554024 2026] [wsgi:error] [pid 4:tid 36] [remote 10.128.2.2:52016] [2026-06-22 08:46:25,553] ERROR in messaging: Timeout publishing message a154b0a6-c826-4fe0-b2e3-f57a141d4d60
[Mon Jun 22 08:46:25.554054 2026] [wsgi:error] [pid 4:tid 36] [remote 10.128.2.2:52016] [2026-06-22 08:46:25] [4] [ERROR] resultsdb.messaging: Timeout publishing message a154b0a6-c826-4fe0-b2e3-f57a141d4d60
[Mon Jun 22 08:46:25.557238 2026] [wsgi:error] [pid 4:tid 38] [remote 10.131.0.2:39756] [2026-06-22 08:46:25,557] DEBUG in common: Preparing to publish message for result id 7190813
[Mon Jun 22 08:46:55.558749 2026] [wsgi:error] [pid 4:tid 38] [remote 10.131.0.2:39756] [2026-06-22 08:46:55,558] ERROR in messaging: Timeout publishing message 097805b3-a077-4082-9154-de97ebc2de25
[Mon Jun 22 08:46:55.558831 2026] [wsgi:error] [pid 4:tid 38] [remote 10.131.0.2:39756] [2026-06-22 08:46:55] [4] [ERROR] resultsdb.messaging: Timeout publishing message 097805b3-a077-4082-9154-de97ebc2de25

etc etc. I rsh'ed into the pod and sure enough the configured fedora-messaging CA cert is the combined one. Should I twiddle it to use the single CA cert or shall I leave it to you @abompard ?

It looks like resultsdb staging is running into this too now: ``` [Mon Jun 22 08:32:55.319564 2026] [wsgi:error] [pid 4:tid 50] [2026-06-22 08:32:55] [4] [ERROR] pika.adapters.twisted_connection: connection_lost: Error([('SSL routines', '', 'certificate verify failed')]) ``` and it's constantly failing to publish messages: ``` [Mon Jun 22 08:45:55.552606 2026] [wsgi:error] [pid 4:tid 36] [remote 10.128.2.2:52016] [2026-06-22 08:45:55,552] DEBUG in common: Preparing to publish message for result id 7190812 [Mon Jun 22 08:46:25.554024 2026] [wsgi:error] [pid 4:tid 36] [remote 10.128.2.2:52016] [2026-06-22 08:46:25,553] ERROR in messaging: Timeout publishing message a154b0a6-c826-4fe0-b2e3-f57a141d4d60 [Mon Jun 22 08:46:25.554054 2026] [wsgi:error] [pid 4:tid 36] [remote 10.128.2.2:52016] [2026-06-22 08:46:25] [4] [ERROR] resultsdb.messaging: Timeout publishing message a154b0a6-c826-4fe0-b2e3-f57a141d4d60 [Mon Jun 22 08:46:25.557238 2026] [wsgi:error] [pid 4:tid 38] [remote 10.131.0.2:39756] [2026-06-22 08:46:25,557] DEBUG in common: Preparing to publish message for result id 7190813 [Mon Jun 22 08:46:55.558749 2026] [wsgi:error] [pid 4:tid 38] [remote 10.131.0.2:39756] [2026-06-22 08:46:55,558] ERROR in messaging: Timeout publishing message 097805b3-a077-4082-9154-de97ebc2de25 [Mon Jun 22 08:46:55.558831 2026] [wsgi:error] [pid 4:tid 38] [remote 10.131.0.2:39756] [2026-06-22 08:46:55] [4] [ERROR] resultsdb.messaging: Timeout publishing message 097805b3-a077-4082-9154-de97ebc2de25 ``` etc etc. I rsh'ed into the pod and sure enough the configured fedora-messaging CA cert is the combined one. Should I twiddle it to use the single CA cert or shall I leave it to you @abompard ?

adamwill referenced this issue from a commit

2026-06-22 14:31:02 +00:00

Drop use of ca-combined.crt (#13422)

adamwill referenced this issue from infra/ansible

2026-06-22 14:31:18 +00:00

Drop use of ca-combined.crt (#13422) #3433

adamwill referenced this issue from a commit

2026-06-22 14:31:42 +00:00

Drop use of ca-combined.crt (#13422)

smoliicek added the

labels

2026-06-22 19:04:51 +00:00

abompard commented

2026-06-23 16:37:33 +00:00

Member

I've fixed the issue in the fedora-messaging library, released a new version and built RPMs. I've restored the old RabbitMQ server certficates until the fedora-messaging update is wildly adopted and we can switch the server certs again.

Thanks a lot for the detailed report, it helped a great deal.

abompard closed this issue

2026-06-23 16:37:33 +00:00

adamwill referenced this issue from quality/fedora_openqa

2026-06-24 11:12:28 +00:00

Support running tests for MRs in dist-git #104