infra/tickets

Fork 0

Pagure staging is not accessible #13128

New issue

Closed

opened 2026-02-09 06:10:06 +00:00 by amedvede · 6 comments

amedvede commented

2026-02-09 06:10:06 +00:00

Description of request

When you trying to access the page it tells Service Unavailable The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
There is no info on the status page about it, and zabbix shows that everything is alright.

### Description of request When you trying to access the [page](https://stg.pagure.io/dashboard/projects) it tells `Service Unavailable The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.` There is no info on the status page about it, and [zabbix](https://zabbix.fedoraproject.org/zabbix.php?name=&evaltype=0&tags%5B0%5D%5Btag%5D=&tags%5B0%5D%5Boperator%5D=0&tags%5B0%5D%5Bvalue%5D=&show_tags=3&tag_name_format=0&tag_priority=&state=-1&filter_name=&filter_show_counter=0&filter_custom_time=0&sort=name&sortorder=ASC&show_details=0&action=latest.view&hostids%5B%5D=11026) shows that everything is alright.

amedvede referenced this issue from releng/tickets

2026-02-09 06:36:24 +00:00

Test fedpkg request-unretirement functionality #13163

zlopez added the

labels

2026-02-09 11:43:30 +00:00

zlopez self-assigned this

2026-02-09 14:20:52 +00:00

zlopez commented

2026-02-09 14:33:39 +00:00

Owner

So I'm checking what is a problem here and the httpd is running OK, but the Anubis is not running a failing on start. I also noticed some kerberos error, but that was resolved by reset of sssd.

I will look into anubis and see if I can figure out why it's not running.

So I'm checking what is a problem here and the `httpd` is running OK, but the `Anubis` is not running a failing on start. I also noticed some kerberos error, but that was resolved by reset of `sssd`. I will look into anubis and see if I can figure out why it's not running.

zlopez commented

2026-02-09 14:49:39 +00:00

Owner

So from what I was able to find, this is happening till 30th January and here is the output from anubis systemd unit.

Feb 09 14:27:44 pagure-stg02.fedoraproject.org systemd[1]: Starting Anubis Container...
Feb 09 14:27:44 pagure-stg02.fedoraproject.org podman[884489]: time="2026-02-09T14:27:44Z" level=error msg="invalid internal status, try resetting the pause process with \"/usr/bin/podman system migrate\": could not find any running process: no such process"
Feb 09 14:27:44 pagure-stg02.fedoraproject.org podman[884502]: time="2026-02-09T14:27:44Z" level=error msg="invalid internal status, try resetting the pause process with \"/usr/bin/podman system migrate\": could not find any running process: no such process"
Feb 09 14:27:44 pagure-stg02.fedoraproject.org systemd[1]: Started Anubis Container.
Feb 09 14:27:44 pagure-stg02.fedoraproject.org podman[884515]: time="2026-02-09T14:27:44Z" level=error msg="invalid internal status, try resetting the pause process with \"/usr/bin/podman system migrate\": could not find any running process: no such process"
Feb 09 14:27:44 pagure-stg02.fedoraproject.org systemd[1]: anubis.service: Main process exited, code=exited, status=1/FAILURE
Feb 09 14:27:44 pagure-stg02.fedoraproject.org systemd[1]: anubis.service: Failed with result 'exit-code'.

I tried to redeploy it by running ansible-playbook /srv/web/infra/ansible/playbooks/groups/pagure.yml -l pagure_stg -t anubis, but it didn't help.

So from what I was able to find, this is happening till 30th January and here is the output from anubis systemd unit. ``` Feb 09 14:27:44 pagure-stg02.fedoraproject.org systemd[1]: Starting Anubis Container... Feb 09 14:27:44 pagure-stg02.fedoraproject.org podman[884489]: time="2026-02-09T14:27:44Z" level=error msg="invalid internal status, try resetting the pause process with \"/usr/bin/podman system migrate\": could not find any running process: no such process" Feb 09 14:27:44 pagure-stg02.fedoraproject.org podman[884502]: time="2026-02-09T14:27:44Z" level=error msg="invalid internal status, try resetting the pause process with \"/usr/bin/podman system migrate\": could not find any running process: no such process" Feb 09 14:27:44 pagure-stg02.fedoraproject.org systemd[1]: Started Anubis Container. Feb 09 14:27:44 pagure-stg02.fedoraproject.org podman[884515]: time="2026-02-09T14:27:44Z" level=error msg="invalid internal status, try resetting the pause process with \"/usr/bin/podman system migrate\": could not find any running process: no such process" Feb 09 14:27:44 pagure-stg02.fedoraproject.org systemd[1]: anubis.service: Main process exited, code=exited, status=1/FAILURE Feb 09 14:27:44 pagure-stg02.fedoraproject.org systemd[1]: anubis.service: Failed with result 'exit-code'. ``` I tried to redeploy it by running `ansible-playbook /srv/web/infra/ansible/playbooks/groups/pagure.yml -l pagure_stg -t anubis`, but it didn't help.

zlopez commented

2026-02-09 14:56:17 +00:00

Owner

Tried to reboot the machine (in case some update is hanging there), but that didn't helped as well.

kevin commented

2026-02-09 15:44:10 +00:00

Owner

Something was strange with the container image.

I did a new pull:

[root@pagure-stg02 tmp]# sudo -u anubis podman pull ghcr.io/techarohq/anubis:latest               
Trying to pull ghcr.io/techarohq/anubis:latest...                                                 
Getting image source signatures                                                                   
Copying blob 267f0d71ccc2 done   |                                                                
Copying blob d270249e0778 done   |                                                                
Copying blob 250c06f7c38e skipped: already exists                                                 
Copying config 0807bcc631 done   | 
Writing manifest to image destination
0807bcc631f077563e59f108a79a92ba1e09cc7772049c3cd6c27b568887ef0f

And after that just doing a 'systemctl restart anubis' worked?

So, I am unsure what happened here.

Something was strange with the container image. I did a new pull: ``` [root@pagure-stg02 tmp]# sudo -u anubis podman pull ghcr.io/techarohq/anubis:latest Trying to pull ghcr.io/techarohq/anubis:latest... Getting image source signatures Copying blob 267f0d71ccc2 done | Copying blob d270249e0778 done | Copying blob 250c06f7c38e skipped: already exists Copying config 0807bcc631 done | Writing manifest to image destination 0807bcc631f077563e59f108a79a92ba1e09cc7772049c3cd6c27b568887ef0f ``` And after that just doing a 'systemctl restart anubis' worked? So, I am unsure what happened here.

zlopez commented

2026-02-10 08:58:49 +00:00

Owner

I tried that as well, but not as anubis used, so maybe that was the problem.

Anyway the issue is now resolved, closing this as fixed.

I tried that as well, but not as anubis used, so maybe that was the problem. Anyway the issue is now resolved, closing this as fixed.

zlopez closed this issue

2026-02-10 08:58:49 +00:00

zlopez added this to the Sprint ( 2026-02-09 to 2026-02-22 ) project

2026-02-10 08:58:55 +00:00

amedvede commented

2026-02-10 09:08:26 +00:00