infra/tickets
24
11
Fork 1
Code Issues 77 Projects Wiki Activity Actions

RHEL10: migrate ipa clusters to rhel10 #13382

New issue
Closed
opened 2026-06-01 21:55:25 +00:00 by kevin · 31 comments
kevin commented 2026-06-01 21:55:25 +00:00
Owner
Copy link

Description of request

We should migrate our rhel9 ipa clusters (stg and then prod) to rhel10.

"9", ipa01.rdu3.fedoraproject.org
"9", ipa01.stg.rdu3.fedoraproject.org
"9", ipa02.rdu3.fedoraproject.org
"9", ipa02.stg.rdu3.fedoraproject.org
"9", ipa03.rdu3.fedoraproject.org
"9", ipa03.stg.rdu3.fedoraproject.org

Basically this involves installing a new rhel10 ipa server, adding it as a replica in the cluster, making it the ca issuer and other items, then retiring the old one and adding more rhel10 replicas until the cluster is all rhel10.

https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/10/html-single/migrating_to_identity_management_on_rhel_10/index

@zlopez did our rhel8->9 move I think. Perhaps he would be willing to work on 9->10?

We will also need to make sure noggin works ok with things in staging before moving on to prod.

### Description of request We should migrate our rhel9 ipa clusters (stg and then prod) to rhel10. "9", ipa01.rdu3.fedoraproject.org "9", ipa01.stg.rdu3.fedoraproject.org "9", ipa02.rdu3.fedoraproject.org "9", ipa02.stg.rdu3.fedoraproject.org "9", ipa03.rdu3.fedoraproject.org "9", ipa03.stg.rdu3.fedoraproject.org Basically this involves installing a new rhel10 ipa server, adding it as a replica in the cluster, making it the ca issuer and other items, then retiring the old one and adding more rhel10 replicas until the cluster is all rhel10. https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/10/html-single/migrating_to_identity_management_on_rhel_10/index @zlopez did our rhel8->9 move I think. Perhaps he would be willing to work on 9->10? We will also need to make sure noggin works ok with things in staging before moving on to prod.
zlopez commented 2026-06-02 07:37:43 +00:00
Owner
Copy link

Yes, I did the previous move. It was painful as I made a few mistakes, but we get it working in the end and the ansible deployment was improved thanks to it.

I can do it for 9->10 as well. I should still have my notes from the previous move as well.

Yes, I did the previous move. It was painful as I made a few mistakes, but we get it working in the end and the ansible deployment was improved thanks to it. I can do it for 9->10 as well. I should still have my notes from the previous move as well.
kevin commented 2026-06-02 16:32:03 +00:00
Author
Owner
Copy link

That would be great if you could!

Feel free to add it to whatever sprint you feel you can do it in.

That would be great if you could! Feel free to add it to whatever sprint you feel you can do it in.
zlopez commented 2026-06-26 09:05:08 +00:00
Owner
Copy link

Currently working on ipa03.stg and trying to get it to RHEL10. The problem I hit right now is that there is no python3-freeipa package on EPEL10. It's used by some scripts that are being run around IPA, like clearing the stage users.

There is a bug open to request it, but no progress yet. I will look into that and see if I can build it by myself in COPR.

Currently working on ipa03.stg and trying to get it to RHEL10. The problem I hit right now is that there is no python3-freeipa package on EPEL10. It's used by some scripts that are being run around IPA, like clearing the stage users. There is a [bug](https://bugzilla.redhat.com/show_bug.cgi?id=2482962) open to request it, but no progress yet. I will look into that and see if I can build it by myself in COPR.
zlopez commented 2026-06-26 09:44:28 +00:00
Owner
Copy link

Found out the package was already built for epel10.2 and epel10.3 https://koji.fedoraproject.org/koji/packageinfo?packageID=31158, but it's not in the EPEL10 repository. Not sure what is happening here.

There is no bodhi update for EPEL10 build, so I try to submit one and see if that will work https://bodhi.fedoraproject.org/updates/?packages=python-freeipa&page=1. But let's ask the maintainer first.

Found out the package was already built for epel10.2 and epel10.3 https://koji.fedoraproject.org/koji/packageinfo?packageID=31158, but it's not in the EPEL10 repository. Not sure what is happening here. There is no bodhi update for EPEL10 build, so I try to submit one and see if that will work https://bodhi.fedoraproject.org/updates/?packages=python-freeipa&page=1. But let's ask the maintainer first.
zlopez commented 2026-07-01 12:48:22 +00:00
Owner
Copy link

So I built few missing packages for RHEL10 for IPA in epel10-infra tag:

  • journal-to-fedora-messaging
    • python-journal-to-fedora-messaging-messages
    • python-pytest-twisted

But it's failing during install on GPG check. I'm not sure what I missed

Fedora Infrastructure tag 10 - x86_64                                                                                                                                                                       163 kB/s | 1.6 kB     00:00    
Importing GPG key 0x47DD8EF9:
 Userid     : ""
 Fingerprint: 1D69 8E02 822F 64CA 1ACB 9A32 8059 815E 47DD 8EF9
 From       : https://infrastructure.fedoraproject.org/repo/infra/RPM-GPG-KEY-INFRA-TAGS
Is this ok [y/N]: y
error: Certificate 8059815E47DD8EF9:
  Policy rejects 8059815E47DD8EF9: No binding signature at time 2026-07-01T12:42:54Z
Key import failed (code 2). Failing package is: journal-to-fedora-messaging-1.0.0-1.el10_2.noarch
 GPG Keys are configured as: https://infrastructure.fedoraproject.org/repo/infra/RPM-GPG-KEY-INFRA-TAGS
Public key for python3-journal-to-fedora-messaging-messages-1.0.1-1.el10_2.noarch.rpm is not installed. Failing package is: python3-journal-to-fedora-messaging-messages-1.0.1-1.el10_2.noarch
 GPG Keys are configured as: https://infrastructure.fedoraproject.org/repo/infra/RPM-GPG-KEY-INFRA-TAGS
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: GPG check FAILED
So I built few missing packages for RHEL10 for IPA in epel10-infra tag: * journal-to-fedora-messaging * python-journal-to-fedora-messaging-messages * python-pytest-twisted But it's failing during install on GPG check. I'm not sure what I missed ``` Fedora Infrastructure tag 10 - x86_64 163 kB/s | 1.6 kB 00:00 Importing GPG key 0x47DD8EF9: Userid : "" Fingerprint: 1D69 8E02 822F 64CA 1ACB 9A32 8059 815E 47DD 8EF9 From : https://infrastructure.fedoraproject.org/repo/infra/RPM-GPG-KEY-INFRA-TAGS Is this ok [y/N]: y error: Certificate 8059815E47DD8EF9: Policy rejects 8059815E47DD8EF9: No binding signature at time 2026-07-01T12:42:54Z Key import failed (code 2). Failing package is: journal-to-fedora-messaging-1.0.0-1.el10_2.noarch GPG Keys are configured as: https://infrastructure.fedoraproject.org/repo/infra/RPM-GPG-KEY-INFRA-TAGS Public key for python3-journal-to-fedora-messaging-messages-1.0.1-1.el10_2.noarch.rpm is not installed. Failing package is: python3-journal-to-fedora-messaging-messages-1.0.1-1.el10_2.noarch GPG Keys are configured as: https://infrastructure.fedoraproject.org/repo/infra/RPM-GPG-KEY-INFRA-TAGS The downloaded packages were saved in cache until the next successful transaction. You can remove cached packages by executing 'dnf clean packages'. Error: GPG check FAILED ```
kevin commented 2026-07-01 18:57:30 +00:00
Author
Owner
Copy link

Sad trombone... that key has some sha1 still in it. ;(

I'll make a new one, but... we either need to have a seperate public key link in the repo file for rhel10, or resign everything.
I guess I'd prefer to just change epel10-infra. Will try and do this later today.

Sad trombone... that key has some sha1 still in it. ;( I'll make a new one, but... we either need to have a seperate public key link in the repo file for rhel10, or resign everything. I guess I'd prefer to just change epel10-infra. Will try and do this later today.
kevin commented 2026-07-02 01:05:48 +00:00
Author
Owner
Copy link

I didn't get to it today. ;( I can try soon... I'm out tomorrow, holiday on friday and out monday tho.

If someone like @jnsamyak wants to make the key in sigul, I'd call it something like 'fedora-infra-20260702' or whatever (so it's different from the existing one) Then it need autopen to have access to it and robosignatory config adjusted to use it for epel10-infra builds and finally a seperate file with the public key for the repo file.

I didn't get to it today. ;( I can try soon... I'm out tomorrow, holiday on friday and out monday tho. If someone like @jnsamyak wants to make the key in sigul, I'd call it something like 'fedora-infra-20260702' or whatever (so it's different from the existing one) Then it need autopen to have access to it and robosignatory config adjusted to use it for epel10-infra builds and finally a seperate file with the public key for the repo file.
zlopez commented 2026-07-02 07:44:23 +00:00
Owner
Copy link

No worries, I'm out on Friday and Monday too. I will work on something else in the meantime.

No worries, I'm out on Friday and Monday too. I will work on something else in the meantime.
kevin commented 2026-07-09 22:36:23 +00:00
Author
Owner
Copy link

ok. I have rolled out a new happier rpm signing key for epel10-infra/stg... I was able to install those packages ok on ipa03.stg.

Let me know if you spot any further issues with it.

ok. I have rolled out a new happier rpm signing key for epel10-infra/stg... I was able to install those packages ok on ipa03.stg. Let me know if you spot any further issues with it.
zlopez commented 2026-07-10 06:53:30 +00:00
Owner
Copy link

Thanks will try to work on this today.

Thanks will try to work on this today.
jnsamyak commented 2026-07-10 07:11:58 +00:00
Member
Copy link

Hello, ah, sad again, my notification settings weren't up to mark (fixed now), I could have chimed in, it seems like exciting/annoying work. But tag me if there is something I can help with 🗡️

Hello, ah, sad again, my notification settings weren't up to mark (fixed now), I could have chimed in, it seems like exciting/annoying work. But tag me if there is something I can help with 🗡️
zlopez commented 2026-07-10 14:45:59 +00:00
Owner
Copy link

ipa03.stg is now migrated, same for ipa02.stg.

I encountered some strange issue with ssh when reinstalling ipa01.stg. The ssh to machine works, but the ansible has problem reaching to it, this happens during running the ipa-replica-install as the ansible is stuck when this task finishes. Cancelling the task and running it again fails when running

ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no
 -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/root/.ansible/cp/1450f539eb"' ipa01.stg.rdu3.fedoraproject.org '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''

I wasn't able to figure out why today and as the authentication on staging works in current state I will continue on this on Monday.

ipa03.stg is now migrated, same for ipa02.stg. I encountered some strange issue with ssh when reinstalling ipa01.stg. The ssh to machine works, but the ansible has problem reaching to it, this happens during running the `ipa-replica-install` as the ansible is stuck when this task finishes. Cancelling the task and running it again fails when running ``` ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/root/.ansible/cp/1450f539eb"' ipa01.stg.rdu3.fedoraproject.org '/bin/sh -c '"'"'echo ~root && sleep 0'"'"'' ``` I wasn't able to figure out why today and as the authentication on staging works in current state I will continue on this on Monday.
zlopez commented 2026-07-10 15:57:00 +00:00
Owner
Copy link

So the IPA on staging is all on RHEL10 now, I'm not sure why the ssh had issues, but it just took much longer to finish.

So the IPA on staging is all on RHEL10 now, I'm not sure why the ssh had issues, but it just took much longer to finish.
kevin commented 2026-07-10 15:57:45 +00:00
Author
Owner
Copy link

We have seen some odd selinux issues around ssh sometimes on newly installed hosts... but only fedora ones, not rhel that I can recall. ;( Happy to help debug this with you...

We have seen some odd selinux issues around ssh sometimes on newly installed hosts... but only fedora ones, not rhel that I can recall. ;( Happy to help debug this with you...
zlopez commented 2026-07-10 16:06:53 +00:00
Owner
Copy link

It works now, so there is no need and the RHEL 10 upgrade was successful on staging :-)

It works now, so there is no need and the RHEL 10 upgrade was successful on staging :-)
zlopez commented 2026-07-17 13:12:49 +00:00
Owner
Copy link

Currently working on ipa03 on production and encountered these problems:

  1. Disconnected topology - after some investigation found leftovers replication agreements on ipa02 LDAP, after re-initialization of ipa02 from ipa01 those were fixed and the installation of ipa03 didn't failed on existing replication agreement anymore
  2. Can't create dogtag principal - I found orphaned principal on ipa01 and removed it, but that didn't help. Here is the error I'm currently trying to resolve
2026-07-14T15:02:10Z DEBUG Creating principal
2026-07-14T15:02:10Z DEBUG Starting external process
2026-07-14T15:02:10Z DEBUG args=['/usr/sbin/kadmin.local', '-q', 'addprinc -randkey dogtag/ipa03.rdu3.fedoraproject.org@FEDORAPROJECT.ORG', '-x', 'ipa-setup-override-restrictions']
2026-07-14T15:02:20Z DEBUG Process finished, return code=0
2026-07-14T15:02:20Z DEBUG stdout=Authenticating as principal host/admin@FEDORAPROJECT.ORG with password.

2026-07-14T15:02:20Z DEBUG stderr=No policy specified for dogtag/ipa03.rdu3.fedoraproject.org@FEDORAPROJECT.ORG; defaulting to no policy
add_principal: Server error while creating "dogtag/ipa03.rdu3.fedoraproject.org@FEDORAPROJECT.ORG".

2026-07-14T15:02:20Z DEBUG Retrieving keytab
2026-07-14T15:02:20Z DEBUG Starting external process
2026-07-14T15:02:20Z DEBUG args=['/usr/sbin/kadmin.local', '-q', 'ktadd -k /etc/pki/pki-tomcat/dogtag.keytab dogtag/ipa03.rdu3.fedoraproject.org@FEDORAPROJECT.ORG', '-x', 'ipa-setup-override-restrictions']
2026-07-14T15:02:20Z DEBUG Process finished, return code=0
2026-07-14T15:02:20Z DEBUG stdout=Authenticating as principal host/admin@FEDORAPROJECT.ORG with password.

2026-07-14T15:02:20Z DEBUG stderr=kadmin.local: Principal dogtag/ipa03.rdu3.fedoraproject.org@FEDORAPROJECT.ORG does not exist.

2026-07-14T15:02:20Z DEBUG   File "/usr/lib/python3.12/site-packages/ipapython/admintool.py", line 219, in execute
    return_value = self.run()
                   ^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/ipapython/install/cli.py", line 343, in run
    return cfgr.run()
           ^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 360, in run
    return self.execute()
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 386, in execute
    for rval in self._executor():
                ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 435, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 468, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 458, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.12/site-packages/six.py", line 719, in reraise
    raise value
  File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 425, in __runner
    step()
  File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 419, in step_next
    return next(self.__gen)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python3.12/site-packages/six.py", line 719, in reraise
    raise value
  File "/usr/lib/python3.12/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
            ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 663, in _configure
    next(executor)
  File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 435, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 468, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 526, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 458, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.12/site-packages/six.py", line 719, in reraise
    raise value
  File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 523, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 458, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.12/site-packages/six.py", line 719, in reraise
    raise value
  File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 425, in __runner
    step()
  File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 419, in step_next
    return next(self.__gen)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python3.12/site-packages/six.py", line 719, in reraise
    raise value
  File "/usr/lib/python3.12/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
            ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/ipapython/install/common.py", line 65, in _install
    for unused in self._installer(self.parent):
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/ipaserver/install/server/__init__.py", line 702, in main
    replica_install(self)
  File "/usr/lib/python3.12/site-packages/ipaserver/install/server/replicainstall.py", line 386, in decorated
    func(installer)
  File "/usr/lib/python3.12/site-packages/ipaserver/install/server/replicainstall.py", line 1434, in install
    ca.install(False, config, options, custodia=custodia)
  File "/usr/lib/python3.12/site-packages/ipaserver/install/ca.py", line 543, in install
    install_step_1(standalone, replica_config, options, custodia=custodia)
  File "/usr/lib/python3.12/site-packages/ipaserver/install/ca.py", line 665, in install_step_1
    ca.setup_lightweight_ca_key_retrieval()
  File "/usr/lib/python3.12/site-packages/ipaserver/install/cainstance.py", line 1341, in setup_lightweight_ca_key_retrieval
    self.__setup_lightweight_ca_key_retrieval_kerberos()
  File "/usr/lib/python3.12/site-packages/ipaserver/install/cainstance.py", line 1365, in __setup_lightweight_ca_key_retrieval_kerberos
    os.chmod(self.keytab, 0o600)

2026-07-14T15:02:20Z DEBUG The ipa-replica-install command failed, exception: FileNotFoundError: [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/dogtag.keytab'
2026-07-14T15:02:20Z ERROR [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/dogtag.keytab'
2026-07-14T15:02:20Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
Currently working on ipa03 on production and encountered these problems: 1. Disconnected topology - after some investigation found leftovers replication agreements on ipa02 LDAP, after re-initialization of ipa02 from ipa01 those were fixed and the installation of ipa03 didn't failed on existing replication agreement anymore 2. Can't create dogtag principal - I found orphaned principal on ipa01 and removed it, but that didn't help. Here is the error I'm currently trying to resolve ``` 2026-07-14T15:02:10Z DEBUG Creating principal 2026-07-14T15:02:10Z DEBUG Starting external process 2026-07-14T15:02:10Z DEBUG args=['/usr/sbin/kadmin.local', '-q', 'addprinc -randkey dogtag/ipa03.rdu3.fedoraproject.org@FEDORAPROJECT.ORG', '-x', 'ipa-setup-override-restrictions'] 2026-07-14T15:02:20Z DEBUG Process finished, return code=0 2026-07-14T15:02:20Z DEBUG stdout=Authenticating as principal host/admin@FEDORAPROJECT.ORG with password. 2026-07-14T15:02:20Z DEBUG stderr=No policy specified for dogtag/ipa03.rdu3.fedoraproject.org@FEDORAPROJECT.ORG; defaulting to no policy add_principal: Server error while creating "dogtag/ipa03.rdu3.fedoraproject.org@FEDORAPROJECT.ORG". 2026-07-14T15:02:20Z DEBUG Retrieving keytab 2026-07-14T15:02:20Z DEBUG Starting external process 2026-07-14T15:02:20Z DEBUG args=['/usr/sbin/kadmin.local', '-q', 'ktadd -k /etc/pki/pki-tomcat/dogtag.keytab dogtag/ipa03.rdu3.fedoraproject.org@FEDORAPROJECT.ORG', '-x', 'ipa-setup-override-restrictions'] 2026-07-14T15:02:20Z DEBUG Process finished, return code=0 2026-07-14T15:02:20Z DEBUG stdout=Authenticating as principal host/admin@FEDORAPROJECT.ORG with password. 2026-07-14T15:02:20Z DEBUG stderr=kadmin.local: Principal dogtag/ipa03.rdu3.fedoraproject.org@FEDORAPROJECT.ORG does not exist. 2026-07-14T15:02:20Z DEBUG File "/usr/lib/python3.12/site-packages/ipapython/admintool.py", line 219, in execute return_value = self.run() ^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/ipapython/install/cli.py", line 343, in run return cfgr.run() ^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 360, in run return self.execute() ^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 386, in execute for rval in self._executor(): ^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 435, in __runner exc_handler(exc_info) File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 468, in _handle_execute_exception self._handle_exception(exc_info) File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 458, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python3.12/site-packages/six.py", line 719, in reraise raise value File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 425, in __runner step() File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 419, in step_next return next(self.__gen) ^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python3.12/site-packages/six.py", line 719, in reraise raise value File "/usr/lib/python3.12/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) ^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 663, in _configure next(executor) File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 435, in __runner exc_handler(exc_info) File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 468, in _handle_execute_exception self._handle_exception(exc_info) File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 526, in _handle_exception self.__parent._handle_exception(exc_info) File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 458, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python3.12/site-packages/six.py", line 719, in reraise raise value File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 523, in _handle_exception super(ComponentBase, self)._handle_exception(exc_info) File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 458, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python3.12/site-packages/six.py", line 719, in reraise raise value File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 425, in __runner step() File "/usr/lib/python3.12/site-packages/ipapython/install/core.py", line 419, in step_next return next(self.__gen) ^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python3.12/site-packages/six.py", line 719, in reraise raise value File "/usr/lib/python3.12/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) ^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/ipapython/install/common.py", line 65, in _install for unused in self._installer(self.parent): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/ipaserver/install/server/__init__.py", line 702, in main replica_install(self) File "/usr/lib/python3.12/site-packages/ipaserver/install/server/replicainstall.py", line 386, in decorated func(installer) File "/usr/lib/python3.12/site-packages/ipaserver/install/server/replicainstall.py", line 1434, in install ca.install(False, config, options, custodia=custodia) File "/usr/lib/python3.12/site-packages/ipaserver/install/ca.py", line 543, in install install_step_1(standalone, replica_config, options, custodia=custodia) File "/usr/lib/python3.12/site-packages/ipaserver/install/ca.py", line 665, in install_step_1 ca.setup_lightweight_ca_key_retrieval() File "/usr/lib/python3.12/site-packages/ipaserver/install/cainstance.py", line 1341, in setup_lightweight_ca_key_retrieval self.__setup_lightweight_ca_key_retrieval_kerberos() File "/usr/lib/python3.12/site-packages/ipaserver/install/cainstance.py", line 1365, in __setup_lightweight_ca_key_retrieval_kerberos os.chmod(self.keytab, 0o600) 2026-07-14T15:02:20Z DEBUG The ipa-replica-install command failed, exception: FileNotFoundError: [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/dogtag.keytab' 2026-07-14T15:02:20Z ERROR [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/dogtag.keytab' 2026-07-14T15:02:20Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information ```
zlopez commented 2026-07-21 14:41:31 +00:00
Owner
Copy link

The ipa03 is now done, replica installed and playbook finished. Will start with ipa02 tomorrow.

The ipa03 is now done, replica installed and playbook finished. Will start with ipa02 tomorrow.
zlopez commented 2026-07-22 13:17:53 +00:00
Owner
Copy link

So ipa01 is still running without changes, but there are some problems with ipa02 installation (different than those I encountered with ipa03), which I'm currently dealing with, and for some reason ipa03 has some replication issues, so I'm trying to fix that first

So ipa01 is still running without changes, but there are some problems with ipa02 installation (different than those I encountered with ipa03), which I'm currently dealing with, and for some reason ipa03 has some replication issues, so I'm trying to fix that first
zlopez commented 2026-07-23 14:23:03 +00:00
Owner
Copy link

So ipa02 is now done as well, hit some new issues, but was able to overcome them. Will continue on ipa01 tomorrow.

So ipa02 is now done as well, hit some new issues, but was able to overcome them. Will continue on ipa01 tomorrow.
zlopez commented 2026-07-24 18:27:29 +00:00
Owner
Copy link

Encountered another new error on ipa01 ([ldap://ipa03.rdu3.fedoraproject.org:389] reports: Update failed! Status: [Error (-1) connection error: Unknown connection error (-1) - Total update aborted] ), but everything should now use ipa02 or ipa03 for everything.

Encountered another new error on ipa01 (`[ldap://ipa03.rdu3.fedoraproject.org:389] reports: Update failed! Status: [Error (-1) connection error: Unknown connection error (-1) - Total update aborted] `), but everything should now use ipa02 or ipa03 for everything.
smoliicek commented 2026-07-24 22:17:12 +00:00
Member
Copy link

The accounts.fp.o interface is currently pretty much non-functional due to this. Users can't login, those who can login / have an open session can't use search and data reporting of the number of people in groups is also broken.

I created an outage on fedorastatus.org in this commit github.com/fedora-infra/statusfpo@0795c0dbb4

The accounts.fp.o interface is currently pretty much non-functional due to this. Users can't login, those who can login / have an open session can't use search and data reporting of the number of people in groups is also broken. I created an outage on fedorastatus.org in this commit https://github.com/fedora-infra/statusfpo/commit/0795c0dbb481eae51d111577f10b9cdcc5aaf74c
kevin commented 2026-07-25 00:52:21 +00:00
Author
Owner
Copy link

I am not sure whats going on here. :( Everything seems to be running...

  • noggin (the app) is running and people can all reach and load it.
  • ipa02/03 status is all ok
  • kerberos via kdcproxy is working fine
  • Users are not locked

I did do a noggin build and scaled it up in case there was a issue there. I can now seem to login ok... but unclear if it's fixed for others or not. So, confirmation from a few folks seeing it would be welcome.

I am not sure whats going on here. :( Everything seems to be running... * noggin (the app) is running and people can all reach and load it. * ipa02/03 status is all ok * kerberos via kdcproxy is working fine * Users are not locked I did do a noggin build and scaled it up in case there was a issue there. I can now seem to login ok... but unclear if it's fixed for others or not. So, confirmation from a few folks seeing it would be welcome.
zlopez commented 2026-07-28 14:27:59 +00:00
Owner
Copy link

Found out that the ipa02 was not in great state and it was blocking ipa01 install, so I'm reinstalling it now. Right now only ipa03 is working correctly.

Hopefully I will get them to working state soon.

Found out that the ipa02 was not in great state and it was blocking ipa01 install, so I'm reinstalling it now. Right now only ipa03 is working correctly. Hopefully I will get them to working state soon.
kevin commented 2026-07-29 00:05:25 +00:00
Author
Owner
Copy link

So, current status: I have tried to point everything I can to ipa03, but...

  • accounts.fedoraproject.org logins work for some users and not others. It doesn't log anything in noggin logs or ipa that I can see, just shows unauthorized. ;( I have no idea why.
  • I also can't seem to login to the ipa web ui anymore. ;(
  • ipa command line still works on ipa03.
  • kerberos and ipsilon have always been working, which helps mitigate things...

I think if we cant get things back up tomorrow, we should engage more folks... perhaps we could get more help from IPA maintainers? And if it's not back tomorrow after I get in, I'm happy to take the baton and keep working on it...

After we get things all back, we should have a retro on this and see if there's ways we could make things more robust in the case of some servers not being available. We seem to have a lot of hard coded assumptions, would be nice to drop those in favor of just DNS or something to make it easier to switch away. Also, would be nice if noggin logged more about it's errors if possible.

So, current status: I have tried to point everything I can to ipa03, but... * accounts.fedoraproject.org logins work for some users and not others. It doesn't log anything in noggin logs or ipa that I can see, just shows unauthorized. ;( I have no idea why. * I also can't seem to login to the ipa web ui anymore. ;( * ipa command line still works on ipa03. * kerberos and ipsilon have always been working, which helps mitigate things... I think if we cant get things back up tomorrow, we should engage more folks... perhaps we could get more help from IPA maintainers? And if it's not back tomorrow after I get in, I'm happy to take the baton and keep working on it... After we get things all back, we should have a retro on this and see if there's ways we could make things more robust in the case of some servers not being available. We seem to have a lot of hard coded assumptions, would be nice to drop those in favor of just DNS or something to make it easier to switch away. Also, would be nice if noggin logged more about it's errors if possible.
zlopez commented 2026-07-29 07:58:43 +00:00
Owner
Copy link

My assumption is that most of the problems are caused how big the LDAP database is as the install problems are mostly timeouts during replication or that the machine is not visible in database yet at the end of installation.

There is also issue with dogtag keytab, I reported it here.

I'm doing my best to get it to working shape again.

My assumption is that most of the problems are caused how big the LDAP database is as the install problems are mostly timeouts during replication or that the machine is not visible in database yet at the end of installation. There is also issue with `dogtag keytab`, I reported it [here](https://redhat.atlassian.net/browse/HMSIDM-241?atlOrigin=eyJpIjoiY2FjNzlkMmJkZjMzNGFkMGI4NTU2MmM1MWMzM2Y1OTQiLCJwIjoiaiJ9). I'm doing my best to get it to working shape again.
zlopez commented 2026-07-29 17:29:54 +00:00
Owner
Copy link

So I wasn't able to finish the ipa02 installation today. Here is the current state of things:

  • ipa01 is currently not installed, just clean RHEL 10 VM
  • ipa02 has failed ipa-replica-install (I will put instruction bellow what to do with it)
    I'm trying to resolve the error from LDAP that is happening during CA replication and causing timeout 
    [29/Jul/2026:17:06:49.461628224 +0000] - WARN - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=caToipa03.rdu3.fedoraproject.org" (ipa03:389): The remote replica has a different database generation ID than the local database.  You may have to reinitialize the remote replica, or the local replica.
[29/Jul/2026:17:06:49.540552138 +0000] - ERR - ipa-topology-plugin - ipa_topo_be_state_changebackend ipaca is going offline; inactivate plugin
[29/Jul/2026:17:06:49.541484047 +0000] - NOTICE - NSMMReplicationPlugin - multisupplier_be_state_change - Replica o=ipaca is going offline; disabling replication
[29/Jul/2026:17:06:49.745332382 +0000] - ERR - cos-plugin - cos_dn_defs_cb - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=fedoraproject,dc=org--no CoS Templates found, which should be added before the CoS Definition.
[29/Jul/2026:17:07:09.817504142 +0000] - INFO - dbmdb_import_monitor_threads - import ipaca: Processed 2322 entries -- average rate 116.1/sec, recent rate 116.0/sec, 
[29/Jul/2026:17:07:29.848193492 +0000] - INFO - dbmdb_import_monitor_threads - import ipaca: Processed 3467 entries -- average rate 86.7/sec, recent rate 86.7/sec, 
[29/Jul/2026:17:07:49.884181967 +0000] - INFO - dbmdb_import_monitor_threads - import ipaca: Processed 3467 entries -- average rate 57.8/sec, recent rate 28.6/sec, 
[29/Jul/2026:17:08:09.916285323 +0000] - INFO - dbmdb_import_monitor_threads - import ipaca: Processed 3467 entries -- average rate 43.3/sec, recent rate 0.0/sec, 
[29/Jul/2026:17:08:29.945888599 +0000] - INFO - dbmdb_import_monitor_threads - import ipaca: Processed 3467 entries -- average rate 34.7/sec, recent rate 0.0/sec, 
[29/Jul/2026:17:08:49.986527739 +0000] - INFO - dbmdb_import_monitor_threads - import ipaca: Processed 3467 entries -- average rate 28.9/sec, recent rate 0.0/sec, 
[29/Jul/2026:17:09:10.021661804 +0000] - INFO - dbmdb_import_monitor_threads - import ipaca: Processed 3467 entries -- average rate 24.8/sec, recent rate 0.0/sec, 
[29/Jul/2026:17:09:30.051237243 +0000] - INFO - dbmdb_import_monitor_threads - import ipaca: Processed 3467 entries -- average rate 21.7/sec, recent rate 0.0/sec, 
[29/Jul/2026:17:09:50.081630441 +0000] - INFO - dbmdb_import_monitor_threads - import ipaca: Processed 3467 entries -- average rate 19.3/sec, recent rate 0.0/sec, 
[29/Jul/2026:17:10:10.110474894 +0000] - INFO - dbmdb_import_monitor_threads - import ipaca: Processed 3467 entries -- average rate 17.3/sec, recent rate 0.0/sec,
  • ipa03 currently OK and running

What I'm doing

  1. [root@ipa02 ~][PROD-RDU3]# ipa-replica-install --setup-ca --admin-password="xxx" --no-host-dns --mkhomedir --no-ntp --unattended --no-ssh --no-sshd --log-file=/var/log/ipainstall.log --domain=fedoraproject.org --server=ipa03.rdu3.fedoraproject.org --dirsrv-config-file=/root/ldif/replica-install.ldif ('xxx' should be replaced by IPA admin password)
  2. Following are the cleaning steps if the installation fails [root@ipa02 ~][PROD-RDU3]# ipa-client-install --uninstall -U
  3. [root@ipa02 ~][PROD-RDU3]# ipa-server-install --uninstall -U (If that hangs on directory server unconfiguration run systemctl kill -s SIGKILL dirsrv@FEDORAPROJECT-ORG in second terminal)
  4. [root@ipa03 ~][PROD-RDU3]# ipa-replica-manage del ipa02.rdu3.fedoraproject.org --force
  5. [root@ipa03 ~][PROD-RDU3]# ipa server-del ipa02.rdu3.fedoraproject.org --force
  6. There are LDAP entries that needs to be deleted manually on ipa03
    [root@ipa03 ~][PROD-RDU3]# ldapdelete -x -D "cn=Directory Manager" -W "cn=meToipa02.rdu3.fedoraproject.org,cn=replica,cn=dc\3Dfedoraproject\2Cdc\3Dorg,cn=mapping tree,cn=config"
    [root@ipa03 ~][PROD-RDU3]# ldapdelete -x -D "cn=Directory Manager" -W "cn=caToipa02.rdu3.fedoraproject.org,cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config"
  7. Clean the tombstone replication (the CA replication is still hanging) 
    [root@ipa03 ~][PROD-RDU3]# ldapsearch -xLLL -D "cn=directory manager" -W -b dc=fedoraproject,dc=org '(& (nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))'
[root@ipa03 ~][PROD-RDU3]# ldapmodify -x -D "cn=directory manager" -W <<EOF                                                                                                        
dn: cn=replica,cn=dc\3Dfedoraproject\2Cdc\3Dorg,cn=mapping tree,cn=config
changetype: modify
replace: nsds5task
nsds5task: CLEANRUV<replica_ID_from_previous_ldapsearch>
EOF

Logs to check:

  • /var/log/ipainstall.log
  • /var/log/dirsrv/slapd-FEDORAPROJECT-ORG/errors
So I wasn't able to finish the ipa02 installation today. Here is the current state of things: * ipa01 is currently not installed, just clean RHEL 10 VM * ipa02 has failed `ipa-replica-install` (I will put instruction bellow what to do with it) I'm trying to resolve the error from LDAP that is happening during CA replication and causing timeout ``` [29/Jul/2026:17:06:49.461628224 +0000] - WARN - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=caToipa03.rdu3.fedoraproject.org" (ipa03:389): The remote replica has a different database generation ID than the local database. You may have to reinitialize the remote replica, or the local replica. [29/Jul/2026:17:06:49.540552138 +0000] - ERR - ipa-topology-plugin - ipa_topo_be_state_changebackend ipaca is going offline; inactivate plugin [29/Jul/2026:17:06:49.541484047 +0000] - NOTICE - NSMMReplicationPlugin - multisupplier_be_state_change - Replica o=ipaca is going offline; disabling replication [29/Jul/2026:17:06:49.745332382 +0000] - ERR - cos-plugin - cos_dn_defs_cb - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=fedoraproject,dc=org--no CoS Templates found, which should be added before the CoS Definition. [29/Jul/2026:17:07:09.817504142 +0000] - INFO - dbmdb_import_monitor_threads - import ipaca: Processed 2322 entries -- average rate 116.1/sec, recent rate 116.0/sec, [29/Jul/2026:17:07:29.848193492 +0000] - INFO - dbmdb_import_monitor_threads - import ipaca: Processed 3467 entries -- average rate 86.7/sec, recent rate 86.7/sec, [29/Jul/2026:17:07:49.884181967 +0000] - INFO - dbmdb_import_monitor_threads - import ipaca: Processed 3467 entries -- average rate 57.8/sec, recent rate 28.6/sec, [29/Jul/2026:17:08:09.916285323 +0000] - INFO - dbmdb_import_monitor_threads - import ipaca: Processed 3467 entries -- average rate 43.3/sec, recent rate 0.0/sec, [29/Jul/2026:17:08:29.945888599 +0000] - INFO - dbmdb_import_monitor_threads - import ipaca: Processed 3467 entries -- average rate 34.7/sec, recent rate 0.0/sec, [29/Jul/2026:17:08:49.986527739 +0000] - INFO - dbmdb_import_monitor_threads - import ipaca: Processed 3467 entries -- average rate 28.9/sec, recent rate 0.0/sec, [29/Jul/2026:17:09:10.021661804 +0000] - INFO - dbmdb_import_monitor_threads - import ipaca: Processed 3467 entries -- average rate 24.8/sec, recent rate 0.0/sec, [29/Jul/2026:17:09:30.051237243 +0000] - INFO - dbmdb_import_monitor_threads - import ipaca: Processed 3467 entries -- average rate 21.7/sec, recent rate 0.0/sec, [29/Jul/2026:17:09:50.081630441 +0000] - INFO - dbmdb_import_monitor_threads - import ipaca: Processed 3467 entries -- average rate 19.3/sec, recent rate 0.0/sec, [29/Jul/2026:17:10:10.110474894 +0000] - INFO - dbmdb_import_monitor_threads - import ipaca: Processed 3467 entries -- average rate 17.3/sec, recent rate 0.0/sec, ``` * ipa03 currently OK and running What I'm doing 1. `[root@ipa02 ~][PROD-RDU3]# ipa-replica-install --setup-ca --admin-password="xxx" --no-host-dns --mkhomedir --no-ntp --unattended --no-ssh --no-sshd --log-file=/var/log/ipainstall.log --domain=fedoraproject.org --server=ipa03.rdu3.fedoraproject.org --dirsrv-config-file=/root/ldif/replica-install.ldif` ('xxx' should be replaced by IPA admin password) 2. Following are the cleaning steps if the installation fails `[root@ipa02 ~][PROD-RDU3]# ipa-client-install --uninstall -U` 3. `[root@ipa02 ~][PROD-RDU3]# ipa-server-install --uninstall -U` (If that hangs on directory server unconfiguration run `systemctl kill -s SIGKILL dirsrv@FEDORAPROJECT-ORG` in second terminal) 4. `[root@ipa03 ~][PROD-RDU3]# ipa-replica-manage del ipa02.rdu3.fedoraproject.org --force` 5. `[root@ipa03 ~][PROD-RDU3]# ipa server-del ipa02.rdu3.fedoraproject.org --force` 6. There are LDAP entries that needs to be deleted manually on ipa03 `[root@ipa03 ~][PROD-RDU3]# ldapdelete -x -D "cn=Directory Manager" -W "cn=meToipa02.rdu3.fedoraproject.org,cn=replica,cn=dc\3Dfedoraproject\2Cdc\3Dorg,cn=mapping tree,cn=config"` `[root@ipa03 ~][PROD-RDU3]# ldapdelete -x -D "cn=Directory Manager" -W "cn=caToipa02.rdu3.fedoraproject.org,cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config"` 7. Clean the tombstone replication (the CA replication is still hanging) ``` [root@ipa03 ~][PROD-RDU3]# ldapsearch -xLLL -D "cn=directory manager" -W -b dc=fedoraproject,dc=org '(& (nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))' [root@ipa03 ~][PROD-RDU3]# ldapmodify -x -D "cn=directory manager" -W <<EOF dn: cn=replica,cn=dc\3Dfedoraproject\2Cdc\3Dorg,cn=mapping tree,cn=config changetype: modify replace: nsds5task nsds5task: CLEANRUV<replica_ID_from_previous_ldapsearch> EOF ``` Logs to check: * `/var/log/ipainstall.log` * `/var/log/dirsrv/slapd-FEDORAPROJECT-ORG/errors`
kevin commented 2026-07-29 20:39:45 +00:00
Author
Owner
Copy link

So, I got it to fail again (after an hour).

I tossed this at a clanker and it noted that there may be a tombstone replication also for the CA not just the dc...

But... on cleaning things up, when I get to step 7, it's now giving me:

[root@ipa03 ~][PROD-RDU3]# ldapmodify -x -D "cn=directory manager" -W <<EOF                                                                                                        
dn: cn=replica,cn=dc\3Dfedoraproject\2Cdc\3Dorg,cn=mapping tree,cn=config
changetype: modify
replace: nsds5task
nsds5task: CLEANRUV124
EOF
Enter LDAP Password: 
modifying entry "cn=replica,cn=dc\3Dfedoraproject\2Cdc\3Dorg,cn=mapping tree,cn=config"
ldap_modify: Server is unwilling to perform (53)

I see:

[root@ipa03 ~][PROD-RDU3]#   ldapsearch -xLLL -D "cn=directory manager" -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" '(objectclass=nsds5replica)' nsDS5ReplicaId
Enter LDAP Password: 
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nsDS5ReplicaId: 125

so there also does seem to be a ca one that needs clearing? But somehow it doesn't want to clear either of them now. (I also tried a CLEANRUV on the ns one and it gave the same unwilling to perform.

I'm not sure what to do now here... I suppose we could sidestep this a bit and just setup ipa01 (since there's no partial replications to it?). But likely we need advise from ipa folks.

So, I got it to fail again (after an hour). I tossed this at a clanker and it noted that there may be a tombstone replication also for the _CA_ not just the dc... But... on cleaning things up, when I get to step 7, it's now giving me: ``` [root@ipa03 ~][PROD-RDU3]# ldapmodify -x -D "cn=directory manager" -W <<EOF dn: cn=replica,cn=dc\3Dfedoraproject\2Cdc\3Dorg,cn=mapping tree,cn=config changetype: modify replace: nsds5task nsds5task: CLEANRUV124 EOF Enter LDAP Password: modifying entry "cn=replica,cn=dc\3Dfedoraproject\2Cdc\3Dorg,cn=mapping tree,cn=config" ldap_modify: Server is unwilling to perform (53) ``` I see: ``` [root@ipa03 ~][PROD-RDU3]# ldapsearch -xLLL -D "cn=directory manager" -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" '(objectclass=nsds5replica)' nsDS5ReplicaId Enter LDAP Password: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsDS5ReplicaId: 125 ``` so there also does seem to be a ca one that needs clearing? But somehow it doesn't want to clear either of them now. (I also tried a CLEANRUV on the ns one and it gave the same unwilling to perform. I'm not sure what to do now here... I suppose we could sidestep this a bit and just setup ipa01 (since there's no partial replications to it?). But likely we need advise from ipa folks.
kevin commented 2026-07-29 20:51:43 +00:00
Author
Owner
Copy link

So...

[root@ipa03 slapd-FEDORAPROJECT-ORG][PROD-RDU3]# ipa-replica-manage list-ruv
Directory Manager password: 

Replica Update Vectors:
        ipa03.rdu3.fedoraproject.org:389: 124
        ipa02.rdu3.fedoraproject.org:389: 187
        ipa02.rdu3.fedoraproject.org:389: 189
Certificate Server Replica Update Vectors:
        ipa03.rdu3.fedoraproject.org:389: 125
        ipa01.rdu3.fedoraproject.org:389: 167
        ipa01.rdu3.fedoraproject.org:389: 163
        ipa03.rdu3.fedoraproject.org:389: 121
        ipa01.rdu3.fedoraproject.org:389: 159
        ipa01.rdu3.fedoraproject.org:389: 161
        ipa02.rdu3.fedoraproject.org:389: 171

So, lets try cleaning up the ipa02 ones... ok, let me do that.

Ok, did that and now re-running replica install on 02.

So... ``` [root@ipa03 slapd-FEDORAPROJECT-ORG][PROD-RDU3]# ipa-replica-manage list-ruv Directory Manager password: Replica Update Vectors: ipa03.rdu3.fedoraproject.org:389: 124 ipa02.rdu3.fedoraproject.org:389: 187 ipa02.rdu3.fedoraproject.org:389: 189 Certificate Server Replica Update Vectors: ipa03.rdu3.fedoraproject.org:389: 125 ipa01.rdu3.fedoraproject.org:389: 167 ipa01.rdu3.fedoraproject.org:389: 163 ipa03.rdu3.fedoraproject.org:389: 121 ipa01.rdu3.fedoraproject.org:389: 159 ipa01.rdu3.fedoraproject.org:389: 161 ipa02.rdu3.fedoraproject.org:389: 171 ``` So, lets try cleaning up the ipa02 ones... ok, let me do that. Ok, did that and now re-running replica install on 02.
kevin commented 2026-07-29 22:19:17 +00:00
Author
Owner
Copy link

Still failed:

...
Starting replication, please wait until this has completed.
Update in progress, 146 seconds elapsed
[ldap://ipa03.rdu3.fedoraproject.org:389] reports: Update failed! Status: [Error (-1) connection error: Unknown connection error (-1) - Total update aborted]

  [error] RuntimeError: Failed to start replication
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

Failed to start replication
The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

real    58m48.763s
user    0m10.118s
sys     0m3.656s

and logs on 03:

[29/Jul/2026:21:47:37.698671135 +0000] - INFO - NSMMReplicationPlugin - repl5_tot_run - Beginning total update of replica "agmt="cn=caToipa02.rdu3.fedoraproject.org" (ipa02:389)".
[29/Jul/2026:21:47:37.703339696 +0000] - WARN - NSMMReplicationPlugin - check_flow_control_tot_init - agmt="cn=caToipa02.rdu3.fedoraproject.org" (ipa02:389) -  Total update flow control gives time (200 msec) to the consumer before sending more entries [ msgid sent: 61, rcv: 10])
If total update fails you can try to increase nsds5ReplicaFlowControlPause and/or decrease nsds5ReplicaFlowControlWindow in the replica agreement configuration
[29/Jul/2026:21:50:03.995095783 +0000] - ERR - NSMMReplicationPlugin - release_replica - agmt="cn=caToipa02.rdu3.fedoraproject.org" (ipa02:389): Unable to send endReplication extended operation (Timed out)
[29/Jul/2026:21:50:03.996191244 +0000] - ERR - NSMMReplicationPlugin - repl5_tot_run - Total update failed for replica "agmt="cn=caToipa02.rdu3.fedoraproject.org" (ipa02:389)", error (-1)
[29/Jul/2026:21:50:03.996606178 +0000] - ERR - NSMMReplicationPlugin - agmt="cn=caToipa02.rdu3.fedoraproject.org" (ipa02:389): Total update flow control triggered 728 times
You may increase nsds5ReplicaFlowControlPause and/or decrease nsds5ReplicaFlowControlWindow in the replica agreement configuration
Still failed: ``` ... Starting replication, please wait until this has completed. Update in progress, 146 seconds elapsed [ldap://ipa03.rdu3.fedoraproject.org:389] reports: Update failed! Status: [Error (-1) connection error: Unknown connection error (-1) - Total update aborted] [error] RuntimeError: Failed to start replication Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up. Failed to start replication The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information real 58m48.763s user 0m10.118s sys 0m3.656s ``` and logs on 03: ``` [29/Jul/2026:21:47:37.698671135 +0000] - INFO - NSMMReplicationPlugin - repl5_tot_run - Beginning total update of replica "agmt="cn=caToipa02.rdu3.fedoraproject.org" (ipa02:389)". [29/Jul/2026:21:47:37.703339696 +0000] - WARN - NSMMReplicationPlugin - check_flow_control_tot_init - agmt="cn=caToipa02.rdu3.fedoraproject.org" (ipa02:389) - Total update flow control gives time (200 msec) to the consumer before sending more entries [ msgid sent: 61, rcv: 10]) If total update fails you can try to increase nsds5ReplicaFlowControlPause and/or decrease nsds5ReplicaFlowControlWindow in the replica agreement configuration [29/Jul/2026:21:50:03.995095783 +0000] - ERR - NSMMReplicationPlugin - release_replica - agmt="cn=caToipa02.rdu3.fedoraproject.org" (ipa02:389): Unable to send endReplication extended operation (Timed out) [29/Jul/2026:21:50:03.996191244 +0000] - ERR - NSMMReplicationPlugin - repl5_tot_run - Total update failed for replica "agmt="cn=caToipa02.rdu3.fedoraproject.org" (ipa02:389)", error (-1) [29/Jul/2026:21:50:03.996606178 +0000] - ERR - NSMMReplicationPlugin - agmt="cn=caToipa02.rdu3.fedoraproject.org" (ipa02:389): Total update flow control triggered 728 times You may increase nsds5ReplicaFlowControlPause and/or decrease nsds5ReplicaFlowControlWindow in the replica agreement configuration ```
kevin commented 2026-07-30 00:53:57 +00:00
Author
Owner
Copy link

I've done all the cleanup steps again...

I guess we should consult with ipa folks and see if they can think of anything else to try.
and/or I guess you could do ipa01 and see if it gets anywhere? (but there's some RUV's that need clearing for ipa01)

I've done all the cleanup steps again... I guess we should consult with ipa folks and see if they can think of anything else to try. and/or I guess you could do ipa01 and see if it gets anywhere? (but there's some RUV's that need clearing for ipa01)
zlopez commented 2026-07-30 15:38:22 +00:00
Owner
Copy link

So ipa01 and ipa02 are now done. Cleaning the CA RUV helped.

There are some replication errors I need to look at.

Closing this as done. Will reopen some smaller ticket for post update tasks.

So ipa01 and ipa02 are now done. Cleaning the CA RUV helped. There are some replication errors I need to look at. Closing this as done. Will reopen some smaller ticket for post update tasks.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
announcement

Archived

  
anubis
Issues related to anubis

  
authentication
Authentication related issues

  
aws
Issues related to Amazon Web Services

  
backlog
Tasks that will take longer then ~4 hours to do

Archived

  
blocked
Issue is blocked

Archived

  
bodhi
Issues with Bodhi

  
ci
Continuous Integration

  
cloud
Items in FedoraInfraCloud or COmmunishift

  
communishift
issues/questions about os.edorainfracloud.org ( Community OpenShift)

  
copr
COPR system (run by other team)

  
database
Issues related to databases

  
day-to-day
These are small self contained tasks that are part of normal operations, but are async from sprint planning.

  
dc-move
Items related to datacenter move

Archived

  
deprecated
Tools which Infrastructure may not be able to fix

Archived

  
dev
Tasks requiring some dev work

  
discourse
outsourced tool

  
dns
Changes or problems with DNS

  
downloads
rsync,dl.fp.o,AWS,torrents

  
easyfix

Archived

  
epel
Issues relating to epel

Archived

  
firmitas

   
forgejo_migration
Request migration to the new forge

  
Gain
High
 task that gets us a lot / is important / makes many people happy

Archived

  
Gain
Low
 tasks that gets us little / isn't very important / makes only 1 person or few happy

Archived

  
Gain
Medium
 task that gets us some / is somewhat important / makes some people happy

Archived

  
gitlab

   
greenwave
Issues with greenwave

  
hardware
Problem is with Fedora Infrastructure Owned/Leased Hardware

  
help wanted

   
high-trouble
task that is complex/long/difficult

Archived

  
koji
Issues with the build system

  
koschei
Items needing Koschei team

  
lists
issues with lists

  
low-trouble
task that is easy/short

Archived

  
medium-trouble
task that is medium complex / difficult

Archived

  
mirrorlists
Way to tell systems where downloads are

  
monitoring

   
Needs investigation

   
odcs
On Demand Compose Service issues

Archived

  
OpenShift
Issues with OpenShift

  
ops
ops problem now...

Archived

  
outage

   
packager_workflow_blocker
Blocks the packager workflow

Archived

  
pagure
Issues with pagure.io

  
permissions
Items which need permissions set in outside tools

  
Priority
Needs Review
 Priority of Needs Review

Archived

  
Priority
Next Meeting
 Priority of Next Meeting

Archived

  
Priority
🔥 URGENT 🔥
 Priority of 🔥 URGENT 🔥

Archived

  
Priority
Waiting on Assignee
 Priority of Waiting on Assignee

Archived

  
Priority
Waiting on External
 Priority of Waiting on External

Archived

  
Priority
Waiting on Reporter
 Priority of Waiting on Reporter

Archived

  
rabbitmq
issues with the message broker

  
release-monitoring
release-monitoring.org (Anitya)

  
releng
For problems which should be in the releng ticket system

Archived

  
request-for-resources

   
s390x
The secondary architecture hardware and problems related with it

  
security

   
SMTP
Email issues outside of lists (DMARC, SPF, aliases)

  
sprint-0

Archived

  
sprint-1

Archived

  
src.fp.o
Issues related to src.fedoraproject.org

  
staging
Issues in staging

  
unfreeze
Patches to be applied after freeze is lifted

  
waiverdb
Issues with waiverdb

  
websites-general
General items with a website focus

  
wiki
Issues with the Wiki

  
Backlog Status
Needs Review
 Work items that have been created or updated but need review, refinement, or approval before they can be considered ready for sprint planning.

  
Backlog Status
Ready
 Work items that have been reviewed, refined, and are ready to be committed to a sprint. These items have clear requirements and can be worked on immediately when capacity is available.

  
chore
Work that needs to be done but doesn't add new functionality or fix bugs. Maintenance tasks that keep the system running smoothly and the codebase clean.

  
documentation
Work focused on creating, updating, or improving documentation for users, developers, or system administrators.

  
points
01
 A task that is simple, small, and has very little uncertainty. It should be easy enough for any team member to pick up and complete quickly, without a lot of discussion or research.

  
points
02
 This is a task that is slightly more complex or larger than a "1." It might involve two or three simple steps, but the outcome is still very clear and there's little uncertainty.

  
points
03
 Medium-sized task with some noticeable complexity. It might require a bit more thought or collaboration. The steps are well-defined, but it's not a trivial change.

  
points
05
 A significant piece of work. It has more complexity and a higher degree of uncertainty than smaller tasks. It might require some research, a design discussion, or collaboration across multiple teams.

  
points
08
 Complex task with a lot of uncertainty. It's often the largest size that a team will consider for a single sprint. It likely requires a detailed breakdown and multiple team members to complete.

  
points
13
 Any task that feels like it's a "13" or more is considered an Epic. These are too large and uncertain to be worked on in a single sprint and should be broken down.

  
Priority
High
 This is for work that is critical and needs to be addressed in the next sprint. It's often for urgent bugs, important features, or a major piece of technical debt that is blocking other work.

  
Priority
Low
 This is for work that is good to do but not urgent. It's a candidate for being worked on when there's extra time or when it rises in importance. It can stay in the backlog for a longer period.

  
Priority
Medium
 This is for important work that should be done soon, but isn't as urgent as a high-priority item. It will likely be considered for the next 1-2 sprints.

  
Sprint Status
Blocked
 Work items that cannot progress due to external dependencies, blockers, or issues that need resolution.

  
Sprint Status
Done
 Work items that have been completed, reviewed, and integrated. No further work is needed within the current sprint.

  
Sprint Status
In Progress
 Work items currently being worked on by a team member. Active development, research, or implementation is happening.

  
Sprint Status
Review
 Work items that are complete but need review, testing, or approval before being marked as done.

  
Sprint Status
To Do
 Work items that are committed to the sprint but haven't been started yet. These items are ready to be picked up by team members.

  
Technical Debt
Work that addresses shortcuts, workarounds, or suboptimal implementations that were done previously to meet deadlines. These items improve code quality, maintainability, or performance without adding new features.

  
Work Item
Bug
 Issues where the system is not working as intended or designed. Bugs represent deviations from expected behavior.

  
Work Item
Epic
 Large initiatives that are broken down into multiple smaller work items (user stories, tasks, bugs). Epics represent significant features or projects that span multiple sprints.

  
Work Item
Spike
 Time-boxed research, investigation, or proof-of-concept work to reduce uncertainty or explore solutions. Spikes help inform future development decisions.

  
Work Item
Task
 Used to break down larger user stories into smaller, manageable implementation pieces. They focus on the specific technical steps needed to deliver the user story's value.

  
Work Item
User Story
 User-facing features or functionality that can be framed from a user perspective, describing what someone wants to accomplish and why.

No labels
announcement
anubis
authentication
aws
backlog
blocked
bodhi
ci
cloud
communishift
copr
database
day-to-day
dc-move
deprecated
dev
discourse
dns
downloads
easyfix
epel
firmitas
forgejo_migration
Gain
High
Gain
Low
Gain
Medium
gitlab
greenwave
hardware
help wanted
high-trouble
koji
koschei
lists
low-trouble
medium-trouble
mirrorlists
monitoring
Needs investigation
odcs
OpenShift
ops
outage
packager_workflow_blocker
pagure
permissions
Priority
Needs Review
Priority
Next Meeting
Priority
🔥 URGENT 🔥
Priority
Waiting on Assignee
Priority
Waiting on External
Priority
Waiting on Reporter
rabbitmq
release-monitoring
releng
request-for-resources
s390x
security
SMTP
sprint-0
sprint-1
src.fp.o
staging
unfreeze
waiverdb
websites-general
wiki
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Sprint ( 2026-07-27 to 2026-08-09)
Assignees
Clear assignees
No assignees
zlopez
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Blocks
#12712 RHEL10 adoption
infra/tickets
Reference
infra/tickets#13382
No description provided.