infra/tickets
24
9
Fork 1
Code Issues 81 Projects Wiki Activity Actions

Signing problem with the latest rawhide kernels #13279

New issue
Closed
opened 2026-04-20 16:06:24 +00:00 by jcline · 22 comments
jcline commented 2026-04-20 16:06:24 +00:00
Copy link

Description of request

The latest rawhide kernel build failed when signing the kernel for Secure Boot: https://koji.fedoraproject.org/koji/taskinfo?taskID=144635563

Something went wrong on the host side with the sigul-pesign-bridge service. Can someone pull the logs for both the aarch64 and x86_64 builders?

### Description of request The latest rawhide kernel build failed when signing the kernel for Secure Boot: https://koji.fedoraproject.org/koji/taskinfo?taskID=144635563 Something went wrong on the host side with the sigul-pesign-bridge service. Can someone pull the logs for both the aarch64 and x86_64 builders?
jcline commented 2026-04-20 16:13:48 +00:00
Author
Copy link

A few thoughts on things to check: infra/ansible#3267 changed how the ACLs got applied to the service. Did sigul-pesign-bridge.service restart and the ACLs failed to apply properly?

Otherwise the logging should cover what went wrong. It is supposed to retry requests for a day but it returned a failure very quickly so it may be it's treating an error as fatal when it should be retrying.

A few thoughts on things to check: https://forge.fedoraproject.org/infra/ansible/pulls/3267 changed how the ACLs got applied to the service. Did sigul-pesign-bridge.service restart and the ACLs failed to apply properly? Otherwise the logging should cover what went wrong. It is supposed to retry requests for a day but it returned a failure very quickly so it may be it's treating an error as fatal when it should be retrying.
james commented 2026-04-20 20:00:41 +00:00
Member
Copy link

The permissions look fine on the host. That PR, and the ones before it, got enough double checks I'd assume it isn't the cause anymore. I think this is the relevant bit from the logs:

Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]: INFO listen:request{request_id="019dac26-5ad5-7100-9218-00cd782732bf"}:sign_attached_with_filetype:sign_pe: siguldry::v1::connection: Sigul server returned status code status_code=9
Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]: ERROR listen:request{request_id="019dac26-5ad5-7100-9218-00cd782732bf"}:sign_attached_with_filetype:sign_pe: siguldry::v1::connection: error=an I/O error occurred: unexpected end of file
Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]: INFO listen:request{request_id="019dac26-5ad5-7100-9218-00cd782732bf"}:sign_attached_with_filetype:sign_pe: siguldry::v1::client: close
Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]: ERROR listen:request{request_id="019dac26-5ad5-7100-9218-00cd782732bf"}:sign_attached_with_filetype: sigul_pesign_bridge::service: signing failed due to an unrecoverable error error=an I/O error occurred: unexpected end of file
Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]: INFO listen:request{request_id="019dac26-5ad5-7100-9218-00cd782732bf"}:sign_attached_with_filetype: sigul_pesign_bridge::service: return=Err(an I/O error occurred: unexpected end of file
Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]: Caused by:
Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]:    unexpected end of file)
Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]: INFO listen:request{request_id="019dac26-5ad5-7100-9218-00c
d782732bf"}:sign_attached_with_filetype: sigul_pesign_bridge::service: close
Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]: INFO listen:request{request_id="019dac26-5ad5-7100-9218-00cd782732bf"}: sigul_pesign_bridge::service: return=Err(an I/O error occurred: unexpected end of file
Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]: Caused by:
Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]:    unexpected end of file)
Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]: INFO listen:request{request_id="019dac26-5ad5-7100-9218-00cd782732bf"}: sigul_pesign_bridge::service: close


Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]: 2026-04-20 18:28:11,946 [WARNING] {2251563} koji.TaskManager:1443 TRACEBACK: Traceback (most recent call last):
Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]:  File "/usr/lib/python3.14/site-packages/koji/daemon.py", line 1421, in runTask
Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]:    response = (handler.run(),)
Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]:                ~~~~~~~~~~~^^
Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]:  File "/usr/lib/python3.14/site-packages/koji/tasks.py", line 343, in run
Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]:    return koji.util.call_with_argcheck(self.handler, self.params, self.opts)
Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]:           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]:  File "/usr/lib/python3.14/site-packages/koji/util.py", line 516, in call_with_argcheck
Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]:    return func(*args, **kwargs)
Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]:  File "/usr/sbin/kojid", line 1663, in handler
Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]:    broot.build(fn, arch)
Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]:    ~~~~~~~~~~~^^^^^^^^^^
Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]:  File "/usr/sbin/kojid", line 631, in build
Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]:    raise koji.BuildError("error building package (arch %s), %s" %
Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]:                          (arch, self._mockResult(rv)))
Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]: koji.BuildError: error building package (arch x86_64), mock exited with status 1; see build.log or root.log for more information

Let me know if you need/want more info. ... going to leave this open for a bit.

The permissions look fine on the host. That PR, and the ones before it, got enough double checks I'd assume it isn't the cause anymore. I think this is the relevant bit from the logs: ``` Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]: INFO listen:request{request_id="019dac26-5ad5-7100-9218-00cd782732bf"}:sign_attached_with_filetype:sign_pe: siguldry::v1::connection: Sigul server returned status code status_code=9 Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]: ERROR listen:request{request_id="019dac26-5ad5-7100-9218-00cd782732bf"}:sign_attached_with_filetype:sign_pe: siguldry::v1::connection: error=an I/O error occurred: unexpected end of file Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]: INFO listen:request{request_id="019dac26-5ad5-7100-9218-00cd782732bf"}:sign_attached_with_filetype:sign_pe: siguldry::v1::client: close Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]: ERROR listen:request{request_id="019dac26-5ad5-7100-9218-00cd782732bf"}:sign_attached_with_filetype: sigul_pesign_bridge::service: signing failed due to an unrecoverable error error=an I/O error occurred: unexpected end of file Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]: INFO listen:request{request_id="019dac26-5ad5-7100-9218-00cd782732bf"}:sign_attached_with_filetype: sigul_pesign_bridge::service: return=Err(an I/O error occurred: unexpected end of file Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]: Caused by: Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]: unexpected end of file) Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]: INFO listen:request{request_id="019dac26-5ad5-7100-9218-00c d782732bf"}:sign_attached_with_filetype: sigul_pesign_bridge::service: close Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]: INFO listen:request{request_id="019dac26-5ad5-7100-9218-00cd782732bf"}: sigul_pesign_bridge::service: return=Err(an I/O error occurred: unexpected end of file Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]: Caused by: Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]: unexpected end of file) Apr 20 18:28:10 buildhw-x86-02 sigul-pesign-bridge[1376146]: INFO listen:request{request_id="019dac26-5ad5-7100-9218-00cd782732bf"}: sigul_pesign_bridge::service: close Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]: 2026-04-20 18:28:11,946 [WARNING] {2251563} koji.TaskManager:1443 TRACEBACK: Traceback (most recent call last): Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]: File "/usr/lib/python3.14/site-packages/koji/daemon.py", line 1421, in runTask Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]: response = (handler.run(),) Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]: ~~~~~~~~~~~^^ Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]: File "/usr/lib/python3.14/site-packages/koji/tasks.py", line 343, in run Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]: return koji.util.call_with_argcheck(self.handler, self.params, self.opts) Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]: File "/usr/lib/python3.14/site-packages/koji/util.py", line 516, in call_with_argcheck Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]: return func(*args, **kwargs) Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]: File "/usr/sbin/kojid", line 1663, in handler Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]: broot.build(fn, arch) Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]: ~~~~~~~~~~~^^^^^^^^^^ Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]: File "/usr/sbin/kojid", line 631, in build Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]: raise koji.BuildError("error building package (arch %s), %s" % Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]: (arch, self._mockResult(rv))) Apr 20 18:28:11 buildhw-x86-02 kojid[2251563]: koji.BuildError: error building package (arch x86_64), mock exited with status 1; see build.log or root.log for more information ``` Let me know if you need/want more info. ... going to leave this open for a bit.
james self-assigned this 2026-04-20 20:00:47 +00:00
jcline commented 2026-04-20 20:24:59 +00:00
Author
Copy link

Okay, so the bridge should be retrying this case instead of stopping, but also something seems weird from the sigul side.

It's returning UNKNOWN_ERROR which unfortunately it uses for lots of things. If it failed in the pesign function there should be an error level log on the signing server that says "pesign command timed out" or "Command failed: %s returned %d (stderr=%s, stdout=%s)" with (hopefully) details on what went wrong.

Okay, so the bridge should be retrying this case instead of stopping, but also something seems weird from the sigul side. It's returning `UNKNOWN_ERROR` which unfortunately it uses for lots of things. If it failed in the pesign function there should be an error level log on the signing server that says `"pesign command timed out"` or `"Command failed: %s returned %d (stderr=%s, stdout=%s)"` with (hopefully) details on what went wrong.
jcline commented 2026-04-20 21:59:12 +00:00
Author
Copy link

I've filed https://bodhi.fedoraproject.org/updates/FEDORA-2026-8fe66d85c2 which changes the bridge behavior to always retry failed requests, even if sigul returns an error code. I think that makes the most sense since if sigul is in a weird state admins can poke it and the build will continue.

If you can't see anything in the sigul server logs we can just retry the kernel build, I suppose, and it might proceed. The shim rebuild also failed so I do suspect something is up with the sigul server: https://kojipkgs.fedoraproject.org//work/tasks/2489/144642489/build.log

I've filed https://bodhi.fedoraproject.org/updates/FEDORA-2026-8fe66d85c2 which changes the bridge behavior to always retry failed requests, even if sigul returns an error code. I think that makes the most sense since if sigul is in a weird state admins can poke it and the build will continue. If you can't see anything in the sigul server logs we can just retry the kernel build, I suppose, and it might proceed. The shim rebuild also failed so I do suspect something is up with the sigul server: https://kojipkgs.fedoraproject.org//work/tasks/2489/144642489/build.log
james commented 2026-04-20 23:06:26 +00:00
Member
Copy link

We are trying to find out who can see the server side logs atm.

Normally Kevin would look, and I assume there's at least one more person who can access them ... but you need more than sysadmin-main to do it, and neither I nor Adam know who else is on the list atm.

We are trying to find out who can see the server side logs atm. Normally Kevin would look, and I assume there's at least one more person who can access them ... but you need more than sysadmin-main to do it, and neither I nor Adam know who else is on the list atm.
james commented 2026-04-21 01:03:04 +00:00
Member
Copy link

Okay, I have access to the server logs now and messages is a lot of this repeated:

Apr 21 00:51:54 sign-vault01 pcscd[16746]: 00002140 openct/proto-t1.c:168:t1_tra
nsceive() T=1 state machine is DEAD. Reset the card first.
Apr 21 00:51:54 sign-vault01 pcscd[16746]: 00000003 ifdwrapper.c:543:IFDTransmit
() Card not transacted: 612
Apr 21 00:51:54 sign-vault01 pcscd[16746]: 00000003 winscard.c:1620:SCardTransmi
t() Card not transacted: 0x80100016

...I assume this means we need to restart sigul, however I don't have that passwd so it'll have to wait a bit longer until one of those people wake up and/or see their messages.

Okay, I have access to the server logs now and messages is a lot of this repeated: ``` Apr 21 00:51:54 sign-vault01 pcscd[16746]: 00002140 openct/proto-t1.c:168:t1_tra nsceive() T=1 state machine is DEAD. Reset the card first. Apr 21 00:51:54 sign-vault01 pcscd[16746]: 00000003 ifdwrapper.c:543:IFDTransmit () Card not transacted: 612 Apr 21 00:51:54 sign-vault01 pcscd[16746]: 00000003 winscard.c:1620:SCardTransmi t() Card not transacted: 0x80100016 ``` ...I assume this means we need to restart sigul, however I don't have that passwd so it'll have to wait a bit longer until one of those people wake up and/or see their messages.
zlopez commented 2026-04-21 07:21:14 +00:00
Owner
Copy link

I restarted sigul and the signing seems to work as it should, but the errors in pcscd didn't go away. But looking at the logs of pcsd this is probably normal.

I restarted sigul and the signing seems to work as it should, but the errors in `pcscd` didn't go away. But looking at the logs of `pcsd` this is probably normal.
zlopez commented 2026-04-21 07:28:19 +00:00
Owner
Copy link

But I found this error in sigul logs:

2026-04-21 02:55:28,814 INFO: Request: 'cert-name' = b'Certificate 4', 'key' = b'fedora-signer', 'op' = b'sign-pe', 'user' = b'sigul-pesign-bridge'
2026-04-21 02:55:29,050 INFO: Connection from CN=sigul-pesign-bridge
2026-04-21 02:55:29,303 INFO: sign_pe_pkcs11: signing request using 'Certificate 4' received
2026-04-21 02:55:29,303 INFO: sign_pe_pkcs11: using key 'pkcs11:model=PKCS%2315%20emulated;manufacturer=Gemalto;serial=740000e2f0ef1078;token=fedora-signer' in 'fedora-signer' token
2026-04-21 02:55:32,727 ERROR: Command failed: ['/usr/bin/pesign', '--sign', '--in', '/tmp/tmpyuwcbl7q', '--out', '/dev/fd/12', '--force', '--certdir', '/tmp/sigul_temp_nss_75tjulo2', '--certificate', 'Certificate 4', '--nofork', '--pinfile', '/dev/fd/10', '--token', 'fedora-signer'] returned 1 (stderr=b'cms_common.c:find_certificate:625: Could not find token "fedora-signer": Success\npesign: Could not find certificate Certificate 4\n', stdout=b'')

To me it seems that key fedora-signer isn't where we expect it.

But I found this error in sigul logs: ``` 2026-04-21 02:55:28,814 INFO: Request: 'cert-name' = b'Certificate 4', 'key' = b'fedora-signer', 'op' = b'sign-pe', 'user' = b'sigul-pesign-bridge' 2026-04-21 02:55:29,050 INFO: Connection from CN=sigul-pesign-bridge 2026-04-21 02:55:29,303 INFO: sign_pe_pkcs11: signing request using 'Certificate 4' received 2026-04-21 02:55:29,303 INFO: sign_pe_pkcs11: using key 'pkcs11:model=PKCS%2315%20emulated;manufacturer=Gemalto;serial=740000e2f0ef1078;token=fedora-signer' in 'fedora-signer' token 2026-04-21 02:55:32,727 ERROR: Command failed: ['/usr/bin/pesign', '--sign', '--in', '/tmp/tmpyuwcbl7q', '--out', '/dev/fd/12', '--force', '--certdir', '/tmp/sigul_temp_nss_75tjulo2', '--certificate', 'Certificate 4', '--nofork', '--pinfile', '/dev/fd/10', '--token', 'fedora-signer'] returned 1 (stderr=b'cms_common.c:find_certificate:625: Could not find token "fedora-signer": Success\npesign: Could not find certificate Certificate 4\n', stdout=b'') ``` To me it seems that key `fedora-signer` isn't where we expect it.
jcline commented 2026-04-21 11:43:07 +00:00
Author
Copy link

@zlopez wrote in #13279 (comment):

But I found this error in sigul logs:

2026-04-21 02:55:28,814 INFO: Request: 'cert-name' = b'Certificate 4', 'key' = b'fedora-signer', 'op' = b'sign-pe', 'user' = b'sigul-pesign-bridge'
2026-04-21 02:55:29,050 INFO: Connection from CN=sigul-pesign-bridge
2026-04-21 02:55:29,303 INFO: sign_pe_pkcs11: signing request using 'Certificate 4' received
2026-04-21 02:55:29,303 INFO: sign_pe_pkcs11: using key 'pkcs11:model=PKCS%2315%20emulated;manufacturer=Gemalto;serial=740000e2f0ef1078;token=fedora-signer' in 'fedora-signer' token
2026-04-21 02:55:32,727 ERROR: Command failed: ['/usr/bin/pesign', '--sign', '--in', '/tmp/tmpyuwcbl7q', '--out', '/dev/fd/12', '--force', '--certdir', '/tmp/sigul_temp_nss_75tjulo2', '--certificate', 'Certificate 4', '--nofork', '--pinfile', '/dev/fd/10', '--token', 'fedora-signer'] returned 1 (stderr=b'cms_common.c:find_certificate:625: Could not find token "fedora-signer": Success\npesign: Could not find certificate Certificate 4\n', stdout=b'')

To me it seems that key fedora-signer isn't where we expect it.

Okay. So the only thing that signs with keys inside that hardware token are packages sent to the secure boot channel (kernel, grub, shim, etc) so I would expect signing to continue working for other things even if this was broken.

That message at the end indicates that the token (e.g. the hardware) is not there, which is not expected. It's unclear to me if restarting pcscd is enough to "reset the card" without doing some digging into the code, or if the card needs a power cycle.

The following should list a token with the label "fedora-signer":

pkcs11-tool --list-token-slot

I suspect it currently won't. You could try restarting pcscd and then running it again to see if it picks up the token. If that doesn't work, I don't have an idea beyond a reboot to power cycle the card.

On Fedora it looks like pcscd is run via a systemd socket with an option to auto exit after inactivity so I'm not sure that restarting it will do anything to solve this, but it seems worth a try.

@zlopez wrote in https://forge.fedoraproject.org/infra/tickets/issues/13279#issuecomment-617523: > But I found this error in sigul logs: > > ```text > 2026-04-21 02:55:28,814 INFO: Request: 'cert-name' = b'Certificate 4', 'key' = b'fedora-signer', 'op' = b'sign-pe', 'user' = b'sigul-pesign-bridge' > 2026-04-21 02:55:29,050 INFO: Connection from CN=sigul-pesign-bridge > 2026-04-21 02:55:29,303 INFO: sign_pe_pkcs11: signing request using 'Certificate 4' received > 2026-04-21 02:55:29,303 INFO: sign_pe_pkcs11: using key 'pkcs11:model=PKCS%2315%20emulated;manufacturer=Gemalto;serial=740000e2f0ef1078;token=fedora-signer' in 'fedora-signer' token > 2026-04-21 02:55:32,727 ERROR: Command failed: ['/usr/bin/pesign', '--sign', '--in', '/tmp/tmpyuwcbl7q', '--out', '/dev/fd/12', '--force', '--certdir', '/tmp/sigul_temp_nss_75tjulo2', '--certificate', 'Certificate 4', '--nofork', '--pinfile', '/dev/fd/10', '--token', 'fedora-signer'] returned 1 (stderr=b'cms_common.c:find_certificate:625: Could not find token "fedora-signer": Success\npesign: Could not find certificate Certificate 4\n', stdout=b'') > ``` > > To me it seems that key `fedora-signer` isn't where we expect it. Okay. So the only thing that signs with keys inside that hardware token are packages sent to the secure boot channel (kernel, grub, shim, etc) so I would expect signing to continue working for other things even if this was broken. That message at the end indicates that the token (e.g. the hardware) is not there, which is not expected. It's unclear to me if restarting pcscd is enough to "reset the card" without doing some digging into the code, or if the card needs a power cycle. The following _should_ list a token with the label "fedora-signer": ```b pkcs11-tool --list-token-slot ``` I suspect it currently won't. You could try restarting pcscd and then running it again to see if it picks up the token. If that doesn't work, I don't have an idea beyond a reboot to power cycle the card. On Fedora it looks like pcscd is run via a systemd socket with an option to auto exit after inactivity so I'm not sure that restarting it will do anything to solve this, but it seems worth a try.
zlopez commented 2026-04-21 11:59:40 +00:00
Owner
Copy link

We found out that this is probably caused by issue with the USB card reader, which contains the key. There is a ticket open for DC folks to unplug and re-plug the card again as the reboot of the machine didn't help.

We found out that this is probably caused by issue with the USB card reader, which contains the key. There is a ticket open for DC folks to unplug and re-plug the card again as the reboot of the machine didn't help.
❤️ 1
james commented 2026-04-21 16:35:12 +00:00
Member
Copy link

Okay, I think this should be working again now.

Okay, I think this should be working again now.
jforbes commented 2026-04-21 16:36:18 +00:00
Copy link

Appreciate it. Kicked off rawhide kernel builds and we will see how it goes.

Appreciate it. Kicked off rawhide kernel builds and we will see how it goes.
gwmngilfen commented 2026-04-21 16:36:29 +00:00
Member
Copy link

@jcline the DC folk have done the needful and @james has restarted sigul - can you retry a build?

@jcline the DC folk have done the needful and @james has restarted sigul - can you retry a build?
jforbes commented 2026-04-21 17:25:16 +00:00
Copy link

Signing has failed again

Signing has failed again
james commented 2026-04-21 20:15:31 +00:00
Member
Copy link

What I know:

  • The logs are full of the above pcscd errors still.
  • There are new pesign errors in sigul_server.log
  • The pcscd errors seemed to have started a couple of days ago with this:
Apr 19 09:11:44 sign-vault01 pcscd[16746]: 00000000 commands.c:1613:CCID_Receive Unknown CCID error: 0x82
Apr 19 09:11:44 sign-vault01 pcscd[16746]: 00000209 openct/proto-t1.c:212:t1_transceive() fatal: transmit/receive failed

I'm pretty sure that I was tired initially and only looked at the sigul_server.log, saw that it was signing packages and decided it might be fixed even though it was doing that fine all the time.

I think there should be USB notifications in dmesg saying it was unplugged and plugged back in, and I don't see those. So maybe the DC did sign-vault02 instead?

Kevin said that we could/should ping pjones, and we should be able to move the USB token from sign-vault02 to sign-vault01 as the USB readers/cards should be identical. Obviously don't want to try that unless nothing else works.

What I know: - The logs are full of the above pcscd errors still. - There are new pesign errors in sigul_server.log - The pcscd errors seemed to have started a couple of days ago with this: ``` Apr 19 09:11:44 sign-vault01 pcscd[16746]: 00000000 commands.c:1613:CCID_Receive Unknown CCID error: 0x82 Apr 19 09:11:44 sign-vault01 pcscd[16746]: 00000209 openct/proto-t1.c:212:t1_transceive() fatal: transmit/receive failed ``` I'm pretty sure that I was tired initially and only looked at the sigul_server.log, saw that it was signing packages and decided it might be fixed even though it was doing that fine all the time. I _think_ there should be USB notifications in dmesg saying it was unplugged and plugged back in, and I don't see those. So maybe the DC did sign-vault02 instead? Kevin said that we could/should ping pjones, and we should be able to move the USB token from `sign-vault02` to `sign-vault01` as the USB readers/cards should be identical. Obviously don't want to try that unless nothing else works.
james commented 2026-04-21 22:08:40 +00:00
Member
Copy link

Worth noting some extra things for anyone else looking at this:

  • I haven't rebooted vault01 since they said they unplugged/replugged the card/reader, mainly in case there was any info. we could glean about if it had been unplugged.
  • I did restart pcscd, and then restarted sigul_server again after that.
  • The serial console was acting a bit weird at times, which I hope wasn't just me.
  • As I mentioned in matrix, and maybe other places, when starting sigul_server you have to enter two passwords. I used "Vault NSS Database password." for the first, which seems very likely to be correct, and for the second I used "Vault Yubikey PIN" which is probably correct? But there's a chance it's "sign vault yubikey puk"? I don't know if the fact it signs rpms means those must be correct.
Worth noting some extra things for anyone else looking at this: - I haven't rebooted vault01 since they said they unplugged/replugged the card/reader, mainly in case there was any info. we could glean about if it had been unplugged. - I did restart pcscd, and then restarted sigul_server again after that. - The serial console was acting a bit weird at times, which I hope wasn't just me. - As I mentioned in matrix, and maybe other places, when starting sigul_server you have to enter two passwords. I used "Vault NSS Database password." for the first, which seems very likely to be correct, and for the second I used "Vault Yubikey PIN" which is probably correct? But there's a chance it's "sign vault yubikey puk"? I don't know if the fact it signs rpms means those must be correct.
gwmngilfen commented 2026-04-22 14:44:34 +00:00
Member
Copy link

I checked with DC-ops and they unplugged/replugged the devices again, and this time I saw it in dmesg, so it was the right machine. However, still no dice:

sign-vault01# pkcs11-tool -L
Available slots:
Slot 0 (0x0): SafeNet eToken 5100 [eToken 5110 SC] 00 00
  (token not recognized)

while 02 lists that and the yubikeys. Since a reboot and a replug haven't helped, I'll ask DC-ops to swap the devices between 01 and 02 and we'll see if that helps.

I checked with DC-ops and they unplugged/replugged the devices again, and this time I saw it in dmesg, so it was the right machine. However, still no dice: ``` sign-vault01# pkcs11-tool -L Available slots: Slot 0 (0x0): SafeNet eToken 5100 [eToken 5110 SC] 00 00 (token not recognized) ``` while 02 lists that *and* the yubikeys. Since a reboot and a replug haven't helped, I'll ask DC-ops to swap the devices between 01 and 02 and we'll see if that helps.
james commented 2026-04-22 19:22:13 +00:00
Member
Copy link

Okay, what we did/know:

  • Eventually worked out there's an extra thumbdrive SafeNet eToken 5100 in both vault01/vault02 which isn't the yubikeys (Greg shows that above).
  • We confirmed the one in vault02 is working
  • We swapped those between machines, so now vault01 is working.
  • In that there's no pscsd errors anymore, and it doesn't show as "token not recognized".
  • However the serial and token name were different, so I changed those to the new ones.
  • We then tried to build and kernel and signing still didn't work.
  • More investigation and it looks like the thumbdrive from vault02 is empty.
  • After trying pesign by hand I got:
authentication failed for token "fedor... (Digital Signature PIN)": The password is locked.
pesign: Could not find certificate Certificate 4
  • Jeremy says that means the PIN was incorrect and we tried bad PINs too many times so now need to use the PUK (PIN Unlock Key) to unlock it ... however it's probably empty anyway, so it doesn't matter.
  • After all of that we are still signing random RPMs.
Okay, what we did/know: - Eventually worked out there's an extra thumbdrive `SafeNet eToken 5100` in both vault01/vault02 which isn't the yubikeys (Greg shows that above). - We confirmed the one in vault02 is working - We swapped those between machines, so now vault01 is working. - In that there's no pscsd errors anymore, and it doesn't show as "token not recognized". - However the serial and token name were different, so I changed those to the new ones. - We then tried to build and kernel and signing **still didn't work**. - More investigation and it **looks** like the thumbdrive from vault02 is empty. - After trying pesign by hand I got: ``` authentication failed for token "fedor... (Digital Signature PIN)": The password is locked. pesign: Could not find certificate Certificate 4 ``` - Jeremy says that means the PIN was incorrect and we tried bad PINs too many times so now need to use the PUK (PIN Unlock Key) to unlock it ... however it's probably empty anyway, so it doesn't matter. - After all of that we are still signing random RPMs.
pjones commented 2026-04-22 20:01:03 +00:00
Copy link

Okay, so:

  • the thumb drive with SafeNet eToken 5100 is where fedora-signer is stored
  • not sure what you mean by "it's probably empty anyway" (what "it"?)
  • yeah, need to unlock with PUK.
Okay, so: - the thumb drive with `SafeNet eToken 5100` is where `fedora-signer` is stored - not sure what you mean by "it's probably empty anyway" (what "it"?) - yeah, need to unlock with PUK.
jcline commented 2026-04-22 20:32:50 +00:00
Author
Copy link

@pjones wrote in #13279 (comment):

Okay, so:

* the thumb drive with `SafeNet eToken 5100` is where `fedora-signer` is stored

* not sure what you mean by "it's probably empty anyway" (what "it"?)

According to pkcs11-tool --list-objects the SafeNet eToken 5100 device had no objects in it. I expected it to have public keys and certs visible without providing --login. For soft tokens and Yubikeys I've used it works that way:

$ pkcs11-tool --list-objects --type cert
Using slot 0 with a present token (0x0)
Certificate Object; type = X.509 cert
  label:      Certificate for PIV Authentication
  ...

Do you happen to remember if the token behaves like that, or is it marking all objects as private?

@pjones wrote in https://forge.fedoraproject.org/infra/tickets/issues/13279#issuecomment-674690: > Okay, so: > > * the thumb drive with `SafeNet eToken 5100` is where `fedora-signer` is stored > > * not sure what you mean by "it's probably empty anyway" (what "it"?) According to `pkcs11-tool --list-objects` the `SafeNet eToken 5100` device had no objects in it. I expected it to have public keys and certs visible without providing `--login`. For soft tokens and Yubikeys I've used it works that way: ``` $ pkcs11-tool --list-objects --type cert Using slot 0 with a present token (0x0) Certificate Object; type = X.509 cert label: Certificate for PIV Authentication ... ``` Do you happen to remember if the token behaves like that, or is it marking all objects as private?
pjones commented 2026-04-22 20:44:00 +00:00
Copy link

@jcline wrote in #13279 (comment):

Do you happen to remember if the token behaves like that, or is it marking all objects as private?

I don't recall, but I don't think I've ever observed any consistency with these devices or tools.

@jcline wrote in https://forge.fedoraproject.org/infra/tickets/issues/13279#issuecomment-674692: > Do you happen to remember if the token behaves like that, or is it marking all objects as private? I don't recall, but I don't think I've ever observed any consistency with these devices or tools.
james commented 2026-04-22 23:22:19 +00:00
Member
Copy link

This is solved now: https://koji.fedoraproject.org/koji/taskinfo?taskID=144715871

This is solved now: https://koji.fedoraproject.org/koji/taskinfo?taskID=144715871
james closed this issue 2026-04-22 23:22:20 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
announcement

Archived

  
anubis
Issues related to anubis

  
authentication
Authentication related issues

  
aws
Issues related to Amazon Web Services

  
backlog
Tasks that will take longer then ~4 hours to do

Archived

  
blocked
Issue is blocked

Archived

  
bodhi
Issues with Bodhi

  
ci
Continuous Integration

  
cloud
Items in FedoraInfraCloud or COmmunishift

  
communishift
issues/questions about os.edorainfracloud.org ( Community OpenShift)

  
copr
COPR system (run by other team)

  
database
Issues related to databases

  
day-to-day
These are small self contained tasks that are part of normal operations, but are async from sprint planning.

  
dc-move
Items related to datacenter move

Archived

  
deprecated
Tools which Infrastructure may not be able to fix

Archived

  
dev
Tasks requiring some dev work

  
discourse
outsourced tool

  
dns
Changes or problems with DNS

  
downloads
rsync,dl.fp.o,AWS,torrents

  
easyfix

Archived

  
epel
Issues relating to epel

Archived

  
firmitas

   
forgejo_migration
Request migration to the new forge

  
Gain
High
 task that gets us a lot / is important / makes many people happy

Archived

  
Gain
Low
 tasks that gets us little / isn't very important / makes only 1 person or few happy

Archived

  
Gain
Medium
 task that gets us some / is somewhat important / makes some people happy

Archived

  
gitlab

   
greenwave
Issues with greenwave

  
hardware
Problem is with Fedora Infrastructure Owned/Leased Hardware

  
help wanted

   
high-trouble
task that is complex/long/difficult

Archived

  
koji
Issues with the build system

  
koschei
Items needing Koschei team

  
lists
issues with lists

  
low-trouble
task that is easy/short

Archived

  
medium-trouble
task that is medium complex / difficult

Archived

  
mirrorlists
Way to tell systems where downloads are

  
monitoring

   
Needs investigation

   
odcs
On Demand Compose Service issues

Archived

  
OpenShift
Issues with OpenShift

  
ops
ops problem now...

Archived

  
outage

   
packager_workflow_blocker
Blocks the packager workflow

Archived

  
pagure
Issues with pagure.io

  
permissions
Items which need permissions set in outside tools

  
Priority
Needs Review
 Priority of Needs Review

Archived

  
Priority
Next Meeting
 Priority of Next Meeting

Archived

  
Priority
🔥 URGENT 🔥
 Priority of 🔥 URGENT 🔥

Archived

  
Priority
Waiting on Assignee
 Priority of Waiting on Assignee

Archived

  
Priority
Waiting on External
 Priority of Waiting on External

Archived

  
Priority
Waiting on Reporter
 Priority of Waiting on Reporter

Archived

  
rabbitmq
issues with the message broker

  
release-monitoring
release-monitoring.org (Anitya)

  
releng
For problems which should be in the releng ticket system

Archived

  
request-for-resources

   
s390x
The secondary architecture hardware and problems related with it

  
security

   
SMTP
Email issues outside of lists (DMARC, SPF, aliases)

  
sprint-0

Archived

  
sprint-1

Archived

  
src.fp.o
Issues related to src.fedoraproject.org

  
staging
Issues in staging

  
unfreeze
Patches to be applied after freeze is lifted

  
waiverdb
Issues with waiverdb

  
websites-general
General items with a website focus

  
wiki
Issues with the Wiki

  
Backlog Status
Needs Review
 Work items that have been created or updated but need review, refinement, or approval before they can be considered ready for sprint planning.

  
Backlog Status
Ready
 Work items that have been reviewed, refined, and are ready to be committed to a sprint. These items have clear requirements and can be worked on immediately when capacity is available.

  
chore
Work that needs to be done but doesn't add new functionality or fix bugs. Maintenance tasks that keep the system running smoothly and the codebase clean.

  
documentation
Work focused on creating, updating, or improving documentation for users, developers, or system administrators.

  
points
01
 A task that is simple, small, and has very little uncertainty. It should be easy enough for any team member to pick up and complete quickly, without a lot of discussion or research.

  
points
02
 This is a task that is slightly more complex or larger than a "1." It might involve two or three simple steps, but the outcome is still very clear and there's little uncertainty.

  
points
03
 Medium-sized task with some noticeable complexity. It might require a bit more thought or collaboration. The steps are well-defined, but it's not a trivial change.

  
points
05
 A significant piece of work. It has more complexity and a higher degree of uncertainty than smaller tasks. It might require some research, a design discussion, or collaboration across multiple teams.

  
points
08
 Complex task with a lot of uncertainty. It's often the largest size that a team will consider for a single sprint. It likely requires a detailed breakdown and multiple team members to complete.

  
points
13
 Any task that feels like it's a "13" or more is considered an Epic. These are too large and uncertain to be worked on in a single sprint and should be broken down.

  
Priority
High
 This is for work that is critical and needs to be addressed in the next sprint. It's often for urgent bugs, important features, or a major piece of technical debt that is blocking other work.

  
Priority
Low
 This is for work that is good to do but not urgent. It's a candidate for being worked on when there's extra time or when it rises in importance. It can stay in the backlog for a longer period.

  
Priority
Medium
 This is for important work that should be done soon, but isn't as urgent as a high-priority item. It will likely be considered for the next 1-2 sprints.

  
Sprint Status
Blocked
 Work items that cannot progress due to external dependencies, blockers, or issues that need resolution.

  
Sprint Status
Done
 Work items that have been completed, reviewed, and integrated. No further work is needed within the current sprint.

  
Sprint Status
In Progress
 Work items currently being worked on by a team member. Active development, research, or implementation is happening.

  
Sprint Status
Review
 Work items that are complete but need review, testing, or approval before being marked as done.

  
Sprint Status
To Do
 Work items that are committed to the sprint but haven't been started yet. These items are ready to be picked up by team members.

  
Technical Debt
Work that addresses shortcuts, workarounds, or suboptimal implementations that were done previously to meet deadlines. These items improve code quality, maintainability, or performance without adding new features.

  
Work Item
Bug
 Issues where the system is not working as intended or designed. Bugs represent deviations from expected behavior.

  
Work Item
Epic
 Large initiatives that are broken down into multiple smaller work items (user stories, tasks, bugs). Epics represent significant features or projects that span multiple sprints.

  
Work Item
Spike
 Time-boxed research, investigation, or proof-of-concept work to reduce uncertainty or explore solutions. Spikes help inform future development decisions.

  
Work Item
Task
 Used to break down larger user stories into smaller, manageable implementation pieces. They focus on the specific technical steps needed to deliver the user story's value.

  
Work Item
User Story
 User-facing features or functionality that can be framed from a user perspective, describing what someone wants to accomplish and why.

No labels
announcement
anubis
authentication
aws
backlog
blocked
bodhi
ci
cloud
communishift
copr
database
day-to-day
dc-move
deprecated
dev
discourse
dns
downloads
easyfix
epel
firmitas
forgejo_migration
Gain
High
Gain
Low
Gain
Medium
gitlab
greenwave
hardware
help wanted
high-trouble
koji
koschei
lists
low-trouble
medium-trouble
mirrorlists
monitoring
Needs investigation
odcs
OpenShift
ops
outage
packager_workflow_blocker
pagure
permissions
Priority
Needs Review
Priority
Next Meeting
Priority
🔥 URGENT 🔥
Priority
Waiting on Assignee
Priority
Waiting on External
Priority
Waiting on Reporter
rabbitmq
release-monitoring
releng
request-for-resources
s390x
security
SMTP
sprint-0
sprint-1
src.fp.o
staging
unfreeze
waiverdb
websites-general
wiki
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
james
6 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
infra/tickets#13279
No description provided.