Possible intermittent communication issue between Kerberos and Packit #13539
Labels
No labels
announcement
anubis
authentication
aws
backlog
blocked
bodhi
ci
cloud
communishift
copr
database
day-to-day
dc-move
deprecated
dev
discourse
dns
downloads
easyfix
epel
firmitas
forgejo_migration
Gain
High
Gain
Low
Gain
Medium
gitlab
greenwave
hardware
help wanted
high-trouble
koji
koschei
lists
low-trouble
medium-trouble
mirrorlists
monitoring
Needs investigation
odcs
OpenShift
ops
outage
packager_workflow_blocker
pagure
permissions
Priority
Needs Review
Priority
Next Meeting
Priority
🔥 URGENT 🔥
Priority
Waiting on Assignee
Priority
Waiting on External
Priority
Waiting on Reporter
rabbitmq
release-monitoring
releng
request-for-resources
s390x
security
SMTP
sprint-0
sprint-1
src.fp.o
staging
unfreeze
waiverdb
websites-general
wiki
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
No milestone
No project
No assignees
6 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
infra/tickets#13539
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description of request
For the past few weeks I noticed that scratch builds on PRs have been failing with:
If we wait some time and re-request the build, it works fine.
Example of PR that still has the failing build: https://src.fedoraproject.org/rpms/llvm/pull-request/638
Could you take a look at this, please?
Today's ELN composes (the most recent attempt was within the hour) have failed with
koji: GSSAPIAuthError: unable to obtain a session (gssapi auth failed: requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://koji.fedoraproject.org/kojihub/ssllogin)Here is a debug log from a worker running
kinit packit@FEDORAPROJECT.ORG -k -t /secrets/fedora.keytab:Is it always the initial scratch build from packit that gets the error? Does it always happen or sporadic?
Are the eln composes always failing? I see the last two I guess... I do see openssl was updated on the eln composer, wonder if that could be related?
For packit, is the time set correctly where it's running? Can you give a exact timestamp for a Preauthentication failure so I can try and match it in logs?
Sorry, pasting again with timestamps (UTC):
It definitely doesn't happen every time, here is one successful kinit for comparison:
@kevin wrote in #13539 (comment):
I haven't seen a failure after retrying yet, but I don't think I have enough data to take conclusions yet.
It's definitely sporadic. Some PRs work well.
I've been digging through logs, but I can't seem to come up with enough to make a theory. ;(
I do see some:
krb5kdc5910: preauth (encrypted_timestamp) verify failure: Preauthentication failed
but not sure they line up. Thats why I asked about time being in sync...
Also this:
One thing that I recently dealt with was the different kvno between the server and the local key.
On IPA side we have this:
Could you try to run
klist -kte /path/to/packit.keytaband share the kvno for the generated packit keytab?In that case the kvno doesn't seem to be the issue.
FWIW, I'm seeing it too on RISC-V Koji; query operations like
riscv-koji list-tagswork, but scratch builds and evenriscv-koji hellofails.This works:
Can you attach output of:
KRB5_TRACE=/dev/stdout riscv-koji hello
(list-tags I think doesn't require auth)
This is what Claude Sonnet 5 deduced from the logs, if that helps:
What differs between replicas
preauth-types list only contains
PA-ENCRYPTED-CHALLENGE (138)(noSPAKE). The client does the keytab-based encrypted-challenge exchange
once and gets a ticket immediately.
includes
PA-SPAKE (151). The client engages SPAKE first (modulereports success), the KDC asks for one more round
("Additional pre-authentication required" again), the client then does
encrypted-challenge — but this time the final AS-REP comes back as
-1765328360 / Preauthentication failedinstead of succeeding.So doing SPAKE and then encrypted-challenge in the same exchange against
ipa01 corrupts/mismatches the preauth state and the KDC rejects it,
whereas going straight to encrypted-challenge (as ipa03 requires) works
fine.
Conclusion
This isn't the keytab, client config, or network — it's a
version/configuration mismatch between the Fedora IPA/KDC replicas behind
id.fedoraproject.org.ipa01is advertisingPA-SPAKEin a way thatbreaks the encrypted-challenge fallback for keytab-based clients, while
ipa03(and presumably other replicas) don't offer SPAKE and work fine.Since replicas are picked essentially at random per attempt, this produces
intermittent failures — consistent with them starting ~1 week ago, likely
from a partial rollout/upgrade across the replicas.
I'm not a fan of meat puppetting, but that does indeed seem like something to look further into.
All our ipa servers are installed the same exact way via ansible, so there should not actually be any differences, but will look and see if I can find anything. Thanks for the pointer.
I can't seem to find any config differences between them so far. ;(
I did restart krb5kdc and httpd on both of them... can you test and see if that changed anything?
(On the theory that ipa01 had a old config in the running service and needed a restart).
@kevin So far so good, I can't reproduce the error locally anymore and the last occurence in Packit Service happened about 30 minutes before your last message 🎉
Excellent. So, lets hope it was just a restart needed then.
Do reopen or file a new ticket if anyone still sees anything or sees anything new.