message plugin for hub #4627

Open
mikem wants to merge 25 commits from mikem/koji:message-plugin into master
Owner

This plugin enables client-side message bus connections. It works similarly to the existing protonmsg plugin, but stores messages to the database. A dedicated client can use the message.poll() and message.ack() methods to fetch and acknowledge messages from the queue. These calls require the new message permission.

This plugin enables client-side message bus connections. It works similarly to the existing protonmsg plugin, but stores messages to the database. A dedicated client can use the `message.poll()` and `message.ack()` methods to fetch and acknowledge messages from the queue. These calls require the new `message` permission.
skip update on duplicate ack
All checks were successful
/ flake8 (pull_request) Successful in 18s
/ bandit (pull_request) Successful in 20s
/ tests (fedora-43@sha256:cf1ef55da0d7dd9251b79263c4d1e2350d4291fcd6dc54a1e751d3fe2cf4614d) (pull_request) Successful in 1m15s
/ tests (fedora-44@sha256:5e1187ee4536674e08961cbe6b1b84dfc9df15a513dee328365909be5a0d5ed8) (pull_request) Successful in 1m11s
/ tests (centos-10@sha256:1f53b2b040e449b0cfce526f1a221e6dbd6648506270220ea333728a6b52a90b) (pull_request) Successful in 1m55s
/ tests (fedora-rawhide@sha256:5e1187ee4536674e08961cbe6b1b84dfc9df15a513dee328365909be5a0d5ed8) (pull_request) Successful in 1m2s
/ tests (centos-9@sha256:c5fe30a62d4b62b46acdf764bf1f7293e9fa4287972ec6a86ccb43209646dc32) (pull_request) Successful in 2m43s
25e6d30b49
mikem added this to the 1.37 milestone 2026-07-22 20:32:10 +00:00
add some documentation
All checks were successful
/ flake8 (pull_request) Successful in 16s
/ bandit (pull_request) Successful in 19s
/ tests (fedora-44@sha256:5e1187ee4536674e08961cbe6b1b84dfc9df15a513dee328365909be5a0d5ed8) (pull_request) Successful in 1m10s
/ tests (fedora-43@sha256:cf1ef55da0d7dd9251b79263c4d1e2350d4291fcd6dc54a1e751d3fe2cf4614d) (pull_request) Successful in 1m30s
/ tests (centos-10@sha256:1f53b2b040e449b0cfce526f1a221e6dbd6648506270220ea333728a6b52a90b) (pull_request) Successful in 1m58s
/ tests (centos-9@sha256:c5fe30a62d4b62b46acdf764bf1f7293e9fa4287972ec6a86ccb43209646dc32) (pull_request) Successful in 2m43s
/ tests (fedora-rawhide@sha256:5e1187ee4536674e08961cbe6b1b84dfc9df15a513dee328365909be5a0d5ed8) (pull_request) Successful in 56s
65ae40a218
Owner

Do we want to handle multiple clients/threads consuming the same data? It would need some locking FOR UPDATE SKIP LOCKED, etc. Maybe docs could be extended that client is using esclusive session if we don't want to support this.

Theoretically there could be also the variant of many clients for many messaging backends. So, queue could be extended to have some tags (like client1 acks it was successful in sending to backend1 and add user_id1 to ack field, while client2 can still fetch/send such message). We wouldn't know how many clients are going to consume the messages, but max_age would clean it later anyway, so it wouldn't be there forever.

Do we want to handle multiple clients/threads consuming the same data? It would need some locking FOR UPDATE SKIP LOCKED, etc. Maybe docs could be extended that client is using esclusive session if we don't want to support this. Theoretically there could be also the variant of many clients for many messaging backends. So, queue could be extended to have some tags (like client1 acks it was successful in sending to backend1 and add user_id1 to ack field, while client2 can still fetch/send such message). We wouldn't know how many clients are going to consume the messages, but ``max_age`` would clean it later anyway, so it wouldn't be there forever.
@ -0,0 +207,4 @@
'arch': 'x86_64'}
sigkey = ''
message.prep_rpm_sign('postRPMSign', sigkey=sigkey, sighash='fedcba9876543210',
build=build, rpm=rpm)
Owner

Some assert here, that message is not in the queue?

Some assert here, that message is not in the queue?
@ -0,0 +509,4 @@
message.parse_bool_str(val)
# the end
Owner

Maybe test for double-ack?

Maybe test for double-ack?
Author
Owner

As coded, a double ack is a no-op (unless it happens to trigger an auto clean)

There is no meaningful way to test double ack behavior here. The ack state is in the db table, and the behavior is handled in the update query, which we are mocking. The update clauses always include sent IS FALSE. We could only be able to test the behavior if we had a working db for the test.

As coded, a double ack is a no-op (unless it happens to trigger an auto clean) There is no meaningful way to test double ack behavior here. The ack state is in the db table, and the behavior is handled in the update query, which we are mocking. The update clauses always include `sent IS FALSE`. We could only be able to test the behavior if we had a working db for the test.
Author
Owner

@tkopecek wrote in #4627 (comment):

Do we want to handle multiple clients/threads consuming the same data? It would need some locking FOR UPDATE SKIP LOCKED, etc. Maybe docs could be extended that client is using esclusive session if we don't want to support this.

Theoretically there could be also the variant of many clients for many messaging backends. So, queue could be extended to have some tags (like client1 acks it was successful in sending to backend1 and add user_id1 to ack field, while client2 can still fetch/send such message). We wouldn't know how many clients are going to consume the messages, but max_age would clean it later anyway, so it wouldn't be there forever.

This is deliberately written for a single dedicated client to handle the queue. I've tried to use the term "dedicated client" numerous places to emphasize this. I went this way because it is simpler to implement. Otherwise we need to track sent state per client.

That said, I certainly have considered allowing for multiple clients. I don't think it's needed for our purposes, but it could be useful in the future.

@tkopecek wrote in https://forge.fedoraproject.org/koji/koji/pulls/4627#issuecomment-1080940: > Do we want to handle multiple clients/threads consuming the same data? It would need some locking FOR UPDATE SKIP LOCKED, etc. Maybe docs could be extended that client is using esclusive session if we don't want to support this. > > Theoretically there could be also the variant of many clients for many messaging backends. So, queue could be extended to have some tags (like client1 acks it was successful in sending to backend1 and add user_id1 to ack field, while client2 can still fetch/send such message). We wouldn't know how many clients are going to consume the messages, but `max_age` would clean it later anyway, so it wouldn't be there forever. This is deliberately written for a single dedicated client to handle the queue. I've tried to use the term "dedicated client" numerous places to emphasize this. I went this way because it is simpler to implement. Otherwise we need to track sent state per client. That said, I certainly have considered allowing for multiple clients. I don't think it's needed for our purposes, but it could be useful in the future.
add assert for test_prep_rpm_sign_no_sigkey
All checks were successful
/ flake8 (pull_request) Successful in 15s
/ bandit (pull_request) Successful in 17s
/ tests (fedora-44@sha256:5e1187ee4536674e08961cbe6b1b84dfc9df15a513dee328365909be5a0d5ed8) (pull_request) Successful in 1m9s
/ tests (fedora-43@sha256:cf1ef55da0d7dd9251b79263c4d1e2350d4291fcd6dc54a1e751d3fe2cf4614d) (pull_request) Successful in 1m13s
/ tests (centos-10@sha256:1f53b2b040e449b0cfce526f1a221e6dbd6648506270220ea333728a6b52a90b) (pull_request) Successful in 1m51s
/ tests (fedora-rawhide@sha256:5e1187ee4536674e08961cbe6b1b84dfc9df15a513dee328365909be5a0d5ed8) (pull_request) Successful in 1m2s
/ tests (centos-9@sha256:c5fe30a62d4b62b46acdf764bf1f7293e9fa4287972ec6a86ccb43209646dc32) (pull_request) Successful in 2m39s
d10ea57696
tkopecek approved these changes 2026-08-14 08:55:07 +00:00
update docs
All checks were successful
/ flake8 (pull_request) Successful in 15s
/ bandit (pull_request) Successful in 18s
/ tests (fedora-44@sha256:5e1187ee4536674e08961cbe6b1b84dfc9df15a513dee328365909be5a0d5ed8) (pull_request) Successful in 1m3s
/ tests (fedora-43@sha256:cf1ef55da0d7dd9251b79263c4d1e2350d4291fcd6dc54a1e751d3fe2cf4614d) (pull_request) Successful in 1m6s
/ tests (centos-10@sha256:1f53b2b040e449b0cfce526f1a221e6dbd6648506270220ea333728a6b52a90b) (pull_request) Successful in 1m47s
/ tests (fedora-rawhide@sha256:5e1187ee4536674e08961cbe6b1b84dfc9df15a513dee328365909be5a0d5ed8) (pull_request) Successful in 1m6s
/ tests (centos-9@sha256:c5fe30a62d4b62b46acdf764bf1f7293e9fa4287972ec6a86ccb43209646dc32) (pull_request) Successful in 2m49s
3a29ef5375
Author
Owner

Alternate implementation - koji/koji!4652

Alternate implementation - koji/koji!4652
All checks were successful
/ flake8 (pull_request) Successful in 15s
/ bandit (pull_request) Successful in 18s
/ tests (fedora-44@sha256:5e1187ee4536674e08961cbe6b1b84dfc9df15a513dee328365909be5a0d5ed8) (pull_request) Successful in 1m3s
/ tests (fedora-43@sha256:cf1ef55da0d7dd9251b79263c4d1e2350d4291fcd6dc54a1e751d3fe2cf4614d) (pull_request) Successful in 1m6s
/ tests (centos-10@sha256:1f53b2b040e449b0cfce526f1a221e6dbd6648506270220ea333728a6b52a90b) (pull_request) Successful in 1m47s
/ tests (fedora-rawhide@sha256:5e1187ee4536674e08961cbe6b1b84dfc9df15a513dee328365909be5a0d5ed8) (pull_request) Successful in 1m6s
/ tests (centos-9@sha256:c5fe30a62d4b62b46acdf764bf1f7293e9fa4287972ec6a86ccb43209646dc32) (pull_request) Successful in 2m49s
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u message-plugin:mikem-message-plugin
git switch mikem-message-plugin

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch master
git merge --no-ff mikem-message-plugin
git switch mikem-message-plugin
git rebase master
git switch master
git merge --ff-only mikem-message-plugin
git switch mikem-message-plugin
git rebase master
git switch master
git merge --no-ff mikem-message-plugin
git switch master
git merge --squash mikem-message-plugin
git switch master
git merge --ff-only mikem-message-plugin
git switch master
git merge mikem-message-plugin
git push origin master
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
koji/koji!4627
No description provided.