deploy a Draupnir instance for our matrix servers

kevin commented

2024-10-19 21:14:20 +00:00

Owner

It would be nice to have a Draupnir bot for at least our popular rooms ( #fedora at least).

https://github.com/the-draupnir-project/Draupnir

IMHO, we should deploy it in the openshift cluster and use a :fedoraproject.org userid for it.

There's a few folks interested in helping out manage and wrangle it too.

Perhaps Ryan would have some cycles for this setup and onboard interested matrix admins?

CC: @farchord @ryanlerch @jflory7

It would be nice to have a Draupnir bot for at least our popular rooms ( #fedora at least). https://github.com/the-draupnir-project/Draupnir IMHO, we should deploy it in the openshift cluster and use a :fedoraproject.org userid for it. There's a few folks interested in helping out manage and wrangle it too. Perhaps Ryan would have some cycles for this setup and onboard interested matrix admins? CC: @farchord @ryanlerch @jflory7

farchord commented

2024-10-19 21:16:00 +00:00

Oh yeah I would love to! IF we can but not right now, I can at least get the ball rolling/bot setup on a spare box that I have.

But like I told kevin, I only have it til february.

Oh yeah I would love to! IF we can but not right now, I can at least get the ball rolling/bot setup on a spare box that I have. But like I told kevin, I only have it til february.

farchord commented

2024-10-19 21:23:42 +00:00

Btw I'm thinking of also offering other fedora matrix server channels to invite the bot once we got it setup. It would be an additional tool.

kevin commented

2024-10-21 19:04:35 +00:00

Author

Owner

So, we have a community member that has spun up an instance for us to use at least for the main fedora channel. ;)

So, this is not urgent, but I would still like us to run our own.

So, we have a community member that has spun up an instance for us to use at least for the main fedora channel. ;) So, this is not urgent, but I would still like us to run our own.

kevin commented

2024-10-21 19:04:36 +00:00

Author

Owner

Metadata Update from @kevin:

Issue priority set to: Waiting on Assignee (was: Needs Review)
Issue tagged with: low-gain, low-trouble, ops

**Metadata Update from @kevin**: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: low-gain, low-trouble, ops

farchord commented

2024-10-21 19:59:05 +00:00

Agreed. Fairly certain that the community member doesn't mind simply copying it over once we do have a tiny corner of da web ready for it.

zlopez commented

2025-07-17 17:46:49 +00:00

Owner

Metadata Update from @zlopez:

Issue marked as depending on: #12323
Issue tagged with: blocked

**Metadata Update from @zlopez**: - Issue marked as depending on: #12323 - Issue tagged with: blocked

zlopez added this to the sprint ( 2026-01-26 to 2026-02-09 ) project

2026-01-22 17:52:11 +00:00

zlopez self-assigned this

2026-01-26 15:34:31 +00:00

kevin modified the project from sprint ( 2026-01-26 to 2026-02-09 ) to Sprint ( 2026-01-26 to 2026-02-08 )

2026-01-26 17:29:23 +00:00

phsmoura removed the

blocked

label

2026-01-29 18:05:22 +00:00

kevin referenced this issue

2026-01-30 16:03:12 +00:00

[MATRIX] Deploy Draupnir moderation bot for Fedora Matrix room #13094

zlopez commented

2026-02-02 14:26:21 +00:00

Owner

Why the blocked label was removed, #12323 is till not closed.

Why the blocked label was removed, https://forge.fedoraproject.org/infra/tickets/issues/12323 is till not closed.

kevin commented

2026-02-02 17:34:08 +00:00

Author

Owner

Well, I guess we thought we should only do this after we moved the server over to an ESS one run by us, but it really doesn't need to depend on that. We can do this anytime...

jflory7 commented

2026-02-02 22:42:36 +00:00

Member

Do we want to keep this issue, or track the project in #13094? There is useful data in that issue like the existing Draupnir config provided by @rorysys. I'm currently on the go right now, otherwise I'll copy over the insights and the config from that other issue later.

kevin commented

2026-02-02 22:54:08 +00:00

Author

Owner

Yeah, I don't care which one really, we can move information from the other one or just close this one and track there.
As long as there is only one ticket and it has all the info it. ;)

Yeah, I don't care which one really, we can move information from the other one or just close this one and track there. As long as there is only one ticket and it has all the info it. ;)

misc commented

2026-02-06 17:35:03 +00:00

Member

I just finished a working role for draupnir: https://gitlab.com/osci/ansible-role-openshift-apps/-/tree/main/draupnir?ref_type=heads

It can be deployed like this (that's a note for future me):

---

- name: "Deploy OpenShift apps"
  hosts: openshift_controller
  module_defaults:
    group/kubernetes.core.k8s:
      api_key: "{{ openshift_dedicated_api_key }}"
      host: "{{ openshift_dedicated_host }}"
  tasks:
    - name: "Deploy bot on OpenShift"
      include_role:
        name: openshift-apps/draupnir
      vars:
        synapse_access_token: "yourtoken"
        project: openshift-tenant
        homeserver_url: https://matrix.example.org/
        management_room: '!aaaaa:example.org'
        wordlist:
          - 'ganon'
        storage_size: '3Gi'

(in case you wonder why Ganon is a blocked word, I called the test bot vah_naboris and the control room was named #hyrule:example.org)

Now, we have to decide where it will run.

I can either 1) copy the code to Fedora repo and have it under Fedora control (and help maintain if needed, assuming I have enough access to debug stuff)
2) keep the bot on our openshift cluster and be responsible for it.

I will have to run something on our cluster anyway, because ansible folks want a bot too, and I guess the devconf folks as well, so it wouldn't be much effort to cut and paste and run 2 or 3 instances.

In any case, I need:

a access key
the management room (the id, it seems the alias didn't work when i tested)
the words list to block if any
the homeserver url

I just finished a working role for draupnir: https://gitlab.com/osci/ansible-role-openshift-apps/-/tree/main/draupnir?ref_type=heads It can be deployed like this (that's a note for future me): ``` --- - name: "Deploy OpenShift apps" hosts: openshift_controller module_defaults: group/kubernetes.core.k8s: api_key: "{{ openshift_dedicated_api_key }}" host: "{{ openshift_dedicated_host }}" tasks: - name: "Deploy bot on OpenShift" include_role: name: openshift-apps/draupnir vars: synapse_access_token: "yourtoken" project: openshift-tenant homeserver_url: https://matrix.example.org/ management_room: '!aaaaa:example.org' wordlist: - 'ganon' storage_size: '3Gi' ``` (in case you wonder why Ganon is a blocked word, I called the test bot vah_naboris and the control room was named #hyrule:example.org) Now, we have to decide where it will run. I can either 1) copy the code to Fedora repo and have it under Fedora control (and help maintain if needed, assuming I have enough access to debug stuff) 2) keep the bot on our openshift cluster and be responsible for it. I will have to run something on our cluster anyway, because ansible folks want a bot too, and I guess the devconf folks as well, so it wouldn't be much effort to cut and paste and run 2 or 3 instances. In any case, I need: - a access key - the management room (the id, it seems the alias didn't work when i tested) - the words list to block if any - the homeserver url

jflory7 commented

2026-02-06 17:54:30 +00:00

Member

Here is a copy of the current Draupnir config, as provided by @rorysys:

---
admin:
  enableMakeRoomAdminCommand: false
autojoinOnlyIfManager: true
automaticallyRedactForReasons:
- '*'
backgroundDelayMS: 10
commands:
  ban:
    defaultReasons:
    - spam
dataPath: /var/lib/draupnir
fasterMembershipChecks: true
homeserverUrl: https://matrix.rory.gay
managementRoom: '#draupnir-fedora-mgmt:rory.gay'
pollReports: false
protections:
  wordlist:
    minutesBeforeTrusting: 0
    words:
    - https://postimg.cc/
rawHomeserverUrl: https://matrix.rory.gay
recordIgnoredInvites: true

Here is a copy of the current Draupnir config, as provided by @rorysys: ```yaml --- admin: enableMakeRoomAdminCommand: false autojoinOnlyIfManager: true automaticallyRedactForReasons: - '*' backgroundDelayMS: 10 commands: ban: defaultReasons: - spam dataPath: /var/lib/draupnir fasterMembershipChecks: true homeserverUrl: https://matrix.rory.gay managementRoom: '#draupnir-fedora-mgmt:rory.gay' pollReports: false protections: wordlist: minutesBeforeTrusting: 0 words: - https://postimg.cc/ rawHomeserverUrl: https://matrix.rory.gay recordIgnoredInvites: true ```

jflory7 commented

2026-02-06 17:59:19 +00:00

Member

Copy-pasting Emma's comment on the other issue into this one, for reference:

@rorysys wrote…

I noted that you want to run the bot on fp.o, however I would recommend at least hosting some instance on fedora.im, even if it isn't the instance protecting the rooms, to handle blocking invites for fedora.im users - this cannot be done across instances, as it requires admin access on the linked homeserver.

I expect we will not be able to do this before we migrate the fedora.im homeserver into our own infrastructure.

Copy-pasting [Emma's comment on the other issue](https://forge.fedoraproject.org/infra/tickets/issues/13094#issuecomment-361152) into this one, for reference: > _**@rorysys wrote…**_ > > > I noted that you want to run the bot on fp.o, however I would recommend at least hosting some instance on fedora.im, even if it isn't the instance protecting the rooms, to handle blocking invites for fedora.im users - this cannot be done across instances, as it requires admin access on the linked homeserver. I expect we will not be able to do this _before_ we migrate the `fedora.im` homeserver into our own infrastructure.

kevin commented

2026-02-06 18:47:44 +00:00

Author

Owner

Thanks misc!

I'd personally really prefer we run it in our openshift, just because it will be so visible and we have a number of folks around at various times.

Is it possible for it to use both a fedora.im account and a fedoraproject.org account?
If not, we should probibly put it on fedora.im for the reason Emma said.

I'm not sure this will need to block on server migration, I would think we could do it anytime?

Thanks misc! I'd personally really prefer we run it in our openshift, just because it will be so visible and we have a number of folks around at various times. Is it possible for it to use both a fedora.im account and a fedoraproject.org account? If not, we should probibly put it on fedora.im for the reason Emma said. I'm not sure this will need to block on server migration, I would think we could do it anytime?

misc commented

2026-02-06 20:57:18 +00:00

Member

Technically there is nothing that prevent having multiple instance, each on a different server, so each with specific access and/or config. I do not think it would be a great idea since it make harder to operate by moderators, but if there is specific access that requires a specific account, then this could be done.

jflory7 commented

2026-02-06 21:42:30 +00:00

Member

My understanding was that the fedoraproject.org server was always the easy part, since it is easy for us today to make new accounts on that homeserver.

But since we don't have SSH/root access on the fedora.im homeserver instance, this is why I assumed we would need to wait until we get the fedora.im homeserver running in our own infrastructure in order to get the fedora.im homeserver-level integration with Draupnir. (FWIW, we do not even have this functionality yet today, which is why the fedoraproject.org account with a "normal" username and password seems feasible right now.)

My understanding was that the `fedoraproject.org` server was always the easy part, since it is easy for us today to make new accounts on that homeserver. But since we don't have SSH/root access on the `fedora.im` homeserver instance, this is why I assumed we would need to wait until we get the `fedora.im` homeserver running in our own infrastructure in order to get the `fedora.im` homeserver-level integration with Draupnir. (FWIW, we do not even have this functionality yet today, which is why the `fedoraproject.org` account with a "normal" username and password seems feasible right now.)

kevin commented

2026-02-07 18:18:28 +00:00

Author

Owner

We have the same access to both servers. :)

The admin web interface will let you add users to both of them, but for fedora.im I'd advise just making a fas account for the bot and logging in with it and then using the access key for the bot.

True that we can't use any of the features that require a patched homeserver, but I think all the things that currently work now should work fine with a new bot.

We have the same access to both servers. :) The admin web interface will let you add users to both of them, but for fedora.im I'd advise just making a fas account for the bot and logging in with it and then using the access key for the bot. True that we can't use any of the features that require a patched homeserver, but I think all the things that currently work now should work fine with a new bot.

kevin modified the project from Sprint ( 2026-01-26 to 2026-02-08 ) to Sprint ( 2026-02-09 to 2026-02-22 )

2026-02-09 00:28:14 +00:00

misc commented

2026-02-09 15:28:08 +00:00

Member

Do we want to still rely on the rory.gay server for the management room of the bot ? I assume that if the server disappear, it will not work correctly, and it would be better to use a alias for that (as I think this can be done on matrix). I do not have the required access to do that.

kevin commented

2026-02-09 18:18:02 +00:00

Author

Owner

I'm not sure I understand. Federated rooms are not tied to any particular server. The room will exist on all matrix servers that have users in the room. Unless you mean for finding it/index? Yes, we can add local addresses for fedora.im/fedoraproject.org there.
Of course the management room should be invite only however, it shouldn't be in the space or anything (IMHO).

I'm not sure I understand. Federated rooms are not tied to any particular server. The room will exist on all matrix servers that have users in the room. Unless you mean for finding it/index? Yes, we can add local addresses for fedora.im/fedoraproject.org there. Of course the management room should be invite only however, it shouldn't be in the space or anything (IMHO).

misc commented

2026-02-10 13:58:58 +00:00

Member

Yes, my concern is that the bot will not be able to join the room if we keep '#draupnir-fedora-mgmt:rory.gay' in the config in case the rory.gay domain disappear (or if the server is shutdown for a period long enough, I assume that there is a cache somewhere and the cache is not eternal).

We could also just use the room ID, but given there is some change (eg version 12 changed the format ), and a warning on how v 12 was not supported (cf https://marewolf.me/posts/draupnir/25/do-not-upgrade-to-v12.html ), I think a alias would be safer, clearer and likely more futureproof.

Yes, my concern is that the bot will not be able to join the room if we keep '#draupnir-fedora-mgmt:rory.gay' in the config in case the rory.gay domain disappear (or if the server is shutdown for a period long enough, I assume that there is a cache somewhere and the cache is not eternal). We could also just use the room ID, but given there is some change (eg [version 12 changed the format](https://spec.matrix.org/v1.17/appendices/#room-ids) ), and a warning on how v 12 was not supported (cf https://marewolf.me/posts/draupnir/25/do-not-upgrade-to-v12.html ), I think a alias would be safer, clearer and likely more futureproof.

kevin commented

2026-02-10 22:45:00 +00:00

Author

Owner

yeah, aliases seem fine to me. We can add both fedora.im and fedoraproject.org ones.

misc commented

2026-02-16 14:28:37 +00:00

Member

So, if we want to deploy, I need a access key. I have the rest. If people can give it to me, either on matrix, or our internal chat, I will do the deployment (likely later today, or tomorrow evening). For the sack of expdency, I will do it on our cluster, but the bot can be moved later.

smilner commented

2026-02-16 15:18:35 +00:00

cc @zlopez

kevin commented

2026-02-16 19:15:01 +00:00

Author

Owner

ok. I have created a infra-mod-bot user, logged into chat.fedoraproject.org with it and placed it's access token in ~misc/infra-mod-bot on batcave01.

We will of course have to make it a moderator in any channels its protecting.

Let me know if you need anything more...

ok. I have created a infra-mod-bot user, logged into chat.fedoraproject.org with it and placed it's access token in ~misc/infra-mod-bot on batcave01. We will of course have to make it a moderator in any channels its protecting. Let me know if you need anything more...

zlopez removed their assignment

2026-02-17 09:10:58 +00:00

zlopez commented

2026-02-17 09:11:34 +00:00

Owner

As this is being worked on by @misc let me remove myself as assignee.

misc commented

2026-02-19 14:53:47 +00:00

Member

I deployed it, but the token seems to not work. I tested with fedora.ems.host and fedoraproject.ems.host, same error message. If I test with curl directly, I have the same error code:

$ curl 'https://fedora.ems.host/_matrix/client/v3/account/whoami?access_token=mat_xxxx' ; echo
{"errcode":"M_UNKNOWN_TOKEN","error":"Token is not active","soft_logout":false}

If I test with a access token I get from chat.fedoraproject.org on my Fedora account, it work:

$ curl 'https://fedora.ems.host/_matrix/client/v3/account/whoami?access_token=syt_xxxx' ; echo
{"user_id":"@misc:fedora.im","is_guest":false,"device_id":"BIIZFORGEZ"}

If I test on fedoraproject.ems.host with my token, it return M_UNKNOWN_TOKEN.

I notice the token do not have the same prefix (mat_ for the one @kevin created, and syt_ for the ones I used for testing, both on my server and on the fedora server). I followed the instructions on https://the-draupnir-project.github.io/draupnir-documentation/bot/setup_draupnir_account but I do not remember how I got the one for the bot on my server, could this be a reason ?

I deployed it, but the token seems to not work. I tested with `fedora.ems.host` and `fedoraproject.ems.host`, same error message. If I test with curl directly, I have the same error code: ``` $ curl 'https://fedora.ems.host/_matrix/client/v3/account/whoami?access_token=mat_xxxx' ; echo {"errcode":"M_UNKNOWN_TOKEN","error":"Token is not active","soft_logout":false} ``` If I test with a access token I get from chat.fedoraproject.org on my Fedora account, it work: ``` $ curl 'https://fedora.ems.host/_matrix/client/v3/account/whoami?access_token=syt_xxxx' ; echo {"user_id":"@misc:fedora.im","is_guest":false,"device_id":"BIIZFORGEZ"} ``` If I test on `fedoraproject.ems.host` with my token, it return `M_UNKNOWN_TOKEN`. I notice the token do not have the same prefix (mat_ for the one @kevin created, and syt_ for the ones I used for testing, both on my server and on the fedora server). I followed the instructions on https://the-draupnir-project.github.io/draupnir-documentation/bot/setup_draupnir_account but I do not remember how I got the one for the bot on my server, could this be a reason ?

kevin commented

2026-02-19 19:38:59 +00:00

Author

Owner

Bah. Apparently the change to MAS changed things around here.

(So this would have worked before we moved to MAS I think)

So, those instructions won't seem to work here. We do not have password auth enabled at all.

I found:
https://element-hq.github.io/matrix-authentication-service/topics/access-token.html

but if I run that with asking for '"urn:matrix:client:api:*"' scope, it gives me an access token, but it's another mat_ one...

Does this need a synapse admin token? thats pretty high priv... our existing bot isn't one. Only a few folks have admin api enabled.

Open to ideas on how to get a token it likes...

Bah. Apparently the change to MAS changed things around here. (So this would have worked before we moved to MAS I think) So, those instructions won't seem to work here. We do not have password auth enabled at all. I found: https://element-hq.github.io/matrix-authentication-service/topics/access-token.html but if I run that with asking for '"urn:matrix:client:api:*"' scope, it gives me an access token, but it's another mat_ one... Does this need a synapse admin token? thats pretty high priv... our existing bot isn't one. Only a few folks have admin api enabled. Open to ideas on how to get a token it likes...

zlopez modified the project from Sprint ( 2026-02-09 to 2026-02-22 ) to Sprint ( 2026-02-23 to 2026-03-08 )

2026-02-23 09:28:20 +00:00

zlopez removed this from the Sprint ( 2026-02-23 to 2026-03-08 ) project

2026-02-23 15:32:16 +00:00

misc commented

2026-02-24 10:53:07 +00:00

Member

We can try the token given by default OAuth scope (eg, urn:matrix:org.matrix.msc2967.client:api:*) and test how it goes. I assume the mat_ prefix is a opaque identifier, so that shouldn't be blocking.

We can try the token given by default OAuth scope (eg, `urn:matrix:org.matrix.msc2967.client:api:*`) and test how it goes. I assume the `mat_` prefix is a opaque identifier, so that shouldn't be blocking.

kevin commented

2026-02-26 19:48:28 +00:00

Author

Owner

ok, same file, I added the token with that scope.

misc commented

2026-02-27 16:50:56 +00:00

Member

So it work (for how long, I do not know as I assume the token need to be renewed). However, now it crash with:

Unable to join Draupnir's management room
caused by: M_FORBIDDEN: You do not belong to any of the required rooms/spaces to join this room.
from unknown: undefined

So it work (for how long, I do not know as I assume the token need to be renewed). However, now it crash with: ``` Unable to join Draupnir's management room caused by: M_FORBIDDEN: You do not belong to any of the required rooms/spaces to join this room. from unknown: undefined ```

kevin commented

2026-02-27 17:04:28 +00:00

Author

Owner

It probibly needs to be invited? I can do that... done.

misc commented

2026-02-27 22:45:30 +00:00

Member

Now, it goes back to M_UNKNOWN_TOKEN (so I guess it expired). It seems the SDK used do not support renewing the token, but the bug say also that this is not needed.

So either we find the right kind of token from MAS, or we have to use a separate server (or go back to irc, as I am quite sure this problem wouldn't exist :p )

Now, it goes back to `M_UNKNOWN_TOKEN` (so I guess it expired). It seems the SDK used do not support [renewing the token](https://github.com/turt2live/matrix-bot-sdk/issues/298), but the bug say also that this is not needed. So either we find the right kind of token from MAS, or we have to use a separate server (or go back to irc, as I am quite sure this problem wouldn't exist :p )

kevin commented

2026-03-02 18:10:09 +00:00

Author

Owner

Perhaps we need to ask EMS folks what we need to do here... ytalk forever! :)

misc commented

2026-03-04 09:47:40 +00:00

Member

Ok so after discussing with EMS folks that directed to use https://admin-beta.element.dev/, the latest code given by @kevin seems to work, but now, the bot crash with:

message: "M_FORBIDDEN: You don't have permission to post that to the room. user_level (0) < send_level (10)",
elaborations: [
'Could not store the changed protection settings',
'Unable to change protection settings'
],
exceptionKind: 'Unknown',
exception: undefined,
uuid: '94ba2347-ee22-47e2-9acf-95ae9595724c',
matrixErrorCode: 'M_FORBIDDEN',
matrixErrorMessage: "You don't have permission to post that to the room. user_level (0) < send_level (10)"
}

and also

Wed, 04 Mar 2026 09:43:17 GMT [ERROR] [ManagementRoomOutput] Failed to log a message to the management room MatrixError: Error during MatrixClient request PUT /_matrix/client/v3/rooms/!sSlPjROlBByPSAUOrm%3Arory.gay/send/m.room.message/1772617397169__inc1: 500 Internal Server Error -- {"errcode":"M_UNKNOWN","error":"Internal server error"}
at /draupnir/utils.js:255:23
at result (/draupnir/utils.js:371:21)
at new Promise (<anonymous>)
at /draupnir/utils.js:370:38
at respIsBuffer (/node_modules/matrix-bot-sdk/lib/http.js:72:38)
at new Promise (<anonymous>)
at doHttpRequest (/node_modules/matrix-bot-sdk/lib/http.js:71:41)
at MatrixClient.doRequest (/node_modules/matrix-bot-sdk/lib/MatrixClient.js:1902:41)
at descriptor.value (/node_modules/matrix-bot-sdk/lib/metrics/decorators.js:19:53)
at MatrixClient.sendRawEvent (/node_modules/matrix-bot-sdk/lib/MatrixClient.js:1250:21) {
body: { errcode: 'M_UNKNOWN', error: 'Internal server error' },
statusCode: 500,
errcode: 'M_UNKNOWN',
error: 'Internal server error',
retryAfterMs: undefined
}

I guess the bot also need some privileges to post on the room.

Ok so after discussing with EMS folks that directed to use https://admin-beta.element.dev/, the latest code given by @kevin seems to work, but now, the bot crash with: ``` message: "M_FORBIDDEN: You don't have permission to post that to the room. user_level (0) < send_level (10)", elaborations: [ 'Could not store the changed protection settings', 'Unable to change protection settings' ], exceptionKind: 'Unknown', exception: undefined, uuid: '94ba2347-ee22-47e2-9acf-95ae9595724c', matrixErrorCode: 'M_FORBIDDEN', matrixErrorMessage: "You don't have permission to post that to the room. user_level (0) < send_level (10)" } ``` and also ``` Wed, 04 Mar 2026 09:43:17 GMT [ERROR] [ManagementRoomOutput] Failed to log a message to the management room MatrixError: Error during MatrixClient request PUT /_matrix/client/v3/rooms/!sSlPjROlBByPSAUOrm%3Arory.gay/send/m.room.message/1772617397169__inc1: 500 Internal Server Error -- {"errcode":"M_UNKNOWN","error":"Internal server error"} at /draupnir/utils.js:255:23 at result (/draupnir/utils.js:371:21) at new Promise (<anonymous>) at /draupnir/utils.js:370:38 at respIsBuffer (/node_modules/matrix-bot-sdk/lib/http.js:72:38) at new Promise (<anonymous>) at doHttpRequest (/node_modules/matrix-bot-sdk/lib/http.js:71:41) at MatrixClient.doRequest (/node_modules/matrix-bot-sdk/lib/MatrixClient.js:1902:41) at descriptor.value (/node_modules/matrix-bot-sdk/lib/metrics/decorators.js:19:53) at MatrixClient.sendRawEvent (/node_modules/matrix-bot-sdk/lib/MatrixClient.js:1250:21) { body: { errcode: 'M_UNKNOWN', error: 'Internal server error' }, statusCode: 500, errcode: 'M_UNKNOWN', error: 'Internal server error', retryAfterMs: undefined } ``` I guess the bot also need some privileges to post on the room.

misc commented

2026-03-04 23:31:12 +00:00

Member

Ok, so for the record, after discussing on Matrix, the bot is now running on a separate room on my server, and I think we will change the alias and get people to join the new room.

misc commented

2026-03-04 23:45:30 +00:00

Member

I spoke too soon, it fail now with

Wed, 04 Mar 2026 23:41:47 GMT [DEBUG] [DraupnirNews] Fetching remote news https://raw.githubusercontent.com/the-draupnir-project/Draupnir/refs/heads/main/src/protections/DraupnirNews/news.json
Wed, 04 Mar 2026 23:41:47 GMT [ERROR] [ActionException] ActionException Unknown 91140047-128b-4151-a365-1fd48ca0328a M_UNKNOWN: Internal server error undefined
Wed, 04 Mar 2026 23:41:47 GMT [ERROR] [DraupnirNews] Unable to notify of news item
Wed, 04 Mar 2026 23:41:47 GMT [ERROR] [ManagementRoomOutput] Failed to log a message to the management room MatrixError: Error during MatrixClient request PUT /_matrix/client/v3/rooms/!FFzYstueTYZfZzbGwS%3Aephaone.org/send/m.room.message/1772667707671__inc2: 500 Internal Server Error -- {"errcode":"M_UNKNOWN","error":"Internal server error"}
at /draupnir/utils.js:255:23
at result (/draupnir/utils.js:371:21)
at new Promise (<anonymous>)
at /draupnir/utils.js:370:38
at respIsBuffer (/node_modules/matrix-bot-sdk/lib/http.js:72:38)
at new Promise (<anonymous>)
at doHttpRequest (/node_modules/matrix-bot-sdk/lib/http.js:71:41)
at MatrixClient.doRequest (/node_modules/matrix-bot-sdk/lib/MatrixClient.js:1902:41)
at descriptor.value (/node_modules/matrix-bot-sdk/lib/metrics/decorators.js:19:53)
at MatrixClient.sendRawEvent (/node_modules/matrix-bot-sdk/lib/MatrixClient.js:1250:21) {
body: { errcode: 'M_UNKNOWN', error: 'Internal server error' },
statusCode: 500,
errcode: 'M_UNKNOWN',
error: 'Internal server error',
retryAfterMs: undefined
}

I spoke too soon, it fail now with ``` Wed, 04 Mar 2026 23:41:47 GMT [DEBUG] [DraupnirNews] Fetching remote news https://raw.githubusercontent.com/the-draupnir-project/Draupnir/refs/heads/main/src/protections/DraupnirNews/news.json Wed, 04 Mar 2026 23:41:47 GMT [ERROR] [ActionException] ActionException Unknown 91140047-128b-4151-a365-1fd48ca0328a M_UNKNOWN: Internal server error undefined Wed, 04 Mar 2026 23:41:47 GMT [ERROR] [DraupnirNews] Unable to notify of news item Wed, 04 Mar 2026 23:41:47 GMT [ERROR] [ManagementRoomOutput] Failed to log a message to the management room MatrixError: Error during MatrixClient request PUT /_matrix/client/v3/rooms/!FFzYstueTYZfZzbGwS%3Aephaone.org/send/m.room.message/1772667707671__inc2: 500 Internal Server Error -- {"errcode":"M_UNKNOWN","error":"Internal server error"} at /draupnir/utils.js:255:23 at result (/draupnir/utils.js:371:21) at new Promise (<anonymous>) at /draupnir/utils.js:370:38 at respIsBuffer (/node_modules/matrix-bot-sdk/lib/http.js:72:38) at new Promise (<anonymous>) at doHttpRequest (/node_modules/matrix-bot-sdk/lib/http.js:71:41) at MatrixClient.doRequest (/node_modules/matrix-bot-sdk/lib/MatrixClient.js:1902:41) at descriptor.value (/node_modules/matrix-bot-sdk/lib/metrics/decorators.js:19:53) at MatrixClient.sendRawEvent (/node_modules/matrix-bot-sdk/lib/MatrixClient.js:1250:21) { body: { errcode: 'M_UNKNOWN', error: 'Internal server error' }, statusCode: 500, errcode: 'M_UNKNOWN', error: 'Internal server error', retryAfterMs: undefined } ```

misc commented

2026-03-05 15:39:16 +00:00

Member

Thanks to the help of folks from Element, we found the issue is due to not having device attached to the access key (documenting to our LLM overload can claim to diagnose the issue later).

😆 1

kevin commented

2026-03-05 19:53:16 +00:00

Author

Owner

new token, hopefully with device allowed...

misc commented

2026-03-05 20:08:18 +00:00

Member

Ok so it finally work, the last step is to create a alias on fedora.im for the room to replace the one on my server, then I will be able to update the config and merge the MR on our repo. Then move that to Fedora repo and cluster in due time.

kevin commented

2026-03-11 23:50:02 +00:00

Author

Owner

ok, so what are the next steps here? invite people to the control room that are in the existing one, then get it added to the lists that our current bot has and then add the bot user to all the channels we want and give it perms?

misc commented

2026-03-26 16:34:58 +00:00

Member

Yeah, I think someone should configure the bot. I have no idea how to use it, but I moved it to the room where it was invited earlier by someone.

jflory7 commented

2026-03-30 02:09:46 +00:00

Member

I can carve out some time to help with this if I can get a step-by-step guide on what to do. Obviously I know the gist but I want to make sure I get the order of operations right and that I don't miss any steps. I haven't dug into how things are done right now but I have admin rights in most Fedora rooms, so I can help roll this out gradually.

FYI, @rorysys! I think we are on the cusp of being able to take over the hosting for the moderation bot.

I can carve out some time to help with this if I can get a step-by-step guide on what to do. Obviously I know the gist but I want to make sure I get the order of operations right and that I don't miss any steps. I haven't dug into how things are done right now but I have admin rights in most Fedora rooms, so I can help roll this out gradually. FYI, @rorysys! I think we are on the cusp of being able to take over the hosting for the moderation bot.

❤️ 1

kevin commented

2026-03-31 23:20:02 +00:00

Author

Owner

I think we need to come up with a migration plan?

Something like:

make sure the new bot is subscribed to all the CME lists the old one is.
invite all the people in the existing control room to the new bot one.
get the list of all protected rooms from the current bot
have the new bot join all those rooms
have someone add moderator perms for the new bot
disable the old bot / have it leave rooms / remove it's moderator perms

I might have missed something though. We will need to coordinate to avoid both bots trying to protect the same rooms at the same time and fighting each other.

I think we need to come up with a migration plan? Something like: * make sure the new bot is subscribed to all the CME lists the old one is. * invite all the people in the existing control room to the new bot one. * get the list of all protected rooms from the current bot * have the new bot join all those rooms * have someone add moderator perms for the new bot * disable the old bot / have it leave rooms / remove it's moderator perms I might have missed something though. We will need to coordinate to avoid both bots trying to protect the same rooms at the same time and fighting each other.

misc commented

2026-04-14 13:34:39 +00:00

Member

do we have SOP for the bot operations ?

farchord commented

2026-04-14 13:36:25 +00:00

Also, could the moderators be added to the room?

kevin commented

2026-04-14 21:49:10 +00:00

Author

Owner

@misc wrote in #12246 (comment):

do we have SOP for the bot operations ?

no, just the upstream documentation. @rorysys might have something more?

@farchord wrote in #12246 (comment):

Also, could the moderators be added to the room?

Yeah, we need to do that.
But we need to be careful adding the new bot to rooms already managed by the old bot because they can get in fights over settings acls.

@misc wrote in https://forge.fedoraproject.org/infra/tickets/issues/12246#issuecomment-615346: > do we have SOP for the bot operations ? no, just the upstream documentation. @rorysys might have something more? @farchord wrote in https://forge.fedoraproject.org/infra/tickets/issues/12246#issuecomment-615347: > Also, could the moderators be added to the room? Yeah, we need to do that. But we need to be careful adding the new bot to rooms already managed by the old bot because they can get in fights over settings acls.

farchord commented

2026-04-15 20:39:14 +00:00

Btw: Emma did say in the Fedora Matrix WG [PRIVATE]:

currently being annoyed by not having my password on hand for the IDP lol
re #12246 - good luck :)
if you need anything from me, you know where to find me

So if you have any questions, bring it in matrix it's likely we wont get an answer here :)

Btw: Emma did say in the Fedora Matrix WG [PRIVATE]: currently being annoyed by not having my password on hand for the IDP lol re #12246 - good luck :) if you need anything from me, you know where to find me So if you have any questions, bring it in matrix it's likely we wont get an answer here :)

farchord commented

2026-04-22 00:53:56 +00:00

If I were going to suggest a list of priorities, and mind you I only have a rudimentary knowledge of Draupnir bots:

1- Migrate the bot to Fedora infra (I'd say that's probably the most important right now)
2- Make sure our bot can alter/use the same lists as the Moderation bot (With Emma's help ofc), and reuse Emma's fedora-coc list if needed
3- Get accepted in the cme-bans group and get the bot added to alter the list (If it's not already done)
4- Get the bot added to some rooms, run it in parallel with the moderation bot, make sure everything works as expected
5- (Once #4 tests are successful) Complete the migration, kick the moderation bot offline.

One thing we can't account for is how good the bot will be under duress. I did let @jflory7 know today and we had a discussion about this, but we're working on laying the groundwork to allow to add some Matrix server-level superadmins by making the Matrix WG more official and setting up some workflows in that regard. Trying to get that done asap, because when attacks happen, being reactive really bites.

If I were going to suggest a list of priorities, and mind you I only have a rudimentary knowledge of Draupnir bots: 1- Migrate the bot to Fedora infra (I'd say that's probably the most important right now) 2- Make sure our bot can alter/use the same lists as the Moderation bot (With Emma's help ofc), and reuse Emma's fedora-coc list if needed 3- Get accepted in the cme-bans group and get the bot added to alter the list (If it's not already done) 4- Get the bot added to _some_ rooms, run it in parallel with the moderation bot, make sure everything works as expected 5- (Once #4 tests are successful) Complete the migration, kick the moderation bot offline. One thing we can't account for is how good the bot will be under duress. I did let @jflory7 know today and we had a discussion about this, but we're working on laying the groundwork to allow to add some Matrix server-level superadmins by making the Matrix WG more official and setting up some workflows in that regard. Trying to get that done asap, because when attacks happen, being reactive really bites.

gwmngilfen self-assigned this

2026-07-27 14:22:14 +00:00

gwmngilfen added this to the Sprint ( 2026-07-27 to 2026-08-09) project

2026-07-27 14:22:27 +00:00

gwmngilfen commented

2026-08-10 10:59:48 +00:00

Member

This has been completed - in so far as the bot is deployed and appears to be working. We have copied the config and policy subscription, and the bot appears to be working in a test room. I've also set some reasonable resource limits in OCP too. We also had a discussion about setting the bot to PL 100 since it's on our official homeserver, and using it for room creation for v12+.

I think we can therefore close this - the Infra side work is done, now we need to work in https://forge.fedoraproject.org/matrix to get the new account added to all the rooms and promoted appropriately. Happy to work on that with you, but this ticket is done :)

This has been completed - in so far as the bot is deployed and appears to be working. We have copied the config and policy subscription, and the bot appears to be working in a test room. I've also set some reasonable resource limits in OCP too. We also had a discussion about setting the bot to PL 100 since it's on our official homeserver, and using it for room creation for v12+. I think we can therefore close this - the Infra side work is done, now we need to work in https://forge.fedoraproject.org/matrix to get the new account added to all the rooms and promoted appropriately. Happy to work on that with you, but this ticket is done :)

gwmngilfen closed this issue

2026-08-10 10:59:48 +00:00

gwmngilfen referenced this issue from matrix/tickets

2026-08-10 11:02:50 +00:00

Move rooms to the new Draupnir instance #12

kevin referenced this issue from matrix/tickets

2026-08-10 17:07:29 +00:00

Move rooms to the new Draupnir instance #12

Rows
Columns

deploy a Draupnir instance for our matrix servers #12246