mini-initiative proposal: skeleton for bridge from fasjson groups to discourse

mattdm commented

2022-10-21 20:07:05 +00:00

Background

This is a follow-up to https://pagure.io/fedora-infrastructure/issue/9580, inspired by my discovery of the Communishift Authorization Operator, which is very similar to (at least part of) what I want.

As Kevin notes at the end of that ticket, upstream doesn't seem to be going in a helpful-for-our-use-case direction with this, so a little system to keep things in sync seems best to me.

Summary

I want to one-way sync FAS group membership information to Discourse (specifically, https://discussion.fedoraproject.org/). If a user is added to or removed from a group in FAS, and that user has a Discourse account and there is a Discourse group of the same name, the Discourse user should be added to (or removed from) the corresponding Discourse group.

As a bonus, this can solve another problem: if an account is locked, disabled, or deleted from FAS, the Discourse account should also be locked.

Not even pseudocode

Here's what I envision:

A ~~webhook~~ Fedora Message Bus topic about group membership change is received (see below), giving us the username to check
fasjson: validate that the account is active and in good standing
fasjson: get list of groups for the user
discourse api: disable the account if it should be disabled
discourse api: get list of all groups
take intersection of the fas and discourse groups (groups will be manually created on Discussion, so only groups that exist will be synced)
discourse api: get list of groups user belongs to
discourse api: remove the user from groups that are in 7 (discourse) but not 6 (fas)
discourse api: add the user to any group that is in 6 (fas) but not 7 (discourse)

But wait! Before you think "that's too many steps for a mini-initiative!" —

I can do the Discourse part!

I am familiar with the Discourse API. But I'm not with fasjson. It doesn't look complicated, but then I realized there's a whole bunch of yak-shaving to do around getting a keytab and setting it up so that's available and authenticated and running in infrastructure. And it looks like you have already shaved that yak! — that is, all of these things are very close to what Communishift Authorization Operator already does.

So,

What would really help me:

Steps 1-3 running somewhere -- maybe as a toddler. That should pass either "this account should be disabled" or the resulting grouplist to....
A stub that does nothing. (This will be quicker for me with python, but I can do this in whatever language you prefer.)
A place to put a secret available to that stub (an API key — I can either provide the key, or you can provide a way for me to add it securely)
Something where when I push to a git repo on a deploy branch (or whatever), that commit gets deployed, replacing the stub (or my previous version).
And access to logs would be a nice bonus

Because most of this is what Communishift Authorization Operator already does, I really hope that this will not be a lot of effort.

Triggering

I had originally imagined this would trigger on webhooks from discourse. We could still do that, but I think it makes more sense to listen to Fedora Message Bus messages for group changes (https://fedmsg2.readthedocs.io/en/latest/topics.html#fas).

However, there is something I don't understand -- there is fas.group.member.remove, but no fas.group.member.add. (There are .sponsor and .approve, though.) Hopefully you know what's going on here.

We'd also want to trigger on the account being deactivated or disabled — fas.user.update, I guess.

These could trigger different paths in my part of the code (remove, add, disable) but that has more risk of getting out of sync. I think it's better to just send user and grouplist and be done.

In case a message gets lost or out of sync, there probably should also be some occasional out-of-band double-check.

The connection to the discourse part

This could all live in the same codebase (maybe all running as one toddler), or the first part (the fas listener) could make a remote call of some sort -- a JSON http POST with the username and a list of groups. (And some auth token!). Maybe something running in OpenShift? I can work with whatever you think makes sense.

The actual code should be pretty simple, and I'll make sure to comment it well so that anyone else can, in the future, fix any issues or make updates.

Okay, but, why?

We're finally getting Enterprise hosting, which will let us combine the Ask and Discussion instances. Having group information will allow us to link permissions for posting in certain areas to group membership, allow teams to polls of just their team, allow teams to get notifications, and so much more. We could also make it so Fedora contributors in certain groups get automatically put to "not a newbie" level (which avoids some frustrations with limits on image uploads, etc.). This will be important as more teams move from mailing lists.

Note: I've edited this a bit since first filed!

Background ------------------ This is a follow-up to https://pagure.io/fedora-infrastructure/issue/9580, inspired by my discovery of the [Communishift Authorization Operator](https://pagure.io/cpe/communishift), which is very similar to (at least part of) what I want. As Kevin notes at the end of that ticket, upstream doesn't seem to be going in a helpful-for-our-use-case direction with this, so a little system to keep things in sync seems best to me. Summary -------------- I want to one-way sync FAS group membership information to Discourse (specifically, https://discussion.fedoraproject.org/). If a user is added to or removed from a group in FAS, and that user has a Discourse account and there is a Discourse group of the same name, the Discourse user should be added to (or removed from) the corresponding Discourse group. As a bonus, this can solve another problem: if an account is locked, disabled, or deleted from FAS, the Discourse account should also be locked. Not even pseudocode ----------- Here's what I envision: 1. A ~~webhook~~ Fedora Message Bus topic about group membership change is received (see below), giving us the username to check 2. fasjson: validate that the account is active and in good standing 3. fasjson: get list of groups for the user 4. discourse api: disable the account if it should be disabled 5. discourse api: get list of all groups 6. take intersection of the fas and discourse groups (groups will be manually created on Discussion, so only groups that exist will be synced) 7. discourse api: get list of groups user belongs to 8. discourse api: remove the user from groups that are in 7 (discourse) but not 6 (fas) 9. discourse api: add the user to any group that is in 6 (fas) but not 7 (discourse) But wait! Before you think "that's too many steps for a mini-initiative!" — I can do the Discourse part! -------- I am familiar with the [Discourse API](https://docs.discourse.org/). But I'm not with fasjson. It doesn't look complicated, but then I realized there's a whole bunch of yak-shaving to do around getting a keytab and setting it up so that's available and authenticated and running in infrastructure. And it looks like you have already shaved that yak! — that is, all of these things are very close to what Communishift Authorization Operator already does. So, What would really help me: ----------------- 1. Steps 1-3 running somewhere -- maybe as a [toddler](https://pagure.io/fedora-infra/toddlers). That should pass either "this account should be disabled" or the resulting grouplist to.... 2. A stub that does nothing. (This will be quicker for me with python, but I can do this in whatever language you prefer.) 3. A place to put a secret available to that stub (an API key — I can either provide the key, or you can provide a way for me to add it securely) 4. Something where when I push to a git repo on a deploy branch (or whatever), that commit gets deployed, replacing the stub (or my previous version). 5. And access to logs would be a nice bonus Because most of this is what Communishift Authorization Operator already does, I really hope that this will not be a lot of effort. Triggering -------- I had originally imagined this would trigger on webhooks from discourse. We could still do that, but I think it makes more sense to listen to Fedora Message Bus messages for group changes (https://fedmsg2.readthedocs.io/en/latest/topics.html#fas). However, there is something I don't understand -- there is `fas.group.member.remove`, but no `fas.group.member.add`. (There are `.sponsor` and `.approve`, though.) Hopefully you know what's going on here. We'd also want to trigger on the account being deactivated or disabled — `fas.user.update`, I guess. These could trigger different paths in my part of the code (remove, add, disable) but that has more risk of getting out of sync. I think it's better to just send user and grouplist and be done. In case a message gets lost or out of sync, there probably should also be some occasional out-of-band double-check. The connection to the discourse part ------------ This could all live in the same codebase (maybe all running as one toddler), or the first part (the fas listener) could make a remote call of some sort -- a JSON http POST with the username and a list of groups. (And some auth token!). Maybe something running in OpenShift? I can work with whatever you think makes sense. The actual code should be pretty simple, and I'll make sure to comment it well so that anyone else can, in the future, fix any issues or make updates. Okay, but, why? ------------- We're finally getting Enterprise hosting, which will let us combine the Ask and Discussion instances. Having group information will allow us to link permissions for posting in certain areas to group membership, allow teams to polls of just their team, allow teams to get notifications, and so much more. We could also make it so Fedora contributors in certain groups get automatically put to "not a newbie" level (which avoids some frustrations with limits on image uploads, etc.). This will be important as more teams move from mailing lists. ------ Note: I've edited this a bit since first filed!

zlopez commented

2022-10-24 08:50:40 +00:00

Owner

Metadata Update from @zlopez:

Issue priority set to: Waiting on Assignee (was: Needs Review)
Issue tagged with: dev, high-gain, high-trouble, mini-initiative

**Metadata Update from @zlopez**: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: dev, high-gain, high-trouble, mini-initiative

mattdm commented

2022-11-22 15:19:29 +00:00

Author

Hey, so, I've tried to make this as not-high-trouble-at-all as possible:

basically a duplicate of something just implemented
can fit into the existing "toddlers" framework rather than being some new service
clearly spec'ing out whta i
leave the part that is out of existing Fedora Account system / fasjson expertise to me"

Is there anything else I can do for y'all, or do differently, to possibly get this down to "medium-trouble" at least? What are the next steps?

Hey, so, I've tried to make this as not-high-trouble-at-all as possible: 1. basically a duplicate of something just implemented 2. can fit into the existing "toddlers" framework rather than being some new service 3. clearly spec'ing out whta i 4. leave the part that is out of existing Fedora Account system / fasjson expertise to me" Is there anything else I can do for y'all, or do differently, to possibly get this down to "medium-trouble" at least? What are the next steps?

dkirwan commented

2022-12-22 06:00:28 +00:00

Member

This is looking very similar to what we did with the communishiftauthorization operator.

@mattdm would you be able to provide an API key on the Discourse side, that has enough CRUD permissions for users and groups that we can start testing against?

This is looking very similar to what we did with the communishiftauthorization operator. @mattdm would you be able to provide an API key on the Discourse side, that has enough CRUD permissions for users and groups that we can start testing against?

kevin commented

2023-02-20 21:11:16 +00:00

Owner

Just confirming here that you are working on this now @dkirwan ? :)

zlopez commented

2023-02-28 13:30:36 +00:00

Owner

This is being worked on by @dkirwan and @lenkaseg.

zlopez commented

2023-02-28 13:30:51 +00:00

Owner

Metadata Update from @zlopez:

Issue assigned to dkirwan

**Metadata Update from @zlopez**: - Issue assigned to dkirwan

dkirwan commented

2023-04-12 10:27:51 +00:00

Member

Hi all, we are pretty close to being feature complete, we do need to test this operator running in staging with something resembling a production load.

We want to add the staging discourse to staging ipa/noggin/ipsilon and while we have the required information to update the openidc.staging.static in the ansible-private repo, it looks like we are missing a plugin on the Discourse side to handle oauth2. If possible, can @lenkaseg and I @dkirwan be added as administrators on the production Discourse instance? We'd like to examine the configuration on Discourse side, and figure out which plugins might be required to be installed on the staging side.

ping @mattdm

Hi all, we are pretty close to being feature complete, we do need to test this operator running in staging with something resembling a production load. We want to add the staging discourse to staging ipa/noggin/ipsilon and while we have the required information to update the `openidc.staging.static` in the ansible-private repo, it looks like we are missing a plugin on the Discourse side to handle oauth2. If possible, can @lenkaseg and I @dkirwan be added as administrators on the production Discourse instance? We'd like to examine the configuration on Discourse side, and figure out which plugins might be required to be installed on the staging side. ping @mattdm

dkirwan commented

2023-04-14 08:07:01 +00:00

Member

Hi all,

We've installed the latest version of fas2discourse operator on our staging OpenShift and configured it to point at the staging Discourse instance:
https://fedoraproject.staged-by-discourse.com.

This instance is now hooked into staging IPA/Noggin and Ipsilon. So community members can login and create their user accounts on this staging Discourse.

Think we are ready to do some proper testing now. Start making the groups that we want, and invite users in these groups to login and see if they are correctly given access.

Sync time set is currently set to run every 20 minutes, this can be lowered to 5 minutes or maybe even 2 minutes but we run the risk of hitting Discourse rate limiting if we are making large updates and go any lower.

Hi all, We've installed the latest version of `fas2discourse operator` on our staging OpenShift and configured it to point at the staging Discourse instance: `https://fedoraproject.staged-by-discourse.com`. This instance is now hooked into staging IPA/Noggin and Ipsilon. So community members can login and create their user accounts on this staging Discourse. Think we are ready to do some proper testing now. Start making the groups that we want, and invite users in these groups to login and see if they are correctly given access. Sync time set is currently set to run every `20 minutes`, this can be lowered to 5 minutes or maybe even 2 minutes but we run the risk of hitting Discourse rate limiting if we are making large updates and go any lower.

dkirwan commented

2023-04-25 13:14:55 +00:00

Member

https://pagure.io/infra-docs-fpo/pull-request/203# PR for the fas2discourse operator SOPs, reviewing in team before merging.

That was the final step btw, the operator appears to be working as expected in staging!

The groups in staging now match what is in production, and what is in IPA/Noggin/FAS

Hoping to deploy to production soon, with a version that is running with functionality for adding/removing users from groups disabled, but logging proposed changes will debug for a few days before enabling full functionality and calling this ticket complete.

https://pagure.io/infra-docs-fpo/pull-request/203# PR for the fas2discourse operator SOPs, reviewing in team before merging. That was the final step btw, the operator appears to be working as expected in staging! The groups in staging now match what is in production, and what is in IPA/Noggin/FAS Hoping to deploy to production soon, with a version that is running with functionality for adding/removing users from groups disabled, but logging proposed changes will debug for a few days before enabling full functionality and calling this ticket complete.

dkirwan commented

2023-05-02 10:48:15 +00:00

Member

We've pushed this into production this morning. Just to be aware, FAS/IPA/Noggin is the single source of truth, so please make sure that we add users to groups there, rather than in the Discourse side.

dkirwan commented

2023-05-03 15:16:57 +00:00

Member

Closing as complete, to raise bugs against this service please open a ticket at: https://pagure.io/cpe/fas2discourse/issues

dkirwan commented

2023-05-03 15:16:58 +00:00

Member

Metadata Update from @dkirwan:

Issue close_status updated to: Fixed
Issue status updated to: Closed (was: Open)

**Metadata Update from @dkirwan**: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)

Rows
Columns

mini-initiative proposal: skeleton for bridge from fasjson groups to discourse #10952

Background

Summary

Not even pseudocode

I can do the Discourse part!

What would really help me:

Triggering

The connection to the discourse part

Okay, but, why?