abompard commented

Member

The old fmn was used to watch all commits on src.fedoraproject.org and send them to the scm-commits@lists.fedoraproject.org list.
Since I took down the old FMN, there's no more emails there. ;(

This list was super high volume, but it was a way to have commits for all packages transparently go out and allow auditing from 3rd parties, etc. It would be super nice if we could make this work again.

I think it would be a nice exercise for someone who wants to learn writing a fedora-messaging consumer. Maybe a toddler would be better btw.

The old fmn was used to watch all commits on src.fedoraproject.org and send them to the scm-commits@lists.fedoraproject.org list. Since I took down the old FMN, there's no more emails there. ;( This list was super high volume, but it was a way to have commits for all packages transparently go out and allow auditing from 3rd parties, etc. It would be super nice if we could make this work again. I think it would be a nice exercise for someone who wants to learn writing a fedora-messaging consumer. Maybe a toddler would be better btw.

zlopez commented

2023-11-23 11:07:58 +00:00

Owner

Metadata Update from @zlopez:

Issue priority set to: Waiting on Assignee (was: Needs Review)
Issue tagged with: dev, medium-gain, medium-trouble

**Metadata Update from @zlopez**: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: dev, medium-gain, medium-trouble

zlopez commented

2023-11-23 11:08:28 +00:00

Owner

Probably a good ticket to bring up on I&R standup.

abompard commented

2023-12-01 06:45:59 +00:00

Author

Member

OK, so let me make a summary of this, after a week of work or so. The solution is composed of 3 elements:

A git hook running on the server that will send messages on push. This hook already exists.
A schema for the messages sent by the git hook. This is what will be used to generate the Subject and the body of the email. This work used to be done by fedmsg_meta_fedora_infrastructure/scm.py on fedmsg.
A fedora-messaging consumer that will receive the messages, use the schema to generate the email components, and send the email. One process to rule them all, and in the darkness of the datacenter, bind them.

Part 3 could be done by anyone new to fedora-messaging and Toddlers, it's simple enough I think. I'm happy to give some mentoring on the fedora-messaging part (I haven't ever written a Toddler yet so I don't know about that part).

Part 1 is... trickier. The git hook is actually installed by Ansible, and has two versions, slightly different from each other, one adapted to Python 3 but not the other, one supporting namespaces but not the other. It's a bit of a mess. And hard to evolve into something that would use schemas as required by Part 2. Let's try to minimize the amount of raw code that we deploy with Ansible, please.

I've taken those git hooks and tried to merge them into a proper repo, retaining the features and the Python 3.6 compatibility (wasn't easy, but Pagure is currently running on Python 3.6), and adding some unit tests and QA tools at the same time.

I've also written the schemas required by Part 2 in another repo.

I don't think we want to deploy code directly from Github on pkgs01 and batcave, so I had those two be packaged into RPMs by Packit into a Copr repo: https://copr.fedorainfracloud.org/coprs/abompard/fedora-messaging-git-hook/.

I'm going to rebuild the SRPMs from this repo into Koji's infra tag each time there's an update in the code even if it's an extra step because this is pretty sensitive code, running on batcave and dist-git on each commit and having full access to the repos. Or do you think it's not necessary?

I'll try on staging today and hopefully it'll work out fine enough. We can work on the consumer (Part 3) in parallel.

OK, so let me make a summary of this, after a week of work or so. The solution is composed of 3 elements: 1. A git hook running on the server that will send messages on push. This hook already exists. 2. A schema for the messages sent by the git hook. This is what will be used to generate the `Subject` and the body of the email. This work used to be done by `fedmsg_meta_fedora_infrastructure/scm.py` on fedmsg. 3. A fedora-messaging consumer that will receive the messages, use the schema to generate the email components, and send the email. One process to rule them all, and in the darkness of the datacenter, bind them. Part 3 could be done by anyone new to fedora-messaging and Toddlers, it's simple enough I think. I'm happy to give some mentoring on the fedora-messaging part (I haven't ever written a Toddler yet so I don't know about that part). Part 1 is... trickier. The git hook is actually installed by Ansible, and has [two](https://pagure.io/fedora-infra/ansible/blob/main/f/roles/git/hooks/files/post-receive-fedora-messaging) [versions](https://pagure.io/fedora-infra/ansible/blob/main/f/roles/batcave/files/fedmsg-announce-commits.py), slightly different from each other, one adapted to Python 3 but not the other, one supporting namespaces but not the other. It's a bit of a mess. And hard to evolve into something that would use schemas as required by Part 2. Let's try to minimize the amount of raw code that we deploy with Ansible, please. I've taken those git hooks and tried to merge them into [a proper repo](https://github.com/fedora-infra/fedora-messaging-git-hook), retaining the features and the Python 3.6 compatibility (wasn't easy, but Pagure is currently running on Python 3.6), and adding some unit tests and QA tools at the same time. I've also written the schemas required by Part 2 in [another repo](https://github.com/fedora-infra/fedora-messaging-git-hook-messages). I don't think we want to deploy code directly from Github on pkgs01 and batcave, so I had those two be packaged into RPMs by Packit into a Copr repo: https://copr.fedorainfracloud.org/coprs/abompard/fedora-messaging-git-hook/. I'm going to rebuild the SRPMs from this repo into Koji's infra tag each time there's an update in the code even if it's an extra step because this is pretty sensitive code, running on batcave and dist-git on each commit and having full access to the repos. Or do you think it's not necessary? I'll try on staging today and hopefully it'll work out fine enough. We can work on the consumer (Part 3) in parallel.

kevin commented

2023-12-04 00:17:29 +00:00

Owner

This all seems kinda complex when it could just send email from the hook, but makes sense. :(

one thing I am a bit worried at with this and toddlers is that it will get a heavy processing flow... if we have a bunch of things in toddlers we need to make sure a problem with one toddler doesn't cause all of them to stop processing... or slow down.

This all seems kinda complex when it could just send email from the hook, but makes sense. :( one thing I am a bit worried at with this and toddlers is that it will get a heavy processing flow... if we have a bunch of things in toddlers we need to make sure a problem with one toddler doesn't cause all of them to stop processing... or slow down.

abompard commented

2023-12-04 07:54:21 +00:00

Author

Member

This all seems kinda complex when it could just send email from the hook, but makes sense. :(

Agreed, if the hook has no other use than sending an email to scm-list, then it's very overkill. But I think we want a fedora-messaging message to be sent when a commit is received nonetheless, so that people can get those from FMN, have it be recorded in datanommer, etc.

one thing I am a bit worried at with this and toddlers is that it will get a heavy processing flow... if we have a bunch of things in toddlers we need to make sure a problem with one toddler doesn't cause all of them to stop processing... or slow down.

Good point, I haven't looked at Toddler's code, but I would bet that processing of a message starts when the processing of the previous message has finished. So one slow toddler will slow down the whole processing. We should however be able to run multiple toddlers pods in parallel, and they would not be waiting on each other.

One more thing: Pagure sends a message on each push, so for dist-git we actually have two messages from the same action. Not a big deal, but maybe a bit superfluous.
We still need this git hook for the batcave repos (such as ansible), though.

> This all seems kinda complex when it could just send email from the hook, but makes sense. :( Agreed, if the hook has no other use than sending an email to scm-list, then it's very overkill. But I think we want a fedora-messaging message to be sent when a commit is received nonetheless, so that people can get those from FMN, have it be recorded in datanommer, etc. > one thing I am a bit worried at with this and toddlers is that it will get a heavy processing flow... if we have a bunch of things in toddlers we need to make sure a problem with one toddler doesn't cause all of them to stop processing... or slow down. Good point, I haven't looked at Toddler's code, but I would bet that processing of a message starts when the processing of the previous message has finished. So one slow toddler will slow down the whole processing. We should however be able to run multiple toddlers pods in parallel, and they would not be waiting on each other. One more thing: Pagure sends a message on each push, so for dist-git we actually have two messages from the same action. Not a big deal, but maybe a bit superfluous. We still need this git hook for the batcave repos (such as ansible), though.

pingou commented

2023-12-04 08:29:04 +00:00

Member

toddlers is actually very good at scaling horizontally. I've had times where it
was stuck due to messages not being in the expected format and the toddler was
thus crashing. I just bumped the number of pods to 50 and let it seat over the
week-end and waited for the next week to make the code change that allowed the
toddler to handle these messages.

So if you're worried about processing being slow, just increase the number of
pods.

toddlers is actually very good at scaling horizontally. I've had times where it was stuck due to messages not being in the expected format and the toddler was thus crashing. I just bumped the number of pods to 50 and let it seat over the week-end and waited for the next week to make the code change that allowed the toddler to handle these messages. So if you're worried about processing being slow, just increase the number of pods.

pingou commented

2023-12-04 16:44:14 +00:00

Member

one thing I am a bit worried at with this and toddlers is that it will get a heavy processing flow... if we have a bunch of things in toddlers we need to make sure a problem with one toddler doesn't cause all of them to stop processing... or slow down.

Good point, I haven't looked at Toddler's code, but I would bet that processing of a message starts when the processing of the previous message has finished. So one slow toddler will slow down the whole processing. We should however be able to run multiple toddlers pods in parallel, and they would not be waiting on each other.

toddlers is actually very good at scaling horizontally. I've had times where it
was stuck due to messages not being in the expected format and the toddler was
thus crashing. I just bumped the number of pods to 50 and let it seat over the
week-end and waited for the next week to make the code change that allowed the
toddler to handle these messages.

So if you're worried about processing being slow, just increase the number of
pods.

> > one thing I am a bit worried at with this and toddlers is that it will get a heavy processing flow... if we have a bunch of things in toddlers we need to make sure a problem with one toddler doesn't cause all of them to stop processing... or slow down. > > Good point, I haven't looked at Toddler's code, but I would bet that processing of a message starts when the processing of the previous message has finished. So one slow toddler will slow down the whole processing. We should however be able to run multiple toddlers pods in parallel, and they would not be waiting on each other. toddlers is actually very good at scaling horizontally. I've had times where it was stuck due to messages not being in the expected format and the toddler was thus crashing. I just bumped the number of pods to 50 and let it seat over the week-end and waited for the next week to make the code change that allowed the toddler to handle these messages. So if you're worried about processing being slow, just increase the number of pods.

kevin commented

2024-01-19 00:21:32 +00:00

Owner

@abompard Any news here. ;( it's been down a while now... makes me sad. ;(

abompard commented

2024-01-19 16:11:39 +00:00

Author

Member

Metadata Update from @abompard:

Issue assigned to abompard

**Metadata Update from @abompard**: - Issue assigned to abompard

abompard commented

2024-01-19 16:13:18 +00:00

Author

Member

Writing a toddler is still up for grabs for anyone who wants to learn how to do that.
That said, no one wants a sad Kevin, so I'll have a look hopefully next week.

Writing a toddler is still up for grabs for anyone who wants to learn how to do that. That said, no one wants a sad Kevin, so I'll have a look hopefully next week.

kevin commented

2024-02-27 20:39:53 +00:00

Owner

@abompard any news here? The lack of emails... so sad. ;)

abompard commented

2024-02-28 13:27:26 +00:00

Author

Member

Got bogged down by mirrormanager, sorry.

nphilipp commented

2024-03-27 12:16:33 +00:00

Member

I’ll take a stab at the toddler in fedora-infra/toddlers#183.

nphilipp commented

2024-04-09 10:27:56 +00:00

Member

The new distgit_commit_processor is merged, but needs new configuration to be merged in Ansible and deployed before it can be tested in staging.

The new `distgit_commit_processor` is merged, but needs [new configuration](https://pagure.io/fedora-infra/ansible/pull-request/1940) to be merged in Ansible and deployed before it can be tested in staging.

nphilipp commented

2024-05-13 13:59:57 +00:00

Member

@zlopez thanks for the review!

@kevin I would deploy the changes myself, but I don’t think I’ve ever deployed on our OpenShift before, and I’m not sure if there’s anything I need to keep in mind (compared to non-OpenShift playbooks). Also, no idea how to test the change, for largely the same reason. 😉

@zlopez thanks for the review! @kevin I would deploy the changes myself, but I don’t think I’ve ever deployed on our OpenShift before, and I’m not sure if there’s anything I need to keep in mind (compared to non-OpenShift playbooks). Also, no idea how to test the change, for largely the same reason. 😉

zlopez commented

2024-05-13 15:00:13 +00:00

Owner

@nphilipp The playbooks are being run the same way, you just need to check how the build and deployment finished in openshift.

kevin commented

2024-05-14 01:31:33 +00:00

Owner

Yeah... for openshift apps ansible runs and templates things to os-control01 or os-control01.stg (or both) and then runs a 'oc apply -f ...' with some validation to make the thing exist in openshift.

Then use the normal console or oc to debug and such.

Testing something like this is kinda hard. We could deploy to staging, but there's not much traffic there, although I suppose doing a commit to a package on src.stg should result in a commit email going out?

Yeah... for openshift apps ansible runs and templates things to os-control01 or os-control01.stg (or both) and then runs a 'oc apply -f ...' with some validation to make the thing exist in openshift. Then use the normal console or oc to debug and such. Testing something like this is kinda hard. We could deploy to staging, but there's not much traffic there, although I suppose doing a commit to a package on src.stg should result in a commit email going out?

kevin commented

2024-08-19 17:24:21 +00:00

Owner

@nphilipp whats the state of things here? I think the toddler was deployed, but had issues and was disabled?
Can we get this over the finish line?

@nphilipp whats the state of things here? I think the toddler was deployed, but had issues and was disabled? Can we get this over the finish line?

kevin commented

2024-11-07 22:19:37 +00:00

Owner

Any news here?

zlopez commented

2024-11-11 09:54:04 +00:00

Owner

The work on the toddler is still in progress. @nphilipp is working on it. The related ticket in toddlers is here https://pagure.io/fedora-infra/toddlers/issue/183, but the deployed version have to be disabled for now as it caused errors.

abompard commented

2025-02-04 12:12:32 +00:00

Author

Member

Metadata Update from @abompard:

Assignee reset

**Metadata Update from @abompard**: - Assignee reset

kevin commented

2025-07-21 19:38:56 +00:00

Owner

Hey, any news here?

Sad that this isn't working for so long.

Hey, any news here? Sad that this isn't working for so long.

zlopez commented

2025-07-22 12:15:15 +00:00

Owner

@nphilipp is currently helping on forgejo, so I'm not sure when he will have time to continue working on this.

zlopez commented

2026-02-12 17:48:07 +00:00

Owner

@nphilipp Did you had time to look at this?

kevin commented

2026-02-12 17:54:58 +00:00

Owner

One idea I had was instead of making a toddler that sent emails to the list, we use public-inbox ( basically a git repo for messages with some stuff on top). That way we could just have the toddler commit directly to that public-inbox, then we could expose it to the world for interested parties to look at/use.

I've not looked at it in a while tho, so I could be wrong that it will work for this... but might be worth exploring.

One idea I had was instead of making a toddler that sent emails to the list, we use public-inbox ( basically a git repo for messages with some stuff on top). That way we could just have the toddler commit directly to that public-inbox, then we could expose it to the world for interested parties to look at/use. I've not looked at it in a while tho, so I could be wrong that it will work for this... but might be worth exploring.

smoliicek commented

2026-02-12 22:07:11 +00:00

Member

im currently trying to make something with the public-inbox idea
assigning this to myself for now, will post updates

im currently trying to make something with the public-inbox idea assigning this to myself for now, will post updates

👍 1

smoliicek self-assigned this

2026-02-12 22:07:16 +00:00

kevin added this to the Sprint ( 2026-02-23 to 2026-03-08 ) project

2026-02-19 17:15:09 +00:00

zlopez removed this from the Sprint ( 2026-02-23 to 2026-03-08 ) project

2026-02-23 15:43:53 +00:00

smoliicek added this to the Sprint ( 2026-03-09 to 2026-03-23 ) project

2026-03-05 18:38:31 +00:00

kevin added

and removed

labels

2026-03-06 19:28:55 +00:00

gwmngilfen commented

2026-05-28 15:48:42 +00:00

Member

Did you get anywhere with this @smoliicek? Anything we can do to help?

smoliicek commented

2026-05-28 17:01:46 +00:00

Member

i was trying to setup the public-inbox thing in my local dev enviroment, with no luck... and then i forgot about this, sorry :(

i can try to spin it up once more, if that doesn't work i will look onto fixing the current setup
one question with this - if public-inbox would be running in our infra, should it run:

wherever lists are now
wherever the smtp server is
on a dedicated vm?

will get on this ASAP

i was trying to setup the public-inbox thing in my local dev enviroment, with no luck... and then i forgot about this, sorry :( i can try to spin it up once more, if that doesn't work i will look onto fixing the current setup one question with this - if public-inbox would be running in our infra, should it run: * wherever lists are now * wherever the smtp server is * on a dedicated vm? will get on this ASAP

kevin commented

2026-05-28 17:47:32 +00:00

Owner

I'd say ideally a openshift pod/project? We could use nfs storage...

smoliicek referenced this issue

2026-05-28 19:25:03 +00:00

communishift enrollment request #13372

smoliicek commented

2026-05-28 19:27:26 +00:00

Member

i filed #13372 for a communishift namespace, will try to spin it up there

i filed https://forge.fedoraproject.org/infra/tickets/issues/13372 for a communishift namespace, will try to spin it up there

smoliicek commented

2026-05-29 20:16:57 +00:00

Member

The POC is currently running in communishift and is exposing it's route here: https://public-inbox-communishift-public-inbox-poc.apps.fedora.cj14.p1.openshiftapps.com/

please check it out!
do note, that due to development, i pretty frequently wipe it's data directory

The POC is currently running in communishift and is exposing it's route here: https://public-inbox-communishift-public-inbox-poc.apps.fedora.cj14.p1.openshiftapps.com/ please check it out! do note, that due to development, i pretty frequently wipe it's data directory

🎉 1

mwinters commented

2026-06-02 23:45:13 +00:00

Contributor

Recapping our Matrix convo today:

I have some concerns about how this solution would pan out long-term. Namely, we had almost exactly 500k git commits in 2025, which is a lot of objects to have in one git repo if you're running this for several years. That might eventually become a UX or performance problem. If there is a "better" solution, that would depend on what the packagers actually need.

Kevin mentioned some offline / historical analytics, which FDWG can provide very well, so I thought we should compare notes.

Use cases

@kevin chimed in with two main categories:

Use case `#1`: Queries / Analytics / Auditing

"Ability for packagers to search commits easily". Example queries:

"How many commits have BuildRequires: foo in them"

"How many commits did user bar make and what was in them"

"Ability for maintainers to make their own local copy of all commits".

All of these things can be provided for today via FDWG, either in "download some data" mode or "query live data" mode. We'd be happy to tailor things to improve UX for packagers.

Questions

The main question here is: what does "easily" mean to packagers? If you're downloading data (in parquet format), DuckDB provides 3 ways to use it: a CLI SQL interface, a GUI SQL interface, and a python library. Would that work? (FYI, if Python is your primary method, there are many many ways to process parquet besides DuckDB if you want to make things harder on yourself for some reason :) )

We also have some (limited but growing) ability to create dashboards and fancy graphs, if you're looking for more aggregate-type info.

Use case `#2`: Alerts

"Ability for packagers to watch commits easily".

Questions

Same question as above: what does "easily" mean to packagers?

What are the expected / ideal channels of delivery? Do we want both push (email / webhook) and pull (something like atom or git pull)?
What is the maximum acceptable latency? Do you need alerts from things that happened 10 minutes ago, or would a daily digest suffice? (What about hourly digest? Etc.)
Do most people want to apply some sort of filter? Or do they want "all commits"?

I'll be honest, the alerting use case sounds like, "Use FMN", but again I don't know what packagers need.

Recapping our Matrix convo today: I have some concerns about how this solution would pan out long-term. Namely, we had almost exactly 500k git commits in 2025, which is a lot of objects to have in one git repo if you're running this for several years. That might eventually become a UX or performance problem. If there is a "better" solution, that would depend on what the packagers actually need. Kevin mentioned some offline / historical analytics, which FDWG can provide very well, so I thought we should compare notes. # Use cases @kevin chimed in with two main categories: ## Use case `#1`: Queries / Analytics / Auditing > - "Ability for packagers to search commits easily". Example queries: > - "How many commits have BuildRequires: foo in them" > - "How many commits did user bar make and what was in them" > - "Ability for maintainers to make their own local copy of all commits". All of these things can be provided for today via FDWG, either in "download some data" mode or "query live data" mode. We'd be happy to tailor things to improve UX for packagers. ### Questions The main question here is: what does "easily" mean to packagers? If you're downloading data (in parquet format), DuckDB provides 3 ways to use it: a CLI SQL interface, a GUI SQL interface, and a python library. Would that work? (FYI, if Python is your primary method, there are many many ways to process parquet besides DuckDB if you want to make things harder on yourself for some reason :) ) We also have some (limited but growing) ability to create dashboards and fancy graphs, if you're looking for more aggregate-type info. ## Use case `#2`: Alerts > - "Ability for packagers to watch commits easily". ### Questions Same question as above: what does "easily" mean to packagers? - What are the expected / ideal channels of delivery? Do we want both push (email / webhook) and pull (something like atom or `git pull`)? - What is the maximum acceptable latency? Do you need alerts from things that happened 10 minutes ago, or would a daily digest suffice? (What about hourly digest? Etc.) - Do most people want to apply some sort of filter? Or do they want "all commits"? I'll be honest, the alerting use case sounds like, "Use FMN", but again I don't know what packagers need.

kevin commented

2026-06-03 22:20:07 +00:00

Owner

https://discussion.fedoraproject.org/t/the-future-plans-for-the-scm-commits-mailing-list/193068 posted for discussing this. ;)

❤️ 1

Rows
Columns

Commits don't end up on the scm-commits list #11641

Use cases

Use case #1: Queries / Analytics / Auditing

Questions

Use case #2: Alerts

Questions

Use case `#1`: Queries / Analytics / Auditing

Use case `#2`: Alerts