fedora 40 and above missing on images on data-analysis.fedoraproject.org/csv-reports/images

lnvso commented

2025-02-27 13:47:51 +00:00

could fedora 40, 41 and 42 be included in the graphs available at https://data-analysis.fedoraproject.org/csv-reports/images/summary.html

phsmoura commented

2025-02-27 20:14:07 +00:00

Member

Metadata Update from @phsmoura:

Issue priority set to: Waiting on Assignee (was: Needs Review)
Issue tagged with: low-gain, low-trouble, ops

**Metadata Update from @phsmoura**: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: low-gain, low-trouble, ops

phsmoura commented

2025-02-27 20:14:23 +00:00

Member

cc: @james

james commented

2025-02-28 01:40:01 +00:00

Member

I know about the numbers behind this, but not what generates the graphs etc.

cc @mattdm ?

I know about the numbers behind this, but not what generates the graphs etc. cc @mattdm ?

smooge commented

2025-02-28 14:08:36 +00:00

This is done via a set of scripts not attached to the work James has been doing. The older data graphs were were to be end-of-lifed around Fedora 40 and replaced with ones generated weekly from the dnf countme data.

The scripts are a python and awk which do a rough count of 1 entry per arch, release,ip/day and then writes out a csv file. That csv file is then run against some rough gnuplot graphs which get updated daily.

All of these were really a short term hack which turned into a permanent solution so updating to newer releases requires a lot of manual hacking of scripts, gnuplot graphs and the existing CSV files to deal with the newer data. At the moment the only data I would consider it useful for is the EPEL-6,8,9 numbers and historical data for other items.

A better solution would be to finish the 'make a graph every week from the countme data' project and keep an older set of pictures with an end-date of when they were last valid (Jun 2024?).

This is done via a set of scripts not attached to the work James has been doing. The older data graphs were were to be end-of-lifed around Fedora 40 and replaced with ones generated weekly from the dnf countme data. The scripts are a python and awk which do a rough count of 1 entry per arch, release,ip/day and then writes out a csv file. That csv file is then run against some rough gnuplot graphs which get updated daily. All of these were really a short term hack which turned into a permanent solution so updating to newer releases requires a lot of manual hacking of scripts, gnuplot graphs and the existing CSV files to deal with the newer data. At the moment the only data I would consider it useful for is the EPEL-6,8,9 numbers and historical data for other items. A better solution would be to finish the 'make a graph every week from the countme data' project and keep an older set of pictures with an end-date of when they were last valid (Jun 2024?).

kevin commented

2025-03-13 23:29:12 +00:00

Owner

The hacky stuff appears to be under roles/web-data-analysis/files/
Perhaps it's not hard to just add 40+ for now?

Or perhaps someone can finish the new thing... but I don't even know the status of that? was someone working on it?

The hacky stuff appears to be under roles/web-data-analysis/files/ Perhaps it's not hard to just add 40+ for now? Or perhaps someone can finish the new thing... but I don't even know the status of that? was someone working on it?

zlopez commented

2025-03-14 08:59:22 +00:00

Owner

I thought that the new thing is already finished and running.

gwmngilfen commented

2025-03-14 12:26:34 +00:00

Member

Last year, I was playing with a rewrite of https://pagure.io/brontosaurusifier which includes a frontend for the graphs. I didn't finish it then, but (obviously) I kep the code and it probably wouldn't be hard to wrap it up. I could make PR for stg or communishift, if people want me to push on with it.

kevin commented

2025-03-15 18:32:16 +00:00

Owner

It would be nice to have for sure...

kevin commented

2025-12-11 17:52:49 +00:00

Owner

9 months later.... anyone willing to look into what it would take to finish off the new one or just trivially add 40+ to the hacky one?

james commented

2026-03-10 16:52:09 +00:00

Member

So ... I spoke to Matthew today, and have had a look at a bunch of the code.

Roughly what happens now:

log01:/etc/cron.d/condense-mirrorlogs.cron (from roles/web-data-analysis/files/) runs...
condense-mirrorlogs.sh which runs...
mirrors-data.awk which creates...
https://data-analysis.fedoraproject.org/csv-reports/mirrors/mirrorsdata-all.csv which is used by...
https://pagure.io/velociraptorizer (on Matthew's machine) to create the images.

...trying to run velociraptorizer is ... not fun. However even then you can't just add F40-F44 because the data from mirrorsdata-all.csv also only goes up to F39 ... so we'd need to alter mirrors-data.awk to generate the new columns and velociraptorizer to use them, and I wouldn't be shocked if smooge is the only one who knows awk that well ;).

Matthew said he hoped he'd have free time on friday to look using the countme data directly, and/or maybe something else to generate graphs. If that doesn't close this issue then we def. need to give it more than low-trouble.

So ... I spoke to Matthew today, and have had a look at a bunch of the code. Roughly what happens now: * log01:/etc/cron.d/condense-mirrorlogs.cron (from roles/web-data-analysis/files/) runs... * condense-mirrorlogs.sh which runs... * mirrors-data.awk which creates... * https://data-analysis.fedoraproject.org/csv-reports/mirrors/mirrorsdata-all.csv which is used by... * https://pagure.io/velociraptorizer (on Matthew's machine) to create the images. ...trying to run `velociraptorizer` is ... not fun. However even then you can't just add F40-F44 because the data from mirrorsdata-all.csv also only goes up to F39 ... so we'd need to alter `mirrors-data.awk` to generate the new columns and velociraptorizer to use them, and I wouldn't be shocked if smooge is the only one who knows awk that well ;). Matthew said he hoped he'd have free time on friday to look using the countme data directly, and/or maybe something else to generate graphs. If that doesn't close this issue then we def. need to give it more than low-trouble.

mattdm commented

2026-03-10 18:56:21 +00:00

Oh, actually -- the images on the server are generated by Smooge's script, not velociraptorizer.

Oh, actually -- the images on the server are generated by Smooge's script, _not_ velociraptorizer.

james commented

2026-03-10 20:42:26 +00:00

Member

On that note, I hacked a python script up to get the weekly unique IP data out of totals.db for specified repo-tags and put them into a single CSV file ... then did what all the cool kids were doing before AI and found a graph as a service thing:

Comparing F40+ (2024-) unique IP hits.
https://datawrapper.dwcdn.net/BzYk3/2/

Comparing F32+ (2020-) unique IP hits.
https://datawrapper.dwcdn.net/cA5zb/1/

EPEL 5+ (2012-) unique IP hits. (for the big numbers)
https://datawrapper.dwcdn.net/iuRY0/1/

On that note, I hacked a python script up to get the weekly unique IP data out of totals.db for specified repo-tags and put them into a single CSV file ... then did what all the cool kids were doing before AI and found a graph as a service thing: Comparing F40+ (2024-) unique IP hits. https://datawrapper.dwcdn.net/BzYk3/2/ Comparing F32+ (2020-) unique IP hits. https://datawrapper.dwcdn.net/cA5zb/1/ EPEL 5+ (2012-) unique IP hits. (for the big numbers) https://datawrapper.dwcdn.net/iuRY0/1/

smooge commented

2026-03-10 20:47:10 +00:00

I just got these in my email box.. do you still need info from me on my messed up really bad program?

smooge commented

2026-03-10 20:50:26 +00:00

The images on the website are all done via a 'POC goes directly into production' method of using gnu plot I wanted to get the velociraptor tooling instead as the graphs were much nicer. I would suggest not using a graph as a service thing because they tend to become 'crap this is costing me money.. better just fill anyone using its browser with porn spam' at some point.

kevin referenced this issue

2026-03-11 23:57:09 +00:00

/etc/cron.daily/sync-http-logs-and-merge.sh on log01 errors #12833

james commented

2026-03-12 13:58:18 +00:00

Member

The images on the website are all done via a 'POC goes directly into production' method of using gnu plot

Yeh, we've all been there to some degree. Although I looked at the git log and it's been in prod. for 10 years now, achievement unlocked ;)

Hopefully we can replace it with another POC before this one hits 11.

I would suggest not using a graph as a service thing

Was def. not planning on using it more than a couple of times, but wanted something as easy as possible to let people see data for the last couple of releases (I do realize how this sounds next to your 10 yr POC).

do you still need info from me

I think we are good. We need to come up with a plan and repoint this issue.

> The images on the website are all done via a 'POC goes directly into production' method of using gnu plot Yeh, we've all been there to some degree. Although I looked at the git log and it's been in prod. for 10 years now, achievement unlocked ;) Hopefully we can replace it with another POC before this one hits 11. > I would suggest not using a graph as a service thing Was def. not planning on using it more than a couple of times, but wanted something as easy as possible to let people see data for the last couple of releases (I do realize how this sounds next to your 10 yr POC). > do you still need info from me I think we are good. We need to come up with a plan and repoint this issue.

❤️ 1

mattdm commented

2026-06-04 18:25:45 +00:00

I am working on moving Brontosaurusifier to Graphana, and as part of that also adding the long-format unique-ip data that James worked on, like two years ago now. That has the big advantage of not ever needing column additions; new things just automatically get picked up.

If there is Hot Demand for velociraptorizer-style static images too, I can keep that for selected data.

But in any case, I thnink the old tooling can be retired.

I am working on moving Brontosaurusifier to Graphana, and as part of that also adding the long-format unique-ip data that James worked on, like two years ago now. That has the big advantage of not ever needing column additions; new things just automatically get picked up. If there is Hot Demand for velociraptorizer-style static images too, I can keep that for selected data. But in any case, I thnink the old tooling can be retired.

Rows
Columns

fedora 40 and above missing on images on data-analysis.fedoraproject.org/csv-reports/images #12426