Could we have fedoraproject-updates-archive.fedoraproject.org for Rawhide? #12366

New issue

Closed

opened 2024-10-01 10:38:14 +00:00 by praiskup · 6 comments

praiskup commented

2024-10-01 10:38:14 +00:00

Member

It would be nice to have a repository with all the RPMs that want to Rawhide,
with the ability to dnf downgrade.

I assume that we don't want to keep the history indefinitely, but could we for
example keep history across say max two %dist versions? Another alternative
is to use /bin/prunerepo (= keep the latest NEVRAs + anything that is not
older than XY months). Or reset the history at branching time. Or.. ideas?

It would be nice to have a repository with all the RPMs that want to Rawhide, with the ability to `dnf downgrade`. I assume that we don't want to keep the history indefinitely, but could we for example keep history across say max two %dist versions? Another alternative is to use `/bin/prunerepo` (= keep the latest NEVRAs + anything that is not older than XY months). Or reset the history at branching time. Or.. ideas?

praiskup commented

2024-10-01 11:05:18 +00:00

Author

Member

The motivation is to be able to downgrade to previous package versions
of packages that previously landed my systems, without going more
inconvenient way like koji-tool (I have to know the NVRs).

Sometimes I'm interested to see dnf diff (change in configuration),
which requires the ability to download the previous NEVRA of the package
that's no longer in the Everything repo.

Another use-case would be to have ability to generate rpm lockfiles, see e.g. this issue:
https://github.com/rpm-software-management/libpkgmanifest/issues/7

It's hard to estimate how much time or effort this could save me, but this
functionality seems essential for regular Rawhide users. Whenever I want to
experiment with older versions, it's often cumbersome and non-trivial. Recently,
I noticed that Fedora’s stable versions include this feature in
fedora-repos-archive.rpm, which is a huge relief.

The motivation is to be able to downgrade to previous package versions of packages that previously landed my systems, without going more inconvenient way like `koji-tool` (I have to know the NVRs). Sometimes I'm interested to see `dnf diff` (change in configuration), which requires the ability to download the previous NEVRA of the package that's no longer in the Everything repo. Another use-case would be to have ability to generate rpm lockfiles, see e.g. this issue: https://github.com/rpm-software-management/libpkgmanifest/issues/7 It's hard to estimate how much time or effort this could save me, but this functionality seems essential for regular Rawhide users. Whenever I want to experiment with older versions, it's often cumbersome and non-trivial. Recently, I noticed that Fedora’s stable versions include this feature in `fedora-repos-archive.rpm`, which is a huge relief.

phsmoura commented

2024-10-01 19:07:24 +00:00

Metadata Update from @phsmoura:

Issue assigned to kevin
Issue tagged with: low-gain, low-trouble, ops

**Metadata Update from @phsmoura**: - Issue assigned to kevin - Issue tagged with: low-gain, low-trouble, ops

james commented

2024-10-01 19:19:45 +00:00

Member

I assume that we don't want to keep the history indefinitely, but could we for
example keep history across say max two %dist versions?

Currently development is ~1.4TB

So assuming we only do this for the binary package repos, and don't do any depsolving, that would be:

development/rawhide/Everything/aarch64/os/Packages = ~100GB
development/rawhide/Everything/x86_64/os/Packages = ~120GB

...so for each older version of a package we'd be adding roughly 220GB, or about a 14% increase.

...adding source/debuginfo makes this bigger, obviously.

Sometimes I'm interested to see dnf diff

You can upgrade the package and then run diff.

> I assume that we don't want to keep the history indefinitely, but could we for example keep history across say max two %dist versions? Currently development is ~1.4TB So assuming we _only_ do this for the binary package repos, and don't do any depsolving, that would be: development/rawhide/Everything/aarch64/os/Packages = ~100GB development/rawhide/Everything/x86_64/os/Packages = ~120GB ...so for each older version of a package we'd be adding roughly 220GB, or about a 14% increase. ...adding source/debuginfo makes this bigger, obviously. > Sometimes I'm interested to see dnf diff You can upgrade the package and then run diff.

james commented

2024-10-01 19:21:04 +00:00

Member

Also dnf history should contain the nvrs, making downloading the packages directly significantly easier.

kevin commented

2024-10-02 17:29:41 +00:00

Owner

So, this repo was setup by @dustymabe for coreos to handle a corner case for them (where people layer packages, but then then a dep of that layered package is no longer available or something).

It's going to a single s3 bucket. It has no caching.

It was not intended for regular fedora users, but as I feared people started using it for this. ;(

rawhide is going to be a LOT more resources. Because there's not just occasional updates, but rather constant updates of big things, mass rebuilds where 100% of packages are rebuilt, etc.

This 'allow multiple versions of packages' thing has been discussed for years and years, and I don't think we should abuse this use case for this.

So, this repo was setup by @dustymabe for coreos to handle a corner case for them (where people layer packages, but then then a dep of that layered package is no longer available or something). It's going to a single s3 bucket. It has no caching. It was not intended for regular fedora users, but as I feared people started using it for this. ;( rawhide is going to be a LOT more resources. Because there's not just occasional updates, but rather constant updates of big things, mass rebuilds where 100% of packages are rebuilt, etc. This 'allow multiple versions of packages' thing has been discussed for years and years, and I don't think we should abuse this use case for this.