Define a more appropriate fallback for the UA mirror

horth commented

2024-11-01 15:15:03 +00:00

As far as I know, recently UA mirrors went down, and currently the default mirror for UA region is routed to RU region.
As you probably know russia is occupied with waging genocidal war in Ukraine at the moment. This poses several problems:

Many ISPs in Ukraine block some or all russian domains, making this repo unavailable without VPN (or overriding ISPs DNS, or using a proxy, I haven't check to be fair);
Although packages are signed with gpg, and security risk is low, it can never be zero, and I wouldn't like to try my luck here.

So I have to ask of you to either:

Change how the fallback is chosen for UA region to avoid RU;
Provide a convenient method to ban mirrors or choose priority mirrors (perhaps Ubuntu UI for package manager could be used as some sort of inspiration).
Preferably both :)

I don't have a specific date it needs to be done before, but I'd appreciate if it's done as soon as possible.
And please accept my heartfelt gratitude for all the great things you do!

As far as I know, recently UA mirrors went down, and currently the default mirror for UA region is routed to RU region. As you probably know russia is occupied with waging genocidal war in Ukraine at the moment. This poses several problems: 1. Many ISPs in Ukraine block some or all russian domains, making this repo unavailable without VPN (or overriding ISPs DNS, or using a proxy, I haven't check to be fair); 2. Although packages are signed with gpg, and security risk is low, it can never be zero, and I wouldn't like to try my luck here. So I have to ask of you to either: - Change how the fallback is chosen for UA region to avoid RU; - Provide a convenient method to ban mirrors or choose priority mirrors (perhaps Ubuntu UI for package manager could be used as some sort of inspiration). Preferably both :) I don't have a specific date it needs to be done before, but I'd appreciate if it's done as soon as possible. And please accept my heartfelt gratitude for all the great things you do!

👍 1

kevin commented

2024-11-01 17:26:43 +00:00

Owner

Yeah, I am not sure what we can do in the mirrormanager backend... will leave that for @adrian and @abompard to comment on.
I would hope we could adjust things there tho.

As a personal workaround, you can add '&country=' to your metalink urls in /etc/yum.repos.d/
You can use 'eu' or 'global' or whatever region you like and you will get mirrors from that region.

Note that things are designed so compromised mirrors don't matter. You get the metalink from us directly, which has a list of mirrors, but also has the checksums of the repomd file.
So, if a mirror tampers with anything, the checksum doesn't match and dnf goes to the next one. If they change packages/files, those checksums are in the repomd file, so it would either never use them or the repomd is wrong. Finally, all packages are signed.

Anyhow, hopefully we can do something here and thanks for bringing this to our attention. ;)

Yeah, I am not sure what we can do in the mirrormanager backend... will leave that for @adrian and @abompard to comment on. I would hope we could adjust things there tho. As a personal workaround, you can add '&country=' to your metalink urls in /etc/yum.repos.d/ You can use 'eu' or 'global' or whatever region you like and you will get mirrors from that region. Note that things are designed so compromised mirrors don't matter. You get the metalink from us directly, which has a list of mirrors, but also has the checksums of the repomd file. So, if a mirror tampers with anything, the checksum doesn't match and dnf goes to the next one. If they change packages/files, those checksums are in the repomd file, so it would either never use them or the repomd is wrong. Finally, all packages are signed. Anyhow, hopefully we can do something here and thanks for bringing this to our attention. ;)

augenauf commented

2024-11-01 23:07:11 +00:00

for reference, related previous discussion: https://discussion.fedoraproject.org/t/looks-like-the-default-mirror-for-ukraine-is-russian-now-and-it-shouldnt-be/135132

I don't think dnf provides a way to exclude mirrors from a certain countries. You should use the "positive list" by modifying your metalink url.
Infra could only remove certain mirrors from their populated list - that's a bigger question.

for reference, related previous discussion: https://discussion.fedoraproject.org/t/looks-like-the-default-mirror-for-ukraine-is-russian-now-and-it-shouldnt-be/135132 I don't think dnf provides a way to exclude mirrors from a certain countries. You should use the "positive list" by modifying your metalink url. Infra could only remove certain mirrors from their populated list - that's a bigger question.

phsmoura commented

2024-11-04 20:03:27 +00:00

Member

Metadata Update from @phsmoura:

Issue priority set to: Waiting on Assignee (was: Needs Review)
Issue tagged with: dev, medium-gain, medium-trouble

**Metadata Update from @phsmoura**: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: dev, medium-gain, medium-trouble

horth commented

2024-11-05 09:38:29 +00:00

Author

Thanks for your answers!
Addressing your recommendations: I did add the whitelists for myself, though I must say this process is not exactly user friendly, and it might stop some users from doing that. Besides, there can be no easy way to know which mirror is used, if it's not blocked and no error is thrown. You can't fix a problem that you don't know is present.
As for security concerns, I must admit, I don't know much about GPG and I trust your judgement there. But my thoughts are along the lines of: checksum can be bruteforced by appending some bytes to the binary (like how some cryptocurrencies are mined), and there are always vulnerabilities to be found, even if we don't know about them yet :) So someone motivated enough with abundant resources could do that or is it definitely impossible?
Also, if the infrastructure team has nothing to do with how fallback is determined, and I agree that removing that mirror altogether is a bigger question, perhaps I could reopen the issue in a more appropriate place, if you could point it to me?

Thanks for your answers! Addressing your recommendations: I did add the whitelists for myself, though I must say this process is not exactly user friendly, and it might stop some users from doing that. Besides, there can be no easy way to know which mirror is used, if it's not blocked and no error is thrown. You can't fix a problem that you don't know is present. As for security concerns, I must admit, I don't know much about GPG and I trust your judgement there. But my thoughts are along the lines of: checksum can be bruteforced by appending some bytes to the binary (like how some cryptocurrencies are mined), and there are always vulnerabilities to be found, even if we don't know about them yet :) So someone motivated enough with abundant resources could do that or is it definitely impossible? Also, if the infrastructure team has nothing to do with how fallback is determined, and I agree that removing that mirror altogether is a bigger question, perhaps I could reopen the issue in a more appropriate place, if you could point it to me?

kevin commented

2024-11-05 17:51:37 +00:00

Owner

Adding or changing the binary rpms would show as a invalid checksum and dnf wouldn't use that mirror. Sure, there's always bugs, but the danger here is very low IMHO.

I was waiting to hear from @adrian and @abompard who work on the mirrormanager code. They should be able to say if this can be adjusted or not.

I suppose we could re-file this upstream, but... it would be waiting on the same folks, so perhaps we just be patient here and wait for them to chime in?

Adding or changing the binary rpms would show as a invalid checksum and dnf wouldn't use that mirror. Sure, there's always bugs, but the danger here is very low IMHO. I was waiting to hear from @adrian and @abompard who work on the mirrormanager code. They should be able to say if this can be adjusted or not. I suppose we could re-file this upstream, but... it would be waiting on the same folks, so perhaps we just be patient here and wait for them to chime in?

horth commented

2024-11-05 17:57:07 +00:00

Author

Sure thing ;)

adrian commented

2024-11-05 18:17:42 +00:00

We do not have defined fallbacks for each country. What we basically do is we look at the country of the client using GeoIP and then we select all mirrors in the same country. The next step is to select all mirrors on the same continent. Mirrors on the same continent are not selected by distance. We just use the GeoIP database to figure out which countries are on the same continent.

If you use the mirrorlist interface and not the metalink interface you can see how MirrorManager decides what mirror to include in the mirrorlist. The following is an example for a host with an IP in the Ukraine:

$ curl "https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-40&arch=x86_64&"
# repo = fedora-40 arch = x86_64 country = UA country = RO country = EE country = LU country = DK country = SE country = DE country = HU country = CZ country = LT country = FI country = BG country = SK country = GR country = PL country = NO country = BY country = RU country = FR country = GB country = CH country = PT country = NL

So we just use what GeoIP gives us and we cannot currently change those results. You can limit mirrors to a single country using:

$ curl "https://mirrors.fedoraproject.org/metalink?repo=fedora-40&arch=x86_64&country=ua"

If one of the mirrors in the returned list is not working DNF will go to the next mirror. So besides a message that some mirror could not be used it should just work like @kevin described earilier.

We do not have defined fallbacks for each country. What we basically do is we look at the country of the client using GeoIP and then we select all mirrors in the same country. The next step is to select all mirrors on the same continent. Mirrors on the same continent are not selected by distance. We just use the GeoIP database to figure out which countries are on the same continent. If you use the mirrorlist interface and not the metalink interface you can see how MirrorManager decides what mirror to include in the mirrorlist. The following is an example for a host with an IP in the Ukraine: ``` $ curl "https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-40&arch=x86_64&" # repo = fedora-40 arch = x86_64 country = UA country = RO country = EE country = LU country = DK country = SE country = DE country = HU country = CZ country = LT country = FI country = BG country = SK country = GR country = PL country = NO country = BY country = RU country = FR country = GB country = CH country = PT country = NL ``` So we just use what GeoIP gives us and we cannot currently change those results. You can limit mirrors to a single country using: ``` $ curl "https://mirrors.fedoraproject.org/metalink?repo=fedora-40&arch=x86_64&country=ua" ``` If one of the mirrors in the returned list is not working DNF will go to the next mirror. So besides a message that some mirror could not be used it should just work like @kevin described earilier.

rokejulianlockhart commented

2024-11-07 23:30:11 +00:00

We do not have defined fallbacks for each country. What we basically do is we look at the country of the client using GeoIP and then we select all mirrors in the same country. The next step is to select all mirrors on the same continent. Mirrors on the same continent are not selected by distance. We just use the GeoIP database to figure out which countries are on the same continent.

@adrian, I dare say that that seems like it might need to change. As an example, a user in China may not want Fedora to automatically request access to domains in countries geographically near to it, for they may be hostile to the PRC and thus blocked by the national firewall. Connecting to those domains may be quite seriously dangerous for the user — law enforcement don't much care for technical excuses.

Likewise, a Ukrainian running Fedora may very reasonably not want their computer to connect to domains of a nation-state at war with them, especially if they possess security clearance (as is more common in wartime, when much of the populace are employed by the armed forces). The military don't tend to care much for technical excuses, although the consequence is less severe.

Expecting the standard user of Fedora to manually configure dnf via the CLI appears unreasonable, especially considering that it may not be obvious to the user that this is even occurring.

> We do not have defined fallbacks for each country. What we basically do is we look at the country of the client using GeoIP and then we select all mirrors in the same country. The next step is to select all mirrors on the same continent. Mirrors on the same continent are not selected by distance. We just use the GeoIP database to figure out which countries are on the same continent. @adrian, I dare say that that seems like it might need to change. As an example, a user in China may not want Fedora to automatically request access to domains in countries geographically near to it, for they may be hostile to the PRC and thus blocked by the national firewall. Connecting to those domains may be quite seriously dangerous for the user — law enforcement don't much care for technical excuses. Likewise, a Ukrainian running Fedora may very reasonably not want their computer to connect to domains of a nation-state at war with them, especially if they possess security clearance (as is more common in wartime, when much of the populace are employed by the armed forces). The military don't tend to care *much* for technical excuses, although the consequence is less severe. Expecting the standard user of Fedora to manually *configure* `dnf` via the CLI appears unreasonable, especially considering that it may not be obvious to the user that this is even occurring.

smooge commented

2024-11-08 13:41:47 +00:00

Most of the code for mirrormanager was written a long time ago when these sorts of concerns were not a pressing item and in fact various times attempts to change it to that were vetoed by a majority of people running mirrors or using Fedora as against the spirit of the Internet. The world has changed a lot but the code of mirrormanager would probably need major rewrites to match it.

All of the people working on it are doing so in spare time so if people have patches which can help, I expect they would be appreciated.

Most of the code for mirrormanager was written a long time ago when these sorts of concerns were not a pressing item and in fact various times attempts to change it to that were vetoed by a majority of people running mirrors or using Fedora as against the spirit of the Internet. The world has changed a lot but the code of mirrormanager would probably need major rewrites to match it. All of the people working on it are doing so in spare time so if people have patches which can help, I expect they would be appreciated.

triatic commented

2024-11-08 15:06:51 +00:00

@rokejulianlockhart I'm curious as how to such things would work in practice. Would Fedora need a panel which meets regularly to discuss international politics and vote on which clients connect to which mirrors?

rokejulianlockhart commented

2024-11-08 16:12:02 +00:00

@rokejulianlockhart I'm curious as how to such things would work in practice. Would Fedora need a panel which meets regularly to discuss international politics and vote on which clients connect to which mirrors?

@triatic, this doesn't necessarily have to be a proactive decision process. It doesn't even have to be political, per se.

The cryptographic signatures of the packages held within every official mirror remain undisputed, and there have been no obvious attempts to attack users. Consequently, it would merely be a matter of acting upon reports from users that they are unable to access a mirror in a specific country, and then checking whether that country actually has deliberately blocked the other (or whether their LAN is just buggered).

If this block is revoked by the nation, this may put undue strain on other mirrors, so a review process in that case might be of use. However, I would be surprised if the mirror operators wouldn't immediately notify Fedora of revocations.

The current blockers appear to entirely technical, as usual (fortunately).

> @rokejulianlockhart I'm curious as how to such things would work in practice. Would Fedora need a panel which meets regularly to discuss international politics and vote on which clients connect to which mirrors? @triatic, this doesn't necessarily have to be a proactive decision process. It doesn't even have to be political, per se. The cryptographic signatures of the packages held within every official mirror remain undisputed, and there have been no obvious attempts to attack users. Consequently, it would merely be a matter of acting upon reports from users that they are unable to access a mirror in a specific country, and then checking whether that country actually has deliberately blocked the other (or whether their LAN is just buggered). If this block is revoked by the nation, this may put undue strain on other mirrors, so a review process in that case might be of use. However, I would be surprised if the mirror operators wouldn't immediately notify Fedora of revocations. The current blockers appear to entirely technical, as usual (fortunately).

augenauf commented

2024-11-08 19:47:15 +00:00

Finally, it is an individual end-user decision on which mirrors shall be used or not used. This can't be fixed on the backend, it's not a matter of the Project or mirrormanager to decide what you want or not want.
Instead, it could be useful to have little tool, say a dnf plugin, to append the country definition to every repo enabled. For example,

 &country=DE,FR,BE,CH,NL,SE,US

This step is easy to do but sort of annoying setting it up by hand. For newbies, it may even be difficult.

I think, in Debian and/or debian-derived distros you have a Gnome GUI to modify repos and select mirrors. Someone up for coding a little tool?

Finally, it is an individual end-user decision on which mirrors shall be used or not used. This can't be fixed on the backend, it's not a matter of the Project or mirrormanager to decide what you want or not want. Instead, it could be useful to have little tool, say a dnf plugin, to append the country definition to every repo enabled. For example, &country=DE,FR,BE,CH,NL,SE,US This step is easy to do but sort of annoying setting it up by hand. For newbies, it may even be difficult. I think, in Debian and/or debian-derived distros you have a Gnome GUI to modify repos and select mirrors. Someone up for coding a little tool?

horth commented

2024-11-08 20:01:49 +00:00

Author

I think this tool is a great way to partially solve this problem!
Another part of the problem remains, however: users need to be informed, that they may have a reason to look for and use this tool. What I mean is: I wouldn't call myself a newbie, but I wouldn't know that I'm routed to a mirror I don't want to use, if it wasn't for dnf printing 404s because my ISP blocks a russian mirror.
Maybe we could agree on some indication of which mirror is currently used in dnf itself? As little as a country code near the progress bar would already be of great help.

I think this tool is a great way to partially solve this problem! Another part of the problem remains, however: users need to be informed, that they may have a reason to look for and use this tool. What I mean is: I wouldn't call myself a newbie, but I wouldn't know that I'm routed to a mirror I don't want to use, if it wasn't for `dnf` printing 404s because my ISP blocks a russian mirror. Maybe we could agree on some indication of which mirror is currently used in `dnf` itself? As little as a country code near the progress bar would already be of great help.

rokejulianlockhart commented

2024-11-08 20:16:58 +00:00

Maybe we could agree on some indication of which mirror is currently used in dnf itself? As little as a country code near the progress bar would already be of great help.

@horth, something on the CLI would be useful for most of us here, but many users don't use it. At all. Perhaps something that integrates with the desktop notification API too, like invent.kde.org/plasma/plasma-workspace/-/merge_requests/4204#note_1057791 does, would be more comprehensive? At the least, ≥ GNOME 41 and KDE Plasma 6.0 would support it immediately.

I'm referring to repository access failures, of course. Merely indicating the country in a notification would be useless; it would instead be useful as additional information in a repository access failure notification.

Maybe we could agree on some indication of which mirror is currently used in dnf itself? As little as a country code near the progress bar would already be of great help.

I'd second that. A standard ISO 3166 Alpha-2 or 3 (or UN/LOCODE) name would be perfect.

Someone up for coding a little tool?

@augenauf, I'll at least try testing it…? Can't assist in development.

> Maybe we could agree on some indication of which mirror is currently used in `dnf` itself? As little as a country code near the progress bar would already be of great help. @horth, something on the CLI would be useful for most of us here, but many users don't use it. At all. Perhaps something that integrates with the desktop notification API too, like [`invent.kde.org/plasma/plasma-workspace/-/merge_requests/4204#note_1057791`](https://invent.kde.org/plasma/plasma-workspace/-/merge_requests/4204#note_1057791) does, would be more comprehensive? At the *least*, ≥ GNOME 41 and KDE Plasma 6.0 would support it immediately. I'm referring to repository access failures, of course. *Merely* indicating the country in a notification would be useless; it would instead be useful as *additional* information in a repository access failure notification. > Maybe we could agree on some indication of which mirror is currently used in dnf itself? As little as a country code near the progress bar would already be of great help. I'd second that. A standard ISO 3166 Alpha-2 or 3 (or UN/LOCODE) name would be perfect. --- > Someone up for coding a little tool? @augenauf, I'll at least try testing it…? Can't assist in development.

horth commented

2024-11-08 20:29:49 +00:00

Author

Hmm, I'm not sure how one would update dnf packages outside of command line, but I would guess that via the "Software" GUI app? I just went to check it, and there is a "Software Repositories" submenu, which seems like it already has a nice base for GUI mirror chooser. But it's kinda hard to find and notice, esp. for a user that doesn't pay much attention to things like updates. So, I'm thinking maybe a set of country flags in the update menu indicating which mirrors are used above the list of updates would do as an alternative for UI?

Hmm, I'm not sure how one would update `dnf` packages outside of command line, but I would guess that via the "Software" GUI app? I just went to check it, and there is a "Software Repositories" submenu, which seems like it already has a nice base for GUI mirror chooser. But it's kinda hard to find and notice, esp. for a user that doesn't pay much attention to things like updates. So, I'm thinking maybe a set of country flags in the update menu indicating which mirrors are used above the list of updates would do as an alternative for UI? [![Screenshot_From_2024-11-08_22-24-52.png](/fedora-infrastructure/issue/raw/files/dddc98536e16b30f384a94088669fb8bd9af4e517a0e2dcd98d6fd252850b12f-Screenshot_From_2024-11-08_22-24-52.png)](/fedora-infrastructure/issue/raw/files/dddc98536e16b30f384a94088669fb8bd9af4e517a0e2dcd98d6fd252850b12f-Screenshot_From_2024-11-08_22-24-52.png)

rokejulianlockhart commented

2024-12-26 20:23:13 +00:00

I'm not sure how one would update dnf packages outside of command line, but I would guess that via the "Software" GUI app?

@horth, there is no "the [...] software GUI app". Rather, that's GNOME Software. Irrespective, I agree, but such ideas would probably be better pitted at bugs.kde.org/enter_bug.cgi?product=Discover and gitlab.gnome.org/GNOME/gnome-software/issues. I'll support any issues you propose there if you link them here.

> I'm not sure how one would update dnf packages outside of command line, but I would guess that via the "Software" GUI app? @horth, there is no "*the* [...] software GUI app". Rather, that's GNOME Software. Irrespective, I agree, but such ideas would probably be better pitted at [`bugs.kde.org/enter_bug.cgi?product=Discover`](https://bugs.kde.org/enter_bug.cgi?product=Discover&component=PackageKit) and [`gitlab.gnome.org/GNOME/gnome-software/issues`](https://gitlab.gnome.org/GNOME/gnome-software/-/issues/new). I'll support any issues you propose there if you link them here.

kevin commented

2025-10-23 16:51:43 +00:00

Owner

So, this ticket has languished sadly. :(

We discussed it today in our backlog discussions.

One question that came up... how are the blocks done there? ie, what does it look like if you just 'dnf update' ? Does it just 404 on the ru mirror? or 401? or timeout?

For a country override, it would take work in mirrormanager, so perhaps we should refile this upstream?

So, this ticket has languished sadly. :( We discussed it today in our backlog discussions. One question that came up... how are the blocks done there? ie, what does it look like if you just 'dnf update' ? Does it just 404 on the ru mirror? or 401? or timeout? For a country override, it would take work in mirrormanager, so perhaps we should refile this upstream?

grandis commented

2026-01-23 10:01:54 +00:00

@kevin wrote in #12270 (comment):

One question that came up... how are the blocks done there? ie, what does it look like if you just 'dnf update' ? Does it just 404 on the ru mirror? or 401? or timeout?

That's how it looks in Terminal when "dnf upgrade" tries to download new packages.

@kevin wrote in https://forge.fedoraproject.org/infra/tickets/issues/12270#issuecomment-270275: > One question that came up... how are the blocks done there? ie, what does it look like if you just 'dnf update' ? Does it just 404 on the ru mirror? or 401? or timeout? That's how it looks in Terminal when "dnf upgrade" tries to download new packages.

Screenshot From 2026-01-23 11-50-24.png

168 KiB

horth commented

2026-01-23 10:09:56 +00:00

Author

@kevin wrote in #12270 (comment):

So, this ticket has languished sadly. :(

We discussed it today in our backlog discussions.

One question that came up... how are the blocks done there? ie, what does it look like if you just 'dnf update' ? Does it just 404 on the ru mirror? or 401? or timeout?

For a country override, it would take work in mirrormanager, so perhaps we should refile this upstream?

It's also a timeout for me, but I suspect it might be ISP dependent.

@kevin wrote in https://forge.fedoraproject.org/infra/tickets/issues/12270#issuecomment-270275: > So, this ticket has languished sadly. :( > > We discussed it today in our backlog discussions. > > One question that came up... how are the blocks done there? ie, what does it look like if you just 'dnf update' ? Does it just 404 on the ru mirror? or 401? or timeout? > > For a country override, it would take work in mirrormanager, so perhaps we should refile this upstream? It's also a timeout for me, but I suspect it might be ISP dependent.

james commented

2026-04-16 18:36:52 +00:00

Member

I pushed opened this issue upstream as it's very unlikely that fedora infra. will have the ability/time to patch MM.

That doesn't mean it'll get fixed soon, but it's maybe got a better chance now.

I pushed opened this issue [upstream](https://github.com/adrianreber/mirrorlist-server/issues/383) as it's very unlikely that fedora infra. will have the ability/time to patch MM. That doesn't mean it'll get fixed soon, but it's maybe got a better chance now.

👍 1

patrikp commented

2026-08-13 15:53:59 +00:00

Member

We came across this ticket during backlog refinement. Seeing how there is now a ticket open upstream, we decided to close this one. Please feel free to continue the discussion/work in this ticket:
https://github.com/adrianreber/mirrorlist-server/issues/383

We came across this ticket during backlog refinement. Seeing how there is now a ticket open upstream, we decided to close this one. Please feel free to continue the discussion/work in this ticket: https://github.com/adrianreber/mirrorlist-server/issues/383

patrikp closed this issue

2026-08-13 15:53:59 +00:00

Rows
Columns

Define a more appropriate fallback for the UA mirror #12270