poor transfer rates and broken quick-fedora-mirror #13335
Labels
No labels
announcement
anubis
authentication
aws
backlog
blocked
bodhi
ci
cloud
communishift
copr
database
day-to-day
dc-move
deprecated
dev
discourse
dns
downloads
easyfix
epel
firmitas
forgejo_migration
Gain
High
Gain
Low
Gain
Medium
gitlab
greenwave
hardware
help wanted
high-trouble
koji
koschei
lists
low-trouble
medium-trouble
mirrorlists
monitoring
Needs investigation
odcs
OpenShift
ops
outage
packager_workflow_blocker
pagure
permissions
Priority
Needs Review
Priority
Next Meeting
Priority
🔥 URGENT 🔥
Priority
Waiting on Assignee
Priority
Waiting on External
Priority
Waiting on Reporter
rabbitmq
release-monitoring
releng
request-for-resources
s390x
security
SMTP
sprint-0
sprint-1
src.fp.o
staging
unfreeze
waiverdb
websites-general
wiki
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
infra/tickets#13335
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description of request
Hello,
I am one of the maintainers of fedora.tu-chemnitz.de.
We are currently struggling to keep our mirror up to date, particularly during the high-traffic periods surrounding new Fedora releases. After monitoring the download rates of dl01 to dl05 and other co-located servers, we observed that transfer speeds consistently remain below 400 KB/s for 98% of the time. Furthermore, we are seeing a high frequency of unexpectedly closed connections and drops.
Until yesterday, we used an extended version of quick-fedora-mirror, which temporarily mitigated these connection issues. However, this is no longer sufficient. After investing weeks of development into a new version of the tool, we have decided to halt that effort, as we believe these fundamental issues must be addressed on the Tier0 side.
We have identified the following critical challenges:
Reason: Slow and unstable connections mean these large images block bandwidth for essential package updates, often failing to complete.
Metadata Inconsistency: Timestamps in fulltimefilelist do not match the actual file timestamps.
Partial File Handling: quick-fedora-mirror does not handle .partial files correctly. Following a connection loss, most downloads are deleted, leading to bloated transfer lists.
State Tracking: The tool saves the last transfer time to a file to identify new content but fails to verify if this timestamp is accurate.
Check-in Errors: The check-in URL used for the MirrorMaster is outdated.
We have consulted with other mirrors (including linux.cz and hs-esslingen.de), who confirmed they are facing identical challenges and have switched to custom rsync scripts. This shift likely leads to out-of-sync mirrors because the fulltimefilelist is no longer utilized.
On a personal and organizational note: I have invested a significant amount of time into troubleshooting this, but we are facing a severe lack of personnel resources. At this point, I can no longer justify further development or extensive troubleshooting to my superiors.
I deeply regret this situation, as we want to remain a reliable source for Tier2 mirrors. However, if we cannot ensure a stable sync from Tier0, we will be forced to reconsider our role as a mirror.
Maybe you can take a look to ftpsync of debian.
Best regards,
Patrick
Maintainer, fedora.tu-chemnitz.de
Hello. Sorry you are seeing issues here. ;(
On the slowness: Can you tell me if you are accessing via ipv4 or ipv6? We have two uplinks, so it would be good to know which one you are hitting. Perhaps you could provide a traceroute from your sync endpoint?
Can you test and see if 'download-ib01.fedoraproject.org' is any faster for you? Thats in another datacenter with different connectivity.
On the other issues... I can look, but don't see why timestamps would not be in sync, our release process updates them at the end after new content is synced. Unless you are seeing us stage content before we have updated fullfiletimelist?
In any case probibly we should make improvements on the actual upstream quick mirror if there are improvements to be had?
Thanks for working on this, and thanks for bringing it up here so we can all work on it and not cause more work for you.
Hi Kevin,
thank you for your reply. It doesn't matter if we are using ipv4 or ipv6.
The trace of ipv4 to dl01.fedoraproject.org:
The trace of ipv4 to dl01.fedoraproject.org:
Attached the Statistics we got, yesterday with poor rates and today with better rates.
Example
grub.cfgfor fullfiletimelist-fedora, resolved:Fr 6. Mär 15:59:47 CET 2026But the stat of file shows:
So my question is: what is this timestamp based on, and how should it be used? I would determine whether a file is up to date based on whether the mtime matches.
ok, thanks for info.
The basic idea of quick-mirror is to save us from the stat storms that regular rsync does. So:
It's really a big win for transferring.
I think the problems you are seeing are at somehow a network or transit layer, not quick mirror, but I could be wrong. Thanks for working through this and trying to track it down.
Did you get a chance to try download-ib01? That would be another good datapoint as to if it's the network path or something else.
@kevin wrote in #13335 (comment):
The mean speed over the last 7 days, resulting :
dl01.fedoraproject.org: 31.1 kB/s
dl02.fedoraproject.org: 73.2 kB/s
dl03.fedoraproject.org: 43.8 kB/s
dl04.fedoraproject.org: 71.5 kB/s
dl05.fedoraproject.org: 73.9 kB/s
download-cc-rdu01.fedoraproject.org: 30.6 kB/s
download-ib01.fedoraproject.org: 17.4 kB/s
dl04 and dl05 are a little bit faster, cause of higher peaks.
fulltimefilelist
Yes, of course! I’m aware of the benefits, which is why we used “quick-fedora-mirror” until Saturday. However, over the past few years, we’ve encountered some issues with the script, which were likely due to poor data transfer rates, especially around release times.
Transfers became slow, connections dropped, and completed transfers remained in the .
tmpdirectory and were discarded the next time quick-fedora-mirror was started. As a result, the list of files to be transferred kept growing. Since quick-fedora-mirror supports only one mirror, you are limited to a slow mirror during the transfer. And as a result, recurring connection drops lead to an ever-growing list of files, that needs to download.Another effect we found, the repomd.xml stopped syncing on 30. April, maybe the reason was, that the timestamp in the "$TIMEFILE" was somehow newer then the timestamp in the fullfiletimelist.
Checkin to mirrormaster did not work.
The "Exclude" option deletes files, so it is not possible to synchronize images and packages separately using quick-fedora-mirror.
A short list of troubleshooting:
Do you track how many Upstream Servers using quick-fedora-mirror? Maybe some servers do bare rsync (as we do since saturday) and so free connections are blocked (as you told).
Yes we did, see above.
dl05% stat /srv/pub/fedora/linux/releases/44/Kinoite/x86_64/os/boot/grub2/grub.cfg
File: /srv/pub/fedora/linux/releases/44/Kinoite/x86_64/os/boot/grub2/grub.cfg
Size: 1490 Blocks: 8 IO Block: 65536 regular file
Device: 0,63 Inode: 5516910 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 263/ UNKNOWN) Gid: ( 263/ UNKNOWN)
Context: system_u:object_r:nfs_t:s0
Access: 2026-05-13 18:55:15.433537000 +0000
Modify: 2026-04-22 14:30:18.734089000 +0000
Change: 2026-04-24 10:26:11.250836000 +0000
Birth: -
So the change time matches as expected on the actual master mirror.
Which repomd.xml? Note that repomd.xml files are 'special' in that they also get checksums included.
I've not heard any reports of other mirrors seeing that, or the other mirrors we run that use quick-mirror-fedora.
checkins are only supported for private mirrors, and they changed a while back from xmlrpc to rest I think.
public mirrors shouldn't need to check in as far as I know.
Sadly, I don't think there's any way for us to know. quick-fedora-mirror and rsync look identical from the server side. ;(
I can look and see if we are hitting rsync server limits, but if we are, you would just get a reject not slowness...
So, this slowness seems like a network issue upstream of us. I can ask our networking folks if there is anything they could do, perhaps they can adjust so traffic from you goes over our other link. Can you provide also a traceroute to download-ib01? That might tell us what provider/network might be involved here (since it's also slow for you).
I guess I can also press for a .eu master mirror, but I am not sure where we could locate it. If we don't do archive, perhaps we could do something in a cloud provider (as long as we can sync to it with reasonable speed).
@kevin wrote in #13335 (comment):
Did you compare the changetime from the file to the changetime in fulltimelist?
Do you have anything similar to: https://mirror-master.debian.org/status/mirror-hierarchy.html to detect outdated mirrors?
We also get rejected sometimes. If I have more time i can update our scripts to track connection rejects.
I contacted Jan of muni.cz and Adrian hs-esslingen to share their experiences, maybe its a peering issue.
Hi all, maintainer of
ftp.fi.muni.cz/ftp.linux.czhere.It indeed seems to be either peering or generally network bandwidth issue. I did a quick test - downloading
fedora-enchilada/fullfiletimelist-fedorausing rsync fromdl01-dl05anddownload-ib01, both IPv4 and IPv6, using this script.I am getting reasonable bandwidth from
dl0[345], and connection errors or poor speed fromdl0[12]:-Yenya
Yeah, dl01/02/03 are all 'public' ones... so they allow any rsync (with a connection limit) and also are mirrors of last resort for http/https dnf requests (they appear at the bottom of the metalink/mirrorlist).
I'm not sure why 03 would be slower than 01/02. I'll look and see if I can see anything on the server end.
I did notice that ftp.hrz.tu-chemnitz.de. is making a lot of rsync connections:
Is it getting dropped or something? or in some kind of infinite loop?
So, any news here? Things better, or worse, or the same?
Any other feedback here? Still slow? better? worse?
From
ftp.fi.muni.cz/ftp.linux.czit is a bit worse today, also more IPv6 failures fordl0[1-3]:Of course I picked a pretty bad day to ask. Yesterday was fedora 45 branching (so there is now a new f45 tree all mirrors have to sync (although it is hardlinked to rawhide)). Also, somehow f44 release dirs got their permissions messed up and some mirrors removed those trees, only to sync again once it was fixed. :( So, lots of mirror activity this week.
When you have failed above, what does that mean? you got a connection refused? rsync full?
@kevin: it simply means rsync returned non-zero return code.
I ran it again, now it is a bit better, only one FAIL, but speeds are still not very good. The rsync error message from dl01/ipv6 was:
Timing data:
I just rolled out the suggestion in #13504 (to switch to bbr)
Can you run again now?
Definitely looks better for me - these are from last few hours:
Hello Kevin,
thanks for your work, you got the best day to ask. :-) Days like these were particularly critical in the past, but currently, I am seeing significantly better download rates.
I will continue to monitor it over the next few weeks.
have a nice weekend.
Thats encouraging... ;)
I'll leave this ticket for next week and see what folks are seeing then...
I'm also going to bump the number of rsyncs from 20 to 30. (That will mean a restart in a bit here, but hopefully less 'failed/full' messages.