fix(fedora-web): Exclude archives from mod_deflate to fix lookaside cache downloads #3170
No reviewers
kevin
Labels
No labels
ai-review-please
freeze-break-request
post-freeze
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
infra/ansible!3170
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "victorkoycheff/ansible:issue-12812-fix-deflate-headers"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Fixes #12812.
What this does:
Updates the
SetEnvIfNoCaseregex inroles/fedora-web/main/files/deflate.confto explicitly exclude common binary/archive extensions frommod_deflate.Why it's needed:
The frontend proxies were aggressively intercepting archive requests (like
.crate,.tar.gz,.rpm) that weren't in the original image exclusion list, attempting to compress them, and appending aContent-Encoding: gzip(orx-gzip) header. Clients likecurlandfedpkgwere honoring this header by double-decompressing the files on the fly, resulting in raw tarballs with.crateextensions and immediate checksum failures.Testing done:
I replicated the issue locally and confirmed that adding
(?:gz|tgz|bz2|bz|xz|iso|tar|crate|zst|rpm|srpm)to the exclusion list prevents Apache from applying gzip encoding to these files, passing them through exactly as they are.Signed-off-by: Victor Koycheff victorkoycheff@gmail.com
f54a3be1ddc31dc3de6eI’m not fully convinced by the explanation, because the clients should only decode one layer of gzip encoding when they see the
Content-Encoding: gzipheader. If clients were indeed served two layers of gzip encoding as described above (equivalent to.crate.gzor.tar.gz.gz), that ought to be slightly wasteful of CPU and bandwidth but basically harmless, as the clients would decode the outer layer and produce the original file intact. Instead, it seems the original files are somehow served unmodified, but with theContent-Encoding: gzipheader added, which causes the clients to decompress the pre-existing gzip layer. Still, whether or not the problem is truly fully understood, and even if the problem may reoccur in the future for some unforeseen archive format, this change ought to be helpful and a big step in the right direction. It’s wise to try to avoid double compression even if the resulting “turducken” were handled properly.c31dc3de6e3e97c302bdThank you for the feedback @music. You're completely right about the mechanics here. The proxies weren't double-compressing; they were just passing the original files through while falsely classifyingthem with a
Content-Encoding: gzipheader. This caused clients to obediently decompress the payload on the fly, resulting in raw files being written to disk under compressed extensions (like.tar.gz), which immediately broke the checksums.I have updated the commit message to accurately reflect this indeed 'false advertising' behavior. ; )
Thanks for the sharp eye
3e97c302bd0f787448f8So, this might be the right solution, but it's in the wrong place. ;(
This deflate.conf is used in the main 'fedoraproject.org' website config.
The issue to be fixed is happening in the src.fedoraproject.org site, which would be unaffected by this.
So, we need it in the right place, but I am not sure where that would be.
The path is:
request -> httpd on proxy -> anubis -> httpd on proxy -> varnish -> haproxy -> pkgs01 backend httpd
So, I guess the question is where the invalid header is being set.
0f787448f8de731d9035de731d903556f5a7f2f5Thanks for the feedback @kevin
I did some more digging into the roles and found the culprit. In
roles/httpd/website/templates/website.conf, there’s a globalSetOutputFilter DEFLATEapplied to thesrc.fedoraproject.orgVirtualHost. Becausesrcdoesn't use the legacydeflate.confexclusions, it was blindly adding theContent-Encoding: gzipheader to archives without actually compressing themI have refactored the PR as follows:
roles/fedora-web/main/files/deflate.conf.SetEnv no-gzip 1andSetEnv dont-vary 1to the<Directory /srv/cache/lookaside>block inroles/distgit/templates/lookaside.conf.Thus ensuring that mod_deflate is explicitly disabled for the lookaside cache at the directory level, regardless of which VirtualHost or alias is serving it.
Let me know if further refinement is needed
So, this is worth trying on staging I think...
Can you put all your changes in a
{% if env == 'staging' %}
block, so prod is not changed ? and remove WIP if you don't see any more changes needed before we try staging.
56f5a7f2f514ab06dabbWIP: fix(fedora-web): Exclude archives from mod_deflate to fix lookaside cache downloadsto fix(fedora-web): Exclude archives from mod_deflate to fix lookaside cache downloads14ab06dabbac57bbb2eaOkay, I wrapped the changes in the staging block (unindented is better, right?)... I also dropped the WIP prefix, so this should be ready to try out on staging whenever. ; )
@ -3,5 +3,12 @@ Alias /lookaside /srv/cache/lookasideOptions Indexes FollowSymLinksAllowOverride NoneRequire all grantedHate to be nit picky, but can you drop this blank line for now?
With it, it means if someone runs the prod playbook this will have "changed" and it may restart things, etc.
Ideally this doesn't change production at all if it's run there.
ac57bbb2ea4af6011bb84af6011bb81fa76e4de0Nit pickiness is more than fine, negligence is the bad one ; )
I did the change as requested @kevin
Thanks! Lets try it in staging then...