Migration doesn't migrate issue attachments properly #321
Labels
No labels
Backlog Status
Needs Review
Backlog Status
Ready
Chore
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
forge/forge#321
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The issue attachments from Pagure repos are not migrated properly. The migrated image hyperlinks are give 404.
Example (image in initial comment): https://pagure.io/fedora-qa/testdays-web/issue/47
Migrated issue: https://forge.stg.fedoraproject.org/quality/testdays-web/issues/47
Pagure image link (works): https://pagure.io/fedora-qa/testdays-web/issue/raw/files/e55e4df624a67f458bfdd0ffcf4439a404140ab10b08fe488bd5d9e428970b17-Screenshot_From_2025-05-15_12-55-51.png
Forge image link (broken): https://forge.stg.fedoraproject.org/fedora-qa/testdays-web/issue/raw/files/e55e4df624a67f458bfdd0ffcf4439a404140ab10b08fe488bd5d9e428970b17-Screenshot_From_2025-05-15_12-55-51.png
Example2 (image in subsequent comment): https://pagure.io/fedora-qa/testdays-web/issue/55
Migrated issue: https://forge.stg.fedoraproject.org/quality/testdays-web/issues/55
Closing this one as a duplicate. WE have this one, where we investigated actaully importing all the attachments:
#262
And this one is still open tracking the redirect route that we are taking:
#275
Thanks, Ryan. Does this mean that I can count on those migrated attachment links to get functional once #275 is in place? Or is it safer to halt our migrations at this point and wait until this is resolved in one way or another?
All the tickets are closed but the attachments still don't work, so I'm reopening this ticket.
For production Forge, all the attachment links still give 404, example:
quality/fedora-easy-karma#58 (reference)
For staging Forge, the attachment tries to load for a long time, and then gives 503, example:
https://forge.stg.fedoraproject.org/quality/testdays-web/issues/55 (reference)
In the proxy logs I see this:
which indicates the issue is with SSL. Though just doing
curl https://stg.pagure.iofrom a console on the proxy works fine, so that's a bit odd.But I also see another issue. If you use a browser debug console you can see the attachment URL it tries to get is something like:
note the start of the URL has both
quality/testdays-web- which is the group/repo on forgejo - andfedora-qa/testdays-web- which is the group/repo on pagure. The HTTP config snippet we put in place is:so I don't think that would work even if the SSL issue was resolved; I think we'd wind up trying to get:
and that's going to be 404. Also I don't understand why the config snippet was written that way - the text between the two match groups is identical on both sides, so wouldn't:
have been much simpler? anyway.
Oh, and there's yet another problem: we migrated from prod pagure to staging forgejo, but the redirect is hardcoded to assume staging pagure. testdays-web doesn't actually exist on staging pagure - there's no https://stg.pagure.io/fedora-qa/testdays-web/ at all. Still, that wouldn't be a problem in production at least.
So...poking at the SSL aspect of this... If I do
curl -4 https://stg.pagure.io/fedora-qa/blocker-reviewfrom either staging proxy (using-4because httpd seems to be using IPv4 so I want to reproduce that), it works instantly. If I doopenssl s_client -4 -showcerts -connect stg.pagure.io:443, it immediately runs most of the way and says "Verify return code: 0 (ok)", then gets stuck atread R BLOCKfor 20-30 seconds, then printsclosedand exits. Not sure why there's the delay at the end, but that mostly looks OK, at least it doesn't indicate any SSL issues.So I don't know why the proxying is failing with SSL errors :/
Aha, I think I see the issue. Further on in
/etc/httpd/conf.d/forge.stg.fedoraproject.org/forge.confon the proxies is this:which I think means we use that as the source of trusted CAs for proxy requests. This seems to be to do with some proxying to do with balancer stuff - it comes from https://pagure.io/fedora-infra/ansible/blob/main/f/roles/httpd/reverseproxy/templates/reversepassproxy.conf . But I think it causes us to not trust the certificate of stg.pagure.io (or pagure.io) because don't use certs signed by the OCP CA, they use Let's Encrypt certs.
If I change
SSLProxyVerify requiretoSSLProxyVerify none, I get a 404 (which is expected because of the path problem and the 'testdays-web doesn't exist on staging pagure' problem) instead of SSL errors. There's still about a two minute delay. I think this might be caused by trying IPv6 and then falling back to IPv4 after a timeout;openssl s_clientwithout the-4behaves much the same.There is a https://pagure.io/fedora-infra/ansible/blob/main/f/roles/httpd/reverseproxy/templates/reversepassproxy.conf-nonopenshift-rdu3 that seems to be a sort of alternative to reversepassproxy.conf. openQA uses that one, for instance. It does not have an SSLProxyCACertificateFile directive. But I'm not totally sure whether using it for forge would be the right fix, and what the implications of that are.
Edit: https://access.redhat.com/solutions/5058261 notes that "with the Apache httpd version 2.4.30 and later SSLProxyCACertificateFile and SSLProxyProtocol is allowed in proxy section context". So we could probably change
reversepassproxy.confto specify the options it wants within each<Proxy>block, not outside them, and that might resolve this.I can't figure out yet how to force ProxyPass to use IPv4, though :/
so, we dealt with the IPv6 thing by fiddling with DNS so the host only resolves to IPv4 internally. Adding this to the httpd config file solves the SSL issue:
and that just leaves the doubled-path thing to figure out, I think. Will look at that next.
https://pagure.io/fedora-infra/ansible/pull-request/3006 fixes the SSL issue and the path issue (which turned out to be due to some...interesting forgejo behavior). There's still the problem of whether it's better for staging forgejo to proxy to prod or staging pagure, but that's at least only an issue for staging.
I did notice another problem while working on this: the text immediately before the image isn't rendered right either. On Pagure it looks like this:
On Forgejo it looks like this:
I assume this is to do with sanitization, but just displaying the HTML tags as plain text ought to be perfectly safe...I've filed a Forgejo issue on that.
Oh, forgot to update here: I went ahead and did a slightly ugly fix to specifically proxy testdays-web requests from staging forgejo to prod pagure, so the test testdays-web migration should be 'really fixed' now. Please take a look and see if everything looks good, if so I think we could close this.
@adamwill can you please summarize the current status? Is this supposed to be fixed both for forge.prod and forge.stg? And why is that testdays-web exception (above) necessary?
I still don't see images in quality/fedora-easy-karma#58 (compare with https://pagure.io/fedora-easy-karma/issue/58 ), so this doesn't seem to be working fully yet.
As for stg vs prod, ideally the migration script should remember where it migrated the project from (whether pagure.prod or pagure.stg), and the images should link to that location (regardless whether it's on forge.stg or forge.prod). If that's not possible/easy to do, then I'd say always redirect to pagure.prod, from both forge.prod and forge.stg. That makes testing migration possible (including images) and will be the majority use case.
OK, looking into the f-e-k case. The 404ing URLs have a three-element path, e.g.
https://forge.fedoraproject.org/quality/fedora-easy-karma/fedora-easy-karma/issue/raw/files/b598b490bcf140b0645078e66813ac3cc79cc318201c632e007d7f3a432a7086-before.png. I'll go back and look at the redirect logic and see what I got wrong.I don't think it's easy to do the 'remember where you migrated from' thing - at least, not in terms of 'get the information where it needs to go'. The redirects (proxies, really) are in apache config on the proxy hosts. That's a long way (in various senses) from the migration script in Forgejo. I also would choose to redirect to prod by default, but I think in Matrix chat somebody else had the opposite opinion...
Ohh. Wait. I think this is really easy: we didn't actually set up the reverse proxying for prod forgejo, it was done only for staging as a test. So right now prod forgejo doesn't proxy attachment requests to pagure at all. I'll fix that.
OK, yeah, looks like that was it. Seems like it's working now.
Great, it seems to be working now, thanks!
Should we keep this open until we resolve the "where should forge.stg redirect to?" question, or close it now?
I'd say let's close it, if you're particularly concerned about that, file a new ticket for it. It's easy enough to generalize the special-casing for whichever repos you want to handle (I can just use slightly fancier templating and a list).
Ok, closing, thanks.