Docs localization migration from pagure.io to forge.fedoraproject.org #52
Labels
No labels
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Depends on
Reference
localization/tickets#52
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
A tracking issue for the migration of Docs localization repositories from pagure.io to the new Fedora Forge.
Progress of the migration is mostly recorded in the comments. If there was a need, another issue can be filed (and linked as dependency).
The plan is:
List of (scheduled?) tasks:
Migrated repositories: Table moved to a wiki page
The plan is, of course, a subject of a discussion and/or modification.
there is a ticket to open for infrastructure team to stop the current automation system. I will open it and link it to this ticket.
I will have to migrate the script by opening a pull request on https://pagure.io/fedora-docs/translations-scripts/ (should we migrate this repository first?)
I suggest to lock all docs repositories in Weblate until our migration is done, and publish a message on top of each components linking to #52 (without sending email to users, we don't want to spam them ;))
after each repository migration, we can remove the message and unlock the repository
Is it meant that we want have translation-scripts under
localization-docs(notdocs) in the forge?I thought if the repositories would be mirroring the ones in pagure, we could wait with the locking repos in Weblate and the locks could not last long. The translation will be present in both forgejo and pagure. But I have no strong opinion here. It would resemble a 'big bang' approach. On the other hand, when we make locks in Weblate, we can 'a pilot' or 'POC' any one selected repository and see if everything in the chain (source -script - pot/po - script - translated sources) works. And if it works we can remove the lock for the repository.
I have to warn that I might not be very quick with preparing the repositories in either forge and Weblate. ;)
I am also not sure if I will be able to create the mirrors without authentization (permissions) to the pagure's fedora-docs-l10n repositories.
It is possible to pull from Pagure to Forgejo without permissions, but to push back indeed requires privileges. See Pulling from a remote repository, and the following section, from Forgejo docs.
I hope we do not need to push, at least for now.
@jibecfed wrote in #52 (comment):
Sorry, I have read this again now. I misunderstood before, I guess. You speak about migration by a pull request. So now I assume the repository will finally stay under the
docsorg (namespace) - under the Docs team responsibility. And you will use PR to adapt the scripts as they will work withourmigrated repositories under thelocalization-docs...?Originally, my idea was that we will create (migrate) all the pagure.io/projects/fedora-docs-l10n repositories to the forge.fp.o/localization-docs and make them mirroring the ones (old) in pagure. And we could work in the 'background' on the changes, while the translation automation would not have to be stopped for the Docs website. But today I have doubts that we can migrate the repositories to the forge and make them mirrors of the pagure ones. So, maybe, we will have to stop the translation automation and change the translations-scripts first, as you suggested. Then migrate repositories 'normally' and work further on the changes in the repositories and weblate. After we will have everything ready, then restart the translation automation.
This is a draft of the announcement which could be published in the Fedora Discussion announcements category. It could also serve as a base for information on the Weblate. Comments, suggestions.... are welcome.
Fedora Documentation translations are not available
Fedora Documentation translations are put on hold as the Fedora Localization Team has started the process of migration from pagure.io to the Fedora Forge. Translation projects of the documentation are locked on the Fedora translation platform. No translation updates can be found in languages on the Fedora Documentation.
The migration involves all repositories which support and ensure the availability of translations of the Fedora Documentation. There is no possibility the migration can be performed 'on the fly' as changes in the repositories, related scripts and continuous integration with the translation platform cannot be dealt with independently. Therefore the translation process of the Fedora Documentation is kept on hold.
We regrettably ask the Fedora contributors, our translation community, to pull back from translating of the Fedora Documentation and wait till the translation automation of the documentation is resumed again.
The progress of migration can be followed in the localization tracker as issue #52.
To migrate a repository from pagure to the forge so as the (new) migrated repository in the forge is a (pull] mirror of the (old) one repository in the pagure:
All repositories from pagure created in the forge - as mirrors; the aim was to preserve the history of changes which can occur before we finally are ready to run translation automation using the (new) repos, thus to make the necessary locking translations in weblate as short as possible.
I migrated a few docs repositories to forgejo (only those related to translations)
https://pagure.io/fedora-docs/localization
migrated as https://forge.fedoraproject.org/docs/localization-stats
https://pagure.io/fedora-docs/translations-scripts
migrated as https://forge.fedoraproject.org/docs/translations-scripts
https://pagure.io/fedora-docs/translated-sources/
migrated as https://forge.fedoraproject.org/docs/translated-sources
I created a request to infrastructure team to change the current automation to use these new repositories: infra/tickets#13171
Before the automation is changed to Fedora Forge repositories, i think we should 1) lock doc projects, 2) make sure forge repositories are updated by git-pulling changes from respective Pagure repository, 3) leave the projects locked (?)
We could also make some repos cleaning - removing unused, renaming...
I will write (hopefully today) my idea into an separate issue so you can voice your opinion if yes or no, and if as a part of the migration or leave it rather after.
@jibecfed I'm working on updating translation-scripts README, and one step is to add a fedmsg hook. This is straightforward in Pagure, but how does this apply in Forgejo?
I don't see anything related to fedmesg hook on https://docs.fedoraproject.org/en-US/forge-documentation/
I suggest to open an issue to get the info https://forge.fedoraproject.org/forge/forge/ (and maybe help them by updating the doc)
Wild guess: this is by default and there is nothing to do.
@peartown wrote in #52 (comment):
I suggest you keep track of weird things in https://fedoraproject.org/wiki/L10N/Forgejo or in the first message of this ticket
We'll handle it after we are done with the technical migration
Tomorrow the post in Community Blog will be published.
I intend to start locking the projects/components in Weblate on Wednesday and place a short notice for the translators with link to the post. After I lock a project I would like to work on its configuration changes (in Weblate and forge). And go project by project.
I would be grateful if you (Rafael?) could supervise my work/changes. ;)
Sure.
The steps are, basically:
How about add an announce about this? It could be to the whole Fedora Weblate instance like "Fedora docs translation projects are locked due to on-going migration to Fedora Forge. See <link to blog post>"
Exactly, something like this I had in my mind. 👍
We might need to set push from mirror to Pagure, or remove mirror setting.
See error from https://translate.fedoraproject.org/projects/fedora-docs-l10n-project/nav/:
See issue comment in https://github.com/go-gitea/gitea/issues/7609:
I assume setting push would solve the read-only status.
Yes, it was expected. I have chosen between having to remove the mirror setting in the forge, or to pull from pagure to forge in the moment of migration. Purpose of the mirroring was to retain the history and a sync'd state.
I would not keep pagure in sync after, ie. mirroring from forge to pagure. Am I not seeing anything...? :)
All translation projects locked in Weblate, except fedora-docs-l10n/release-notes. I tried (twice) to push over 1300 outgoing commits, but it failed. So I am not sure what to do with the project/repositories.
localization-docs/project repository is transferred to a normal repository (mirroring setting removed) in the forge. I also added the gitea webhook for it. But we are probably not able to push changes from Weblate into it, unless jibec logs in to the forge as weblatebot, then weblatebot should be synced/added as a member to localization-docs.
weblatebot is not listed as a member in https://forge.fedoraproject.org/org/localization-docs/teams/members, which might explain why we're getting access denied:
To confirm my previous message, I manually added weblatebot as member of https://forge.fedoraproject.org/localization-docs/project (Settings > Collaborators > Add collaborator weblatebot) and the translation push worked just fine. I removed it as this was a simple test.
for reference: forge/forge#448
for reference:
Gitea credentials in Fedora Weblate - https://care.weblate.org/#ticket/zoom/12178
Issue for Forgejo hook, pull requests and documentation - https://github.com/WeblateOrg/weblate/issues/18453
Just to announce that the gitea (forgejo) pull requests are now enabled in Fedora Weblate. (I have not tested the use yet.)
Another thing, as we are still not able to get the weblatebot user added into the localization-docs, see the comment above, we might, as a sort of a workaround, add it into the repositories manually and continue with the migration. Rafael and me tested that adding the weblatebot into the repository works. As a side note, I am also still not sure we, me and Jean-Baptiste, understood each other correctly what should and was done regarding the login of weblatebot into the forge.fedoraproject.org. @jibecfed, can you please confirm here that you did login as the weblatebot (fas) user into the forge.fedoraproject.org last week (I am really sorry for asking again)?
F44 release is approaching fast and there will likely not be Docs translation updates with the release.
Findings of March 30th Summary:
(Yesterday, me and rffontenelle tried to look further on the possible ways how to proceed with the migration.)
Finding I: Setting two-way mirrors, ie. where two repositories would mirror each other is not possible (because of read-only status of the forge repositories, no push from another remote(s) is possible).
If Weblate was set to push to the forge repositories:
Finding II: Forge and pagure repositories likely would not longer stay in sync.
Finding III: Stop (and changes) in translation automation is desirable. (#52 (comment))
If translation automation was stoped:
A possible question of keeping changes (pushes from translation automation) just in Weblate, ie. to unset pushing configuration(s) in Weblate in the translation projects, in the meantime the translation automation is not changed can be further explored.
Finding IV: Further steps can be taken without significant (negative) effect:
Added
https://translate.fedoraproject.org/hooks/gitea/webhook to all repositories under localization-docs.Added a request for stopping the translation automation in infra/tickets#13171. But there is the infra freeze unfortunately.
Now that translations scripts automation is stopped, I migrated the localization-docs/websites to Forge (i.e. all settings in Weblate and in Forge). See steps below.
For each project you're going to migrate, do the following:
l10n repository in forge.repository.org
translate.fedoraproject.org
<TOKEN>andREPO_NAMEaccordingly)https://weblatebot:<TOKEN>@forge.fedoraproject.org/localization-docs/<REPO_NAME>.gitDone migrating the following projects (expand for its URLs):
localization-docs/websites
weblate: https://translate.fedoraproject.org/projects/fedora-docs-l10n-websites/forge l10n: https://forge.fedoraproject.org/localization-docs/websites
docs.fp.o: https://docs.fedoraproject.org/en-US/websites/
upstream: https://gitlab.com/fedora/websites-apps/documentation/-/blob/main/modules/ROOT
localization-docs/teleirc-sig
weblate: https://translate.fedoraproject.org/projects/fedora-docs-l10n-teleirc-sig/forge l10n: https://forge.fedoraproject.org/localization-docs/teleirc-sig
docs.fp.o: https://docs.fedoraproject.org/en-US/teleirc-sig/
upstream: https://pagure.io/sig-teleirc/infrastructure
localization-docs/remix-building
weblate: https://translate.fedoraproject.org/projects/fedora-docs-l10n-remix-building/forge l10n: https://forge.fedoraproject.org/localization-docs/remix-building
docs.fp.o: https://docs.fedoraproject.org/en-US/remix-building/
upstream: https://pagure.io/fedora-docs/remix-building
I have just moved the table of projects/repos from OP to a wiki page. To enable edition by other people than me.
All Weblate translation projects migrated to Forge and unlocked, with a few exceptions listed below.
Pending answer, unsure how to proceed:
Should we create Weblate translation projects for the following repositories? These localization repository were migrated to Forge, but there are (and probably there never were) no Weblate translation projects for them:
I recommend removing the Weblate translation projects below because they are not published in docs.fp.o, which means they won't be built nor updated by the l10n infrastructure, so it would be a lose of translation effort:
Should we remove these localization repositories from https://forge.fedoraproject.org/localization-docs/ ?:
NOTE: unmigrated dead project:
@peartown can you have automation turned on?
This is my summary of darknao's discussion posts regarding of modifications needed to the translation scripts in order the translation automation works in prod:
script needs write access to everything under forge.fedoraproject.org/localization-docs/* and forge.fedoraproject.org/docs/translated-sources
support in that script to use a proper git token in the pushurl, (a way to pass that to the script: args, environment variables, config file) preferably through environment variables so we can easily configure that credential in the infra side
we need a forge user with write access to all the required repositories, and create an access token with appropriate privileges that we can pass to the script
We also discussed that the forge user could very probably be weblatebot.
I am afraid I am not really helpful here with my scripting skills. Can you prepare the changes in the script?
for now, automation is blocked by forge/forge#27
I don't know how to resolve this without ssh auth
All localization-docs repositories set with Forgejo webhook URL. I used the attached Bash script, for the record.
FYI, Forgejo webhooks to Fedora Weblate do not work [Ticket#2013652] https://care.weblate.org/#ticket/zoom/12432https://care.weblate.org/#ticket/zoom/12432
Since weblatebot is now a member of the localization-docs organization (thanks @peartown for letting us know), I removed weblatebot as collaborator from all of its repositories.
Testing docs/translations-scripts#54, I manually updated PO and POT files of all localization-docs repositories.
I didn't update docs/localization-stats and docs/translated-sources because I don't have access.
You can remove those as we migrated to a unified doc repo: https://forge.fedoraproject.org/atomic-desktops/docs
I think time has come to close this ticket. What do you think?
All the repositories are migrated.
Updated translation scripts are in place and working.
Probably some 'cleaning' tasks, regarding weblate or pagure, still remain, but...
The only thing left is the handle Pagure fedora-docs-l10n repositories. I requested deletion (as suggested by Jean-Baptiste in #l10n Matrix room): infra/tickets#13428
@siosm: translation projects removed in favor of https://translate.fedoraproject.org/projects/fedora-docs-l10n-atomic-desktops/, thanks
I have drafted a short announcement about the translations availability for the Community Blog.
Pasting here for possible suggestions/improvements/comments (as I am not sure how much accessible the draft commblog post is):
Fedora Documentation translations again available
Updates to translations of Fedora Documentation are again available.
As announced on March 3rd, the unavailability of translation updates was due to the migration of the
translation repositories and necessary tools from Pagure to the Fedora Forge.
It took longer than expected but we are pleased to report this undertaking came finally to the end.
Used links:
Fedora Documentation -> docs.fedoraproject.org
announced -> previous commblog of March 3
undertaking -> this issue
Cool, yes I can access the draft
my suggestion:
My suggest would be to start all our communications with something like "Dear translators and allies"
I can suggest text for whatever part you ask for, but your writing is fine
It is great that you are working on fully finish this task by this public communication
I am afraid things again happened much faster then I expected. The post has been already published on commblog (without me knowing). I was expecting Tuesday or so.
But definitely I would welcome any help with real statistics to paint the effort we have spent on it. I was just afraid that with my pace of work these days it could take me perhaps weeks to prepare the statistics myself and include it in the post.
If you agree we could prepare another post where we can provide a such information (and retrospective?)?