Fedora Docs internationalization

This is a prototype of a multi-language Fedora Docs build pipeline. Work is tracked in https://fedoraproject.org/wiki/User:Jibecfed/fedoradoc-antora-localization.

The internationalization work with:

• Fedora Docs sources — sources of the docs in an adoc format. For full list of these, see site.yml in docs-fp-o.
• Translation scripts — https://forge.fedoraproject.org/docs/translations-scripts — scripts to convert from adoc to pot and from pot to adoc.
• Translation repos — https://forge.fedoraproject.org/localization-docs — repositories holding POT and PO files, acting as a bridge between the Fedora Docs site and translators. This way, translators can choose any workflow with a translation engine of their choice. The only requirement is that they need to work with POT and PO files.
• Translated sources — https://forge.fedoraproject.org/docs/translated-sources — all translated adoc files used to build the translated site.

How to run

Make sure Podman is installed on your system. You can install Podman using the following command:

$ sudo dnf install podman

You can build the container image with Podman:

$ cd translations-scripts
$ podman build . -t builder:latest

Two environment variables are required in order to allow the scripts to push to the above remote repos (the translation repos and the translated sources repo above). Through these variables your appropriate git credentials (forge username and access token) are passed to the scripts. For more info see the Forge User Documentation.

To set and export the environment variables enter (replace forge_username with your valid username and access_token with your valid access token):

$ export UPDATE_DOCS_TRANS_NAME=forge_username UPDATE_DOCS_TRANS_SEC=access_token

To run the build.py script inside a container:

$ podman run -e UPDATE_DOCS_TRANS_NAME -e UPDATE_DOCS_TRANS_SEC builder:latest [args]

Alternatively, you can mount your own local repository inside the container, for development purpose:

$ podman run -e UPDATE_DOCS_TRANS_NAME -e UPDATE_DOCS_TRANS_SEC -v ./:/workspace:Z builder:latest [args]

Submitting sources for translation

The build.py script will find list of repositories to be translated in the file site.yml from the docs-fp-o repo. It clones all of the specified source repositories, all of the matching translation repos, and converts the English sources into POT files. The output of this script are local copies of the translation repos, updated with the newly generated POT files.

It takes the following arguments :

  --clone_sources {true,false}
                   Should we clone/update sources or use local content?
  --commit_l10n {true,false}
                   Should we commit changes on l10n repos?
  --commit_tsources {true,false}
                   Should we commit translated sources?
  --nopush {true,false}
                   Should we not push to remote repositories?
  --component COMPONENT
                   Antora component to convert, all if unset
  --module MODULE
                   Antora module to convert (ROOT is the default value in Antora), all if unset
  --version VERSION
                   Antora version to convert, all if unset
  --clone-po4a {true,false}
                   Should we clone and use a local checkout of po4a?
  --clone-po4a-version CLONE_PO4A_VERSION
                   What po4a version from the local checkout should we use?

The build.py script will create three folders:

• $(pwd)/sources — Fedora Docs sources
• $(pwd)/l10n — Translation repos
• $(pwd)/translated-sources — Translated sources

Publish translated sources

$ cd translated-sources
$ git add .
$ git commit -m "automatic update of translated content"
$ git push

Rebuilding the translated website

The official Fedora Docs docs-fp-o repo holds Antora playbooks for each language of the docs, and build scripts for each language.

Note: The playbooks are constructed manually and might not be up-to-date. Please make sure to update them based on the main site.yml and the outputs in the translated sources repo.

Clone the repository, run a build of the translated sites (each language is its own site), and run a local preview:

$ git clone https://forge.fedoraproject.org/docs/docs-fp-o.git
$ cd docs-fp-o/
$ ./build-translated.sh
$ ./preview.sh

Now you should be able to see, for example, the French site, on http://localhost:8080/fr/.

Maintenance

When you run the build.py process, an "errors.txt" file will be generated. It contains missing localization-docs repositories.

Adding a new repository

Example data:

• component name: doc-example
• It corresponds to the public URL: https://docs.fedoraproject.org/en-US/doc-example
• The content itself comes from: https://forge.fedoraproject.org/doc-example

Process:

• Log in https://forge.fedoraproject.org, browse localization-docs namespace, and press New repository button:

  • Keep owner: localization-docs
  • Set for Repository name: doc-example
  • Set for Description: translation of doc-example
  • Select "Initialize repository"
  • Press "Create repository" button

• In the top-right menu icon > Settings > Webhooks:

  • Press "Add webhook" button, select "Forgejo"
  • URL: https://translate.fedoraproject.org/hooks/forgejo/

• Log in https://translate.fedoraproject.org, follow: Add > Add new translation project

  • Set for project name: fedora-docs-l10n/doc-example
  • Set for slug: fedora-docs-l10n-doc-example
  • Set for Project website: https://forge.fedoraproject.org/localization-docs/doc-example
  • Set for Translation instructions: Mailing list for translators: <trans@lists.fedoraproject.org> ([listinfo](https://lists.fedoraproject.org/admin/lists/trans.lists.fedoraproject.org/))

• Click the add new translation component

  • Set the component name as "nav" (use antora if no nav)
  • Version control system: git
  • URL is: https://forge.fedoraproject.org/localization-docs/doc-example.git
  • Validate and select the nav pattern
  • Set Repository push URL: ssh://git@forge.fedoraproject.org/localization-docs/doc-example.git
  • Disable "Edit base file"
  • set License to match the documentation's license (e.g. CC-SA-3.0)
  • Save

• Open the component and access Addons

  • Automatic translation
  • Contributors in comment
  • Update PO files to match POT (msgmerge)
  • Squash Git commits: Commit squashing Per author
  • Component discovery: This addon automatically adds or removes components to the project based on file changes in the version control system.
    • Regular expression to match translation files against: po/(?P<language>[^/]*)/(?P<component>.*)\.po
    • File format: gettext PO file
    • Customize the component name: {{ component }}
    • File for new translations: pot/{{ component }}.pot
    • Language filter: ^[^.]+$
    • Clone addons from the main component to the newly created ones: True
    • Remove components for inexistent files: True