koji-flatpak

koji-flatpak is a Koji plugin to build Flatpaks container images from RPMs, using flatpak-module-tools.

flatpak-module-tools is used in two ways: it is imported into the builder plugin and used for parsing and validating container.yaml files, but the heavy lifting is done by installing flatpak-module-tools into the buildroot and running flatpak-module assemble there.

Since the usage within the plugin is minimal, there is no requirement that the two copies of flatpak-module-tools match; updating the version used by the builder should only be necessary if the plugin code is changed to parse additional files from container.yaml.

Setup: builder

Install flatpak_builder_plugin.py as /usr/lib/koji-builder-plugins/flatpak.py. If you want to push the results to a registry, create a file /etc/kojid/plugins/flatpak.conf with contents:

[push]
registry=docker://registry.example.com/mynamespace
authfile=/etc/kojid/containers/auth.json

And call skopeo login registry.example.com --authfile /etc/kojid/containers/auth.json to store credentials that can push images to the namespace.

Setup: hub

Install flatpak_hub_plugin.py as /usr/lib/koji-hub-plugins/flatpak.py.

Setup: cli

Install flatpak_cli_plugin.py as /usr/lib/python<version>/site-packages/koji_cli_plugins/flatpak.py.

Set up: koji data

# Create source and dest tags and a target for building Flatpaks
koji add-tag f39-flatpak-container-build --parent f39-build --arches x86_64
koji add-tag f39-flatpak-updates-candidate
koji add-target f39-flatpak-candidate f38-flatpak-container-build f39-flatpak-updates-candidate

# Add a group called flatpak-tools with the tools necessary to build 
koji add-group f39-flatpak-container-build flatpak-build
koji add-group-pkg f39-flatpak-container-build flatpak-build flatpak-module-tools dnf tar

# Add extra data to the build tag pointing to a) the tag where to find runtimes
# b) the tag to take runtime packages from c) the tag to take applications from
koji edit_tag f39-flatpak-container-build \
    -x flatpak.runtime_tag=f39-flatpak-updates-candidate \
    -x flatpak.runtime_package_tag=f39-flatpak-runtime-build \
    -x flatpak.app_package_tag=f39-flatpak-app-build

# Can alternatively use a dist-repo for the app (or runtime) package source
koji edit_tag f39-flatpak-container-build \
    -x flatpak.app_package_tag=f39-flatpak-app-build \
    -x flatpak.app_package_dist_repo=true

# The dist-repo could be created with the tag2distrepo plugin
koji edit-tag f39-flatpak-app \
   -x tag2distrepo.enabled=true \
   -x tag2distrepo.inherit=false

## Usage

Usage: koji flatpak-build [options] <scm_url> (Specify the --help global option for a list of other help options)"

Options: -h, --help show this help message and exit --arch-override=ARCH_OVERRIDE Override build arches --nowait Don't wait on build --scratch Perform a scratch build --skip-tag Do not attempt to tag package --wait Wait on the build, even if running in the background

## Development

Running tests involves having the plugin code and dependencies installed into
a virtual environment.

``` sh
dnf --setopt=install_weak_deps=False tox python3.8
tox devenv .venv
. .venv/bin/activate

You can then run the tests like:

pytest

Or more formally, run all tests via tox.

tox

Note that installing the plugins as koji_flatpak.plugins.* is only useful for the purposes of running tests, and has no relationship to how they need to be installed for actual usage. (See above.)

I would recommend using vscode to develop this project; the Python support via pylance catches a lot of errors as you type. (Using mypy should be possible too, but will require some additional annotations.) You'll need to configure vscode to use the python interpreter .venv/bin/python.

Python feature usage specified to Python 3.8 or newer to match flatpak-module-tools, which in turn matches OSBS. Koji doesn't have a clear specification of the minimum required version of Python 3 (and still at least nominally supports Python), but for Fedora, Koji is deployed on Python 3.11 (as of June 2023), so if there was a strong version to require a newer Python, that likely wouldn't be a problem.

Some of the features to avoid include:

• Union types with | - Use Optional[str] rather than str | None.
• Using builtin collections rather than the types from typing - use Dict[str] not dict[str].

Development Koji instance

The Dockerfiles and config files in images/ define a container-based test environment for Koji that works with rootless podman.

There are three containers; since private networks don't work with rootless podman, they are part of the same pod to share the loopback interface.

• postgresql - postgresql server
• hubweb - apache server that serves the xmlrpc "hub" interface to koji, the web frontend. There's also the ability to bind-mount in local git repositories to build from.
• builder - a single builder node

You can configure it by creating a .manage.conf.sh file at the toplevel of koji-flatpak.

# shellcheck shell=bash
# shellcheck disable=SC2034

# Registry to push successful builds (use 'manage login' to log in)
push_registry=docker://quay.io/myuser

# Optional: local directories to export over git to use to build from:
# (manage koji flatpakBuild f39-flatpak-app git+http://koji-flatpak-hubweb:8080/git/flatpaks/eog)
git_exports=(
    "$HOME/Source/flatpak-module-tools":flatpak-module-tools
    "$HOME/fedora-git/flatpaks/eog":flatpaks/eog
)

Running it looks like:

# Generate TLS certificates for the components
tools/generate-cert.sh
# *Modify your host* to trust the certificates and add a /etc/hosts
# entry for koji-flatpak.local.fishsoup.net.
tools/trust.local.sh
# Set up a shell alias and completion for 'manage'
. tools/bash-setup.sh
# Build the containers
manage build
# start them up, with the builder in the foreground
manage start postgresql hubweb

Now you'll need to create all the tags and targets, and external repositories.

podman exec -u kojiadmin koji-flatpak-hubweb setup-koji-data.sh

And then you can start the builder and try some builds

# Watching the builder can be useful, but could just 'manage start' to start
# any missing containers in the background
manage start --no-detach builder

# Only if you've configured a registry to push to
manage login
# enter credentials - for quay.io, use a robot account. For a robot account
# owned by your user, you'll have to individually create each repository and
# give the robot account write access there.
# See http://docs.quay.io/issues/no-create-permission.html
# In the below, the robot will need access to repositories named f38/flatpak-runtime
# and eog.

# In a different terminal
. tools/bash-setup.sh
manage koji regen-repo --wait f38-flatpak-runtime-build
manage koji build f38-flatpak-runtime 'git+https://pagure.io/flatpak-module-tools#without-modules'
manage koji build f38-flatpak-runtime 'git+https://src.fedoraproject.org/rpms/flatpak-runtime-config?#f38'
manage koji build f38-flatpak-runtime 'git+https://src.fedoraproject.org/forks/otaylor/rpms/flatpak-rpm-macros.git#without-modules'

Go to https://koji-flatpak.local.fishsoup.net:8443/koji/ in your browser to see them in action. And build a runtime:

manage koji regen-repo --wait f38-flatpak-container-build
manage koji regen-repo --wait f38-flatpak-runtime-packages
manage koji add-pkg f38-flatpak-updates-candidate --owner=kojiadmin flatpak-runtime
manage koji flatpak-build f38-flatpak-candidate 'git+https://src.fedoraproject.org/forks/otaylor/flatpaks/flatpak-runtime#without-modules'

And when that finishes, an application Flatpak

manage koji regen-repo --wait f38-flatpak-app-build
manage koji add-pkg f38-flatpak-app --owner=kojiadmin gnome-desktop3 libpeas libportal exempi eog
manage koji build f38-flatpak-app 'git+https://src.fedoraproject.org/rpms/gnome-desktop3#f38'
manage koji build f38-flatpak-app 'git+https://src.fedoraproject.org/rpms/libpeas#f38'
manage koji build f38-flatpak-app 'git+https://src.fedoraproject.org/rpms/libportal#f38'
manage koji build f38-flatpak-app 'git+https://src.fedoraproject.org/rpms/exempi#f38'
# Wait for those to finish
manage koji regen-repo --wait f38-flatpak-app-build
manage koji build f38-flatpak-app 'git+https://src.fedoraproject.org/rpms/eog#f38'
# Wait for that to finish
manage koji regen-repo --wait f38-flatpak-app-packages
manage koji add-pkg f38-flatpak-updates-candidate --owner=kojiadmin eog
manage koji flatpak-build f38-flatpak-candidate 'git+https://src.fedoraproject.org/forks/otaylor/flatpaks/eog#without-modules'

Phew. Now you are ready to hack on koji-flatpak! (Try to lean on the test suite to reduce cycle times.)