Consider using a caching proxy or local registry for podman tests

adamwill commented

2026-01-21 21:04:22 +00:00

Owner

In the last day or two, the podman test is frequently failing (on both arches, on both instances). There are various failures, but they all seem to happen when a test is trying to pull something from the real quay.io for some reason. One common one is a timeout doing a pull, like this:

not ok 96 [070] build with copy-from referencing the base image
# (from function `run_podman' in file /usr/share/podman/test/system/helpers.bash, line 598,
#  in test file /usr/share/podman/test/system/070-build.bats, line 890)
#   `run_podman build --jobs 1 -t ${target} -f ${containerfile2} ${tmpdir}' failed
#
# [13:44:38.416312159] # podman  image exists quay.io/libpod/alpine:latest
# [13:44:38.445043131] [ rc=1 (expected) ]
#
# [13:44:38.451284745] # podman  build --jobs 1 -t derived-t96-aecwt3ct -f /tmp/podman_bats.7wWEDM/build-test/Containerfile2 /tmp/podman_bats.7wWEDM/build-test
# [13:46:38.475138625] [2/2] STEP 1/2: FROM quay.io/libpod/alpine AS final
# Trying to pull quay.io/libpod/alpine:latest...
# Getting image source signatures
# Copying blob sha256:9d16cba9fb961d1aafec9542f2bf7cb64acfc55245f9e4eb5abecd4cdc38d749
# timeout: sending signal TERM to command ‘podman’
# [13:46:38.484258602] [ rc=124 (** EXPECTED 0 **) ]
# *** TIMED OUT ***
#
# [13:46:38.499217824] # podman  rm -t 0 -a -f
#
# [13:46:38.546122560] # podman  rmi -f b-t96-aecwt3ct
#
# [13:46:38.579566655] # podman  image prune -f
# # [teardown]

another I've seen is an auto-update test (from 255-auto-update.bats) getting a 504 response from quay.io and retrying; the retry worked, but the test was considered 'failed' because there was a warning logged for the 504 and the test fails on unexpected warnings.

This started around noon UTC on 2026-01-20. Before that failures were very rare, now 80-90% of attempts are failing.

I've rebooted the entire prod instance (server and workers) but that doesn't seem to be helping.

For now I've set the test to auto-retry 5 times, but that's only a mitigation and might hide genuine bugs. It'd be good to figure out what's going on here but I'm really not sure.

@lsm5 any ideas? Do you happen to know who we could ask about this on the quay.io end?

In the last day or two, the podman test is frequently failing (on both arches, on both instances). There are various failures, but they all seem to happen when a test is trying to pull something from the real quay.io for some reason. One common one is a timeout doing a pull, like this: ``` not ok 96 [070] build with copy-from referencing the base image # (from function `run_podman' in file /usr/share/podman/test/system/helpers.bash, line 598, # in test file /usr/share/podman/test/system/070-build.bats, line 890) # `run_podman build --jobs 1 -t ${target} -f ${containerfile2} ${tmpdir}' failed # # [13:44:38.416312159] # podman image exists quay.io/libpod/alpine:latest # [13:44:38.445043131] [ rc=1 (expected) ] # # [13:44:38.451284745] # podman build --jobs 1 -t derived-t96-aecwt3ct -f /tmp/podman_bats.7wWEDM/build-test/Containerfile2 /tmp/podman_bats.7wWEDM/build-test # [13:46:38.475138625] [2/2] STEP 1/2: FROM quay.io/libpod/alpine AS final # Trying to pull quay.io/libpod/alpine:latest... # Getting image source signatures # Copying blob sha256:9d16cba9fb961d1aafec9542f2bf7cb64acfc55245f9e4eb5abecd4cdc38d749 # timeout: sending signal TERM to command ‘podman’ # [13:46:38.484258602] [ rc=124 (** EXPECTED 0 **) ] # *** TIMED OUT *** # # [13:46:38.499217824] # podman rm -t 0 -a -f # # [13:46:38.546122560] # podman rmi -f b-t96-aecwt3ct # # [13:46:38.579566655] # podman image prune -f # # [teardown] ``` another I've seen is an auto-update test (from 255-auto-update.bats) getting a 504 response from quay.io and retrying; the retry worked, but the test was considered 'failed' because there was a warning logged for the 504 and the test fails on unexpected warnings. This started around noon UTC on 2026-01-20. Before that failures were very rare, now 80-90% of attempts are failing. I've rebooted the entire prod instance (server and workers) but that doesn't seem to be helping. For now I've set the test to auto-retry 5 times, but that's only a mitigation and might hide genuine bugs. It'd be good to figure out what's going on here but I'm really not sure. @lsm5 any ideas? Do you happen to know who we could ask about this on the quay.io end?

adamwill commented

2026-01-22 18:49:11 +00:00

Author

Owner

Paul Holzinger suggested using a caching proxy to mitigate this, that's probably a good idea anyway to reduce unnecessary repeated traffic to quay.io for every run of this test. We already implemented one for Flathub, so we could more or less copy that for this.

adamwill commented

2026-01-22 18:49:17 +00:00

Author

Owner

Metadata Update from @adamwill:

Custom field story_points adjusted to 5

**Metadata Update from @adamwill**: - Custom field story_points adjusted to 5

lsm5 commented

2026-01-23 09:43:36 +00:00

Sorry I was out yesterday. Paul's suggestion would be the way to go.

adamwill self-assigned this

2026-01-23 20:47:38 +00:00

adamwill added the

points

05

label

2026-01-23 20:49:02 +00:00

luap99 commented

2026-01-23 22:05:35 +00:00

Contributor

@adamwill FYI we use the registries.conf mirror feature for this:
github.com/containers/podman@1af4caf888/test/registries-cached.conf (L20-L23)

Basically we point to a local registry that has all our test images already embeeded, since we use custom GCE/AWS images we pull them when we create the VM disk images and then there is no network pull needed during the actul CI runtime on PRs.

Not sure how well that would map to openqa, I guess as long as you have the registry somehwere more local that should help.

For system tests we only need a few images, I don't know if there are others as hoc in some tests but the main ones are:

quay.io/libpod/testimage:20241011
quay.io/libpod/testimage:00000004
quay.io/libpod/systemd-image:20240124

github.com/containers/podman@1af4caf888/test/system/helpers.bash (L14-L33)

A full list of what we cache is here github.com/containers/automation_images@4240f9dbb2/cache_images/local-cache-registry (L56-L100)

Though that is much more than needed for system tests.

Hope this helps!

@adamwill FYI we use the registries.conf mirror feature for this: https://github.com/containers/podman/blob/1af4caf888924e837e9c6038c30d67bfcc7c0196/test/registries-cached.conf#L20-L23 Basically we point to a local registry that has all our test images already embeeded, since we use custom GCE/AWS images we pull them when we create the VM disk images and then there is no network pull needed during the actul CI runtime on PRs. Not sure how well that would map to openqa, I guess as long as you have the registry somehwere more local that should help. For system tests we only need a few images, I don't know if there are others as hoc in some tests but the main ones are: - quay.io/libpod/testimage:20241011 - quay.io/libpod/testimage:00000004 - quay.io/libpod/systemd-image:20240124 https://github.com/containers/podman/blob/1af4caf888924e837e9c6038c30d67bfcc7c0196/test/system/helpers.bash#L14-L33 A full list of what we cache is here https://github.com/containers/automation_images/blob/4240f9dbb2851c73b424b6190831159ea716198f/cache_images/local-cache-registry#L56-L100 Though that is much more than needed for system tests. Hope this helps!

adamwill commented

2026-01-23 22:29:03 +00:00

Author

Owner

The issue is we're running podman's test suite, that's maintained by podman, and that test suite refers to quay.io directly. I don't really want to hack it up with a bunch of sed commands before running it.

So, a caching proxy for quay.io seems like the best option, on the face of it at least.

edit: oh, I see how that works. hmm. It might be doable that way, yeah. I'll have to experiment when I can.

In the meantime the errors went away, so it's a bit less urgent now...

The issue is we're running podman's test suite, that's maintained by podman, and that test suite refers to quay.io directly. I don't really want to hack it up with a bunch of sed commands before running it. So, a caching proxy for quay.io seems like the best option, on the face of it at least. edit: oh, I see how that works. hmm. It *might* be doable that way, yeah. I'll have to experiment when I can. In the meantime the errors went away, so it's a bit less urgent now...

adamwill added the

labels

2026-02-10 02:24:31 +00:00

adamwill changed title from ~~podman tests frequently fail due to quay.io connectivity issues(?)~~ to Consider using a caching proxy or local registry for podman tests

2026-04-07 23:21:04 +00:00

adamwill added

and removed

labels

2026-04-07 23:21:16 +00:00

adamwill commented

2026-05-11 22:34:50 +00:00

Author