This should enable dist-git PR testing in prod, and make *only*
prod report status back to dist-git (staging no longer should).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
See infra/tickets#13422
for details on this. It seems like rabbitmq staging has been
switch to the 'new' CA cert, and the consumers don't actually
read/trust both certs in the combined CA cert file, only the
first (old) one, so they don't trust the new one. This should
deploy and use a new file with only the new CA cert in it. We
copy it out of the private repo for convenience but it's not
actually private, hence the 0644 perms.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I'm having auth issues on openQA staging, I believe it *may* be
to do with the changes Aurelien made in
6fe8b98de21747dc16d59dcf5096da719a541296 . Let's see if using the
old key and cert helps.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This *should* configure a new queue and consumer for openQA to
report dist-git PR test results back. It's all a bit speculative
ATM, may need some tweaking. Mostly applied only on staging for
now.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
AI caught that I've been cargo-culting a dumb message queue
config around from somewhere or other forever. Non-durable,
exclusive, and auto-delete is just about the worst config for a
queue you want to survive transitory issues like the server or
client going down or whatever. This might actually explain the
occasional issues we have with test results not showing up in
resultsdb, even.
This brings the config in line with most other message queues in
infra.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Recent openQA has a mechanism where it refuses to schedule jobs
if the free space for various dirs (results, asset, archive) is
less than X%, where the default X is 5. We use a very large share
for these things and expand it only on need, so it's usually quite
close to 100% full. Let's try only needing 0.5% to be free.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We only have two job groups, so the front page is a bit sad and
empty. Let's show 10 builds per group, not 3.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
it never worked anyway (ffmpeg always showed up, *somehow*) and
on the new workers it doesn't seem to be an issue.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We weren't actually applying the quota we had defined (only) for
lab, so both lab and prod had the upstream default (which now
seems to be 200GB, not 100GB). Let's fix it so we do apply the
value, and set it to 250GB for both prod and stg, because we're
now aiming to have full parity in the update test sets between
aarch64 and x86_64, and we have the space on the rdu3 hosts.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's this annoying pattern where the NFS mount fails on boot
and then the worker services all start up and take jobs, but they
instafail because the share isn't there.
Ideally we could handle this very easily with Restart= directives
but systemd has...*opinions* about this:
https://github.com/systemd/systemd/issues/4468https://github.com/systemd/systemd/issues/1312
so we have to do some fairly awkward hacks to just express:
* Retry the NFS mount if it fails
* Don't start the workers unless the NFS mount is up
* Retry the workers after a while if they were blocked
It's ugly, but in testing this same config on one worker it seems
to work...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This reverts commit 4dc01bc892 and
a follow-up commit. I'm having trouble getting things to work
and want to see if it works if we go back to having the openQA
bridge be br0, and rename the bridge used for the system's bonded
network connection to something else instead.
I thought/assumed/knew/something? that resultsdb_conventions_fedora
required resultsdb_conventions, but right now it seems it doesn't.
It *should*, but I can't fix it right now as the buildsystem is
down, so let's just install it here...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
On the new rdu3 worker hosts, br0 already exists and is the main
system 'interface' (it's a bridge on two bonded physical interfaces
connected to different switches, to make networking upgrades
easier). So we can't call our openvswitch bridge 'br0' any more.
Let's try calling it 'openqabr0' and see if anything explodes.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
OpenID support in FAS is going away. openQA has OAuth2 support.
I've tested this config to work with manual edits on lab, now
ansiblizing it (for lab only to start with).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This will unify all the handlers to use first uppercase letter for
ansible-lint to stop complaining.
I went through all `notify:` occurrences and fixed them by running
```
set TEXT "text_to_replace"; set REPLACEMENT "replacement_text"; git grep
-rlz "$TEXT" . | xargs -0 sed -i "s/$TEXT/$REPLACEMENT/g"
```
Then I went through all the changes and removed the ones that wasn't
expected to be changed.
Fixes https://pagure.io/fedora-infrastructure/issue/12391
Signed-off-by: Michal Konecny <mkonecny@redhat.com>
fix 1900 failures of the following case issue:
`name[casing]: All names should start with an uppercase letter.`
Signed-off-by: Ryan Lerch <rlerch@redhat.com>
This is an awful hack to deal with
https://github.com/os-autoinst/os-autoinst/issues/2549 while we
try and fix it properly. This finds stuck qemu processes by
parsing the journal messages of the workers, and kills them.
workers stuck in the broken state should then recover on the
next checkin with the server. I tested this manually on all the
worker hosts and it...seemed to work, mostly. I'll keep an eye on
things after deploying it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
In various roles I maintain I use `python3 -m pip install` to
directly install a Python project (usually a fedora-messaging
consumer), to avoid the pointless bureaucracy of packaging them.
The roles install all the deps of these projects as packages
first, so pip doesn't have to install any deps, it only installs
the project itself. Well...that's the idea. It's possible for
this to go wrong (say I forget to update the roles when adding
a dep to the project), and in that case I think we'd rather have
things blow up (so I know something's wrong) than have pip
silently install some random upstream wheel system-wide to make
it work. The intent is that all the deps still come from proper
Fedora packages, only these projects themselves get installed
directly.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is a new feature in openQA that prevents worker hosts
picking up new jobs if their load average is above a certain
threshold. It defaults to 40. Our big worker hosts tend to run
above this, so let's bump it on those.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
network-scripts-openvswitch was removed in f40 and network-scripts
is going away in f41; we really need to get off using them.
This attempts to implement the same setup using NetworkManager,
based on a few different NM/ovs references, and the source of
openQA upstream's os-autoinst-setup-multi-machine . It might
need a bit of tweaking, so for now, we make it a separate task
and use it only on p09-worker01 for testing. This doesn't handle
tearing down the old network-scripts-based config as that's
pretty complex and will only need to happen once; I'll do it
manually before trying this out.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We don't want it there - see earlier commits - but I didn't
notice it's actually explicitly listed here for all arches,
which breaks stuff on aarch64 now we told dnf to exclude it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
