Fedora synchronous/real-time chat has moved primarily from IRC to
Matrix, so this reference should point to the right place.
Signed-off-by: Neal Gompa <ngompa@fedoraproject.org>
Some kind of bug is currently giving us composes with invalid
IDs - e.g. "[[ release.id_prefix.title() ]]-37-updates-20230429.0"
(obviously a template substitution problem). This causes us to
blow up because fedfind's parse_cid raises an exception in this
case and we don't handle it. So...handle it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Autocloud was retired years ago, we don't need any of this any
more. Simplifies things quite a bit. Yay.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Ran the page through the w3c validator and discovered that we're
supposed to declare the page's language, and the bottom margin
of the table had a typo in the unit ("pm" not "px").
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The way we had it before, you either set both or none, but turns
out now I moved it to infra I want to set htmlfile but leave
datafile as default. And that didn't work. This makes that work
and is clearer anyhow.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is a way to get rid of things we just don't build any more
from the list - flavors that were removed or renamed, arches we
stopped doing image builds for (hi, i386!)
It's a slightly crude approach, because it's actually possible
for an image we are still trying to build to fail for this long.
It's probably happened before. But even there the impact should
be fairly limited; the old image will no longer be downloadable
(nightly composes get garbage collected after about two weeks)
and whenever the image finally *does* build again, we'll add it
back in automatically. So all that's lost is the *information*
on when it last successfully built, which is possibly slightly
interesting to someone but the benefit of the cleanup is greater,
I think.
Trying to do this in an 'exactly correct' way is very hard as
compose metadata is silent on the topic of what images *should
have* been built.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
fedfind caches collections data anyway, so this is effectively
duplicating that. (Also the one-day expiry wasn't working
because fedfind's collections cache had no expiry; I've moved
the one-day expiry logic to the fedfind cache in 4.2.8).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I switched implementation designs halfway through writing it,
and as usually happens, that meant the conditional got left in
the wrong state...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The image dict used in test_image is pretty ancient, and was
actually failing due to fedfind adjusting the image dict for
https://pagure.io/pungi/issue/417 . Let's just update the test
to use a much newer image dict (from the final F29 nightly).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We cache it so we're not constantly re-discovering this every
time a test completes (that seems a little excessive), but
caching it *forever* is wrong too, as every few months we
either branch or cut a release. Currently I have to remember to
restart fedmsg-hub when this happens, or else nightlies keeps
showing the table for the release that was cut, or *doesn't*
add a table for the new Branched release.
This way, I don't have to remember that, within a day or so of
the branch happening or a release happening, nightlies will
refresh and clean up the data automatically.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I added a helper to fedfind for generating 'image identifiers',
set up such that both fedora_openqa_schedule and this tool can
use it to generate the openQA flavor name, so the code doesn't
have to be copied into both.
Also run stored image data through fedfind `correct_image` when
reading it from disk; the intent here is to make sure the
image identifiers for the stored image dicts and newly-imported
ones will match. See https://pagure.io/pungi/issue/417 for
more background on this whole mess.
openQA now treats 'softfailed' as a distinct result. For now,
we'll continue to just treat these as passes (we might get
smarter about this in future).
autocloud is now testing after compose completion, so we don't
need the hack any more. the fedmsg format has also changed, we
need to handle the new one. good news, we have a compose ID now,
so we can do a better image match and do the 'update "built"
if image not found' loop for autocloud too.
this is a hack to deal with #6 for now: autocloud runs on Koji
fedmsgs, so it runs before the compose is complete, and may
finish before the compose is complete, in which case we don't
know what to do with the result. As a temporary hack to deal
with this, just update the last day's worth of autocloud results
every time a compose completes.
do more of the stringification in __str__ rather than __init__,
so self.contents isn't so heavily modified. self.contents will
always be a list of string-able things; if passed a string or
Element as 'contents' it will just be made into a one-item list,
if passed a list of strings or list of Elements it's kept as
self.contents unchanged. Then in __str__ we do the 'stringify
everything in the list and decide on newlines' stuff.
get_jobs is a simpler job getting thing I just added to the
client; we don't need the elaboration of iterate_jobs any more.
Handle connection errors (we didn't before). Handle 'no jobs'
and 'still running jobs' cases differently now.
There was a bug in the 'update the 'last built' data if we
don't find the image' loop: it would update the data alright,
but it wouldn't re-do the match, because that's inside the
'while' loop and once the data was added we broke out of the
loop. So rejig the loop logic a bit so the match is re-done
after the data is updated.
not a great improvement here, but good practice. IMPORTANT:
the API does not expose 'softfailed' state and there is no
accounting for it in the constants:
https://github.com/os-autoinst/openQA/issues/648
if that gets fixed this code will need to change.
the consumer was broken due to one of my trademark indentation
fails: the update function only got run for Autocloud messages
because it was indented into that block...
also add a test consumer (handy for, well, testing) which
listens for dev not prod messages and writes to files in /tmp.
Set up logging in the consumers, and do a bit more logging
throughout, especially some more debug logging.
three bug fixes that showed up during testing of the failed
test link stuff.
A cut/paste error in the new get_cell() meant untested links
were green instead of blue.
The image groups are sorted by a weight calculation done by
fedfind. This took arch into account. This was a problem,
because for nightlies the image groups include images of many
arches, so the weight was somewhat unpredictable: nightlies
just picks some random image (the first in the list for the
group) to do the weight score with, and if it happened to pick
an ARM image for a group that includes ARM images, that group
would be heavily penalized. This is why the 'Server DVD' group
would sometimes show up way down the list. So in fedfind 2.4.5
I made it possible to tell fedfind to ignore the arch, and now
we do that (and the weight is calculated solely from the sub
variant).
update_openqa() didn't check for dupes, so if you call it more
than once on the same compose it would add duplicates of
already-stored job dicts. This wasn't obviously visible before
but with the links, it was pretty clear. So this is fixed from
two ends: update_openqa() now won't add a job if it's already
in the list (I hope python's 'dict in list' is up to this...),
and prune_jobs checks through the list of openqa jobs for ones
with the same job ID and removes dupes.
Oh man I love this so much! Huge thanks to Robery Mayr for help
with the HTML/CSS here. You can now hover over the link for any
image with test fails to show a tooltip with links to each of
the failed tests in openQA, or the Autocloud test output page.
Super cool. Woohoo. All in pure HTML/CSS, no stinky JS here.
this isn't quite ready for prime-time: it works, but it's ugly.
I think the right way to do it might be to add a double-height
row when there are failures, with the arch and lng cells
spanning both rows and two cells in the last-built column, one
for the image link and one for the failure links. then we can
play with the height of the failure links row. but that needs
quite a change to the logic, might have to split out the result
row code into a separate function for sanity. i'll play with it
tomorrow.
ahh, a good old completely mixed up commit, sloppy development
is awesome. This adds a major new feature - autocloud results
support - and fixes some completely critical bugs at the same
time:
* Image.__cmp__ was a terrible idea that wasn't even being used
for its intended purpose any more, but sure messed with the
'image in self.data' check in _prune_data
* get_latests(flat=True) was returning a list of lists of images
not a list of images, so _prune_data would remove everything
after removing __cmp__
* the sorting for latest images (you know, the core function of
the entire goddamn tool) was exactly incorrect, so we got the
OLDEST image
* pruning data after the 'get latest builds' step but before
the 'update test results' steps during seed_data meant we
threw out a bunch of images we actually might want as last
known good and had to go get them again
* the new clever-clever partial stuff in the fedmsg consumers
was using 'nightlies' before assigning it
Basically everything except the goddamn HTML writer was utterly
screwed. Good job me. This version works though! I promise!
make Element a bit smarter. If 'contents' is a list, we str()
each element of the list, join the strings and check how long
the result is; if it's <81 chars we take the result, otherwise
we join the strings with newlines and take that.
If 'contents' is anything but a list, we just stringify it and
take the result.
This lets us make ReleaseTable rather simpler and just throw
Elements, lists of elements, and strings in as 'contents' as we
go.
avoids the need for an annoying attributes dict. Deal with
'class' as a special case. There are some others, apparently,
but hopefully I'm not gonna run into them.
We *could* just use lxml or something, but I'm trying to avoid
the dep for now...
