Track 'compose critical packages' similar to critical-path #12943
Labels
No labels
after freeze
automation
backlog
blocked
change-ack
change-nak
change-noreleng
changes
Closed As
Can't Fix
Closed As
Duplicate
Closed As
Fixed
Closed As
Fixed with Explanation
Closed As
Get back later
Closed As
Grooming
Closed As
Insufficient data
Closed As
Invalid
Closed As
It's all good
Closed As
taiga
Closed As
upstream
day-to-day
dev
docs
easyfix
epel
f26
f27
f28
f29
f30
f31
f32
f33
f34
f35
f36
f37
f38
f39
f40
f41
f42
f43
f44
f45
fedora
groomed
high-gain
high-trouble
in-progress
in-review
investigation
legal
low-gain
low-trouble
mass rebuild
medium-gain
medium-trouble
meeting
mini-initiative
new_artifact
ops
pdc_retirement
rawhide
RCA
review
script
sidetarget
sprint-0
sprint-1
sprint-2
sprint-3
sprint-4
sprint-5
unfrozen
waiting on external
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
releng/tickets#12943
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
We have the critical path concept and the critpath.py script which records the critical path in a format Bodhi uses. However, the critical path concept is functional and was intended for identifying packages whose updates should get more manual testing. This leaves a fairly large gap: we have no mechanism for identifying what I'd call "release critical packages" or "compose critical packages". I think of this as, approximately, "all the packages in all the release-blocking images and all the packages needed to build the release-blocking images". Over the years we've somewhat stretched the "critical path" concept a bit to cover such packages - for instance, we have the
critical-path-composeandcritical-path-buildgroups, which list (some of) the tools needed to build composes and packages respectively - but I kinda don't want to keep stretching it. There are cases where a package is on the Workstation live, for instance, so I think we ought to run appropriate tests to make sure its dependencies don't break, but it's clearly not really part of the "critical path" as we defined that years ago. I feel like, for clarity, we should have a new concept.I think we should have a mechanism for identifying and recording these packages, similar to critpath.py - it could even be critpath.py, I go back and forth on that. It should definitely produce a similar format for Bodhi to use as an input.
I have two practical reasons for doing this:
I've been thinking about and researching this for a while, but I wanted to write it down in a ticket now so the idea's out there and we can prioritize it appropriately.
When do you need this? (YYYY/MM/DD)
No specific date, but I feel like it's kinda important and want to prioritize work on it.
When is this no longer needed or useful? (YYYY/MM/DD)
N/A
If we cannot complete your request, what is the impact?
We will continue to miss things like "oh, this relatively unimportant package on Workstation broke its dependencies and now the compose fails".
Note, the question of how to identify these is an interesting one. You can just resolve out the environment groups in comps, but that does leave some stuff out, I think. Kiwi does have a feature to do a sort of 'dry run' build of a given image and report out all the packages it would install; I was playing around with that a few weeks ago, I ran into a couple of issues but don't entirely remember what they were. Another approach is to parse the logs from the media build tool (Kiwi or whatever) right when a compose finishes.
For the compose tools themselves it's a bit harder, I think we just have to write them down. In theory you could try to sort of scrape them out of the various scripts that actually run builds, but that would probably be so fragile it wouldn't really save any work over writing them down, or be more reliable/accurate.
This a great idea to do; for Kiwi I've also previously given you a script that lists the packages for a profile.
For image-builder you can get the list of comps groups and packages that are to be installed in the buildroot and other pipelines with
image-builder describe $imagetype.We could probably automate running both based on the profiles/image types being built in the pungi configs for a given branch and put them into one comps group?
However, for bootable containers and ostree commits; these are built differently and I am not sure how easily we can get their inputs. For commits we can probably do something based on the yaml files they use as inputs.
If someone wants some help on gathering this set of packages feel free to poke me somewhere (Matrix, e-mail) :)
For both kiwi and image-builder you can also use the rpmlist or sbom outputs to get the solved set of packages though that might be less useful.
Metadata Update from @phsmoura:
Yes, I mentioned this above. I did find at least one problem with it: dnf variable substitutions aren't applied on this Kiwi codepath, so it doesn't actually work with our unmodified templates. You have to edit the relevant repositories file and do the
$releaseverand$basearchsubstitutions manually to make it work. I can't remember what the other issue was.There's no point using comps groups on paths where we can get the package list from a tool, it's an unnecessary intermediate step. The thing that writes out the 'compose critical package' list can just do it directly using whatever tool can produce the list.
and yeah, it might actually be interesting to have pungi do this at compose time if it fits in, I'll have to look at that.
OK, I finally got back to looking at this today, did some preliminary research and prep. As things stand we have release blocking images built three ways:
I found viable ways to get the info for Kiwi and Lorax, didn't look at IB yet. Next week I'm going to look into two different ways of implementing this - in
critpath.pyor in pungi itself - and see which seems to feel best.Here's my rough notes, dumping here for safety...
kiwi Cloud/aarch64/images/Fedora-Cloud-Base-Generic-RELEASE_MILESTONE.aarch64.qcow2
kiwi Cloud/x86_64/images/Fedora-Cloud-Base-Generic-RELEASE_MILESTONE.x86_64.qcow2
kiwi Cloud/x86_64/images/Fedora-Cloud-Base-AmazonEC2-RELEASE_MILESTONE.x86_64.raw.xz
kiwi Container/x86_64/images/Fedora-Container-Toolbox-RELEASE_MILESTONE.x86_64.oci.tar.xz
kiwi Container/aarch64/images/Fedora-Container-Toolbox-RELEASE_MILESTONE.aarch64.oci.tar.xz
lorax Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-RELEASE_MILESTONE.iso
ib/ot IoT/aarch64/images/Fedora-IoT-RELEASE_MILESTONE.aarch64.raw.xz
ib/ot IoT/aarch64/iso/Fedora-IoT-IoT-ostree-aarch64-RELEASE_MILESTONE.iso
ib/ot IoT/x86_64/images/Fedora-IoT-RELEASE_MILESTONE.x86_64.raw.xz
ib/ot IoT/x86_64/iso/Fedora-IoT-IoT-ostree-x86_64-RELEASE_MILESTONE.iso
createiso Server/aarch64/iso/Fedora-Server-dvd-aarch64-RELEASE_MILESTONE.iso
lorax Server/aarch64/iso/Fedora-Server-netinst-aarch64-RELEASE_MILESTONE.iso
createiso Server/x86_64/iso/Fedora-Server-dvd-x86_64-RELEASE_MILESTONE.iso
lorax Server/x86_64/iso/Fedora-Server-netinst-x86_64-RELEASE_MILESTONE.iso
ib Spins/aarch64/images/Fedora-Minimal-RELEASE_MILESTONE_.aarch64.raw.xz
kiwi Workstation/aarch64/images/Fedora-Workstation-Disk-RELEASE_MILESTONE-aarch64.raw.xz
kiwi Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-RELEASE_MILESTONE.iso
kiwi KDE/aarch64/images/Fedora-KDE-Desktop-Disk-RELEASE_MILESTONE.aarch64.raw.xz
kiwi KDE/x86_64/iso/Fedora-KDE-Desktop-Live-RELEASE_MILESTONE.x86_64.iso
Kiwi
So: have @kiwi-build as a 'release critical group' in critpath.py. Use
kiwi --resolve-package-listto get the package list from the relevant template. OR: logged to .packages in the Koji task, e.g. https://kojipkgs.fedoraproject.org//work/tasks/1027/139121027/Fedora-Workstation-Disk-Rawhide-20251121.n.0.aarch64.packages . Read out of that (if we can easily find the Koji task).Lorax
So: read both lists out of latest compose, parse.
@adamwill you can grab the packages out of the manifest attached to the build, or the SBOM, or if you prefer I can add a handier format for you. Let me know what you prefer.
Alternatively you can look at the pungi config to see which image type is being built and do
image-builder describe --distro fedora-XX $imagetypeto get YAML output of the non-depsolved list.another note: image-builder uses the
@image-builder-buildgroup, so that's also critical.welp, I finally forced myself to get started on this damn thing today, so far I did this:
that's probably enough to make me finish it next week, heh.
https://pagure.io/pungi/pull-request/1898 should make parsing the pungi logs for Kiwi tasks a hell of a lot easier.
Also note that Koji actually knows what packages are included in an image artifact. They are logged in the database, and are viewable in Koji itself. For example: https://koji.fedoraproject.org/koji/rpmlist?imageID=1633260&type=image
hmm, that might potentially be useful I guess. I can't find an XML-RPC API method for it though...tried various search terms. Anyhow, reading the .packages file is easy once we know the task ID, so it's not really an issue.
@adamwill wrote in #12943 (comment):
Note that (for Brew in this case, but same difference)
image-builderwill also be implementing the rpmlist standard output soon-ish; so it would generalize better over both kiwi/image-builder instead of having custom parsing for both 🙂 (see: https://redhat.atlassian.net/browse/HMS-10051)Pungi logs the 'required' deliverables for each compose in
logs/global/deliverables.json. This is hugely useful, but the format isn't good enough. The logging is done viautil.failable:it passes through
variant, arch, deliverable, subvarianttocompose._log_failed_deliverables, which logsvariant, arch, subvariant. (This is the same place where we produce the failure log lines that we see in failed-composes tickets).variant, arch, subvariant isn't enough to uniquely identify the images, though. We also need at least image type and format. Also, since in https://pagure.io/pungi/pull-request/1898 I had pungi log the Kiwi profile when logging the Koji task ID - not the subvariant - that sets up an unfortunate impedance mismatch, given the obvious plan here is "read the 'required' deliverables list then find the Koji task for each one".
I'll stew on this for a bit then propose further pungi changes.
oooh, no, there's a much easier way to do it. Many many thanks to whichever hero of pungi's past decided to give each task in a phase an index. They all look like this:
so all we need to do is make
failablealso lognum, and have the lines that log Koji task creation include the phase and the num. Then cross-referencing should be simple.Actually, let's just skip a step and put the task IDs in the JSON file, eh?
I'd say this is now more or less blocked by that pungi PR - basing this script off the
deliverables-v2.jsonwhich that PR would create is clearly the right way to do it. We would just need to read that file, gather the Koji task IDs of all the 'required' images, then run the appropriate 'pull a package list out of this task' function for each one.I can maybe do more work on the 'generate a package list' functions for different image types while we wait, I guess.
Track 'release critical packages' similar to critical-pathto Track 'compose critical packages' similar to critical-pathI've been thinking about this as 'compose critical' packages for a while now, so updating the title to that.
This has been stalled waiting on the Pungi change to go into production. I had lots of other stuff to do so I didn't rush it through. The updates should be going stable in the next day or two, so it should be possible to get back to this.
The Pungi change is now in production for Rawhide composes and we have usable deliverables metadata, so this is unblocked. woo. I'll add it to my next sprint...
Erf, so a fun discovery as I work through this: the Kiwi tasks are created with arch * and then spawn subtasks for each arch. So in deliverables-v2.json we get the 'parent' task and we're gonna have to work from that down to the subtasks for blocking arches
I've sent a PR to add my current version of the script to the releng tooling repo, so at least other people can see what I've got so far.