releng/tickets

Fork 2

Include installed_pkgs.log in Koji output #11532

New issue

Open

opened 2023-07-11 17:12:48 +00:00 by sgallagh · 8 comments

sgallagh commented

2023-07-11 17:12:48 +00:00

Member

Describe the issue

Currently, the Content Resolver (https://tiny.distro.builders/) relies on unreliable parsing of root.log to identify the build dependencies for a build. All of the information it needs would actually be provided by installed_pkgs.log if we were including that in the output.

When do you need this? (YYYY/MM/DD)
Before the F39 mass-rebuild starts.
When is this no longer needed or useful? (YYYY/MM/DD)
Useful forever.
If we cannot complete your request, what is the impact?
We have to continue to use unreliable sources to determine the packages present in a build.

Additionally, we plan to enhance the ELN DistroBuildSync tool to take advantage of this for more reliable rebuilds of side-tags.

* Describe the issue Currently, the Content Resolver (https://tiny.distro.builders/) relies on unreliable parsing of root.log to identify the build dependencies for a build. All of the information it needs would actually be provided by installed_pkgs.log if we were including that in the output. * When do you need this? (YYYY/MM/DD) Before the F39 mass-rebuild starts. * When is this no longer needed or useful? (YYYY/MM/DD) Useful forever. * If we cannot complete your request, what is the impact? We have to continue to use unreliable sources to determine the packages present in a build. Additionally, we plan to enhance the ELN DistroBuildSync tool to take advantage of this for more reliable rebuilds of side-tags.

sgallagh commented

2023-07-11 17:13:38 +00:00

Author

Member

See https://rpm-software-management.github.io/mock/Plugin-PackageState for details on adding the necessary mock config.

phsmoura commented

2023-07-11 18:13:40 +00:00

Metadata Update from @phsmoura:

Issue tagged with: medium-gain, medium-trouble, ops

**Metadata Update from @phsmoura**: - Issue tagged with: medium-gain, medium-trouble, ops

asamalik commented

2023-07-12 07:27:20 +00:00

That won't work well, because there's no distinction between directly required and dependencies. That distinction is quite important when you're trying to cut build dependencies — you need to know what's an actual direct build dependency to cut, and what was dragged in there as a runtime dependency of the build dependency.

We have it in the CentOS Stream koji: https://kojihub.stream.centos.org/kojifiles/packages/autofs/5.1.4/108.el8/data/logs/aarch64/installed_pkgs.log

^^ see it's just one long list of everything.

The root log splits it into two sections: https://kojihub.stream.centos.org/kojifiles/packages/autofs/5.1.4/108.el8/data/logs/aarch64/root.log

Installing group/module packages: - directly required
Installing dependencies: - dependencies

Content Resolver only takes the directly rquired, and resolves the dependencies itself. If you only use the single list you'll lose the distinction.

Parsing root logs was really the last resort, believe me, I looked at installed_pkgs.log before.

Ideally we'd have arch-specific SRPMs available in a repo. Or actually just the repodata would be enough, we don't need the packages. Just need to have a way to query the dependencies, which are in many cases different per architecture.

That won't work well, because there's no distinction between *directly required* and *dependencies*. That distinction is quite important when you're trying to cut build dependencies — you need to know what's an actual direct build dependency to cut, and what was dragged in there as a runtime dependency of the build dependency. We have it in the CentOS Stream koji: https://kojihub.stream.centos.org/kojifiles/packages/autofs/5.1.4/108.el8/data/logs/aarch64/installed_pkgs.log ^^ see it's just one long list of everything. The root log splits it into two sections: https://kojihub.stream.centos.org/kojifiles/packages/autofs/5.1.4/108.el8/data/logs/aarch64/root.log 1. *Installing group/module packages:* - directly required 2. *Installing dependencies:* - dependencies Content Resolver only takes the directly rquired, and resolves the dependencies itself. If you only use the single list you'll lose the distinction. --- Parsing root logs was really the last resort, believe me, I looked at installed_pkgs.log before. Ideally we'd have arch-specific SRPMs available in a repo. Or actually just the repodata would be enough, we don't need the packages. Just need to have a way to query the dependencies, which are in many cases different per architecture.

kevin commented

2023-07-28 00:41:02 +00:00

Owner

In light of that... is this still desired?

sgallagh commented

2023-08-01 15:13:56 +00:00

Author

Member

It's still desirable for the ELN build sync; We're looking at this as a way to potentially make rebuilds more reliable there by replicating the buildroot more exactly.

churchyard commented

2023-08-02 11:17:29 +00:00

Member

It's also desirable to debug sudden build failures. Such lists should be easier to diff.

kevin commented

2026-05-04 15:55:11 +00:00

Owner

So, it's been two years... still wanted?

The place to add it would be: roles/koji_builder/templates/builders/site-defaults.cfg

it does mean every single build will have another log file/take up more space, but I suppose it's not a super big deal.

So, it's been two years... still wanted? The place to add it would be: roles/koji_builder/templates/builders/site-defaults.cfg it does mean every single build will have another log file/take up more space, but I suppose it's not a super big deal.

sgallagh commented

2026-05-04 16:37:35 +00:00

Author

Member

@kevin wrote in #11532 (comment):

So, it's been two years... still wanted?

The place to add it would be: roles/koji_builder/templates/builders/site-defaults.cfg

it does mean every single build will have another log file/take up more space, but I suppose it's not a super big deal.

Funnily-enough, I was just discussing this topic this morning with @yselkowitz

The output of that discussion was this:

installed-pkgs.log is not going to do what we want, because it includes everything installed in the container, not just the dependencies installed by BuildRequires:
There's no information available about what is a direct dependency vs an indirect one.

Sometime in the future, I think we need to discuss with the DNF and mock teams about having a mode of operation that produces a machine-readable output format showing the fully-resolved graph it used to determine what it installs. As things stand right now, installed-pkgs.log isn't the right choice for us (Content Resolver).

@kevin wrote in https://forge.fedoraproject.org/releng/tickets/issues/11532#issuecomment-692860: > So, it's been two years... still wanted? > > The place to add it would be: roles/koji_builder/templates/builders/site-defaults.cfg > > it does mean every single build will have another log file/take up more space, but I suppose it's not a super big deal. Funnily-enough, I was just discussing this topic this morning with @yselkowitz The output of that discussion was this: 1. `installed-pkgs.log` is not going to do what we want, because it includes everything installed in the container, not just the dependencies installed by `BuildRequires:` 2. There's no information available about what is a direct dependency vs an indirect one. Sometime in the future, I think we need to discuss with the DNF and mock teams about having a mode of operation that produces a machine-readable output format showing the fully-resolved graph it used to determine what it installs. As things stand right now, `installed-pkgs.log` isn't the right choice for us (Content Resolver).