need_rebuild.py reports false positives due to Koji query filtering (non-RPMs and target mismatch) #13429

New issue

Open

opened 2026-07-18 21:39:28 +00:00 by jesusdelosreyes98 · 3 comments

jesusdelosreyes98 commented

2026-07-18 21:39:28 +00:00

Describe the issue

The mass rebuild tracking script (need_rebuild.py) currently reports inaccurate "needs rebuild" lists. As noted by nirik and Fabio Valentini in recent chat discussions, there are two distinct symptoms of this issue:

1. Non-RPM items are listed as failed packages
Items like Fedora-Cloud-Base-GCE are appearing on the need-rebuild list.

Root Cause: The script uses listPackages(target, inherited=True) to populate the initial list. This pulls in all Koji packages for the tag without filtering by btype. Consequently, Image and Module builds are evaluated alongside standard RPMs, causing them to falsely flag as failed rebuilds.

2. Successfully rebuilt packages are ignored
Packages that successfully completed their mass rebuild (e.g., PyQt-builder) are still flagged as needing a rebuild.

Root Cause: After pulling builds via listBuilds(..., state=1), the script iterates through them and calls getTaskInfo(..., request=True) to inspect the task request tuple. It does a rigid string match on the target parameter. If a maintainer uses a side-tag (e.g., f45-python), or if the bot uses a specific rebuild target, the script may discard the valid build.

Proposed Solution:
Instead of fetching all packages and looping through listBuilds and getTaskInfo, we can optimize and fix both bugs by querying the tags directly:

Use listTagged(target, type='rpm', latest=True) to fetch only RPM builds and immediately exclude Cloud/Image builds.
Evaluate success based on the build's presence in the correct target tag and its creation_time relative to the mass-rebuild epoch, rather than relying on strict string matching of the request target parameter.

I am currently working on a patch for this and will submit a PR shortly.

When do you need this? (YYYY/MM/DD)

Before the next mass rebuild cycle.

When is this no longer needed or useful? (YYYY/MM/DD)

N/A

If we cannot complete this, what is the impact? [Dependencies/Blocker]

The need_rebuild.py tracker will continue to display false positives, causing the Release Engineering team to waste time manually verifying packages and OS images that do not actually need to be rebuilt.

Checklist

I have checked existing issues for duplicates
I have filled out all the fields above
I have provided relevant links or references (if applicable)

## Describe the issue The mass rebuild tracking script (`need_rebuild.py`) currently reports inaccurate "needs rebuild" lists. As noted by `nirik` and Fabio Valentini in recent chat discussions, there are two distinct symptoms of this issue: **1. Non-RPM items are listed as failed packages** Items like `Fedora-Cloud-Base-GCE` are appearing on the need-rebuild list. * **Root Cause:** The script uses `listPackages(target, inherited=True)` to populate the initial list. This pulls in all Koji packages for the tag without filtering by `btype`. Consequently, Image and Module builds are evaluated alongside standard RPMs, causing them to falsely flag as failed rebuilds. **2. Successfully rebuilt packages are ignored** Packages that successfully completed their mass rebuild (e.g., `PyQt-builder`) are still flagged as needing a rebuild. * **Root Cause:** After pulling builds via `listBuilds(..., state=1)`, the script iterates through them and calls `getTaskInfo(..., request=True)` to inspect the task request tuple. It does a rigid string match on the target parameter. If a maintainer uses a side-tag (e.g., `f45-python`), or if the bot uses a specific rebuild target, the script may discard the valid build. **Proposed Solution:** Instead of fetching all packages and looping through `listBuilds` and `getTaskInfo`, we can optimize and fix both bugs by querying the tags directly: 1. Use `listTagged(target, type='rpm', latest=True)` to fetch only RPM builds and immediately exclude Cloud/Image builds. 2. Evaluate success based on the build's presence in the correct target tag and its `creation_time` relative to the mass-rebuild epoch, rather than relying on strict string matching of the `request` target parameter. I am currently working on a patch for this and will submit a PR shortly. ## When do you need this? (YYYY/MM/DD) Before the next mass rebuild cycle. ## When is this no longer needed or useful? (YYYY/MM/DD) N/A ## If we cannot complete this, what is the impact? [Dependencies/Blocker] The `need_rebuild.py` tracker will continue to display false positives, causing the Release Engineering team to waste time manually verifying packages and OS images that do not actually need to be rebuilt. --- ## Checklist - [x] I have checked existing issues for duplicates - [x] I have filled out all the fields above - [x] I have provided relevant links or references (if applicable)

jnsamyak commented

2026-07-19 02:52:30 +00:00

Owner

The script uses listPackages(target, inherited=True) to populate the initial list. This pulls in all Koji packages for the tag without filtering by btype. Consequently, Image and Module builds are evaluated alongside standard RPMs, causing them to falsely flag as failed rebuilds.

This happens because we have Fedora-* based in our dist-git; for this, we have PKG_SKIP_LIST, make sure to use that instead of adding bigger logic for this altogether.

After pulling builds via listBuilds(..., state=1), the script iterates through them and calls getTaskInfo(..., request=True) to inspect the task request tuple. It does a rigid string match on the target parameter. If a maintainer uses a side-tag (e.g., f45-python), or if the bot uses a specific rebuild target, the script may discard the valid build.

For the context, in mass-rebuilds, we use the f45-rebuild tag with releng as the user filing it, so we might need to look at what is currently missing from the current logic.

> The script uses listPackages(target, inherited=True) to populate the initial list. This pulls in all Koji packages for the tag without filtering by btype. Consequently, Image and Module builds are evaluated alongside standard RPMs, causing them to falsely flag as failed rebuilds. This happens because we have Fedora-* based in our dist-git; for this, we have PKG_SKIP_LIST, make sure to use that instead of adding bigger logic for this altogether. > After pulling builds via listBuilds(..., state=1), the script iterates through them and calls getTaskInfo(..., request=True) to inspect the task request tuple. It does a rigid string match on the target parameter. If a maintainer uses a side-tag (e.g., f45-python), or if the bot uses a specific rebuild target, the script may discard the valid build. For the context, in mass-rebuilds, we use the f45-rebuild tag with releng as the user filing it, so we might need to look at what is currently missing from the current logic.

jnsamyak commented

2026-07-19 02:52:49 +00:00

Owner

If you any questions feel free to reach out.

decathorpe commented

2026-07-21 08:13:51 +00:00

Contributor

I found at least one issue with the script:

The "epoch" is set to '2026-07-15 18:00:00.000000' but quite a few package builds were created before this date, for example, nirik's PyQt-builder (created at 'Wed, 15 Jul 2026 17:53:59 UTC', 6 minutes before the epoch set in the need_rebuild.py script). It looks like at least some false positives (mostly packages with names that start with capital letters) should get fixed by fixing the epoch in need_rebuild.py.

I found at least one issue with the script: The "epoch" is set to `'2026-07-15 18:00:00.000000'` but quite a few package builds were created *before* this date, for example, nirik's PyQt-builder (created at `'Wed, 15 Jul 2026 17:53:59 UTC'`, 6 minutes *before* the epoch set in the need_rebuild.py script). It looks like at least some false positives (mostly packages with names that start with capital letters) should get fixed by fixing the epoch in need_rebuild.py.