releng/tickets

Fork 2

Packages that fail to build SRPM are not reported during the mass rebuild bugzillas #8601

New issue

Open

opened 2019-08-09 17:34:09 +00:00 by churchyard · 18 comments

churchyard commented

2019-08-09 17:34:09 +00:00

Contributor

See for example: https://bugzilla.redhat.com/show_bug.cgi?id=1739634

The package cannot create SRPM:

$ fedpkg srpm 


error: Bad source: .../python-chm/pychm-0.8.4.1.tar.gz: No such file or directory
Could not execute srpm: Failed to execute command.

And hence it was not rebuilt since Fedora 29:

https://koji.fedoraproject.org/koji/packageinfo?packageID=3571

But no FTBFS bug was reported.

See for example: https://bugzilla.redhat.com/show_bug.cgi?id=1739634 The package cannot create SRPM: ``` $ fedpkg srpm error: Bad source: .../python-chm/pychm-0.8.4.1.tar.gz: No such file or directory Could not execute srpm: Failed to execute command. ``` And hence it was not rebuilt since Fedora 29: https://koji.fedoraproject.org/koji/packageinfo?packageID=3571 But no FTBFS bug was reported.

ignatenkobrain commented

2019-08-12 15:47:31 +00:00

Contributor

Nice finding!

kevin commented

2019-08-19 21:21:48 +00:00

Owner

Yeah, this is because koji doesn't even send the 'start' fedmsg until after this, so there's not an indication a build was attempted.

Perhaps we could add in another set of messages there before/after srpm creation?

not sure how else to detect them. ;(

Yeah, this is because koji doesn't even send the 'start' fedmsg until after this, so there's not an indication a build was attempted. Perhaps we could add in another set of messages there before/after srpm creation? not sure how else to detect them. ;(

churchyard commented

2019-08-19 21:22:44 +00:00

Author

Contributor

Well we do attempt to build them, so a check after the mass rebuild for packages we never heard back from would do.

pbrobinson commented

2019-08-22 08:54:43 +00:00

Contributor

Not entirely the same failure but it seems tcl wasn't bumped at all.

churchyard commented

2019-11-14 00:02:38 +00:00

Author

Contributor

tcl has https://src.fedoraproject.org/rpms/tcl/blob/master/f/noautobuild - something we should probably stop using.

humaton commented

2021-04-13 15:56:06 +00:00

Owner

Metadata Update from @humaton:

Issue tagged with: mini-initiative

**Metadata Update from @humaton**: - Issue tagged with: mini-initiative

mohanboddu commented

2021-04-20 15:14:48 +00:00

Contributor

Filed upstream ticket - https://pagure.io/koji/issue/2825

mohanboddu commented

2021-05-25 16:32:20 +00:00

Contributor

Metadata Update from @mohanboddu:

Issue tagged with: backlog

**Metadata Update from @mohanboddu**: - Issue tagged with: backlog

vondruch commented

2022-01-31 10:39:12 +00:00

Member

@humaton Just FYI, this is related to the MLT rebuild we have discussed via IRC

churchyard commented

2022-07-25 21:49:33 +00:00

Author

Contributor

This could be solved by not only reporting bugzillas for packages listed in 37-failures.html, but also the remaining packages listed only in f37-need-rebuild.html. The bugzillas could be slightly different (they could say: "Your package was not rebuilt in the mass rebuild, investigate why"). That could also help discover packages like https://pagure.io/releng/issue/10920

This could be solved by not only reporting bugzillas for packages listed in [37-failures.html](https://kojipkgs.fedoraproject.org/mass-rebuild/f37-failures.html), but also the remaining packages listed only in [f37-need-rebuild.html](https://kojipkgs.fedoraproject.org/mass-rebuild/f37-need-rebuild.html). The bugzillas could be slightly different (they could say: "Your package was not rebuilt in the mass rebuild, investigate why"). That could also help discover packages like https://pagure.io/releng/issue/10920

james commented

2023-08-28 18:55:46 +00:00

Member

I created a test cli application that can get this data out of koji (how old buildtime data is for each package in a tag): github.com/CentOS/sync2git@1cba9c84b1

...obviously can do a lot of changes from here, at the moment it will give you lists of packages with a buildtime older than a certain duration or a historgram of the buildtime of all packages in a tag. Eg. Fedora-38:

2021-08-13     72 
2021-09-12    136 
2021-10-13      0 
2021-11-12      0 
2021-12-12     57 
2022-01-12   1054 
2022-02-11    116 
2022-03-13      7 
2022-04-13     19 
2022-05-13      6 
2022-06-12    177 
2022-07-13   2238 
2022-08-12    291 
2022-09-11    415 
2022-10-12    358 
2022-11-11    714 
2022-12-11    684 
2023-01-11 214615 ##################################################
2023-02-10  38332 ########
2023-03-12  29175 ######

So ~208 packages with buildtimes before 2021-10-13. Note that I don't filter to .src packages atm. but you can do that in the lists easily:

% wc -l work/lp-f38-50w
4151 work/lp-f38-50w
% fgrep .src work/lp-f38-50w | wc -l
380

...anyway, I guess the next step is working out if we want to tweak how we present the data so a human can look at it or add code so it can just email the owners on it's own?

I created a test cli application that can get this data out of koji (how old buildtime data is for each package in a tag): https://github.com/CentOS/sync2git/commit/1cba9c84b1587d633d3183aa147c8b25a0aadafa ...obviously can do a lot of changes from here, at the moment it will give you lists of packages with a buildtime older than a certain duration or a historgram of the buildtime of all packages in a tag. Eg. Fedora-38: ``` 2021-08-13 72 2021-09-12 136 2021-10-13 0 2021-11-12 0 2021-12-12 57 2022-01-12 1054 2022-02-11 116 2022-03-13 7 2022-04-13 19 2022-05-13 6 2022-06-12 177 2022-07-13 2238 2022-08-12 291 2022-09-11 415 2022-10-12 358 2022-11-11 714 2022-12-11 684 2023-01-11 214615 ################################################## 2023-02-10 38332 ######## 2023-03-12 29175 ###### ``` So ~208 packages with buildtimes before 2021-10-13. Note that I don't filter to .src packages atm. but you can do that in the lists easily: ``` % wc -l work/lp-f38-50w 4151 work/lp-f38-50w % fgrep .src work/lp-f38-50w | wc -l 380 ``` ...anyway, I guess the next step is working out if we want to tweak how we present the data so a human can look at it or add code so it can just email the owners on it's own?

james commented

2023-08-28 22:01:48 +00:00

Member

FYI, filtered to source packages (f38):

2021-08-13     6 
2021-09-12     7 
2021-10-13     0 
2021-11-12     0 
2021-12-12     4 
2022-01-12   123 
2022-02-11     8 
2022-03-13     2 
2022-04-13     2 
2022-05-13     1 
2022-06-12    11 
2022-07-13   198 
2022-08-12    19 
2022-09-11    37 
2022-10-12    23 
2022-11-11    23 
2022-12-11    32 
2023-01-11 19257 ##################################################
2023-02-10  2102 #####
2023-03-12  1253 ###

FYI, filtered to source packages (f38): ``` 2021-08-13 6 2021-09-12 7 2021-10-13 0 2021-11-12 0 2021-12-12 4 2022-01-12 123 2022-02-11 8 2022-03-13 2 2022-04-13 2 2022-05-13 1 2022-06-12 11 2022-07-13 198 2022-08-12 19 2022-09-11 37 2022-10-12 23 2022-11-11 23 2022-12-11 32 2023-01-11 19257 ################################################## 2023-02-10 2102 ##### 2023-03-12 1253 ### ```

kevin commented

2023-08-29 19:57:55 +00:00

Owner

Thanks for working on this!

Ideally, next step would be to use this information and adjust the existing https://pagure.io/releng/blob/main/f/scripts/mass_rebuild_file_bugs.py script to use it.

ie, in addition to the ones that it finds that failed, also find the ones that haven't been rebuilt since the mass rebuild started and report bugs on them as not having been built. That might catch more than the ones that just failed to make a src.rpm in koji, but ideally any that were skipped for any reason should get a bug so the maintainer(s) can fix it.

We may want to I guess before that mail the devel list with the list we have and ask them if there's any false positives we need to exclude?

The only one that leaps to mind is shim...

Thanks for working on this! Ideally, next step would be to use this information and adjust the existing https://pagure.io/releng/blob/main/f/scripts/mass_rebuild_file_bugs.py script to use it. ie, in addition to the ones that it finds that failed, also find the ones that haven't been rebuilt since the mass rebuild started and report bugs on them as not having been built. That might catch more than the ones that just failed to make a src.rpm in koji, but ideally any that were skipped for any reason should get a bug so the maintainer(s) can fix it. We may want to I guess before that mail the devel list with the list we have and ask them if there's any false positives we need to exclude? The only one that leaps to mind is shim...