Allow query more than one repo and other improvements

sergiomb commented

2024-04-15 17:31:56 +00:00

Contributor

I'm sending a batch of commits , you can merge commits, one by one and test that should run .
The goal was simplify the logic and make more easy to understand , optimize , i.e. make less calls and allow query various repos at same time , can be useful for testing updates and updates-testing repos , EPEL repos and 3rd party repos .

Let me know , what is the best way to present the commits

I'm sending a batch of commits , you can merge commits, one by one and test that should run . The goal was simplify the logic and make more easy to understand , optimize , i.e. make less calls and allow query various repos at same time , can be useful for testing updates and updates-testing repos , EPEL repos and 3rd party repos . Let me know , what is the best way to present the commits

sergiomb commented

2024-04-15 17:36:25 +00:00

Author

Contributor

Forgot to tell find_unblocked_orphans.py can be used to check dependencies of a package.

for example:
find_unblocked_orphans.py --max_deps 300 --skip-orphans glib gtk+ GConf2 libbonoboui libgnome libgnomeui > glib_orphans.txt

will list packages that depend to build or to install on glib gtk+ GConf2 libbonoboui libgnome libgnomeui

Forgot to tell find_unblocked_orphans.py can be used to check dependencies of a package. for example: find_unblocked_orphans.py --max_deps 300 --skip-orphans glib gtk+ GConf2 libbonoboui libgnome libgnomeui > glib_orphans.txt will list packages that depend to build or to install on glib gtk+ GConf2 libbonoboui libgnome libgnomeui

sergiomb commented

2024-04-16 23:28:50 +00:00

Author

Contributor

@gotmax23 this PR is for orphaned-packages-process/find_unblocked_orphans.py, maybe you can review it or test it ...

Thanks

@gotmax23 this PR is for orphaned-packages-process/find_unblocked_orphans.py, maybe you can review it or test it ... Thanks

sergiomb commented

2024-04-29 18:06:36 +00:00

Author

Contributor

rebased onto f7a450f884042897a15f4c6b96fa4c6cd34af263

sergiomb commented

2024-05-01 23:40:01 +00:00

Author

Contributor

rebased onto 0e2d94425d8f27ce9643ee83ef26d22c1f685eaa

sergiomb commented

2024-05-01 23:48:08 +00:00

Author

Contributor

I placed the first 3 commits in another PR, so they can be merged first and to facilitate review

https://pagure.io/releng/pull-request/12090

I placed the first 3 commits in another PR, so they can be merged first and to facilitate review https://pagure.io/releng/pull-request/12090

sergiomb commented

2024-05-02 02:56:18 +00:00

Author

Contributor

12 new commits added

allow user mock config with --mock-config option
dep_checker now can run standalone.
add rpmfusion repos
move maintainer information out of depchecker
calculate the package's branch, to avoid errors when trying to obtain information from the maintainer when packages is outside of Fedora.
make SRPM one independent function
move class DepcChecker and PagureInfo to new files
Allow search dependencies in more than one repository
Simplify the code logic, allow do mapping by repo
Fix when report starts and only call koji when we calculate orphans
trivial fix
minor simplification

**12 new commits added** * ``allow user mock config with --mock-config option`` * ``dep_checker now can run standalone.`` * ``add rpmfusion repos`` * ``move maintainer information out of depchecker`` * ``calculate the package's branch, to avoid errors when trying to obtain information from the maintainer when packages is outside of Fedora.`` * ``make SRPM one independent function`` * ``move class DepcChecker and PagureInfo to new files`` * ``Allow search dependencies in more than one repository`` * ``Simplify the code logic, allow do mapping by repo`` * ``Fix when report starts and only call koji when we calculate orphans`` * ``trivial fix`` * ``minor simplification``

sergiomb commented

2024-05-04 10:26:05 +00:00

Author

Contributor

1 new commit added

get just binary latest() packages while sources rpm get it all

**1 new commit added** * ``get just binary latest() packages while sources rpm get it all``

sergiomb commented

2024-05-06 14:46:57 +00:00

Author

Contributor

13 new commits added

if given packages doesn't not exist in repos, exit with error
just latest packages while sources get all packages
allow user mock config with --mock-config option
dep_checker now can run standalone.
add rpmfusion repos
move class DepcChecker and PagureInfo to new files
move maintainer information out of depchecker
calculate the package's branch, to avoid errors when trying to obtain information from the maintainer when packages is outside of Fedora.
Allow search dependencies in more than one repository
Simplify the code logic, allow do mapping by repo
Fix when report starts and only call koji when we calculate orphans
trivial fix
minor simplification

**13 new commits added** * ``if given packages doesn't not exist in repos, exit with error`` * ``just latest packages while sources get all packages`` * ``allow user mock config with --mock-config option`` * ``dep_checker now can run standalone.`` * ``add rpmfusion repos`` * ``move class DepcChecker and PagureInfo to new files`` * ``move maintainer information out of depchecker`` * ``calculate the package's branch, to avoid errors when trying to obtain information from the maintainer when packages is outside of Fedora.`` * ``Allow search dependencies in more than one repository`` * ``Simplify the code logic, allow do mapping by repo`` * ``Fix when report starts and only call koji when we calculate orphans`` * ``trivial fix`` * ``minor simplification``

sergiomb commented

2024-05-11 01:35:03 +00:00

Author

Contributor

15 new commits added

Allow not check src.rpm dependencies
does not stop when the package source is not found and print the package which source package was not found
if given packages doesn't not exist in repos, exit with error
allow user mock config with --mock-config option v2
dep_checker now can run standalone.
add rpmfusion repos
move maintainer information out of depchecker
calculate the package's branch, to avoid errors when trying to obtain information from the maintainer when packages is outside of Fedora.
make SRPM one independent function
move class DepcChecker and PagureInfo to new files
Allow search dependencies in more than one repository
Simplify the code logic, allow do mapping by repo
Fix when report starts and only call koji when we calculate orphans
trivial fix
minor simplification

**15 new commits added** * ``Allow not check src.rpm dependencies`` * ``does not stop when the package source is not found and print the package which source package was not found`` * ``if given packages doesn't not exist in repos, exit with error`` * ``allow user mock config with --mock-config option v2`` * ``dep_checker now can run standalone.`` * ``add rpmfusion repos`` * ``move maintainer information out of depchecker`` * ``calculate the package's branch, to avoid errors when trying to obtain information from the maintainer when packages is outside of Fedora.`` * ``make SRPM one independent function`` * ``move class DepcChecker and PagureInfo to new files`` * ``Allow search dependencies in more than one repository`` * ``Simplify the code logic, allow do mapping by repo`` * ``Fix when report starts and only call koji when we calculate orphans`` * ``trivial fix`` * ``minor simplification``

sergiomb commented

2024-05-11 01:41:33 +00:00

Author

Contributor

rebased onto 24622051ebb7df7f5f9d814d4541ffbcfa666ae4

sergiomb commented

2024-05-11 01:48:40 +00:00

Author

Contributor

on commits of 2024-05-06 , I had introduce a bug, I had to rollback all the code , now it seems to me fine again

sergiomb commented

2024-05-18 01:37:32 +00:00

Author

Contributor

16 new commits added

dnf latest only and simplications
by default use mock config, metalink and mirrors
Allow not check src.rpm dependencies
if given packages doesn't not exist in repos, exit with error
allow user mock config with --mock-config option v2
dep_checker now can run standalone.
add rpmfusion repos
move maintainer information out of depchecker
calculate the package's branch, to avoid errors when trying to obtain information from the maintainer when packages is outside of Fedora.
simplified SRPM function and make it independent of the class
move class DepcChecker and PagureInfo to new files
Allow search dependencies in more than one repository
Simplify the code logic, allow do mapping by repo
Fix when report starts and only call koji when we calculate orphans
trivial fix
minor simplification

**16 new commits added** * ``dnf latest only and simplications`` * ``by default use mock config, metalink and mirrors`` * ``Allow not check src.rpm dependencies`` * ``if given packages doesn't not exist in repos, exit with error`` * ``allow user mock config with --mock-config option v2`` * ``dep_checker now can run standalone.`` * ``add rpmfusion repos`` * ``move maintainer information out of depchecker`` * ``calculate the package's branch, to avoid errors when trying to obtain information from the maintainer when packages is outside of Fedora.`` * ``simplified SRPM function and make it independent of the class`` * ``move class DepcChecker and PagureInfo to new files`` * ``Allow search dependencies in more than one repository`` * ``Simplify the code logic, allow do mapping by repo`` * ``Fix when report starts and only call koji when we calculate orphans`` * ``trivial fix`` * ``minor simplification``

humaton commented

2024-06-03 18:21:22 +00:00

Owner

Sorry for the lag on this. The separate PR is merged.

sergiomb commented

2024-06-03 21:06:01 +00:00

Author

Contributor

wow ! I will check the state 1 , I got more things in mind to add related with this I'd query deps with extra repo like copr etc.

Many thanks

wow ! I will check the state 1 , I got more things in mind to add related with this I'd query deps with extra repo like copr etc. Many thanks

sergiomb commented

2024-06-03 21:08:30 +00:00

Author

Contributor

ah only the trivial fix was merged , I will prepare this one a little better .

thank you

ah only the trivial fix was merged , I will prepare this one a little better . thank you

humaton commented

2024-06-04 07:05:19 +00:00

Owner

It looks good, by reading the code. but can you please rebase it? I will be able to test the changes and merge or report problems.

sergiomb commented

2024-09-19 00:51:36 +00:00

Author

Contributor

rebased onto 94f7c8ae0a

rebased onto 94f7c8ae0afd7216824a4079a3f6f1adadc57785

kevin commented

2025-01-06 21:00:58 +00:00

Owner

Whats the status here?

sergiomb commented

2025-01-07 14:51:38 +00:00

Author

Contributor

Hello, I have testing my new code , is it good for me, looks like is finished but I need rearrange the commits , to be well documented and more easy to understand , i had done some loops in the commits that I want solve, before sending to you .
For now, I have 2 new commits , that are minor changes https://pagure.io/releng/pull-request/12519

Hello, I have testing my new code , is it good for me, looks like is finished but I need rearrange the commits , to be well documented and more easy to understand , i had done some loops in the commits that I want solve, before sending to you . For now, I have 2 new commits , that are minor changes https://pagure.io/releng/pull-request/12519

sergiomb commented

2025-01-07 14:56:48 +00:00

Author

Contributor

rebased onto 606438f8dc

rebased onto 606438f8dc5e46a0c7a219bd31a948ea1ea4943a

sergiomb commented

2025-01-13 19:15:53 +00:00

Author

Contributor

rebased onto 8deb9fd249

rebased onto 8deb9fd249ee58f70ab9e987b23b9f531e507c20

sergiomb commented

2025-01-30 20:04:54 +00:00

Author

Contributor

rebased onto 1bfc541fe5

rebased onto 1bfc541fe5917b370195b4a2ec7d5b195d98d85e

sergiomb commented

2025-02-24 23:16:39 +00:00

Author

Contributor

rebased onto 00c112cb10

rebased onto 00c112cb10814d43c5331ef0fb079a3468636025

sergiomb commented

2025-02-26 12:22:09 +00:00

Author

Contributor

11 new commits added

change default of max_dep default in dep_checker to 0
The subdict variable is always set, no need to check
add rpmfusion repos
Latest developments
dnf latest only
Allow not check src.rpm dependencies
Allow use mock configurations to query repos
dep_checker now can run standalone.
Move class DepcChecker and PagureInfo to new files
Move maintainer information from depchecker class
Allow search dependencies in more than one repository

**11 new commits added** * ``change default of max_dep default in dep_checker to 0`` * ``The subdict variable is always set, no need to check`` * ``add rpmfusion repos`` * ``Latest developments`` * ``dnf latest only`` * ``Allow not check src.rpm dependencies`` * ``Allow use mock configurations to query repos`` * ``dep_checker now can run standalone.`` * ``Move class DepcChecker and PagureInfo to new files`` * ``Move maintainer information from depchecker class`` * ``Allow search dependencies in more than one repository``

sergiomb commented

2025-02-26 12:27:35 +00:00

Author

Contributor

The development is finished , we now can query packages in all fedora, epel branches etc and with or without 3rd party repos , for example :

./dep_checker.py --mock-repo fedora+rpmfusion_nonfree-rawhide-x86_64 GConf2

./dep_checker.py --mock-repo epel+rpmfusion_nonfree-8-x86_64  clamav

@gotmax23 I separate the first 3 commits to review and merge, here: https://pagure.io/releng/pull-request/12524 , with some development that allow us to split find_unblocked_orphans.py in multiple files .

The development is finished , we now can query packages in all fedora, epel branches etc and with or without 3rd party repos , for example : ./dep_checker.py --mock-repo fedora+rpmfusion_nonfree-rawhide-x86_64 GConf2 ./dep_checker.py --mock-repo epel+rpmfusion_nonfree-8-x86_64 clamav @gotmax23 I separate the first 3 commits to review and merge, here: https://pagure.io/releng/pull-request/12524 , with some development that allow us to split find_unblocked_orphans.py in multiple files .

james commented

2025-02-26 17:52:31 +00:00

Member

Not tested anything, just looked at the code.

tl;dr It all seems good.

Minor comments per. patch:

Subject: [PATCH 3/11] Move class DepcChecker and PagureInfo to new files

Having a common.py or misc.py or utils etc. is always tempting, but sucks in the future when 666 unrelated things end up in there.

Having a releases_conf.py for the top parts and textui.py for the eprint/wrapping would be better.

Also future people will be happier if you add a comment at the top of the new dep_checker.py giving an example of how to run it and/or roughly what you'd use it to do.

Subject: [PATCH 5/11] Allow use mock configurations to query repos

Feels like there should be a better way to build the mock repos. than doing it by hand in the option parsing?
Even just adding a decent comment saying "this bit parses repos. from mock config" or whatever would help.

Subject: [PATCH 7/11] dnf latest only

Did you check the speed of self.dnfquery.latest(1).filter(provides=base_provide) vs self.dnfquery.filter(provides=base_provide).latest(1) ... would assume the later is better, esp. on big repos. with multiple packages (but could be 0.5 seconds or something useless ;).

Subject: [PATCH 11/11] change default of max_dep default in dep_checker to 0

I assume 0 is inifinate? Does this work fine with deploops?

Not tested anything, just looked at the code. tl;dr It all seems good. Minor comments per. patch: Subject: [PATCH 3/11] Move class DepcChecker and PagureInfo to new files Having a common.py or misc.py or utils etc. is always tempting, but sucks in the future when 666 unrelated things end up in there. Having a releases_conf.py for the top parts and textui.py for the eprint/wrapping would be better. Also future people will be happier if you add a comment at the top of the new dep_checker.py giving an example of how to run it and/or roughly what you'd use it to do. Subject: [PATCH 5/11] Allow use mock configurations to query repos Feels like there should be a better way to build the mock repos. than doing it by hand in the option parsing? Even just adding a decent comment saying "this bit parses repos. from mock config" or whatever would help. Subject: [PATCH 7/11] dnf latest only Did you check the speed of self.dnfquery.latest(1).filter(provides=base_provide) vs self.dnfquery.filter(provides=base_provide).latest(1) ... would assume the later is better, esp. on big repos. with multiple packages (but could be 0.5 seconds or something useless ;). Subject: [PATCH 11/11] change default of max_dep default in dep_checker to 0 I assume 0 is inifinate? Does this work fine with deploops?

sergiomb commented

2025-02-26 19:28:46 +00:00

Author

Contributor

I assume 0 is inifinate? Does this work fine with deploops?

is the opposite , none deps except the direct ones

> I assume 0 is inifinate? Does this work fine with deploops? is the opposite , none deps except the direct ones

gotmax23 commented

2025-02-26 22:11:33 +00:00

Member

I will add this to my TODO for this weekend. Thanks for working on this! I am concerned that some of the changes here go beyond the scope of this script's intended use—which is to facilitate the orphaned packages process, not serve as a general repository querying tool.

sergiomb commented

2025-02-26 22:57:02 +00:00

Author

Contributor

I will add this to my TODO for this weekend. Thanks for working on this! I am concerned that some of the changes here go beyond the scope of this script's intended use—which is to facilitate the orphaned packages process, not serve as a general repository querying tool.

First yes, it is to go beyond the scope. I found that find_unblocked_orphans.py calculate very well the rpm dependencies, so the goal is expand it to calculate rpm deps in other ways.
To preserve the scope I keept find_unblocked_orphans.py as is , exactly with same features and have some freedom on dep_checker.py

> I will add this to my TODO for this weekend. Thanks for working on this! I am concerned that some of the changes here go beyond the scope of this script's intended use—which is to facilitate the orphaned packages process, not serve as a general repository querying tool. First yes, it is to go beyond the scope. I found that find_unblocked_orphans.py calculate very well the rpm dependencies, so the goal is expand it to calculate rpm deps in other ways. To preserve the scope I keept find_unblocked_orphans.py as is , exactly with same features and have some freedom on dep_checker.py

sergiomb commented

2025-02-26 23:03:08 +00:00

Author

Contributor

I assume 0 is inifinate? Does this work fine with deploops?

is the opposite , none deps except the direct ones

But is a good idea , can be set infinity max_deps, also max_deps calculation itself can be improved. About dep loops , I think is not a problem because since one rpm is added is not added again, I think

> > I assume 0 is inifinate? Does this work fine with deploops? > > is the opposite , none deps except the direct ones But is a good idea , can be set infinity max_deps, also max_deps calculation itself can be improved. About dep loops , I think is not a problem because since one rpm is added is not added again, I think

sergiomb commented

2025-03-03 20:07:36 +00:00

Author

Contributor

Subject: [PATCH 3/11] Move class DepcChecker and PagureInfo to new files

Having a common.py or misc.py or utils etc. is always tempting, but sucks in the future when 666 unrelated things end up in there.

Having a releases_conf.py for the top parts and textui.py for the eprint/wrapping would be better.

Also future people will be happier if you add a comment at the top of the new dep_checker.py giving an example of how to run it and/or roughly what you'd use it to do.

all comments of @james are pertinent

Since we can improve the patch 3 and later , I suggest just merge the first two, to certificate the code is good and the PR for that is here: https://pagure.io/releng/pull-request/12524

The first patch is needed for the second patch to identify the repos, the second patch is necessary, to be able to separate the Pagure information class, from the dep checker class .

> Subject: [PATCH 3/11] Move class DepcChecker and PagureInfo to new files > > Having a common.py or misc.py or utils etc. is always tempting, but sucks in the future when 666 unrelated things end up in there. > > Having a releases_conf.py for the top parts and textui.py for the eprint/wrapping would be better. > > > Also future people will be happier if you add a comment at the top of the new dep_checker.py giving an example of how to run it and/or roughly what you'd use it to do. all comments of @james are pertinent Since we can improve the patch 3 and later , I suggest just merge the first two, to certificate the code is good and the PR for that is here: https://pagure.io/releng/pull-request/12524 The first patch is needed for the second patch to identify the repos, the second patch is necessary, to be able to separate the Pagure information class, from the dep checker class .

gotmax23 commented

2025-03-05 16:20:48 +00:00

Member

I don't think we can have references to rpmfusion here

gotmax23 commented

2025-03-05 16:22:09 +00:00

Member

Parsing mock configs seems completely out of scope here and adds a fair amount of extra code.

sergiomb commented

2025-03-05 18:57:15 +00:00

Author

Contributor

rebased onto d61b031990

rebased onto d61b03199063520359def285f450a06740c20da6

sergiomb commented

2025-03-05 19:00:59 +00:00

Author

Contributor

removed rpmfusion repos commit
Using mock configs allow us check deps on any version or distribution , like James wrote, we can improve it

- removed rpmfusion repos commit - Using mock configs allow us check deps on any version or distribution , like James wrote, we can improve it

sergiomb commented

2025-03-06 14:29:05 +00:00

Author

Contributor

Parsing mock configs seems completely out of scope here and adds a fair amount of extra code.

To be clear , this PR is supposed to extended the orphaned-packages-process and also calculate deps on any repo , any distro , any arch etc that is my first goal , after we can add support to other things link weak deps for example .

This tool is very useful to me, today I'm updating MLT I need to know If i can update movit with soname bump on epel 9 and I ran :

./dep_checker.py --mock-repo epel+rpmfusion_nonfree-9-x86_64 movit

that use dnf cache and mirrorlist which is much faster and causes less load on koji

the result is only mlt depend on movit, so I can update it also .

So the split of find_unblocked_orphans.py have two goals, one be more organized , second is develop dep_checker without interfere on find_unblocked_orphans.py calculations . find_unblocked_orphans.py works exactly in the same way .

> Parsing mock configs seems completely out of scope here and adds a fair amount of extra code. To be clear , this PR is supposed to extended the orphaned-packages-process and also calculate deps on any repo , any distro , any arch etc that is my first goal , after we can add support to other things link weak deps for example . This tool is very useful to me, today I'm updating MLT I need to know If i can update movit with soname bump on epel 9 and I ran : ./dep_checker.py --mock-repo epel+rpmfusion_nonfree-9-x86_64 movit that use dnf cache and mirrorlist which is much faster and causes less load on koji the result is only mlt depend on movit, so I can update it also . So the split of find_unblocked_orphans.py have two goals, one be more organized , second is develop dep_checker without interfere on find_unblocked_orphans.py calculations . find_unblocked_orphans.py works exactly in the same way .

gotmax23 commented

2025-04-03 17:48:15 +00:00

Member

So I'm still thinking this might be better maintained as a fork in a separate repo. These changes add extra complication outside of the script's main purpose and will make it harder to convert the script to use the libdnf5 which will need to happen eventually. I'm curious to hear what releng members think.

kevin commented

2025-04-03 19:22:39 +00:00

Owner

Yeah, I mean, I am ok helping other uses, but this seems like it has added a lot, so perhaps it should be seperate?

sergiomb commented

2025-04-11 22:38:28 +00:00

Author

Contributor

Hi,

So I'm still thinking this might be better maintained as a fork in a separate repo. These changes add extra complication outside of the script's main purpose and will make it harder to convert the script to use the libdnf5 which will need to happen eventually. I'm curious to hear what releng members think.

I understand the problem of adding things that are not related to the real purpose.

so I will do first only commit that improve organization and orphaned-packages-process script
the 2 next commits already do that more or less (https://pagure.io/releng/pull-request/12524 )

"Query more than one repo" is a new feature , is it interesting to orphaned-packages-process script ?

about convert to libdnf5 , I will check , I don't think that would be difficult

Best regards

Hi, > So I'm still thinking this might be better maintained as a fork in a separate repo. These changes add extra complication outside of the script's main purpose and will make it harder to convert the script to use the libdnf5 which will need to happen eventually. I'm curious to hear what releng members think. I understand the problem of adding things that are not related to the real purpose. so I will do first only commit that improve organization and orphaned-packages-process script the 2 next commits already do that more or less (https://pagure.io/releng/pull-request/12524 ) "Query more than one repo" is a new feature , is it interesting to orphaned-packages-process script ? about convert to libdnf5 , I will check , I don't think that would be difficult Best regards

jnsamyak commented

2025-10-02 08:45:53 +00:00

Owner

Any updates on this, is this pending for review? We have moved this script to a new path, if it is still required can you rebase your pr, and we can get it in review please?

jnsamyak closed this pull request

2026-01-15 17:02:05 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

Rows
Columns

Allow query more than one repo and other improvements #12065

Pull request closed