Allow turning on opt-in gating

pingou commented

2018-08-07 10:17:44 +00:00

Currently bodhi is querying greenwave for every update to ask if the update can go through and greenwave is configured to not enforce any tests, so all updates are going through.

Greenwave recently got the feature allowing packagers to define their own requirements, per package.

In addition, one of the main issue with enforcing tests was the UX around waiving false-negative. This is improved in the up-coming bodhi release (3.9, deployed in staging) via the bodhi client. The following release improves a little more the web UI (merged in git). One last item needs to be fixed to make the web UI as flexible as the CLI, I will try to work on it sooner rather than later.

I would like FESCo's approval to turn on greenwave's opt-in feature in Fedora once bodhi 3.9 has been deployed in production. I have tested the feature in stage and got an update blocked as expected. Another deployment of greenwave is using it with success as well.

The documentation about greenwave's opt-in feature can be found at: https://docs.pagure.org/greenwave/package-specific-policies.html

Currently bodhi is querying greenwave for every update to ask if the update can go through and greenwave is configured to not enforce any tests, so all updates are going through. Greenwave recently got the feature allowing packagers to define their own requirements, per package. In addition, one of the main issue with enforcing tests was the UX around waiving false-negative. This is improved in the up-coming bodhi release (3.9, deployed in staging) via the bodhi client. The following release improves a little more the web UI (merged in git). One last item needs to be fixed to make the web UI as flexible as the CLI, I will try to work on it sooner rather than later. I would like FESCo's approval to turn on greenwave's opt-in feature in Fedora once bodhi 3.9 has been deployed in production. I have tested the feature in stage and got an update blocked as expected. Another deployment of greenwave is using it with success as well. The documentation about greenwave's opt-in feature can be found at: https://docs.pagure.org/greenwave/package-specific-policies.html

jsmith commented

2018-08-09 07:15:14 +00:00

+1. Seems reasonable.

sgallagh commented

2018-08-09 07:28:18 +00:00

Owner

Before we vote, can you post a guide on what packagers would need to do in order to opt-in (a step-by-step HOWTO would be ideal)? The current documentation is super-technical on what the YAML file looks like, but there's nothing like a quick-start guide to help people on the way.

Unless you're intending ONLY greenwave devs to be opting in at this point, of course.

Before we vote, can you post a guide on what packagers would need to do in order to opt-in (a step-by-step HOWTO would be ideal)? The current documentation is super-technical on what the YAML file looks like, but there's nothing like a quick-start guide to help people on the way. Unless you're intending ONLY greenwave devs to be opting in at this point, of course.

pingou commented

2018-08-09 09:39:53 +00:00

Author

can you post a guide on what packagers would need to do in order to opt-in

All they would have to do is add a gating.yaml file in the dist-git repo of the packages they want to have gated.

Unless you're intending ONLY greenwave devs to be opting in at this point, of course.

I expect they will be among the first ones to do so, especially at the beginning to ensure things work properly.

I definitely want to expand the documentation on how to get a package gated and what a package can be gated on once we confirmed everything works as expected.

> can you post a guide on what packagers would need to do in order to opt-in All they would have to do is add a gating.yaml file in the dist-git repo of the packages they want to have gated. > Unless you're intending ONLY greenwave devs to be opting in at this point, of course. I expect they will be among the first ones to do so, especially at the beginning to ensure things work properly. I definitely want to expand the documentation on how to get a package gated and what a package can be gated on once we confirmed everything works as expected.

till commented

2018-08-10 13:04:18 +00:00

In https://pagure.io/fesco/issue/1872#comment-511656 we decided to allow opt-in gating when the devs think it is ready, therefore +1.

I agree with @sgallagh that there should be better documentation and IMHO also a better user interface. Ideally it would be just a knob in dist-git pagure to enable opt-in or a fedpkg command.

In https://pagure.io/fesco/issue/1872#comment-511656 we decided to allow opt-in gating when the devs think it is ready, therefore +1. I agree with @sgallagh that there should be better documentation and IMHO also a better user interface. Ideally it would be just a knob in dist-git pagure to enable opt-in or a fedpkg command.

zbyszek commented

2018-08-10 13:57:13 +00:00

Owner

The docs under https://docs.pagure.org/greenwave/package-specific-policies.html seem pretty clear. I'm not sure that we actually need to support this through the UI. After all we are all command-line animals and we can expect maintainers to configure a simple text file. I'd even say that most users will probably be more happy with that interface then with a graphical web ui.

strong +1 from me

The docs under https://docs.pagure.org/greenwave/package-specific-policies.html seem pretty clear. I'm not sure that we actually need to support this through the UI. After all we are all command-line animals and we can expect maintainers to configure a simple text file. I'd even say that most users will probably be more happy with that interface then with a graphical web ui. strong +1 from me

till commented

2018-08-10 14:26:18 +00:00

The docs under https://docs.pagure.org/greenwave/package-specific-policies.html seem pretty clear. I'm not sure that we actually need to support this through the UI. After all we are all command-line animals and we can expect maintainers to configure a simple text file. I'd even say that most users will probably be more happy with that interface then with a graphical web ui.

The bad user interface contributed to the problems with the first run and requiring now a complex yaml file for a simple task like opting-in to gating seems like it would repeat the mistakes. All in all we want big participation, therefore it should be as easy as possible to participate. When I glance at the current documentation it requires to read a lot and leads to more open questions. The first code snippet is:

--- !Policy
product_versions:
  - fedora-26
decision_context: bodhi_update_push_testing
rules:
  - !PassingTestCaseRule {test_case_name: dist.depcheck}

Fedora 26 is already EOL, so it is a bad example. Also I guess maintainers would just enable it for all releases, what would be the value here? Do they need to manually update the file for each release?
What does decision_context mean? Which other values make sense here?
Does the exclamation mark mean negation like it does in other contexts?
What tests can be listed there? How can one enable all tests?

It would be a lot user friendly if there is a tool where one can choose from the valid choices instead of having to figure out the internal details one is not caring about. This will also make it less likely that maintainers accidentally create invalid gating files. Teaching maintainers this file format is IMHO time that could be better spent on other tasks that require human intelligence. And having a proper interface does not prevent you from changing the raw data if you prefer this. And last but not least, fedpkg is just a wrapper for everything else so that maintainers do not have to care about too many internals such as build targets, tags or URLs to make maintaining pkgs easier.

Btw. is there a way for a maintainer to check now if gating would block current updates? IIRC Bodhi does not show whether ignored tests would have led to a block.

> The docs under https://docs.pagure.org/greenwave/package-specific-policies.html seem pretty clear. I'm not sure that we actually need to support this through the UI. After all we are all command-line animals and we can expect maintainers to configure a simple text file. I'd even say that most users will probably be more happy with that interface then with a graphical web ui. The bad user interface contributed to the problems with the first run and requiring now a complex yaml file for a simple task like opting-in to gating seems like it would repeat the mistakes. All in all we want big participation, therefore it should be as easy as possible to participate. When I glance at the current documentation it requires to read a lot and leads to more open questions. The first code snippet is: ``` --- !Policy product_versions: - fedora-26 decision_context: bodhi_update_push_testing rules: - !PassingTestCaseRule {test_case_name: dist.depcheck} ``` - Fedora 26 is already EOL, so it is a bad example. Also I guess maintainers would just enable it for all releases, what would be the value here? Do they need to manually update the file for each release? - What does decision_context mean? Which other values make sense here? - Does the exclamation mark mean negation like it does in other contexts? - What tests can be listed there? How can one enable all tests? It would be a lot user friendly if there is a tool where one can choose from the valid choices instead of having to figure out the internal details one is not caring about. This will also make it less likely that maintainers accidentally create invalid gating files. Teaching maintainers this file format is IMHO time that could be better spent on other tasks that require human intelligence. And having a proper interface does not prevent you from changing the raw data if you prefer this. And last but not least, fedpkg is just a wrapper for everything else so that maintainers do not have to care about too many internals such as build targets, tags or URLs to make maintaining pkgs easier. Btw. is there a way for a maintainer to check now if gating would block current updates? IIRC Bodhi does not show whether ignored tests would have led to a block.

kevin commented

2018-08-15 05:36:50 +00:00

Owner

+1 to allow opt in, but yeah, still more work is needed before enabling everyone.

jwboyer commented

2018-08-15 10:36:08 +00:00

+1

bowlofeggs commented

2018-08-17 12:57:33 +00:00

I'm a +1 to opt-in, and I agree that the methods for opting in (i.e., the not-well-documented yaml file) are not going to work well at scale (but are fine for testing the system until it is broadly useful).

maxamillion commented

2018-08-17 19:14:35 +00:00

+1 to opt-in

psabata commented

2018-08-20 10:33:25 +00:00

Also +1 on the opt-in.

lholecek commented

2018-08-20 10:42:55 +00:00

@till

Fedora 26 is already EOL, so it is a bad example. Also I guess maintainers would just enable it for all releases, what would be the value here? Do they need to manually update the file for each release?

Better value would be fedora-* (see PR).

What does decision_context mean? Which other values make sense here?

That's a bug. It doesn't have any meaning in gating.yaml yet (see the issue).

Does the exclamation mark mean negation like it does in other contexts?

If you mean exclamation mark in rules (!PassingTestCaseRule), it's just part of YAML syntax. Looks like this would be also worth mentioning in documentation.

What tests can be listed there? How can one enable all tests?

You have to list the tests explicitly, there is no predefined list with test names (test name is just arbitrary string which is part of test result in ResultsDB).

@till > Fedora 26 is already EOL, so it is a bad example. Also I guess maintainers would just enable it for all releases, what would be the value here? Do they need to manually update the file for each release? Better value would be `fedora-*` (see [PR](https://pagure.io/greenwave/pull-request/283)). > What does decision_context mean? Which other values make sense here? That's a bug. It doesn't have any meaning in `gating.yaml` yet (see [the issue](https://pagure.io/greenwave/issue/282)). > Does the exclamation mark mean negation like it does in other contexts? If you mean exclamation mark in rules (`!PassingTestCaseRule`), it's just part of [YAML syntax](http://yaml.org/refcard.html). Looks like this would be also worth mentioning in documentation. > What tests can be listed there? How can one enable all tests? You have to list the tests explicitly, there is no predefined list with test names (test name is just arbitrary string which is part of test result in ResultsDB).

psabata commented

2018-08-25 21:57:04 +00:00

I see six +1. I consider this approved.

psabata commented

2018-08-25 21:57:48 +00:00

Metadata Update from @psabata:

Issue close_status updated to: Fixed
Issue status updated to: Closed (was: Open)

**Metadata Update from @psabata**: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)

Rows
Columns

Allow turning on opt-in gating #1966