Policy revamp: Package Removal for Long-standing FTBFS and FTI bugs

churchyard commented

2019-03-19 10:43:36 +00:00

Current policy to change: https://fedoraproject.org/wiki/Fails_to_build_from_source

Context: https://pagure.io/fesco/issue/1973 and https://pagure.io/fesco/issue/1970

Policy proposal:

Package Removal for Long-standing FTBFS and FTI bugs

Glossary:

FTBFS = Fails To Build From Sources
FTI = Fails To Install (usually due to broken dependencies)

Packages which fail to build or fail to install will be orphaned and/or retired after a period of time.

If a package fails to build from sources or fails to install, any concerned party can file a bug in Bugzilla blocking a FTBFS/FTI tracker, providing information about the failure.
- A bug about build failure needs to block the FTBFS tracker for the appropriate release, eg. F30FTBFS for Fedora 30, see list below.
- A bug about installation failure needs to block the FTI tracker for the appropriate release, eg. F30FTI for Fedora 30, see list below.
- One bug can block multiple trackers if convenient. Reporters are encouraged to search for duplicates first.
Maintainers should either fix and close the bug or acknowledge that they are working on a solution by setting the state to ASSIGNED.
If an FTBFS or FTI bug remains in the NEW state for at least 1 week, any concerned party can set a NEEDINFO for the maintainer to respond.
If the bug remains in NEW state for at least another 3 weeks after the NEEDINFO (= at least for 4 weeks in total), any concerned party can send an e-mail reminder with the Bugzilla link to <component_name>-maintainers@fedoraproject.org, cc'ing the devel mailing list (so there is a public record) and commenting on the bug about doing so.
If the bug remains in NEW state for at least another 4 weeks after that e-mail and comment (= at least 8 weeks in total), the package will be orphaned. Orphaning can be requested via a releng issue.
The normal Orphaned package that needs new maintainers procedure will be followed for the packages orphaned in this way, leading to their retirement if nobody adopts them.
A week before the mass branching, any packages which still have open FTBFS bugs from the previous release will be retired. This can be requested via a releng issue.
A week before the scheduled beta freeze, any packages which have open FTI bugs in the NEW state with at least 8 weekly reminders will be** retired** from the relevant release and rawhide (in addition to being orphaned). (Releng ticket for this needs to be opened at least a week before the freeze, but can be opened sooner.)
The previous point repeats for the final freeze.

(Effectively, packages will be retired after 14 weeks or sooner if there is no maintainer response and the package is orphaned, or after 6½ months if the maintainer responds to FTBFS but the bug is not fixed.)

When releng performs the mass rebuild, releng opens FTBFS bugs for any packages which fail to build. Anyone can send the weekly reminders and request packages to be orphaned/retired – in other words, the procedure can be applied manually. At any point, releng can act as an concerned user and automate any steps mentioned above (for example, the weekly reminders).

Anytime a releng ticket is open, please cross reference it from the bug report.

Notes:

@zbyszek and @pviktori helped to craft this. Thanks.

I've originally wanted to add a note about stalling (setting to ASSIGNED, doing nothing), but let's not clutter the policy, such cases can be dealt with individually.

I also think that 8 weeks is too much, but that was taken from the existing policy and I don't want this to be blocked on discussion about the time. However I'd appreciate to hear your opinions on this as well.

Should we demand that the weekly reminders summarize what is happening if there is no response? Should we demand that this policy is linked from them if it wasn't already linked?

Current policy to change: https://fedoraproject.org/wiki/Fails_to_build_from_source Context: https://pagure.io/fesco/issue/1973 and https://pagure.io/fesco/issue/1970 ## Policy proposal: ---- ### Package Removal for Long-standing FTBFS and FTI bugs Glossary: * _FTBFS_ = Fails To Build From Sources * _FTI_ = Fails To Install (usually due to broken dependencies) Packages which fail to build or fail to install will be orphaned and/or retired after a period of time. 1. If a package **fails to build from sources** or **fails to install**, any concerned party can **file a bug** in Bugzilla blocking a FTBFS/FTI tracker, providing information about the failure. * A bug about build failure needs to block the FTBFS tracker for the appropriate release, eg. F30FTBFS for Fedora 30, see list below. * A bug about installation failure needs to block the FTI tracker for the appropriate release, eg. F30FTI for Fedora 30, see list below. * One bug can block multiple trackers if convenient. Reporters are encouraged to search for duplicates first. 2. Maintainers should either fix and close the bug or acknowledge that they are working on a solution by setting the state to ASSIGNED. 3. If an FTBFS or FTI bug remains in the NEW state for at least **1 week**, any concerned party can set a **NEEDINFO** for the maintainer to respond. 4. If the bug remains in NEW state for at least another 3 weeks after the NEEDINFO (= at least for **4 weeks** in total), any concerned party can send an **e-mail reminder** with the Bugzilla link to `<component_name>-maintainers@fedoraproject.org`, cc'ing the [devel mailing list](https://lists.fedoraproject.org/admin/lists/devel.lists.fedoraproject.org/) (so there is a public record) and commenting on the bug about doing so. 5. If the bug remains in NEW state for at least another 4 weeks after that e-mail and comment (= at least **8 weeks** in total), the package will be **orphaned**. Orphaning can be requested via a releng issue. 6. The normal [Orphaned package that needs new maintainers procedure](https://fedoraproject.org/wiki/Orphaned_package_that_need_new_maintainers) will be followed for the packages orphaned in this way, leading to their **retirement** if nobody adopts them. 7. A week before the **mass branching**, any packages which still have open FTBFS bugs from the previous release will be **retired**. This can be requested via a releng issue. 8. A week before the scheduled **beta freeze**, any packages which have open FTI bugs in the NEW state with at least 8 weekly reminders will be** retired** from the relevant release and rawhide (in addition to being orphaned). (Releng ticket for this needs to be opened at least a week before the freeze, but can be opened sooner.) 9. The previous point repeats for the **final freeze**. (Effectively, packages will be retired after 14 weeks or sooner if there is no maintainer response and the package is orphaned, or after 6½ months if the maintainer responds to FTBFS but the bug is not fixed.) When releng performs the [mass rebuild](https://docs.pagure.org/releng/sop_mass_rebuild.html), releng opens FTBFS bugs for any packages which fail to build. Anyone can send the weekly reminders and request packages to be orphaned/retired – in other words, the procedure can be applied manually. At any point, releng can act as an concerned user and automate any steps mentioned above (for example, the weekly reminders). Anytime a releng ticket is open, please cross reference it from the bug report. ---- ## Notes: @zbyszek and @pviktori helped to craft this. Thanks. I've originally wanted to add a note about stalling (setting to ASSIGNED, doing nothing), but let's not clutter the policy, such cases can be dealt with individually. I also think that 8 weeks is too much, but that was taken from the existing policy and I don't want this to be blocked on discussion about the time. However I'd appreciate to hear your opinions on this as well. Should we demand that the weekly reminders summarize what is happening if there is no response? Should we demand that this policy is linked from them if it wasn't already linked?

zbyszek commented

2019-03-19 11:51:22 +00:00

Owner

Current policy to change: https://fedoraproject.org/wiki/Fails_to_build_from_source

We should move this under https://docs.fedoraproject.org/en-US/fesco/ too.

> Current policy to change: https://fedoraproject.org/wiki/Fails_to_build_from_source We should move this under https://docs.fedoraproject.org/en-US/fesco/ too.

sgallagh commented

2019-03-19 12:37:53 +00:00

Owner

Can we get a little clarification on what constitutes a weekly reminder? Is it just a comment on the Bugzilla? Is it an email sent to packagename-owner@fp.o?

I think for any bug that is in NEW at least, there should be some mechanism to reach out to all comainainers via packagename-owner@fp.o in case the default assignee/CC list on Bugzilla contains only lapsed maintainers.

Can we get a little clarification on what constitutes a weekly reminder? Is it just a comment on the Bugzilla? Is it an email sent to packagename-owner@fp.o? I think for any bug that is in NEW at least, there should be some mechanism to reach out to all comainainers via packagename-owner@fp.o in case the default assignee/CC list on Bugzilla contains only lapsed maintainers.

churchyard commented

2019-03-19 12:56:14 +00:00

Author

When I drafted this, I've imagined a bugzilla comment (possibly with NEEDINFO).

If we agree that e-mailing is necessary as well, i suppose we could make it one of the steps (e.g. instead of e-mailing every week, only do it after 2 weeks or something like that).

When I drafted this, I've imagined a bugzilla comment (possibly with NEEDINFO). If we agree that e-mailing is necessary as well, i suppose we could make it one of the steps (e.g. instead of e-mailing every week, only do it after 2 weeks or something like that).

bowlofeggs commented

2019-03-19 14:48:05 +00:00

On Tue, 2019-03-19 at 10:43 +0000, Miro Hron=C4=8Dok wrote:

I also think that 8 weeks is too much, but that was taken from the
existing policy and I don't want this to be blocked on discussion
about the time. However I'd appreciate to hear your opinions on this
as well.

I actually don't mind 8 weeks. Sometimes people go on long vacations
(3-4 weeks) so if we made it much shorter they wouldn't have much time
to respond.

Should we demand that the weekly reminders summarize what is
happening if there is no response? Should we demand that this policy
is linked from them if it wasn't already linked?

The current non-responsive maintainer policy I think suffers from some
confusion about what precisely needs to be done. People often have
different interpretations about what it means. Sometimes they file a
dedicated ticket, sometimes they just comment on existing tickets. I
think it would be good if we were clear about what we want to be done
here.

I think the weekly reminder messages could say that the author intends
to have the package orphaned if there is no response.

One question I have though is whether expecting someone to write 8
messages 1 week apart each is too much of a burden? I think I would be
annoyed with having to do that. We do this for nonresponsive
maintainer, but at least it's only 3 weeks. Bugzilla does have a nag
feature for needsinfo, though I don't think it sends out reminders very
often. I wonder if we could use that instead. Or here's a maybe better
idea: what if we have a bot that looks for bugs that block the
FTI/FTBFS tracker bugs and we have it write these weekly reminders on
the tickets? Then the bot could file a releng issue at the end of the 8
weeks. This way humans don't have to carry the burden, and the bot can
put the links to the policy and what not.

On Tue, 2019-03-19 at 10:43 +0000, Miro Hron=C4=8Dok wrote: > I also think that 8 weeks is too much, but that was taken from the > existing policy and I don't want this to be blocked on discussion > about the time. However I'd appreciate to hear your opinions on this > as well. I actually don't mind 8 weeks. Sometimes people go on long vacations (3-4 weeks) so if we made it much shorter they wouldn't have much time to respond. > Should we demand that the weekly reminders summarize what is > happening if there is no response? Should we demand that this policy > is linked from them if it wasn't already linked? The current non-responsive maintainer policy I think suffers from some confusion about what precisely needs to be done. People often have different interpretations about what it means. Sometimes they file a dedicated ticket, sometimes they just comment on existing tickets. I think it would be good if we were clear about what we want to be done here. I think the weekly reminder messages could say that the author intends to have the package orphaned if there is no response. One question I have though is whether expecting someone to write 8 messages 1 week apart each is too much of a burden? I think I would be annoyed with having to do that. We do this for nonresponsive maintainer, but at least it's only 3 weeks. Bugzilla does have a nag feature for needsinfo, though I don't think it sends out reminders very often. I wonder if we could use that instead. Or here's a maybe better idea: what if we have a bot that looks for bugs that block the FTI/FTBFS tracker bugs and we have it write these weekly reminders on the tickets? Then the bot could file a releng issue at the end of the 8 weeks. This way humans don't have to carry the burden, and the bot can put the links to the policy and what not.

bowlofeggs commented

2019-03-19 14:48:45 +00:00

On Tue, 2019-03-19 at 12:37 +0000, Stephen Gallagher wrote:

Can we get a little clarification on what constitutes a weekly
reminder? Is it just a comment on the Bugzilla? Is it an email sent
to packagename-owner@fp.o?

I recommend using something public so we can verify that the policy was
followed.

On Tue, 2019-03-19 at 12:37 +0000, Stephen Gallagher wrote: > Can we get a little clarification on what constitutes a weekly > reminder? Is it just a comment on the Bugzilla? Is it an email sent > to packagename-owner@fp.o? I recommend using something public so we can verify that the policy was followed.

bowlofeggs commented

2019-03-19 14:49:26 +00:00

If we made the bot I mentioned above, it could write on BZ and also write the packagename-owner@fpo address.

churchyard commented

2019-03-19 15:48:07 +00:00

Author

Or here's a maybe better
idea: what if we have a bot that looks for bugs that block the
FTI/FTBFS tracker bugs and we have it write these weekly reminders on
the tickets? Then the bot could file a releng issue at the end of the 8
weeks. This way humans don't have to carry the burden, and the bot can
put the links to the policy and what not.

Excellent idea. But let's not block the policy on this. Everything about this needs to be automated, but wasn't (in years), so this policy aims at: anybody can follow this now.

Writing robots and releng scripts is good, but we tend to build policies on nonexistent automation. Maybe we even get somebody who writes such robot based on this policy (I certainly might once I do it three times manually).

> Or here's a maybe better > idea: what if we have a bot that looks for bugs that block the > FTI/FTBFS tracker bugs and we have it write these weekly reminders on > the tickets? Then the bot could file a releng issue at the end of the 8 > weeks. This way humans don't have to carry the burden, and the bot can > put the links to the policy and what not. Excellent idea. But let's not block the policy on this. Everything about this needs to be automated, but wasn't (in years), so this policy aims at: anybody can follow this now. Writing robots and releng scripts is good, but we tend to build policies on nonexistent automation. Maybe we even get somebody who writes such robot based on this policy (I certainly might once I do it three times manually).

bowlofeggs commented

2019-03-19 15:55:04 +00:00

On Tue, 2019-03-19 at 15:48 +0000, Miro Hron=C4=8Dok wrote:

Excellent idea. But let's not block the policy on this. Everything
about this needs to be automated, but wasn't (in years), so this
policy aims at: anybody can follow this now.
=20
Writing robots and releng scripts is good, but we tend to build
policies on nonexistent automation. Maybe we even get somebody who
writes such robot based on this policy (I certainly might once I do
it three times manually).

Well the requirement in the proposal to have humans remember to do a
weekly task for 8 weeks seems like a burden to me. I wouldn't mind if
we had a bot do that, but I think it might be too much to ask of a
human. What if we relaxed that a bit, if we don't want to require a bot
to do it?

We could make it so you have to write one comment, then 4 weeks later a
second comment, then 4 weeks after that you file the releng ticket and
link it in the original BZ?

On Tue, 2019-03-19 at 15:48 +0000, Miro Hron=C4=8Dok wrote: > Excellent idea. But let's not block the policy on this. Everything > about this needs to be automated, but wasn't (in years), so this > policy aims at: anybody can follow this now. >=20 > Writing robots and releng scripts is good, but we tend to build > policies on nonexistent automation. Maybe we even get somebody who > writes such robot based on this policy (I certainly might once I do > it three times manually). Well the requirement in the proposal to have humans remember to do a weekly task for 8 weeks seems like a burden to me. I wouldn't mind if we had a bot do that, but I think it might be too much to ask of a human. What if we relaxed that a bit, if we don't want to require a bot to do it? We could make it so you have to write one comment, then 4 weeks later a second comment, then 4 weeks after that you file the releng ticket and link it in the original BZ?

churchyard commented

2019-03-19 16:02:38 +00:00

Author

I would agree to that relaxing.

ignatenkobrain commented

2019-03-19 17:38:26 +00:00

How do you decide what FTI?

churchyard commented

2019-03-19 18:08:27 +00:00

Author

I was thinking you spin a mock, enable modular repos, install the package in it.

churchyard commented

2019-03-19 18:09:49 +00:00

Author

But the definition is not there on purpose. i guess if the package fails to install only if certain conditions are met, an interested party can still open the bug. The maintainer can always work with the reporter to understand those conditions.

till commented

2019-03-20 19:11:47 +00:00

just a comment: a tracker like "F30_fails_to_install" is a lot easier to understand for everyone than new acronyms like FTI.

kevin commented

2019-03-22 18:20:03 +00:00

Owner

Bugzilla used to at least send a email about needinfo every week. I am not sure if it still does after the upgrade to 5, but we could check that. I think a bug with needinfo that reminded every week should be enough.

churchyard commented

2019-03-23 14:11:41 +00:00

Author

I think a bug with needinfo that reminded every week should be enough.

Works for me, but let's check it. https://bugzilla.redhat.com/show_bug.cgi?id=1692033

I'll amend the proposal and incorporate the received feedback early next week.

> I think a bug with needinfo that reminded every week should be enough. Works for me, but let's check it. https://bugzilla.redhat.com/show_bug.cgi?id=1692033 --- I'll amend the proposal and incorporate the received feedback early next week.

zbyszek commented

2019-03-25 08:17:42 +00:00

Owner

Metadata Update from @zbyszek:

Issue tagged with: meeting

**Metadata Update from @zbyszek**: - Issue tagged with: meeting

zbyszek commented

2019-03-25 19:08:10 +00:00

Owner

This was discussed during today's FESCo meeting:
ACTION: mhroncok to update the proposal

This was discussed during today's FESCo meeting: ACTION: mhroncok to update the proposal

zbyszek commented

2019-03-25 19:08:10 +00:00

Owner

Metadata Update from @zbyszek:

Issue untagged with: meeting

**Metadata Update from @zbyszek**: - Issue **un**tagged with: meeting

sgallagh commented

2019-04-01 13:18:34 +00:00

Owner

Any updates here?

churchyard commented

2019-04-01 18:54:42 +00:00

Author

@kevin Did you get any NEEDINFO e-mail? @zbyszek suggested on the IRC meeting last week that such e-mails are no longer delivered.

kevin commented

2019-04-01 18:59:37 +00:00

Owner

Sadly, I did not, so it seems they are no longer sent. ;(

Too bad really. I can ask bugzilla admins about it...

Sadly, I did not, so it seems they are no longer sent. ;( Too bad really. I can ask bugzilla admins about it...

churchyard commented

2019-04-01 20:22:32 +00:00

Author

OK, I'll amend the proposal expecting we can sort things out with bugzilla admins.

If we find out we cannot, it can be amended once more.

OK, I'll amend the proposal expecting we can sort things out with bugzilla admins. If we find out we cannot, it can be amended once more.

churchyard commented

2019-04-01 20:58:35 +00:00

Author

I've amended the proposal. tl;dr diff:

you open the bug
after 1 week, you set NEEDINFO
after +3 (=4) weeks, you send e-mail + comment
after +4 (=8) weeks, the package is orphaned

I've amended the proposal. tl;dr diff: * you open the bug * after 1 week, you set NEEDINFO * after +3 (=4) weeks, you send e-mail + comment * after +4 (=8) weeks, the package is orphaned

bowlofeggs commented

2019-04-03 17:30:15 +00:00

On Mon, 2019-04-01 at 20:58 +0000, Miro Hron=C4=8Dok wrote:

I've amended the proposal. tl;dr diff:
=20

you open the bug

after 1 week, you set NEEDINFO

after +3 (=3D4) weeks, you send e-mail + comment

after +4 (=3D4) weeks, the package is orphaned

+1

On Mon, 2019-04-01 at 20:58 +0000, Miro Hron=C4=8Dok wrote: > I've amended the proposal. tl;dr diff: >=20 > * you open the bug > * after 1 week, you set NEEDINFO > * after +3 (=3D4) weeks, you send e-mail + comment > * after +4 (=3D4) weeks, the package is orphaned +1

churchyard commented

2019-04-07 09:58:46 +00:00

Author

Metadata Update from @churchyard:

Issue tagged with: meeting

**Metadata Update from @churchyard**: - Issue tagged with: meeting

sgallagh commented

2019-04-09 14:59:56 +00:00

Owner

https://pagure.io/fesco/issue/2109#comment-563753 is approved, with one amendment that we add an email to pkg-owner@fp.o at the NEEDINFO step (+8, 0, -0)