Re: adding KFAILs to C++ testsuite

[Michael Elizabeth Chastain <mec at shout dot net>]

Hi Fernando,

> We hope to have 0 FAILed tests as a baseline, independent on the 
> platform, target etc.  Tjis way, if you make some changes and you see 
> something != 0 you can start looking for what you have broken ;-)

Sort of, in my opinion.  I regard "0 FAILs" as an unattainable goal
(although I remember that Daniel J had one configuration with 0 FAILs
with some patches he had that got lost in patch-approval limbo).

New FAILs are going to come in all the time as things change.  I don't
want to drop everything and nag people at high priority every time a
FAIL happens.

My goals are:

  At any moment, every covered configuration has 10 or fewer FAILs.
  Each new FAIL spends a maximum of 4 weeks before becoming something else
    (it gets fixed, or it becomes a KFAIL or an XFAIL).

Here are some life cycles:

  # bug gets fixed quickly
  Person #1 runs a test run and notices a new FAIL.
  Person #1 files a bug report on it.
  A week later, Person #2 fixes the bug.
  Someone closes the bug report.

  # bug does not get fixed quickly
  Person #1 runs a test run and notices a new FAIL.
  Person #1 files a bug report on it.
  Several weeks later, nothing further has happened.
  Someone (person #1 or someone else) edits the test suite -> KFAIL.

I think it's too much work + test suite thrashing to edit the test
suite after every run where a new FAIL happens.  I'd like to give
people a little time to just fix the bugs and save a bunch of
test suite patch submission activity.

If the # of un-KFAIL'ed FAIL's starts to pile up and gets to be more
than 10, then I would start complaining to people to fix more bugs or
analyze more bugs or review more KFAIL patches before doing anything else.
I personally can analyze bugs and submit good KFAIL patches (and commit
them in gdb.c++) so I think that my goals are attainable.

Michael C