Re: [RFA:] sim-defs.exp: support xfail

[Andrew Cagney <cagney at gnu dot org>]

Hans-Peter Nilsson wrote:
> > Date: Wed, 10 Nov 2004 09:01:56 -0500
> > From: Andrew Cagney <cagney@gnu.org>
>
>
> > Hans-Peter Nilsson wrote:
> >
> > > Directly taken from the same option in the ld run_dump_test.
> > > Ok to commit?
> > > 2004-11-10  Hans-Peter Nilsson  <hp@axis.com>
> > >
> > > 	* lib/sim-defs.exp (run_sim_test): Support "xfail" option.
> >
> > What do you mean by "xfail"? (Hint, check a current dejagnu document 
> > where it describes kfail :-)
>
>
> I know about kfail, but I'm agnostic.  I just mean what happens
> for the ld run_dump_test "xfail" option; a simple setup_xfail.
> The purpose is to be able to keep failing tests around but not
> get the total results marked as partially failing.  Reasons for
> the failure would be unspecified, matching both xfail and kfail
> usage.
>
> To me, it doesn't really matter whether it's kfail or xfail
> (vivid descriptions of kfail vs. xfail to /dev/null).  In the
> sim testsuite I believe it'll be used very sparingly, perhaps
> not even for things that are checked in.  But it'd be Nice to
> Have.
>
> Is the change acceptable by naming the option "kfail" and
> calling "setup_kfail"?  Or do you want both?
>
> brgds, H-P
> PS.  BTW, the dejagnu documentation on
> <URL:http://www.gnu.org/software/dejagnu/manual/> is not
> current; KFAIL isn't mentioned there.

Ok, really weird, the date on that page says 2004 but the manual is 
missing (from dejagnu 1.4.4):

@item KFAIL
A test is known to fail in some environment(s) due to a known bug
in the tool being tested (identified by a bug id string).  This
exists so that, after a bug is identified and properly registered
in a bug tracking database (Gnats, for instance), the count of
failures can be kept as zero.  Having zero as a baseline in all
platforms allow the tool developers to immediately detect regressions
caused by changes (which may affect some platforms and not others).
The connection with a bug tracking database allows for automatic
generation of the BUGS section of man pages or Release Notes, as
well as a "Bugs Fixed this Release" section (by comparing to a
previous release set of known failures).
The procedure @code{setup_kfail} is used to indicate a failure is
known to exist.

@cindex KFAIL, avoiding for POSIX
As with @code{XFAIL}, @sc{posix} tests must return @code{FAIL} rather
than @code{KFAIL} even if a failure was due to a known bug.

I can't think of a single reason for a simulator test that failing not 
being for a known bug.

However, for consistency with dejagnu, we might as well support both. 
Feel free to add the necessary glue to cover all of xfail and kfail and 
kpass.

Andrew