From: Thomas Huth <thuth@redhat.com>
To: Nicholas Piggin <npiggin@gmail.com>,
Andrew Jones <andrew.jones@linux.dev>
Cc: Laurent Vivier <lvivier@redhat.com>,
linuxppc-dev@lists.ozlabs.org, kvm@vger.kernel.org
Subject: Re: [kvm-unit-tests PATCH v9 02/31] report: Add known failure reporting option
Date: Mon, 6 May 2024 09:25:37 +0200 [thread overview]
Message-ID: <aed85321-7e8e-4202-9f91-791229ef9455@redhat.com> (raw)
In-Reply-To: <20240504122841.1177683-3-npiggin@gmail.com>
On 04/05/2024 14.28, Nicholas Piggin wrote:
> There are times we would like to test a function that is known to fail
> in some conditions due to a bug in implementation (QEMU, KVM, or even
> hardware). It would be nice to count these as known failures and not
> report a summary failure.
>
> xfail is not the same thing, xfail means failure is required and a pass
> causes the test to fail. So add kfail for known failures.
Actually, I wonder whether that's not rather a bug in report_xfail()
instead. Currently, when you call report_xfail(true, ...), the result is
*always* counted as a failure, either as an expected failure (if the test
really failed), or as a normal failure (if the test succeeded). What's the
point of counting a successful test as a failure??
Andrew, you've originally introduced report_xfail in commit a5af7b8a67e,
could you please comment on this?
IMHO we should rather do something like this instead:
diff --git a/lib/report.c b/lib/report.c
--- a/lib/report.c
+++ b/lib/report.c
@@ -98,7 +98,7 @@ static void va_report(const char *msg_fmt,
skipped++;
else if (xfail && !pass)
xfailures++;
- else if (xfail || !pass)
+ else if (!xfail && !pass)
failures++;
spin_unlock(&lock);
Thomas
next prev parent reply other threads:[~2024-05-06 7:25 UTC|newest]
Thread overview: 49+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-05-04 12:28 [kvm-unit-tests PATCH v9 00/31] powerpc improvements Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 01/31] doc: update unittests doc Nicholas Piggin
2024-05-06 7:03 ` Thomas Huth
2024-05-07 3:57 ` Nicholas Piggin
2024-05-06 8:02 ` Andrew Jones
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 02/31] report: Add known failure reporting option Nicholas Piggin
2024-05-06 7:25 ` Thomas Huth [this message]
2024-05-06 8:01 ` Andrew Jones
2024-05-06 10:19 ` Thomas Huth
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 03/31] powerpc: Mark known failing tests as kfail Nicholas Piggin
2024-05-06 7:37 ` Thomas Huth
2024-05-07 4:07 ` Nicholas Piggin
2024-05-07 11:44 ` Thomas Huth
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 04/31] powerpc: Update unittests for latest QEMU version Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 05/31] powerpc/sprs: Specify SPRs with data rather than code Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 06/31] powerpc/sprs: Avoid taking PMU interrupts caused by register fuzzing Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 07/31] scripts: allow machine option to be specified in unittests.cfg Nicholas Piggin
2024-05-07 15:08 ` Thomas Huth
2024-05-08 12:27 ` Nicholas Piggin
2024-05-08 12:55 ` Thomas Huth
2024-05-08 12:58 ` Thomas Huth
2024-05-08 13:36 ` Thomas Huth
2024-05-09 5:44 ` Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 08/31] scripts: Accommodate powerpc powernv machine differences Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 09/31] powerpc: Support powernv machine with QEMU TCG Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 10/31] powerpc: Fix emulator illegal instruction test for powernv Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 11/31] powerpc/sprs: Test hypervisor registers on powernv machine Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 12/31] powerpc: general interrupt tests Nicholas Piggin
2024-05-07 12:12 ` Thomas Huth
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 13/31] powerpc: Add rtas stop-self support Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 14/31] powerpc: Remove broken SMP exception stack setup Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 15/31] powerpc: Enable page alloc operations Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 16/31] powerpc: add SMP and IPI support Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 17/31] powerpc: Add cpu_relax Nicholas Piggin
2024-05-07 13:44 ` Thomas Huth
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 18/31] powerpc: Permit ACCEL=tcg,thread=single Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 19/31] powerpc: Avoid using larx/stcx. in spinlocks when only one CPU is running Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 20/31] powerpc: Add atomics tests Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 21/31] powerpc: Add timebase tests Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 22/31] powerpc: Add MMU support Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 23/31] common/sieve: Use vmalloc.h for setup_mmu definition Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 24/31] common/sieve: Support machines without MMU Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 25/31] powerpc: Add sieve.c common test Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 26/31] powerpc: add usermode support Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 27/31] powerpc: add pmu tests Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 28/31] configure: Make arch_libdir a first-class entity Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 29/31] powerpc: Remove remnants of ppc64 directory and build structure Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 30/31] powerpc: Add facility to query TCG or KVM host Nicholas Piggin
2024-05-04 12:28 ` [kvm-unit-tests PATCH v9 31/31] powerpc: gitlab CI update Nicholas Piggin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=aed85321-7e8e-4202-9f91-791229ef9455@redhat.com \
--to=thuth@redhat.com \
--cc=andrew.jones@linux.dev \
--cc=kvm@vger.kernel.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=lvivier@redhat.com \
--cc=npiggin@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).