From: Felipe Contreras <felipe.contreras@gmail.com>
To: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
"Calvin Wan" <calvinwan@google.com>
Cc: git@vger.kernel.org
Subject: Re: [RFC PATCH 0/2] add an external testing library for unit tests
Date: Tue, 02 May 2023 09:28:11 -0600 [thread overview]
Message-ID: <64512c0b1237b_1ba2d29426@chronos.notmuch> (raw)
In-Reply-To: <230502.861qjyj0cb.gmgdl@evledraar.gmail.com>
Ævar Arnfjörð Bjarmason wrote:
> We already have pure-C libraries that we add a really shim to unit test,
> the most extreme example of this is t0032-reftable-unittest.sh, whose
> body is simply (excluding comments):
>
> #!/bin/sh
> test_description='reftable unittests'
>
> TEST_PASSES_SANITIZE_LEAK=true
> . ./test-lib.sh
>
> test_expect_success 'unittests' '
> TMPDIR=$(pwd) && export TMPDIR &&
> test-tool reftable
> '
>
> test_done
Yeah, but an output of 'ok 1 - unittests' is not very useful, neither is an
output of 'not ok 1 - unittests'.
This completely misses the point of a TAP interface, which is to parse the
status of individual test cases, or even individual assertions.
If all we are doing is check the exit code of some program, then we don't need
TAP.
> Now, that goes into reftable/test_framework.h which basically implements
> its own mini-test framework, so that's at least a *partial* argument for
> what you're suggesting here, but note that it doesn't emit TAP, it just
> returns an exit code, the EXPECT() etc. is purely internal. I.e. "what
> should we return?".
You are just describing the status quo.
I think this is a naturaistic fallacy: confusing what is versus what ought to
be.
Can we test C code with our current testing framework? Yes.
Should we? That's the actual question.
> None of those are perfect, but I think the current arrangement is rather
> ideal.
I think misuing TAP is far from ideal.
In my view an ideal framework would:
1. Be able to test C code
2. Report individual test cases success/failure
3. Report relevant context in the case of failure (actual vs. expected)
4. Don't create forks on every individual test case or assertion
5. Properly handle crashes
Doing basically the following is not an ideal framework:
echo '1..1'
test-tool reftable > /dev/null 2>&1 && echo 'ok 1' || { echo 'not ok 1'; false; }
Yes, it works. But "it works" has never been a good enough reason to stay on
the status quo.
> We can write most or all of the test in C, but we just do so by
> calling a function that returns an exit code.
Yes, we *can*, but should we?
---
If we can test C code in a way that individual test case failures are reported
(as is the intention with TAP), why would we reject that in favor of the status
quo which basically is just reporting the exit code of the whole test file?
--
Felipe Contreras
prev parent reply other threads:[~2023-05-02 15:28 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-04-27 17:50 [RFC PATCH 0/2] add an external testing library for unit tests Calvin Wan
2023-04-27 17:50 ` [RFC PATCH 1/2] Add C TAP harness Calvin Wan
2023-04-27 18:29 ` SZEDER Gábor
2023-04-27 18:38 ` Calvin Wan
2023-04-27 20:15 ` Phillip Wood
2023-04-28 16:31 ` Calvin Wan
2023-05-02 15:46 ` Felipe Contreras
2023-05-10 15:46 ` Phillip Wood
2023-05-11 23:16 ` Glen Choo
2023-05-18 10:04 ` Phillip Wood
2023-06-21 15:57 ` Linus Arver
2023-06-26 13:15 ` Phillip Wood
2023-06-28 21:17 ` Linus Arver
2023-06-29 5:52 ` Oswald Buddenhagen
2023-06-30 9:48 ` Phillip Wood
2023-05-02 15:54 ` Felipe Contreras
2023-05-02 16:39 ` Ævar Arnfjörð Bjarmason
2023-05-02 18:11 ` Felipe Contreras
2023-05-02 16:34 ` Ævar Arnfjörð Bjarmason
2023-05-10 8:18 ` Phillip Wood
2023-04-27 17:50 ` [RFC PATCH 2/2] unit test: add basic example Calvin Wan
2023-04-27 18:47 ` Junio C Hamano
2023-05-02 15:58 ` Felipe Contreras
2023-04-27 18:39 ` [RFC PATCH 0/2] add an external testing library for unit tests Junio C Hamano
2023-04-27 18:46 ` Calvin Wan
2023-04-27 21:35 ` brian m. carlson
2023-05-02 4:18 ` Felipe Contreras
2023-05-02 13:52 ` Ævar Arnfjörð Bjarmason
2023-05-02 15:28 ` Felipe Contreras [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=64512c0b1237b_1ba2d29426@chronos.notmuch \
--to=felipe.contreras@gmail.com \
--cc=avarab@gmail.com \
--cc=calvinwan@google.com \
--cc=git@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).