From: Felipe Contreras <felipe.contreras@gmail.com>
To: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
"Felipe Contreras" <felipe.contreras@gmail.com>
Cc: Phillip Wood <phillip.wood123@gmail.com>,
Calvin Wan <calvinwan@google.com>,
git@vger.kernel.org
Subject: Re: [RFC PATCH 1/2] Add C TAP harness
Date: Tue, 02 May 2023 12:11:10 -0600 [thread overview]
Message-ID: <6451523ec8b47_1ba2d29427@chronos.notmuch> (raw)
In-Reply-To: <230502.86sfcehecl.gmgdl@evledraar.gmail.com>
Ævar Arnfjörð Bjarmason wrote:
> On Tue, May 02 2023, Felipe Contreras wrote:
>
> > Phillip Wood wrote:
> >
> >> Unfortunately this library doesn't seem to offer any of those features.
> >> It does support a lazy test plan but uses atexit() so will not detect if
> >> the test program exits before all the tests have run.
> >
> > I think there's a fundamental misunderstanding of how we use TAP.
> >
> > If a program generates this output:
> >
> > 1..3
> > ok 1 - test 1
> > ok 2 - test 2
> >
> > That's clearly not complete. It shouldn't be the job a test script to check for
> > those cases.
> >
> > If you run the programm through a TAP harness such as prove, you get:
> >
> > foo.t .. Failed 1/3 subtests
> >
> > Test Summary Report
> > -------------------
> > foo.t (Wstat: 0 Tests: 2 Failed: 0)
> > Parse errors: Bad plan. You planned 3 tests but ran 2.
> > Files=1, Tests=2, 0 wallclock secs ( 0.01 usr + 0.00 sys = 0.01 CPU)
> > Result: FAIL
> >
> > Why do we bother generaing a TAP output if we are not going to take advantage
> > of it?
>
> (As the person who added the TAP output to git.git)
>
> Yeah, we could do the "plan ahead", but it would mean that tests would
> need to pre-declare the number of tests they have.
I'm not advocating for planning ahead.
In the case of the shell testing framework it makes sense to not plan
ahead and just print the test plan at the end with test_done(), as
that's very convenient.
But $subject is a new proposal for C unit tests with a test plan
specified ahead, therefore if the program exits before all the tests
have run, that should be obvious from the output. I'm not saying that is
good or desirable, merely that it is the case.
I personaly don't even think C is the way to write unit tests.
> In the Perl world that's the usual pattern, but as it involves having a:
>
> plan tests => 123;
>
> At the top of the file that's guaranteed to give you merge conflicts if
> two topics add/remove tests in different parts of the file.
>
> It also doesn't work well in cases where the number of tests is hard to
> determine in advance, i.e. when they're determined programatically.
>
> I don't think there's any practical downside to using the "test_done"
> pattern to print a plan at the end as far as missing tests go.
>
> There *is* a minor practical downside to it though, which is that we'll
> get output like "25/?" or whatever, but not "25/100", as we don't know
> yet that we've got a total of 100 tests.
>
> But I think that's a minor drawback, and really only matters if you're
> eyeballing the prove(1) output of a very slow test as it scrolls by.
>
> I think on balance the "plan at the end" approach we're using now is
> much better, and would also be better in a future (or hypothetical)
> pure-C test framework.
>
> Well, there are ways to avoid the painful conflicts, e.g. by mandating
> that all tests are driven by callbacks in an array or something, so then
> you won't get merge conflicts on the "plan" line, as it'll just be "the
> number of tests is the number of items in this array".
>
> But such a thing is painful to mandate, and has its own drawbacks,
> i.e. not being able to do a "test" at anything less than a function
> callback level of granularity.
If you are using an old-school language that doesn't have support for
anonymous functions, then yes, that is pretty much your only nice
maintenable option.
But the list of languages that don't have support for that is rather
small, it's pretty much limited to shell, C, Fortran and COBOL. Even
Perl has support for anonymous functions.
In all modern languages you can write test cases using anonymous
functions such as (JavaScript's ava [1]):
test('test case', t => {
t.is('actual', 'expected');
});
This can be done in Perl as well (see Test::Functional [2]).
So at the end of the script it is known how many test cases there are
*before* running any of them.
This is precisely the approach I followed in my new testing framework,
which I sent as a proposal to the ml [3].
require 'testox'
test('one') { ok(true) }
test('two') { ok(true) }
test('three') { ok(true) }
test('four') { ok(true) }
puts '# num tests: %s' % TestOx.tests.length
No test case is run until the script ends in at_exit:
# num tests: 4
1..4
ok 1 - one
ok 2 - two
ok 3 - three
ok 4 - four
There's no need to specify the number of test cases.
I don't see why we insist in using ancient languages like C and shell
for things that neither C nor shell are good at. Other languages exist
in 2023. Other languages exist for a reason.
Cheers.
[1] https://github.com/avajs/ava
[2] https://metacpan.org/pod/Test::Functional
[3] https://lore.kernel.org/git/20230502041113.103385-1-felipe.contreras@gmail.com/T/#t
--
Felipe Contreras
next prev parent reply other threads:[~2023-05-02 18:11 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-04-27 17:50 [RFC PATCH 0/2] add an external testing library for unit tests Calvin Wan
2023-04-27 17:50 ` [RFC PATCH 1/2] Add C TAP harness Calvin Wan
2023-04-27 18:29 ` SZEDER Gábor
2023-04-27 18:38 ` Calvin Wan
2023-04-27 20:15 ` Phillip Wood
2023-04-28 16:31 ` Calvin Wan
2023-05-02 15:46 ` Felipe Contreras
2023-05-10 15:46 ` Phillip Wood
2023-05-11 23:16 ` Glen Choo
2023-05-18 10:04 ` Phillip Wood
2023-06-21 15:57 ` Linus Arver
2023-06-26 13:15 ` Phillip Wood
2023-06-28 21:17 ` Linus Arver
2023-06-29 5:52 ` Oswald Buddenhagen
2023-06-30 9:48 ` Phillip Wood
2023-05-02 15:54 ` Felipe Contreras
2023-05-02 16:39 ` Ævar Arnfjörð Bjarmason
2023-05-02 18:11 ` Felipe Contreras [this message]
2023-05-02 16:34 ` Ævar Arnfjörð Bjarmason
2023-05-10 8:18 ` Phillip Wood
2023-04-27 17:50 ` [RFC PATCH 2/2] unit test: add basic example Calvin Wan
2023-04-27 18:47 ` Junio C Hamano
2023-05-02 15:58 ` Felipe Contreras
2023-04-27 18:39 ` [RFC PATCH 0/2] add an external testing library for unit tests Junio C Hamano
2023-04-27 18:46 ` Calvin Wan
2023-04-27 21:35 ` brian m. carlson
2023-05-02 4:18 ` Felipe Contreras
2023-05-02 13:52 ` Ævar Arnfjörð Bjarmason
2023-05-02 15:28 ` Felipe Contreras
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6451523ec8b47_1ba2d29427@chronos.notmuch \
--to=felipe.contreras@gmail.com \
--cc=avarab@gmail.com \
--cc=calvinwan@google.com \
--cc=git@vger.kernel.org \
--cc=phillip.wood123@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).