From: Josh Steadmon <steadmon@google.com>
To: phillip.wood@dunelm.org.uk
Cc: git@vger.kernel.org, calvinwan@google.com, szeder.dev@gmail.com,
chooglen@google.com, avarab@gmail.com, gitster@pobox.com,
sandals@crustytoothpaste.net
Subject: Re: [RFC PATCH v3 1/1] unit tests: Add a project plan document
Date: Fri, 30 Jun 2023 15:35:01 -0700 [thread overview]
Message-ID: <ZJ9Ylb+/UOtRNKKy@google.com> (raw)
In-Reply-To: <a99a5134-3bac-64d4-b4e7-f02e8578090a@gmail.com>
On 2023.06.30 15:07, Phillip Wood wrote:
> Hi Josh
>
> Thanks for putting this together, I think it is really helpful to have a
> comparison of the various options. Sorry for the slow reply, I was off the
> list for a couple of weeks.
Thank you for the review! Unfortunately I didn't see it in time for any
of these to make it into v4, but I'll keep them all as TODOs.
> On 10/06/2023 00:25, Josh Steadmon wrote:
> > diff --git a/Documentation/technical/unit-tests.txt b/Documentation/technical/unit-tests.txt
> > new file mode 100644
> > index 0000000000..dac8062a43
> > --- /dev/null
> > +++ b/Documentation/technical/unit-tests.txt
> > +== Definitions
> > +
> > +For the purposes of this document, we'll use *test framework* to refer to
> > +projects that support writing test cases and running tests within the context
> > +of a single executable. *Test harness* will refer to projects that manage
> > +running multiple executables (each of which may contain multiple test cases) and
> > +aggregating their results.
>
> Thanks for adding this, it is really helpful to have definitions for what we
> mean by "test framework" and "test harness" within the git project. It might
> be worth mentioning somewhere that we already use prove as a test harness
> when running our integration tests.
Yeah, I'll try to clarify this, probably not in V4 though. I'll note it
as a remaining TODO.
> > +==== Parallel execution
> > +
> > +Ideally, we will build up a significant collection of unit tests cases, most
> > +likely split across multiple executables. It will be necessary to run these
> > +tests in parallel to enable fast develop-test-debug cycles.
>
> This is a good point, though I think it is really a property of the harness
> rather than the framework so we might want to indicate in the table whether
> a framework provides parallelism itself or relies on the harness providing
> it.
Same here.
> > [...]
> > +==== Major platform support
> > +
> > +At a bare minimum, unit-testing must work on Linux, MacOS, and Windows.
>
> I think we'd want to be able to run unit tests on *BSD and NonStop as well,
> especially as I think some of the platform dependent code probably lends
> itself to being unit tested. I suspect a framework that covers Linux and
> MacOS would probably run on those platforms as well (I don't think NonStop
> has complete POSIX support but it is hard to imagine a test framework doing
> anything very exotic)
Yes, unfortunately I don't have easy access to either of these, but I'll
try to figure out how to evaluate these.
> > [...]
> > +==== Mock support
> > +
> > +Unit test authors may wish to test code that interacts with objects that may be
> > +inconvenient to handle in a test (e.g. interacting with a network service).
> > +Mocking allows test authors to provide a fake implementation of these objects
> > +for more convenient tests.
>
> Do we have any idea what sort of thing we're likely to want to mock and what
> we want that support to look like?
Not at the moment. Another TODO for v5.
> > +==== Signal & exception handling
> > +
> > +The test framework must fail gracefully when test cases are themselves buggy or
> > +when they are interrupted by signals during runtime.
>
> I had assumed that it would be enough for the test harness to detect if a
> test executable was killed by a signal or exited early due to a bug in the
> test script. That requires the framework to have robust support for lazy
> test plans but I'm not sure that we need it to catch and recover from things
> like SIGSEGV.
I think as long as a SIGSEGV in the test code doesn't cause the entire
test run to crash, we'll be OK. Agreed that this is really more of a
harness feature.
> > +==== Coverage reports
> > +
> > +It may be convenient to generate coverage reports when running unit tests
> > +(although it may be possible to accomplish this regardless of test framework /
> > +harness support).
>
> I agree this would be useful, though perhaps we should build it on our
> existing gcov usage.
>
> Related to this do we want timing reports from the harness or the framework?
I'll add this as well in V5.
> > +
> > +=== Comparison
> > +
> > +[format="csv",options="header",width="75%"]
> > +|=====
> > +Framework,"TAP support","Diagnostic output","Parallel execution","Vendorable / ubiquitous","Maintainable / extensible","Major platform support","Lazy test planning","Runtime- skippable tests","Scheduling / re-running",Mocks,"Signal & exception handling","Coverage reports"
> > +https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[Custom Git impl.],[lime-background]#True#,[lime-background]#True#,?,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,?,?,[red-background]#False#,?,?
> > +https://cmocka.org/[cmocka],[lime-background]#True#,[lime-background]#True#,?,[red-background]#False#,[yellow-background]#Partial#,[yellow-background]#Partial#,[yellow-background]#Partial#,?,?,[lime-background]#True#,?,?
> > +https://libcheck.github.io/check/[Check],[lime-background]#True#,[lime-background]#True#,?,[red-background]#False#,[yellow-background]#Partial#,[lime-background]#True#,[yellow-background]#Partial#,?,?,[red-background]#False#,?,?
> > +https://github.com/rra/c-tap-harness/[C TAP],[lime-background]#True#,[red-background]#False#,?,[lime-background]#True#,[yellow-background]#Partial#,[yellow-background]#Partial#,[yellow-background]#Partial#,?,?,[red-background]#False#,?,?
> > +https://github.com/silentbicycle/greatest[Greatest],[yellow-background]#Partial#,?,?,[lime-background]#True#,[yellow-background]#Partial#,?,[yellow-background]#Partial#,?,?,[red-background]#False#,?,?
> > +https://github.com/Snaipe/Criterion[Criterion],[lime-background]#True#,?,?,[red-background]#False#,?,[lime-background]#True#,?,?,?,[red-background]#False#,?,?
> > +https://github.com/zorgnax/libtap[libtap],[lime-background]#True#,?,?,?,?,?,?,?,?,?,?,?
> > +https://nemequ.github.io/munit/[µnit],?,?,?,?,?,?,?,?,?,?,?,?
> > +https://github.com/google/cmockery[cmockery],?,?,?,?,?,?,?,?,?,[lime-background]#True#,?,?
> > +https://github.com/lpabon/cmockery2[cmockery2],?,?,?,?,?,?,?,?,?,[lime-background]#True#,?,?
> > +https://github.com/ThrowTheSwitch/Unity[Unity],?,?,?,?,?,?,?,?,?,?,?,?
> > +https://github.com/siu/minunit[minunit],?,?,?,?,?,?,?,?,?,?,?,?
> > +https://cunit.sourceforge.net/[CUnit],?,?,?,?,?,?,?,?,?,?,?,?
> > +https://www.kindahl.net/mytap/doc/index.html[MyTAP],[lime-background]#True#,?,?,?,?,?,?,?,?,?,?,?
> > +|=====
>
> Thanks for going through these projects, hopefully we can use this
> information to make a decision on a framework soon.
>
> Best Wishes
>
> Phillip
prev parent reply other threads:[~2023-06-30 22:35 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-17 23:56 [PATCH RFC v2 0/4] Add an external testing library for unit tests steadmon
2023-05-17 23:56 ` [PATCH RFC v2 1/4] common-main: split common_exit() into a new file steadmon
2023-05-18 17:17 ` Junio C Hamano
2023-07-14 23:38 ` Splitting common-main (Was: Re: [PATCH RFC v2 1/4] common-main: split common_exit() into a new file) Josh Steadmon
2023-07-15 0:34 ` Splitting common-main Junio C Hamano
2023-08-14 13:09 ` Splitting common-main (Was: Re: [PATCH RFC v2 1/4] common-main: split common_exit() into a new file) Jeff Hostetler
2023-05-17 23:56 ` [PATCH RFC v2 2/4] unit tests: Add a project plan document steadmon
2023-05-18 13:13 ` Phillip Wood
2023-05-18 20:15 ` Glen Choo
2023-05-24 17:40 ` Josh Steadmon
2023-06-01 9:19 ` Phillip Wood
2023-05-17 23:56 ` [PATCH RFC v2 3/4] Add C TAP harness steadmon
2023-05-18 13:15 ` Phillip Wood
2023-05-18 20:50 ` Josh Steadmon
2023-05-17 23:56 ` [PATCH RFC v2 4/4] unit test: add basic example and build rules steadmon
2023-05-18 13:32 ` Phillip Wood
2023-06-09 23:25 ` [RFC PATCH v3 0/1] Add a project document for adding unit tests Josh Steadmon
2023-06-09 23:25 ` [RFC PATCH v3 1/1] unit tests: Add a project plan document Josh Steadmon
2023-06-13 22:30 ` Junio C Hamano
2023-06-30 22:18 ` Josh Steadmon
2023-06-29 19:42 ` Linus Arver
2023-06-29 20:48 ` Josh Steadmon
2023-06-30 19:31 ` Linus Arver
2023-07-06 18:24 ` Glen Choo
2023-07-06 19:02 ` Junio C Hamano
2023-07-06 22:48 ` Glen Choo
2023-06-30 21:33 ` Josh Steadmon
2023-06-29 21:21 ` Junio C Hamano
2023-06-30 0:11 ` Linus Arver
2023-06-30 14:07 ` Phillip Wood
2023-06-30 18:47 ` K Wan
2023-06-30 22:35 ` Josh Steadmon [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZJ9Ylb+/UOtRNKKy@google.com \
--to=steadmon@google.com \
--cc=avarab@gmail.com \
--cc=calvinwan@google.com \
--cc=chooglen@google.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=phillip.wood@dunelm.org.uk \
--cc=sandals@crustytoothpaste.net \
--cc=szeder.dev@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).