From: K Wan <calvinwan@gmail.com>
To: phillip.wood@dunelm.org.uk
Cc: Josh Steadmon <steadmon@google.com>,
git@vger.kernel.org, szeder.dev@gmail.com, chooglen@google.com,
avarab@gmail.com, gitster@pobox.com,
sandals@crustytoothpaste.net
Subject: Re: [RFC PATCH v3 1/1] unit tests: Add a project plan document
Date: Fri, 30 Jun 2023 10:47:06 -0800 [thread overview]
Message-ID: <667C1DBB-CF32-4BAA-8887-2B343CD9EA12@gmail.com> (raw)
In-Reply-To: <a99a5134-3bac-64d4-b4e7-f02e8578090a@gmail.com>
Hi
Pleasure exclude my email from this discussion.
Thank you
> On Jun 30, 2023, at 6:08 AM, Phillip Wood <phillip.wood123@gmail.com> wrote:
>
> Hi Josh
>
> Thanks for putting this together, I think it is really helpful to have a comparison of the various options. Sorry for the slow reply, I was off the list for a couple of weeks.
>
>> On 10/06/2023 00:25, Josh Steadmon wrote:
>> diff --git a/Documentation/technical/unit-tests.txt b/Documentation/technical/unit-tests.txt
>> new file mode 100644
>> index 0000000000..dac8062a43
>> --- /dev/null
>> +++ b/Documentation/technical/unit-tests.txt
>> @@ -0,0 +1,141 @@
>> += Unit Testing
>
> I've deleted the sections I agree with to avoid quoting parts that are not relevant to my comments.
>
>> +== Definitions
>> +
>> +For the purposes of this document, we'll use *test framework* to refer to
>> +projects that support writing test cases and running tests within the context
>> +of a single executable. *Test harness* will refer to projects that manage
>> +running multiple executables (each of which may contain multiple test cases) and
>> +aggregating their results.
>
> Thanks for adding this, it is really helpful to have definitions for what we mean by "test framework" and "test harness" within the git project. It might be worth mentioning somewhere that we already use prove as a test harness when running our integration tests.
>
>> +In reality, these terms are not strictly defined, and many of the projects
>> +discussed below contain features from both categories.
>
>> +
>> +== Choosing a framework & harness
>> +
>> +=== Desired features
>> +
>> [...]
>> +==== Parallel execution
>> +
>> +Ideally, we will build up a significant collection of unit tests cases, most
>> +likely split across multiple executables. It will be necessary to run these
>> +tests in parallel to enable fast develop-test-debug cycles.
>
> This is a good point, though I think it is really a property of the harness rather than the framework so we might want to indicate in the table whether a framework provides parallelism itself or relies on the harness providing it.
>
> > [...]
>> +==== Major platform support
>> +
>> +At a bare minimum, unit-testing must work on Linux, MacOS, and Windows.
>
> I think we'd want to be able to run unit tests on *BSD and NonStop as well, especially as I think some of the platform dependent code probably lends itself to being unit tested. I suspect a framework that covers Linux and MacOS would probably run on those platforms as well (I don't think NonStop has complete POSIX support but it is hard to imagine a test framework doing anything very exotic)
>
>> [...]
>> +==== Mock support
>> +
>> +Unit test authors may wish to test code that interacts with objects that may be
>> +inconvenient to handle in a test (e.g. interacting with a network service).
>> +Mocking allows test authors to provide a fake implementation of these objects
>> +for more convenient tests.
>
> Do we have any idea what sort of thing we're likely to want to mock and what we want that support to look like?
>
>> +==== Signal & exception handling
>> +
>> +The test framework must fail gracefully when test cases are themselves buggy or
>> +when they are interrupted by signals during runtime.
>
> I had assumed that it would be enough for the test harness to detect if a test executable was killed by a signal or exited early due to a bug in the test script. That requires the framework to have robust support for lazy test plans but I'm not sure that we need it to catch and recover from things like SIGSEGV.
>
>> +==== Coverage reports
>> +
>> +It may be convenient to generate coverage reports when running unit tests
>> +(although it may be possible to accomplish this regardless of test framework /
>> +harness support).
>
> I agree this would be useful, though perhaps we should build it on our existing gcov usage.
>
> Related to this do we want timing reports from the harness or the framework?
>
>> +
>> +=== Comparison
>> +
>> +[format="csv",options="header",width="75%"]
>> +|=====
>> +Framework,"TAP support","Diagnostic output","Parallel execution","Vendorable / ubiquitous","Maintainable / extensible","Major platform support","Lazy test planning","Runtime- skippable tests","Scheduling / re-running",Mocks,"Signal & exception handling","Coverage reports"
>> +https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[Custom Git impl.],[lime-background]#True#,[lime-background]#True#,?,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,?,?,[red-background]#False#,?,?
>> +https://cmocka.org/[cmocka],[lime-background]#True#,[lime-background]#True#,?,[red-background]#False#,[yellow-background]#Partial#,[yellow-background]#Partial#,[yellow-background]#Partial#,?,?,[lime-background]#True#,?,?
>> +https://libcheck.github.io/check/[Check],[lime-background]#True#,[lime-background]#True#,?,[red-background]#False#,[yellow-background]#Partial#,[lime-background]#True#,[yellow-background]#Partial#,?,?,[red-background]#False#,?,?
>> +https://github.com/rra/c-tap-harness/[C TAP],[lime-background]#True#,[red-background]#False#,?,[lime-background]#True#,[yellow-background]#Partial#,[yellow-background]#Partial#,[yellow-background]#Partial#,?,?,[red-background]#False#,?,?
>> +https://github.com/silentbicycle/greatest[Greatest],[yellow-background]#Partial#,?,?,[lime-background]#True#,[yellow-background]#Partial#,?,[yellow-background]#Partial#,?,?,[red-background]#False#,?,?
>> +https://github.com/Snaipe/Criterion[Criterion],[lime-background]#True#,?,?,[red-background]#False#,?,[lime-background]#True#,?,?,?,[red-background]#False#,?,?
>> +https://github.com/zorgnax/libtap[libtap],[lime-background]#True#,?,?,?,?,?,?,?,?,?,?,?
>> +https://nemequ.github.io/munit/[µnit],?,?,?,?,?,?,?,?,?,?,?,?
>> +https://github.com/google/cmockery[cmockery],?,?,?,?,?,?,?,?,?,[lime-background]#True#,?,?
>> +https://github.com/lpabon/cmockery2[cmockery2],?,?,?,?,?,?,?,?,?,[lime-background]#True#,?,?
>> +https://github.com/ThrowTheSwitch/Unity[Unity],?,?,?,?,?,?,?,?,?,?,?,?
>> +https://github.com/siu/minunit[minunit],?,?,?,?,?,?,?,?,?,?,?,?
>> +https://cunit.sourceforge.net/[CUnit],?,?,?,?,?,?,?,?,?,?,?,?
>> +https://www.kindahl.net/mytap/doc/index.html[MyTAP],[lime-background]#True#,?,?,?,?,?,?,?,?,?,?,?
>> +|=====
>
> Thanks for going through these projects, hopefully we can use this information to make a decision on a framework soon.
>
> Best Wishes
>
> Phillip
next prev parent reply other threads:[~2023-06-30 18:47 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-17 23:56 [PATCH RFC v2 0/4] Add an external testing library for unit tests steadmon
2023-05-17 23:56 ` [PATCH RFC v2 1/4] common-main: split common_exit() into a new file steadmon
2023-05-18 17:17 ` Junio C Hamano
2023-07-14 23:38 ` Splitting common-main (Was: Re: [PATCH RFC v2 1/4] common-main: split common_exit() into a new file) Josh Steadmon
2023-07-15 0:34 ` Splitting common-main Junio C Hamano
2023-08-14 13:09 ` Splitting common-main (Was: Re: [PATCH RFC v2 1/4] common-main: split common_exit() into a new file) Jeff Hostetler
2023-05-17 23:56 ` [PATCH RFC v2 2/4] unit tests: Add a project plan document steadmon
2023-05-18 13:13 ` Phillip Wood
2023-05-18 20:15 ` Glen Choo
2023-05-24 17:40 ` Josh Steadmon
2023-06-01 9:19 ` Phillip Wood
2023-05-17 23:56 ` [PATCH RFC v2 3/4] Add C TAP harness steadmon
2023-05-18 13:15 ` Phillip Wood
2023-05-18 20:50 ` Josh Steadmon
2023-05-17 23:56 ` [PATCH RFC v2 4/4] unit test: add basic example and build rules steadmon
2023-05-18 13:32 ` Phillip Wood
2023-06-09 23:25 ` [RFC PATCH v3 0/1] Add a project document for adding unit tests Josh Steadmon
2023-06-09 23:25 ` [RFC PATCH v3 1/1] unit tests: Add a project plan document Josh Steadmon
2023-06-13 22:30 ` Junio C Hamano
2023-06-30 22:18 ` Josh Steadmon
2023-06-29 19:42 ` Linus Arver
2023-06-29 20:48 ` Josh Steadmon
2023-06-30 19:31 ` Linus Arver
2023-07-06 18:24 ` Glen Choo
2023-07-06 19:02 ` Junio C Hamano
2023-07-06 22:48 ` Glen Choo
2023-06-30 21:33 ` Josh Steadmon
2023-06-29 21:21 ` Junio C Hamano
2023-06-30 0:11 ` Linus Arver
2023-06-30 14:07 ` Phillip Wood
2023-06-30 18:47 ` K Wan [this message]
2023-06-30 22:35 ` Josh Steadmon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=667C1DBB-CF32-4BAA-8887-2B343CD9EA12@gmail.com \
--to=calvinwan@gmail.com \
--cc=avarab@gmail.com \
--cc=chooglen@google.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=phillip.wood@dunelm.org.uk \
--cc=sandals@crustytoothpaste.net \
--cc=steadmon@google.com \
--cc=szeder.dev@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).