Git Mailing List Archive mirror
 help / color / mirror / Atom feed
From: Glen Choo <chooglen@google.com>
To: steadmon@google.com, git@vger.kernel.org
Cc: Josh Steadmon <steadmon@google.com>,
	calvinwan@gmail.com, szeder.dev@gmail.com,
	phillip.wood123@gmail.com, avarab@gmail.com, gitster@pobox.com,
	sandals@crustytoothpaste.net
Subject: Re: [PATCH RFC v2 2/4] unit tests: Add a project plan document
Date: Thu, 18 May 2023 13:15:26 -0700	[thread overview]
Message-ID: <kl6lzg61xuox.fsf@chooglen-macbookpro.roam.corp.google.com> (raw)
In-Reply-To: <20230517-unit-tests-v2-v2-2-21b5b60f4b32@google.com>

steadmon@google.com writes:

> Describe what we hope to accomplish by implementing unit tests, and
> explain some open questions and milestones.

Thanks! I found this very helpful.

> diff --git a/Documentation/technical/unit-tests.txt b/Documentation/technical/unit-tests.txt
> new file mode 100644
> index 0000000000..7c575e6ef7
> --- /dev/null
> +++ b/Documentation/technical/unit-tests.txt
> @@ -0,0 +1,47 @@
> += Unit Testing
> +
> +In our current testing environment, we spend a significant amount of effort
> +crafting end-to-end tests for error conditions that could easily be captured by
> +unit tests (or we simply forgo some hard-to-setup and rare error conditions).
> +Unit tests additionally provide stability to the codebase and can simplify
> +debugging through isolation. Writing unit tests in pure C, rather than with our
> +current shell/test-tool helper setup, simplifies test setup, simplifies passing
> +data around (no shell-isms required), and reduces testing runtime by not
> +spawning a separate process for every test invocation.

The stated goals make sense to me, and I believe they are worth
restating. I believe this is mostly taken from Calvin's v1 cover letter

  https://lore.kernel.org/git/20230427175007.902278-1-calvinwan@google.com

so perhaps he should receive some writing credit in a commit trailer
(Helped-by?).

> +== Open questions
> +
> +=== TAP harness
> +
> +We'll need to decide on a TAP harness. The C TAP library is easy to integrate,
> +but has a few drawbacks:
> +* (copy objections from lore thread)
> +* We may need to carry local patches against C TAP. We'll need to decide how to
> +  manage these. We could vendor the code in and modify them directly, or use a
> +  submodule (but then we'll need to decide on where to host the submodule with
> +  our patches on top).
> +
> +Phillip Wood has also proposed a new implementation of a TAP harness (linked
> +above). While it hasn't been thoroughly reviewed yet, it looks to support a few
> +nice features that C TAP does not, e.g. lazy test plans and skippable tests.

A third option would be to pick another, more mature third party testing
library. As I mentioned in

  https://lore.kernel.org/git/kl6lpm76zcg7.fsf@chooglen-macbookpro.roam.corp.google.com

my primary concern is the maintainability and extensibility of a third
party library that (no offense to the original author) is not used very
widely, is relatively underdocumented, is missing features that we want,
and whose maintenance policy is relatively unknown to us. I'm not
opposed to taking in a third party testing framework, but we need to be
sure that we can rely on it instead of being something that requires
active upkeep.

I don't what sorts of testing libraries exist for C or how widely they
are used, but a quick web search gives some candidates that seem like
plausible alternatives to C TAP Harness:

- cmocka https://cmocka.org/ supports TAP, assert macros and mocking,
  and is used by other projects (their website indicates Samba, OpenVPN,
  etc). The documentation is a bit lacking IMO. It's apparently a fork
  of cmockery, which I'm not familiar with.

- Check https://libcheck.github.io/check/ supports TAP and has
  relatively good documentation, though the last release seems to have
  been 3 years ago.

- µnit https://nemequ.github.io/munit/ has a shiny website with nice
  docs (and a handy list of other unit test frameworks we can look at).
  The last release also seems to be ~3 years ago. Not sure if this
  supports TAP.

For flexibility, I also think it's reasonable for us to roll our own
testing library. I think it is perfectly fine for our unit test
framework to be suboptimal at the beginning, but owning the code makes
it relatively easy to fix bugs and extend it to our liking.

  parent reply	other threads:[~2023-05-18 20:15 UTC|newest]

Thread overview: 32+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-05-17 23:56 [PATCH RFC v2 0/4] Add an external testing library for unit tests steadmon
2023-05-17 23:56 ` [PATCH RFC v2 1/4] common-main: split common_exit() into a new file steadmon
2023-05-18 17:17   ` Junio C Hamano
2023-07-14 23:38     ` Splitting common-main (Was: Re: [PATCH RFC v2 1/4] common-main: split common_exit() into a new file) Josh Steadmon
2023-07-15  0:34       ` Splitting common-main Junio C Hamano
2023-08-14 13:09       ` Splitting common-main (Was: Re: [PATCH RFC v2 1/4] common-main: split common_exit() into a new file) Jeff Hostetler
2023-05-17 23:56 ` [PATCH RFC v2 2/4] unit tests: Add a project plan document steadmon
2023-05-18 13:13   ` Phillip Wood
2023-05-18 20:15   ` Glen Choo [this message]
2023-05-24 17:40     ` Josh Steadmon
2023-06-01  9:19     ` Phillip Wood
2023-05-17 23:56 ` [PATCH RFC v2 3/4] Add C TAP harness steadmon
2023-05-18 13:15   ` Phillip Wood
2023-05-18 20:50     ` Josh Steadmon
2023-05-17 23:56 ` [PATCH RFC v2 4/4] unit test: add basic example and build rules steadmon
2023-05-18 13:32   ` Phillip Wood
2023-06-09 23:25 ` [RFC PATCH v3 0/1] Add a project document for adding unit tests Josh Steadmon
2023-06-09 23:25 ` [RFC PATCH v3 1/1] unit tests: Add a project plan document Josh Steadmon
2023-06-13 22:30   ` Junio C Hamano
2023-06-30 22:18     ` Josh Steadmon
2023-06-29 19:42   ` Linus Arver
2023-06-29 20:48     ` Josh Steadmon
2023-06-30 19:31       ` Linus Arver
2023-07-06 18:24         ` Glen Choo
2023-07-06 19:02           ` Junio C Hamano
2023-07-06 22:48             ` Glen Choo
2023-06-30 21:33       ` Josh Steadmon
2023-06-29 21:21     ` Junio C Hamano
2023-06-30  0:11       ` Linus Arver
2023-06-30 14:07   ` Phillip Wood
2023-06-30 18:47     ` K Wan
2023-06-30 22:35     ` Josh Steadmon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=kl6lzg61xuox.fsf@chooglen-macbookpro.roam.corp.google.com \
    --to=chooglen@google.com \
    --cc=avarab@gmail.com \
    --cc=calvinwan@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=phillip.wood123@gmail.com \
    --cc=sandals@crustytoothpaste.net \
    --cc=steadmon@google.com \
    --cc=szeder.dev@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).