From: Josh Steadmon <steadmon@google.com>
To: git@vger.kernel.org
Cc: calvinwan@gmail.com, szeder.dev@gmail.com,
phillip.wood123@gmail.com, chooglen@google.com, avarab@gmail.com,
gitster@pobox.com, sandals@crustytoothpaste.net
Subject: [RFC PATCH v3 1/1] unit tests: Add a project plan document
Date: Fri, 9 Jun 2023 16:25:30 -0700 [thread overview]
Message-ID: <8afdb215d7e10ca16a2ce8226b4127b3d8a2d971.1686352386.git.steadmon@google.com> (raw)
In-Reply-To: <20230517-unit-tests-v2-v2-0-21b5b60f4b32@google.com>
In our current testing environment, we spend a significant amount of
effort crafting end-to-end tests for error conditions that could easily
be captured by unit tests (or we simply forgo some hard-to-setup and
rare error conditions).Describe what we hope to accomplish by
implementing unit tests, and explain some open questions and milestones.
Discuss desired features for test frameworks/harnesses, and provide a
preliminary comparison of several different frameworks.
Signed-off-by: Josh Steadmon <steadmon@google.com>
Coauthored-by: Calvin Wan <calvinwan@google.com>
---
Documentation/Makefile | 1 +
Documentation/technical/unit-tests.txt | 141 +++++++++++++++++++++++++
2 files changed, 142 insertions(+)
create mode 100644 Documentation/technical/unit-tests.txt
diff --git a/Documentation/Makefile b/Documentation/Makefile
index b629176d7d..3f2383a12c 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -122,6 +122,7 @@ TECH_DOCS += technical/scalar
TECH_DOCS += technical/send-pack-pipeline
TECH_DOCS += technical/shallow
TECH_DOCS += technical/trivial-merge
+TECH_DOCS += technical/unit-tests
SP_ARTICLES += $(TECH_DOCS)
SP_ARTICLES += technical/api-index
diff --git a/Documentation/technical/unit-tests.txt b/Documentation/technical/unit-tests.txt
new file mode 100644
index 0000000000..dac8062a43
--- /dev/null
+++ b/Documentation/technical/unit-tests.txt
@@ -0,0 +1,141 @@
+= Unit Testing
+
+In our current testing environment, we spend a significant amount of effort
+crafting end-to-end tests for error conditions that could easily be captured by
+unit tests (or we simply forgo some hard-to-setup and rare error conditions).
+Unit tests additionally provide stability to the codebase and can simplify
+debugging through isolation. Writing unit tests in pure C, rather than with our
+current shell/test-tool helper setup, simplifies test setup, simplifies passing
+data around (no shell-isms required), and reduces testing runtime by not
+spawning a separate process for every test invocation.
+
+We believe that a large body of unit tests, living alongside the existing test
+suite, will improve code quality for the Git project.
+
+== Definitions
+
+For the purposes of this document, we'll use *test framework* to refer to
+projects that support writing test cases and running tests within the context
+of a single executable. *Test harness* will refer to projects that manage
+running multiple executables (each of which may contain multiple test cases) and
+aggregating their results.
+
+In reality, these terms are not strictly defined, and many of the projects
+discussed below contain features from both categories.
+
+
+== Choosing a framework & harness
+
+=== Desired features
+
+==== TAP support
+
+The https://testanything.org/[Test Anything Protocol] is a text-based interface
+that allows tests to communicate with a test harness. It is already used by
+Git's integration test suite. Supporting TAP output is a mandatory feature for
+any prospective test framework.
+
+==== Diagnostic output
+
+When a test case fails, the framework must generate enough diagnostic output to
+help developers find the appropriate test case in source code in order to debug
+the failure.
+
+==== Parallel execution
+
+Ideally, we will build up a significant collection of unit tests cases, most
+likely split across multiple executables. It will be necessary to run these
+tests in parallel to enable fast develop-test-debug cycles.
+
+==== Vendorable or ubiquitous
+
+If possible, we want to avoid forcing Git developers to install new tools just
+to run unit tests. So any prospective frameworks and harnesses must either be
+vendorable (meaning, we can copy their source directly into Git's repository),
+or so ubiquitous that it is reasonable to expect that most developers will have
+the tools installed already.
+
+==== Maintainable / extensible
+
+It is unlikely that any pre-existing project perfectly fits our needs, so any
+project we select will need to be actively maintained and open to accepting
+changes. Alternatively, assuming we are vendoring the source into our repo, it
+must be simple enough that Git developers can feel comfortable making changes as
+needed to our version.
+
+==== Major platform support
+
+At a bare minimum, unit-testing must work on Linux, MacOS, and Windows.
+
+==== Lazy test planning
+
+TAP supports the notion of _test plans_, which communicate which test cases are
+expected to run, or which tests actually ran. This allows test harnesses to
+detect if the TAP output has been truncated, or if some tests were skipped due
+to errors or bugs.
+
+The test framework should handle creating plans at runtime, rather than
+requiring test developers to manually create plans, which leads to both human-
+and merge-errors.
+
+==== Skippable tests
+
+Test authors may wish to skip certain test cases based on runtime circumstances,
+so the framework should support this.
+
+==== Test scheduling / re-running
+
+The test harness scheduling should be configurable so that e.g. developers can
+choose to run slow tests first, or to run only tests that failed in a previous
+run.
+
+==== Mock support
+
+Unit test authors may wish to test code that interacts with objects that may be
+inconvenient to handle in a test (e.g. interacting with a network service).
+Mocking allows test authors to provide a fake implementation of these objects
+for more convenient tests.
+
+==== Signal & exception handling
+
+The test framework must fail gracefully when test cases are themselves buggy or
+when they are interrupted by signals during runtime.
+
+==== Coverage reports
+
+It may be convenient to generate coverage reports when running unit tests
+(although it may be possible to accomplish this regardless of test framework /
+harness support).
+
+
+=== Comparison
+
+[format="csv",options="header",width="75%"]
+|=====
+Framework,"TAP support","Diagnostic output","Parallel execution","Vendorable / ubiquitous","Maintainable / extensible","Major platform support","Lazy test planning","Runtime- skippable tests","Scheduling / re-running",Mocks,"Signal & exception handling","Coverage reports"
+https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[Custom Git impl.],[lime-background]#True#,[lime-background]#True#,?,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,?,?,[red-background]#False#,?,?
+https://cmocka.org/[cmocka],[lime-background]#True#,[lime-background]#True#,?,[red-background]#False#,[yellow-background]#Partial#,[yellow-background]#Partial#,[yellow-background]#Partial#,?,?,[lime-background]#True#,?,?
+https://libcheck.github.io/check/[Check],[lime-background]#True#,[lime-background]#True#,?,[red-background]#False#,[yellow-background]#Partial#,[lime-background]#True#,[yellow-background]#Partial#,?,?,[red-background]#False#,?,?
+https://github.com/rra/c-tap-harness/[C TAP],[lime-background]#True#,[red-background]#False#,?,[lime-background]#True#,[yellow-background]#Partial#,[yellow-background]#Partial#,[yellow-background]#Partial#,?,?,[red-background]#False#,?,?
+https://github.com/silentbicycle/greatest[Greatest],[yellow-background]#Partial#,?,?,[lime-background]#True#,[yellow-background]#Partial#,?,[yellow-background]#Partial#,?,?,[red-background]#False#,?,?
+https://github.com/Snaipe/Criterion[Criterion],[lime-background]#True#,?,?,[red-background]#False#,?,[lime-background]#True#,?,?,?,[red-background]#False#,?,?
+https://github.com/zorgnax/libtap[libtap],[lime-background]#True#,?,?,?,?,?,?,?,?,?,?,?
+https://nemequ.github.io/munit/[µnit],?,?,?,?,?,?,?,?,?,?,?,?
+https://github.com/google/cmockery[cmockery],?,?,?,?,?,?,?,?,?,[lime-background]#True#,?,?
+https://github.com/lpabon/cmockery2[cmockery2],?,?,?,?,?,?,?,?,?,[lime-background]#True#,?,?
+https://github.com/ThrowTheSwitch/Unity[Unity],?,?,?,?,?,?,?,?,?,?,?,?
+https://github.com/siu/minunit[minunit],?,?,?,?,?,?,?,?,?,?,?,?
+https://cunit.sourceforge.net/[CUnit],?,?,?,?,?,?,?,?,?,?,?,?
+https://www.kindahl.net/mytap/doc/index.html[MyTAP],[lime-background]#True#,?,?,?,?,?,?,?,?,?,?,?
+|=====
+
+== Milestones
+
+* Settle on final framework
+* Add useful tests of library-like code
+* Integrate with Makefile
+* Integrate with CI
+* Integrate with
+ https://lore.kernel.org/git/20230502211454.1673000-1-calvinwan@google.com/[stdlib
+ work]
+* Run alongside regular `make test` target
--
2.41.0.162.gfafddb0af9-goog
next prev parent reply other threads:[~2023-06-09 23:25 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-17 23:56 [PATCH RFC v2 0/4] Add an external testing library for unit tests steadmon
2023-05-17 23:56 ` [PATCH RFC v2 1/4] common-main: split common_exit() into a new file steadmon
2023-05-18 17:17 ` Junio C Hamano
2023-07-14 23:38 ` Splitting common-main (Was: Re: [PATCH RFC v2 1/4] common-main: split common_exit() into a new file) Josh Steadmon
2023-07-15 0:34 ` Splitting common-main Junio C Hamano
2023-08-14 13:09 ` Splitting common-main (Was: Re: [PATCH RFC v2 1/4] common-main: split common_exit() into a new file) Jeff Hostetler
2023-05-17 23:56 ` [PATCH RFC v2 2/4] unit tests: Add a project plan document steadmon
2023-05-18 13:13 ` Phillip Wood
2023-05-18 20:15 ` Glen Choo
2023-05-24 17:40 ` Josh Steadmon
2023-06-01 9:19 ` Phillip Wood
2023-05-17 23:56 ` [PATCH RFC v2 3/4] Add C TAP harness steadmon
2023-05-18 13:15 ` Phillip Wood
2023-05-18 20:50 ` Josh Steadmon
2023-05-17 23:56 ` [PATCH RFC v2 4/4] unit test: add basic example and build rules steadmon
2023-05-18 13:32 ` Phillip Wood
2023-06-09 23:25 ` [RFC PATCH v3 0/1] Add a project document for adding unit tests Josh Steadmon
2023-06-09 23:25 ` Josh Steadmon [this message]
2023-06-13 22:30 ` [RFC PATCH v3 1/1] unit tests: Add a project plan document Junio C Hamano
2023-06-30 22:18 ` Josh Steadmon
2023-06-29 19:42 ` Linus Arver
2023-06-29 20:48 ` Josh Steadmon
2023-06-30 19:31 ` Linus Arver
2023-07-06 18:24 ` Glen Choo
2023-07-06 19:02 ` Junio C Hamano
2023-07-06 22:48 ` Glen Choo
2023-06-30 21:33 ` Josh Steadmon
2023-06-29 21:21 ` Junio C Hamano
2023-06-30 0:11 ` Linus Arver
2023-06-30 14:07 ` Phillip Wood
2023-06-30 18:47 ` K Wan
2023-06-30 22:35 ` Josh Steadmon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=8afdb215d7e10ca16a2ce8226b4127b3d8a2d971.1686352386.git.steadmon@google.com \
--to=steadmon@google.com \
--cc=avarab@gmail.com \
--cc=calvinwan@gmail.com \
--cc=chooglen@google.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=phillip.wood123@gmail.com \
--cc=sandals@crustytoothpaste.net \
--cc=szeder.dev@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).