Git Mailing List Archive mirror
 help / color / mirror / Atom feed
From: Josh Steadmon <>
Subject: [RFC PATCH v3 1/1] unit tests: Add a project plan document
Date: Fri,  9 Jun 2023 16:25:30 -0700	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

In our current testing environment, we spend a significant amount of
effort crafting end-to-end tests for error conditions that could easily
be captured by unit tests (or we simply forgo some hard-to-setup and
rare error conditions).Describe what we hope to accomplish by
implementing unit tests, and explain some open questions and milestones.
Discuss desired features for test frameworks/harnesses, and provide a
preliminary comparison of several different frameworks.

Signed-off-by: Josh Steadmon <>
Coauthored-by: Calvin Wan <>
 Documentation/Makefile                 |   1 +
 Documentation/technical/unit-tests.txt | 141 +++++++++++++++++++++++++
 2 files changed, 142 insertions(+)
 create mode 100644 Documentation/technical/unit-tests.txt

diff --git a/Documentation/Makefile b/Documentation/Makefile
index b629176d7d..3f2383a12c 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -122,6 +122,7 @@ TECH_DOCS += technical/scalar
 TECH_DOCS += technical/send-pack-pipeline
 TECH_DOCS += technical/shallow
 TECH_DOCS += technical/trivial-merge
+TECH_DOCS += technical/unit-tests
 SP_ARTICLES += technical/api-index
diff --git a/Documentation/technical/unit-tests.txt b/Documentation/technical/unit-tests.txt
new file mode 100644
index 0000000000..dac8062a43
--- /dev/null
+++ b/Documentation/technical/unit-tests.txt
@@ -0,0 +1,141 @@
+= Unit Testing
+In our current testing environment, we spend a significant amount of effort
+crafting end-to-end tests for error conditions that could easily be captured by
+unit tests (or we simply forgo some hard-to-setup and rare error conditions).
+Unit tests additionally provide stability to the codebase and can simplify
+debugging through isolation. Writing unit tests in pure C, rather than with our
+current shell/test-tool helper setup, simplifies test setup, simplifies passing
+data around (no shell-isms required), and reduces testing runtime by not
+spawning a separate process for every test invocation.
+We believe that a large body of unit tests, living alongside the existing test
+suite, will improve code quality for the Git project.
+== Definitions
+For the purposes of this document, we'll use *test framework* to refer to
+projects that support writing test cases and running tests within the context
+of a single executable. *Test harness* will refer to projects that manage
+running multiple executables (each of which may contain multiple test cases) and
+aggregating their results.
+In reality, these terms are not strictly defined, and many of the projects
+discussed below contain features from both categories.
+== Choosing a framework & harness
+=== Desired features
+==== TAP support
+The[Test Anything Protocol] is a text-based interface
+that allows tests to communicate with a test harness. It is already used by
+Git's integration test suite. Supporting TAP output is a mandatory feature for
+any prospective test framework.
+==== Diagnostic output
+When a test case fails, the framework must generate enough diagnostic output to
+help developers find the appropriate test case in source code in order to debug
+the failure.
+==== Parallel execution
+Ideally, we will build up a significant collection of unit tests cases, most
+likely split across multiple executables. It will be necessary to run these
+tests in parallel to enable fast develop-test-debug cycles.
+==== Vendorable or ubiquitous
+If possible, we want to avoid forcing Git developers to install new tools just
+to run unit tests. So any prospective frameworks and harnesses must either be
+vendorable (meaning, we can copy their source directly into Git's repository),
+or so ubiquitous that it is reasonable to expect that most developers will have
+the tools installed already.
+==== Maintainable / extensible
+It is unlikely that any pre-existing project perfectly fits our needs, so any
+project we select will need to be actively maintained and open to accepting
+changes. Alternatively, assuming we are vendoring the source into our repo, it
+must be simple enough that Git developers can feel comfortable making changes as
+needed to our version.
+==== Major platform support
+At a bare minimum, unit-testing must work on Linux, MacOS, and Windows.
+==== Lazy test planning
+TAP supports the notion of _test plans_, which communicate which test cases are
+expected to run, or which tests actually ran. This allows test harnesses to
+detect if the TAP output has been truncated, or if some tests were skipped due
+to errors or bugs.
+The test framework should handle creating plans at runtime, rather than
+requiring test developers to manually create plans, which leads to both human-
+and merge-errors.
+==== Skippable tests
+Test authors may wish to skip certain test cases based on runtime circumstances,
+so the framework should support this.
+==== Test scheduling / re-running
+The test harness scheduling should be configurable so that e.g. developers can
+choose to run slow tests first, or to run only tests that failed in a previous
+==== Mock support
+Unit test authors may wish to test code that interacts with objects that may be
+inconvenient to handle in a test (e.g. interacting with a network service).
+Mocking allows test authors to provide a fake implementation of these objects
+for more convenient tests.
+==== Signal & exception handling
+The test framework must fail gracefully when test cases are themselves buggy or
+when they are interrupted by signals during runtime.
+==== Coverage reports
+It may be convenient to generate coverage reports when running unit tests
+(although it may be possible to accomplish this regardless of test framework /
+harness support).
+=== Comparison
+Framework,"TAP support","Diagnostic output","Parallel execution","Vendorable / ubiquitous","Maintainable / extensible","Major platform support","Lazy test planning","Runtime- skippable tests","Scheduling / re-running",Mocks,"Signal & exception handling","Coverage reports"
+[Custom Git impl.],[lime-background]#True#,[lime-background]#True#,?,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,?,?,[red-background]#False#,?,?
+[C TAP],[lime-background]#True#,[red-background]#False#,?,[lime-background]#True#,[yellow-background]#Partial#,[yellow-background]#Partial#,[yellow-background]#Partial#,?,?,[red-background]#False#,?,?
+== Milestones
+* Settle on final framework
+* Add useful tests of library-like code
+* Integrate with Makefile
+* Integrate with CI
+* Integrate with
+  work]
+* Run alongside regular `make test` target

  parent reply	other threads:[~2023-06-09 23:25 UTC|newest]

Thread overview: 32+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-05-17 23:56 [PATCH RFC v2 0/4] Add an external testing library for unit tests steadmon
2023-05-17 23:56 ` [PATCH RFC v2 1/4] common-main: split common_exit() into a new file steadmon
2023-05-18 17:17   ` Junio C Hamano
2023-07-14 23:38     ` Splitting common-main (Was: Re: [PATCH RFC v2 1/4] common-main: split common_exit() into a new file) Josh Steadmon
2023-07-15  0:34       ` Splitting common-main Junio C Hamano
2023-08-14 13:09       ` Splitting common-main (Was: Re: [PATCH RFC v2 1/4] common-main: split common_exit() into a new file) Jeff Hostetler
2023-05-17 23:56 ` [PATCH RFC v2 2/4] unit tests: Add a project plan document steadmon
2023-05-18 13:13   ` Phillip Wood
2023-05-18 20:15   ` Glen Choo
2023-05-24 17:40     ` Josh Steadmon
2023-06-01  9:19     ` Phillip Wood
2023-05-17 23:56 ` [PATCH RFC v2 3/4] Add C TAP harness steadmon
2023-05-18 13:15   ` Phillip Wood
2023-05-18 20:50     ` Josh Steadmon
2023-05-17 23:56 ` [PATCH RFC v2 4/4] unit test: add basic example and build rules steadmon
2023-05-18 13:32   ` Phillip Wood
2023-06-09 23:25 ` [RFC PATCH v3 0/1] Add a project document for adding unit tests Josh Steadmon
2023-06-09 23:25 ` Josh Steadmon [this message]
2023-06-13 22:30   ` [RFC PATCH v3 1/1] unit tests: Add a project plan document Junio C Hamano
2023-06-30 22:18     ` Josh Steadmon
2023-06-29 19:42   ` Linus Arver
2023-06-29 20:48     ` Josh Steadmon
2023-06-30 19:31       ` Linus Arver
2023-07-06 18:24         ` Glen Choo
2023-07-06 19:02           ` Junio C Hamano
2023-07-06 22:48             ` Glen Choo
2023-06-30 21:33       ` Josh Steadmon
2023-06-29 21:21     ` Junio C Hamano
2023-06-30  0:11       ` Linus Arver
2023-06-30 14:07   ` Phillip Wood
2023-06-30 18:47     ` K Wan
2023-06-30 22:35     ` Josh Steadmon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).