From: Josh Steadmon <steadmon@google.com>
To: git@vger.kernel.org
Cc: johannes.schindelin@gmx.de, peff@peff.net,
phillip.wood@dunelm.org.uk, gitster@pobox.com
Subject: [PATCH v3 0/7] test-tool: add unit test suite runner
Date: Fri, 23 Feb 2024 15:33:49 -0800 [thread overview]
Message-ID: <cover.1708728717.git.steadmon@google.com> (raw)
In-Reply-To: <cover.1705443632.git.steadmon@google.com>
Please note: this series has once again been rebased onto the latest
jk/unit-tests-buildfix.
For various reasons (see discussion at [1]) we would like an alternative
to `prove` for running test suites (including the unit tests) on
Windows.
[1] https://lore.kernel.org/git/pull.1613.git.1699894837844.gitgitgadget@gmail.com/
This series extends the existing `test-tool run-command testsuite` to
support running unit tests. In addition, it includes some small
cleanups:
* move t-basic out of the unit-tests directory
* don't hardcode the shell for running tests in `test-tool ... testsuite`
* don't hardcode a test name filter in `test-tool ... testsuite`
* add a test wrapper script to allow unit tests and the shell test suite
to run in a single `prove` process
Changes in V3:
* Added new patch (#7) to use the new test-tool runner for unit tests in
Windows CI.
* Restored the explicit sort call in t/Makefile, for backwards
compatibility with older GNU Make versions.
* Rebased the series on the latest jk/unit-tests-buildfix branch.
* Fixed header include order in patch #1.
* Removed a paragraph in patch #1's commit message that is obsolete now
that we're building the list of test files from the sources rather
than by globbing.
* Added a note in patch #2 that setting a NULL suite.shell_path will be
used in a later commit.
* Clarified up some sloppy wording in commit messages and comments in
t/helper/test-run-command.c.
Changes in V2:
* Rebased the series on the latest jk/unit-tests-buildfix branch.
* Patch 1: move t-basic to a test-tool subcommand rather than a new
executable under t/t0080/
* New patch 2: get the shell path from TEST_SHELL_PATH in
`test-tool run-command testsuite`
* New patch 3: remove the hardcoded filename filter in
`test-tool run-command testsuite`
* Patch 4 (previously 2): simplified now that we no longer need to add
any command-line flags to support unit tests
* Patch 5 (previously 3): avoid trying to run cmake *.pdb files by using
the unit test list built in the makefile in jk/unit-tests-buildfix.
Jeff King (1):
t/Makefile: run unit tests alongside shell tests
Josh Steadmon (6):
t0080: turn t-basic unit test into a helper
test-tool run-command testsuite: get shell from env
test-tool run-command testsuite: remove hardcoded filter
test-tool run-command testsuite: support unit tests
unit tests: add rule for running with test-tool
ci: use test-tool as unit test runner on Windows
Makefile | 6 ++--
ci/run-build-and-tests.sh | 2 --
ci/run-test-slice.sh | 2 +-
t/Makefile | 14 ++++++++--
.../t-basic.c => helper/test-example-tap.c} | 5 ++--
t/helper/test-run-command.c | 28 +++++++++++++++----
t/helper/test-tool.c | 1 +
t/helper/test-tool.h | 1 +
t/run-test.sh | 13 +++++++++
t/t0080-unit-test-output.sh | 24 ++++++++--------
10 files changed, 67 insertions(+), 29 deletions(-)
rename t/{unit-tests/t-basic.c => helper/test-example-tap.c} (95%)
create mode 100755 t/run-test.sh
Range-diff against v2:
1: da756b4bfb ! 1: 6777451100 t0080: turn t-basic unit test into a helper
@@ Commit message
t0080-unit-test-output.sh, so let's move it to
t/helper/test-example-tap.c and adjust Makefiles as necessary.
- This has the additional benefit that test harnesses seeking to run all
- unit tests can find them with a simple glob of "t/unit-tests/bin/t-*",
- with no exceptions needed. This will be important in a later patch where
- we add support for running the unit tests via a test-tool subcommand.
-
## Makefile ##
@@ Makefile: perf: all
$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(filter %.a,$^) $(LIBS)
## t/Makefile ##
-@@ t/Makefile: TINTEROP = $(sort $(wildcard interop/i[0-9][0-9][0-9][0-9]-*.sh))
- CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.test)))
+@@ t/Makefile: CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.tes
CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl
UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c)
--UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))
+ UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))
-UNIT_TESTS = $(sort $(filter-out unit-tests/bin/t-basic%,$(UNIT_TEST_PROGRAMS)))
-+UNIT_TESTS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))
++UNIT_TESTS = $(sort $(UNIT_TEST_PROGRAMS))
# `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)
# checks all tests in all scripts via a single invocation, so tell individual
@@ t/Makefile: TINTEROP = $(sort $(wildcard interop/i[0-9][0-9][0-9][0-9]-*.sh))
## t/unit-tests/t-basic.c => t/helper/test-example-tap.c ##
@@
-#include "test-lib.h"
-+#include "t/unit-tests/test-lib.h"
+#include "test-tool.h"
++#include "t/unit-tests/test-lib.h"
/*
* The purpose of this "unit test" is to verify a few invariants of the unit
2: c8448406d7 ! 2: 24f47f8fc7 test-tool run-command testsuite: get shell from env
@@ Commit message
Add a shell_path field in struct testsuite so that we can pass this to
the task runner callback. If this is non-null, we'll use it as the
argv[0] of the subprocess. Otherwise, we'll just execute the test
- program directly.
+ program directly. We will use this feature in a later commit to enable
+ running binary executable unit tests.
- When setting up the struct testsuite in testsuite(), use the value
- of TEST_SHELL_PATH if it's set, otherwise default to `sh`.
+ However, for now when setting up the struct testsuite in testsuite(),
+ use the value of TEST_SHELL_PATH if it's set, otherwise keep the
+ original behavior by defaulting to `sh`.
[1] https://lore.kernel.org/git/20240123005913.GB835964@coredump.intra.peff.net/
3: e1b89ae93e = 3: 4a16a3ec24 test-tool run-command testsuite: remove hardcoded filter
4: b5665386b5 ! 4: abc9a7afe8 test-tool run-command testsuite: support unit tests
@@ Commit message
test-tool run-command testsuite: support unit tests
Teach the testsuite runner in `test-tool run-command testsuite` how to
- run unit tests: if TEST_SHELL_PATH is not set, assume that we're running
- the programs directly from CWD, rather than defaulting to "sh" as an
- interpreter.
+ run unit tests: if TEST_SHELL_PATH is not set, run the programs directly
+ from CWD, rather than defaulting to "sh" as an interpreter.
With this change, you can now use test-tool to run the unit tests:
$ make
@@ t/helper/test-run-command.c: static int testsuite(int argc, const char **argv)
max_jobs = online_cpus();
+ /*
-+ * If we run without a shell, we have to provide the relative path to
-+ * the executables.
++ * If we run without a shell, execute the programs directly from CWD.
+ */
suite.shell_path = getenv("TEST_SHELL_PATH");
if (!suite.shell_path)
5: f2746703d5 ! 5: a8bbff2c6b unit tests: add rule for running with test-tool
@@ Makefile: $(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o $(UNIT_
$(MAKE) -C t/ unit-tests
## t/Makefile ##
-@@ t/Makefile: CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.tes
- CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl
+@@ t/Makefile: CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl
UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c)
- UNIT_TESTS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))
+ UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))
+ UNIT_TESTS = $(sort $(UNIT_TEST_PROGRAMS))
+UNIT_TESTS_NO_DIR = $(notdir $(UNIT_TESTS))
# `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)
6: cd7467a7bd ! 6: cfcc4bd427 t/Makefile: run unit tests alongside shell tests
@@ Commit message
twice in CI, as discussed in [1].
Additionally, this moves the unit tests into the main dev workflow, so
- that errors can be spotted more quickly.
-
- NEEDS WORK: as discussed in previous commits in this series, there's a
- desire to avoid `prove` specifically and (IIUC) unnecessary
- fork()/exec()ing in general on Windows. This change adds an extra exec()
- for each shell and unit test execution, will that be a problem for
- Windows?
+ that errors can be spotted more quickly. Accordingly, we remove the
+ separate unit tests step for Linux CI. (We leave the Windows CI
+ unit-test step as-is, because the sharding scheme there involves
+ selecting specific test files rather than running `make test`.)
[1] https://lore.kernel.org/git/pull.1613.git.1699894837844.gitgitgadget@gmail.com/
Signed-off-by: Jeff King <peff@peff.net>
+ ## ci/run-build-and-tests.sh ##
+@@ ci/run-build-and-tests.sh: if test -n "$run_tests"
+ then
+ group "Run tests" make test ||
+ handle_failed_tests
+- group "Run unit tests" \
+- make DEFAULT_UNIT_TEST_TARGET=unit-tests-prove unit-tests
+ fi
+ check_unignored_build_artifacts
+
+
## t/Makefile ##
@@ t/Makefile: failed:
test -z "$$failed" || $(MAKE) $$failed
-: ---------- > 7: cbf37e0ddc ci: use test-tool as unit test runner on Windows
base-commit: 4904a4d08cc085716df12ce713ae7ee3d5ecb75a
--
2.44.0.rc0.258.g7320e95886-goog
next prev parent reply other threads:[~2024-02-23 23:33 UTC|newest]
Thread overview: 99+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-01-16 22:22 [RFC PATCH 0/4] test-tool: add unit test suite runner Josh Steadmon
2024-01-16 22:22 ` [RFC PATCH 1/4] t0080: turn t-basic unit test into a helper Josh Steadmon
2024-01-16 22:54 ` Junio C Hamano
2024-01-23 0:43 ` Jeff King
2024-02-01 19:40 ` Josh Steadmon
2024-01-16 22:22 ` [RFC PATCH 2/4] test-tool run-command testsuite: support unit tests Josh Steadmon
2024-01-16 23:18 ` Junio C Hamano
2024-01-16 23:40 ` Junio C Hamano
2024-01-23 0:59 ` Jeff King
2024-02-01 22:06 ` Josh Steadmon
2024-02-01 22:26 ` Junio C Hamano
2024-02-01 23:08 ` Josh Steadmon
2024-02-01 23:14 ` Josh Steadmon
2024-01-16 22:23 ` [RFC PATCH 3/4] unit tests: add rule for running with test-tool Josh Steadmon
2024-01-16 22:23 ` [RFC PATCH 4/4] t/Makefile: run unit tests alongside shell tests Josh Steadmon
2024-01-16 23:24 ` [RFC PATCH 0/4] test-tool: add unit test suite runner Junio C Hamano
2024-02-03 0:50 ` [RFC PATCH v2 0/6] " Josh Steadmon
2024-02-03 0:50 ` [RFC PATCH v2 1/6] t0080: turn t-basic unit test into a helper Josh Steadmon
2024-02-07 20:55 ` Junio C Hamano
2024-02-12 20:42 ` Josh Steadmon
2024-02-07 22:58 ` Jeff King
2024-02-08 0:09 ` Junio C Hamano
2024-02-12 20:53 ` Josh Steadmon
2024-02-12 21:27 ` Junio C Hamano
2024-02-13 7:41 ` Jeff King
2024-02-13 14:36 ` rsbecker
2024-02-22 23:57 ` Josh Steadmon
2024-02-23 0:06 ` rsbecker
2024-02-13 17:28 ` Junio C Hamano
2024-02-22 23:55 ` Josh Steadmon
2024-02-03 0:50 ` [RFC PATCH v2 2/6] test-tool run-command testsuite: get shell from env Josh Steadmon
2024-02-07 20:55 ` Junio C Hamano
2024-02-12 21:35 ` Josh Steadmon
2024-02-03 0:50 ` [RFC PATCH v2 3/6] test-tool run-command testsuite: remove hardcoded filter Josh Steadmon
2024-02-07 20:55 ` Junio C Hamano
2024-02-12 22:48 ` Josh Steadmon
2024-02-12 22:51 ` Junio C Hamano
2024-02-03 0:50 ` [RFC PATCH v2 4/6] test-tool run-command testsuite: support unit tests Josh Steadmon
2024-02-05 16:16 ` phillip.wood123
2024-02-12 21:15 ` Josh Steadmon
2024-02-07 20:48 ` Junio C Hamano
2024-02-23 22:45 ` Josh Steadmon
2024-02-03 0:50 ` [RFC PATCH v2 5/6] unit tests: add rule for running with test-tool Josh Steadmon
2024-02-07 20:50 ` Junio C Hamano
2024-02-03 0:50 ` [RFC PATCH v2 6/6] t/Makefile: run unit tests alongside shell tests Josh Steadmon
2024-02-07 20:55 ` Junio C Hamano
2024-02-07 22:43 ` Jeff King
2024-02-07 23:26 ` Junio C Hamano
2024-02-03 18:52 ` [RFC PATCH v2 0/6] test-tool: add unit test suite runner Junio C Hamano
2024-02-12 22:50 ` Josh Steadmon
2024-02-07 21:14 ` Junio C Hamano
2024-02-23 23:33 ` Josh Steadmon [this message]
2024-02-23 23:33 ` [PATCH v3 1/7] t0080: turn t-basic unit test into a helper Josh Steadmon
2024-02-23 23:33 ` [PATCH v3 2/7] test-tool run-command testsuite: get shell from env Josh Steadmon
2024-02-23 23:33 ` [PATCH v3 3/7] test-tool run-command testsuite: remove hardcoded filter Josh Steadmon
2024-02-23 23:33 ` [PATCH v3 4/7] test-tool run-command testsuite: support unit tests Josh Steadmon
2024-02-23 23:33 ` [PATCH v3 5/7] unit tests: add rule for running with test-tool Josh Steadmon
2024-02-23 23:33 ` [PATCH v3 6/7] t/Makefile: run unit tests alongside shell tests Josh Steadmon
2024-03-27 8:58 ` Jeff King
2024-04-11 18:44 ` Josh Steadmon
2024-04-12 4:29 ` Jeff King
2024-04-24 18:57 ` Josh Steadmon
2024-02-23 23:33 ` [PATCH v3 7/7] ci: use test-tool as unit test runner on Windows Josh Steadmon
2024-03-26 21:33 ` [PATCH v3 0/7] test-tool: add unit test suite runner Junio C Hamano
2024-03-27 9:00 ` Jeff King
2024-04-24 19:14 ` [PATCH v4 " Josh Steadmon
2024-04-24 19:14 ` [PATCH v4 1/7] t0080: turn t-basic unit test into a helper Josh Steadmon
2024-04-24 19:14 ` [PATCH v4 2/7] test-tool run-command testsuite: get shell from env Josh Steadmon
2024-04-24 19:14 ` [PATCH v4 3/7] test-tool run-command testsuite: remove hardcoded filter Josh Steadmon
2024-04-24 19:14 ` [PATCH v4 4/7] test-tool run-command testsuite: support unit tests Josh Steadmon
2024-04-24 19:14 ` [PATCH v4 5/7] unit tests: add rule for running with test-tool Josh Steadmon
2024-04-24 19:14 ` [PATCH v4 6/7] t/Makefile: run unit tests alongside shell tests Josh Steadmon
2024-04-24 21:25 ` Junio C Hamano
2024-04-30 19:49 ` Josh Steadmon
2024-05-03 18:02 ` Jeff King
2024-05-03 19:17 ` Junio C Hamano
2024-05-06 19:58 ` Josh Steadmon
2024-04-24 19:14 ` [PATCH v4 7/7] ci: use test-tool as unit test runner on Windows Josh Steadmon
2024-04-30 19:55 ` [PATCH v5 0/7] test-tool: add unit test suite runner Josh Steadmon
2024-04-30 19:55 ` [PATCH v5 1/7] t0080: turn t-basic unit test into a helper Josh Steadmon
2024-04-30 19:55 ` [PATCH v5 2/7] test-tool run-command testsuite: get shell from env Josh Steadmon
2024-04-30 19:55 ` [PATCH v5 3/7] test-tool run-command testsuite: remove hardcoded filter Josh Steadmon
2024-04-30 19:55 ` [PATCH v5 4/7] test-tool run-command testsuite: support unit tests Josh Steadmon
2024-04-30 19:55 ` [PATCH v5 5/7] unit tests: add rule for running with test-tool Josh Steadmon
2024-04-30 19:55 ` [PATCH v5 6/7] t/Makefile: run unit tests alongside shell tests Josh Steadmon
2024-04-30 21:05 ` Junio C Hamano
2024-05-06 19:58 ` Josh Steadmon
2024-04-30 19:55 ` [PATCH v5 7/7] ci: use test-tool as unit test runner on Windows Josh Steadmon
2024-04-30 21:15 ` [PATCH v5 0/7] test-tool: add unit test suite runner Junio C Hamano
2024-05-06 20:02 ` Josh Steadmon
2024-05-06 19:57 ` [PATCH v6 " Josh Steadmon
2024-05-06 19:57 ` [PATCH v6 1/7] t0080: turn t-basic unit test into a helper Josh Steadmon
2024-05-06 19:57 ` [PATCH v6 2/7] test-tool run-command testsuite: get shell from env Josh Steadmon
2024-05-06 19:57 ` [PATCH v6 3/7] test-tool run-command testsuite: remove hardcoded filter Josh Steadmon
2024-05-06 19:57 ` [PATCH v6 4/7] test-tool run-command testsuite: support unit tests Josh Steadmon
2024-05-06 19:57 ` [PATCH v6 5/7] unit tests: add rule for running with test-tool Josh Steadmon
2024-05-06 19:57 ` [PATCH v6 6/7] t/Makefile: run unit tests alongside shell tests Josh Steadmon
2024-05-06 19:57 ` [PATCH v6 7/7] ci: use test-tool as unit test runner on Windows Josh Steadmon
2024-05-06 21:11 ` [PATCH v6 0/7] test-tool: add unit test suite runner Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cover.1708728717.git.steadmon@google.com \
--to=steadmon@google.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=johannes.schindelin@gmx.de \
--cc=peff@peff.net \
--cc=phillip.wood@dunelm.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).