From: Josh Steadmon <steadmon@google.com>
To: git@vger.kernel.org
Cc: johannes.schindelin@gmx.de, peff@peff.net,
phillip.wood@dunelm.org.uk, gitster@pobox.com
Subject: [RFC PATCH v2 0/6] test-tool: add unit test suite runner
Date: Fri, 2 Feb 2024 16:50:25 -0800 [thread overview]
Message-ID: <cover.1706921262.git.steadmon@google.com> (raw)
In-Reply-To: <cover.1705443632.git.steadmon@google.com>
Please note: this series has been rebased onto jk/unit-tests-buildfix.
For various reasons (see discussion at [1]) we would like an alternative
to `prove` for running test suites (including the unit tests) on
Windows.
This series extends the existing `test-tool run-command testsuite` to
support running unit tests. In addition, it includes some small
cleanups:
* move t-basic out of the unit-tests directory
* don't hardcode the shell for running tests in `test-tool ... testsuite`
* don't hardcode a test name filter in `test-tool ... testsuite`
* add a test wrapper script to allow unit tests and the shell test suite
to run in a single `prove` process
Some known remaining bits of work:
* We should investigate switching the Windows CI to use `test-tool`
instead of prove. However, Windows CI seems broken on
jk/unit-tests-buildfix, and I haven't had time to determine why.
* We should determine whether it is confusing or otherwise harmful to
people's workflow to have the unit tests run in parallel with shell
tests when using prove as the default test target.
[1] https://lore.kernel.org/git/pull.1613.git.1699894837844.gitgitgadget@gmail.com/
Changes in V2:
* Patch 1: move t-basic to a test-tool subcommand rather than a new
executable under t/t0080/
* New patch 2: get the shell path from TEST_SHELL_PATH in
`test-tool run-command testsuite`
* New patch 3: remove the hardcoded filename filter in
`test-tool run-command testsuite`
* Patch 4 (previously 2): simplified now that we no longer need to add
any command-line flags to support unit tests
* Patch 5 (previously 3): avoid trying to run cmake *.pdb files by using
the unit test list built in the makefile in jk/unit-tests-buildfix.
Jeff King (1):
t/Makefile: run unit tests alongside shell tests
Josh Steadmon (5):
t0080: turn t-basic unit test into a helper
test-tool run-command testsuite: get shell from env
test-tool run-command testsuite: remove hardcoded filter
test-tool run-command testsuite: support unit tests
unit tests: add rule for running with test-tool
Makefile | 6 ++--
t/Makefile | 15 +++++++---
.../t-basic.c => helper/test-example-tap.c} | 5 ++--
t/helper/test-run-command.c | 29 +++++++++++++++----
t/helper/test-tool.c | 1 +
t/helper/test-tool.h | 1 +
t/run-test.sh | 13 +++++++++
t/t0080-unit-test-output.sh | 24 +++++++--------
8 files changed, 67 insertions(+), 27 deletions(-)
rename t/{unit-tests/t-basic.c => helper/test-example-tap.c} (95%)
create mode 100755 t/run-test.sh
Range-diff against v1:
1: a9f67ed703 < -: ---------- t0080: turn t-basic unit test into a helper
-: ---------- > 1: da756b4bfb t0080: turn t-basic unit test into a helper
-: ---------- > 2: c8448406d7 test-tool run-command testsuite: get shell from env
-: ---------- > 3: e1b89ae93e test-tool run-command testsuite: remove hardcoded filter
2: 5ecbc976e6 ! 4: b5665386b5 test-tool run-command testsuite: support unit tests
@@ Commit message
test-tool run-command testsuite: support unit tests
Teach the testsuite runner in `test-tool run-command testsuite` how to
- run unit tests, by adding two new flags:
+ run unit tests: if TEST_SHELL_PATH is not set, assume that we're running
+ the programs directly from CWD, rather than defaulting to "sh" as an
+ interpreter.
- First, "--(no-)run-in-shell" allows the test-tool to exec the unit-test
- binaries directly, rather than trying to interpret them as shell
- scripts.
-
- Second "--(no-)require-shell-test-pattern" bypasses the check that the
- test filenames match the expected t####-*.sh pattern.
-
- With these changes, you can now use test-tool to run the unit tests:
+ With this change, you can now use test-tool to run the unit tests:
$ make
$ cd t/unit-tests/bin
- $ ../../helper/test-tool run-command testsuite --no-run-in-shell \
- --no-require-shell-test-pattern
+ $ ../../helper/test-tool run-command testsuite
This should be helpful on Windows to allow running tests without
requiring Perl (for `prove`), as discussed in [1] and [2].
+ This again breaks backwards compatibility, as it is now required to set
+ TEST_SHELL_PATH properly for executing shell scripts, but again, as
+ noted in [2], there are no longer any such invocations in our codebase.
+
[1] https://lore.kernel.org/git/nycvar.QRO.7.76.6.2109091323150.59@tvgsbejvaqbjf.bet/
[2] https://lore.kernel.org/git/850ea42c-f103-68d5-896b-9120e2628686@gmx.de/
## t/helper/test-run-command.c ##
-@@ t/helper/test-run-command.c: static int task_finished(int result UNUSED,
- struct testsuite {
- struct string_list tests, failed;
- int next;
-- int quiet, immediate, verbose, verbose_log, trace, write_junit_xml;
-+ int quiet, immediate, verbose, verbose_log, trace, write_junit_xml, run_in_shell;
- };
- #define TESTSUITE_INIT { \
- .tests = STRING_LIST_INIT_DUP, \
- .failed = STRING_LIST_INIT_DUP, \
-+ .run_in_shell = 1, \
- }
-
- static int next_test(struct child_process *cp, struct strbuf *err, void *cb,
-@@ t/helper/test-run-command.c: static int next_test(struct child_process *cp, struct strbuf *err, void *cb,
- return 0;
-
- test = suite->tests.items[suite->next++].string;
-- strvec_pushl(&cp->args, "sh", test, NULL);
-+ if (suite->run_in_shell)
-+ strvec_push(&cp->args, "sh");
-+ strvec_push(&cp->args, test);
- if (suite->quiet)
- strvec_push(&cp->args, "--quiet");
- if (suite->immediate)
-@@ t/helper/test-run-command.c: static const char * const testsuite_usage[] = {
- static int testsuite(int argc, const char **argv)
- {
- struct testsuite suite = TESTSUITE_INIT;
-- int max_jobs = 1, i, ret = 0;
-+ int max_jobs = 1, i, ret = 0, require_shell_test_pattern = 1;
- DIR *dir;
- struct dirent *d;
- struct option options[] = {
-@@ t/helper/test-run-command.c: static int testsuite(int argc, const char **argv)
- OPT_BOOL('x', "trace", &suite.trace, "trace shell commands"),
- OPT_BOOL(0, "write-junit-xml", &suite.write_junit_xml,
- "write JUnit-style XML files"),
-+ OPT_BOOL(0, "run-in-shell", &suite.run_in_shell,
-+ "run programs in the suite via `sh`"),
-+ OPT_BOOL(0, "require-shell-test-pattern", &require_shell_test_pattern,
-+ "require programs to match 't####-*.sh'"),
- OPT_END()
- };
- struct run_process_parallel_opts opts = {
@@ t/helper/test-run-command.c: static int testsuite(int argc, const char **argv)
.task_finished = test_finished,
.data = &suite,
@@ t/helper/test-run-command.c: static int testsuite(int argc, const char **argv)
argc = parse_options(argc, argv, NULL, options,
testsuite_usage, PARSE_OPT_STOP_AT_NON_OPTION);
-
+@@ t/helper/test-run-command.c: static int testsuite(int argc, const char **argv)
if (max_jobs <= 0)
max_jobs = online_cpus();
+
+ /*
+ * If we run without a shell, we have to provide the relative path to
+ * the executables.
+ */
-+ if (!suite.run_in_shell)
+ suite.shell_path = getenv("TEST_SHELL_PATH");
+ if (!suite.shell_path)
+- suite.shell_path = "sh";
+ strbuf_addstr(&progpath, "./");
+ path_prefix_len = progpath.len;
dir = opendir(".");
if (!dir)
@@ t/helper/test-run-command.c: static int testsuite(int argc, const char **argv)
- while ((d = readdir(dir))) {
- const char *p = d->d_name;
-
-- if (*p != 't' || !isdigit(p[1]) || !isdigit(p[2]) ||
-- !isdigit(p[3]) || !isdigit(p[4]) || p[5] != '-' ||
-- !ends_with(p, ".sh"))
-+ if (!strcmp(p, ".") || !strcmp(p, ".."))
- continue;
-+ if (require_shell_test_pattern)
-+ if (*p != 't' || !isdigit(p[1]) || !isdigit(p[2]) ||
-+ !isdigit(p[3]) || !isdigit(p[4]) || p[5] != '-' ||
-+ !ends_with(p, ".sh"))
-+ continue;
/* No pattern: match all */
if (!argc) {
3: 5b34c851cd ! 5: f2746703d5 unit tests: add rule for running with test-tool
@@ Commit message
`make DEFAULT_UNIT_TEST_TARGET=unit-tests-test-tool unit-tests`, or by
setting DEFAULT_UNIT_TEST_TARGET in config.mak.
- NEEDS WORK: we need to exclude .pdb files generated by cmake [see
- 0df903d402 (unit-tests: do not mistake `.pdb` files for being
- executable, 2023-09-25)]
-
## Makefile ##
-@@ Makefile: $(UNIT_TEST_HELPER_PROGS): %$X: %.o $(UNIT_TEST_DIR)/test-lib.o $(GITLIBS) GIT-L
+@@ Makefile: $(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o $(UNIT_TEST_DIR)/
.PHONY: build-unit-tests unit-tests
build-unit-tests: $(UNIT_TEST_PROGS)
@@ Makefile: $(UNIT_TEST_HELPER_PROGS): %$X: %.o $(UNIT_TEST_DIR)/test-lib.o $(GITL
$(MAKE) -C t/ unit-tests
## t/Makefile ##
+@@ t/Makefile: CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.tes
+ CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl
+ UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c)
+ UNIT_TESTS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))
++UNIT_TESTS_NO_DIR = $(notdir $(UNIT_TESTS))
+
+ # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`)
+ # checks all tests in all scripts via a single invocation, so tell individual
@@ t/Makefile: $(T):
$(UNIT_TESTS):
@echo "*** $@ ***"; $@
@@ t/Makefile: unit-tests-raw: $(UNIT_TESTS)
+ @echo "*** test-tool - unit tests **"
+ ( \
+ cd unit-tests/bin && \
-+ ../../helper/test-tool run-command testsuite --no-run-in-shell --no-require-shell-test-pattern \
++ ../../helper/test-tool$X run-command testsuite $(UNIT_TESTS_NO_DIR)\
+ )
+
pre-clean:
4: c823265f0d = 6: cd7467a7bd t/Makefile: run unit tests alongside shell tests
base-commit: 799d449105dc1f6e77fa1ebaea4f6d8bdc6537cf
--
2.43.0.594.gd9cf4e227d-goog
next prev parent reply other threads:[~2024-02-03 0:50 UTC|newest]
Thread overview: 99+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-01-16 22:22 [RFC PATCH 0/4] test-tool: add unit test suite runner Josh Steadmon
2024-01-16 22:22 ` [RFC PATCH 1/4] t0080: turn t-basic unit test into a helper Josh Steadmon
2024-01-16 22:54 ` Junio C Hamano
2024-01-23 0:43 ` Jeff King
2024-02-01 19:40 ` Josh Steadmon
2024-01-16 22:22 ` [RFC PATCH 2/4] test-tool run-command testsuite: support unit tests Josh Steadmon
2024-01-16 23:18 ` Junio C Hamano
2024-01-16 23:40 ` Junio C Hamano
2024-01-23 0:59 ` Jeff King
2024-02-01 22:06 ` Josh Steadmon
2024-02-01 22:26 ` Junio C Hamano
2024-02-01 23:08 ` Josh Steadmon
2024-02-01 23:14 ` Josh Steadmon
2024-01-16 22:23 ` [RFC PATCH 3/4] unit tests: add rule for running with test-tool Josh Steadmon
2024-01-16 22:23 ` [RFC PATCH 4/4] t/Makefile: run unit tests alongside shell tests Josh Steadmon
2024-01-16 23:24 ` [RFC PATCH 0/4] test-tool: add unit test suite runner Junio C Hamano
2024-02-03 0:50 ` Josh Steadmon [this message]
2024-02-03 0:50 ` [RFC PATCH v2 1/6] t0080: turn t-basic unit test into a helper Josh Steadmon
2024-02-07 20:55 ` Junio C Hamano
2024-02-12 20:42 ` Josh Steadmon
2024-02-07 22:58 ` Jeff King
2024-02-08 0:09 ` Junio C Hamano
2024-02-12 20:53 ` Josh Steadmon
2024-02-12 21:27 ` Junio C Hamano
2024-02-13 7:41 ` Jeff King
2024-02-13 14:36 ` rsbecker
2024-02-22 23:57 ` Josh Steadmon
2024-02-23 0:06 ` rsbecker
2024-02-13 17:28 ` Junio C Hamano
2024-02-22 23:55 ` Josh Steadmon
2024-02-03 0:50 ` [RFC PATCH v2 2/6] test-tool run-command testsuite: get shell from env Josh Steadmon
2024-02-07 20:55 ` Junio C Hamano
2024-02-12 21:35 ` Josh Steadmon
2024-02-03 0:50 ` [RFC PATCH v2 3/6] test-tool run-command testsuite: remove hardcoded filter Josh Steadmon
2024-02-07 20:55 ` Junio C Hamano
2024-02-12 22:48 ` Josh Steadmon
2024-02-12 22:51 ` Junio C Hamano
2024-02-03 0:50 ` [RFC PATCH v2 4/6] test-tool run-command testsuite: support unit tests Josh Steadmon
2024-02-05 16:16 ` phillip.wood123
2024-02-12 21:15 ` Josh Steadmon
2024-02-07 20:48 ` Junio C Hamano
2024-02-23 22:45 ` Josh Steadmon
2024-02-03 0:50 ` [RFC PATCH v2 5/6] unit tests: add rule for running with test-tool Josh Steadmon
2024-02-07 20:50 ` Junio C Hamano
2024-02-03 0:50 ` [RFC PATCH v2 6/6] t/Makefile: run unit tests alongside shell tests Josh Steadmon
2024-02-07 20:55 ` Junio C Hamano
2024-02-07 22:43 ` Jeff King
2024-02-07 23:26 ` Junio C Hamano
2024-02-03 18:52 ` [RFC PATCH v2 0/6] test-tool: add unit test suite runner Junio C Hamano
2024-02-12 22:50 ` Josh Steadmon
2024-02-07 21:14 ` Junio C Hamano
2024-02-23 23:33 ` [PATCH v3 0/7] " Josh Steadmon
2024-02-23 23:33 ` [PATCH v3 1/7] t0080: turn t-basic unit test into a helper Josh Steadmon
2024-02-23 23:33 ` [PATCH v3 2/7] test-tool run-command testsuite: get shell from env Josh Steadmon
2024-02-23 23:33 ` [PATCH v3 3/7] test-tool run-command testsuite: remove hardcoded filter Josh Steadmon
2024-02-23 23:33 ` [PATCH v3 4/7] test-tool run-command testsuite: support unit tests Josh Steadmon
2024-02-23 23:33 ` [PATCH v3 5/7] unit tests: add rule for running with test-tool Josh Steadmon
2024-02-23 23:33 ` [PATCH v3 6/7] t/Makefile: run unit tests alongside shell tests Josh Steadmon
2024-03-27 8:58 ` Jeff King
2024-04-11 18:44 ` Josh Steadmon
2024-04-12 4:29 ` Jeff King
2024-04-24 18:57 ` Josh Steadmon
2024-02-23 23:33 ` [PATCH v3 7/7] ci: use test-tool as unit test runner on Windows Josh Steadmon
2024-03-26 21:33 ` [PATCH v3 0/7] test-tool: add unit test suite runner Junio C Hamano
2024-03-27 9:00 ` Jeff King
2024-04-24 19:14 ` [PATCH v4 " Josh Steadmon
2024-04-24 19:14 ` [PATCH v4 1/7] t0080: turn t-basic unit test into a helper Josh Steadmon
2024-04-24 19:14 ` [PATCH v4 2/7] test-tool run-command testsuite: get shell from env Josh Steadmon
2024-04-24 19:14 ` [PATCH v4 3/7] test-tool run-command testsuite: remove hardcoded filter Josh Steadmon
2024-04-24 19:14 ` [PATCH v4 4/7] test-tool run-command testsuite: support unit tests Josh Steadmon
2024-04-24 19:14 ` [PATCH v4 5/7] unit tests: add rule for running with test-tool Josh Steadmon
2024-04-24 19:14 ` [PATCH v4 6/7] t/Makefile: run unit tests alongside shell tests Josh Steadmon
2024-04-24 21:25 ` Junio C Hamano
2024-04-30 19:49 ` Josh Steadmon
2024-05-03 18:02 ` Jeff King
2024-05-03 19:17 ` Junio C Hamano
2024-05-06 19:58 ` Josh Steadmon
2024-04-24 19:14 ` [PATCH v4 7/7] ci: use test-tool as unit test runner on Windows Josh Steadmon
2024-04-30 19:55 ` [PATCH v5 0/7] test-tool: add unit test suite runner Josh Steadmon
2024-04-30 19:55 ` [PATCH v5 1/7] t0080: turn t-basic unit test into a helper Josh Steadmon
2024-04-30 19:55 ` [PATCH v5 2/7] test-tool run-command testsuite: get shell from env Josh Steadmon
2024-04-30 19:55 ` [PATCH v5 3/7] test-tool run-command testsuite: remove hardcoded filter Josh Steadmon
2024-04-30 19:55 ` [PATCH v5 4/7] test-tool run-command testsuite: support unit tests Josh Steadmon
2024-04-30 19:55 ` [PATCH v5 5/7] unit tests: add rule for running with test-tool Josh Steadmon
2024-04-30 19:55 ` [PATCH v5 6/7] t/Makefile: run unit tests alongside shell tests Josh Steadmon
2024-04-30 21:05 ` Junio C Hamano
2024-05-06 19:58 ` Josh Steadmon
2024-04-30 19:55 ` [PATCH v5 7/7] ci: use test-tool as unit test runner on Windows Josh Steadmon
2024-04-30 21:15 ` [PATCH v5 0/7] test-tool: add unit test suite runner Junio C Hamano
2024-05-06 20:02 ` Josh Steadmon
2024-05-06 19:57 ` [PATCH v6 " Josh Steadmon
2024-05-06 19:57 ` [PATCH v6 1/7] t0080: turn t-basic unit test into a helper Josh Steadmon
2024-05-06 19:57 ` [PATCH v6 2/7] test-tool run-command testsuite: get shell from env Josh Steadmon
2024-05-06 19:57 ` [PATCH v6 3/7] test-tool run-command testsuite: remove hardcoded filter Josh Steadmon
2024-05-06 19:57 ` [PATCH v6 4/7] test-tool run-command testsuite: support unit tests Josh Steadmon
2024-05-06 19:57 ` [PATCH v6 5/7] unit tests: add rule for running with test-tool Josh Steadmon
2024-05-06 19:57 ` [PATCH v6 6/7] t/Makefile: run unit tests alongside shell tests Josh Steadmon
2024-05-06 19:57 ` [PATCH v6 7/7] ci: use test-tool as unit test runner on Windows Josh Steadmon
2024-05-06 21:11 ` [PATCH v6 0/7] test-tool: add unit test suite runner Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cover.1706921262.git.steadmon@google.com \
--to=steadmon@google.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=johannes.schindelin@gmx.de \
--cc=peff@peff.net \
--cc=phillip.wood@dunelm.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).