automake: Parallel Test Harness
1
1 15.2.3 Parallel Test Harness
1 ----------------------------
1
1 By default, Automake generated a parallel (concurrent) test harness. It
1 features automatic collection of the test scripts output in ‘.log’
1 files, concurrent execution of tests with ‘make -j’, specification of
1 inter-test dependencies, lazy reruns of tests that have not completed in
1 a prior run, and hard errors for exceptional failures.
1
1 The parallel test harness operates by defining a set of ‘make’ rules
1 that run the test scripts listed in ‘TESTS’, and, for each such script,
1 save its output in a corresponding ‘.log’ file and its results (and
1 other “metadata”, ⇒API for Custom Test Drivers) in a
1 corresponding ‘.trs’ (as in Test ReSults) file. The ‘.log’ file will
1 contain all the output emitted by the test on its standard output and
1 its standard error. The ‘.trs’ file will contain, among the other
1 things, the results of the test cases run by the script.
1
1 The parallel test harness will also create a summary log file,
1 ‘TEST_SUITE_LOG’, which defaults to ‘test-suite.log’ and requires a
1 ‘.log’ suffix. This file depends upon all the ‘.log’ and ‘.trs’ files
1 created for the test scripts listed in ‘TESTS’.
1
1 As with the serial harness above, by default one status line is
1 printed per completed test, and a short summary after the suite has
1 completed. However, standard output and standard error of the test are
1 redirected to a per-test log file, so that parallel execution does not
1 produce intermingled output. The output from failed tests is collected
1 in the ‘test-suite.log’ file. If the variable ‘VERBOSE’ is set, this
1 file is output after the summary.
1
1 Each couple of ‘.log’ and ‘.trs’ files is created when the
1 corresponding test has completed. The set of log files is listed in the
1 read-only variable ‘TEST_LOGS’, and defaults to ‘TESTS’, with the
1 executable extension if any (⇒EXEEXT), as well as any suffix
1 listed in ‘TEST_EXTENSIONS’ removed, and ‘.log’ appended. Results are
1 undefined if a test file name ends in several concatenated suffixes.
1 ‘TEST_EXTENSIONS’ defaults to ‘.test’; it can be overridden by the user,
1 in which case any extension listed in it must be constituted by a dot,
1 followed by a non-digit alphabetic character, followed by any number of
1 alphabetic characters. For example, ‘.sh’, ‘.T’ and ‘.t1’ are valid
1 extensions, while ‘.x-y’, ‘.6c’ and ‘.t.1’ are not.
1
1 It is important to note that, due to current limitations (unlikely to
1 be lifted), configure substitutions in the definition of ‘TESTS’ can
1 only work if they will expand to a list of tests that have a suffix
1 listed in ‘TEST_EXTENSIONS’.
1
1 For tests that match an extension ‘.EXT’ listed in ‘TEST_EXTENSIONS’,
1 you can provide a custom “test runner” using the variable
1 ‘EXT_LOG_COMPILER’ (note the upper-case extension) and pass options in
1 ‘AM_EXT_LOG_FLAGS’ and allow the user to pass options in
1 ‘EXT_LOG_FLAGS’. It will cause all tests with this extension to be
1 called with this runner. For all tests without a registered extension,
1 the variables ‘LOG_COMPILER’, ‘AM_LOG_FLAGS’, and ‘LOG_FLAGS’ may be
1 used. For example,
1
1 TESTS = foo.pl bar.py baz
1 TEST_EXTENSIONS = .pl .py
1 PL_LOG_COMPILER = $(PERL)
1 AM_PL_LOG_FLAGS = -w
1 PY_LOG_COMPILER = $(PYTHON)
1 AM_PY_LOG_FLAGS = -v
1 LOG_COMPILER = ./wrapper-script
1 AM_LOG_FLAGS = -d
1
1 will invoke ‘$(PERL) -w foo.pl’, ‘$(PYTHON) -v bar.py’, and
1 ‘./wrapper-script -d baz’ to produce ‘foo.log’, ‘bar.log’, and
1 ‘baz.log’, respectively. The ‘foo.trs’, ‘bar.trs’ and ‘baz.trs’ files
1 will be automatically produced as a side-effect.
1
1 It’s important to note that, differently from what we’ve seen for the
1 serial test harness (⇒Serial Test Harness), the
1 ‘AM_TESTS_ENVIRONMENT’ and ‘TESTS_ENVIRONMENT’ variables _cannot_ be
1 used to define a custom test runner; the ‘LOG_COMPILER’ and ‘LOG_FLAGS’
1 (or their extension-specific counterparts) should be used instead:
1
1 ## This is WRONG!
1 AM_TESTS_ENVIRONMENT = PERL5LIB='$(srcdir)/lib' $(PERL) -Mstrict -w
1
1 ## Do this instead.
1 AM_TESTS_ENVIRONMENT = PERL5LIB='$(srcdir)/lib'; export PERL5LIB;
1 LOG_COMPILER = $(PERL)
1 AM_LOG_FLAGS = -Mstrict -w
1
1 By default, the test suite harness will run all tests, but there are
1 several ways to limit the set of tests that are run:
1
1 • You can set the ‘TESTS’ variable. For example, you can use a
1 command like this to run only a subset of the tests:
1
1 env TESTS="foo.test bar.test" make -e check
1
1 Note however that the command above will unconditionally overwrite
1 the ‘test-suite.log’ file, thus clobbering the recorded results of
1 any previous testsuite run. This might be undesirable for packages
1 whose testsuite takes long time to execute. Luckily, this problem
1 can easily be avoided by overriding also ‘TEST_SUITE_LOG’ at
1 runtime; for example,
1
1 env TEST_SUITE_LOG=partial.log TESTS="..." make -e check
1
1 will write the result of the partial testsuite runs to the
1 ‘partial.log’, without touching ‘test-suite.log’.
1
1 • You can set the ‘TEST_LOGS’ variable. By default, this variable is
1 computed at ‘make’ run time from the value of ‘TESTS’ as described
1 above. For example, you can use the following:
1
1 set x subset*.log; shift
1 env TEST_LOGS="foo.log $*" make -e check
1
1 The comments made above about ‘TEST_SUITE_LOG’ overriding applies
1 here too.
1
1 • By default, the test harness removes all old per-test ‘.log’ and
1 ‘.trs’ files before it starts running tests to regenerate them.
1 The variable ‘RECHECK_LOGS’ contains the set of ‘.log’ (and, by
1 implication, ‘.trs’) files which are removed. ‘RECHECK_LOGS’
1 defaults to ‘TEST_LOGS’, which means all tests need to be
1 rechecked. By overriding this variable, you can choose which tests
1 need to be reconsidered. For example, you can lazily rerun only
1 those tests which are outdated, i.e., older than their prerequisite
1 test files, by setting this variable to the empty value:
1
1 env RECHECK_LOGS= make -e check
1
1 • You can ensure that all tests are rerun which have failed or passed
1 unexpectedly, by running ‘make recheck’ in the test directory.
1 This convenience target will set ‘RECHECK_LOGS’ appropriately
1 before invoking the main test harness.
1
1 In order to guarantee an ordering between tests even with ‘make -jN’,
1 dependencies between the corresponding ‘.log’ files may be specified
1 through usual ‘make’ dependencies. For example, the following snippet
1 lets the test named ‘foo-execute.test’ depend upon completion of the
1 test ‘foo-compile.test’:
1
1 TESTS = foo-compile.test foo-execute.test
1 foo-execute.log: foo-compile.log
1
1 Please note that this ordering ignores the _results_ of required tests,
1 thus the test ‘foo-execute.test’ is run even if the test
1 ‘foo-compile.test’ failed or was skipped beforehand. Further, please
1 note that specifying such dependencies currently works only for tests
1 that end in one of the suffixes listed in ‘TEST_EXTENSIONS’.
1
1 Tests without such specified dependencies may be run concurrently
1 with parallel ‘make -jN’, so be sure they are prepared for concurrent
1 execution.
1
1 The combination of lazy test execution and correct dependencies
1 between tests and their sources may be exploited for efficient unit
1 testing during development. To further speed up the edit-compile-test
1 cycle, it may even be useful to specify compiled programs in
1 ‘EXTRA_PROGRAMS’ instead of with ‘check_PROGRAMS’, as the former allows
1 intertwined compilation and test execution (but note that
1 ‘EXTRA_PROGRAMS’ are not cleaned automatically, ⇒Uniform).
1
1 The variables ‘TESTS’ and ‘XFAIL_TESTS’ may contain conditional parts
1 as well as configure substitutions. In the latter case, however,
1 certain restrictions apply: substituted test names must end with a
1 nonempty test suffix like ‘.test’, so that one of the inference rules
1 generated by ‘automake’ can apply. For literal test names, ‘automake’
1 can generate per-target rules to avoid this limitation.
1
1 Please note that it is currently not possible to use ‘$(srcdir)/’ or
1 ‘$(top_srcdir)/’ in the ‘TESTS’ variable. This technical limitation is
1 necessary to avoid generating test logs in the source tree and has the
1 unfortunate consequence that it is not possible to specify distributed
1 tests that are themselves generated by means of explicit rules, in a way
11 that is portable to all ‘make’ implementations (⇒(autoconf)Make
Target Lookup, the semantics of FreeBSD and OpenBSD ‘make’ conflict
1 with this). In case of doubt you may want to require to use GNU ‘make’,
1 or work around the issue with inference rules to generate the tests.
1