automake: Scripts-based Testsuites
1
1 15.2.1 Scripts-based Testsuites
1 -------------------------------
1
1 If the special variable ‘TESTS’ is defined, its value is taken to be a
1 list of programs or scripts to run in order to do the testing. Under
1 the appropriate circumstances, it’s possible for ‘TESTS’ to list also
1 data files to be passed to one or more test scripts defined by different
1 means (the so-called “log compilers”, ⇒Parallel Test Harness).
1
1 Test scripts can be executed serially or concurrently. Automake
1 supports both these kinds of test execution, with the parallel test
1 harness being the default. The concurrent test harness relies on the
1 concurrence capabilities (if any) offered by the underlying ‘make’
1 implementation, and can thus only be as good as those are.
1
1 By default, only the exit statuses of the test scripts are considered
1 when determining the testsuite outcome. But Automake allows also the
11 use of more complex test protocols, either standard (⇒Using the TAP
test protocol) or custom (⇒Custom Test Drivers). Note that you
1 can’t enable such protocols when the serial harness is used, though. In
1 the rest of this section we are going to concentrate mostly on
1 protocol-less tests, since we cover test protocols in a later section
1 (again, ⇒Custom Test Drivers).
1
1 When no test protocol is in use, an exit status of 0 from a test
1 script will denote a success, an exit status of 77 a skipped test, an
1 exit status of 99 an hard error, and any other exit status will denote a
1 failure.
1
1 You may define the variable ‘XFAIL_TESTS’ to a list of tests (usually
1 a subset of ‘TESTS’) that are expected to fail; this will effectively
1 reverse the result of those tests (with the provision that skips and
1 hard errors remain untouched). You may also instruct the testsuite
1 harness to treat hard errors like simple failures, by defining the
1 ‘DISABLE_HARD_ERRORS’ make variable to a nonempty value.
1
1 Note however that, for tests based on more complex test protocols,
1 the exact effects of ‘XFAIL_TESTS’ and ‘DISABLE_HARD_ERRORS’ might
1 change, or they might even have no effect at all (for example, in tests
1 using TAP, there is no way to disable hard errors, and the
1 ‘DISABLE_HARD_ERRORS’ variable has no effect on them).
1
1 The result of each test case run by the scripts in ‘TESTS’ will be
1 printed on standard output, along with the test name. For test
1 protocols that allow more test cases per test script (such as TAP), a
1 number, identifier and/or brief description specific for the single test
1 case is expected to be printed in addition to the name of the test
1 script. The possible results (whose meanings should be clear from the
1 previous ⇒Generalities about Testing) are ‘PASS’, ‘FAIL’, ‘SKIP’,
1 ‘XFAIL’, ‘XPASS’ and ‘ERROR’. Here is an example of output from an
1 hypothetical testsuite that uses both plain and TAP tests:
1 PASS: foo.sh
1 PASS: zardoz.tap 1 - Daemon started
1 PASS: zardoz.tap 2 - Daemon responding
1 SKIP: zardoz.tap 3 - Daemon uses /proc # SKIP /proc is not mounted
1 PASS: zardoz.tap 4 - Daemon stopped
1 SKIP: bar.sh
1 PASS: mu.tap 1
1 XFAIL: mu.tap 2 # TODO frobnication not yet implemented
1
1 A testsuite summary (expected to report at least the number of run,
1 skipped and failed tests) will be printed at the end of the testsuite
1 run.
1
1 If the standard output is connected to a capable terminal, then the
1 test results and the summary are colored appropriately. The developer
1 and the user can disable colored output by setting the ‘make’ variable
1 ‘AM_COLOR_TESTS=no’; the user can in addition force colored output even
1 without a connecting terminal with ‘AM_COLOR_TESTS=always’. It’s also
1 worth noting that some ‘make’ implementations, when used in parallel
11 mode, have slightly different semantics (⇒(autoconf)Parallel
make), which can break the automatic detection of a connection to a
1 capable terminal. If this is the case, the user will have to resort to
1 the use of ‘AM_COLOR_TESTS=always’ in order to have the testsuite output
1 colorized.
1
1 Test programs that need data files should look for them in ‘srcdir’
1 (which is both a make variable and an environment variable made
1 available to the tests), so that they work when building in a separate
1 directory (⇒Build Directories (autoconf)Build Directories.), and
11 in particular for the ‘distcheck’ rule (⇒Checking the
Distribution).
1
1 The ‘AM_TESTS_ENVIRONMENT’ and ‘TESTS_ENVIRONMENT’ variables can be
1 used to run initialization code and set environment variables for the
1 test scripts. The former variable is developer-reserved, and can be
1 defined in the ‘Makefile.am’, while the latter is reserved for the user,
1 which can employ it to extend or override the settings in the former;
1 for this to work portably, however, the contents of a non-empty
1 ‘AM_TESTS_ENVIRONMENT’ _must_ be terminated by a semicolon.
1
1 The ‘AM_TESTS_FD_REDIRECT’ variable can be used to define file
1 descriptor redirections for the test scripts. One might think that
1 ‘AM_TESTS_ENVIRONMENT’ could be used for this purpose, but experience
1 has shown that doing so portably is practically impossible. The main
1 hurdle is constituted by Korn shells, which usually set the
1 close-on-exec flag on file descriptors opened with the ‘exec’ builtin,
1 thus rendering an idiom like ‘AM_TESTS_ENVIRONMENT = exec 9>&2;’
1 ineffectual. This issue also affects some Bourne shells, such as the
1 HP-UX’s ‘/bin/sh’,
1
1 AM_TESTS_ENVIRONMENT = \
1 ## Some environment initializations are kept in a separate shell
1 ## file 'tests-env.sh', which can make it easier to also run tests
1 ## from the command line.
1 . $(srcdir)/tests-env.sh; \
1 ## On Solaris, prefer more POSIX-compliant versions of the standard
1 ## tools by default.
1 if test -d /usr/xpg4/bin; then \
1 PATH=/usr/xpg4/bin:$$PATH; export PATH; \
1 fi;
1 ## With this, the test scripts will be able to print diagnostic
1 ## messages to the original standard error stream, even if the test
1 ## driver redirects the stderr of the test scripts to a log file
1 ## before executing them.
1 AM_TESTS_FD_REDIRECT = 9>&2
1
1 Note however that ‘AM_TESTS_ENVIRONMENT’ is, for historical and
11 implementation reasons, _not_ supported by the serial harness (⇒
Serial Test Harness).
1
1 Automake ensures that each file listed in ‘TESTS’ is built before it
1 is run; you can list both source and derived programs (or scripts) in
1 ‘TESTS’; the generated rule will look both in ‘srcdir’ and ‘.’. For
1 instance, you might want to run a C program as a test. To do this you
1 would list its name in ‘TESTS’ and also in ‘check_PROGRAMS’, and then
1 specify it as you would any other program.
1
1 Programs listed in ‘check_PROGRAMS’ (and ‘check_LIBRARIES’,
1 ‘check_LTLIBRARIES’...) are only built during ‘make check’, not during
1 ‘make all’. You should list there any program needed by your tests that
1 does not need to be built by ‘make all’. Note that ‘check_PROGRAMS’ are
1 _not_ automatically added to ‘TESTS’ because ‘check_PROGRAMS’ usually
1 lists programs used by the tests, not the tests themselves. Of course
1 you can set ‘TESTS = $(check_PROGRAMS)’ if all your programs are test
1 cases.
1