automake: Overview of Custom Test Drivers Support

1 
1 15.3.1 Overview of Custom Test Drivers Support
1 ----------------------------------------------
1 
1 Starting from Automake version 1.12, the parallel test harness allows
1 the package authors to use third-party custom test drivers, in case the
1 default ones are inadequate for their purposes, or do not support their
1 testing protocol of choice.
1 
1    A custom test driver is expected to properly run the test programs
1 passed to it (including the command-line arguments passed to those
1 programs, if any), to analyze their execution and outcome, to create the
1 ‘.log’ and ‘.trs’ files associated to these test runs, and to display
1 the test results on the console.  It is responsibility of the author of
1 the test driver to ensure that it implements all the above steps
1 meaningfully and correctly; Automake isn’t and can’t be of any help
1 here.  On the other hand, the Automake-provided code for testsuite
1 summary generation offers support for test drivers allowing several test
1 results per test script, if they take care to register such results
1 properly (⇒Log files generation and test results recording).
1 
1    The exact details of how test scripts’ results are to be determined
1 and analyzed is left to the individual drivers.  Some drivers might only
1 consider the test script exit status (this is done for example by the
1 default test driver used by the parallel test harness, described in the
1 previous section).  Other drivers might implement more complex and
1 advanced test protocols, which might require them to parse and
1 interpreter the output emitted by the test script they’re running
1 (examples of such protocols are TAP and SubUnit).
1 
1    It’s very important to note that, even when using custom test
1 drivers, most of the infrastructure described in the previous section
1 about the parallel harness remains in place; this includes:
1 
1    • list of test scripts defined in ‘TESTS’, and overridable at runtime
1      through the redefinition of ‘TESTS’ or ‘TEST_LOGS’;
1    • concurrency through the use of ‘make’’s option ‘-j’;
1    • per-test ‘.log’ and ‘.trs’ files, and generation of a summary
1      ‘.log’ file from them;
1    • ‘recheck’ target, ‘RECHECK_LOGS’ variable, and lazy reruns of
1      tests;
1    • inter-test dependencies;
1    • support for ‘check_*’ variables (‘check_PROGRAMS’,
1      ‘check_LIBRARIES’, ...);
1    • use of ‘VERBOSE’ environment variable to get verbose output on
1      testsuite failures;
1    • definition and honoring of ‘TESTS_ENVIRONMENT’,
1      ‘AM_TESTS_ENVIRONMENT’ and ‘AM_TESTS_FD_REDIRECT’ variables;
1    • definition of generic and extension-specific ‘LOG_COMPILER’ and
1      ‘LOG_FLAGS’ variables.
1 
1 On the other hand, the exact semantics of how (and if) testsuite output
1 colorization, ‘XFAIL_TESTS’, and hard errors are supported and handled
1 is left to the individual test drivers.
1