automake: Use TAP with the Automake test harness
1
1 15.4.2 Use TAP with the Automake test harness
1 ---------------------------------------------
1
1 Currently, the TAP driver that comes with Automake requires some by-hand
1 steps on the developer’s part (this situation should hopefully be
1 improved in future Automake versions). You’ll have to grab the
1 ‘tap-driver.sh’ script from the Automake distribution by hand, copy it
1 in your source tree, and use the Automake support for third-party test
1 drivers to instruct the harness to use the ‘tap-driver.sh’ script and
1 the awk program found by ‘AM_INIT_AUTOMAKE’ to run your TAP-producing
1 tests. See the example below for clarification.
1
11 Apart from the options common to all the Automake test drivers (⇒
Command-line arguments for test drivers), the ‘tap-driver.sh’ supports
1 the following options, whose names are chosen for enhanced compatibility
1 with the ‘prove’ utility.
1
1 ‘--ignore-exit’
1 Causes the test driver to ignore the exit status of the test
1 scripts; by default, the driver will report an error if the script
1 exits with a non-zero status. This option has effect also on
1 non-zero exit statuses due to termination by a signal.
1 ‘--comments’
1 Instruct the test driver to display TAP diagnostic (i.e., lines
1 beginning with the ‘#’ character) in the testsuite progress output
1 too; by default, TAP diagnostic is only copied to the ‘.log’ file.
1 ‘--no-comments’
1 Revert the effects of ‘--comments’.
1 ‘--merge’
1 Instruct the test driver to merge the test scripts’ standard error
1 into their standard output. This is necessary if you want to
1 ensure that diagnostics from the test scripts are displayed in the
1 correct order relative to test results; this can be of great help
1 in debugging (especially if your test scripts are shell scripts run
1 with shell tracing active). As a downside, this option might cause
1 the test harness to get confused if anything that appears on
1 standard error looks like a test result.
1 ‘--no-merge’
1 Revert the effects of ‘--merge’.
1 ‘--diagnostic-string=STRING’
1 Change the string that introduces TAP diagnostic from the default
1 value of “‘#’” to ‘STRING’. This can be useful if your TAP-based
1 test scripts produce verbose output on which they have limited
1 control (because, say, the output comes from other tools invoked in
1 the scripts), and it might contain text that gets spuriously
1 interpreted as TAP diagnostic: such an issue can be solved by
1 redefining the string that activates TAP diagnostic to a value you
1 know won’t appear by chance in the tests’ output. Note however
1 that this feature is non-standard, as the “official” TAP protocol
1 does not allow for such a customization; so don’t use it if you can
1 avoid it.
1
1 Here is an example of how the TAP driver can be set up and used.
1
1 % cat configure.ac
1 AC_INIT([GNU Try Tap], [1.0], [bug-automake@gnu.org])
1 AC_CONFIG_AUX_DIR([build-aux])
1 AM_INIT_AUTOMAKE([foreign -Wall -Werror])
1 AC_CONFIG_FILES([Makefile])
1 AC_REQUIRE_AUX_FILE([tap-driver.sh])
1 AC_OUTPUT
1
1 % cat Makefile.am
1 TEST_LOG_DRIVER = env AM_TAP_AWK='$(AWK)' $(SHELL) \
1 $(top_srcdir)/build-aux/tap-driver.sh
1 TESTS = foo.test bar.test baz.test
1 EXTRA_DIST = $(TESTS)
1
1 % cat foo.test
1 #!/bin/sh
1 echo 1..4 # Number of tests to be executed.
1 echo 'ok 1 - Swallows fly'
1 echo 'not ok 2 - Caterpillars fly # TODO metamorphosis in progress'
1 echo 'ok 3 - Pigs fly # SKIP not enough acid'
1 echo '# I just love word plays ...'
1 echo 'ok 4 - Flies fly too :-)'
1
1 % cat bar.test
1 #!/bin/sh
1 echo 1..3
1 echo 'not ok 1 - Bummer, this test has failed.'
1 echo 'ok 2 - This passed though.'
1 echo 'Bail out! Ennui kicking in, sorry...'
1 echo 'ok 3 - This will not be seen.'
1
1 % cat baz.test
1 #!/bin/sh
1 echo 1..1
1 echo ok 1
1 # Exit with error, even if all the tests have been successful.
1 exit 7
1
1 % cp PREFIX/share/automake-APIVERSION/tap-driver.sh .
1 % autoreconf -vi && ./configure && make check
1 ...
1 PASS: foo.test 1 - Swallows fly
1 XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
1 SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
1 PASS: foo.test 4 - Flies fly too :-)
1 FAIL: bar.test 1 - Bummer, this test has failed.
1 PASS: bar.test 2 - This passed though.
1 ERROR: bar.test - Bail out! Ennui kicking in, sorry...
1 PASS: baz.test 1
1 ERROR: baz.test - exited with status 7
1 ...
1 Please report to bug-automake@gnu.org
1 ...
1 % echo exit status: $?
1 exit status: 1
1
1 % env TEST_LOG_DRIVER_FLAGS='--comments --ignore-exit' \
1 TESTS='foo.test baz.test' make -e check
1 ...
1 PASS: foo.test 1 - Swallows fly
1 XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
1 SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
1 # foo.test: I just love word plays...
1 PASS: foo.test 4 - Flies fly too :-)
1 PASS: baz.test 1
1 ...
1 % echo exit status: $?
1 exit status: 0
1