automake: Use TAP with the Automake test harness

1 
1 15.4.2 Use TAP with the Automake test harness
1 ---------------------------------------------
1 
1 Currently, the TAP driver that comes with Automake requires some by-hand
1 steps on the developer’s part (this situation should hopefully be
1 improved in future Automake versions).  You’ll have to grab the
1 ‘tap-driver.sh’ script from the Automake distribution by hand, copy it
1 in your source tree, and use the Automake support for third-party test
1 drivers to instruct the harness to use the ‘tap-driver.sh’ script and
1 the awk program found by ‘AM_INIT_AUTOMAKE’ to run your TAP-producing
1 tests.  See the example below for clarification.
1 
11    Apart from the options common to all the Automake test drivers (⇒
 Command-line arguments for test drivers), the ‘tap-driver.sh’ supports
1 the following options, whose names are chosen for enhanced compatibility
1 with the ‘prove’ utility.
1 
1 ‘--ignore-exit’
1      Causes the test driver to ignore the exit status of the test
1      scripts; by default, the driver will report an error if the script
1      exits with a non-zero status.  This option has effect also on
1      non-zero exit statuses due to termination by a signal.
1 ‘--comments’
1      Instruct the test driver to display TAP diagnostic (i.e., lines
1      beginning with the ‘#’ character) in the testsuite progress output
1      too; by default, TAP diagnostic is only copied to the ‘.log’ file.
1 ‘--no-comments’
1      Revert the effects of ‘--comments’.
1 ‘--merge’
1      Instruct the test driver to merge the test scripts’ standard error
1      into their standard output.  This is necessary if you want to
1      ensure that diagnostics from the test scripts are displayed in the
1      correct order relative to test results; this can be of great help
1      in debugging (especially if your test scripts are shell scripts run
1      with shell tracing active).  As a downside, this option might cause
1      the test harness to get confused if anything that appears on
1      standard error looks like a test result.
1 ‘--no-merge’
1      Revert the effects of ‘--merge’.
1 ‘--diagnostic-string=STRING’
1      Change the string that introduces TAP diagnostic from the default
1      value of “‘#’” to ‘STRING’.  This can be useful if your TAP-based
1      test scripts produce verbose output on which they have limited
1      control (because, say, the output comes from other tools invoked in
1      the scripts), and it might contain text that gets spuriously
1      interpreted as TAP diagnostic: such an issue can be solved by
1      redefining the string that activates TAP diagnostic to a value you
1      know won’t appear by chance in the tests’ output.  Note however
1      that this feature is non-standard, as the “official” TAP protocol
1      does not allow for such a customization; so don’t use it if you can
1      avoid it.
1 
1 Here is an example of how the TAP driver can be set up and used.
1 
1      % cat configure.ac
1      AC_INIT([GNU Try Tap], [1.0], [bug-automake@gnu.org])
1      AC_CONFIG_AUX_DIR([build-aux])
1      AM_INIT_AUTOMAKE([foreign -Wall -Werror])
1      AC_CONFIG_FILES([Makefile])
1      AC_REQUIRE_AUX_FILE([tap-driver.sh])
1      AC_OUTPUT
1 
1      % cat Makefile.am
1      TEST_LOG_DRIVER = env AM_TAP_AWK='$(AWK)' $(SHELL) \
1                        $(top_srcdir)/build-aux/tap-driver.sh
1      TESTS = foo.test bar.test baz.test
1      EXTRA_DIST = $(TESTS)
1 
1      % cat foo.test
1      #!/bin/sh
1      echo 1..4 # Number of tests to be executed.
1      echo 'ok 1 - Swallows fly'
1      echo 'not ok 2 - Caterpillars fly # TODO metamorphosis in progress'
1      echo 'ok 3 - Pigs fly # SKIP not enough acid'
1      echo '# I just love word plays ...'
1      echo 'ok 4 - Flies fly too :-)'
1 
1      % cat bar.test
1      #!/bin/sh
1      echo 1..3
1      echo 'not ok 1 - Bummer, this test has failed.'
1      echo 'ok 2 - This passed though.'
1      echo 'Bail out! Ennui kicking in, sorry...'
1      echo 'ok 3 - This will not be seen.'
1 
1      % cat baz.test
1      #!/bin/sh
1      echo 1..1
1      echo ok 1
1      # Exit with error, even if all the tests have been successful.
1      exit 7
1 
1      % cp PREFIX/share/automake-APIVERSION/tap-driver.sh .
1      % autoreconf -vi && ./configure && make check
1      ...
1      PASS: foo.test 1 - Swallows fly
1      XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
1      SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
1      PASS: foo.test 4 - Flies fly too :-)
1      FAIL: bar.test 1 - Bummer, this test has failed.
1      PASS: bar.test 2 - This passed though.
1      ERROR: bar.test - Bail out! Ennui kicking in, sorry...
1      PASS: baz.test 1
1      ERROR: baz.test - exited with status 7
1      ...
1      Please report to bug-automake@gnu.org
1      ...
1      % echo exit status: $?
1      exit status: 1
1 
1      % env TEST_LOG_DRIVER_FLAGS='--comments --ignore-exit' \
1            TESTS='foo.test baz.test' make -e check
1      ...
1      PASS: foo.test 1 - Swallows fly
1      XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
1      SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
1      # foo.test: I just love word plays...
1      PASS: foo.test 4 - Flies fly too :-)
1      PASS: baz.test 1
1      ...
1      % echo exit status: $?
1      exit status: 0
1