autoconf: Writing Testsuites
1
1 19.2 Writing `testsuite.at'
1 ===========================
1
1 The `testsuite.at' is a Bourne shell script making use of special
1 Autotest M4 macros. It often contains a call to `AT_INIT' near its
1 beginning followed by one call to `m4_include' per source file for
1 tests. Each such included file, or the remainder of `testsuite.at' if
1 include files are not used, contain a sequence of test groups. Each
1 test group begins with a call to `AT_SETUP', then an arbitrary number
1 of shell commands or calls to `AT_CHECK', and then completes with a
1 call to `AT_CLEANUP'. Multiple test groups can be categorized by a
1 call to `AT_BANNER'.
1
1 All of the public Autotest macros have all-uppercase names in the
1 namespace `^AT_' to prevent them from accidentally conflicting with
1 other text; Autoconf also reserves the namespace `^_AT_' for internal
1 macros. All shell variables used in the testsuite for internal
1 purposes have mostly-lowercase names starting with `at_'. Autotest
1 also uses here-document delimiters in the namespace `^_AT[A-Z]', and
1 makes use of the file system namespace `^at-'.
1
11 Since Autoconf is built on top of M4sugar (⇒Programming in
M4sugar) and M4sh (⇒Programming in M4sh), you must also be
1 aware of those namespaces (`^_?\(m4\|AS\)_'). In general, you _should
1 not use_ the namespace of a package that does not own the macro or
1 shell code you are writing.
1
1 -- Macro: AT_INIT ([NAME])
1 Initialize Autotest. Giving a NAME to the test suite is
1 encouraged if your package includes several test suites. Before
1 this macro is called, `AT_PACKAGE_STRING' and
1 `AT_PACKAGE_BUGREPORT' must be defined, which are used to display
1 information about the testsuite to the user. Typically, these
11 macros are provided by a file `package.m4' built by `make' (⇒
Making testsuite Scripts), in order to inherit the package name,
1 version, and bug reporting address from `configure.ac'.
1
1 -- Macro: AT_COPYRIGHT (COPYRIGHT-NOTICE)
1 State that, in addition to the Free Software Foundation's
1 copyright on the Autotest macros, parts of your test suite are
1 covered by COPYRIGHT-NOTICE.
1
1 The COPYRIGHT-NOTICE shows up in both the head of `testsuite' and
1 in `testsuite --version'.
1
1 -- Macro: AT_ARG_OPTION (OPTIONS, HELP-TEXT, [ACTION-IF-GIVEN],
1 [ACTION-IF-NOT-GIVEN])
1 Accept options from the space-separated list OPTIONS, a list that
1 has leading dashes removed from the options. Long options will be
1 prefixed with `--', single-character options with `-'. The first
1 word in this list is the primary OPTION, any others are assumed to
1 be short-hand aliases. The variable associated with it is
1 `at_arg_OPTION', with any dashes in OPTION replaced with
1 underscores.
1
1 If the user passes `--OPTION' to the `testsuite', the variable
1 will be set to `:'. If the user does not pass the option, or
1 passes `--no-OPTION', then the variable will be set to `false'.
1
1 ACTION-IF-GIVEN is run each time the option is encountered; here,
1 the variable `at_optarg' will be set to `:' or `false' as
1 appropriate. `at_optarg' is actually just a copy of
1 `at_arg_OPTION'.
1
1 ACTION-IF-NOT-GIVEN will be run once after option parsing is
1 complete and if no option from OPTIONS was used.
1
1 HELP-TEXT is added to the end of the list of options shown in
1 `testsuite --help' (⇒AS_HELP_STRING).
1
1 It is recommended that you use a package-specific prefix to OPTIONS
1 names in order to avoid clashes with future Autotest built-in
1 options.
1
1 -- Macro: AT_ARG_OPTION_ARG (OPTIONS, HELP-TEXT, [ACTION-IF-GIVEN],
1 [ACTION-IF-NOT-GIVEN])
1 Accept options with arguments from the space-separated list
1 OPTIONS, a list that has leading dashes removed from the options.
1 Long options will be prefixed with `--', single-character options
1 with `-'. The first word in this list is the primary OPTION, any
1 others are assumed to be short-hand aliases. The variable
1 associated with it is `at_arg_OPTION', with any dashes in OPTION
1 replaced with underscores.
1
1 If the user passes `--OPTION=ARG' or `--OPTION ARG' to the
1 `testsuite', the variable will be set to `ARG'.
1
1 ACTION-IF-GIVEN is run each time the option is encountered; here,
1 the variable `at_optarg' will be set to `ARG'. `at_optarg' is
1 actually just a copy of `at_arg_OPTION'.
1
1 ACTION-IF-NOT-GIVEN will be run once after option parsing is
1 complete and if no option from OPTIONS was used.
1
1 HELP-TEXT is added to the end of the list of options shown in
1 `testsuite --help' (⇒AS_HELP_STRING).
1
1 It is recommended that you use a package-specific prefix to OPTIONS
1 names in order to avoid clashes with future Autotest built-in
1 options.
1
1 -- Macro: AT_COLOR_TESTS
1 Enable colored test results by default when the output is
1 connected to a terminal.
1
1 -- Macro: AT_TESTED (EXECUTABLES)
1 Log the file name and answer to `--version' of each program in
1 space-separated list EXECUTABLES. Several invocations register
1 new executables, in other words, don't fear registering one program
1 several times.
1
1 Autotest test suites rely on `PATH' to find the tested program.
1 This avoids the need to generate absolute names of the various
1 tools, and makes it possible to test installed programs.
1 Therefore, knowing which programs are being exercised is crucial
1 to understanding problems in the test suite itself, or its
1 occasional misuses. It is a good idea to also subscribe foreign
1 programs you depend upon, to avoid incompatible diagnostics.
1
1
1 -- Macro: AT_BANNER (TEST-CATEGORY-NAME)
1 This macro identifies the start of a category of related test
1 groups. When the resulting `testsuite' is invoked with more than
1 one test group to run, its output will include a banner containing
1 TEST-CATEGORY-NAME prior to any tests run from that category. The
1 banner should be no more than about 40 or 50 characters. A blank
1 banner indicates uncategorized tests; an empty line will be
1 inserted after tests from an earlier category, effectively ending
1 that category.
1
1 -- Macro: AT_SETUP (TEST-GROUP-NAME)
1 This macro starts a group of related tests, all to be executed in
1 the same subshell. It accepts a single argument, which holds a
1 few words (no more than about 30 or 40 characters) quickly
1 describing the purpose of the test group being started.
1 TEST-GROUP-NAME must not expand to unbalanced quotes, although
1 quadrigraphs can be used.
1
1 -- Macro: AT_KEYWORDS (KEYWORDS)
1 Associate the space-separated list of KEYWORDS to the enclosing
1 test group. This makes it possible to run "slices" of the test
1 suite. For instance, if some of your test groups exercise some
1 `foo' feature, then using `AT_KEYWORDS(foo)' lets you run
1 `./testsuite -k foo' to run exclusively these test groups. The
1 TEST-GROUP-NAME of the test group is automatically recorded to
1 `AT_KEYWORDS'.
1
1 Several invocations within a test group accumulate new keywords.
1 In other words, don't fear registering the same keyword several
1 times in a test group.
1
1 -- Macro: AT_CAPTURE_FILE (FILE)
1 If the current test group fails, log the contents of FILE.
1 Several identical calls within one test group have no additional
1 effect.
1
1 -- Macro: AT_FAIL_IF (SHELL-CONDITION)
1 Make the test group fail and skip the rest of its execution, if
1 SHELL-CONDITION is true. SHELL-CONDITION is a shell expression
1 such as a `test' command. Tests before `AT_FAIL_IF' will be
1 executed and may still cause the test group to be skipped. You
1 can instantiate this macro many times from within the same test
1 group.
1
1 You should use this macro only for very simple failure conditions.
1 If the SHELL-CONDITION could emit any kind of output you should
1 instead use `AT_CHECK' like
1 AT_CHECK([if SHELL-CONDITION; then exit 99; fi])
1 so that such output is properly recorded in the `testsuite.log'
1 file.
1
1 -- Macro: AT_SKIP_IF (SHELL-CONDITION)
1 Determine whether the test should be skipped because it requires
1 features that are unsupported on the machine under test.
1 SHELL-CONDITION is a shell expression such as a `test' command.
1 Tests before `AT_SKIP_IF' will be executed and may still cause the
1 test group to fail. You can instantiate this macro many times
1 from within the same test group.
1
1 You should use this macro only for very simple skip conditions.
1 If the SHELL-CONDITION could emit any kind of output you should
1 instead use `AT_CHECK' like
1 AT_CHECK([if SHELL-CONDITION; then exit 77; fi])
1 so that such output is properly recorded in the `testsuite.log'
1 file.
1
1 -- Macro: AT_XFAIL_IF (SHELL-CONDITION)
1 Determine whether the test is expected to fail because it is a
1 known bug (for unsupported features, you should skip the test).
1 SHELL-CONDITION is a shell expression such as a `test' command;
1 you can instantiate this macro many times from within the same
1 test group, and one of the conditions is enough to turn the test
1 into an expected failure.
1
1 -- Macro: AT_CLEANUP
1 End the current test group.
1
1
1 -- Macro: AT_DATA (FILE, CONTENTS)
1 Initialize an input data FILE with given CONTENTS. Of course, the
1 CONTENTS have to be properly quoted between square brackets to
1 protect against included commas or spurious M4 expansion.
1 CONTENTS must be empty or end with a newline. FILE must be a
1 single shell word that expands into a single file name.
1
1 -- Macro: AT_CHECK (COMMANDS, [STATUS = `0'], [STDOUT], [STDERR],
1 [RUN-IF-FAIL], [RUN-IF-PASS])
1 -- Macro: AT_CHECK_UNQUOTED (COMMANDS, [STATUS = `0'], [STDOUT],
1 [STDERR], [RUN-IF-FAIL], [RUN-IF-PASS])
1 Execute a test by performing given shell COMMANDS in a subshell.
1 COMMANDS is output as-is, so shell expansions are honored. These
1 commands should normally exit with STATUS, while producing expected
1 STDOUT and STDERR contents. If COMMANDS exit with unexpected
1 status 77, then the rest of the test group is skipped. If
1 COMMANDS exit with unexpected status 99, then the test group is
1 immediately failed. Otherwise, if this test fails, run shell
1 commands RUN-IF-FAIL or, if this test passes, run shell commands
1 RUN-IF-PASS, both inside the current shell execution environment.
1 At the beginning of RUN-IF-FAIL and RUN-IF-PASS, the status of
1 COMMANDS is available in the `at_status' shell variable.
1
1 This macro must be invoked in between `AT_SETUP' and `AT_CLEANUP'.
1
1 If STATUS is the literal `ignore', then the corresponding exit
1 status is not checked, except for the special cases of 77 (skip)
1 and 99 (hard failure). The existence of hard failures allows one
1 to mark a test as an expected failure with `AT_XFAIL_IF' because a
1 feature has not yet been implemented, but to still distinguish
1 between gracefully handling the missing feature and dumping core.
1 A hard failure also inhibits post-test actions in RUN-IF-FAIL.
1
1 If the value of the STDOUT or STDERR parameter is one of the
1 literals in the following table, then the test treats the output
1 according to the rules of that literal. Otherwise, the value of
1 the parameter is treated as text that must exactly match the
1 output given by COMMANDS on standard output and standard error
1 (including an empty parameter for no output); any differences are
1 captured in the testsuite log and the test is failed (unless an
1 unexpected exit status of 77 skipped the test instead). The
1 difference between `AT_CHECK' and `AT_CHECK_UNQUOTED' is that only
1 the latter performs shell variable expansion (`$'), command
1 substitution (``'), and backslash escaping (`\') on comparison
1 text given in the STDOUT and STDERR arguments; if the text
1 includes a trailing newline, this would be the same as if it were
1 specified via an unquoted here-document. (However, there is no
1 difference in the interpretation of COMMANDS).
1
1 `ignore'
1 The content of the output is ignored, but still captured in
1 the test group log (if the testsuite is run with option `-v',
1 the test group log is displayed as the test is run; if the
1 test group later fails, the test group log is also copied
1 into the overall testsuite log). This action is valid for
1 both STDOUT and STDERR.
1
1 `ignore-nolog'
1 The content of the output is ignored, and nothing is captured
1 in the log files. If COMMANDS are likely to produce binary
1 output (including long lines) or large amounts of output,
1 then logging the output can make it harder to locate details
1 related to subsequent tests within the group, and could
1 potentially corrupt terminal display of a user running
1 `testsuite -v'.
1
1 `stdout'
1 For the STDOUT parameter, capture the content of standard
1 output to both the file `stdout' and the test group log.
1 Subsequent commands in the test group can then post-process
1 the file. This action is often used when it is desired to
1 use `grep' to look for a substring in the output, or when the
1 output must be post-processed to normalize error messages
1 into a common form.
1
1 `stderr'
1 Like `stdout', except that it only works for the STDERR
1 parameter, and the standard error capture file will be named
1 `stderr'.
1
1 `stdout-nolog'
1 `stderr-nolog'
1 Like `stdout' or `stderr', except that the captured output is
1 not duplicated into the test group log. This action is
1 particularly useful for an intermediate check that produces
1 large amounts of data, which will be followed by another
1 check that filters down to the relevant data, as it makes it
1 easier to locate details in the log.
1
1 `expout'
1 For the STDOUT parameter, compare standard output contents
1 with the previously created file `expout', and list any
1 differences in the testsuite log.
1
1 `experr'
1 Like `expout', except that it only works for the STDERR
1 parameter, and the standard error contents are compared with
1 `experr'.
1
1 -- Macro: AT_CHECK_EUNIT (MODULE, TEST-SPEC, [ERLFLAGS],
1 [RUN-IF-FAIL], [RUN-IF-PASS])
1 Initialize and execute an Erlang module named MODULE that performs
1 tests following the TEST-SPEC EUnit test specification. TEST-SPEC
1 must be a valid EUnit test specification, as defined in the EUnit
1 Reference Manual (http://erlang.org/doc/apps/eunit/index.html).
1 ERLFLAGS are optional command-line options passed to the Erlang
1 interpreter to execute the test Erlang module. Typically,
1 ERLFLAGS defines at least the paths to directories containing the
1 compiled Erlang modules under test, as `-pa path1 path2 ...'.
1
1 For example, the unit tests associated with Erlang module `testme',
1 which compiled code is in subdirectory `src', can be performed
1 with:
1
1 AT_CHECK_EUNIT([testme_testsuite], [{module, testme}],
1 [-pa "${abs_top_builddir}/src"])
1
1 This macro must be invoked in between `AT_SETUP' and `AT_CLEANUP'.
1
1 Variables `ERL', `ERLC', and (optionally) `ERLCFLAGS' must be
1 defined as the path of the Erlang interpreter, the path of the
1 Erlang compiler, and the command-line flags to pass to the
1 compiler, respectively. Those variables should be configured in
1 `configure.ac' using the `AC_ERLANG_PATH_ERL' and
1 `AC_ERLANG_PATH_ERLC' macros, and the configured values of those
1 variables are automatically defined in the testsuite. If `ERL' or
1 `ERLC' is not defined, the test group is skipped.
1
1 If the EUnit library cannot be found, i.e. if module `eunit' cannot
1 be loaded, the test group is skipped. Otherwise, if TEST-SPEC is
1 an invalid EUnit test specification, the test group fails.
1 Otherwise, if the EUnit test passes, shell commands RUN-IF-PASS
1 are executed or, if the EUnit test fails, shell commands
1 RUN-IF-FAIL are executed and the test group fails.
1
1 Only the generated test Erlang module is automatically compiled and
1 executed. If TEST-SPEC involves testing other Erlang modules,
1 e.g. module `testme' in the example above, those modules must be
1 already compiled.
1
1 If the testsuite is run in verbose mode, with option `--verbose',
1 EUnit is also run in verbose mode to output more details about
1 individual unit tests.
1