automake: Log files generation and test results recording
1
1 15.3.3.2 Log files generation and test results recording
1 ........................................................
1
1 The test driver must correctly generate the files specified by the
1 ‘--log-file’ and ‘--trs-file’ option (even when the tested program fails
1 or crashes).
1
1 The ‘.log’ file should ideally contain all the output produced by the
1 tested program, plus optionally other information that might facilitate
1 debugging or analysis of bug reports. Apart from that, its format is
1 basically free.
1
1 The ‘.trs’ file is used to register some metadata through the use of
1 custom reStructuredText fields. This metadata is expected to be
1 employed in various ways by the parallel test harness; for example, to
1 count the test results when printing the testsuite summary, or to decide
1 which tests to re-run upon ‘make recheck’. Unrecognized metadata in a
1 ‘.trs’ file is currently ignored by the harness, but this might change
1 in the future. The list of currently recognized metadata follows.
1
1 ‘:test-result:’
1 The test driver must use this field to register the results of
1 _each_ test case run by a test script file. Several
1 ‘:test-result:’ fields can be present in the same ‘.trs’ file; this
1 is done in order to support test protocols that allow a single test
1 script to run more test cases.
1
1 The only recognized test results are currently ‘PASS’, ‘XFAIL’,
1 ‘SKIP’, ‘FAIL’, ‘XPASS’ and ‘ERROR’. These results, when declared
1 with ‘:test-result:’, can be optionally followed by text holding
1 the name and/or a brief description of the corresponding test; the
1 harness will ignore such extra text when generating
1 ‘test-suite.log’ and preparing the testsuite summary.
1
1 ‘:recheck:’
1 If this field is present and defined to ‘no’, then the
1 corresponding test script will _not_ be run upon a ‘make recheck’.
1 What happens when two or more ‘:recheck:’ fields are present in the
1 same ‘.trs’ file is undefined behaviour.
1
1 ‘:copy-in-global-log:’
1 If this field is present and defined to ‘no’, then the content of
1 the ‘.log’ file will _not_ be copied into the global
1 ‘test-suite.log’. We allow to forsake such copying because, while
1 it can be useful in debugging and analysis of bug report, it can
1 also be just a waste of space in normal situations, e.g., when a
1 test script is successful. What happens when two or more
1 ‘:copy-in-global-log:’ fields are present in the same ‘.trs’ file
1 is undefined behaviour.
1
1 ‘:test-global-result:’
1 This is used to declare the "global result" of the script.
1 Currently, the value of this field is needed only to be reported
1 (more or less verbatim) in the generated global log file
1 ‘$(TEST_SUITE_LOG)’, so it’s quite free-form. For example, a test
1 script which run 10 test cases, 6 of which pass and 4 of which are
1 skipped, could reasonably have a ‘PASS/SKIP’ value for this field,
1 while a test script which run 19 successful tests and one failed
1 test could have an ‘ALMOST PASSED’ value. What happens when two or
1 more ‘:test-global-result:’ fields are present in the same ‘.trs’
1 file is undefined behaviour.
1
1 Let’s see a small example. Assume a ‘.trs’ file contains the following
1 lines:
1
1 :test-result: PASS server starts
1 :global-log-copy: no
1 :test-result: PASS HTTP/1.1 request
1 :test-result: FAIL HTTP/1.0 request
1 :recheck: yes
1 :test-result: SKIP HTTPS request (TLS library wasn't available)
1 :test-result: PASS server stops
1
1 Then the corresponding test script will be re-run by ‘make check’, will
1 contribute with _five_ test results to the testsuite summary (three of
1 these tests being successful, one failed, and one skipped), and the
1 content of the corresponding ‘.log’ file will _not_ be copied in the
1 global log file ‘test-suite.log’.
1