automake: Log files generation and test results recording

1 
1 15.3.3.2 Log files generation and test results recording
1 ........................................................
1 
1 The test driver must correctly generate the files specified by the
1 ‘--log-file’ and ‘--trs-file’ option (even when the tested program fails
1 or crashes).
1 
1    The ‘.log’ file should ideally contain all the output produced by the
1 tested program, plus optionally other information that might facilitate
1 debugging or analysis of bug reports.  Apart from that, its format is
1 basically free.
1 
1    The ‘.trs’ file is used to register some metadata through the use of
1 custom reStructuredText fields.  This metadata is expected to be
1 employed in various ways by the parallel test harness; for example, to
1 count the test results when printing the testsuite summary, or to decide
1 which tests to re-run upon ‘make recheck’.  Unrecognized metadata in a
1 ‘.trs’ file is currently ignored by the harness, but this might change
1 in the future.  The list of currently recognized metadata follows.
1 
1 ‘:test-result:’
1      The test driver must use this field to register the results of
1      _each_ test case run by a test script file.  Several
1      ‘:test-result:’ fields can be present in the same ‘.trs’ file; this
1      is done in order to support test protocols that allow a single test
1      script to run more test cases.
1 
1      The only recognized test results are currently ‘PASS’, ‘XFAIL’,
1      ‘SKIP’, ‘FAIL’, ‘XPASS’ and ‘ERROR’.  These results, when declared
1      with ‘:test-result:’, can be optionally followed by text holding
1      the name and/or a brief description of the corresponding test; the
1      harness will ignore such extra text when generating
1      ‘test-suite.log’ and preparing the testsuite summary.
1 
1 ‘:recheck:’
1      If this field is present and defined to ‘no’, then the
1      corresponding test script will _not_ be run upon a ‘make recheck’.
1      What happens when two or more ‘:recheck:’ fields are present in the
1      same ‘.trs’ file is undefined behaviour.
1 
1 ‘:copy-in-global-log:’
1      If this field is present and defined to ‘no’, then the content of
1      the ‘.log’ file will _not_ be copied into the global
1      ‘test-suite.log’.  We allow to forsake such copying because, while
1      it can be useful in debugging and analysis of bug report, it can
1      also be just a waste of space in normal situations, e.g., when a
1      test script is successful.  What happens when two or more
1      ‘:copy-in-global-log:’ fields are present in the same ‘.trs’ file
1      is undefined behaviour.
1 
1 ‘:test-global-result:’
1      This is used to declare the "global result" of the script.
1      Currently, the value of this field is needed only to be reported
1      (more or less verbatim) in the generated global log file
1      ‘$(TEST_SUITE_LOG)’, so it’s quite free-form.  For example, a test
1      script which run 10 test cases, 6 of which pass and 4 of which are
1      skipped, could reasonably have a ‘PASS/SKIP’ value for this field,
1      while a test script which run 19 successful tests and one failed
1      test could have an ‘ALMOST PASSED’ value.  What happens when two or
1      more ‘:test-global-result:’ fields are present in the same ‘.trs’
1      file is undefined behaviour.
1 
1 Let’s see a small example.  Assume a ‘.trs’ file contains the following
1 lines:
1 
1      :test-result: PASS server starts
1      :global-log-copy: no
1      :test-result: PASS HTTP/1.1 request
1      :test-result: FAIL HTTP/1.0 request
1      :recheck: yes
1      :test-result: SKIP HTTPS request (TLS library wasn't available)
1      :test-result: PASS server stops
1 
1 Then the corresponding test script will be re-run by ‘make check’, will
1 contribute with _five_ test results to the testsuite summary (three of
1 these tests being successful, one failed, and one skipped), and the
1 content of the corresponding ‘.log’ file will _not_ be copied in the
1 global log file ‘test-suite.log’.
1