automake: Generalities about Testing

1 
1 15.1 Generalities about Testing
1 ===============================
1 
1 The purpose of testing is to determine whether a program or system
1 behaves as expected (e.g., known inputs produce the expected outputs,
1 error conditions are correctly handled or reported, and older bugs do
1 not resurface).
1 
1    The minimal unit of testing is usually called _test case_, or simply
1 _test_.  How a test case is defined or delimited, and even what exactly
1 _constitutes_ a test case, depends heavily on the testing paradigm
1 and/or framework in use, so we won’t attempt any more precise
1 definition.  The set of the test cases for a given program or system
1 constitutes its _testsuite_.
1 
1    A _test harness_ (also _testsuite harness_) is a program or software
1 component that executes all (or part of) the defined test cases,
1 analyzes their outcomes, and report or register these outcomes
1 appropriately.  Again, the details of how this is accomplished (and how
1 the developer and user can influence it or interface with it) varies
1 wildly, and we’ll attempt no precise definition.
1 
1    A test is said to _pass_ when it can determine that the condition or
1 behaviour it means to verify holds, and is said to _fail_ when it can
1 determine that such condition of behaviour does _not_ hold.
1 
1    Sometimes, tests can rely on non-portable tools or prerequisites, or
1 simply make no sense on a given system (for example, a test checking a
1 Windows-specific feature makes no sense on a GNU/Linux system).  In this
1 case, accordingly to the definition above, the tests can neither be
1 considered passed nor failed; instead, they are _skipped_ – i.e., they
1 are not run, or their result is anyway ignored for what concerns the
1 count of failures an successes.  Skips are usually explicitly reported
1 though, so that the user will be aware that not all of the testsuite has
1 really run.
1 
1    It’s not uncommon, especially during early development stages, that
1 some tests fail for known reasons, and that the developer doesn’t want
1 to tackle these failures immediately (this is especially true when the
1 failing tests deal with corner cases).  In this situation, the better
1 policy is to declare that each of those failures is an _expected
1 failure_ (or _xfail_).  In case a test that is expected to fail ends up
1 passing instead, many testing environments will flag the result as a
1 special kind of failure called _unexpected pass_ (or _xpass_).
1 
1    Many testing environments and frameworks distinguish between test
1 failures and hard errors.  As we’ve seen, a test failure happens when
1 some invariant or expected behaviour of the software under test is not
1 met.  An _hard error_ happens when e.g., the set-up of a test case
1 scenario fails, or when some other unexpected or highly undesirable
1 condition is encountered (for example, the program under test
1 experiences a segmentation fault).
1