gccinstall: Testing
1
1 6 Installing GCC: Testing
1 *************************
1
1 Before you install GCC, we encourage you to run the testsuites and to
1 compare your results with results from a similar configuration that have
1 been submitted to the gcc-testresults mailing list. Some of these
1 archived results are linked from the build status lists at
1 <http://gcc.gnu.org/buildstat.html>, although not everyone who reports a
1 successful build runs the testsuites and submits the results. This step
1 is optional and may require you to download additional software, but it
1 can give you confidence in your new GCC installation or point out
1 problems before you install and start using your new GCC.
1
1 First, you must have downloaded the testsuites. These are part of
1 the full distribution, but if you downloaded the "core" compiler plus
1 any front ends, you must download the testsuites separately.
1
1 Second, you must have the testing tools installed. This includes
1 DejaGnu, Tcl, and Expect; the DejaGnu site has links to these. For
1 running the BRIG frontend tests, a tool to assemble the binary BRIGs
1 from HSAIL text, HSAILasm must be installed.
1
1 If the directories where 'runtest' and 'expect' were installed are
1 not in the 'PATH', you may need to set the following environment
1 variables appropriately, as in the following example (which assumes that
1 DejaGnu has been installed under '/usr/local'):
1
1 TCL_LIBRARY = /usr/local/share/tcl8.0
1 DEJAGNULIBS = /usr/local/share/dejagnu
1
1 (On systems such as Cygwin, these paths are required to be actual
1 paths, not mounts or links; presumably this is due to some lack of
1 portability in the DejaGnu code.)
1
1 Finally, you can run the testsuite (which may take a long time):
1 cd OBJDIR; make -k check
1
1 This will test various components of GCC, such as compiler front ends
1 and runtime libraries. While running the testsuite, DejaGnu might emit
1 some harmless messages resembling 'WARNING: Couldn't find the global
1 config file.' or 'WARNING: Couldn't find tool init file' that can be
1 ignored.
1
1 If you are testing a cross-compiler, you may want to run the
1 testsuite on a simulator as described at
1 <http://gcc.gnu.org/simtest-howto.html>.
1
1 6.1 How can you run the testsuite on selected tests?
1 ====================================================
1
1 In order to run sets of tests selectively, there are targets 'make
1 check-gcc' and language specific 'make check-c', 'make check-c++', 'make
1 check-fortran', 'make check-ada', 'make check-objc', 'make
1 check-obj-c++', 'make check-lto' in the 'gcc' subdirectory of the object
1 directory. You can also just run 'make check' in a subdirectory of the
1 object directory.
1
1 A more selective way to just run all 'gcc' execute tests in the
1 testsuite is to use
1
1 make check-gcc RUNTESTFLAGS="execute.exp OTHER-OPTIONS"
1
1 Likewise, in order to run only the 'g++' "old-deja" tests in the
1 testsuite with filenames matching '9805*', you would use
1
1 make check-g++ RUNTESTFLAGS="old-deja.exp=9805* OTHER-OPTIONS"
1
1 The file-matching expression following FILENAME'.exp=' is treated as
1 a series of whitespace-delimited glob expressions so that multiple
1 patterns may be passed, although any whitespace must either be escaped
1 or surrounded by single quotes if multiple expressions are desired. For
1 example,
1
1 make check-g++ RUNTESTFLAGS="old-deja.exp=9805*\ virtual2.c OTHER-OPTIONS"
1 make check-g++ RUNTESTFLAGS="'old-deja.exp=9805* virtual2.c' OTHER-OPTIONS"
1
1 The '*.exp' files are located in the testsuite directories of the GCC
1 source, the most important ones being 'compile.exp', 'execute.exp',
1 'dg.exp' and 'old-deja.exp'. To get a list of the possible '*.exp'
1 files, pipe the output of 'make check' into a file and look at the
1 'Running ... .exp' lines.
1
1 6.2 Passing options and running multiple testsuites
1 ===================================================
1
1 You can pass multiple options to the testsuite using the
1 '--target_board' option of DejaGNU, either passed as part of
1 'RUNTESTFLAGS', or directly to 'runtest' if you prefer to work outside
1 the makefiles. For example,
1
1 make check-g++ RUNTESTFLAGS="--target_board=unix/-O3/-fmerge-constants"
1
1 will run the standard 'g++' testsuites ("unix" is the target name for
1 a standard native testsuite situation), passing '-O3 -fmerge-constants'
1 to the compiler on every test, i.e., slashes separate options.
1
1 You can run the testsuites multiple times using combinations of
1 options with a syntax similar to the brace expansion of popular shells:
1
1 ..."--target_board=arm-sim\{-mhard-float,-msoft-float\}\{-O1,-O2,-O3,\}"
1
1 (Note the empty option caused by the trailing comma in the final
1 group.) The following will run each testsuite eight times using the
1 'arm-sim' target, as if you had specified all possible combinations
1 yourself:
1
1 --target_board='arm-sim/-mhard-float/-O1 \
1 arm-sim/-mhard-float/-O2 \
1 arm-sim/-mhard-float/-O3 \
1 arm-sim/-mhard-float \
1 arm-sim/-msoft-float/-O1 \
1 arm-sim/-msoft-float/-O2 \
1 arm-sim/-msoft-float/-O3 \
1 arm-sim/-msoft-float'
1
1 They can be combined as many times as you wish, in arbitrary ways.
1 This list:
1
1 ..."--target_board=unix/-Wextra\{-O3,-fno-strength\}\{-fomit-frame,\}"
1
1 will generate four combinations, all involving '-Wextra'.
1
1 The disadvantage to this method is that the testsuites are run in
1 serial, which is a waste on multiprocessor systems. For users with GNU
1 Make and a shell which performs brace expansion, you can run the
1 testsuites in parallel by having the shell perform the combinations and
1 'make' do the parallel runs. Instead of using '--target_board', use a
1 special makefile target:
1
1 make -jN check-TESTSUITE//TEST-TARGET/OPTION1/OPTION2/...
1
1 For example,
1
1 make -j3 check-gcc//sh-hms-sim/{-m1,-m2,-m3,-m3e,-m4}/{,-nofpu}
1
1 will run three concurrent "make-gcc" testsuites, eventually testing
1 all ten combinations as described above. Note that this is currently
1 only supported in the 'gcc' subdirectory. (To see how this works, try
1 typing 'echo' before the example given here.)
1
1 6.3 How to interpret test results
1 =================================
1
1 The result of running the testsuite are various '*.sum' and '*.log'
1 files in the testsuite subdirectories. The '*.log' files contain a
1 detailed log of the compiler invocations and the corresponding results,
1 the '*.sum' files summarize the results. These summaries contain status
1 codes for all tests:
1
1 * PASS: the test passed as expected
1 * XPASS: the test unexpectedly passed
1 * FAIL: the test unexpectedly failed
1 * XFAIL: the test failed as expected
1 * UNSUPPORTED: the test is not supported on this platform
1 * ERROR: the testsuite detected an error
1 * WARNING: the testsuite detected a possible problem
1
1 It is normal for some tests to report unexpected failures. At the
1 current time the testing harness does not allow fine grained control
1 over whether or not a test is expected to fail. This problem should be
1 fixed in future releases.
1
1 6.4 Submitting test results
1 ===========================
1
1 If you want to report the results to the GCC project, use the
1 'contrib/test_summary' shell script. Start it in the OBJDIR with
1
1 SRCDIR/contrib/test_summary -p your_commentary.txt \
1 -m gcc-testresults@gcc.gnu.org |sh
1
1 This script uses the 'Mail' program to send the results, so make sure
1 it is in your 'PATH'. The file 'your_commentary.txt' is prepended to
1 the testsuite summary and should contain any special remarks you have on
1 your results or your build environment. Please do not edit the
1 testsuite result block or the subject line, as these messages may be
1 automatically processed.
1