diff options
| author | Luc Van Oostenryck <luc.vanoostenryck@gmail.com> | 2017-02-13 00:28:56 +0100 |
|---|---|---|
| committer | Christopher Li <sparse@chrisli.org> | 2017-02-13 09:38:12 +0800 |
| commit | 2958bb18c166468848d7046e04fbfcb3daba447e (patch) | |
| tree | c3aee97940c6117bdcd9045202aced98f2ceb2f3 /validation/test-suite | |
| parent | 9ab8a6a72c38f5fc3d4cb3f2a403f9ab83c3206f (diff) | |
| download | sparse-dev-2958bb18c166468848d7046e04fbfcb3daba447e.tar.gz | |
testsuite: report as error tests known to fail but which succeed
Such situation may simply show that what was tested is now fixed
and that it's juste the test annotation which need to be adapted,
but can be a sign that something else is broken.
Reporting the exact result (failure/success, known-to-fail/expect-to-succeed)
make the testsuite more useful and allow to use more efficiently
git-bisect or other automated testing tools.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
Signed-off-by: Christopher Li <sparse@chrisli.org>
Diffstat (limited to 'validation/test-suite')
| -rwxr-xr-x | validation/test-suite | 29 |
1 files changed, 21 insertions, 8 deletions
diff --git a/validation/test-suite b/validation/test-suite index 0d874e07..8692bfe9 100755 --- a/validation/test-suite +++ b/validation/test-suite @@ -11,6 +11,10 @@ if [ ! -x "$default_path/sparse-llvm" ]; then disabled_cmds="sparsec sparsei sparse-llvm" fi +# flags: +# - some tests gave an unexpected result +failed=0 + # counts: # - tests that have not been converted to test-suite format # - tests that are disabled @@ -163,18 +167,27 @@ do_test() test_failed=1 fi - if [ "$test_failed" -eq "1" ]; then - ko_tests=`expr $ko_tests + 1` - get_tag "check-known-to-fail" $file - if [ "$?" -eq "0" ]; then + get_tag "check-known-to-fail" $file + must_fail=`expr "$?" = 0` + known_ko_tests=`expr $known_ko_tests + $must_fail` + + [ "$test_failed" -eq "$must_fail" ] || failed=1 + + if [ "$must_fail" -eq "1" ]; then + if [ "$test_failed" -eq "1" ]; then echo "info: test '$file' is known to fail" - known_ko_tests=`expr $known_ko_tests + 1` + else + echo "info: test '$file' is known to fail but succeed!" + test_failed=1 fi - return 1 + fi + + if [ "$test_failed" -eq "1" ]; then + ko_tests=`expr $ko_tests + 1` else ok_tests=`expr $ok_tests + 1` - return 0 fi + return $test_failed } do_test_suite() @@ -273,5 +286,5 @@ case "$1" in ;; esac -exit 0 +exit $failed |