| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
* add a symbolic checker
|
|
Some instruction simplifications can be quite tricky and thus easy to
get wrong. Often, they also are hard to test (for example, you can
test it with a few input values but of course not all combinations).
I'm used to validate some of these with an independent tool
(Alive cfr. [1], [2]) which is quite neat but has some issues:
1) This tool doesn't work with Sparse's IR or C source but it needs to
have the tests written in its own language (very close to LLVM's IR).
So it can be used to test if the logic of a simplification but
not if implemented correctly.
2) This tool isn't maintained anymore (has some bugs too) and it's
successor (Alive2 [3]) is not quite as handy to me (I miss the
pre-conditions a lot).
So, this patch implement the same idea but fully integrated with
Sparse. This mean that you can write a test in C, let Sparse process
and simplify it and then directly validate it and not only for
a few values but symbolically, for all possible values.
Note: Of course, this is not totally stand-alone and depends on
an external library for the solver (Boolector, see [4], [5]).
Note: Currently, it's just a proof of concept and, except the
included tests, it's only very slightly tested (and untested
with anything more complex than a few instructions).
[1] https://blog.regehr.org/archives/1170
[2] https://www.cs.utah.edu/~regehr/papers/pldi15.pdf
[3] https://blog.regehr.org/archives/1722
[4] https://boolector.github.io/
[5] https://boolector.github.io/papers/BrummayerBiere-TACAS09.pdf
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Because laziness is a virtue, add an option '-r' to the 'format'
subcommand of the testsuite to quickly create a test template for
linearized code which should just return 1.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
In testcases' tags, if a value contains 'check-' then this
value will be used as the tagname instead of the value.
Fix this by adding a bit more context in the regexp used for parsing these.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
The current tags check-output-contains/excludes/pattern are
quite powerful and the new check-output-match is easy to use
but it can be even simpler. Indeed, a lot of IR simplifications/
canonicalizations can be tested by checking that the expression
to be tested is equivalent to another one. This is less precise
than some more specific tests but:
* it's a big advantage because it's less sensitive to 'noise'
like the exact number used by the pseudos or to the results
of some new simplifications or canonicalizations
* very often, this equivalence is what really matters and not
the exact transformation.
So, add a new utra-simple-to-use tag: just ask that all functions
of the tests return the same specified value (usually 1):
check-output-returns: <value>
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
The current tags check-output-contains/excludes/pattern are
quite powerful, universal, but they often need 'complex' regular
expressions with escaping which make them not so nice to read.
For testing IR results, a very common pattern is:
this instruction must have this (kind of) operand.
So, make a new tag for this. It does nothing than can't be done
with done with the previous ones, on the contrary, but is much
simpler to use:
check-output-match(instruction): operand
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
This flag facilitates the creation of testcases for preprocessing.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
The subcommand 'format help' is broken because the of the way
arguments are parsed without validating the number of arguments.
Fix this by parsing all arguments (even if there is only one)
and validate the number of arguments at the end of the loop.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
For some testcases, the testsuite use the command 'timeout'
to ensure that the test finish after a reasonable amount of
time. This is mainly used for some testcases which, in the past,
were stuck in an infinite loop. This the command 'timeout' is
used with an extra option (-k 1s) to issue a second kill signal
in case the first one would have been ignored.
However, this extra option is not supported on all implementations
(Alpine) and its use seems a bit paranoid for sparse.
So, remove this extra option.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
This simplifies finding the offending test when the build ended with
KO: out of 584 tests, 527 passed, 57 failed
56 of them are known to fail
Signed-off-by: Uwe Kleine-König <uwe@kleine-koenig.org>
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
This allow, for example, to simulate running the testsuite on a 32bit
machine or running the testsuite with some extra debugging flags.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
It's certainly worth to have some tests but to not
slow down the testsuite and to not create a dependency
on python this test need to be run explicitely with:
./test-suite doc/cdoc.cdoc
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Ideally, the testcases should be universal but it happen than
some of them need to test some specificities or are meaningless
or plainly wrong in some situations. In such cases, these tests
must but ignored.
Currently, the only the only mechanisms a test are:
1) ignoring the tests depending on a tool which cannot be compiled
(like, for example, those using sparse-llvm when LLVM is not
installed.
2) some rather corse criteria using the name of the arch used
to run the tests.
Allow more flexibility by allowing to exclude some tests based on
the success or failure of an arbitrary condition via _Static_assert().
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Ideally, the testcases should be universal but it happen than
some of them need to test some specificities or are meaningless
or plainly wrong in some situations. In such cases, these tests
must but ignored.
Currently, the only the only mechanisms a test are:
1) ignoring the tests depending on a tool which cannot be compiled
(like, for example, those using sparse-llvm when LLVM is not
installed.
2) some rather corse criteria using the name of the arch used
to run the tests.
Allow more flexibility by allowing to exclude some tests based on
the evaluation of some pre-processor expression at test-time.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Since the patterns in the testcases are evaluated in the shell
script, the backslash used to escape characters special to the
pattern need itself to be escaped. Theer is a few cases where
it wasn't done so, partly because 'format -l' gave a single
escape in its template.
Fix all occurences neededing this double-escape as well as the
'format -l' template.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
"append" was used instead of "$append" when formatting
a new test, with the result that the infos for the test
system were always appended to the test fiel, which is
maybe often but not always desirable.
So, add the missing '$' when using the variable.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
During testing it's sometimes useful to force some default arguments
for all commands. An example of this is using '-m32' which essentially
allow to run the tessuite on an 64bit machine as-if run a 32-bit one.
Allow this by using the environment variable 'SPARSE_TEST_ARGS' to
hole default arguments for the test commands.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Commit 399c43889 (testsuite: get options from env too) allowed
the testsuite to takes extra options from the environment but
did it in a crude way involving exec.
Change this by using 'set --' instead of doing an 'exec'.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Currently the testsuite use 'eval echo $cmd' to expand
the name of the test file to be given on the command line.
This has the annoying consequence to go a bit too far in
the expansion of variables and to destroy any quotes and
whitespaces escaping that would have done.
Fix this by doing the eval later, when effectively executing
the command.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
The 'format' command create the information needed for the testcase
from the input file and output this on stdout. The developper must
then add this to the input file.
Let's do this automatically by adding an option '-a' to the 'format'
command to directly append the infos to the input file.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
The getopt loop used to bear by default and only some
options had to explicitly call 'shift' and 'continue'
to process further elements.
Change this to a 'normal' loop, shifting the next arg by default
and breaking of the loop when needed.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
During development, it is very useful to be able to run only
some of the tests, maybe a whole class.
Help this by allowing to run the testsuite on only a subdir
of the 'validation/' directory.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
The text for the testsuite usage used 'none' as if it was an
option/keyword while it only meant the absence of arguments.
Make the text clearer by removing the 'none' and being explicit
about the absence of arguments.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
This is a preparatory step to allow to run only a part
of the testsuite (a subdir).
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
This is a preparatory step to allow to run only a part
of the testsuite (a subdir).
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
This is a preparatory step to allow to run only a part
of the testsuite (a subdir).
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Making a typo in one of the 'check-...' tags can make
a testcase useless and thus incapable of detecting a regression.
Add some validation to these tags in order to detect wrong
tags.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
The flag 'quiet' is used to quiets unwanted error messages,
for example for testcases known to fail, but this flag is reset
too late so that the beginning of the next testcases will run
with the value for the previous case.
Fix this by reseting the flag at the begining of each testcase.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Allow this new helper to indicate wich file trigger the
warning and replace the existing call to 'echo "warning: ...'.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
So, we can use them inside get_tag_value().
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
It was too ugly (and a bit longish).
Remove it.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
as it can, for example, be needed via arg_file() -> error()
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Emit the short command witht he same sacing as done
in the kernel: 2 + 8.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
The old syntax is kept for now to avoid too much conflits
in pending branches.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Sometimes we want to only launch a few tests, not the whole
testsuite. the 'single' option allow to test a single case
but it's too restrictive.
Changes this by allowing arbirary arguments as files to be tested.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
POSIX tell that the interpretation of the '-n' is implementation
independent. So avoid it.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Tests taking a huge or infinite amout of time is
a problem for the testsuite. Furthermore, it's most
often the sign of a problem somewhere.
Fix this by adding some support to use an optional
timeout for each testcase command.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Some testcase (the ones related to sparse-llvm) are disabled if
the needed support is not present. This is done by checking
the name of the command used by the testcase.
The previous possible presence of './' before the command meant
that the command was checked via the 'basename' command, which
need to ne fork+execed.
Since the './' have now been stripped from all command names
this is no more needed.
Change this by comparing directly the command name.
This speedup the testsuite by another 6%.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
The tessuite use a few counters to keep track of things.
These counters are incremented via the 'expr' command.
But this command is not a shell builtin and need thus
to be fork+execed which need more CPU ressources than
using some builtin operation.
Change this by doing all the arithmetic via the portable
'$(( ... ))' shell builtin.
This speedup the testsuite by another 10%.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
For some testcase, it is checked if the output contains or not
some given pattern, or we're checking the number of time a
pattern is present in the output.
This is done with grep and some glue.
But for most testcases there is nothing to check and this
grepping is just wasted CPU cycles.
Fix this by using the already known tags to see if we need to
do this grepping or not.
This speedup the testsuite by another 15%.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Currently the testsuite always try to extract the expected
output from the test case description.
This is done with grep & sed.
But in some case we're not interested to compare the actual
output with the expected one. And in those cases the grep & sed
are just wasted CPU cycles.
Fix this by extracting the expected output only when we know
that it won't be ignored.
This speedup the testsuite by a modest 1%.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
The test cases contain annotations with the following:
check-<tagname>[: <value>]
These are extracted with grep & sed but this is done
separately for each tags which means that we need fork+exec
two processes for each possible tags.
Change this by trying to get all the tag+value in once by storing
the result in a variables instead of doing the grep & sed thing
at each time we need to check the tag.
This speedup the testsuite by around 30% for me.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
When a test succeed, remove all temporary/result files
but leave them for failed tests.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
Signed-off-by: Christopher Li <sparse@chrisli.org>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
Signed-off-by: Christopher Li <sparse@chrisli.org>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
Signed-off-by: Christopher Li <sparse@chrisli.org>
|
|
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
Signed-off-by: Christopher Li <sparse@chrisli.org>
|
|
Complement the 'check-output-contains/excludes' tags to also
be able to specify the number of times a given pattern should
occurs in the output.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
Signed-off-by: Christopher Li <sparse@chrisli.org>
|
|
Currently the test suite always check the exit value and the output
of the command used for the test. This is fine and allow use to catch
the most common situations:
- failure or crash (via the exit value)
- (un)expected output (like when testing the result of the preprocessor)
- (un)expected errors & warnings (like when testing sparse's warnings)
But sometimes, we're not interested in the output as such because it
can't be compared textually to some reference. This occurs systematically
when testing the output of test-linearize or test-unssa which emits
labels names which are in fact pointer values and which exact output
is very sensitive to any change in processing order, optimizations, ...
But useful tests can be easily made by just checking for the presence
or absence of some identifiers, or more generally some patterns.
This patch allow to do that by adding support for two new tags
(check-output-contains & check-output-excludes), telling to test suite
to verifiy that the given patterns are effectively present ot absent
from the output of the tested file.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
Signed-off-by: Christopher Li <sparse@chrisli.org>
|
|
The test suite must be run from the validation/ dir
which is sometimes slightly annoying.
Change that by letting the script to first do a cd to
the validation dir.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
Signed-off-by: Christopher Li <sparse@chrisli.org>
|
|
Such situation may simply show that what was tested is now fixed
and that it's juste the test annotation which need to be adapted,
but can be a sign that something else is broken.
Reporting the exact result (failure/success, known-to-fail/expect-to-succeed)
make the testsuite more useful and allow to use more efficiently
git-bisect or other automated testing tools.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
Signed-off-by: Christopher Li <sparse@chrisli.org>
|
|
Currently the test suite always check the exit value and the output
of the command used for the test. This is fine and allow use to catch
the most common situations:
- failure or crash (via the exit value)
- (un)expected output (like when testing the result of the preprocessor)
- (un)expected errors & warnings (like when testing sparse's warnings)
But sometimes, we're not interested in the output or the output (as is)
is simply not meaningful for the test or can't be compared textually
to some reference.
This patch add two new tags (check-output-ignore & check-error-ignore),
telling to test suite to ignore the content of stdout or stderr when
testing this file.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
Signed-off-by: Christopher Li <sparse@chrisli.org>
|
|
The use of the '==' operator in a test/[ conditional is a non-POSIX
bash extension. In order to avoid test failures on systems that do
not have bash as the system shell (/bin/sh), replace the use of the
'==' operator with the POSIX compatible '=' operator.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Christopher Li <sparse@chrisli.org>
|
|
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Christopher Li <sparse@chrisli.org>
|
|
Sparse is often share compile flags. So Werror is usually
mean gcc should treat warning as error. Apply the same
option to sparse will cause the Linux kernel checking fail
the build. We don't want that.
Rename the sparse option to -Wsparse-error. It allow caller
to control gcc and sparse behavior seperately.
It also make sparse return error status only when -Wsparse-error
is present.
Signed-off-by: Christopher Li <sparse@chrisli.org>
|
|
Make sparse fail and return an error code if a warning is encountered
and -Werror is specified or a hard error is found. This allows to use
sparse in automated build systems to more easily catch new sparse
warnings.
The validation script is extended to parse the expected output message
for an error message and validate the a non zero return value if such
a error message is found.
Also changes cgcc to die if the checker fails.
Signed-off-by: Thomas Graf <tgraf@suug.ch>
Signed-off-by: Christopher Li <sparse@chrisli.org>
|
|
This patch changes 'make check' output to show sparse output compared to
expected results upon unexpected test failure. For example,
static-forward-decl.c output would look like this if it would not be tagged as
"known to fail":
TEST static forward declaration (static-forward-decl.c)
error: actual error text does not match expected error text.
--- static-forward-decl.c.error.expected 2011-08-22 06:29:40.000000000 +0000
+++ static-forward-decl.c.error.got 2011-08-22 06:29:40.000000000 +0000
@@ -0,0 +1 @@
+static-forward-decl.c:3:5: warning: symbol 'f' was not declared. Should it be static?
error: see static-forward-decl.c.error.* for further investigation.
info: test 'static-forward-decl.c' is known to fail
This makes it easier to detect and analyze test breakage.
Cc: Christopher Li <sparse@chrisli.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
Signed-off-by: Christopher Li <sparse@chrisli.org>
|
|
Impact:
- On an 'unhandled' test issue a warning, and
- in case of a 'known to fail' test a info message.
Signed-off-by: Hannes Eder <hannes@hanneseder.net>
Signed-off-by: Christopher Li <sparse@chrisli.org>
|
|
test-suite merged stdout and stderr, which relied on the ordering of data from
the two streams in the merged stream. This made test-suite frequently fail on
tests with both output and errors, when the ordering didn't happen to match
what the test assumed. Handle each of the streams separately, and update the
tests accordingly.
Also clean up the output of "test-suite format" to avoid outputting values
equal to the defaults.
Signed-off-by: Josh Triplett <josh@freedesktop.org>
|
|
This patch introduces test-suite, a simple script that makes test cases
verification easier. Test cases in the validation directory are annotated
and this script parses them to check if the actual result is the one
expected by the test writer.
Signed-off-by: Damien Lespiau <damien.lespiau@gmail.com>
|
