diff options
| author | Luc Van Oostenryck <luc.vanoostenryck@gmail.com> | 2018-02-24 18:43:50 +0100 |
|---|---|---|
| committer | Luc Van Oostenryck <luc.vanoostenryck@gmail.com> | 2018-05-21 02:49:43 +0200 |
| commit | b048f49bd28074bcbdc7506f1a61b48c842633ed (patch) | |
| tree | 0e65b4252c72cd1115911c3ed4c011d04b52d437 | |
| parent | dc9a969bdba0d48d36daf104d39a1390cbd83847 (diff) | |
| download | sparse-dev-b048f49bd28074bcbdc7506f1a61b48c842633ed.tar.gz | |
doc: the testsuite doc in reST
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
| -rw-r--r-- | Documentation/index.rst | 1 | ||||
| -rw-r--r-- | Documentation/test-suite | 151 | ||||
| -rw-r--r-- | Documentation/test-suite.rst | 169 |
3 files changed, 170 insertions, 151 deletions
diff --git a/Documentation/index.rst b/Documentation/index.rst index af807448..637c4435 100644 --- a/Documentation/index.rst +++ b/Documentation/index.rst @@ -18,6 +18,7 @@ Developer documentation .. toctree:: :maxdepth: 1 + test-suite dev-options IR diff --git a/Documentation/test-suite b/Documentation/test-suite deleted file mode 100644 index bf4518a2..00000000 --- a/Documentation/test-suite +++ /dev/null @@ -1,151 +0,0 @@ - - - Sparse test suite - ~~~~~~~~~~~~~~~~~ - -Sparse has a number of test cases in its validation directory. The test-suite -script aims at making automated checking of these tests possible. It works by -embedding tags in C comments in the test cases. - -check-name: <name> - Name of the test. This is the only mandatory tag. - -check-description: <description ...> - A description of what the test checks. - -check-command: <command arg ...> - There are different kinds of tests. Some can validate the sparse - preprocessor, while others will use sparse, cgcc, or even other backends - of the library. check-command allows you to give a custom command to - run the test-case. - The '$file' string is special. It will be expanded to the file name at - run time. - It defaults to "sparse $file". - -check-arch-ignore: <arch[|...]> -check-arch-only: <arch[|...]> - Ignore the test if the current architecture (as returned by 'uname -m') - match or not one of the archs given in the pattern. - -check-assert: <condition> - Ignore the test if the given condition is false when evaluated as a - static assertion (_Static_assert). - -check-cpp-if: <condition> - Ignore the test if the given condition is false when evaluated - by sparse's pre-processor. - -check-exit-value: <value> - The expected exit value of check-command. It defaults to 0. - -check-timeout: <timeout> - The maximum expected duration of check-command, in seconds. - It defaults to 1. - -check-output-start / check-output-end - The expected output (stdout and stderr) of check-command lies between - those two tags. It defaults to no output. - -check-output-ignore / check-error-ignore - Don't check the expected output (stdout or stderr) of check-command - (useful when this output is not comparable or if you're only interested - in the exit value). - By default this check is done. - -check-known-to-fail - Mark the test as being known to fail. - -check-output-contains: <pattern> - Check that the output (stdout) contains the given pattern. - Several such tags can be given, in which case the output - must contains all the patterns. - -check-output-excludes: <pattern> - Similar than the above one, but with opposite logic. - Check that the output (stdout) doesn't contain the given pattern. - Several such tags can be given, in which case the output - must contains none of the patterns. - -check-output-pattern(<nbr>): <pattern> -check-output-pattern(<min>,<max>): <pattern> - Similar to the contains/excludes above, but with full control - of the number of times the pattern should occur in the output. - If <min> or <max> is '-' the corresponding check is ignored. - - Using test-suite - ~~~~~~~~~~~~~~~~ - -The test-suite script is called through the check target of the Makefile. It -will try to check every test case it finds (find validation -name '*.c'). - -It can be called to check a single test with: -$ cd validation -$ ./test-suite single preprocessor/preprocessor1.c - TEST Preprocessor #1 (preprocessor/preprocessor1.c) -preprocessor/preprocessor1.c passed ! - - - Writing a test - ~~~~~~~~~~~~~~ - -test-suite comes with a format command to make a test easier to write: - - test-suite format file [name [cmd]] - -name: - check-name value. If no name is provided, it defaults to the file name. -cmd: - check-command value. If no cmd is provided, it defaults to - "sparse $file". - -The output of the test-suite format command can be redirected into the -test case to create a test-suite formatted file. - -$ ./test-suite format bad-assignment.c Assignment >> bad-assignment.c -$ cat !$ -cat bad-assignment.c -/* - * check-name: bad assignment - * - * check-command: sparse $file - * check-exit-value: 1 - * - * check-output-start -bad-assignment.c:3:6: error: Expected ; at end of statement -bad-assignment.c:3:6: error: got \ - * check-output-end - */ - -You can define the check-command you want to use for the test. $file will be -extended to the file name at run time. - -$ ./test-suite format validation/preprocessor2.c "Preprocessor #2" \ - "sparse -E \$file" >> validation/preprocessor2.c -$ cat !$ -cat validation/preprocessor2.c -/* - * This one we happen to get right. - * - * It should result in a simple - * - * a + b - * - * for a proper preprocessor. - */ -#define TWO a, b - -#define UNARY(x) BINARY(x) -#define BINARY(x, y) x + y - -UNARY(TWO) -/* - * check-name: Preprocessor #2 - * - * check-command: sparse -E $file - * check-exit-value: 0 - * - * check-output-start - -a + b - * check-output-end - */ diff --git a/Documentation/test-suite.rst b/Documentation/test-suite.rst new file mode 100644 index 00000000..333106ee --- /dev/null +++ b/Documentation/test-suite.rst @@ -0,0 +1,169 @@ +Test suite +########## + +Sparse has a number of test cases in its validation directory. The test-suite +script aims at making automated checking of these tests possible. It works by +embedding tags in C comments in the test cases. + +Tag's syntax +============ + +``check-name:`` *name* + + Name of the test. This is the only mandatory tag. + +``check-description:`` *description ...* + + A description of what the test checks. + +``check-command:`` *command arg ...* + + There are different kinds of tests. Some can validate the sparse + preprocessor, while others will use sparse, cgcc, or even other backends + of the library. check-command allows you to give a custom command to + run the test-case. + The ``$file`` string is special. It will be expanded to the file name at + run time. + It defaults to ``sparse $file``. + +``check-arch-ignore:`` *arch[|...]* + +``check-arch-only:`` *arch[|...]* + + Ignore the test if the current architecture (as returned by ``uname -m``) + matches or not one of the archs given in the pattern. + +``check-assert:`` *condition* + + Ignore the test if the given condition is false when evaluated as a + static assertion (``_Static_assert``). + +``check-cpp-if:`` *condition* + + Ignore the test if the given condition is false when evaluated + by sparse's pre-processor. + +``check-exit-value:`` *value* + + The expected exit value of check-command. It defaults to 0. + +``check-timeout:`` *timeout* + + The maximum expected duration of check-command, in seconds. + It defaults to 1. + +``check-output-start`` / ``check-output-end`` + + The expected output (stdout and stderr) of check-command lies between + those two tags. It defaults to no output. + +``check-output-ignore`` / ``check-error-ignore`` + + Don't check the expected output (stdout or stderr) of check-command + (useful when this output is not comparable or if you're only interested + in the exit value). By default this check is done. + +``check-known-to-fail`` + + Mark the test as being known to fail. + +``check-output-contains:`` *pattern* + + Check that the output (stdout) contains the given pattern. + Several such tags can be given, in which case the output + must contains all the patterns. + +``check-output-excludes:`` *pattern* + + Similar than the above one, but with opposite logic. + Check that the output (stdout) doesn't contain the given pattern. + Several such tags can be given, in which case the output + must contains none of the patterns. + +``check-output-pattern(``\ *nbr*\ ``):`` *pattern* + +``check-output-pattern(``\ *min*\ ``,``\ *max*\ ``):`` *pattern* + + Similar to the contains/excludes above, but with full control + of the number of times the pattern should occur in the output. + If *min* or *max* is ``-`` the corresponding check is ignored. + +Using test-suite +================ + +The test-suite script is called through the check target of the Makefile. It +will try to check every test case it finds (``find validation -name '*.c'``). +It can be called to check a single test with:: + + $ cd validation + $ ./test-suite single preprocessor/preprocessor1.c + TEST Preprocessor #1 (preprocessor/preprocessor1.c) + preprocessor/preprocessor1.c passed ! + + +Writing a test +============== + +The test-suite comes with a format command to make a test easier to write:: + + test-suite format [-a] [-l] [-f] file [name [cmd]] + +`name:` check-name value + If no name is provided, it defaults to the file name. + +`cmd:` check-command value + If no cmd is provided, it defaults to ``sparse $file``. + +The output of the test-suite format command can be redirected into the +test case to create a test-suite formatted file.:: + + $ ./test-suite format bad-assignment.c Assignment >> bad-assignment.c + $ cat !$ + cat bad-assignment.c + /* + * check-name: bad assignment + * + * check-command: sparse $file + * check-exit-value: 1 + * + * check-output-start + bad-assignment.c:3:6: error: Expected ; at end of statement + bad-assignment.c:3:6: error: got \ + * check-output-end + */ + +The same effect without the redirection can be achieved by using the ``-a`` +option. + +You can define the check-command you want to use for the test.:: + + $ ./test-suite format -a validation/preprocessor2.c "Preprocessor #2" \ + "sparse -E \$file" + $ cat !$ + cat validation/preprocessor2.c + /* + * This one we happen to get right. + * + * It should result in a simple + * + * a + b + * + * for a proper preprocessor. + */ + #define TWO a, b + + #define UNARY(x) BINARY(x) + #define BINARY(x, y) x + y + + UNARY(TWO) + /* + * check-name: Preprocessor #2 + * + * check-command: sparse -E $file + * check-exit-value: 0 + * + * check-output-start + + a + b + * check-output-end + */ |