1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
|
Test suite
##########
Sparse has a number of test cases in its validation directory. The test-suite
script aims at making automated checking of these tests possible. It works by
embedding tags in C comments in the test cases.
Tag's syntax
============
``check-name:`` *name*
Name of the test. This is the only mandatory tag.
``check-description:`` *description ...*
A description of what the test checks.
``check-command:`` *command arg ...*
There are different kinds of tests. Some can validate the sparse
preprocessor, while others will use sparse, cgcc, or even other backends
of the library. check-command allows you to give a custom command to
run the test-case.
The ``$file`` string is special. It will be expanded to the file name at
run time.
It defaults to ``sparse $file``.
``check-arch-ignore:`` *arch[|...]*
``check-arch-only:`` *arch[|...]*
Ignore the test if the current architecture (as returned by ``uname -m``)
matches or not one of the archs given in the pattern.
``check-assert:`` *condition*
Ignore the test if the given condition is false when evaluated as a
static assertion (``_Static_assert``).
``check-cpp-if:`` *condition*
Ignore the test if the given condition is false when evaluated
by sparse's pre-processor.
``check-exit-value:`` *value*
The expected exit value of check-command. It defaults to 0.
``check-timeout:`` *timeout*
The maximum expected duration of check-command, in seconds.
It defaults to 1.
``check-output-start`` / ``check-output-end``
The expected output (stdout and stderr) of check-command lies between
those two tags. It defaults to no output.
``check-output-ignore`` / ``check-error-ignore``
Don't check the expected output (stdout or stderr) of check-command
(useful when this output is not comparable or if you're only interested
in the exit value). By default this check is done.
``check-known-to-fail``
Mark the test as being known to fail.
``check-output-contains:`` *pattern*
Check that the output (stdout) contains the given pattern.
Several such tags can be given, in which case the output
must contains all the patterns.
``check-output-excludes:`` *pattern*
Similar than the above one, but with opposite logic.
Check that the output (stdout) doesn't contain the given pattern.
Several such tags can be given, in which case the output
must contains none of the patterns.
``check-output-pattern(``\ *nbr*\ ``):`` *pattern*
``check-output-pattern(``\ *min*\ ``,``\ *max*\ ``):`` *pattern*
Similar to the contains/excludes above, but with full control
of the number of times the pattern should occur in the output.
If *min* or *max* is ``-`` the corresponding check is ignored.
``check-output-match(``\ *start*\ ``):`` *pattern*
Check that in the output (stdout) all lines starting with the
first pattern also contains the second pattern. This should be
reserved for matching IR instructions since the '.$size' suffix
is ignored in the first pattern but is expected to be followed
by a space character.
``check-output-returns:`` *value*
Check that in the output (stdout) all IR return instructions
have the given value.
Using test-suite
================
The test-suite script is called through the check target of the Makefile. It
will try to check every test case it finds (``find validation -name '*.c'``).
It can be called to check a single test with::
$ cd validation
$ ./test-suite single preprocessor/preprocessor1.c
TEST Preprocessor #1 (preprocessor/preprocessor1.c)
preprocessor/preprocessor1.c passed !
Writing a test
==============
The test-suite comes with a format command to make a test easier to write::
test-suite format [-a] [-l] [-f] file [name [cmd]]
`name:` check-name value
If no name is provided, it defaults to the file name.
`cmd:` check-command value
If no cmd is provided, it defaults to ``sparse $file``.
The output of the test-suite format command can be redirected into the
test case to create a test-suite formatted file.::
$ ./test-suite format bad-assignment.c Assignment >> bad-assignment.c
$ cat !$
cat bad-assignment.c
/*
* check-name: bad assignment
*
* check-command: sparse $file
* check-exit-value: 1
*
* check-output-start
bad-assignment.c:3:6: error: Expected ; at end of statement
bad-assignment.c:3:6: error: got \
* check-output-end
*/
The same effect without the redirection can be achieved by using the ``-a``
option.
You can define the check-command you want to use for the test.::
$ ./test-suite format -a validation/preprocessor2.c "Preprocessor #2" \
"sparse -E \$file"
$ cat !$
cat validation/preprocessor2.c
/*
* This one we happen to get right.
*
* It should result in a simple
*
* a + b
*
* for a proper preprocessor.
*/
#define TWO a, b
#define UNARY(x) BINARY(x)
#define BINARY(x, y) x + y
UNARY(TWO)
/*
* check-name: Preprocessor #2
*
* check-command: sparse -E $file
* check-exit-value: 0
*
* check-output-start
a + b
* check-output-end
*/
|
