Content uploaded by Alain Denise
Author content
All content in this area was uploaded by Alain Denise
Content may be subject to copyright.
L R I
A GENERIC METHOD FOR STATISTICAL
TESTING
DENISE A / GAUDEL M C / GOURAUD S D
Unité Mixte de Recherche 8623
CNRS-Université Paris Sud – LRI
04/2004
Rapport de Recherche N° 1386
CNRS – Université de Paris Sud
Centre d’Orsay
LABORATOIRE DE RECHERCHE EN INFORMATIQUE
Bâtiment 490
91405 ORSAY Cedex (France)
A Generic Method for Statistical Testing
A. Denise, M.-C. Gaudel and S.-D. Gouraud
email= denise,mcg,gouraud @lri.fr
L.R.I., Universit´e Paris-Sud,
91405 Orsay Cedex, France.
Abstract
This paper addresses the problem of selecting finite test
sets and automating this selection. Among these methods,
some are deterministic and some are statistical. The kind of
statistical testing we considerhas been inspired by the work
of Thevenod-Fosse andWaeselynck. There, the choice of the
distribution on the input domain is guided by the structure
of the program or the form of its specification.
In the present paper, we describe a new generic method
for performing statistical testing according to any given
graphical description of the behavior of the system under
test. This method can be fully automated. Its main origi-
nality is that it exploits recent results and tools in combina-
torics, precisely in the area of random generation of combi-
natorial structures.
Uniform random generation routines are used for draw-
ing paths from the set of execution paths or traces of the
system under test. Then a constraint resolution step is per-
formed, aiming to design a set of test data that activate the
generated paths. This approachapplies to a number of clas-
sical coverage criteria. Moreover, we show how linear pro-
gramming techniques may help to improve the quality of
test, i.e. the probabilities for the elements to be covered by
the test process.
The paper presents the method in its generality. Then,
in the last section, experimental results on applying it to
structural statistical software testing are reported.
1 Introduction
In the area of software testing, numerous methods have
been proposed and used for selecting finite test sets and au-
tomating this selection. Among these methods some are
deterministic and some are probabilistic. Depending on the
authors, methods of this last class are called random testing
or statistical testing.
Random testing as in [11, 12, 28], consists in selecting
test data uniformly at random from the input domain of the
program. When the random selection is based on some op-
erational profile, it is sometimes called statistical or opera-
tional testing and can be used to make reliability estimates
[26]. In this article, the kind of statistical testing we con-
sider has been inspired by the work of Th´evenod-Fosse and
Waeselynck [31]. There, the choice of the distribution on
the input domain is guided by some coverage criteria of ei-
ther the program (structural statistical testing) or some spec-
ification (functional statistical testing).
In the recent years, the general problem of studying
and simulating random processes has particularly benefit-
ted from progresses in the area of random generation of
combinatorial structures. The seminal works of Wilf and
Nijenhuis in the late 70’s [35, 27] have led to efficient al-
gorithms for generating uniformly at random a variety of
combinatorial structures. In 1994, Flajolet, Zimmermann
and Van Cutsem [13] have widely generalized and system-
atized the approach. Briefly, their approach is based on a
non-ambiguous recursive decomposition of the combinato-
rial structures to be generated. Their work constitutes the
basis of powerful tools for uniform random generation of
complex entities, as graphs, trees, words, paths, etc. In the
present paper, we explore the idea of using such concepts
and tools for random software testing.
Actually, there are several ways to use uniform genera-
tion in the area of testing.
As mentioned above, a natural idea is to uniformly draw
data from the input domain. This approach of random test-
ing, was studied some time ago for numerical data [11, 12],
and turned out to have an uneven detection power when ap-
plied to realistic complex programs [4, 31]. This is con-
firmed by recent comparisons and evaluations [28].
In this paper, we follow another idea: We describe a
generic method for using these tools as soon as there is a
graphical description of the behavior of the system under
test. It may be the control graph of the program, or some
specification, either directly graphical (State-charts, Petri
nets) or indirectly via some semantics in terms of transi-
tion systems, automaton, state machines, Kripke structures,
etc. Such behavioral graphs can be described as combina-
0
1
2
3 4 7
6
5
b
a
i
c
d
e
g
f
h
j
k
Figure 1 : A graph with starting and ending vertices
torial structures. Therefore, uniform generation can be used
for drawing paths from the set of execution paths or traces
of the system under test, or, more efficiently, among some
subsets satisfying some coverage conditions.
Our approach was sketched in a previous paper [16],
where we presented a first example for structural statisti-
cal testing. This paper presents the method in its generality,
and avoids the heuristic used in [16] for the implementation
of the all-statements and all-branches criteria.
The paper is organized as follows: Section 2 presents
our general method to count and uniformly draw paths in a
graph, based on some translation into a combinatorial struc-
ture specification; Section 3 recalls some basic notions of
statistical testing, coverage criteria and test quality; In Sec-
tion 4 we present a path generation scheme guided by test
quality, andin Section 5 we discuss the issue of deriving test
inputs, once a set of paths has been generated. In the two
last sections, we give some experimental results on apply-
ing our method to structural statistical testing and we sketch
some perspectives.
2 Combinatorial preliminaries
We present here some combinatorial concepts and meth-
ods which will be used in the sequel of the paper. Let us
consider a connected directed graph
where vertices, as
well as edges, are labeled in such a way that any two dis-
tinct vertices (resp edges) have distinct labels. Furthermore,
there exist two vertices (starting vertex) and (ending
vertex) such that, for any vertex , there exists a path from
to and a path from to in . Figure 1 presents such
a graph, where vertices are labeled with numbers from to
and edges are labeled with letters from ’a’ to ’k’; vertices
and are the starting and ending vertices respectively. If
is a positive integer, (resp. ) denotes the set of
paths of length (resp. whose length is ) in from
to , and denotes the whole (possibly infinite) set of
paths from to .
2.1 Uniform random generation of paths in a
graph
Our aim is, given an integer , to generate uniformly at
random (u.a.r.) one or several paths of length from
to . Uniformly means that all paths in have the
same probability to be generated. At first, let us focus on a
slightly different problem: the generation of paths of length
exactly. We will see further that a small change in the
graph allows to generate paths of length . Remark that
generally the number of paths of length grows exponen-
tially with .
The principle of the generation process is simple: Start-
ing from vertex , draw a path step by step. At each step,
the process consists in choosing a successor of the current
vertex and going to it. The problem is to proceed in such a
way that only (and all) paths of length can be generated,
and that they are equiprobably distributed. This is done by
choosing successors with suitable probabilities. Given any
vertex , let denote the number of paths of length
which connect to the end vertex . Suppose that, at
any step of the generation, we are on vertex which has
successors denoted , , ..., . In addition, suppose that
edges remain to be crossed in order to get a path
of length to . Then the condition for uniformity is that
the probability of choosing vertex equals
. In other words, the probability to go to
any successor of must be proportional to the number of
paths of suitable length from this successor to .
So we need to compute the numbers for any
and any vertex of the graph. This can be done by
using the following recurrence rules:
if
otherwise
for
where means that there exists an edge from to
(note that may be equal to if loops are allowed in the
graph). Table 1 presents the recurrence rules which corre-
spond to the graph of Figure 1.
Now the generation scheme is as follows:
Preprocessingstage: Compute a tableofthe ’s for
all and all vertices.
Generation stage: Draw the path according to the
scheme seen above.
Note that the preprocessing stage must be done only once,
whatever thenumberof paths to be generated. Easy compu-
tations show that the memory space requirement is
integer numbers, where stands for the number of ver-
tices in the graph. The number of arithmetic operations
2
Table 1 : Recurrences for the .
needed for the preprocessing stage, as well as for the gen-
eration stage, is linear in and in . This ensures a very
efficient generation process.
Now we address the problem of generating paths of
length instead of exactly . The only change is the fol-
lowing: Add to the graph a new vertex which becomes
the new start vertex, with an edge from to and a loop
edge from to itself. Each path of length from to
in this new graph crosses times the loop edge for some
such that and once the one from to . With
this path we obviously associate a path of length in
the previous graph. It is straightforward to verify that any
path of length can be generated in such a way, and the
generation is uniform.
Note that the above developments are a special case of
a general method of generation of combinatorial structures,
which has been first addressed by Wilf [35] and then gen-
eralized and systematized by Flajolet, Zimmermann and
Van Cutsem [13]. More precisely, the problem of gener-
ating paths of a given length in is equivalent to the one
of uniform randomgeneration of words of so-called regular
languages, which has first been discussed in [17]. Indeed,
a regular language is defined by a particular labeled graph
called finite state automaton, and any word of the language
corresponds to a path in the automaton. We show in Table
2 the set of words which correspond to the paths of length
10 of the graph of Figure 1.
In our implementation, the generation of paths is pro-
grammedin MuPAD, using the CS package. MuPAD[29] is
a formal and algebraic calculus tool, developed at the Uni-
versity of Paderborn. CS [8, 9], is a package devoted to
countingandrandomlygenerating combinatorial structures,
based on the general notion of “decomposable structures”
defined in [13]. CS is now part of the MuPAD-Combinat
package [18] which is freely available at the following ad-
dress:
http://mupad-combinat.sourceforge.net/
length words
3 bdk
4 acfk, bdkj
5 acegj, acfhj
7 bdhicfk
8 acegicfk, acfhicfk, bdhicegj, bdhicfhj
9 acegicegj, acegicfhj, acfhicegj, acfhicfhj
Table 2 : The 14 paths of length from vertex to vertex
.
2.2 Constraints on paths and graphs transforma-
tions
As we will see in Section 4, ourmethodofstatistical test-
ing involves counting and random generation of paths sub-
ject to additional constraints. In this subsection, we show
how to change the graph in order to take into account such
constraints.
Let us focus first on a rathersimple constraint: we aim to
construct, given a labeled connected graph and an edge
label , a graph whose set of paths is equal to the set of
paths of which cross edge labeled . This can be done by
using the following procedure:
1. Create a copy
of graph , in which the edges are
labeled exactly as the edges of , and in which any
vertex label in becomes in .
2. Suppose that the edge labeled joins vertex to vertex
in G. Then delete this edge and replace it with a new
edge labeled between vertex (in ) and vertex
(in ).
3. set as the ending vertex, instead of (but re-
mains the start vertex.)
4. Delete all the vertices (and their adjacent edges) to
which no path from exists.
5. Delete all the vertices (and their adjacent edges) from
which no path to exists.
This concludes the construction of . Figure 2 shows the
result of the procedure, given the graph of Figure 1 and the
edge labeled ’e’. This transformation can be done in linear
time and linear memory requirement with respect to the size
ofthegraph. Note that steps 4 and 5 are not mandatory: they
are used only to ”clean” the final graph by deleting useless
elements.
Like in the previous subsection, this process can be
stated in terms of operations on regular languages: the
graph may be seen as a finite automaton of the regu-
lar language which is the intersection of the language of
and the (regular) language of words which contain at least
3
c
e
f
h
j
k
0
1
2
3
6
5
b
a
i
c
f
h
d
i
g
e
1’
3’ 4’
6’
7’
5’
Figure 2 : Graph which contains only the paths of the graph of
Figure 1 which cross edge labeled ’e’.
once the letter
. This approach can be generalized in or-
der to perform more complextransformations of graph as
for example, constructing a graph which contains the paths
which cross exactly times a given edge, or which cross
two or more given edges, or which take times a given cy-
cle in the graph. Roughly, it suffices to be able to express
the desired constraint in terms of a regular language, i.e.
to design a regular expression or a finite automaton which
recognizes the whole set of words which satisfy the con-
straint. Then a standard algorithm for intersecting regular
languages (see e.g. [20]) gives
. In our special case, this
general method consists exactly in the procedure described
above. Very similar procedures apply if we are given ver-
tices instead of edges.
3 Coverage criteria and statistical testing
The idea of combining coverage criteria and random
testing aims at overcoming some drawbacks of both ap-
proaches.
Applying coverage criteria corresponds to a decomposi-
tion of the input domain into some (very often non disjoint)
sub-domains: Each element to be covered defines a sub-
domain that is the subset of the inputs that cause its execu-
tion. The main drawback here is that these sub-domains are
generally not homogeneous, i.e. some of their inputs may
result in a failure, and some others may yield correct results.
Random testing lessens this drawback since it allows in-
tensive test campaigns where the same element of the pro-
gram may be executed several times with different data.
However, in its pure uniform version it induces a bad cov-
erage of cases corresponding to small input sub-domains.
In [31, 32, 33, 34], Th´evenod-Fosse and Waeselynck de-
veloped a statistical testing method where the input distri-
bution takes into account some coverage criteria in order
to avoid the existence of low probability cases. They have
reported several experiments, which led to the conclusion
that their approach has a better fault detection power than
uniform random testing and deterministic testing based on
classical coverage criteria. However, the construction of
the input distribution is difficult since it requires the reso-
lution of as many equations as paths in the program (or the
specification). For large programs, or in presence of loops,
the construction is empirical, basedon preliminary observa-
tions of the behavior of the program [34].
Here, we avoid the construction of the distribution by
using the tools presented in Section 2 for generating paths.
Before presenting our approach in detail, we must state pre-
cisely what it means for a statistical method to take into
account a coverage criteria.
A notion of test quality for statistical testing methods has
been defined first in [30]. We slightly reformulate it for our
context.
Let
be some description of a system under test.
may be a specification or a program, depending on the kind
of test we are interested in (functional or structural). We
assume that is based on a graph (or, more generally, on
some kind of combinatorial structure). On the basis of this
graph, it is possible to define coverage criteria: all-vertices,
all-edges, all-paths-of a certain-kind, etc. More precisely, a
coverage criterion characterizes for a givendescription
a set of elements of the underlying graph (noted
in the sequel when and are obvious). In the case of de-
terministic testing, the criterion is satisfied if every element
of the set is exercised by at least one test.
In the case of statistical testing, the satisfaction of a cov-
erage criteria
by a testing method for a description is
characterizedby the minimal probability of cover-
ing any element of when drawing tests. In [30],
is called the test qualityof the method with respect
to .
The test quality can be easily stated if
is known. Indeed, one gets ,
since when drawing tests, the probability of reaching an
element is one minus the probability of not reaching it
times.
Let us come back to the example of Section 2, where the
set of all paths of Figure 1 has been expressed as a specifi-
cation of some combinatorial structure, and the CS system
is used for uniformly drawing among paths of length .
Let us note the set of such paths, as in Section 2. Con-
sidering the coverage criterion “all paths of length ”,
4
noted below , we get the following test quality:
In the example, choosing allows the coverage of all
elementary paths. Since there are 14 paths of length less or
equal to 10 (see Table 2) we have:
Table 3 gives the number of tests requiredfor four values
of test quality, for the criterion “all paths of length ”.
0.9 0.99 0.999 0.9999
32 63 94 125
Table 3 : Number of tests required for a test quality
The assessment of test quality is more complicated in
general. Let us consider more practicable coverage criteria,
such as “all-vertices” or “all-edges”, and some given statis-
tical testing method. The elements to be covered generally
have different probabilities to be reached by a test. Some of
them are covered by all the tests, for instance the initial and
terminalvertices and mentioned in Section 2. Some of
them may have a very weak probability, due to the structure
of the behavioral graph or to some specificity of the testing
method. For instance, in our example edges
and appear
in 5 paths of length only. Edges and appear in
9 such paths. It means that drawing uniformly from
leads to a probability of to reach edge , and to reach
edge .
Let and for any
the probability for the element to be exercised during the
execution of a test generated by the considered statistical
testing method. Then
(1)
where . Consequently, the
number of tests required to reach a given quality
is
By definition of the test quality, is just .
Thus, from the formula above one immediately deduces that
for any given , for any given , maximizing the quality
of a statistical testing method with respect to a coverage
criteria reduces to maximizing , i. e. .
4 Generation of paths guided by the QoT
4.1 General scheme.
Now let us consider a given coverage criterion . As
a preliminary remark, note that the set of elements
must be finite, otherwise the quality of test would be zero.
This implies, in particular, that the coverage criterion “all
paths” is irrelevant as soon as there is a cycle in the descrip-
tion, like in our example (figure 1). Thus, this criterion has
to be bounded by additional conditions, for example “all
paths of length ”, “all paths of length between given
and ”, or “all paths which take at most times each cy-
cle in the graph”. For the sake of simplicity, we consider in
the following that paths are generated within , the set
of paths of length that go from to .
We consider two cases, according to the nature of the el-
ements of . If denotes a set of paths in the
graph, we immediately state that the quality of test is opti-
mal if the paths of are generated uniformly, i.e. any
path has the same probability to be generated.
Indeed, if the probability of one or several paths was greater
than , then there would exist at least one path
with probability less than , therefore the quality
of test would be lower. We saw in Section 2.1 how to gen-
erate uniformly random paths of given length in a graph,
and how to modify the graph in order to fit with the crite-
rion “all paths of length ”. The method easily applies
to other criteria that involve paths, as those given above, by
ways similar to the ones seen in Section 2.2.
Now, weconsiderthe case wherethe elements of
are not paths, but are constitutive elements of the graph as,
for example, vertices, edges, or cycles. Clearly, uniform
generation of paths does not ensure optimal quality of test
in this case. Ideally, the distribution on paths should en-
sure that the minimal probability to reach any element of
is maximal. Unfortunately, computing this distri-
bution would require the resolution as as many equations as
paths. This is generally impracticable. Thus we propose to
generate a path in two steps:
1. pick at random one element of , according to
a suitable probability distribution (which will be dis-
cussed in Section 4.2);
2. generate uniformly at random one path of length
that goes through . (This ensures a balanced coverage
of the set of paths which cross .)
Algorithms for achieving the second step are detailed in
Section 2. The next subsection deals with the first step.
5
4.2 Probabilitydistributionforanoptimal quality
of test.
Theproblemconsistsinchoosingthesuitableprobability
distribution over in order to maximizethe quality of
test. Given , with , the
probability for a the element (for any in ) to
be reached by a path is
where
is the number of paths of which takes element
;
is the number of paths which take both elements
and ; (note that = and = );
is the probability of choosing element during step
1 of the above process.
Indeed, the probability of choosing element in step 1 is
; and the probability of reaching by drawing a random
path which goes through another element is . The
above equation simplifies in
(2)
since . Note that coefficients and are easily
computed by ways given in Section 2.
Now we have to determine probabilities
with , which maximize
. This can be stated as a linear
programming problem:
Maximize under the constraints:
where the ’s are computed as in Equation (2). Standard
methods lead to a solution in time polynomial according to
.
Let us illustrate this with our example. Given the cov-
erage criterion “all the edges” and given , Table 4
presents the coefficients , where and denote letters
from ’a’ to ’k’. For example, the value ’9’ in row ’f’ and
column ’c’ means that , i.e. there are exactly 9
paths of length lower or equal to 10 from to which
cross both edges and in the graph of Figure 1.
The corresponding linear program is shown in Table 5.
Each line, but the last one, is an inequation which corre-
sponds to a row in Table 4. The first term of the inequation
is , the value to be maximized. The second term is one
a b c d e f g h i j k
a 9 0 9 0 5 7 5 5 6 6 3
b 0 5 3 5 1 2 1 4 3 3 2
c 9 3 12 3 6 9 6 8 9 8 4
d 0 5 3 5 1 2 1 4 3 3 2
e 5 1 6 1 6 3 6 3 5 5 1
f 7 2 9 2 3 9 3 7 7 5 4
g
5 1 6 1 6 3 6 3 5 5 1
h 5 4 8 4 3 7 3 9 7 7 2
i 6 3 9 3 5 7 5 7 9 6 3
j 6 3 8 3 5 5 5 7 6 9 0
k 3 2 4 2 1 4 1 2 3 0 5
Table 4 : Table of the .
of the ’s, computed according to Formula 2. For exam-
ple, the first line means that must be lower or equal to
, the probability of reaching edge ’a’ with a random path.
By maximizing , one maximizes the lowest , so that
the quality of test is optimal. The last line ensures that the
probabilities that we are searching for sum to 1.
Solving this linear program leads to
, while
and . This gives , therefore the optimal
quality of test equals , according to Formula 1.
5 From paths to input data
So farwe havepresenteda generic methodforgenerating
execution paths in a way that maximizes test quality. This
method relies on existing algorithms and tools and can be
fully automated. A last step is to generate, for every path,
input values that will cause its execution.
5.1 The trivial case of finite models
First let us consider thecase where the graphical descrip-
tion corresponds to a finite model of the system under test
(Finite State Automaton, Finite State Machine, etc) [7]. Ev-
ery edge of the description is labeled by some symbol of
a finite alphabet that represents some input or event. This
symbol may be coupled with some other symbol indicating
some expected reaction (output, action). Coverage criteria
typically used in such cases are variants of transition cover-
age, the main problem being state identification before and
after a tested transition. The method described in Section 4
can be used to draw paths for such coverage criteria. Then,
given a path, the test data for executing it is just the se-
quence of inputs labeling its edges, possibly followed by
some additional inputs in order to observe that the system
under test is in the expected state [21, 6].
6
Table 5 : The linear program.
5.2 The general case of infinite models
The problem is more difficult as soon as the model un-
derlying the description is not finite [21]. It is the case
for various sorts of Extended Finite State Machines [19],
State-charts [5], or the control graph of pieces of code [16],
namely any description including non-trivial data types and
guards.
An example is given in Figure 3, using a notation
close to UML state charts. It is presented in details
in [22]. In this example, the variable is of a given
type ; every message has a priority. The
variable is of type . The opera-
tion returns the oldest message of the queue with the
best priority. The boxes labeled by and
denote infinite classes of states, (as
many states as possible values for the variable).
A possible trace (or more exactly class of traces) is:
Buffer(Q)
!Q.get()/Q.remove()
?ready[not Q.isEmpty()]/_
?M/Q.add(M)
actionguardevent
_/Q.init()
?M/Q.add(M)
ClientReady(Q)
Figure 3 : A state-chart specification of a buffer with priorities
Given some path or some trace made of conditioned
statements or guarded commands, how to find some inputs
triggering its execution? It is a classical issue in struc-
tural testing, or in functional testing based on specifica-
tions with data types. Constructing, via symbolic evalua-
tion techniques, the predicate characterizing the input do-
main of the path, can solve it: This predicate is the con-
junction of the guards (or conditions) encountered on the
path, adequately updated in function of the variables as-
signments (see for instance [16]). Then the problem re-
duces to a constraint-solving problem. Any data satisfy-
ing the above predicate is an input executing the path. At
this stage, the tool to be used is highly dependent on the
kind of guards and data types allowed in the description:
There exist a lot of specialized constraint solvers for var-
ious types of variables and constraints, which ensure ter-
mination and completeness. However, in full generality
the problem is only semi-decidable and a general-purpose
solver may not terminate when searching for a solution.
Lately, significant advances have been achieved with the in-
troduction of powerful heuristics and randomization tech-
niques, such as those used in the LOFT and GATEL tools
[23, 24] or the BZ-tools in [2]. Other uses of constraint
solvers for test generation are reported in [3, 14, 25]. A
classical difficulty at this stage is that unfeasible paths may
arise. For instance, in the example of Figure 3, all paths
beginning by
are
unfeasible since the method assigns an empty state to
the Queue. In the next section we show how we cope with
this problem in the prototype that we have developed for
structural statistical testing.
7
6 The AuGuSTe prototype
In order to validate the applicability of our approach,
Sandrine Gouraud has developed a tool for statistical struc-
tural testing [15]. The programs under test are written in
a small programming language inspired from C. The data
types are booleans, integers, and arrays. The constraint
solver is the one of GATEL [24], extended to arrays.
When an unfeasiblepathis detected (or suspected) by the
constraint solver, it is rejected and another path is drawn.
This strategy does not affect the uniform distribution on
paths: any feasible path is still drawn with uniform prob-
ability. This ensures that, if the coverage criterion involves
paths only (like e.g. ”all paths of length ”), the quality
of test stays optimal. However, in other cases, it may de-
crease with regard to its theoretical value, depending on the
distribution of unfeasible paths in the graph.
Actually, our first experiments with AuGuSTe show that
the difference may be significant in presence of big numbers
of unfeasible paths. We are currently investigating methods
for improving this experimental quality of test. For exam-
ple, in some cases a number of unfeasible paths can be de-
tected by static analysis of the description of the system.
Then the combinatorial specification of the graph can be
modified in order to avoid these paths.
lines paths coverage cardinality
criterion of
FCT1 30 17 all paths 17
FCT2 43 9 all paths 9
FCT3 135 33 all paths 33
FCT4 77 all branches 41
Table 6 : The four tested programs
Our tool has been used for testing the same set of four
C programs as in [33] (see also table 6), where Pascale
Th´evenod-Fosse, H´el`ene Waeselynck and Yves Crouzet
presented the first experimental evaluation of the detection
power of statistical structural testing. Thanks to them, it
was possible to reuse the same sets of mutants and to replay
almost the same set of experiments. In [33], the statistical
method is different from here, since it is based on the ex-
plicit construction of a distribution on the input domain, ei-
ther analytically, or empirically (when there is some loop).
In our case, we draw paths and then use constraint-solving
tools to produce inputs. Of course, this induces a distri-
bution on the input domain. As this distribution is highly
dependent on the implementation of the constraint solver,
it remains implicit. The only way to compare the detection
power of the two methods is by experiments. Despite the
difference between the two approaches, the results of the
experiments are quite similar (see Table 8), with the advan-
tage that our new approach is fully automated.
More than 10000 experiments were performed on 2914
mutants (see Table 7). Test data were generated in order to
obtain a quality of test of
. The test generation
time for the programs FCT1, FCT2 and FCT3 was few min-
utes. The differences between the mutation scores obtained
for FCT3 (see Table 8) are mainly due to a problem of non
independence of the test experiments because some global
variables are not initialized in this program.
tests mutants
FCT1 170 279
FCT2 80 563
FCT3 1467
FCT4 605
Table 7 : Number of mutants and tests for each program
Mutation scores
LAAS LRI
FCT1 1 1
FCT2 1 1
min=1 min=0.9951
FCT3 exp=1 exp=0.9989
max=1 max=1
min=0.9898 min=0.9854
FCT4(1) exp=0.9901 exp=0.9854
max=0.9915 max=0.9854
min=0.9898 min=0.9634
FCT4(2) exp=0.9901 exp=0.9762
max=0.9915 max=0.9854
Table 8 : Mutation scores
The fourth program, FCT4, was the most difficult and
the most interesting. It contains both a loop and a huge
number of unfeasible paths. The coverage criterion was “all
the edges with maximal path length of 234”, in number of
edges of the control graph (thus much more in number of
statements). The length 234 was chosen according to the
characteristics of the loop. Consequently, the predicates to
be solved were rather long too: At least one out of two ver-
tices on a path corresponds to a decision point, thus to a
condition to be added to the predicate.
In order to reduce the number of unfeasible paths, we
have adapted the combinatorial structure according to the
characteristics of the program. This modification can be
done by some simple static analysis, without altering the
program. This manipulation reduces the test generation
time (drawing paths and solving predicates) from a week
8
to two hours. The preprocessing stage (construction of the
combinatorial structure and counting the number of paths)
lasted two days: it means constructing the table for
41 edges. This preprocessing stage could be improved by
avoiding unnecessary intersections, using dominator tree
[1] for determining equivalences between , and optimiz-
ing combinatorial structures. We are currently working in
this direction.
The obtained mutation scores are presented in Table 8,
line FCT4(1). The line FCT4(2) correspond to mutation
scores obtained by using for the
a distribution based on
the heuristic presented in [16]. The results are slightlylower
but the preprocessing stage lasts one hour only.
These first experiments let think that the method scales
up well.
7 Conclusion and perspectives
We have shown how uniform generation of combina-
torial structures can be used for statistical testing as soon
as some graphical description of the program under test is
available. If the description is at the program level (control
flow graph), our method applies to structural statistical test-
ing. If the description is at the specification level, it applies
to functional statistical testing.
This paper brings two main novelties with respect to our
previous paper [16], where we presented a first application
of combinatorial methods to the main criteria of structural
testing: First, we give a generalization of the method to any
graphical description and to any coverage criteria ; More-
over, in Section 4, we show how to build a probability distri-
bution on the elements to be covered for any given criteria.
This distribution ensures an optimal quality of test when
associated with the methods of Section 2.2 for randomly
generating paths constrained to cross these elements. This
replaces the heuristic used in [16] for the implementation of
the all-statements and all-branches criteria.
As reported in Section 6, this approach has been vali-
dated on realistic examples.
It exhibits a similar detection power as Th´evenod-Fosse
andWaeselynck’s method, which is betterthan pure random
testing or deterministic testing.
More generally, we think that this approach provides a
basis for a new class of tools in the domain of testing, com-
bining random generation of combinatorial structures, lin-
ear programming techniques, and constraint solvers.
Some interesting perspectives are still open. The CS tool
can deal with languages more complex than regular lan-
guages (for instance, with cardinality constraints such as
paths with the same number of iterations in loop 1 and loop
2). Practically, it means that it could be possible to com-
pile behavioral graphs into more elaborated combinatorial
structures, taking into account some knowledge on the sys-
tem under test, or some results of static analysis. This could
improve significantly the efficiency of the tools, by elimi-
nating some major sources of unfeasible paths.
Another possibility worth to explore is the use of the new
approach proposed recently by Flajolet & al. [10] for ran-
dom generation of combinatorial structures : It is based on
Boltzmann models and could avoid the introduction on a
bound on the length of the considered paths.
Acknowledgments
We are indebted to Claire Kenyon for some fruitful dis-
cussions on the optimization part of this work. For our
experiments we wish to acknowledge : the testing group
in LAAS for providing us the library of mutants, Sylvie
Corteel for the combinatorial structures library and Bruno
Marre for his help in extending and using the constraint
solver. This work was partially funded by the European
community (IST Project 1999-11585: DSoS).
References
[1] H. Agrawal. Dominators, super blocks, and program cov-
erage. In Proceedings of Principles of Programming Lan-
guages (POPL’94), pages 25–34, 1994.
[2] F. Ambert, F. Bouquet, S. Chemin, S. Guenaud, B. Legeard,
F. Peureux, M. Utting, and N. Vacele. BZ-testing-tools: A
tool-set for test generation from Z and B using constraint
logic programming. In FATES’02, Formal Approaches to
Testing of Software, Workshop of CONCUR’02, pages 105–
120, Brn, Czech Republic, August 2002.
[3] F. Barray, P. Codognet, D. Diaz, and H. Michel. Code-based
test generation for validation of functionnal processor de-
scriptions. In Proceedings TACAS 2003, Springer Verlag,
pages 569–584. january 2003. LNCS 2619.
[4] P. Bishop, D. Esp, M. Barnes, P. Humphreys, G. Dahll,
and J. Lahti. PODS-A project on diverse software. In
IEEETransactions on Software Engineering, volume SE-12,
pages 929–941, 1986.
[5] L. C. Briand, J. Cui, and Y. Labiche. Towards auto-
mated support for deriving test data from UML statecharts.
In Proceedings of ACM/IEEE International Conference on
the Unified Modeling Language (UML’03), pages 249–264,
2003.
[6] E. Brinskma and J. Tretmans. Testing transition systems:
An annotated bibliography. In LNCS, volume 2067, pages
187–195, 2001.
[7] T. Chow. Testing software design modeled by finite-state
machines. In IEEE Transactions on Software Engineering,
volume SE-4,n
3, pages 178–187, 1978.
[8] S. Corteel, A. Denise, I. Dutour, F. Sarron, and P. Zim-
mermann. CS web page. http://dept-info.labri.u-
bordeaux.fr/
dutour/CS/.
[9] A. Denise, I. Dutour, and P. Zimmermann. CS:
a package for counting and generating combina-
torial structures. mathPAD, 8(1):22–29, 1998.
http://www.mupad.de/mathpad.shtml.
9
[10] P. Duchon, P. Flajolet, G. Louchard, and G. Schaeffer. Ran-
dom sampling from Boltzmann principes. In P. W. et al,
editor, ICALP 2002, LNCS 2380, pages 501–513. Springer-
Verlag Berlin Heidelberg.
[11] J. Duran and S. Ntafos. A report on random testing. In
5th IEEE International Conference on Software Engineer-
ing, pages 179–183, San Diego, march 1981.
[12] J. Duran and S. Ntafos. An evaluation of random testing. In
IEEE Transactions on Software Engineering,volume SE-10,
pages 438–444, july 1984.
[13] P. Flajolet, P. Zimmermann, and B. Van Cutsem. A calculus
for the random generation of labelled combinatorial struc-
tures. Theoretical Computer Science, 132:1–35, 1994.
[14] A. Gottlieb, B. Botella, and M. Rueher. A CLP framework
for computing structural test data. In Proceedins CL2000,
Springer Verlag, pages 399–413. july 2000. LNAI 1891.
[15] S.-D. Gouraud. G´en´eration de test `a l’aide d’outils combina-
toires: premiers r´esultats exp´erimentaux. Technical report,
LRI, Universit´e Paris II, Orsay, France, 2003. RR No1364.
[16] S.-D. Gouraud, A. Denise, M.-C. Gaudel, and B. Marre. A
new way of automating statistical testing methods. In Auto-
mated Software Engineering Conference, IEEE, pages 5–12,
2001.
[17] T. Hickey and J. Cohen. Uniform random generation
of strings in a context-free language. SIAM. J. Comput,
12(4):645–655, 1983.
[18] F. Hivert and N. Thi´ery. MuPAD-Combinat, an open-source
package for research in algebraic combinatories. Preprint
available at http://mupad-combinat.sourceforge.net.
[19] H. Hong, I. Lee, O. Sokolsky, and H. Ural. Data flow testing
as model checking. In Proceedings of 25th International
Conference on Software Engineering (ICSE’03), pages 232–
242, 2003.
[20] J. Hopcroft and J. Ullman. Introduction to Automata Theory,
Languages and Computation. Addison Wesley, 1979.
[21] D. Lee and M. Yannakakis. Principles and methods of test-
ing finite state machines - a survey. In Proceedings of the
IEEE, volume 84, pages 1090–1123, aoˆut 1996.
[22] G. Lestiennes and M.-C. Gaudel. Testing processes from
formal specifications with inputs, outputs and data types. In
13th IEEE International Symposium on Software Reliability
Engineering (ISSRE-2002), pages 3–14, Annapolis, 2002.
[23] B. Marre. LOFT, a tool for assisting selecting of test data
sets from algebraic specifications. In LNCS, number 915.
Springer-Verlag, 1995.
[24] B. Marre and A. Arnould. Test sequences generation
from LUSTRE descriptions : GATEL. In 15h IEEE Inter-
national Conference on Automated Software Engineering,
pages 229–237, 2000.
[25] C. Meudec. Atgen : automatic test data generation using
constraint logic programming and symbolic execution. Soft-
ware Testing, Verification and Reliability, 11(2):81–96, june
2001.
[26] J. Musa, G. Fuoco, N. Irving, , D. Krofl, and B. Juhli. The
operational profile. In M. R. Lyu, editor, Handbook on Soft-
ware Reliability Engineering, pages 167–218. IEEE Com-
puter Society Press, McGraw-Hill, 1996.
[27] A. Nijenhuis and H. Wilf. Combinatorial algorithms. Aca-
demic Press Inc., 1979.
[28] S. C. Ntafos. On comparisons of random, partition, and pro-
portional partition testing. IEEE Transactions on Software
Engineering, 27(10):949–960, october 2001.
[29] The MuPAD Group (Benno Fuchssteiner et al.). Mu-
PAD User’s Manual - MuPAD Version 1.2.2 Multi Pro-
cessing Algebra Data Tool. John Wiley and sons, 1996.
http://www.mupad.de/.
[30] P. Th´evenod-Fosse. Software validation by means of statisti-
cal testing: Retrospect and future direction. In International
Working Conference on Dependable Computing for Critical
Applications, pages 15–22, 1989.
[31] P. Th´evenod-Fosse and H. Waeselynck. An investigation of
software statistical testing. The Journal of Software Test-
ing, Verification and Reliability, 1(2):5–26, july-september
1991.
[32] P. Th´evenod-Fosse and H.Waeselynck. Statemate applied to
statistical software testing. ACM International Symposium
on Software Testing and Analysis (ISSTA), pages 99–109,
june 1993.
[33] P. Th´evenod-Fosse, H. Waeselynck, and Y. Crouzet. An
experimental study on software structural testing: deter-
ministic versus random input generation. 21st IEEE An-
nual International Symposium on Fault-Tolerant Computing
(FTCS’21), pages 410–417, 1991.
[34] P. Th´evenod-Fosse, H. Waeselynck, and Y. Crouzet. Soft-
ware statistical testing. In B.Randell, JC.Laprie, H.Kopetz,
and B.Littlewood, editors, Predictably dependable com-
puting systems, ISBN 3-540-59334-9, pages 253–272.
Springer, 1995.
[35] H. Wilf. A unified setting for sequencing, ranking, and se-
lection algorithms for combinatorial objects. Advances in
Mathematics, 24:281–291, 1977.
10
