A probabilistic analysis of coverage methods.

Full text

Page 1

A Probabilistic Analysis of
Coverage Methods
Research Thesis
SUBMITTED IN PARTIAL FULFILLMENT OF THE
REQUIREMENTS FOR THE DEGREE OF
MASTER OF SCIENCE
IN
INFORMATION MANAGEMENT ENGINEERING
Ekaterina Kutsy
Submitted to the Senate of
the Technion - Israel Institute of Technology
SIVAN 5768 - HAIFA - JUNE 2008

Page 2

Table of Contents
Table of Contents ii
List of Figures iv
Abstractv
Acknowledgements vii
1Introduction 1
2A cross-product functional coverage model4
3Testing Model 1
3.1 Model definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2Test generation strategies. . . . . . . . . . . . . . . . . . . . . . . .
3.3The probability to expose an error with the Directed and Random
sampling strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4A comparison of the Directed and Random sampling strategies . . . .
7
7
8
9
11
4Testing Model 2
4.1Model definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2Test generation strategies. . . . . . . . . . . . . . . . . . . . . . . .
4.3Random, Round-Robin and Hybrid sampling strategies: recurring case
4.3.1 The probability to expose an error
4.3.2The probability to expose an error with a cost . . . . . . . . .
4.3.3 Expected number of samples until error detection . . . . . . .
4.4The Random and Round-Robin sampling strategies: non-recurring case 24
4.4.1The probability to expose an error
4.5The Directed strategy. . . . . . . . . . . . . . . . . . . . . . . . . .
13
13
13
15
15
19
21
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .24
24
ii

Page 3

4.5.1
4.5.2
4.5.3
4.5.4
Reduction to a search problem . . . . . . . . . . . . . . . . . .
The optimal strategy in terms of cost: recurring case . . . . .
The optimal strategy in terms of cost: non-recurring case . . .
The optimal strategy in terms of probability to expose an error
24
25
27
28
5Testing Model 3
5.1Model definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2Test generation strategies. . . . . . . . . . . . . . . . . . . . . . . .
5.3Expected sampling cost . . . . . . . . . . . . . . . . . . . . . . . . . .
5.4Optimal sampling strategy in terms of expected sampling cost . . . .
32
32
32
33
34
6 Testing Model 4
6.1Model definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2 Expected sampling cost . . . . . . . . . . . . . . . . . . . . . . . . . .
6.3Two sampling strategies . . . . . . . . . . . . . . . . . . . . . . . . .
6.3.1Strategy I . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.3.2Strategy II . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
35
35
35
36
36
38
7Related work39
Bibliography42
A Optimal number of directed samples in the Hybrid strategy43
B A proof of Proposition 2: recurring case 45
iii

Page 4

List of Figures
4.1 The optimal number of directed samples in example 8 . . . . . . . . .20
6.1 Illustrating the two options of switching to level i + 1 after j or j + 1
samples in level i. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .37
iv

Page 5

Abstract
Testing of hardware and software frequently cost more than the development of the
product itself. As a result there is a constant effort, both in the industry and in
the academia, to make this process as efficient as possible. Automatic test genera-
tors are tools that generate test vectors (i.e., input sequences) that are run on the
tested program / hardware, and the outputs are then compared to some predefined
specification (ideally this process is fully automatic). The simplest type of automatic
test generation is generating random input vectors. There are various analytic results
showing the effectiveness of such testing, assuming a uniform distribution of the tests.
However, uniformly generated tests are not the only way tests are generated in the
industry.
Hardware testing is typically done by modeling the environment of the tested
component with a set of constraints. Every solution to this set of constraints is a
test, and hence such constraints-based test generators produce a large number of
solutions. A typical scenario is that of beginning the testing process by defining a
coverage model, which is a partition of the sampling space to coverage goals according
to the developer’s knowledge of the expected functionality of the tested product.
Then, using a constraint solver, various constraints are added in order to direct the
constraint solver to goals that were not yet covered. Another possible scenario is that
random sampling is combined with such directed sampling in various ways. Directed
sampling has a cost, since a more constrained model is many times harder to solve.
There is therefore a trade-off between cheaper sampling – achieved by generating
random samples – and directed sampling that have a higher probability of covering
v