Engineering Abstractions in Model Checking and Testing

Abstract

Abstractions are used in model checking to tackle problems like state space explosion or modeling of IO. The application of these abstractions in real software development processes, however, lacks engineering support. This is one reason why model checking is not widely used in practice yet and testing is still state of the art in falsification. We show how user-defined abstractions can be integrated into a Java PathFinder setting with tools like AspectJ or Javassist and discuss implications of remaining weaknesses of these tools. We believe that a principled engineering approach to designing and implementing abstractions will improve the applicability of model checking in practice.

Full text

Page 1

Engineering Abstractions in Model Checking and Testing
Michael Achenbach and Klaus Ostermann
Department of Computer Science
University of Aarhus
Aarhus, Danmark
{ma,ko}@cs.au.dk
Abstract— Abstractions are used in model checking to tackle
problems like state space explosion or modeling of IO. The
application of these abstractions in real software development
processes, however, lacks engineering support. This is one
reason why model checking is not widely used in practice yet
and testing is still state of the art in falsification. We show
how user-defined abstractions can be integrated into a Java
PathFinder setting with tools like AspectJ or Javassist and
discuss implications of remaining weaknesses of these tools.
We believe that a principled engineering approach to designing
and implementing abstractions will improve the applicability
of model checking in practice.
I. INTRODUCTION
All methods of program analysis have to deal with the
curse of Rice’s theorem, by which all interesting properties
of a program are not decidable. Therefore, abstractions
which approximate program entities are necessary to say
anything useful about the program. Abstract interpretation
is probably the best-known theory of approximation [1],
since it gives a precise mathematical framework to design
abstractions that are sound with respect to the operational
meaning of the program. Typically, one can distinguish two
different kinds of abstractions: Over-approximating abstrac-
tions, which are sound (but incomplete) from the verification
perspective, and under-approximating abstractions, which
are sound (but incomplete) from the falsification perspective.
Types in static type checking and predicate abstraction [2]
are examples of over-approximating abstractions, whereas
systematic test input creation methods [3], [4] and test cases
in general are examples of under-approximating abstractions.
In this work, we are interested in user-defined abstractions
of program entities. For example, an abstraction of a list may
only store the size of the list but not the actual list elements.
An abstraction of a file stream for checking exception han-
dling may throw exceptions whenever the file is used instead
of accessing any physical file. An abstraction of an integer
may only record whether the integer is zero or not. We
are not concerned with the soundness of such abstractions
from the perspective of verification or falsification — this
is of course an important and challenging problem, but it
is not in the focus of this paper. Rather, our focus is on
the engineering challenge: How can such abstractions be
programmed and integrated into a software system?
We will analyze this problem in the context of Java
PathFinder (JPF) [5], an explicit state model checker for
Java. Moreover, we apply JPF in the falsification setting,
i.e., for error detection by exploration. Software model
checking has received a lot of attention in the last decade [5],
[6]. Recent developments show, however, that due to the
state space explosion problem and other obstacles, model
checking is not applicable to large programs without the
use of abstractions [5]. We believe that one obstacle to
the practical application of such abstractions is the absence
of a principled engineering approach for their design and
implementation.
In practice, program testing is still the state of the art
of falsification. It is applied in all phases of software de-
velopment and used in unit, integration, and system testing.
In integration testing, mock objects — which are, in our
terminology, user-defined abstractions — are used to abstract
non-implemented or missing program parts [7]. In software
model checking, predefined abstractions for primitive types
(such as abstracting integers to their sign) are well-known.
Recently, also abstractions for some specific reference types
like linked lists or trees have been proposed [8], [9]. Due
to engineering difficulties, however, generalized user-defined
abstractions for complex types (such as classes) have re-
ceived little attention.
The hypothesis of this work is that user-defined abstrac-
tions are both useful and feasible for model checking and
testing. The contributions of our paper are as follows:
• We make the case for applying user-defined abstrac-
tions in testing and model-checking.
• We identify the engineering requirements for integrat-
ing such abstractions into a software project.
• We analyze different modularization technologies,
namely AspectJ, Javassist, and JPF’s Java Model In-
terface, with regard to their applicability to integrate
abstractions. We also identify a list of limitations of
these approaches.
• We discuss potential extensions to these tools that will
make them more applicable to abstraction engineering.
The remainder of this paper is organized as follows: In
Section II, we motivate user-defined abstractions for model
checking and testing and present examples that will be
2009 Ninth IEEE International Working Conference on Source Code Analysis and Manipulation2009 Ninth IEEE International Working Conference on Source Code Analysis and Manipulation
978-0-7695-3793-1/09 $25.00 © 2009 IEEE
DOI 10.1109/SCAM.2009.25DOI 10.1109/SCAM.2009.25
137 137978-0-7695-3793-1/09 $26.00 © 2009 IEEE

Page 2

used throughout the paper. Section III identifies the require-
ments of abstraction engineering, exemplifies how current
engineering methods perform, and compares the strengths
and weaknesses of these methods. Section IV discusses
concepts that address the limitations of the analyzed methods
and presents research challenges that focus on remaining
weaknesses. In Section V, we give an overview of related
approaches, Section VI concludes.
II. THE NEED FOR ABSTRACTIONS
The use of abstractions to deal with the state space
explosion problem in model checking is well-known in
literature. There are other issues which motivate the usage
of abstractions of program entities, such as the usage of
I/O resources like networks or databases, which may not be
available during the analysis. In Test Driven Development,
another concept of abstraction is used to enable integration
testing [10]. So called mock objects are used to simulate
missing program parts at runtime [7], [11]. Typically, mock
classes are alternative implementations of some concrete
classes, either handwritten or generated. They are introduced
into the system under test for simulating behavior, monitor-
ing, or checking of specifications.
While mock objects often are practical ad-hoc solutions,
their expressiveness is limited. Using an explicit state model
checker as test driver would enrich modeling possibilities,
e.g., the behavior of a mock object could be modeled with
non-deterministic choice. The term abstraction used through-
out the paper therefore addresses abstractions in a broader
sense, generalizing both over- and under-approximations as
well as generated or handwritten mock objects. Furthermore,
our notion of abstraction includes also specifications defined
in source entities, e.g., user-defined method preconditions.
We focus on the abstraction of classes, which in Java
excludes primitive types, such as integer and boolean. We
discuss the abstraction of primitive types in Section V.
In the following, we show two examples where user-
defined abstractions are useful for analyzing runtime be-
havior. Later we discuss different engineering methods to
implement and integrate such abstractions.
A. Example: IO Abstractions
The example shown in Figure 1 describes a typical Java
implementation for reading the contents of a file. When
an exception is catched during execution, some logging
data is written to a separate log file using the method
logException. Our first goal is to verify the robustness
of this code regarding exceptions, i.e., that no unchecked
exception can be thrown to the surrounding scope. Our
second goal is to check if all executions obey the IO
stream protocol. As shown in Figure 2, a valid execution
never reaches the error state and ends in the acceptance
state closed. A systematic test of the code in Figure 1
should reveal a possible null pointer exception at the line
class ClientCode {
String
String
String
FileReader r = null ;
try {
r = new FileReader ( f i l e );
BufferedReader in =
new BufferedReader ( r );
try {
while (( ln=in . readLine ( ) )
output += ln + ”\n” ;
}
} finally {
in . close ( ) ;
}
} catch ( IOException e ) {
logException ( e );
}
finally {
try {
r . close ( ) ;
} catch ( IOException e ) {
logException ( e );
}
}
return output ;
}
void logException ( Exception e){
File logFile = new File ( ” error . log ” );
. . .
}
}
readFile ( File
output = ”” ;
ln = null ;
f i l e ) {
!= null ) {
/ /(∗)
Figure 1.
exception data to another file
Sample Java class that reads the contents of a file and logs
closed
opened
start
error
close
read
read
close
read,close
Figure 2.Automaton modelling the open/ close behavior of IO streams
138138

Page 3

HashSet union ( HashSet set1 , HashSet set2 ) {
HashSetresult = new HashSet ( ) ;
result . addAll ( set1 );
result . addAll ( set2 );
return
result ;
}
HashSet set1 = new HashSet ( ) ;
HashSet set2 = new HashSet ( ) ;
set1 . add (1);
set2 . add (2);
assert union ( set1 , set2 ) . contains (1);
assert union ( set1 , set2 ) . contains (2);
Figure 3. Toy example with set union implementation and some test cases
marked with (*), which can occur if FileReader causes
a FileNotFoundException during construction. Be-
cause files can be arbitrarily large and are located on
hard drive, we can test only some suspicious executions
systematically. These include executions where the system
file reader enters a failure state. It is, however, difficult
to put the system file reader into all its possible failure
states. Therefore, a replacement of the class FileReader
should simulate this behavior and integrate the FSM from
Figure 2. We assume that the code used in logException
works correct. The challenge here is to test readFile
independently, i.e., replacements for File or FileReader
used in the context of readFile should not influence the
concrete behavior of logException. We show different
implementations of such replacements in Section III and
discuss the challenge of using different versions of class
File at runtime in Section IV.
B. Example: Finite Set Abstraction
The toy example in Figure 3 shows a set union implemen-
tation with sets in Java. The union method creates a new
result set for each call locally. The example also contains
a selection of functional test cases. A broader verification
of behaviors and, hence, a reduction of the state space can
be achieved by using a finite abstraction for sets containing
{InSet,NotInSet} regarding one specific item [12]. The
partial functional specification can then be defined as:
union(InSet,NotInSet) = InSet
union(NotInSet,InSet) = InSet
union(NotInSet,NotInSet) = NotInSet
Such an abstraction could be implemented as a replacement
of the concrete HashSet class.
III. ENGINEERING ABSTRACTIONS
Every phase in software development like design, imple-
mentation, or testing has corresponding engineering methods
and tools. Currently, there exists no such method that fo-
cuses on the engineering of abstractions for software model
checking. In this section, we analyze the requirements of
such a method and make the case, to which degree current
engineering methods fulfill these. For illustration, we use the
IO example from Section II-A, since this example focuses
on different important aspects like abstractions of different
classes, different abstractions of one class, and modeling
of system classes. In the following subsections, we discuss
how to use AspectJ [13], Javassist [14], and the Java Model
Interface (JMI) of Java PathFinder [15] for abstraction
engineering, and thereafter we compare the methods. In the
following, we define requirements and criteria for abstraction
engineering that address important conceptual and technical
aspects of engineering and design:
Expressiveness denotes to which degree concrete code can
be refined or replaced by abstractions. We distinguish
behavioral replacement, where only the internal be-
havior of methods can be modified, structural refine-
ment, where a class can be refined with additional data,
and structural replacement, where data can also be
removed from a class or where a complete class can be
exchanged.
Scoping defines the level of control that is given to specify
when to use which abstraction or replacement. We
distinguish between global scoping, where only one
replacement can be used at runtime, and local scop-
ing, where time and place of different usages can
be controlled in more detail. In the example from
Section II-A, local scoping would allow the execution
of the client code with an abstraction of File to test
the readFile method, but in the same time use the
original File in the scope of the logException
method. We discuss different variants of local scoping
in Section IV.
Abstraction dependencies arise if abstractions of different
classes access each other using a bigger interface
than visible to the client code. In our example from
Section II-A, e.g., a dependency would arise, if the
abstraction of File would implement a new method
which should be accessible within the abstraction of
FileReader. For type safety reasons, such a depen-
dency must not be visible to the class ClientCode. If
the used engineering method does not support abstrac-
tion dependencies, the structural interfaces of File
and FileReader in the abstraction must equal the
concrete structural interfaces, i.e., all method signatures
must be identical.
Static checking is a process that verifies the consistency
between abstraction and program to analyze. The ab-
139139

Page 4

sence of static checking could for instance lead to in-
compatible abstractions or missing methods at runtime.
In the example from Section II-A, an abstraction of
FileReader needs to provide the methods close
and read for the class ClientCode to be executable.
An abstraction of File, however, is only accessed
by the abstraction of FileReader. Hence, it only
needs to provide a structural interface visible in the
abstraction of FileReader.
Non-invasiveness states if a modification of the client code
can be avoided. E.g., in the IO example it is not
desirable to modify the code of the class ClientCode
and, e.g., expose the local variable r to enable the in-
troduction of different FileReader implementations.
Traceability denotes if the engineering method provides a
mapping from executed code to source code, i.e., if a
bytecode error trace can be mapped to a corresponding
trace of source artifacts. This increases understandabil-
ity of counter examples and enables further analyses
of the source code as well as visualization of the error
trace using source entities in an IDE.
System library support is a technical criterion that fo-
cuses on the Java programming language. It denotes if
an abstraction technique can manipulate runtime library
and core classes of Java.
Without the use of advanced program transformation tech-
niques, the application of abstractions often requires heavy
design changes. Either abstraction or simulation code is
directly implemented in the class containing the behavior
to abstract, or the client code uses unique interfaces for
concrete and abstraction classes. The introduction of the
corresponding object at runtime, however, might require the
exposure of local variables and other design changes that
are not desirable. In the following, we call this approach the
hand-written approach and compare it after an introduction
to Java PathFinder with AspectJ, Javassist, and JMI.
A. Java PathFinder
Java PathFinder (JPF) is implemented as a virtual machine
for Java and is able to execute analyzed programs with all
language features of Java [5]. JPF has an API for modeling
non-deterministic choice and execution pruning in the source
code of the analyzed program. For example, rather than
assigning a specific integer to a variable, one can specify
a range of integers (say, 1 to 10), and all possible execution
paths resulting from these non-deterministic choices are then
systematically executed. Pruning of executions can be mod-
eled with a special condition in the source code. Dependent
on its evaluation at runtime, the execution path containing
this condition is pruned. The main techniques in JPF for this
purpose are backtracking and state matching. Backtracking
enables resetting the state of the analyzed program to the
next non-deterministic choice point. State matching prevents
the repetition of already analyzed states. JPF can in fact be
class OpenCloseFSM extends FSM {
OpenCloseFSM () {
/ / Add transitions
. . .
}
/ /Proceeds one transition
/ / name from the
/ / nextstate . E. g .
/ / proceeds from ”opened” to ”closed”
void proceed ( String
super . proceed ( action );
of the stream FSM
with the given
stateto the
action ”close”
current
action ) {
/ /
/ /
assert ! currentState . equals ( ” error ” );
Assert
never be reached .
that the ”error” state can
}
}
Figure 4.Implementation of the stream FSM from Figure 2
seen as a generalization of traditional testing methods in
the falsification setting [15]. Executing test cases with JPF
has several benefits: Backtracking saves computation time
compared to testing, since in testing each run executes from
the beginning. State matching automatically rules out re-
dundant test cases. Moreover, analyzing program executions
with an explicit state model checker includes different thread
interleavings of multi-threaded applications.
B. AspectJ
Our implementation of the IO example of Section II-A
with AspectJ [13] explores three different variants of intro-
ducing abstractions into client code:
• adding state with inheritance,
• using inter-type declarations,
• mapping additional state to existing objects.
We assume some familiarity with AspectJ pointcut and
advice — for further reading we refer to [16]. In our
examples, we implement the stream FSM from Figure 2
with the class OpenCloseFSM as shown in Figure 4. The
class FSM (not shown here) maintains the state of the FSM
in the field currentState and stores the available states
and transitions. These states and transitions are initialized in
the OpenCloseFSM constructor. The method proceed of
FSM performs one state transition on currentState. The
implementation of proceed in OpenCloseFSM keeps
track if the error state can be reached, e.g., if read is applied
in the closed state.
Inheritance: The example in Figure 5 shows the in-
troduction of state with inheritance. The around advice
intercepts constructor calls of FileReader using the
call pointcut and creates an instance of the new subclass
AbstractFileReader instead. We model the possible
non-existence of the file to read using the non-deterministic
140140

Page 5

class
AbstractFileReader
extends FileReader {
FSM fsm ;
/ / Dummy constructor
public AbstractFileReader ( File
throws FileNotFoundException {
super ( in );
}
public void close () throws IOException {
/ /Models the occurrence of an
/ /exception during closing
if
( Verify . getBoolean ( ) )
throw new IOException ( ) ;
in )
/ /
/ /
fsm . proceed ( ” close ” );
Proceeds to the next
of the stream FSM
state
}
/ /
. . .
Similar forother FileReader methods
}
/ /
FileReader around ()
throws FileNotFoundException :
call ( FileReader .new ( . . ) ) {
AbstractFileReader
try {
result = AbstractFileReader
. class . newInstance ( ) ;
} catch ( Exception e ) {
result . fsm = new OpenCloseFSM ( ) ;
Circumvents the ”FileReader” constructor
result = null ;
. . .
}
/ /
if
Models the
( Verify . getBoolean ( ) )
throw new FileNotFoundException ( ) ;
existenceof thef i l e
return
result ;
}
Figure 5.
inheritance
Abstraction engineering with AspectJ — adding state with
choice command of JPF, Verify.getBoolean(). This
leads to different execution paths of the analysis, some
paths with a FileNotFoundException thrown, oth-
ers without. The new close method models the pos-
sibility of an IOException and implements the state
transition close of the stream FSM. The construction
of AbstractFileReader, however, suffers from the
mandatory call to a super constructor, which should be
circumvented in the first place. This is a technical and not
a conceptual problem of the underlying virtual machine that
executes the code. The Java Virtual Machine does not allow
aspect Abstraction {
/ /Models the
/ /read in a new f i e l d
boolean File . exists
= Verify . getBoolean ( ) ;
existenceof the
of class ”File”
f i l eto
FSM FileReader . fsm = new OpenCloseFSM ( ) ;
/ /
FileReader .new( boolean exists )
throws FileNotFoundException {
if
(! exists )
throw new FileNotFoundException ( ) ;
}
/ /Circumvents the
/ /with the new constructor above
FileReader around ( File
throws FileNotFoundException :
call ( FileReader .new( File ))
&& args ( f i l e ) {
assert
f i l e!= null ;
return new FileReader ( f i l e . exists );
}
/ /Circumvents the ”close” method
void around ( FileReader
throws IOException :
execution (∗ FileReader . close ( ) )
&& target ( reader ) {
/ /Models thepossible
/ / an exception during closing
if
( Verify . getBoolean ( ) )
throw new IOException ( ) ;
Constructor modelingf i l eexistence
originalconstructor
f i l e )
reader )
occurrence of
/ /
/ /
reader . fsm . proceed ( ” close ” );
Proceeds to the next
of the stream FSM
state
}
/ /
/ /
/ /
/ /
boolean around ( File
! cflow ( call (∗ ∗. logException ( . . ) ) )
&& call (∗ File . exists ( ) )
&& target ( f i l e ) {
return
f i l e . exists ;
}
/ / Similarlyforother ”File”
/ / and ”FileReader” methods
. . .
Around advice demonstrating
scoping : the
only circumvented
flowis not in ”logException”
local
method ” exists ” is
i fthecontrol
f i l e ) :
}
Figure 6.
inter-type declarations
Abstraction engineering with AspectJ — state introduced with
141141