No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
SealTest: A Simple Library for Test Sequence Generation
Abstract
SealTest is a Java library for generating test sequences based on a formal specification. It allows a user to easily define a wide range of coverage metrics using multiple specification languages. Its simple and generic architecture makes it a useful testing tool for dynamic software systems, as well as an appropriate research testbed for implementing and experimentally comparing test sequence generation algorithms.
... A sequence generation library called SealTest [18] implements a pseudo-random greedy algorithm to generate sequences according to a coverage metric, based on the random greedy algorithm described by [26]. Analysis of the source code shows that this algorithm generates k random walks of a randomly selected length inside the specification, and adds to its test suite the sequence that produces the largest increase in coverage; if no sequence increases the coverage, it regenerates a new set of k candidates. ...
... We chose to implement the concepts described in this paper by extending the open source Java library called SealTest mentioned in Section II-C. [18]. This choice was made for two reasons. ...
... Unlike HECATE, these approaches (a) do not aim to generate a failure-revealing test case, but rather aim to satisfy coverage criteria [82], [73], or to identify performancedriven metamorphic relations [83]. Lastly, some approaches target formalisms named 'test sequences' [84], [85], [86], [87], [88] that are not related to Simulink and are significantly different from Simulink Test Sequences. ...
Simulation-based software testing supports engineers in finding faults in Simulink® models. It typically relies on search algorithms that iteratively generate test inputs used to exercise models in simulation to detect design errors. While simulation-based software testing techniques are effective in many practical scenarios, they are typically not fully integrated within the Simulink environment and require additional manual effort. Many techniques require engineers to specify requirements using logical languages that are neither intuitive nor fully supported by Simulink, thereby limiting their adoption in industry.
This work presents HECATE, a testing approach for Simulink models using Test Sequence and Test Assessment blocks from Simulink® TestTM. Unlike existing testing techniques, HECATE uses information from Simulink models to guide the search-based exploration. Specifically, HECATE relies on information provided by the Test Sequence and Test Assessment blocks to guide the search procedure. Across a benchmark of 18 Simulink models from different domains and industries, our comparison of HECATE with the state-of-the-art testing tool S-Taliro indicates that HECATE is both more effective (more failure-revealing test cases) and efficient (less iterations and computational time) than S-Taliro for ≈94% and ≈83% of benchmark models respectively. Furthermore, HECATE successfully generated a failure-revealing test case for a representative case study from the automotive domain demonstrating its practical usefulness.</p
... Unlike HECATE, these approaches (a) do not aim to generate a failure-revealing test case, but rather aim to satisfy coverage criteria [82], [73], or to identify performancedriven metamorphic relations [83]. Lastly, some approaches target formalisms named 'test sequences' [84], [85], [86], [87], [88] that are not related to Simulink and are significantly different from Simulink Test Sequences. ...
Simulation-based software testing supports engineers in finding faults in Simulink models. It typically relies on search algorithms that iteratively generate test inputs used to exercise models in simulation to detect design errors. While simulation-based software testing techniques are effective in many practical scenarios, they are typically not fully integrated within the Simulink environment and require additional manual effort. Many techniques require engineers to specify requirements using logical languages that are neither intuitive nor fully supported by Simulink, thereby limiting their adoption in industry. This work presents HECATE, a testing approach for Simulink models using Test Sequence and Test Assessment blocks from Simulink Test. Unlike existing testing techniques, HECATE uses information from Simulink models to guide the search-based exploration. Specifically, HECATE relies on information provided by the Test Sequence and Test Assessment blocks to guide the search procedure. Across a benchmark of 16 Simulink models from different domains and industries, our comparison of HECATE with the state-of-the-art testing tool S-TALIRO indicates that HECATE is both more effective (more failure-revealing test cases) and efficient (less iterations and computational time) than S-TALIRO for ~94% and ~81% of benchmark models respectively. Furthermore, HECATE successfully generated a failure-revealing test case for a representative case study from the automotive domain demonstrating its practical usefulness.
This paper introduces a new approach at classifying event traces with respect to some property expressed in Linear Temporal Logic generalizing the classical Boolean outcome. We produce from the evaluation of the formula on a given trace a data structure called a trace hologram. When such holograms are interpreted as equivalence classes, we show how manipulating them produce a clustering of event traces into various categories, depending on the precise way in which each group of traces violate the specification. The approach has been integrated into in an existing bug tracker in an entirely automated fashion, and experimented on a set of traces extracted from the execution of a real-world program.
To find defects in software, one needs test cases that execute the software systematically, and oracles that assess the correctness of the observed behavior when running these test cases. This paper presents EvoSuite, a tool that automatically generates test cases with assertions for classes written in Java code. To achieve this, EvoSuite applies a novel hybrid approach that generates and optimizes whole test suites towards satisfying a coverage criterion. For the produced test suites, EvoSuite suggests possible oracles by adding small and effective sets of assertions that concisely summarize the current behavior; these assertions allow the developer to detect deviations from expected behavior, and to capture the current behavior in order to protect against future defects breaking this behavior.
Although the majority of software testing in industry is conducted at the system level, most formal research has focused on the unit level. As a result, most system-level testing techniques are only described informally. This paper presents formal testing criteria for system level testing that are based on formal specifications of the software. Software testing can only be formalized and quantified when a solid basis for test generation can be defined. Formal specifications represent a significant opportunity for testing because they precisely describe what functions the software is supposed to provide in a form that can be automatically manipulated.
This paper presents general criteria for generating test inputs from state-based specifications. The criteria include techniques for generating tests at several levels of abstraction for specifications (transition predicates, transitions, pairs of transitions and sequences of transitions). These techniques provide coverage criteria that are based on the specifications and are made up of several parts, including test prefixes that contain inputs necessary to put the software into the appropriate state for the test values. The test generation process includes several steps for transforming specifications to tests. These criteria have been applied to a case study to compare their ability to detect seeded faults. Copyright
We propose a novel testing technique for object-oriented programs. Based on the state and activity models of a system, we construct an intermediate representation, which we have named state-activity diagram (SAD). We generate test cases to achieve state-activity coverage of SADs. We have empirically evaluated the effectiveness of our approach. The results show that the proposed technique could detect seeded integration testing faults which could not be detected by the related approaches.
Software Testing is the process of testing the software in order to ensure that it is free of errors andproduces the desired outputs in any given situation. Model based testing is an approach in whichsoftware is viewed as a set of states. A usage model describes software on the basis of its statistical usagedata. One of the major problems faced in such an approach is the generation of optimal sets of testsequences. The model discussed in this paper is a Markov chain based usage model. The analyticaloperations and results associated with Markov chains make them an appropriate choice for checking thefeasibility of test sequences while they are being generated. The statistical data about the estimated usagehas been used to build a stochastic model of the software under test. This paper proposes a technique togenerate optimized test sequences from a markov chain based usage model. The proposed technique usesant colony optimization as its basis and also incorporates factors like cost and criticality of various statesin the model. It further takes into consideration the average number of visits to any state and the trade-offbetween cost considerations and optimality of the test coverage.
The paper discusses the application of state diagrams in UML to
class testing. A set of coverage criteria is proposed based on control
and data flow in UML state diagrams and it is shown how to generate test
cases satisfying these criteria from UML state diagrams. First, control
flow is identified by transforming UML state diagrams into extended
finite state machines (EFSMs). The hierarchical and concurrent structure
of states is flattened and broadcast communications are eliminated in
the resulting EFSMs. Second, data flow is identified by transforming
EFSMs into flow graphs to which conventional data flow analysis
techniques can be applied
Context
The generation of dynamic test sequences from a formal specification, complementing traditional testing methods in order to find errors in the source code.
Objective
In this paper we extend one specific combinatorial test approach, the Classification Tree Method (CTM), with transition information to generate test sequences. Although we use CTM, this extension is also possible for any combinatorial testing method.
Method
The generation of minimal test sequences that fulfill the demanded coverage criteria is an NP-hard problem. Therefore, search-based approaches are required to find such (near) optimal test sequences.
Results
The experimental analysis compares the search-based technique with a greedy algorithm on a set of 12 hierarchical concurrent models of programs extracted from the literature. Our proposed search-based approaches (GTSG and ACOts) are able to generate test sequences by finding the shortest valid path to achieve full class (state) and transition coverage.
Conclusion
The extended classification tree is useful for generating of test sequences. Moreover, the experimental analysis reveals that our search-based approaches are better than the greedy deterministic approach, especially in the most complex instances. All presented algorithms are actually integrated into a professional tool for functional testing.
We present a broad extension of the conventional formalism of state machines and state diagrams, that is relevant to the specification and design of complex discrete-event systems, such as multi-computer real-time systems, communication protocols and digital control units. Our diagrams, which we call statecharts, extend conventional state-transition diagrams with essentially three olements, dealing, respectively, with the notions of hierarchy, concurrency and communication. These transform the language of state diagrams into a highly structured' and economical description language. Statecharts are thus compact and expressive--small diagrams can express complex behavior--as well as compositional and modular. When coupled with the capabilities of computerized graphics, statecharts enable viewing the description at different levels of detail, and make even very large specifications manageable and comprehensible. In fact, we intend to demonstrate here that statecharts counter many of the objections raised against conventional state diagrams, and thus appear to render specification by diagrams an attractive and plausible approach. Statecharts can be used either as a stand-alone behavioral description or as part of a more general design methodology that deals also with the system's other aspects, such as functional decomposition and data-flow specification. We also discuss some practical experience that was gained over the last three years in applying the statechart formalism to the specification of a particularly complex system.
Software testing plays a major role in software development because it accounts for a large part of the development cost. Moreover, manual testing technique always makes a problem. Consequently, this paper proposes the automatic testing technique to solve partially the testing process. This technique can automatically generate and select test cases from UML statechart diagrams. Firstly, we transform this diagram into intermediate diagram, called Testing Flow Graph (TFG), explicitly identify flows of UML statechart diagrams and enhance for testing. Secondly, from TFG we generate test case using the testing criteria that is the coverage of the state and transition of diagrams. Finally, the evaluation is performed using mutation analysis to assess the fault revealing power of our test cases.
The combinatorial test design and combinatorial interac-
tion testing are well studied topics. For the generation of dynamic test
sequences from a formal specification of combinatorial problems, there
has not been much work yet. The classification tree method imple-
ments aspects from the field of combinatorial testing. We will extend
the classification tree by additional information to allow the interpre-
tation of the classification tree as a hierarchical concurrent state ma-
chine. Using this state machine, our new approach then uses a Multi-
agent System to generate test sequences by finding and rating valid
paths through the state machine.
The combinatorial test design and combinatorial interaction testing are well studied topics. For the generation of dynamic test sequences from a formal specification of combinatorial problems, there has not been much work yet. The classification tree method implements aspects from the field of combinatorial testing. This paper extends the classification tree with additional information to allow the interpretation of the classification tree as a hierarchical concurrent state machine. Using this state machine, our new approach then uses a Multi-agent System to generate test sequences by finding and rating valid paths through the state machine.
Finite state machines and pushdown systems are frequently used in model based testing. In such testing, the system under test is abstractly modeled as a finite state machine having a finite set of states and a labeled transition relation between the states. A pushdown system, additionally, has an unbounded stack. Test inputs are then generated by enumer- ating a set of sequences of transitions labels from the model. There has been a lot of research that focussed on generation of test input sequences satisfying various coverage criteria. In this paper, we consider the problem of generating a set of test input sequences that satisfy certain coverage criteria— cover all transition labels or cover all length-n transition label sequences at least once—while minimizing the sum of the length of the sequences in the set. We show that these optimal test input generation problems can be reduced to integer linear programming (ILP) problems. We also prove that our optimal test input generation problems are NP-Complete. We report our experimental results on a prototype implementation for finite states machines.
We present a technique that improves random test generation by incorporating feedback obtained from executing test inputs as they are created. Our technique builds inputs incrementally by randomly selecting a method call to apply and finding arguments from among previously-constructed inputs. As soon as an input is built, it is executed and checked against a set of contracts and filters. The result of the execution determines whether the input is redundant, illegal, contract-violating, or useful for generating more inputs. The technique outputs a test suite consisting of unit tests for the classes under test. Passing tests can be used to ensure that code contracts are preserved across program changes; failing tests (that violate one or more contract) point to potential errors that should be corrected. Our experimental results indicate that feedback-directed random test generation can outperform systematic and undirected random test generation, in terms of coverage and error detection. On four small but nontrivial data structures (used previously in the literature), our technique achieves higher or equal block and predicate coverage than model checking (with and without abstraction) and undirected random generation. On 14 large, widely-used libraries (comprising 780KLOC), feedback-directed random test generation finds many previously-unknown errors, not found by either model checking or undirected random generation.