Conference Paper

Infrastructure support for controlled experimentation with software testing and regression testing techniques

Dept. of Comput. Sci. & Eng., Nebraska Univ., Lincoln, NE, USA
DOI: 10.1109/ISESE.2004.1334894 Conference: Empirical Software Engineering, 2004. ISESE '04. Proceedings. 2004 International Symposium on
Source: IEEE Xplore

ABSTRACT Where the creation, understanding, and assessment of software testing and regression testing techniques are concerned, controlled experimentation is an indispensable research methodology. Obtaining the infrastructure necessary to support such experimentation, however, is difficult and expensive. As a result, progress in experimentation with testing techniques has been slow, and empirical data on the costs and effectiveness of techniques remains relatively scarce. To help address this problem, we have been designing and constructing infrastructure to support controlled experimentation with testing and regression testing techniques. This paper reports on the challenges faced by researchers experimenting with testing techniques, including those that inform the design of our infrastructure. The paper then describes the infrastructure that we are creating in response to these challenges, and that we are now making available to other researchers, and discusses the impact that this infrastructure has and can be expected to have.

  • [Show abstract] [Hide abstract]
    ABSTRACT: Test-suite reduction technique aims to find a subset of the test suite while still satisfying the original test requirements; therefore, it can save the cost of software testing. Because many test cases have been removed, the testing information is also lost. Fault localization is a technique using testing information to locate the fault and widely used by programmers to debug programs, so it suffers from the side effects of test-suite reduction. How to reduce the test-suite size in software testing with the premise of retaining or improving fault-localization effectiveness has become the hot spot in the area of software debugging recently. In this paper, we propose a new approach that selectively keeps the limited redundant test cases in the reduced set; it makes the new reduced set relatively redundant compared to the original one, and we expect that it retains or even improves fault-localization effectiveness. We also describe a framework that implements our approach and conduct a set of empirical studies for evaluation. The results show that our approach can retain or even improve fault-localization effectiveness as expected.
    Proceedings of the 2010 ACM Symposium on Applied Computing (SAC), Sierre, Switzerland, March 22-26, 2010; 01/2010
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: AMPLE locates likely failure-causing classes by comparing method call sequences of passing and failing runs. A differ-ence in method call sequences, such as multiple deallocation of the same resource, is likely to point to the erroneous class. In this paper, we describe the implementation of AMPLE as well as its evaluation.
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Case studies can help companies to evaluate the benefits of testing techniques and tools before their possible incorporation into the testing processes. Although general guidelines and organizational frame-works exist describing what a case study should consist of, no general methodological framework exists that can be instantiated to easily de-sign case studies to evaluate different testing techniques. In this paper we define a first version of a general methodological framework for eval-uating software testing techniques, that focusses on the evaluation of effectiveness and efficiency. Using this framework, (1) software testing practitioners can more easily define case studies through an instantia-tion of the framework, (2) results can be better compared since they are all executed according to a similar design, and (3) the gap in existing work on methodological evaluation frameworks will be narrowed.

Full-text (2 Sources)

Available from