Conference Paper

Infrastructure support for controlled experimentation with software testing and regression testing techniques

Dept. of Comput. Sci. & Eng., Nebraska Univ., Lincoln, NE, USA
DOI: 10.1109/ISESE.2004.1334894 Conference: Empirical Software Engineering, 2004. ISESE '04. Proceedings. 2004 International Symposium on
Source: IEEE Xplore

ABSTRACT Where the creation, understanding, and assessment of software testing and regression testing techniques are concerned, controlled experimentation is an indispensable research methodology. Obtaining the infrastructure necessary to support such experimentation, however, is difficult and expensive. As a result, progress in experimentation with testing techniques has been slow, and empirical data on the costs and effectiveness of techniques remains relatively scarce. To help address this problem, we have been designing and constructing infrastructure to support controlled experimentation with testing and regression testing techniques. This paper reports on the challenges faced by researchers experimenting with testing techniques, including those that inform the design of our infrastructure. The paper then describes the infrastructure that we are creating in response to these challenges, and that we are now making available to other researchers, and discusses the impact that this infrastructure has and can be expected to have.

0 Bookmarks
 · 
108 Views
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Case studies can help companies to evaluate the benefits of testing techniques and tools before their possible incorporation into the testing processes. Although general guidelines and organizational frame-works exist describing what a case study should consist of, no general methodological framework exists that can be instantiated to easily de-sign case studies to evaluate different testing techniques. In this paper we define a first version of a general methodological framework for eval-uating software testing techniques, that focusses on the evaluation of effectiveness and efficiency. Using this framework, (1) software testing practitioners can more easily define case studies through an instantia-tion of the framework, (2) results can be better compared since they are all executed according to a similar design, and (3) the gap in existing work on methodological evaluation frameworks will be narrowed.
  • [Show abstract] [Hide abstract]
    ABSTRACT: Regression testing is a well-established practice in software development, but in recent years it has seen a change of status and emphasis with the increasing popularity of agile methods, which stress the central role of regression testing in maintaining software quality. The objectives of this article are to investigate regression testing strategies in agile development teams and identify the factors that can influence the adoption and implementation of this practice. We have used a mixed methods approach to our research, beginning with an analysis of the literature to identify research themes related to the adoption of regression testing techniques under agile methodologies, from which we developed an analytical framework for the study. This was followed by three exploratory case studies that we used to exercise the main elements of the framework, develop some key themes of interest, and devise a questionnaire for the final stage of the study, an on-line survey to explore the main issues identified in the case studies across different contexts. Within our specific sample, our results suggest that organizational maturity is a key factor in effective regression testing practices and that the adoption of such practices is helped by a coherent testing philosophy and change management processes. We also found that the return on investment in automated regression testing was positive for our respondents and that adopting these practices in the context of agile methods had been a relatively painless process for the organizations in our survey. We conclude that investing in regression testing tools and processes is likely to be beneficial for organizations. However, further work is needed in assessing how organizational culture impacts on the quality process and the financial outcomes for commercial software development organizations.
    Software Quality Control 11/2014; · 0.88 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: There exists a real need in industry to have guidelines on what testing techniques use for different testing objectives, and how usable (effective, efficient, satisfactory) these techniques are. Up to date, these guidelines do not exist. Such guidelines could be obtained by doing secondary studies on a body of evidence consisting of case studies evaluating and comparing testing techniques and tools. However, such a body of evidence is also lacking. In this paper, we will make a first step towards creating such body of evidence by defining a general methodological evaluation framework that can simplify the design of case studies for comparing software testing tools, and make the results more precise, reliable, and easy to com-pare. Using this framework, (1) software testing practitioners can more easily define case studies through an instantiation of the framework, (2) results can be better compared since they are all executed according to a similar design, (3) the gap in existing work on methodological evaluation frameworks will be narrowed, and (4) a body of evidence will be initiated. By means of validating the framework, we will present successful applications of this methodological framework to various case studies for evaluating testing tools in an industrial environment with real objects and real subjects.
    QSIC 2012; 08/2012

Preview (2 Sources)

Download
0 Downloads
Available from