Conference Paper

Empirical studies of test case prioritization in a JUnit testing environment

Dept. of Comput. Sci. & Eng., Nebraska Univ., Lincoln, NE, USA
DOI: 10.1109/ISSRE.2004.18 Conference: Software Reliability Engineering, 2004. ISSRE 2004. 15th International Symposium on
Source: DBLP

ABSTRACT Test case prioritization provides a way to run test cases with the highest priority earliest. Numerous empirical studies have shown that prioritization can improve a test suite's rate of fault detection, but the extent to which these results generalize is an open question because the studies have all focused on a single procedural language, C, and a few specific types of test suites, in particular, Java and the JUnit testing framework are being used extensively in practice, and the effectiveness of prioritization techniques on Java systems tested under JUnit has not been investigated. We have therefore designed and performed a controlled experiment examining whether test case prioritization can be effective on Java programs tested under JUnit, and comparing the results to those achieved in earlier studies. Our analyses show that test case prioritization can significantly improve the rate of fault detection of JUnit test suites, but also reveal differences with respect to previous studies that can be related to the language and testing paradigm.

  • Source
    • "Rothermel et al. [53] provided empirical evidence of the usefulness of the prioritization techniques with respect to the random ordering by measuring the ability to early detect faults. A similar analysis was performed by Do et al. [18] using the Java unit test framework (JUnit). Elbaum et al. [22] considered further testing criteria with different granularity, e.g., statement coverage or function coverage. "
    [Show abstract] [Hide abstract]
    ABSTRACT: A way to reduce the cost of regression testing consists of selecting or prioritizing subsets of test cases from a test suite according to some criteria. Besides greedy algorithms, cost cognizant additional greedy algorithms, multi-objective optimization algorithms, and Multi-Objective Genetic Algorithms (MOGAs), have also been proposed to tackle this problem. However, previous studies have shown that there is no clear winner between greedy and MOGAs, and that their combination does not necessarily produce better results. In this paper we show that the optimality of MOGAs can be significantly improved by diversifying the solutions (sub-sets of the test suite) generated during the search process. Specifically, we introduce a new MOGA, coined as DIV-GA (DIversity based Genetic Algorithm), based on the mechanisms of orthogonal design and orthogonal evolution that increase diversity by injecting new orthogonal individuals during the search process. Results of an empirical study conducted on eleven programs show that DIV-GA outperforms both greedy algorithms and the traditional MOGAs from the optimality point of view. Moreover, the solutions (sub-sets of the test suite) provided by DIV-GA are able to detect more faults than the other algorithms, while keeping the same test execution cost.
    IEEE Transactions on Software Engineering 10/2014; 41(4). DOI:10.1109/TSE.2014.2364175 · 2.29 Impact Factor
  • Source
    • "TestRank Architecture the basic traced unit is at the method level granularity. Finer granularity techniques do not necessarily yield significant improvement [7], especially in Java programs [15]. This is significant since coarse granularity analysis is much faster, especially for large or CPU intensive code bases. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Test-Driven Development (TDD) is characterized by repeated execution of a test suite, enabling developers to change code with confidence. However, running an entire test suite after every small code change is not always cost effective. Therefore, regression test selection (RTS) techniques are important for TDD. Particularly challenging for TDD is the task of selecting a small subset of tests that are most likely to detect a regression fault in a given small and localized code change. We present cost-bounded RTS techniques based on both dynamic program analysis and natural-language analysis. We implemented our techniques in a tool called Test Rank, and evaluated its effectiveness on two open-source projects. We show that using these techniques, developers can accelerate their development cycle, while maintaining a high bug detection rate, whether actually following TDD, or in any methodology that combines testing during development.
    Software Testing, Verification and Validation Workshops (ICSTW), 2011 IEEE Fourth International Conference on; 04/2011
  • Source
    • "Test case prioritization problem is introduced and defined to re-order test cases such that more faults are revealed as early as possible [1]. Prioritization techniques are based on different coverage criteria such as statement coverage, block coverage, probability of exposing faults, etc. [1] [2] [3]. The time-aware test case prioritization problem was introduced by Walcott et al. in [4]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Regression testing is often performed with a time budget and it does not allow executing all test cases. Test case prioritization techniques re-order test cases to increase the rate of fault detection. Several time-aware test case prioritization techniques have been proposed to satisfy a time budget. Since it is difficult to collect the time cost of each test case in some cases, a natural question is whether it is worth using such information when prioritizing test cases. In this paper, two most popular criteria: statement coverage and fault detection are considered for time-aware test case prioritization. We investigate whether the time cost of each test case affects the effectiveness of prioritization techniques, i.e. the rate of statement coverage and the rate of fault detection. Our empirical study shows that: although the techniques considering the time cost of each test case are slightly better than the techniques not considering such information in some cases, they have no significant difference in most cases.
    Proceedings of the 2011 ACM Symposium on Applied Computing (SAC), TaiChung, Taiwan, March 21 - 24, 2011; 01/2011
Show more


1 Download
Available from