Article

Software Test Case Generation Tools and Techniques: A Review

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Software Industry is evolving at a very fast pace since last two decades. Many software developments, testing and test case generation approaches have evolved in last two decades to deliver quality products and services. Testing plays a vital role to ensure the quality and reliability of software products. In this paper authors attempted to conduct a systematic study of testing tools and techniques. Six most popular e-resources called IEEE, Springer, Association for Computing Machinery (ACM), Elsevier, Wiley and Google Scholar to download 738 manuscripts out of which 125 were selected to conduct the study. Out of 125 manuscripts selected, a good number approx. 79% are from reputed journals and around 21% are from good conference of repute. Testing tools discussed in this paper have broadly been divided into five different categories: open source, academic and research, commercial, academic and open source, and commercial & open source. The paper also discusses several benchmarked datasets viz. Evosuite 10, SF100 Corpus, Defects4J repository, Neo4j, JSON, Mocha JS, and Node JS to name a few. Aim of this paper is to make the researchers aware of the various test case generation tools and techniques introduced in the last 11 years with their salient features.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

Article
Full-text available
Hybrid meta-heuristics algorithms have gained popularity in recent years to solve t-way test suite generation problems due to better exploration and exploitation capabilities of the hybridization. This paper presents the implementation of meta-heuristic search algorithms that are Migrating Birds Optimization (MBO) algorithm and Genetic Algorithm (GA) hybrid to a t-way test data generation strategy. The proposed strategy is called Elitist Hybrid MBO-GA Strategy (EMBO-GA). Based on the published benchmarking results, the result of these strategies are competitive with most existing strategies in terms of the generated test size in many of the parameter configurations. In the case where this strategy is not the most optimal, the resulting test size is sufficiently competitive.
Article
Full-text available
In the present paper, we investigate an approach to intelligent support of the software white-box testing process based on an evolutionary paradigm. As a part of this approach, we solve the urgent problem of automated generation of the optimal set of test data that provides maximum statement coverage of the code when it is used in the testing process. We propose the formulation of a fitness function containing two terms, and, accordingly, two versions for implementing genetic algorithms (GA). The first term of the fitness function is responsible for the complexity of the code statements executed on the path generated by the current individual test case (current set of statements). The second term formulates the maximum possible difference between the current set of statements and the set of statements covered by the remaining test cases in the population. Using only the first term does not make it possible to obtain 100 percent statement coverage by generated test cases in one population, and therefore implies repeated launch of the GA with changed weights of the code statements which requires recompiling the code under the test. By using both terms of the proposed fitness function, we obtain maximum statement coverage and population diversity in one launch of the GA. Optimal relation between the two terms of fitness function was obtained for two very different programs under testing.
Conference Paper
Full-text available
T-way test suite generation strategy based on Ant Colony algorithm (TTSGA) has been developed to support t-way variable strength testing which tackles exhaustive testing issues. It employs the ant colony optimization algorithm to generate near-optimal number of test suite size. Even though the test suite size is smaller than exhaustive testing, the strategy covers every possible combination of interacting parameters. The strategy has been evaluated by using benchmarked experiments. Results obtained were compared with other existing strategies that support variable strength. It was found that TTSGA produces comparable results with other existing strategies especially for higher strength configurations. Two non-parametric tests, which are Wilcoxon Rank and Friedman test, have been conducted to analyze the results statistically between TTSGA and HSS as only both strategies have complete experiments results. Although the results shows that there is no significant difference of test suite size among them, TTSGA is in the first rank in the Friedman test.
Article
Full-text available
The level and quality of automation dramatically affects software testing activities, determines costs and effectiveness of the testing process, and largely impacts on the quality of the final product. While costs and benefits of automating many testing activities in industrial practice (including managing the quality process, executing large test suites, and managing regression test suites) are well understood and documented, the benefits and obstacles of automatically generating system test suites in industrial practice are not well reported yet, despite the recent progresses of automated test case generation tools. Proprietary tools for automatically generating test cases are becoming common practice in large software organisations, and commercial tools are becoming available for some application domains and testing levels. However, generating system test cases in small and medium-size software companies is still largely a manual, inefficient and ad-hoc activity. This paper reports our experience in introducing techniques for automatically generating system test suites in a medium-size company. We describe the technical and organisational obstacles that we faced when introducing automatic test case generation in the development process of the company, and present the solutions that we successfully experienced in that context. In particular, the paper discusses the problems of automating the generation of test cases by referring to a customised ERP application that the medium-size company developed for a third party multinational company, and presents ABT2.0, the test case generator that we developed by tailoring ABT, a research state-of-the-art GUI test generator, to their industrial environment. This paper presents the new features of ABT2.0, and discusses how these new features address the issues that we faced.
Article
Full-text available
Specification-based testing methods generate test data without the knowledge of the structure of the program. However, the quality of these test data are not well ensured to detect bugs when non-functional changes are introduced to the program. To generate test data effectively, we propose a new method that combines formal specifications with the genetic algorithm (GA). In this method, formal specifications are reformed by GA in order to be used to generate input values that can kill as many mutants of the target program as possible. Two classic examples are presented to demonstrate how the method works. The result shows that the proposed method can help effectively generate test cases to kill the program mutants, which contributes to the further maintenance of software.
Article
Full-text available
Testing an object-oriented software is harder than testing a structural program due to inheritance, states, behaviour of different objects, association, and polymorphism properties of the object-oriented software, which is established based on classes. The classes should be carefully tested using predefined test cases to optimize coverage goals. Because generating test cases manually is labour-intensive, search-based software testing methods that maximize coverage criteria by a search algorithm can be used to generate test cases automatically. Artificial Bee Colony (ABC) algorithm is a powerful search tool that performs balanced exploration and exploitation. In this study, two new multi-criteria and combinatorial ABC algorithms using mutation and crossover operators are proposed to generate a test suite that maximizes a fitness function combining various goals for object-oriented software. First, an archive-based ABC algorithm is proposed to keep covered targets in the archive to use available search resource efficiently. Second, the archive-based ABC algorithm is improved to bring more diversity in the population. To investigate the effect of the archive, the basic ABC algorithm and the archive-based ABC algorithms are compared on a set of classes from SF110 corpus. To validate efficiency of the proposed methods, they are compared to state-of-the-art evolutionary test suite generators and random testing methodology on the classes divided into difficulty levels based on their weight method class values. The experimental results reveal that introducing an archive into the ABC algorithm provides fast convergence compared to the basic ABC and the improved archive-based ABC produces higher coverage for the problems with both low and high difficulty levels.
Article
Full-text available
Software testing is leading toward automation that reduces the effort to find errors or bugs. The identification of test cases and its critical domain requirements is done with generation of test cases. The brooding characteristic of the cuckoo bird is explained through the adaptive cuckoo search meta-heuristic algorithm (ACSA) that further narrates that host nest is used by the cuckoo bird for laying their eggs and the next generation also sees the best quality eggs from the host bird’s nest. This paper focuses on the adoption of ACSA for analysis, generation, and optimization of random test cases. In addition to that, the present work also explains the model driven approach to automatically generate and optimize the test cases with the help of unified modeling language diagram like sequence diagram. Then, the respective sequence diagram is converted into a sequence diagram graph that shows the flow of sequences being produced. Thereafter, it is optimized using ACSA by taking a case study of withdrawal operation of ATM transaction. The said approach is also evaluated in terms of efficiency and usefulness for generating the test cases through simulated experiments. In addition to that, the projected approach also identifies the operational faults as well as message faults.
Article
Full-text available
Software testing is a very important technique to design the faultless software and takes approximately 60% of resources for the software development. It is the process of executing a program or application to detect the software bugs. In software development life cycle, the testing phase takes around 60% of cost and time. Test case generation is a method to identify the test data and satisfy the software testing criteria. Test case generation is a vital concept used in software testing, that can be derived from the user requirements specification. An automatic test case technique determines automatically where the test cases or test data generates utilizing search based optimization method. In this paper, Cuckoo Search and Bee Colony Algorithm (CSBCA) method is used for optimization of test cases and generation of path convergence within minimal execution time. The performance of the proposed CSBCA was compared with the performance of existing methods such as Particle Swarm Optimization (PSO), Cuckoo Search (CS), Bee Colony Algorithm (BCA), and Firefly Algorithm (FA).
Article
Full-text available
Automatic test-suite generation tools are often complex and their behavior is not predictable. To provide a minimum baseline that test-suite generators should be able to surpass, we present PRTest, a random black-box test-suite generator for C programs: To create a test, PRTest natively executes the program under test and creates a new, random test value whenever an input value is required. After execution, PRTest checks whether any new program branches were covered and, if this is the case, the created test is added to the test suite. This way, tests are rapidly created either until a crash is found, or until the user aborts the creation. While this naive mechanism is not competitive with more sophisticated, state-of-the-art test-suite generation tools, it is able to provide a good baseline for Test-Comp and a fast alternative for automatic test-suite generation for programs with simple control flow. PRTest is publicly available and open source.
Article
Full-text available
Automation of test data generation is of prime importance in software testing because of the high cost and time incurred in manual testing. This paper proposes an Improved Crow Search Algorithm (ICSA) to automate the generation of test suites using the concept of mutation testing by simulating the intelligent behaviour of crows and Cauchy distribution. The Crow Search Algorithm suffers from the problem of search solutions getting trapped into the local search. The ICSA attempts to enhance the exploration capabilities of the metaheuristic algorithm by utilizing the concept of Cauchy random number. The concept of Mutation Sensitivity Testing has been used for defining the fitness function for the search based approach. The fitness function used, aids in finding optimal test suite which can achieve high detection score for the Program Under Test. The empirical evaluation of the proposed approach with other popular meta-heuristics, prove the effectiveness of ICSA for test suite generation using the concepts of mutation testing.
Article
Full-text available
Finite-state machines (FSMs) and the W method have been widely used in software testing. However, the W method fails to detect post-processing errors in the implementation under test (IUT) because it ends testing when it encounters a previously visited state. To alleviate this issue, we propose an enhanced fault-detection W method. The proposed method does not stop the test, even if it has reached a previously visited state; it continues to test and check the points that the W method misses. Through various case studies, we demonstrated software testing using the W method and the proposed method. From the results, it can be inferred that the proposed method can more explicitly determine the consistency between design and implementation, and it is a better option for testing larger software. Unfortunately, the testing time of the proposed method is approximately 1.4 times longer than that of the W method because of the added paths. However, our method is more appropriate than the W method for software testing in safety-critical systems, even if this method is time consuming. This is because the error-free characteristics of a safety-critical system are more important than anything else. As a result, our method can be used to increase software reliability in safety-critical embedded systems.
Article
Full-text available
Automatic test case generation is usually based on models of the software under test. However, those models may not exist or may be outdated and so, the test case generation must resort to other artifacts. In a software maintenance context, test cases must adapt to software changes and should be improved continuously to test adequately the new versions of the software. Mutation testing is a fault-based testing technique that evaluates the quality of the tests by applying simple changes to the source code and checking afterwards if the tests are able to detects those changes. This paper presents a web testing approach in which test cases are generated from user execution traces as a way to deal with the absence of models. In addition, it applies mutation operators over those test cases to enrich the test suite. The mutation operators were designed so as to mimic possible real failures. The additional tests are analyzed, and those that generate different outcomes are kept because they exercise additional behavior of the web application under test. At the end, the overall approach is illustrated and validated in a case study.
Article
Full-text available
Software testing is one of most crucial phase in software development life cycle (SDLC). The main function of testing is to cater bugs between interactions of the inputs. It is not possible to eliminate all bugs in one system but by using a suitable testing optimization, it can provide a good enough solution for it. Reducing effort on this phase is not only could lead to numerous bugs between the input interactions, but it also leads to a greater loss such as loss of profits, reputations and even loss of life. Generally, there are three categories of combinatorial testing techniques which is computational, metaheuristic, and hyper heuristic. Ant colony optimization (ACO) is a heuristic technique where its mimic the nature of ants in finding the best route from the nest to the food node and vice versa. Many optimization problems have been solved by using ACO. This paper is to proposed a self-adapting ant colony optimization algorithm using fuzzy logic (ACOF) for combinatorial test suite generation, where it will dynamically determine number of ants and edge selection (i.e. either to explore or to exploit) based on percentage of remaining tuple list and covered test cases.
Article
Full-text available
There exists a direct need to automate the process of Test Suite Generation to get the most optimal results as testing accounts for more than 40% of total cost. One method to solve this problem is the use of Meta-Heuristic Algorithms which iteratively improve the test data to reach the most optimized test suites. This study focuses on the performance evaluation of six Meta-Heuristic Algorithms namely: Hill Climbing Algorithm (HCA), Particle Swarm Optimization (PSO), Firefly Algorithm (FA), Cuckoo Search Algorithm (CS), Bat Algorithm (BA) and Artificial Bee Colony Algorithm (ABC) by using their standard implementation to optimize the Path Coverage and Branch Coverage produced by the Test Data. The goal of the study was to find the best suited algorithm to narrow down future research in the field of Test Automation for Path Coverage Based Optimization Approaches. Each algorithm was first implemented to automatically generate test suites based on the program under test. This was followed by the performance evaluation of each algorithm for five programs written in Java. The algorithms were compared using process metrics: average time, best time, worst time and product metrics: path coverage & objective function values of the generated test suites. Results indicated ABC as the best suited algorithm as it gave the most optimal Test Suites in reasonable time. BA was found to be the fastest but produced less optimal results. FA was found to be the slowest algorithm while CS, PSO and HCA performed in between. These results show the relative performance of the six algorithms for this scenario and may be used by future researchers to narrow down and improve the best performing algorithms for Path Coverage Based Optimization Approaches.
Article
Full-text available
Test case generation is a multi objective problem as the goal is to achieve multiple targets. In existing work, emphasis is given to generate test cases to achieve maximum path coverage. For quality testing, coverage of critical path is more important than percentage of code coverage. The objective of this paper is to generate test cases to achieve maximum path coverage with a challenge of covering a critical path, within the available test resources. At the time of automatic test case generation, a path is critical if the probability of covering the path is low. Search based techniques use metaheuristic algorithms for automated test case generator. Fitness function plays an important role in searching techniques. We propose a fitness function, Improved Combined Fitness (ICF) function, using Adaptive Particle Swarm Optimization (APSO), to generate test cases automatically based on path coverage criteria. We have conducted experiments on three well-known case studies and observed that though both Particle Swarm Optimization (PSO) and APSO with the existing fitness functions, branch distance function and branch distance combined with approximation level, give maximum path coverage, sometimes fail to achieve critical path. Our proposed ICF function applied on APSO gives better result in terms of number of path coverage.
Article
Full-text available
Software testing is very time consuming, labor-intensive and complex process. It is found that 50% of the resources of the software development are consumed for testing. Testing can be done in two different ways such as manual testing and automatic testing. Automatic testing can overcomes the limitations of manual testing by decreasing the cost and time of testing process. Path testing is the strongest coverage criteria among all white box testing techniques as it can detect about 65% of defects present in a SUT. With the help of path testing, the test cases are created and executed for all possible paths which results in 100% statement coverage and 100% branch coverage .This paper presents a systematic review of test data generation and optimization for path testing using Evolutionary Algorithms (EAs). Different EAs like GA, PSO, ACO, and ABCO based methods has been already proposed for automatic test case generation and optimization to achieve maximum path coverage.
Article
Full-text available
The object‐oriented (OO) systems have emerged as the core systems in every field. Test case generation (TCG) for these systems has been identified as one of the crucial activity of software testing. Many researchers have been working in the area of TCG to raise the quality and effectiveness of the OO software systems. It is impossible to test the system exhaustively because of limitations of time, monetary cost and human efforts involved in the process of generation of all test cases. Presently, there is no adequate method that considers the contextual demand for the generation of test cases. Therefore, this study proposes a contextual demand‐based TCG for OO systems using test paths or scenarios of sequence diagrams (SDs). Unlike the existing approaches, the proposed method considers flexible approach for generation of test cases as per the contextual demand. For the implementation of the proposed technique, two case studies, i.e. the sample SD and the SD of automated teller machine (ATM) system, have been considered. The evaluation of the proposed method showed the significant consideration of the user demand for generating the final test cases.
Conference Paper
Full-text available
Over the last decades, white-box search-based techniques have been applied to automate the design and the execution of test cases. While most of the research effort has been devoted to unit-level testing, integration-level test case generation requires to solve several open challenges, such as the combinatorial explosion of conditions or precondition failures. This paper summarizes these challenges in white-box testing and highlights possible research directions to overcome them.
Article
Full-text available
Software engineering comprises of several disciplines. Testing, the subject of this paper, is an important phase which is still largely unpredictable, expensive and ad hoc. Software testing has already been framed as an optimization problem i.e. to solve problems in software testing various optimization techniques can be used. To adequately test the software, various rules, called test adequacy criteria are used. In this paper, a review of optimization techniques used in domains: test case generation, selection, minimization and prioritization of testing has been presented. The review is conducted by selecting quality literature from journals, conferences, workshops and symposium. State of the art issues in software testing has been addressed by analyzing the reviewed literature on the basis of the domain of testing, test adequacy criteria and optimization technique used. After analysis of the considered literature, authors were able to conclude the limitations of existing work, list of traditional and recently proposed adequacy criteria as well as combinations of adequacy criteria for multi-objective optimization and optimization techniques which may be used in optimization of software testing.
Article
Full-text available
This work introduces a heuristic-guided branching search algorithm for model-based, mutation-driven test-case generation. The algorithm is designed towards the efficient and computationally tractable exploration of discrete, non-deterministic models with huge state spaces. Asynchronous parallel processing is a key feature of the algorithm. The algorithm is inspired by the successful path planning algorithm Rapidly exploring Random Trees (RRT). We adapt RRT in several aspects towards test-case generation. Most notably, we introduce parametrized heuristics for start and successor state selection, as well as a mechanism to construct test cases from the data produced during the search. We implemented our algorithm in the existing test-case generation framework MoMuT. We present an extensive evaluation of the proposed heuristics and parameters of the algorithm, based on a diverse set of demanding models obtained in an industrial context. In total, we continuously utilized 128 CPU cores on three servers for several weeks to gather the experimental data presented. We show that branching search works well and the use of multiple heuristics is justified. With our new algorithm, we are now able to process models consisting of over 2,300 concurrent objects. To our knowledge, there is no other mutation-driven test-case generation tool that is able to process models of this magnitude.
Article
Full-text available
What if we could know that a program is buggy, even if we could not tell whether or not its observed output is correct? This is one of the key strengths of metamorphic testing, a technique where failures are not revealed by checking an individual concrete output, but by checking the relations among the inputs and outputs of multiple executions of the program under test. Two decades after its introduction, metamorphic testing has become a fully-fledged testing technique with successful applications in multiple domains, including online search engines, autonomous machinery, compilers, Web APIs, and deep learning programs, among others. This article serves as a hands-on entry point for newcomers to metamorphic testing, describing examples, possible applications, and current limitations, providing readers with the basics for the application of the technique in their own projects.
Conference Paper
Full-text available
Metamorphic testing is a well known approach to tackle the oracle problem in software testing. This technique requires the use of source test cases that serve as seeds for the generation of follow-up test cases. Systematic design of test cases is crucial for the test quality. Thus, source test case generation strategy can make a big impact on the fault detection effectiveness of metamorphic testing. Most of the previous studies on metamorphic testing have used either random test data or existing test cases as source test cases. There has been limited research done on systematic source test case generation for metamorphic testing. This paper provides a comprehensive evaluation on the impact of source test case generation techniques on the fault finding effectiveness of metamorphic testing. We evaluated the effectiveness of line coverage, branch coverage, weak mutation and random test generation strategies for source test case generation. The experiments are conducted with 77 methods from 4 open source code repositories. Our results show that by systematically creating source test cases, we can significantly increase the fault finding effectiveness of metamorphic testing. Further, in this paper we introduce a simple metamorphic testing tool called "METtester" that we use to conduct metamorphic testing on these methods.
Article
Full-text available
Traditional test case generation approaches focus on design and implementation models while a large percentage of software errors are caused by the lack of understanding in the early phases. One of the most important models in the early phases of software development is business process model which closely resembles the real world and captures the requirements precisely. The aim of this paper is to present a model-based approach to automatically generate test cases from business process models. We first model business processes and convert them to state graphs. Then, the graphs are traversed and transformed to the input format of the "Spec explorer" tool that generates the test cases. Furthermore, we conduct a study to evaluate the impact of process characterizations on the performance of the proposed method.
Article
Full-text available
High code coverage is measured by the process of software testing typically using automatic test case generation tools. This standard approach is usually used for unit testing to improve software reliability. Most automated test case generation tools focused just on code coverage without considering its cost and redundancy between generated test cases. To obtain optimized high code coverage and to ensure minimum cost and redundancy a Multi-Objectives Evolutionary Algorithm approach (MOEA) is set in motion. An efficient approach is proposed and applied to different algorithms from MOEA Frame from the separate library with three fitness functions for Coverage, Cost, and Redundancy. Four MEOA algorithms have been proven reliable to reach above the 90 percent code coverage: NSGAII, Random, SMSEMOA, and ε-MOEA. These four algorithms are the key factors behind the MOEA approach.
Article
Full-text available
The generation of multiple-path test cases can greatly enhance the efficiency of path-wise testing. Various methods adopting meta-heuristic algorithm to generate multiple-path test cases have been proposed, but existing methods focus on improving the meta-heuristic algorithm to get better test case generation efficiency, and test cases covering each path needs to be generated by meta-heuristic algorithm searching. To improve efficiency, a test case generation method for multiple-path coverage is proposed in this study, which combines a particle swarm optimisation (PSO) algorithm with metamorphic relations (MRs). The method first generates a test case using the PSO algorithm, and then generates new test cases by repeatedly using MRs between test cases. This method reduces evolving numbers of PSO algorithm. The experimental results show that the proposed method can significantly enhance the efficiency in terms of fitness evaluations and time consumption.
Article
Full-text available
Testing gives means pertaining to assuring software performance. As it has been proved that the software testing phase is one of the most critical and important phases in the software development lifecycle as it lead to increase in cost, time and effort. The total aim of software industry is actually to make a certain start associated with high quality software for the end user. Software testing has quite a few underlying concerns, which are very important and need to pay attention on these issues which are effectively generating, prioritization of test cases, etc. To over come from these issues we need to pay attention and focus. In this paper we are discussing various techniques which are used to improve automated test case generation and comparing it with other existing techniques. In this we are also proposing new work which can be done in future.
Article
Artificial intelligence systems, such as Sentiment Analysis (SA) systems, typically learn from large amounts of data that may reflect human bias. Consequently, such systems may exhibit unintended demographic bias against specific characteristics (e.g., gender, occupation, country-of-origin, etc.). Such bias manifests in an SA system when it predicts different sentiments for similar texts that differ only in the characteristic of individuals described. To automatically uncover bias in SA systems, this paper presents BiasFinder, an approach that can discover biased predictions in SA systems via metamorphic testing. A key feature of BiasFinder is the automatic curation of suitable templates from any given text inputs, using various Natural Language Processing (NLP) techniques to identify words that describe demographic characteristics. Next, BiasFinder generates new texts from these templates by mutating words associated with a class of a characteristic (e.g., gender-specific words such as female names, she, her). These texts are then used to tease out bias in an SA system. BiasFinder identifies a bias-uncovering test case (BTC) when an SA system predicts different sentiments for texts that differ only in words associated with a different class (e.g., male vs. female) of a target characteristic (e.g., gender). We evaluate BiasFinder on 10 SA systems and 2 large scale datasets, and the results show that BiasFinder can create more BTCs than two popular baselines. We also conduct an annotation study and find that human annotators consistently think that test cases generated by BiasFinder are more fluent than the two baselines.
Article
Search-based techniques have been successfully used to automate test case generation. Such approaches allocate a fixed search budget to generate test cases aiming at maximizing code coverage. The search budget plays a crucial role; due to the hugeness of the search space, the higher the assigned budget, the higher the expected coverage. Code components have different structural properties that may affect the ability of search-based techniques to achieve a high coverage level. Thus, allocating a fixed search budget for all the components is not recommended and a component-specific search budget should be preferred. However, deciding the budget to assign to a given component is not a trivial task. In this article, we introduce Budget Optimization for Testing (BOT), an approach to adaptively allocate the search budget to the classes under test. BOT requires information about the branch coverage that will be achieved on each class with a given search budget. Therefore, we also introduce BRANCHOS, an approach that predicts coverage in a budget-aware way. The results of our experiments show that (i) BRANCHOS can approximate the branch coverage in time with a low error, and (ii) BOT can significantly increase the coverage achieved by a test generation tool and the effectiveness of generated tests.
Article
Context Agent-based models play an important role in simulating complex emergent phenomena and supporting critical decisions. In this context, a software fault may result in poorly informed decisions that lead to disastrous consequences. The ability to rigorously test these models is therefore essential. Objective Our objective is to summarise the state-of-the-art techniques for test case generation in agent-based models and identify future research directions. Method We have conducted a systematic literature review in which we pose five research questions related to the key aspects of test case generation in agent-based models: What are the information artifacts used to generate tests? How are these tests generated? How is a verdict assigned to a generated test? How is the adequacy of a generated test suite measured? What level of abstraction of an agent-based model is targeted by a generated test? Results Out of the 464 initial search results, we identified 24 primary publications. Based on these primary publications, we formed a taxonomy to summarise the state-of-the-art techniques for test case generation in agent-based models. Our results show that whilst the majority of techniques are effective for testing functional requirements at the agent and integration levels of abstraction, there are comparatively few techniques capable of testing society-level behaviour. Furthermore, the majority of techniques cannot test non-functional requirements or “soft goals”. Conclusions This paper reports insights into the key developments and open challenges concerning test case generation in agent-based models that may be of interest to both researchers and practitioners. In particular, we identify the need for test case generation techniques that focus on societal and non-functional behaviour, and a more thorough evaluation using realistic case studies that feature challenging properties associated with a typical agent-based model.
Article
Context Typically, search-based test data generation methods search on a population of program input values. Program input values can be regarded as solutions to underlying path constraints over program input parameters. One way to discover these path constraints is to use the symbolic execution method. Search-based methods attempt to find input values which are solutions to these path constraints, without knowing the actual constraints. Objective In this paper, we show that we can search for the underlying path constraints using search-based methods, without resorting to symbolic execution. Trying to discover the exact or a good enough approximation of the underlying constraints may lead to a more targeted search, compared to directly searching for program input values. Besides, the construction of approximate constraints by searching may help to avoid some problems of symbolic execution. Method The proposed method uses genetic programming for learning constraints on program input parameters. Results To evaluate the performance of the proposed approach, a series of experiments have been conducted on a number of different benchmark programs. For 91.8% of benchmark programs, the proposed method achieved the best efficiency among the competitive algorithms. Conclusion The results show that, if constraint solving can be provided for some or all parameter types of the methods of programs under test, our approach can improve the efficiency and effectiveness of search-based test data generation.
Chapter
Software testing is an important process of detecting bugs in the software product thereby a quality software product is developed. Verification and Validation (V & V) activities are the effective methods employed to achieve quality. Static and dynamic testing activities are performed during V & V. During static testing, the program code is not executed while in dynamic testing (Black Box and White Box), the execution of the program code is performed. Effective test cases are designed by both these methods. Tables are employed to represent test case documentation. The most beneficial representation - State table based testing, for generating test inputs is explained with the help of state graphs and state tables. This technique is mainly employed in embedded system software testing, real time applications and web application based software product testing where time constraints are a major criteria. Automatic generation of test cases will help to reduce time overhead in testing activities. Our study is to develop optimum test cases by a modified Ant Colony Optimization (ACO) technique in an automated method and it ensures maximum coverage. The prediction model used in this paper ensures better accuracy of the design of test inputs. A comparison of the similar optimization techniques was also discussed that is used in automated test case generation. A case study of the various states during the execution of a task in an operating system has been presented to illustrate our approach.
Article
In this article, test diversity has been suggested to be a valid way to improve test suite effectiveness. Extended finite state machine (EFSM) is a widely used formal model, but little attention is paid on the test suite generation with more diversity. EFSM test suite generation involves test paths generation and test data generation. Considering the discrepancy between test paths has a more crucial impact on the diversity of test suite, compared with the difference between test data, this article, therefore, mainly concerns the test paths generation with more diversity for EFSM models. Hence, the factors that influence the discrepancy between test paths are investigated. Then based on these factors, an integrated distance metric is designed to evaluate the dissimilarity between test paths, and a diversity measurement for EFSM test suite is presented. Furthermore, a diversity-oriented test suite generation (DOTSG) method is proposed where a dissimilarity-based fitness function and diversity-oriented update strategy are adopted in traditional coverage-oriented EFSM test suite generation (COTSG) by genetic algorithm. The experimental results show that, compared to COTSG, our DOTSG can not only generate more diverse test suite to satisfy a certain coverage criteria, improving the fault detection capability of the test suite, but also decrease the evolution time cost and the size of test suite generated.
Article
Model‐based test case generation techniques provide a mechanism to derive tests systematically. This study provides a systematic mapping of test case generation techniques based on UML interaction diagrams. The study compares the test case generation techniques regarding their capabilities and limitations, and it also assesses the reporting quality of the primary studies. We can conclude that the studies presenting test case generation techniques using UML interaction diagrams were not following the guidelines for research methods (eg, case studies or experiments). Solutions were not empirically evaluated in industrial contexts. Our study revealed that better tool support is needed to introduce the UML interaction diagram–based test case generation techniques in the industry.
Article
Meta-heuristic Artificial Bee Colony Algorithm finds its applications in the optimization of numerical problems. The intelligent searching behaviour of honey bees forms the base of this algorithm. The Artificial Bee Colony Algorithm is responsible for performing a global search along with a local search. One of the major usage areas of Artificial Bee Colony Algorithm is software testing, such as in structural testing and test suite optimization. The implementation of Artificial Bee Colony Algorithm in the field of data flow testing is still unexplored. In data flow testing, the definition-use paths which are not definition-clear paths are the potential trouble spots. The main aim of this paper is to present a simple and novel algorithm by making use of artificial bee colony algorithm in the field of data flow testing to find out and prioritize the definition-use paths which are not definition-clear paths.
Article
Software testing is one of the most labor-intensive and crucial phases of the software development life cycle. It accounts for about 50–60% of the total cost of a project and requires a substantial amount of effort to ensure that the quality of the software is not compromised. Hence, optimizing the process of software testing has been of paramount importance in the IT industry. In recent times, there has been a significant shift toward software test automation to ensure the quality of the software while minimizing the cost and effort. This study proposes a statistical model analyzing the various factors that affect the testing cost, quality and effort by studying the behavioral pattern of professionals employed in the field of testing. The paper studies the factors, project duration, Software Complexity (SC), testing tools and automated test suite generation, while contrasting it with the overall software test automation, with the aim of finding a measure of how much automation affects the cost, quality and effort in practice. The results indicated that automation accounted for a variation of about 25% in each of the factors: cost, quality and effort much more than factors such as project duration, SC and testing tool . Further, the analysis indicated that the effect of SC on cost and effort was extremely low. This suggests that automation improved the quality while reducing the cost and effort irrespective of the complexity of the software being developed.
Article
Testing is one of the most important phases in the development of any product or software. Various types of software testing exist that have to be done to meet the need of the software. Regression testing is one of the crucial phases of testing where testing of a program is done for the original test build along with the modifications. In this article, various studies proposed by the authors have been analysed focusing on test cases generation and their approach toward web application. A detailed study was conducted on Regression Test Case Generation and its approaches toward web application. From our detailed study, we have found that very few approaches and methodologies have been found that provide the real tool for test case generation. There is a need of an automated regression testing tool to generate the regression test cases directly based on user requirements. These test cases have to be generated and implemented by the tool so that the reduction in the overall effort and cost can be achieved. From our study, we have also found that regression testing for web applications was not investigated much, but in today's scenario web applications are an integral part of our daily life and so that needs to be tested for regression testing.
Conference Paper
Graphical User Interface (GUI) visualizes computer programs for the purpose of facilitating interaction between users and various computing devices. Today's computers, smart phones and even small devices such as watches are equipped with GUIs. Unlike command based interaction, GUI uses images, labels, push buttons, radio buttons, etc. for the effective communication of users with a software system. GUI testing is a critical part of software testing as it is the door to the actual functionality of software. For the quality assurance, GUI functional testing of a software validates proper interaction between the interface and the user without considering any coding details. In this paper, a strategy based on fuzzy Adaptive Teaching Learning-based Optimization (ATLBO) algorithm, a variant of the basic Teaching Learning-based Optimization (TLBO) algorithm, for GUI functional testing is proposed. ATLBO utilizes Event-Interaction Graph (EIG) for the generation of quality test cases. The proposed strategy has produced competitive experimental results against the basic TLBO and other test case generation algorithms.