Figure 6 - available via license: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International
Content may be subject to copyright.
% of Mutants killed by all six MRs using 4 test suite strategies (Branch, Line Coverage, Weak Mutation and Random)
Source publication
Metamorphic testing is a well known approach to tackle the oracle problem in software testing. This technique requires the use of source test cases that serve as seeds for the generation of follow-up test cases. Systematic design of test cases is crucial for the test quality. Thus, source test case generation strategy can make a big impact on the f...
Context in source publication
Context 1
... see how each source test case generation technique performs with individual MRs, Figure 6 illustrates the percentage of mutants killed by all six MRs separately using weak mutation, line, branch coverage and random test suites. Weak mutation has the highest percentage of killed mutants in all the six MRs. ...
Similar publications
Software testing is an essential part of the software development cycle to improve the code quality. Typically, a unit test consists of a test prefix and a test oracle which captures the developer's intended behaviour. A known limitation of traditional test generation techniques (e.g. Randoop and Evosuite) is that they produce test oracles that cap...
Metamorphic testing is a well known approach to tackle the oracle problem in software testing. This technique requires the use of source test cases that serve as seeds for the generation of follow-up test cases. Systematic design of test cases is crucial for the test quality. Thus, source test case generation strategy can make a big impact on the f...
Citations
... A systematic literature review (SLR) methodology was employed to synthesize existing research on software security testing. This method provides evidence-based insights, identifies research gaps, and guides new research directions [37] and [38]. ...
________________________________________________________________________________________ Abstract: Software systems are integral to modern organizations, necessitating rigorous testing to ensure security and integrity. However, with the evolution of technology, vulnerabilities and threats to software security are on the rise. Metaheuristic algorithms (MHS) or evolutionary techniques have emerged as valuable tools in addressing these challenges. This research aims to explore and evaluate evolutionary software security testing techniques comprehensively. Specific objectives include analyzing different test cases and strategies, identifying commonly targeted security vulnerabilities, assessing cost-effective and scalable testing techniques, and developing a framework for selecting optimal evolutionary testing methods. The methodology employs a systematic literature review across five major databases, selecting 52 relevant papers. Findings indicate prevalent security vulnerabilities such as Cross-site scripting XSS, Buffer overflow/stack overflow, SQL/XML injection, etc. The commonly used genetic algorithms for software security testing are Genetic algorithm, Particle swarm optimization, and Simulated annealing. Cost-effective and scalable MHS algorithms are ranked, with the Genetic algorithm emerging as the most effective. Additionally, a model for selecting and utilizing MHS algorithms is proposed based on research findings. This study offers valuable insights for researchers and practitioners, outlining future research avenues and providing practical guidelines for employing MHS algorithms in software security testing.
Metamorphic testing is a technique that uses metamorphic relations (i.e., necessary properties of the software under test), to construct new test cases (i.e., follow-up test cases), from existing test cases (i.e., source test cases). Metamorphic testing allows for the verification of testing results without the need of test oracles (a mechanism to detect the correctness of the outcomes of a program), and it has been widely used in many application domains to detect real-world faults. Numerous investigations have been conducted to further improve the effectiveness of metamorphic testing. Recent studies have emerged suggesting a new research direction on the generation and selection of source test cases that are effective in fault detection. Herein, we present two important findings: i) a mutant reduction strategy that is applied to increase the testing efficiency of source test cases, and ii) a test suite minimization technique to help reduce the testing costs without trading off fault-finding effectiveness. To validate our results, an empirical study was conducted to demonstrate the increase in efficiency and fault-finding effectiveness of source test cases. The results from the experiment provide evidence to support our claims.
Metamorphic testing is a technique that makes use of some necessary properties of the software under test, termed as metamorphic relations, to construct new test cases, namely follow-up test cases, based on some existing test cases, namely source test cases. Due to the ability of verifying testing results without the need of test oracles, it has been widely used in many application domains and detected lots of real-life faults. Numerous investigations have been conducted to further improve the effectiveness of metamorphic testing, most of which were focused on the identification and selection of “good” metamorphic relations. Recently, a few studies emerged on the research direction of how to generate and select source test cases that are effective in fault detection. In this paper, we propose a novel approach to generating source test cases based on their associated path constraints, which are obtained through symbolic execution. The path distance among test cases is leveraged to guide the prioritization of source test cases, which further improve the efficiency. A tool has been developed to automate the proposed approach as much as possible. Empirical studies have also been conducted to evaluate the fault-detection effectiveness of the approach. The results show that this approach enhances both the performance and automation of metamorphic testing. It also highlights interesting research directions for further improving metamorphic testing.
Automatic software fault localization serves as a significant role in helping developers find bugs efficiently. Existing approaches can be categorized into static methods and dynamic ones, which have improved the fault locating greatly by analyzing static features from the source code or tracking dynamic behaviors during the runtime respectively. However, the localization accuracy is still far from satisfactory for users. To enhance the capability of detecting software faults with the statement granularity, this paper proposes ALBFL, a novel neural ranking model, combining the static and dynamic features could obtain excellent fault localization accuracy. Firstly, ALBFL learns the semantic features of the source code by a transformer encoder. Then, it integrates them with other static features and dynamic features, i.e., statistical features, Spectrum-Based Fault Localization (SBFL) features, and mutation features, through a self-attention layer. Next, in order to evaluate the faulty possibility of each software statement, the integration results output by self-attention layer are fed into the LambdaRank model, which ranks the suspicious statements in descending order. Finally, we test our model on the authoritative dataset–defect4J, in which consists of 5 open-source projects and a total of 357 faulty programs. It shows that the defect statements identified by ALBFL are three times more than 11 traditional SBFL methods, and outperform two state-of-the-art approaches by more than 14% in [email protected]
Context: Automatic software fault localization serves as a significant purpose in helping developers solve bugs efficiently. Existing approaches for software fault localization can be categorized into static methods and dynamic ones, which have improved the fault locating ability greatly by analyzing static features from the source code or tracking dynamic behaviors during the runtime respectively. However, the accuracy of fault localization is still unsatisfactory.
Objective: To enhance the capability of detecting software faults with the statement granularity, this paper puts forward ALBFL, a novel neural ranking model that combines the static and dynamic features, which obtains excellent fault localization accuracy. Firstly, ALBFL learns the semantic features of the source code by a transformer encoder. Then, it exploits a self-attention layer to integrate those static features and dynamic features. Finally, those integrated features are fed into a LambdaRank model, which can list the suspicious statements in descending order by their ranked scores.
Method: The experiments are conducted on an authoritative dataset (i.e., Defect4J), which includes 5 open-source projects, 357 faulty programs in total. We evaluate the effectiveness of ALBFL, effectiveness of combining features, effectiveness of model components and aggregation on method level.
Result: The results reflect that ALBFL identifies triple more faulty statements than 11 traditional SBFL methods and outperforms 2 state-of-the-art approaches by on average 14% on ranking faults in the first position.
Conclusions: To improve the precision of automatic software fault localization, ALBFL combines neural network ranking model equipped with the self-attention layer and the transformer encoder, which can take full use of various techniques to judge whether a code statement is fault-inducing or not. Moreover, the joint architecture of ALBFL is capable of training the integration of these features under various strategies so as to improve accuracy further. In the future, we plan to exploit more features so as to improve our method's efficiency and accuracy.
