Figure 2 - uploaded by Prashanta Saha
Content may be subject to copyright.
Source publication
Metamorphic testing is a well known approach to tackle the oracle problem in software testing. This technique requires the use of source test cases that serve as seeds for the generation of follow-up test cases. Systematic design of test cases is crucial for the test quality. Thus, source test case generation strategy can make a big impact on the f...
Contexts in source publication
Context 1
... [17] is a simple tool that we are developing to auto- mate the MT process on a given Java program. This tool allows users to specify MRs and source test cases through a simple XML file. METtester transforms the source test cases according to the specified MRs and conducts MT on the given program. Figure 2 shows the high level architecture of the tool. Below we describe the important components of the ...
Similar publications
Software testing is an essential part of the software development cycle to improve the code quality. Typically, a unit test consists of a test prefix and a test oracle which captures the developer's intended behaviour. A known limitation of traditional test generation techniques (e.g. Randoop and Evosuite) is that they produce test oracles that cap...
Citations
... [cs.SE] 30 Dec 2024 amined the characteristic of effective MRs and proposed some qualitative guidelines for selecting effective MRs [19], [20]. Apart from the identification of MRs, a variety of source test input generation approaches have been proposed [21], [22], and studies have investigated their impact on the fault detection effectiveness of MT [23], [24]. ...
Metamorphic testing (MT) is a simple yet effective technique to alleviate the oracle problem in software testing. The underlying idea of MT is to test a software system by checking whether metamorphic relations (MRs) hold among multiple test inputs (including source and follow-up inputs) and the actual output of their executions. Since MRs and source inputs are two essential components of MT, considerable efforts have been made to examine the systematic identification of MRs and the effective generation of source inputs, which has greatly enriched the fundamental theory of MT since its invention. However, few studies have investigated the test adequacy assessment issue of MT, which hinders the objective measurement of MT's test quality as well as the effective construction of test suites. Although in the context of traditional software testing, there exist a number of test adequacy criteria that specify testing requirements to constitute an adequate test from various perspectives, they are not in line with MT's focus which is to test the software under testing (SUT) from the perspective of necessary properties. In this paper, we proposed a new set of criteria that specifies testing requirements from the perspective of necessary properties satisfied by the SUT, and designed a test adequacy measurement that evaluates the degree of adequacy based on both MRs and source inputs. The experimental results have shown that the proposed measurement can effectively indicate the fault detection effectiveness of test suites, i.e., test suites with increased test adequacy usually exhibit higher effectiveness in fault detection. Our work made an attempt to assess the test adequacy of MT from a new perspective, and our criteria and measurement provide a new approach to evaluate the test quality of MT and provide guidelines for constructing effective test suites of MT.
... Among many solutions to test the oracle problem, metamorphic testing (MT) is the most popular technique that tackles the oracle problem in software testing of IPAs [5]. MT was first proposed by Chen et al. in 1998 [6]. ...
... The foremost step is the generation of source test cases (original test cases). In literature, source test cases are generated either through some traditional test case generation techniques as discussed earlier or through some tool such as EvoSuite (it generates source test cases automatically through coverage criterion) [5]. Nowadays, few researches are emerged in the direction of generation and selection of source test cases that are effective in fault detection [24]. ...
Testing an intricate plexus of advanced software system architecture is quite challenging due to the absence of test oracle. Metamorphic testing is a popular technique to alleviate the test oracle problem. The effectiveness of metamorphic testing is dependent on metamorphic relations (MRs). MRs represent the essential properties of the system under test and are evaluated by their fault detection rates. The existing techniques for the evaluation of MRs are not comprehensive, as very few mutation operators are used to generate very few mutants. In this research, we have proposed six new MRs for dilation and erosion operations. The fault detection rate of six newly proposed MRs is determined using mutation testing. We have used eight applicable mutation operators and determined their effectiveness. By using these applicable operators, we have ensured that all the possible numbers of mutants are generated, which shows that all the faults in the system under test are fully identified. Results of the evaluation of four MRs for edge detection show an improvement in all the respective MRs, especially in MR1 and MR4, with a fault detection rate of 76.54% and 69.13%, respectively, which is 32% and 24% higher than the existing technique. The fault detection rate of MR2 and MR3 is also improved by 1%. Similarly, results of dilation and erosion show that out of 8 MRs, the fault detection rates of four MRs are higher than the existing technique. In the proposed technique, MR1 is improved by 39%, MR4 is improved by 0.5%, MR6 is improved by 17%, and MR8 is improved by 29%. We have also compared the results of our proposed MRs with the existing MRs of dilation and erosion operations. Results show that the proposed MRs complement the existing MRs effectively as the new MRs can find those faults that are not identified by the existing MRs.
... In fact, most of the previous studies in MT have used randomly generated test cases or existing test cases as source test cases when conducting MT [5]- [8], [12]. Our previous work showed that the effectiveness of MT can be improved by systematically generating the source test cases based on some coverage criteria such as line, branch, and weak mutation (WM) [9]. But, it is sub-optimal to use a combination of all the coverage-based techniques to test numerical programs. ...
... For example, there exist some unique characteristics of weather systems that help testers to find the correct MRs. We can create source test cases using techniques like random testing [12], structural testing [13], or search-based testing [9]. Follow-up test cases are generated by applying the input transformation specified by the MRs. ...
Metamorphic testing is a technique that uses metamorphic relations (i.e., necessary properties of the software under test), to construct new test cases (i.e., follow-up test cases), from existing test cases (i.e., source test cases). Metamorphic testing allows for the verification of testing results without the need of test oracles (a mechanism to detect the correctness of the outcomes of a program), and it has been widely used in many application domains to detect real-world faults. Numerous investigations have been conducted to further improve the effectiveness of metamorphic testing. Recent studies have emerged suggesting a new research direction on the generation and selection of source test cases that are effective in fault detection. Herein, we present two important findings: i) a mutant reduction strategy that is applied to increase the testing efficiency of source test cases, and ii) a test suite minimization technique to help reduce the testing costs without trading off fault-finding effectiveness. To validate our results, an empirical study was conducted to demonstrate the increase in efficiency and fault-finding effectiveness of source test cases. The results from the experiment provide evidence to support our claims.
... (2) Metamorphic relations should not be formally described. (Tao et al., 2010;Hui and Huang, 2013;Bandaru and Albert Mayan, 2016;Saha and Kanewala, 2018;Lv et al., 2018;Segura et al., 2020;Asyrofi et al., 2021) Test Case Generation Techniques ...
Software Industry is evolving at a very fast pace since last two decades. Many software developments, testing and test case generation approaches have evolved in last two decades to deliver quality products and services. Testing plays a vital role to ensure the quality and reliability of software products. In this paper authors attempted to conduct a systematic study of testing tools and techniques. Six most popular e-resources called IEEE, Springer, Association for Computing Machinery (ACM), Elsevier, Wiley and Google Scholar to download 738 manuscripts out of which 125 were selected to conduct the study. Out of 125 manuscripts selected, a good number approx. 79% are from reputed journals and around 21% are from good conference of repute. Testing tools discussed in this paper have broadly been divided into five different categories: open source, academic and research, commercial, academic and open source, and commercial & open source. The paper also discusses several benchmarked datasets viz. Evosuite 10, SF100 Corpus, Defects4J repository, Neo4j, JSON, Mocha JS, and Node JS to name a few. Aim of this paper is to make the researchers aware of the various test case generation tools and techniques introduced in the last 11 years with their salient features.
... Previous studies have used special case [9] and random testing [10] techniques to generate source test cases. Further, previous studies have shown that using coverage-based test inputs as source inputs would improve the fault detection effectiveness of MT compared to random test inputs [11]. As shown in the above To satisfy this MR the follow-up test output should be greater than the source output. ...
... Finally, source test cases were generated by replacing the symbolic inputs with real values. Saha et al. [11] applied a coveragebased testing technique to generate test cases for MT. They compared their results with randomly generated test cases, and it outperforms the effectiveness of randomly generated test suite. ...
... Sometimes, the correctness of the program outcomes is not easy to specifythe test oracle problem. This problem was studied in [15] for scientific software and a metamorphic testing was proposed, it specifies output changes related to input changes. ...
... The heat map for alert types revealed bigger number of some critical types: unavailable sender (187 and 146 for log11 and 12, respectively), SenderCriticalError (533-2045 for logs [13][14][15], AccountJobinactive (555-1405 for logs 11 and 10), transactioBad Message (218-251 for logs 1-3), securityviolation (200-604 for logs 18,11,12), transactionResponseTimeout (100-604 for logs 13). Five logs did not comprise alerts, for 4 logs alerts constituted a fraction of percent, 3 logs with about 2-5%, the remaining ones below 2%. ...
... The heatmaps for registered exceptions (25 types) showed for most logs low values (0-10) per exception, and 10-50 exceptions per log file in total. However, two exceptions FileProcessingFileServer criticalerror and SourceDirectorynotFoundError showed very big values (533-2045 for logs [13][14][15]. They correlated with Source Scheduler (552-2047) and alert type SchedulerErrorCritical (533-2045) for the same logs. ...
Software reliability depends on the performed tests. Bug detection and diagnosis are based on test outcome (oracle) analysis. Most practical test reports do not provide sufficient information for localizing and correcting bugs. We have found the need to extend the space of test result observation in data and time perspectives. This resulted in tracing supplementary test result features in event logs. They are explored with combined text mining and log parsing techniques. Another important point is correlating test life cycle with project development history journaled in issue tracking and software version control repositories. Dealing with the outlined problems, neglected in the literature, we have introduced original analysis schemes. They focus on assessing test coverage, reasons of low diagnosability, and test result profiles. Multidimensional investigation of test features and their management is supported with the developed test infrastructure. This assures a holistic insight into the test efficiency to identify test scheme deficiencies (e.g., functional inadequacy, aging, insufficient coverage) and possible improvements (test set updates). Our studies have been verified in relevance to a real commercial project (industrial case study) and confronted with the experience of testers engaged in other projects.
... It is worthwhile to study whether and how the concept can be used in the construction of follow-up test cases. A promising direction is to consider more sophisticated distance measures and coverage criteria discussed in Saha and Kanewala (2018) for prioritizing the source test cases, with an aim to further improve the fault detection efficiency of MT. In addition, compared with random prioritization, it is worthwhile to study the improvement of fault detection efficiency achieved by different prioritization strategies. ...
Metamorphic testing is a technique that makes use of some necessary properties of the software under test, termed as metamorphic relations, to construct new test cases, namely follow-up test cases, based on some existing test cases, namely source test cases. Due to the ability of verifying testing results without the need of test oracles, it has been widely used in many application domains and detected lots of real-life faults. Numerous investigations have been conducted to further improve the effectiveness of metamorphic testing, most of which were focused on the identification and selection of “good” metamorphic relations. Recently, a few studies emerged on the research direction of how to generate and select source test cases that are effective in fault detection. In this paper, we propose a novel approach to generating source test cases based on their associated path constraints, which are obtained through symbolic execution. The path distance among test cases is leveraged to guide the prioritization of source test cases, which further improve the efficiency. A tool has been developed to automate the proposed approach as much as possible. Empirical studies have also been conducted to evaluate the fault-detection effectiveness of the approach. The results show that this approach enhances both the performance and automation of metamorphic testing. It also highlights interesting research directions for further improving metamorphic testing.
... Much work has been done on automated test case generation, including the development of publicly available tools [3]. The main focus of this work has been on developing efficient methods to generate test inputs to achieve a particular target such as coverage and weak mutation score [5]. However, there has been relatively less attention paid on utilizing effective oracles to improve the fault detection effectiveness of these automatically generated test cases. ...
Automated test generation has helped to reduce the cost of software testing. However, developing effective test oracles for these automatically generated test inputs is a challenging task. Therefore , most automated test generation tools use trivial oracles that reduce the fault detection effectiveness of these automatically generated test cases. In this work, we provide results of an empirical study showing that utilizing metamorphic relations can increase the fault detection effectiveness of automatically generated test cases.
... Much work has been done on automated test case generation, including the development of publicly available tools [3]. The main focus of this work has been on developing efficient methods to generate test inputs to achieve a particular target such as coverage and weak mutation score [5]. However, there has been relatively less attention paid on utilizing effective oracles to improve the fault detection effectiveness of these automatically generated test cases. ...
Automated test generation has helped to reduce the cost of software testing. However, developing effective test oracles for these automatically generated test inputs is a challenging task. Therefore, most automated test generation tools use trivial oracles that reduce the fault detection effectiveness of these automatically generated test cases. In this work, we provide results of an empirical study showing that utilizing metamorphic relations can increase the fault detection effectiveness of automatically generated test cases.
... Then, source test cases are generated using techniques like random testing, structural testing or search based testing and the corresponding follow-up test cases are constructed based on the MRs. In our previous studies we investigated how the fault detection effectiveness of MT varies with various source test case generation techniques such as different structural coverage based approaches and our results show that coverage based source test case generation outperforms randomly generated source test cases [15]. After executing the source and the follow-up test cases on the SUT we can check if there is a change in the output that matches the MR, if not the MR is considered as violated. ...
... In summary, varying the size of the random sample data has no significant effect on the fault detection effectiveness of the considered MRs. But it might be possible to increase the fault detection effectiveness by generating test data based on some test coverage criterion as we discussed in our previous study [15]. ...
