Chapter

Practical Mutation Testing for Smart Contracts

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Solidity smart contracts operate in a hostile environment, which introduces the need for the adequate application of testing techniques to ensure mitigation of the risk of a security incident. Mutation testing is one such technique. It allows for the evaluation of the efficiency of a test suite in detecting faults in a program, allowing developers to both assess and improve the quality of their test suites. In this paper, we propose a mutation testing framework and implement a prototype implementation called Vertigo that targets Solidity contracts for the Ethereum blockchain. We also show that mutation testing can be used to assess the test suites of real-world projects.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... Except the survey in [25] which investigated efforts on mutation testing tools while giving the pros and cons of them. The studied tools are only five : MuSC [23], SuMo [4], Deviant [5], Vertigo [10] and RegularMutator [11]. ...
... The development of a mutation testing framework and its application to the field of smart contracts were both studied by the authors in [10]. They demonstrated how developers may use mutation testing to evaluate the effectiveness of their test suite and make improvements to it in order to make it more efficient. ...
... -Security testing [27,26,7,11,2,1,23]: concerns a cybersecurity technique that organizations use to identify, test and highlight vulnerabilities in their security posture. -Functional testing [4,6,10,5]: concerns a type of testing that seeks to establish whether each application feature works as the software requirements. -Scalability testing [9]: concerns a testing of a software application to measure its capability to scale up or scale out in terms of any of its non-functional capability. ...
Chapter
Full-text available
Once being deployed on the blockchain, smart contracts cannot be altered, requiring more testing. A fault-based testing technique called mutation testing (MT) can significantly increase the utility of a test for smart contracts. MT is a type of white-box testing which is mainly used for unit testing. In fact, certain statements of the source code are changed to check if the test cases are able to find errors in source code. The main objective of MT is ensuring the quality of test cases in terms of robustness in the way that it should fail the mutated source code. In this paper, our goal is to identify and classify the main applications of mutation testing of smart contracts by providing a quick review on the application perspective of mutation testing based on a collection of several papers. In particular, we analysed in which quality assurance processes mutation testing of smart contracts is used, which mutation tools and which mutation operators are employed.
... The last vulnerability we found is the Short Address Attack, 65 and thus can be a good candidate to become a new element of the SWC Registry. Some approaches (e.g., the one proposed by Honig et al. 66 ) base their approach on such a registry. Therefore, the fact that such vulnerabilities are missing from the SWC registry is a relevant issue. ...
... 70 ✓ ✓ Bhargavan et al. 89 ✓ ✓ ✓ ✓ Chang et al. 90 ✓ ✓ Chapman et al. 50 ✓ ✓ Chatterjee et al. 83 ✓ ✓ ✓ Chen et al. 47 ✓ ✓ Dong et al. 91 ✓ ✓ El-Dosuky et al. 92 ✓ ✓ Feist et al. 93 ✓ 42 ✓ ✓ Grech et al. 44 ✓ ✓ ✓ ✓ ✓ ✓ Grossman et al. 77 ✓ ✓ He et al. 54 ✓ ✓ ✓ ✓ ✓ ✓ ✓ Hirai et al. 56 ✓ ✓ ✓ ✓ Honig et al. 66 ✓ ✓ Jiang et al. 95 ✓ 85 ✓ Kolluri et al. 59 ✓ ✓ Krupp et al. 69 ✓ ✓ ✓ Lahbib et al. 96 ✓ ✓ Lai et al. 43 ✓ ✓ Li et al. 51 ✓ ✓ Liao et al. 64 ✓ 45 ✓ ✓ ✓ Liu et al. 39 ✓ ✓ Luu et al. 72 ✓ 41 ✓ ✓ Nguyen et al. 48 ✓ ✓ ✓ ✓ ✓ ✓ ✓ Nikolic et al. 68 ✓ ✓ ✓ ✓ Park et al. 61 ✓ ✓ ✓ ✓ Peng et al. 98 ✓ ✓ ✓ Sergey et al. 55 ✓ ✓ ✓ Shishkin et al. 99 ✓ ✓ Song et al. 100 ✓ ✓ ✓ ✓ ✓ Tian et al. 101 ✓ ✓ ✓ Tikhomirov et al. 73 ✓ 74 ✓ ✓ ✓ Tsankov et al. 40 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Wang et al. 102 ✓ ✓ ✓ ✓ ✓ Wang et al. 76 ✓ ✓ Wang et al. 52 ✓ ✓ Ye et al. 103 ✓ ✓ ✓ ✓ ✓ Zhang et al. 104 ✓ ✓ ✓ ✓ ✓ Zhang et al. 105 ✓ ✓ ✓ Zhang et al. 81 ✓ ✓ ✓ ✓ ...
... In terms of Public Tool or Framework, we found that only 25 (i.e., 37.3%) research papers publicly share a prototype tool implementing the approach presented in the article. 21 out of 25 studies provide their tool through a GitHub repository 58,66,59 . 4 studies release their tool using an external provider (e.g., self-hosted webpage), i.e., Albert et al. 82 , Klomp et al. 85 , Luu et al. 72 and Securify 40 . ...
Article
Full-text available
Blockchain is a platform of distributed elaboration, which allows users to provide software for a huge range of next‐generation decentralized applications without involving reliable third parties. Smart contracts (SCs) are an important component in blockchain applications: they are programmatic agreements among two or more parties that cannot be rescinded. Furthermore, SCs have an important characteristic: they allow users to implement reliable transactions without involving third parties. However, the advantages of SCs have a price. Like any program, SCs can contain bugs, some of which may also constitute security threats. Writing correct and secure SCs can be extremely difficult because, once deployed, they cannot be modified. Although SCs have been recently introduced, a large number of approaches have been proposed to find bugs and vulnerabilities in SCs. In this article, we present a systematic literature review on the approaches for the automated detection of bugs and vulnerabilities in SCs. We survey 68 papers published between 2015 and 2020, and we annotate each paper according to our classification framework to provide quantitative results and find possible areas not explored yet. Finally, we identify the open problems in this research field to provide possible directions to future researchers.
... Despite having high branch or statement coverage, the authors found that even well-designed test suites may result in low mutation scores. Also, some other similar works are trying to propose 9 mutation operators such as [30,31] or even scalability of mutation testing in smart contracts [32]. Andesta et al. [33] conducted an extensive study on known bugs in smart contracts and proposed 49 Solidity-specific mutation operators across 10 classes based on these faults. ...
... Despite having high branch or statement coverage, the authors found that even well-designed test suites may result in low mutation scores. Also, some other similar works are trying to propose mutation operators such as [30,31] or even scalability of mutation testing in smart contracts [32]. ...
Preprint
Full-text available
Smart contracts are autonomous and immutable pieces of code that are deployed on blockchain networks and run by miners. They were first introduced by Ethereum in 2014 and have since been used for various applications such as security tokens, voting, gambling, non-fungible tokens, self-sovereign identities, stock taking, decentralized finances, decentralized exchanges, and atomic swaps. Since smart contracts are immutable, their bugs cannot be fixed, which may lead to significant monetary losses. While many researchers have focused on testing smart contracts, our recent work has highlighted a gap between test adequacy and test data generation, despite numerous efforts in both fields. Our framework, Griffin, tackles this deficiency by employing a targeted symbolic execution technique for generating test data. This tool can be used in diverse applications, such as killing the survived mutants in mutation testing, validating static analysis alarms, creating counter-examples for safety conditions, and reaching manually selected lines of code. This paper discusses how smart contracts differ from legacy software in targeted symbolic execution and how these differences can affect the tool structure, leading us to propose an enhanced version of the control-flow graph for Solidity smart contracts called CFG+. We also discuss how Griffin can utilize custom heuristics to explore the program space and find the test data that reaches a target line while considering a safety condition in a reasonable execution time. We conducted experiments involving an extensive set of smart contracts, target lines, and safety conditions based on real-world faults and test suites from related tools. The results of our evaluation demonstrate that Griffin can effectively identify the required test data within a reasonable timeframe.
... Mutation testing has attracted many researchers as a strong technique for evaluating smart contracts. There are several papers on proposing novel mutation operators for Solidity smart contracts, including MuSC [3], Vertigo [4], Deviant [5], SuMo [3], and [6]. The focus of these works is on selecting effective test cases from a test suite based on mutation testing. ...
... The author concludes even test suites with a high branch or statement coverage may achieve low mutation scores. Honig et al. designed a mutation testing infrastructure and a prototype project called Vertigo [4] with two Solidity-specific mutation operators and four enhanced mutation operators from the PIT [15] mutation testing framework for Java and JVM. MuSC [12] also proposes 15 mutation Solidity-specific operators and evaluated their work with four real-world Solidity Smart contracts. ...
Conference Paper
Ethereum Smart contracts are pieces of code that are run on this blockchain. The correctness of smart contracts is important as they are immutable, their source can be seen by everyone, and they transfer Ether. In this paper, we propose a framework for the automated generation of a set of effective test cases for a given smart contract. We use symbolic execution for generation and mutation testing for selection of test cases. We have evaluated our tool on a set of smart contracts, and our results show how mutation can reduce the size of test suites generated by symbolic execution. Also, by analyzing the survived mutants, we have interesting results about effective test cases that cannot be generated by the symbolic execution engine for smart contracts.
... In the literature, the generation of test inputs for smart contracts mainly relies on fuzzing and mutation [27], [28], [25], [29], [30], [31], [32], [33]. For example, Jiang et al. [34] proposed to build seed inputs with the valid input domain and the inputs frequently used by some data types in smart contracts, to further fuzz inputs for testing ABI of smart contracts. ...
... Ma et al. [36] combined the feedback-directed mutation with the parameter data types of functions to fuzz test inputs for smart contracts. Honig et al. [28] proposed a mutation testing framework for the Solidity contracts deployed on the Ethereum blockchain. Harvey [27] and sFuzz [32] are proposed by leveraging the fuzzing approach to discover the vulnerabilities for smart contracts as well. ...
... We introduce smart contract specific mutations that inject common mistakes that are made by smart contract developers. An example is a forgotten or wrong function modifier [12]. A modifier can express conditions that have to be fulfilled for the execution of a function, e.g., that the caller of the function is the owner of the contract. ...
... The difference to our approach is that their mutations do not consider access control faults and they are not concerned with inspecting the gas usage. Honig et al. [12] do take access control into account, but not the gas limit. The evaluation of both papers is also limited to 4 respectively 2 contracts. ...
Preprint
The blockchain is a key technology that has been adopted in many application areas to increase security and reliability and to avoid the need for a central trusted authority. One of its essential underlying foundations are smart contracts, which are executable programs for managing data or assets on the blockchain. It is crucial that smart contracts are tested thoroughly due to their immutable nature and since even small bugs can lead to huge monetary losses. However, it is not enough to just test smart contracts, it is also important to ensure the quality and completeness of the tests. Hence, we introduce new smart contract specific mutation operators as well as a novel killing condition that is able to detect a deviation in the gas consumptions, i.e., in the monetary value that is required to perform transactions. Moreover, we establish a baseline for mutation testing of smart contracts by applying our method to a replay test suite and by testing about a thousand contracts.
... Much research has been done to combat this problem, but it mainly focused on development-time fault-removal techniques, such as mutation testing [2], static analysis [3], and vulnerability detection via Machine Learning (ML) [4], to name a few. However, fault-tolerant computing also traditionally knows and employs a set of runtime patterns. ...
... Honig et al. [11] propose a prototype implementation, called Vertigo, that targets Smart Contracts written in Solidity. Vertigo also relies on the functionalities exposed by the Truffle interface to enable testing; however, it discourages the use the of Ganache simulator, as some Truffle tests might not use its clean environment feature, leading to unreliable test results. ...
Chapter
Mutation testing is a powerful test adequacy assessment technique that can guarantee the deployment of more reliable Smart Contract code. Developers add new features, fix bugs, and refactor modern distributed applications at a quick pace, thus they must perform continuous re-testing to ensure that the project evolution does not break existing functionalities. However, regularly re-running the entire test suite can be time intensive, especially when mutation testing is involved. This paper presents ReSuMo, the first regression mutation testing approach and tool for Solidity Smart Contracts. ReSuMo uses a static, file-level technique to select a subset of Smart Contracts to mutate and a subset of test files to re-run during a regression mutation testing campaign. ReSuMo incrementally updates the mutation testing results considering the outcomes of the old program version; in this way, it can speed up mutation testing on evolving projects without compromising the mutation score.KeywordsMutation testingRegression testingSmart contract
... There has been a lot of work using Mutation Testing for verification [46], [47], [48]. This technique involves mutation of the contract code to either analyze using existing test cases for faults or to generate vulnerability-free contracts. ...
Article
Full-text available
Blockchain-based smart contracts enable the creation of decentralized applications, which often handle assets of considerable value. While the underlying platforms guarantee the correctness of smart-contract execution, they cannot ensure that the code of a contract is correct. Today, as evidenced by a number of recent security breaches, developers still have a hard time making contracts that work properly.Even though these incidents often exploit contract interaction, prior work on smart-contract verification, vulnerability discovery, and secure development typically considers only individual contracts in isolation. To address this gap, we introduce the VeriSolid framework for the formal verification of contracts that are specified using a abstract state machine based model with rigorous operational semantics. Our model-based approach allows developers to reason about and verify the behavior of a set of interacting contracts at a high level of abstraction. VeriSolid allows the generation of Solidity code that is functionally and behaviorally equivalent to verified models, which enables the creation of correct-by-design smart contracts. We additionally introduce a graphical notation (called deployment diagrams ) for specifying possible interactions between contract types. Based on this notation, we present a framework for the automated verification, generation, and deployment of contracts that conform to a deployment diagram. To demonstrate the applicability of VeriSolid , we translate existing Ethereum Improvement Proposal (EIP) specifications to temporal properties for two of the most popular contract interfaces: ERC20 and ERC721. We also show you how to write code for the ERC20 and ERC721 interfaces in a way that is safe, and we do this by using VeriSolid . We evaluate our framework on 726 contracts that are currently deployed on the Ethereum blockchain, which include 267 ERC20 and 459 ERC721 contracts. Our experiments indicate that 18% of ERC20 contracts and 4% of ERC721 contracts fail to satisfy the EIP specifications.
... The classification is in line with the official Solidity documentation, though specific aspects were later refined or removed depending on our findings and the implemented mutation operators. As said this leads to the identification of 7 new Solidity mutation operators, while the remaining operators find some correspondence in existing works (Andesta et al., 2020;Chapman et al., 2019;Honig et al., 2019;Hartel and Schumi, 2020;Wu et al., 2019) even though many of them have been reconsidered, and slightly modified. SuMo also includes those operators that have been demonstrated to be effective, while mutation operators that generate a significant number of invalid mutants were excluded from the set. ...
Article
Blockchain technologies have had a rather disruptive impact on many sectors of the contemporary society. The establishment of virtual currencies is probably the most representative case. Nonetheless, the inherent support to trustworthy electronic interactions has widened the possible adoption contexts. In the last years, the introduction of Smart Contracts has further increased the potential impact of such technologies. These self-enforcing programs have interesting peculiarities (e.g., code immutability) that require innovative testing strategies. This paper presents a mutation testing approach for assessing the quality of test suites accompanying Smart Contracts written in Solidity, the language used by the Ethereum Blockchain. Specifically, we propose a novel suite of mutation operators capable of simulating a wide variety of traditional programming errors and Solidity-specific faults. The operators come in two flavors: Optimized, for faster mutation testing campaigns, and Non-Optimized, for performing a more thorough adequacy assessment. We implemented our approach in a proof-of-concept work, SuMo (SOlidity MUtator), and we evaluated its effectiveness on a set of real-world Solidity projects. The experiments highlighted a recurrent low mutation score for the test suites shipped with the selected applications. Moreover, analyzing the surviving mutants of a selected project helped us to identify faulty test cases and Smart Contract code. These results suggest that SuMo can concretely improve the fault-detection capabilities of a test suite, and help to deliver more reliable Solidity code.
... Honig J.J, Everts M.H, Huis man M [27] proposes a mutation testing framework called vertigo. As discussed in above papers related to mutation testing here also the author considers mutation testing rather than code coverage metric as relating to general strategies for the nature of the test suite, it estimates the suite's viability at distinguishing deficiencies in the source code straightforwardly. ...
... Also, recently Honig et. al. [48] provide a prototype for a mutation testing tool named Vertigo that uses Truffle CLI to evaluate the mutants using specified test cases. But in this work, authors do not propose any impressive Solidity-specific mutation operators. ...
Conference Paper
Smart contracts are immutable, verifiable, and autonomous pieces of code that can be deployed and ran on blockchain networks like Ethereum. Due to the immutability nature of blockchain, no change is possible on a deployed smart contract or a verified transaction. On the other hand, there are millions of dollars carried by smart contracts in Ethereum blockchain, and hence, a faulty smart contract can lead to a huge monetary loss. Therefore, it is important for smart contract developers to fully test and check the correctness of their code before deploying it on the blockchain. In this paper, we propose a testing mechanism for smart contracts in the Solidity language, based on mutation testing. We analyzed a comprehensive list of known bugs in Solidity smart contracts and designed 10 classes of mutation operators inspired by the real faults. Our experimental results show that our proposed mutation operators can regenerate 10 out of 15 famous faulty smart contracts, which have resulted in millions of dollars loss. We also have extended the Universal Mutator tool with our mutation operators to generate mutants for smart contracts written in Solidity, automatically.
Article
Full-text available
Smart contracts (SC) are programs embodying certain business logic stored on a blockchain network like Ethereum. The execution of transactions on SC has a cost, measured in gas units, that depends on the low‐level operations performed. Therefore, a poor choice of high‐level language constructs could lead to overcharging users for their transactions. Thus, a testing process focused on possible deviations of the gas used in diverse scenarios could provide substantial global savings. This paper presents a gas‐centered mutation testing approach for taking care of the gas consumed by Solidity SCs. This approach can be useful to improve the test quality to detect gas‐related problems, reason about performance issues that only manifest in certain situations, and identify alternative more optimal implementations. We define and implement several mutation operators specifically designed to perturb gas consumption while preserving contract semantics in general. Our experiments using several real‐world SCs show the feasibility of the technique, with some mutants reproducing meaningful differences in the consumption and exposing some gas limits not tight enough in historic transactions. Therefore, our approach is shown to be a good ally to prevent the appearance of gas‐related issues and lays the groundwork for researchers seeking to improve performance testing practices.
Chapter
Test design and evaluation are both successful with the testing technique known as mutation testing. Although mutation testing has been developed over many years to be suitable and effective with various software systems and programming languages, it has not yet been widely used in industry. The high computational cost is one of the main barriers preventing developers and testers from adopting mutation testing. Particularly, a major challenge and labor-intensive factor in mutation testing is the requirement to manually identify effective mutants. Mutation testing can be applied to smart contracts a feature in the blockchain platform having various applications in many fields like health care, supply chain, education, banking, government services, etc. The nature of smart contract requires exhaustive testing like mutation testing for defect free smart contract to be deployed in blockchain. The test suite effectiveness is identified using the mutation analysis. The smart contract code when mutated produces numerous mutants which includes effective and useless mutants. Executing all the mutants makes mutation testing expensive. To reduce the number of mutants to be executed, various sampling methods are used in previous study. In order to overcome this constraint, this research suggests a machine learning-based method for classifying effective mutants for reducing the number of mutants. Various classifiers in machine learning are compared for prediction of effective mutants in smart contract code. The effective mutants classified can be applied to perform mutation testing for smart contracts.
Article
Full-text available
Mutation testing is a powerful test adequacy assessment technique that can guarantee the deployment of deeply scrutinized and more reliable Smart Contract code. However, regularly evaluating the test suite during the evolution of a project can be unfeasible due to the high costs of the assessment. This paper presents ReSuMo, the first regression mutation testing approach and tool for Solidity Smart Contracts. ReSuMo uses a static and file-level technique to select a subset of Smart Contracts to mutate, and a subset of test files to re-run during a regression mutation testing campaign. After each mutation testing run, ReSuMo incrementally updates the results using the test outcomes registered on the previous program revision. In this way, it can speed up the mutation testing process for evolving projects, while providing a comprehensive adequacy assessment of the whole test suite.
Article
Full-text available
Abstract - Smart contracts are codes for executing transactions over the blockchain. Smart contracts play a major role in executing the transactions, which are immutable in nature, and avoid third-party involvement in transactions. Smart contracts are developed in many languages. One of the most popular languages is solidity. This paper focuses on smart contracts written using solidity. Smart contracts are vulnerable and have faced huge losses like with DAO attacks. Smart contracts require exhaustive testing to avoid such loss. Testing is to be qualified; hence mutation testing is the right choice. Mutation testing for smart contracts focuses on vulnerability detection by inserting faults in the code. Also qualifies the test suite executed against a smart contract. The pros and cons of various tools available for this purpose are discussed in this paper. Finally suggested improving the tools to perform the mutation analysis of smart contracts better. Keywords - Mutation Operators, Smart Contract, Solidity.
Article
Blockchain technologies, also known as Distributed Ledger Technologies (DLT), are increasingly being explored in many applications, especially in the presence of (potential) dis-/mis-/un-trust among organizations and individuals. Today, there exists a plethora of DLT platforms on the market, which makes it challenging for system designers to decide what platform they should adopt and implement. Although a few DLT comparison frameworks have been proposed in the literature, they often fail in covering all performance and functional aspects, adding that they too rarely build upon standardized criteria and recommendations. Given this state of affairs, the present paper considers a recent and exhaustive set of assessment criteria recommended by the ITU (International Telecommunication Union). Those criteria (about fifty) are nonetheless mostly defined in a textual form, which may pose interpretation problems during the implementation process. To avoid this, a systematic literature review regarding each ITU criterion is conducted with a twofold objective: (i) to understand to what extent a given criterion is considered/evaluated by the literature; (ii) to come up with ‘formal’ metric definition (i.e., on a mathematical or experimental ground) based, whenever possible, on the current literature. Following this formalization stage, a decision support tool called CREDO-DLT, which stands for “multiCRiteria-basEd ranking Of Distributed Ledger Technology platforms”, is developed using AHP and TOPSIS, which is publicly made available to help decision-maker to select the most suitable DLT platform alternative (i.e., that best suits their needs and requirements). A use case scenario in the context of energy communities is proposed to show the practicality of CREDO-DLT.
Article
Smart contracts are automated or self-enforcing contracts that can be used to exchange assets without having to place trust in third parties. Many commercial transactions use smart contracts due to their potential benefits in terms of secure peer-to-peer transactions independent of external parties. Experience shows that many commonly used smart contracts are vulnerable to serious malicious attacks, which may enable attackers to steal valuable assets of involving parties. There is, therefore, a need to apply analysis and automated repair techniques to detect and repair bugs in smart contracts before being deployed. In this work, we present the first general-purpose automated smart contract repair approach that is also gas-aware. Our repair method is search-based and searches among mutations of the buggy contract. Our method also considers the gas usage of the candidate patches by leveraging our novel notion of gas dominance relationship . We have made our smart contract repair tool SCRepair available open-source, for investigation by the wider community.
Article
Full-text available
Blockchain has become particularly popular due to its promise to support business-critical services in very different domains (e.g., retail, healthcare). Blockchain systems rely on complex middleware, like Ethereum or Hyperledger Fabric, that allow running smart contracts, which specify business logic in cooperative applications. The presence of software defects in these contracts has notably caused failures, including severe security problems. In this paper, we use software-implemented fault injection (SWIFI) to assess the behavior of permissioned blockchain systems in the presence of faulty smart contracts. We emulate the occurrence of general software faults and also blockchain-specific software faults (e.g., missing require on transaction sender) in smart contracts code and observe the impact on the overall system dependability in terms of reliability and integrity. We also analyze the effectiveness of formal verification and runtime protection mechanisms in detecting the injected faults. Results indicate that formal verification and runtime protections have to complement built-in platform checks to guarantee proper dependability of blockchain systems. The work presented in this paper allows smart contract developers to become aware of possible faults in smart contracts and to understand the impact of their presence. It also provides valuable information for middleware developers to improve the overall fault tolerance of their systems.
Chapter
It is crucial that smart contracts are tested thoroughly due to their immutable nature. Even small bugs in smart contracts can lead to huge monetary losses. However, testing is not enough; it is also important to ensure the quality and completeness of the tests. There are already several approaches that tackle this challenge with mutation testing, but their effectiveness is questionable since they only considered small contract samples. Hence, we evaluate the quality of smart contract mutation testing at scale. We choose the most promising of the existing (smart contract specific) mutation operators, analyse their effectiveness in terms of killability and highlight severe vulnerabilities that can be injected with the mutations. Moreover, we improve the existing mutation methods by introducing a novel killing condition that is able to detect a deviation in the gas consumption, i.e., in the monetary value that is required to perform transactions.
Article
Full-text available
Mutation Testing is a fault-based software testing technique that has been widely studied for over three decades. The literature on Mutation Testing has contributed a set of approaches, tools, developments and empirical results which have not been surveyed in detail until now. This paper provides a comprehensive analysis and survey of Mutation Testing. The paper also presents the results of several development trend analyses. These analyses provide evidence that Mutation Testing techniques and tools are reaching a state of maturity and applicability, while the topic of Mutation Testing itself is the subject of increasing interest.
Chapter
Ensuring correctness of smart contracts is paramount to ensuring trust in blockchain-based systems. This paper studies the safety and security of smart contracts in the Azure Blockchain Workbench, an enterprise Blockchain-as-a-Service offering from Microsoft. In particular, we formalize semantic conformance of smart contracts against a state machine workflow with access-control policy and propose an approach to reducing semantic conformance checking to safety verification using program instrumentation. We develop a new Solidity program verifier VeriSol that is based on translation to Boogie, and have applied it to analyze all application contracts shipped with the Azure Blockchain Workbench and found previously unknown bugs in these published smart contracts. After fixing these bugs, VeriSol was able to successfully perform full verification for all of these contracts.
Conference Paper
Permissionless blockchains allow the execution of arbitrary programs (called smart contracts), enabling mutually untrusted entities to interact without relying on trusted third parties. Despite their potential, repeated security concerns have shaken the trust in handling billions of USD by smart contracts. To address this problem, we present Securify, a security analyzer for Ethereum smart contracts that is scalable, fully automated, and able to prove contract behaviors as safe/unsafe with respect to a given property. Securify's analysis consists of two steps. First, it symbolically analyzes the contract's dependency graph to extract precise semantic information from the code. Then, it checks compliance and violation patterns that capture sufficient conditions for proving if a property holds or not. To enable extensibility, all patterns are specified in a designated domain-specific language. Securify is publicly released, it has analyzed >18K contracts submitted by its users, and is regularly used to conduct security audits by experts. We present an extensive evaluation of Securify over real-world Ethereum smart contracts and demonstrate that it can effectively prove the correctness of smart contracts and discover critical violations.
Conference Paper
Mutation testing is widely used in research (even if not in practice). Mutation testing tools usually target only one programming language and rely on parsing a program to generate mutants, or operate not at the source level but on compiled bytecode. Unfortunately, developing a robust mutation testing tool for a new language in this paradigm is a difficult and time-consuming undertaking. Moreover, bytecode/intermediate language mutants are difficult for programmers to read and understand. This paper presents a simple tool, called universalmutator, based on regular-expression-defined transformations of source code. The primary drawback of such an approach is that our tool can generate invalid mutants that do not compile, and sometimes fails to generate mutants that a parser-based tool would have produced. Additionally, it is incompatible with some approaches to improving the efficiency of mutation testing. However, the regexp-based approach provides multiple compensating advantages. First, our tool is easy to adapt to new languages; e.g., we present here the first mutation tool for Apple's Swift programming language. Second, the method makes handling multi-language programs and systems simple, because the same tool can support every language. Finally, our approach makes it easy for users to add custom, project-specific mutations.
Article
Smart contracts---stateful executable objects hosted on blockchains like Ethereum---carry billions of dollars worth of coins and cannot be updated once deployed. We present a new systematic characterization of a class of trace vulnerabilities, which result from analyzing multiple invocations of a contract over its lifetime. We focus attention on three example properties of such trace vulnerabilities: finding contracts that either lock funds indefinitely, leak them carelessly to arbitrary users, or can be killed by anyone. We implemented MAIAN, the first tool for precisely specifying and reasoning about trace properties, which employs inter-procedural symbolic analysis and concrete validator for exhibiting real exploits. Our analysis of nearly one million contracts flags 34,200 (2,365 distinct) contracts vulnerable, in 10 seconds per contract. On a subset of3,759 contracts which we sampled for concrete validation and manual analysis, we reproduce real exploits at a true positive rate of 89%, yielding exploits for3,686 contracts. Our tool finds exploits for the infamous Parity bug that indirectly locked 200 million dollars worth in Ether, which previous analyses failed to capture.
Conference Paper
Cryptocurrencies record transactions in a decentralized data structure called a blockchain. Two of the most popular cryptocurrencies, Bitcoin and Ethereum, support the feature to encode rules or scripts for processing transactions. This feature has evolved to give practical shape to the ideas of smart contracts, or full-fledged programs that are run on blockchains. Recently, Ethereum's smart contract system has seen steady adoption, supporting tens of thousands of contracts, holding millions dollars worth of virtual coins. In this paper, we investigate the security of running smart contracts based on Ethereum in an open distributed network like those of cryptocurrencies. We introduce several new security problems in which an adversary can manipulate smart contract execution to gain profit. These bugs suggest subtle gaps in the understanding of the distributed semantics of the underlying platform. As a refinement, we propose ways to enhance the operational semantics of Ethereum to make contracts less vulnerable. For developers writing contracts for the existing Ethereum system, we build a symbolic execution tool called Oyente to find potential security bugs. Among 19, 336 existing Ethereum contracts, Oyente flags 8, 833 of them as vulnerable, including the TheDAO bug which led to a 60 million US dollar loss in June 2016. We also discuss the severity of other attacks for several case studies which have source code available and confirm the attacks (which target only our accounts) in the main Ethereum network. This article is summarized in: the morning paper an interesting/influential/important paper from the world of CS every weekday morning, as selected by Adrian Colyer
Conference Paper
The paper reports on a first experimental comparison of software errors generated by real faults and by 1st-order mutations. The experiments were conducted on a program developed by a student from the industrial specification of a critical software from the civil nuclear field. Emphasis was put on the analysis of errors produced upon activation of 12 real faults by focusing on the mechanisms of error creation, masking, and propagation up to failure occurrence, and on the comparison of these errors with those created by 24 mutations. The results involve a total of 3730 errors recorded from program execution traces: 1458 errors were produced by the real faults, and the 2272 others by the mutations. They are in favor of a suitable consistency between errors generated by mutations and by real faults: 85% of the 2272 errors due to the mutations were also produced by the real faults. Moreover, it was observed that although the studied mutations were simple faults, they can create erroneous behaviors as complex as those identified for the real faults. This lends support to the representativeness of errors due to mutations.
Conference Paper
Mutation testing is a powerful, but computationally expensive, technique for unit testing software. This expense has prevented mutation form becoming widely used in practical situations, but recent engineering advances have given us techniques and algorithms for significantly reducing the cost of mutation testing. These technique include a new algorithmic execution technique include a new algorithmic execution technique called schema-based mutation, a reduction technique called selective mutation, heuristics for detecting equivalent mutants, and algorithms for automatic test data generation. This paper reviews experimentation with these advances and outlines a design for a system that will approximate mutation, but in a way that will be accessible to every day programmers. We envision a system to which a programmer can submit a program unit and get back a set of input/output pairs that are guaranteed to form an effective test of the unit by being close to mutation adequate. We believe this system could be efficient enough to be adopted by leading-edge software developers. Full automation in unit testing has the potential to dramatically change the economic balance between testing and development, by reducing the cost of testing from the major part of the total development cost to a small fraction.
Article
Both theoretical and empirical arguments suggest that specifications and implementations are equally important sources of information for generating test cases. Nevertheless, the majority of test generation procedures described in the literature deal only with the program source, ignoring specifications. In this paper we outline a procedure for measuring test case effectiveness using specifications given in predicate calculus form. This method is similar to the mutation analysis method of testing programs.
Article
: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : xiii 1. INTRODUCTION : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 1 1.1 Problem and Motivation : : : : : : : : : : : : : : : : : : : : : : : : 2 1.2 Scope of This Research : : : : : : : : : : : : : : : : : : : : : : : : : 2 1.3 Organization of This Dissertation : : : : : : : : : : : : : : : : : : : 5 2. AN OVERVIEW OF MUTATION AND DATA FLOW TESTING : : : : 9 2.1 Mutation Testing : : : : : : : : : : : : : : : : : : : : : : : : : : : : 9 2.2 Alternate Mutation Testing : : : : : : : : : : : : : : : : : : : : : : 10 2.2.1 Randomly Selected x% Mutation Testing : : : : : : : : : : : 10 2.2.2 Constrained Mutation Testing : : : : : : : : : : : : : : : : : 11 2.2.3 Other Forms of Mutation Testing : : : : : : : : : : : : : : : 12 2.3 Data Flow Testing : : : : : : : : : : : : : : : : : : : : : : : : : : : 14 2.4 Testing Tools : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 16 3. A FORMALEVALUATIONOF M...
Code Issue Leads to $60 Million Ether Theft - CoinDesk
  • Dao The
  • Attacked
Mutation clustering. Master’s thesis, King’s College London
  • S Hussain
Formal specification and verification of smart contracts for Azure blockchain
  • Y Wang
Vandal: a scalable security analysis framework for smart contracts
  • L Brent