Conference PaperPDF Available

Uncovering Smart Contract VM Bugs Via Differential Fuzzing

November 2021

DOI:10.1145/3503921.3503923

Conference: ROOTS'21: Reversing and Offensive-oriented Trends Symposium

Authors:

Dominik Maier

Technische Universität Berlin

Jean-Pierre Seifert

Technische Universität Berlin

Content uploaded by Dominik Maier

Content may be subject to copyright.

Uncovering Smart Contract VM Bugs

Via Dierential Fuzzing

Dominik Maier

dmaier@sect.tu-berlin.de

Technische Universität Berlin

Berlin, Germany

Fabian Fäßler

fabi@fabif.de

Technische Universität Berlin

Berlin, Germany

Jean-Pierre Seifert

jean-pierre.seifert@tu-berlin.de

Technische Universität Berlin

Berlin, Germany

ABSTRACT

The ongoing public interest in blockchains and smart contracts has

brought a rise to a magnitude of dierent blockchain implementa-

tions. The rate at which new concepts are envisioned and imple-

mented makes it hard to vet their impact on security. Especially

smart contract platforms, executing untrusted code, are very com-

plex by design. Still, people put their trust and money into chains

that may lack proper testing. A behavior deviation for edge cases

of single op-codes is a critical bug class in this brave new world. It

can be abused for Denial of Service against the blockchain, chain

splits, double-spending, or direct attacks on applications operating

on the blockchain. In this paper, we propose an automated method-

ology to uncover such dierences. Through coverage-guided and

state-guided fuzzing, we explore smart contract virtual machine

behavior against multiple VMs in parallel. We develop NeoDi, the

rst framework for feedback-guided dierential fuzzing of smart

contract VMs. We discuss real, monetary consequences our tool pre-

vents. NeoDi can be ported to new smart contract platforms with

ease. Apart from fuzzing Ethereum VMs, NeoDi found a range

of critical dierentials in VMs for the Neo blockchain. Moreover,

through a higher-layer semantics mutator, we uncovered semantic

discrepancies between Neo smart contracts written in Python when

executed on the blockchain vs. classic CPython. Along the way,

NeoDi uncovered memory corruptions in the C# Neo VM.

CCS CONCEPTS

•Security and privacy →

Domain-specic security and privacy

architectures;Software and application security.

KEYWORDS

Dierential Fuzzing, State-Aware, Smart Contract VM

ACM Reference Format:

Dominik Maier, Fabian Fäßler, and Jean-Pierre Seifert. 2021. Uncovering

Smart Contract VM Bugs Via Dierential Fuzzing. In Reversing and Oensive-

oriented Trends Symposium (ROOTS’21), November 18–19, 2021, Vienna, Aus-

tria. ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/3503921.

3503923

Permission to make digital or hard copies of all or part of this work for personal or

classroom use is granted without fee provided that copies are not made or distributed

for prot or commercial advantage and that copies bear this notice and the full citation

on the rst page. Copyrights for components of this work owned by others than the

author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or

republish, to post on servers or to redistribute to lists, requires prior specic permission

and/or a fee. Request permissions from permissions@acm.org.

ROOTS’21, November 18–19, 2021, Vienna, Austria

ACM ISBN 978-1-4503-9602-8/21/11. . . $15.00

https://doi.org/10.1145/3503921.3503923

1 INTRODUCTION

Blockchains and cryptocurrencies are often associated with money

and investments. This paper focuses on a very dierent aspect to

them, the underlying

smart contract virtual machines

and their

security. Systems like Ethereum [

] and Neo [

] extend the con-

cept of a traditional blockchain with such a built-in virtual machine:

Turing-complete, yet limited by articial mechanics, to guarantee

the program’s termination. Every execution of the contracts is pub-

licly recorded, and every node can run each smart contract to verify

the transactions. For computer security research, they introduce

a new and foreign execution environment when compared to tra-

ditional CPUs and operating systems. Like in an ordinary OS, the

virtual machine executing the smart contract denes an interface

to interact with various components. Without les, sockets, and

threading, but with direct access to the blockchain, smart contract

VMs create weird computing environments to explore and research.

From these new platforms, new security issues and bug classes

arise.

For common chains, all participating smart contract VMs need

to behave consistently. As we show in this paper, even the slight-

est dierence in op-code behavior can have drastic consequences.

They reach from stolen money, over chain splits, to the complete

collapse of the ecosystem. To mitigate security bugs before they

reach production, we create NeoDi, a purpose-built tool to un-

cover dierences automatically. It leverages dierential fuzzing,

generating valid input and feeding it back to the target to nd dif-

ferences in an automated fashion. In over 2

5k LOC, NeoDi packs

a fully-featured tool for security research. On top of coverage, a

common feedback mechanism for fuzzing, NeoDi can take state

progression into account, backed by the virtual machine. While

ding the two largest virtual machines for Ethereum only uncovers

false positives, likely due to rigorous manual testing, we uncover

a wide variety of bugs for the other ecosystem we test: we take a

deep dive into virtual machines for Neo, a less researched chain,

albeit still spanning a 24-hour trading volume of 1

5bn dollars in

early 2021. Here, NeoDi nds critical bugs, including dierences

between alternative VMs, as well as memory corruptions in the

pre-release code of Neo’s main chain. Since the Neo ecosystem of-

fers the option to write smart contracts in traditional programming

languages, such as Python, they open the door for further fuzzable

dierences: python’s language semantics.

In contrast to traditional fuzzing, the condition a dierential

fuzzer strives to trigger is not corrupted memory or a crash [

but a dierence in output or state. NeoDi can compare targets of

dierent ecosystems and programming languages, such as NeoPy-

thon and the Neo VM in C#, and support feedback mechanisms that

go beyond code coverage. NeoDi found a wide range of dierences

ROOTS’21, November 18–19, 2021, Vienna, Austria Maier et al.

between the Neo VM implemented in C#—used by the main consen-

sus nodes—and the neo-python VM implemented in Python—used

by many client programs that want to interact with the blockchain,

such as wallets or web applications [

]. Moreover, NeoDi uncov-

ered semantic dierences between CPython and neo-boa Python

smart contracts for Neo, as well dierences on opcode-level and

memory corruptions in its VM, written in C#.

Our contributions are as follows:

•

We develop, evaluate, and open-source NeoDi, a purpose-

built platform for dierential fuzzing of smart contract VMs.

NeoDi is a generalized framework for dierential, feedback-

driven fuzzing, applicable to a broad range of VMs.

•

We implement NeoDi back ends to dierential fuzz the

openethereum against the geth Ethereum VM, as well as the

Neo VM against neo-python.

•

As an additional target, we di the CPython language se-

mantics against smart contracts written in Python, nding

semantic gaps in smart contract programming languages

and the associated security ramications.

•

We discuss how discrepancies in smart contract VMs can be

exploited beyond the blockchain network itself. For example,

we describe how to attack applications built on top of the

blockchain.

•

NeoDi helped nd and x critical bugs in the Neo smart

contract ecosystem, including dierences and: memory cor-

ruptions.

2 BACKGROUND

We are convinced building tooling applicable spanning multiple

smart contract platforms will add valuable insight for the research

community. In this section, we take a closer look at the Neo smart

contract platform. By the time of writing this paper, the Neo blockchain

had a reported market cap of about 3

7billion dollars and a 24h

trading volume of over 1.5billion dollars.

2.1 The Neo Smart Contract Ecosystem

Many people trust the Neo ecosystem to handle monetary assets.

Neo started to get research attention, for example, through work by

Qin et al., analyzing Neo’s dBFT scheme [

]. Neo is a proof-of-stake

blockchain using a small number of consensus nodes by design,

voted on by its stakeholders. As opposed to Ethereum, which uses

purpose-built languages like Solidity for smart contracts, Neo oers

front ends for a variety of traditional languages. The Neo home-

page lists support for 8 major programming languages, including

Python,Go,JavaScript,Java and C#, with an additional planned

support of Cand C++. The Neo compilers are typically not full

compilers but use the languages’ ocial compilers to transform

code into an intermediate representation, which is then translated

to AVM code, the bytecode for the Neo VM. The main consensus

nodes run the ocial VM, written in C#, and make up the core Neo

blockchain network. To interact with the blockchain—for example,

to send transactions or to invoke smart contracts—users need to use

client applications. The Neo ecosystem oers SDKs for developers

to build such applications on top of the Neo blockchain and to

interact with smart contracts. The SDKs have to understand all the

transactions in the Neo blockchain and thus include their own Neo

VM implementation to execute the smart contracts locally.

Neo Assets

The native assets of the Neo blockchain are NEO,

the main currency, and Neo GAS (GAS), primarily used to pay for

the deployment and execution of smart contracts. NeoGAS is being

generated and distributed over time to Neo wallets, proportionally

to the amount of Neo owned. Along with the typical verication

of asset transactions, each node also includes the Neo VM, able to

execute smart contracts in the form of bytecode. A certain GAS cost

is associated with each opcode and syscall. The consensus algorithm

used by the consensus nodes is called delegated Byzantine Fault

Tolerance and is based on Practical Byzantine Fault Tolerance [

The following sections will briey introduce the Neo smart contract

ecosystem.

The Neo Virtual Machine

The Neo VM is responsible for ex-

ecuting smart contracts uploaded to the Neo blockchain. Every

participant in the blockchain network, including wallets and ex-

changes, needs their own Neo VM to execute contracts. In contrast

to traditional process VMs, Neo VMs don’t provide access to hard-

ware or the lesystem but interact with data on the blockchain.

They abide by the Neo bytecode format and provide APIs to access,

for example, permanent storage and blockchain transaction data.

The ocial Neo VM used by the consensus nodes is implemented in

C# [

]. Client programs that want to interact with the blockchain,

such as wallets or web applications, often make use of alternative

VMs, matching their respective programming languages. For ex-

ample in go and Python.Evaluating smart contracts locally allows

clients to read data from a contract’s storage directly and to react to

events without having to query slow APIs or trusting a centralized

node.

The C# VM

The architecture of the main VM is depicted in Fig. 1.

Figure 1: Neo VM architecture: The main component of the

Neo VM is the ExecutionEngine. It executes each smart con-

tract in independent ExecutionContexts, ensuring strict sep-

aration between smart contracts. Each contract can work

on two seperate stacks, the AltStack and ExecutionStack.

Through the Interop Service, contracts can access external

data and state.

The ExecutionEngine is the core of the Neo VM and connects the

Uncovering Smart Contract VM Bugs

Via Dierential Fuzzing ROOTS’21, November 18–19, 2021, Vienna, Austria

various components. It also contains the ExecuteInstruction() func-

tion that will interpret and execute any opcode. When a contract is

executed, an ExecutionContext is created and stored in the Invoca-

tionStack of the ExecutionEngine. This ExecutionContext contains

the Script, the bytecode of the smart contract, as well as two stacks.

The two stacks are the EvaluationStack, for storing results, and

AltStack for temporary data. For each call to other contracts, a

new ExecutionContext is created and placed on the Invocation-

Stack. This means the contracts are not sharing their Evaluation-

or AltStack with each other, which is a benecial security property.

Persistent Storage

Most data, such as blockchain transactions

and the contract’s permanent key-value storage, is accessed through

the SYSCALL opcode. The permanent key-value store, or rather the

values themselves, are not directly stored in the blockchain. The

blockchain only records transactions, which include transactions

that call smart contracts. Executed smart contracts can read, write

and delete data in this storage. In order to get the current state of

the storage, all transactions have to be replayed, and smart contract

invocations need to be executed. The storage might store authenti-

cation information or balances of tokens. Giving another contract

access to this storage context object authorizes this other contract

to access the same storage. This creates an attractive attack surface

for Neo smart contracts.

Storage-related issues could stem from bugs in the implemen-

tation itself, or programmers might carelessly authorize untrust-

worthy contracts. The potential impact of developers willingly

authorizing another contract is similar to what can happen with

DELEGATECALL in Ethereum smart contracts [

]. This call will

also execute another contract and authorize (delegate) the other

contract to interact with the same storage.

3 RELATED WORK

Dierential fuzzing has been used to test compilers, and libraries

since before the concept of smart contracts were envisioned.

CSmith

A notable example for dierential testing is CSmith

by Yang et al., which auto-generates C programs and hunts for

dierences in program behavior compiled with dierent compilers

and optimization levels. It found a wide variety of bugs in dierent

C compilers [25].

DifFuzz

With DifFuzz, Nilizadeh et al. propose the use of dif-

ferential fuzzing to nd side channels by observing the program

behavior concerning a resource [17].

NEZHA

Petsios et al. proposed NEZHA, a generalized approach

to dierential fuzzing of conventional binaries, which is more ef-

cient than existing tools at nding bugs in complex software,

targeted mainly towards binary libraries. Instead of using the code

coverage of a single target as feedback, they use multiple feedback

maps for the targets under test [23].

HyDi

HyDi by Noller et al. makes use of similar concepts as

NEZHA, for example, tracking changes of branch coverage in one

of the targets under test, but also makes use of symbolic execution

and static analysis. The tool has a range of divergence heuristics to

improve the selection of the mutated inputs [18].

EVMFuzz

EVMFuzz [

] is a fuzzer that was proposed by Fu et al.

for dierential fuzzing smart contract VMs. It generates contracts

and feeds them into multiple Ethereum VMs, in order to nd their

discrepancies. The tool works by mutating smart contracts on the

Solidity language level.

4 NEODIFF

Diff

Generator

VM1

VM0

Diff

Analyzer Minimizer

NeoDiff

Results

Tracer

𝛕 Map

𝝈0..n ,𝛕0..n 𝝈0..n ,𝛕0..n

𝝈diff ,𝛕diff , Code

𝛕0...n , Code

𝛕0...n , Code𝛕

min

Code

Feedback

𝛕 , Code

Mutator

Target speciﬁc

Figure 2: NeoDi Building Blocks and Data Flow

This section discusses the concepts behind NeoDi. The NeoDi

fuzzer performs dierential testing [

] on alternative implementa-

tions of the same VM in order to nd discrepancies between their

execution.

4.1 Design

NeoDi is feedback-guided yet still able to explore multiple tested

virtual machines at the same time. With options for dierent light-

weight and portable feedback mechanisms, we can adapt NeoDi

to VMs written in a range of dierent languages across multiple

smart contract platforms. NeoDi is purpose-built from scratch

to a) be quickly adaptable to multiple targets b) mutate complex

bytecode c) work with additional state feedback and waypoints.

In contrast to prior work, such as EVMFuzz, NeoDi takes a

lower-layer approach, generating valid opcode sequences through

feedback-based mechanisms directly. Consequently, our approach

will emit any possible opcode sequence, while the Solidity compiler

may not emit sequences that don’t have a meaning on its higher

language level. We expect smart contract VMs to be less tested

against non-standard opcode sequences that are never emitted by

higher-level languages.

In the following sections, the important pieces of NeoDi, as

depicted in Fig. 2 will be discussed in detail.

Di Generator

The Di Generator generates code to be exe-

cuted by the VMs. For raw bytecode, the feedback-based mutator

will yield good results, applying a set of useful mutations to the

test cases, such as splicing and appending of high-coverage in-

put, random byte ips, and more. In comparison to language-level

fuzzers, like EVMFuzz [

], bytecode level mutators are also able to

ROOTS’21, November 18–19, 2021, Vienna, Austria Maier et al.

nd uncommon and invalid bytecode sequences that the high-level

language compiler will never emit.

Mutators

It’s possible to specify custom mutators to customize

the fuzzer for specic targets. By default, NeoDi will apply one

of the following mutations in a loop, starting from an empty input,

until it reaches a minimum contract length:

•

Random Bytes: A random amount (0x1 - 0xF) random bytes

get appended to the contract.

•

Full Splice: Parts of a random second contract are appended.

Favors test cases that produced new coverage.

•

Partial Splice: Parts of a random second contract are ap-

pended. Favors test cases that produced new coverage.

•

Byte Insert: NeoDi inserts a random byte at a random place

into the current contract.

•

Special Operations: Special important VM opcodes, such as

PUSHBYTES for ETH, are pushed.

In contrast to existing coverage-guided fuzzers, like AFL++app,

neo-di can randomly generate completely new contracts, with the

goal of not getting stuck on a local maximum.

The NeoDi Evaluator will then extract bytecode sequences of

successful runs to be used by the next NeoDi Generator run.

On top of the predened mutations, it’s easy to dene chain-

specic mutators. As a concrete example, for semantic dierences

in Python, we implement a custom mutator that generates Python

code. It emits valid Python code, each block according to a state

value, which may be mutated. The generated code from the Di

Generator is then passed on to both VMs to be executed.

Di Analyzer

The instrumentation of the VMs must be adapted

to produce a valid trace for the Di Analyzer. The trace returned

from the run of a VM is a list of executed opcodes together with

a so-called type-hash

and a state-hash

. The Di Analyzer can

then compare

of both traces to nd dis and use

to classify

if new states were reached. We assume a state to diverge if any

state-hash

during execution diers between VMs, for the same

input.

Minimizer

In case of a di, the oending opcode included

, as well as the code that caused the di, is stored in NeoDi

Results. The part of the code that executed cleanly is forwarded

to the Minimizer, together with any

of this run. The Minimizer

uses the instrumented VMs to nd the shortest byte sequence that

results in a specic

and stores it in the

Map. The

Map is

growing over the course of the fuzzing session and can be used by

the Di Generator as feedback for further mutations.

𝛕0𝝈0

Opcode 𝛕1𝝈1

𝛕2𝝈2

Time

Stack Memory

Smart Contract VM Execution Trace NeoDiff Data

Opcode Stack Memory

State Subset

Figure 3: Relation between VM internal execution states and

the NeoDi type-hash τand state-hash σ

4.2 State-Hash

To identify dierential execution, NeoDif makes use of a so-called

state-hash

. Fig. 3 shows that

will take a subset from the current

state during execution, as well as every previous state. A researcher

has to dene which information is important for a di and include

it in the hash. In a stack-based VM, this includes a probabilistic

sample of the stack; if there are registers,

should include register

values, and if there is additional memory, this memory should be

taken into consideration. Of course, there is a trade-o between

execution speed and precision that has to be taken into account.

The state-hash

must be continuously updated, for each opcode,

if possible. NeoDi then uses the state-hash to detect dis in any

of the executed operations.

4.3 Type-Hash

As can be seen in Fig. 3, type-hash

also uses state information

from the execution trace, but it is not chained with previous states

during execution—it only represents a single opcode state. The

name type-hash stems from their purpose in the Neo VM, as the

Neo VM supports various types such as Integer, ByteArray, Array,

Boolean, Map, and Struct. Even though EVM only supports 256-

bit ints as types, we derived virtual types, such as valid addresses,

small ints, bools, that are known to lead to other executions inside

opcodes. While NeoDi can also work on traditional code cover-

age, if available, the type-hash

helps to sort and categorize dis

and allows to quickly recognize patterns—for example if all dis

happened due to the top stack item being of a certain type. For

the type-hash, NeoDi uses the type of the top two stack items,

prexed with the current opcode. In Neo, if the ADD instruction is

executed while an Integer (1) and a ByteArray (2) are on the stack,

the type-hash τof this moment would be ADD_12.

Type-hashes are similar to previously proposed human-guided

feedback mechanisms like waypoints by Padhye et al. [

] and

annotations for Ijon, by Aschermann et al. [

], although both in-

crease the engineering overhead, while

is specically designed as

a lightweight coverage metric for dierential VM fuzzing and other

exotic targets. Instrumenting VMs, written in a range of program-

ming languages, with full code coverage information per opcode

can be an additional engineering task, whereas type-hashes work

well across virtual machines for the same smart contract platform,

making them a good t for this use case.

To evaluate the eectiveness of type-hashes, we tested how many

times a newly-found type-hash results in improved code coverage

of the underlying VM. For this reason, we instrumented the neo-

python VM with code tracing, logging the execution trace for each

opcode and their type-hash. The Fig. 4 shows for each opcode the

ratio between nding a new type-hash and it also being new code

coverage. It shows that nding a new

in 86% of cases (median)

also reaches new code coverage internally, making it a benecial

feedback channel.

Still, this does not imply that type-hashing will pick up all

changes in coverage for each of the tested VMs. Using actual code

coverage as

is supported by the di engine, and test cases can be

shared with AFL++ [6].

Uncovering Smart Contract VM Bugs

Via Dierential Fuzzing ROOTS’21, November 18–19, 2021, Vienna, Austria

0x0 0x1a 0x34 0x4f 0x6a 0x84 0x9e 0xb8 0xd2 0xec 0xﬀ

NEO Opcode

20%

40%

60%

80%

100%

% of Typehashes generating new Coverage

mean (83.66%)

median (85.96%)

Figure 4: Percentage of new type-hashes reaching new cover-

age per opcode in neo-python: over 83% of new type-hashes

also lead to new coverage.

4.4 Minimization

After bytecode executions, NeoDi collects the feedback. When it

discovers a new type-hash

, it minimizes the code to reach this

. The goal is to use the shortest possible smart contract with the

same semantics, as the contracts tend to grow fairly big otherwise.

The minimization re-runs the VMs with an increasing number

of bytes of the test case. For each byte, it checks if any of the new

are included in the trace and stores them in the

Map. Once

all type-hashes

are found during execution, we assume that the

test case is minimal. This byte-for-byte minimization makes the

minimization part of NeoDi portable to other VMs. An issue is

that bytecode might not execute linearly due to jump opcodes. This

could result in the opcode of interest that leads to a new type-hash,

to sit in the middle of the code. Though most opcodes are executed

linearly and thus the best-eort approach should suce.

4.5 Di Triaging

Fuzzing VMs for dierentials is an iterative process.To take the

burden o of triaging, Neo Dis groups dis based on the type-hash

of a potentially oending opcode. Simply re-running the VMs

and looking at the produced states allows very easy verication

of these dis. For example, through NeoDi we quickly found go-

ethereum and openethereum return dierent values for opcodes like

GASLIMIT. However, it was a false-positive in this case, caused by

dierent gas limit default congurations in each chain. The security

researcher using NeoDi can then either ignore false positives or

adjust the fuzzer conguration.

4.5.1 C# Neo VM against neo-python. To target the Neo VM, we

included the current opcode and stack values in each step of the

execution in the state-hash

. To instrument the Neo VM with a

tracer that produces output for the Di Analyzer, we call each VM’s

ExecutionEngine directly. As both the Python VM and the C# VM

share a similar code design, this approach works for both. As Neo

VM knows dierent types,

was chosen to use the opcode and the

type of the topmost stack elements.

4.5.2 Geth against Openethereum. To instrument the Ethereum

VM, we feed the existing JSON trace outputs from go-ethereum(geth)

and openethereum (formerly parity) into NeoDi. In contrast to

the Neo VM, the Ethereum VM does not have a concept of types.

Everything is encoded in uint256 values. So we instead created

based on heuristical virtal types. Though in this case, it is still possi-

ble to dene virtual types to serve a similar purpose. For example, if

a given value on the stack looks like a potential address, we assign

it the virtual type of Address. Similarly, we interpret values that are

higher or lower than addresses as another virtual type, that poten-

tially leads to other code paths in the opcode execution handlers.

In total, for EVM, we deduct the following 6 virtual types, based on

value ranges:

(1) Empty Value not set

(2) Boolean Values 0 or 1

(3) Ethereum Address Exactly 160 bit

(4) ByteArray More than 160 bits

(5) Large Number More than 64 bits

(6) Number All other values

Using these values as

, we were able to plug Ethereum into

the dierential fuzzer NeoDi, using only a few lines of additional

tracing code and some custom mutations.

4.5.3 CPython aginst Neo Python. We developed a custom mutator

for NeoDi, which is able to produce valid Python scripts that

can also run as valid Neo smart contracts with various random

expressions, followed by a return of a unique random identier

and whatever value was calculated before. For Python ding, the

state-hash consists of the hash of articially crafted return tuples,

containing the value calculated in the previous expression and a

unique id of the executed code path.

5 EVALUATION

To test the dierential fuzzer NeoDi, we implemented target-

specic code for multiple targets. In total, we ran NeoDi against

3 Targets, with 2 VMs each: the Neo C# VM against neo-python,

and the two largest Ethereum VMs, geth (go) and openethereum

(rust). As the third backend, we pivoted NeoDi to fuzz language

semantics. Namely, we di CPython against Neo smart contracts

written in Python and compiled with neo-boa to the Neo VM.

5.1 C# Neo VM against neo-python

We fuzzed neo-python:v0.9.1 against the neo-vm:2.4.3 C# VM. To

instrument the Neo VM with a tracer that produces output for the

Di Analyzer, NeoDi executes each VM’s ExecutionEngine directly.

We patched in a persistent mode, receiving inputs from NeoDi

without the need to respawn the VM, allowing us to reach thousands

of execs per second. The execution speed depends heavily on the

length and kind of contracts currently emitted (i.e., loops drag down

the execution speed). As Neo VM knows dierent types, we set

to use the opcode and the type of the topmost stack elements.

Using Neo as a base, we evaluated four mutation strategies, each

in a 20 opcode, 20o, and a 500 opcode, 500o, variant - stopping the

execution of the VM at a maximum of 20, or 500 executed opcodes,

respectively. All mutation strategies nd dis, with the purely ran-

dom, random20o and random500o nding the most dierences at

rst glance. However, digging deeper, the pure-random dis are not

ROOTS’21, November 18–19, 2021, Vienna, Austria Maier et al.

0.0 0.2 0.4 0.6 0.8 1.0

Execs Accumulated ×108

Unique Opcodes

Unique Opcode Diﬀs

mut1p20o

mut20p20o

mut1p500o

mut20p500o

random20o

random500o

coverage20o

coverage500o

(a) Dis on Opcodes

0.0 0.2 0.4 0.6 0.8 1.0

Execs Accumulated ×108

100

200

300

400

Unique Diﬀs

Unique Typehash Diﬀs

mut1p20o

mut20p20o

mut1p500o

mut20p500o

random20o

random500o

coverage20o

coverage500o

(b) Dis with new τ

0.0 0.2 0.4 0.6 0.8 1.0

Execs Accumulated ×108

200

400

600

800

1000

1200

1400

1600

Typehashes Found

Typehashes

mut1p20o

mut20p20o

mut1p500o

mut20p500o

random20o

random500o

coverage20o

coverage500o

Figure 5: Fuzzing Results for multiple NeoDi runs of neo-python and the main Neo VM. We compare multiple mutators and

purely random generation, as a baseline

diverse, as seen in Fig. 5a, after days of runtime, never exceeding

22 unique ding opcodes.

The dierentmutation strategies we evaluated are:

(1) random

True-Random fuzzing with fully random bytes,

used as a base reference

(2) mut1p

Hand-written custom mutator for Neo, with snippets

to push special values to the stack. With low probability, it

will use the coverage τ-feedback.

(3) mut20p

Same mutations as mut1p—but will use feedback-

based mutations with 20 times the initial probability.

(4) coverage

NeoDi default mutation strategy, with initial ran-

dom byte sequences, mostly relying on coverage

-feedback

with random byte ips and splicing.

True random also lags behind all other on type-hashes, as seen

in Fig. 5c indicating the success of our mutation strategies. At

this metric, the type-hash-guided variants do very well. Fig. 5b,

showing the dis with unique type-hash proves that the pure-

random variants cannot compete against the custom mutators and

coverage-guided mutators. We draw another conclusion regarding

the size of the test case: while the 20p variants are faster initially,

they get overtaken by their larger variants eventually—probably

because complex operations cannot be expressed in 20 opcodes

easily. All mutation strategies found a wide range of real dis in

Neo. They are further discussed in-depth in Sect. 6.

5.2 Geth against Openethereum

For EVM, we perform the evaluation using the best mutation strat-

egy according to our prior experiments on Neo: a single mutator

with a very high likelihood of feedback-based mutations. State-

hashes σfor EVM include the opcode, the stack, the memory, and

the gas cost. A dierence in any of these would result in an ex-

ploitable discrepancy. To instrument the Ethereum VM, existing

JSON trace outputs from go-ethereum(geth) and openethereum

(formerly parity) can be reused and fed into NeoDi.

All in all, the Fuzzer found six instances on dierent opcodes,

see Fig. 6a. After triaging, all dierences could be traced back to

dierent behavior on empty chains between geth and openethereum.

We could not identify any security-critical dierences for existing

chains. More details about the dis can be found in the discussion

in Sect. 6.3

5.3 CPython aginst Neo Python

To di Python code, we developed a grammar-based custom mu-

tator for NeoDi, not based on

, which is able to produce valid

Python scripts that can also be run as valid Neo smart contracts.

We could not simply get an opcode trace, as the executed CPython

bytecode is fundamentally dierent from Neo bytecode. Instead,

for Python ding, the state-hash consists of the hash of articially

crafted return tuples, containing the value calculated in the previous

expression and a unique id of the executed code path.

NeoDi was able to nd several semantic dierences for the

Python language. After over 20k executions, NeoDi had already

identied a range of semantic dis between CPython and Neo Smart

contracts:

neo-python: 14 dis vs. Python2.7 | 13 dis vs. Python3.7

Neo VM: 11 dis vs. Python2.7 | 10 dis vs. Python3.7

In fact, what can be seen here is an unintended additional discov-

ery of regular VM dis between C# and neo-python. We explain

the security impact of semantic dierences for smart contracts in

Sect. 6.4.

6 DISCUSSION

In the following section, we present an in-depth security analysis

of the fuzz targets after analyzing the results of NeoDi.

6.1 Security Impact of Dierences in Smart

Contract VMs

To understand why dierences between smart contract VMs can be

classied as security-critical bugs, this section will briey introduce

three categories of dierences, together with their possible attacks.

6.1.1 On-Chain Di. On-Chain dis describe dis between VMs

that are deployed as the participating nodes of a blockchain net-

work. In the case of Ethereum, this refers to the miners who have

to conrm all transactions. For each execution of a smart contract

on the chain, the bytecode runs on all nodes that need to agree

Uncovering Smart Contract VM Bugs

Via Dierential Fuzzing ROOTS’21, November 18–19, 2021, Vienna, Austria

Uncovering Smart Contract VM Bugs

Via Dierential Fuzzing ROOTS’21, November 18–19, 2021, Vienna, Austria

0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6

Execs Accumulated ×107

Unique Opcodes

Unique Opcode Diﬀs

EVM run (Parity & Geth)

(a) Dis on Opcodes

0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6

Execs Accumulated ×107

Unique Diﬀs

Unique Typehash Diﬀs

EVM run (Parity & Geth)

(b) Dis with new θ

01234

Execs Accumulated ×107

500

1000

1500

2000

Typehashes Found

Typehashes

EVM run (Parity & Geth)

Figure 6: The results of our fuzzing run, comparing the Parity and Geth EVM VMs

with the propagated result in the chain when a block is forged.

Implementations between nodes on the chain must never diverge.

In a proof-of-work-based chain like Ethereum, a di between al-

ternative implementations with many active nodes can lead to a

split of the overall hashing power. Depending on the bug, the VM

implementation used by the majority mining power will keep the

main chain going, while the miners using the minority implemen-

tation may not agree and will therefore split o the chain, with a

dierent result for this transaction.

6.1.2 Blockchain Application Di. The issue described in the pre-

vious section can further be applied to applications built upon the

blockchain—including wallets, exchanges, DApps, and more. These

applications must understand new transactions and verify their

correctness by executing them. Local execution results must match

those on the main network. Else the application will suer from a

local chain split, resulting in a "wrong" view of the blockchain’s

state.

6.1.3 Contract Semantics Di. Language semantic dierences are

a discrepancy between the smart contract programming language

implementation and the original programming language. Semantic

dis happen if smart contracts can be written in languages bor-

rowed from other execution environments. In the case of Neo, smart

contracts can be implemented in many dierent languages, for ex-

ample, Python. If the behavior diers from the spec, developers may

introduce bugs. Their expectation of how the code should behave

diers from the actual implementation. A malicious developer can

abuse these dierent behaviors to hide backdoors in plain sight, as

even experienced Python developers have no chance to spot them.

6.2 PoC Attack on Neo VM Dierences

To show why VM discrepancies in Neo are an actual security issue,

we present an example attack for blockchain application dis.

First, we set up neo-local [

], a project to run a personal Neo

blockchain in Docker containers. It sets up a private Neo network

for testing, using the ocial C# node [

]. It also exposes a neo-

python [

] prompt to be used as a client to interact with the

blockchain. We set up a project that will use the Python VM to

execute the contracts locally and a C# VM that will be executing

contracts on the nodes. The attacks can be reproduced using the

version neo-local:0.12. With the neo-python prompt, it is possible to

compile and deploy the contract to the private chain. We depict this

in Listing 1. The resulting SmartContract.Contract.Create transac-

tion event also shows the deployed contract address, as well as the

transaction identier (line 8). After we deploy the contract, we can

invoke it using its address. A local test invocation, performed by

neo-python using the Python VM, returns the hex-encoded string

Python. After providing the wallet’s password, the transaction is

sent to the actual private blockchain network. Once the transaction

is broadcasted, the local wallet using neo-python will receive the

new transaction and execute it locally. As can be seen, the result of

the contract is indeed the string Python (line 22).

1neo>w al l et open neo -privnet .wallet

2neo>s c bu i ld /smart -contracts /div .p y

3Sav ed o ut pu t to /smart -contracts /div .avm

4neo>s c dep lo y /smart -contracts /div .avm False False False 00 0 7

5Pl e as e f i l l o ut the fo l lo w in g co n t r ac t d e t a i l s :

6[Co nt ra ct Name] > DIV PoC

7[ . . . ]

8[I19 1 00 6 1 3 : 2 4 : 4 6 EventHub: 6 2 ] [ SmartContract .Contract .Create ][6562] [0

f904e38fb40891daf11685a29f25a6716d331e7] [ t x 0

c86ab2fcce081f28259a9018e78dc8f8681046dfa06f414df4e521e28978657] { '

type':'InteropInterface','value ': { 'v er si o n ': 0 , 'hash ':'0

x0f904e38fb40891daf11685a29f25a6716d331e7','script':'58 c56b516a0052

[...]6c7566006c7566','parameters ': [ 'Signature'] , 'returntype':'

String','name':'DIV PoC ','code_version':' ' ,'author':'' ,'email ':

'' ,'description ':'' ,'properties ': { 'st o ra ge ':False ,'

dynamic_invoke':False ,'p ay ab le ':False } } }

9[ . . . ]

10 neo>s c in vok e 0x0f904e38fb40891daf11685a29f25a6716d331e7

11 [ . . . ]

12 ------------------------------------------------------------

13 Te s t i nv ok e s u c c e s s f u l

14 To tal op er ati ons : 3 9

15 Re s ul t s [ { 'type':'ByteArray ','value':'507974686 f6e'} ]

16 Inv ok e TX GAS co s t : 0 . 0

17 Inv oke TX f ee : 0 . 0 0 0 1

18 ------------------------------------------------------------

19 [password] > ∗∗∗

20 [ . . . ]

21 Rel ay ed Tx : 7740d42b12ecf4f7c9d92222b6eb629a2757c37e47b2cb6dffca3242b7ed98d8

22 [I19 1 00 6 1 3 : 2 5 : 0 3 EventHub: 6 2 ] [ SmartContract .Execution .Success ][6565] [0

f904e38fb40891daf11685a29f25a6716d331e7] [ t x 7740

d42b12ecf4f7c9d92222b6eb629a2757c37e47b2cb6dffca3242b7ed98d8] { 'type ':

'Array','value': [ { 'type ':'ByteArray','value ':b'Python'} ] }

23 [ . . . ]

Listing 1: DIV test contract deploy

To see what happened in the actual node running the C# Neo

VM, the JSON RPC endpoint exposed by one of the main nodes can

be used. This will return the result hex-encoded "437368617270"—it

returned the string Csharp instead.

Locally, the Python VM came to a dierent result when executing

the contract, compared to the real C# VMs running on the consensus

nodes!