Advanced Study of SDN/OpenFlow controllers
Applied Research Center for
Moscow State University
Applied Research Center for
Moscow State University
Applied Research Center for
Moscow State University
Applied Research Center for
Moscow State University
This paper presents an independent comprehensive anal-
ysis of the eﬃciency indexes of popular open source SD-
N/OpenFlow controllers (NOX, POX, Beacon, Floodlight,
MuL, Maestro, Ryu). The analysed indexes include per-
formance, scalability, reliability, and security. For testing
purposes we developed the new framework called hcprobe.
The test bed and the methodology we used are discussed
in detail so that everyone could reproduce our experiments.
The result of the evaluation show that modern SDN/Open-
Flow controllers are not ready to be used in production and
have to be improved in order to increase all above mentioned
Categories and Subject Descriptors
C.2.1 [Computer-Communication Networks]: Network
Architecture and Design; C.4 [Performance of Systems]
Design, Measurement, Performance, Reliability, Security
SDN, OpenFlow, Controllers evaluation, Throughput, La-
Software Deﬁned Networking (SDN) is the most discussed
topic of networking technology of recent years . It brings a
lot of new capabilities and allows to solve many hard prob-
lems of legacy networks. The approach proposed by the
SDN paradigm is to move network’s intelligence out from
the packet switching devices and to put it into the logically
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for proﬁt or commercial advantage and that copies
bear this notice and the full citation on the ﬁrst page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior speciﬁc
permission and/or a fee.
CEE-SECR ’13 Moscow, Russia
Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$15.00.
centralized controller. The forwarding decisions are done
ﬁrst in the controller, and then moves down to the overseen
switches which simply execute these decisions. This gives
us a lot of beneﬁts like global controlling and viewing whole
network at a time that helpful for automating network op-
erations, better server/network utilization, and etc. Under-
standing all beneﬁts Google and Microsoft recently switched
to SDN in their data centers [2, 3].
In SDN/OpenFlow paradigm switches provide common
and vendor-agnostic protocol for remote controller called
OpenFlow . This protocol provides the controller a way
to discover the OpenFlow-compatible switches, deﬁne the
matching rules for the switching hardware and collect statis-
tics from switching devices. A switch presents a ﬂow ta-
ble abstraction: each ﬂow table entry contains a set of
packet ﬁelds to match, and an action (such as send-out-port,
modify-ﬁeld, or drop). When an OpenFlow Switch receives
a packet it has never seen before, for which it has no match-
ing ﬂow entries, it sends this packet to the controller via
a packet in message. The controller then makes a decision
on how to handle this packet. It can drop the packet, send
packet to a port via a packet out message, or it can add a
ﬂow entry directing the switch on how to forward similar
packets in the future via ﬂow mod message.
An SDN/OpenFlow controller for networks is like an op-
erating system for computers. But the current state of the
SDN/OpenFlow controller market may be compared to early
operating systems for mainframes, which existed 40-50 years
ago. Operating systems were control software that provided
libraries of support code to assist programs in runtime oper-
ations just as today’s controllers are special control software,
which interacts with switching devices and provides a pro-
grammatic interface for user-written network management
Early operating systems were extremely diverse, with each
vendor or customer producing one or more operating sys-
tems speciﬁc to their particular mainframe computers. Ev-
ery operating system could have radically diﬀerent models
of command and operating procedures. The question of op-
erating system eﬃciency was of vital importance and, ac-
cordingly, was studied a lot . However, it was diﬃcult to
reproduce eﬃciency evaluation experiments applied to oper-
ating systems because of a lack of methodology descriptions,
a mixture of running applications that created the workload
in the experiments, unique hardware conﬁgurations, etc .
The same is true for today’s SDN/OpenFlow controllers.
At present, there are more than 30 diﬀerent Open-
Flow controllers created by diﬀerent vendors/universi-
ties/research groups, written in diﬀerent languages, and us-
ing diﬀerent runtime multi-threading techniques. Thus, the
issue of eﬃciency evaluation has emerged.
The latest evaluation of SDN/OpenFlow controllers, pub-
lished in 2012 , covered a limited number of controllers
and focused on controller performance only. During the
last year new controllers appeared, and the old ones were
updated. This paper was intended as impartial compara-
tive analysis of the eﬀectiveness of the most popular and
widely used controllers. In our research we treat the term
”controller eﬀectiveness” as a list of indexes, namely: perfor-
mance, scalability, reliability, and security. Later on we will
introduce the meaning of every index from this list.
For testing controllers we used a well known tool
cbench . However, we were not satisﬁed with the capabil-
ities of cbench, and so we developed our own framework
for SDN/Open-Flow controllers testing, which we called
hcprobe . We developed our own solution because the
capabilities of cbench, the traditional benchmark used for
controllers evaluation, cover only a small range of possible
test cases. It is perfectly ﬁtting for rough performance test-
ing, but it does not provide the ﬁne-tuning of test param-
eters. We use hcprobe to create a set of advanced test-
ing scenarios, which allow us to get a deeper insight into
the controller reliability and security issues: the number of
failures/faults during long-term testing, uptime for a given
workload proﬁle, and processing malformed OpenFlow mes-
sages. With cbench and hcprobe we performed an experi-
mental evaluation of seven freely available OpenFlow con-
trollers: NOX , POX , Beacon , Floodlight ,
MuL , Maestro , and Ryu .
Similar to conventional benchmarks, hcpobe emulates any
number of OpenFlow switches and hosts connected to a con-
troller. Using hcprobe, one can test and analyse several in-
dexes of controller operation in a ﬂexible manner. Hcprobe
allows to specify patterns for generating OpenFlow messages
(including malformed ones), set traﬃc proﬁle, etc. It is writ-
ten in Haskell and allows users to easily create their own
scenarios for controller testing.
The key features of hcprobe are:
•correct OpenFlow and network stack packet genera-
•implementation in a high-level language, which makes
it easier to extend;
•API for custom tests design;
•introducing an embedded domain-speciﬁc language
(eDSL) for creating tests.
The architecture of hcprobe (see Figure 1):
•Network datagram: The library, which provides
typeclass-based interface for easy creation of packets
for network protocols: Ethernet frames, ARP pack-
ets, IP packets, UDP packets, TCP packets. It allows
to set a custom value for any ﬁeld of the header and
generate a custom payload.
Figure 1: The hcrobe architecture
•OpenFlow: The library for parsing and creating
•Conﬁguration: The library that provides the rou-
tines for conﬁguration of the test’s parameters using
command line options or a conﬁguration ﬁle.
•FakeSwitch: Basic ”fake switch”, which implements
the initial handshake, manages the interaction with the
controller and provides callback/queue-based interface
for writing tests.
•eDSL: Provides easier interface for creating tests.
Using these modules, it is possible to create speciﬁed tests
for SDN/OpenFlow controllers, including functional, secu-
rity and performance tests. The most convenient way to cre-
ate tests is to use the embedded domain-speciﬁc language.
Hcprobe domain-speciﬁc language provides the following
•create OpenFlow switches;
•write programs describing the switch logic and run pro-
grams for diﬀerent switches simultaneously;
•create various types of OpenFlow messages with cus-
tom ﬁeld values and also generate custom payload for
•aggregate statistics on message exchange between
OpenFlow controller and switches.
For a basic example, which illustrates the above features,
see Figure 2.
Figure 2: The hcprobe test example (incorrect Pack-
etIn OF message)
So hcprobe provides the framework for creating various
test cases to study the behaviour of OpenFlow controllers
processing diﬀerent types of messages. One can generate all
types of switch-to-controller OpenFlow messages and replay
diﬀerent communication scenarios, even the most sophisti-
cated ones, which cannot be easily reproduced using hard-
ware and software OpenFlow switches. The tool could be
useful for developers to perform functionality, regression and
safety testing of controllers.
3. EXPERIMENTAL SETUP
This section explains our testing methodology and tech-
nical details of experimental evaluation.
3.1 Test bed
The test bed consists of two servers connected with a
10Gb link. Each server has two processors Intel Xeon E5645
with 6 cores (12 threads), 2.4GHz and 48 GB RAM. Both
servers are running Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-
23-generic x86 64) with default network settings. One of
the servers is used to launch the controllers. We bind all
controllers to the cores of the same processor in order to
decrease the inﬂuence of communication between the pro-
cessors. The second server is used for traﬃc generation ac-
cording to a certain test scenario. We use two server setup
instead of a single server conﬁguration where both a con-
troller and a traﬃc generator run on the same server. Our
previous measurements have shown that the throughput of
internal traﬃc is rather unpredictable and unstable, with
throughput surging or dropping by an order. Also, using
two servers we can guarantee that traﬃc generation doesn’t
aﬀect the controller operation.
Based on the study of available materials on twenty four
SDN/OpenFlow controllers, we chose the following seven
open source controllers:
•NOX  is a multi-threaded C++-based controller
written on top of Boost library.
•POX  is a single-threaded Python-based controller.
It is widely used for fast prototyping of network appli-
cations in research.
•Beacon  is a multi-threaded Java-based controller
that relies on OSGi and Spring frameworks.
•Floodlight  is a multi-threaded Java-based con-
troller that uses Netty framework.
•MUL  is a multi-threaded C-based controller writ-
ten on top of libevent and glib.
•Maestro  is a multi-threaded Java-based controller.
•Ryu  is Python-based controller that uses gevent
wrapper of libevent.
Initially we included in our study Trema , Flower 
and Nettle  controllers. But after several experiments we
understood that they were not ready for evaluation under
Each tested controller runs an L2 learning switching appli-
cation provided by the controller. There are several reasons
for this choice. It is quite simple and at the same time repre-
sentative. It uses all of the controller’s internal mechanisms,
and also shows how eﬀective the chosen programming lan-
guage is by implementing single hash lookup.
We relied on the latest available sources of all controllers
dated April, 2013. We run all controllers with the rec-
ommended settings for performance and latency testing, if
Our testing methodology includes performance and scala-
bility measurements as well as advanced functional analysis
such as reliability and security. The meaning of each term
is explained below.
Performance of an SDN/OpenFlow controller is deﬁned
by two characteristics: throughput and latency. The goal
is to obtain maximum throughput (number of outstanding
packets, ﬂows/sec) and minimum latency (response time,
ms) for each controller. The changes in this parameters
when adding more switches and hosts to the network or
adding more CPU cores to the server where the controller
runs show the scalability of the controller.
We analyse the correlation between controller’s through-
put and the following parameters:
1. the number of switches connected to the controller (1,
4, 8, 16, 64, 256), having a ﬁxed number of hosts per
2. the number of unique hosts per switch (103, 104, 105,
106, 107), having a ﬁxed number of switches (32).
3. the number of available CPU cores (1–12).
The latency is measured with one switch, which sends
requests, each time waiting for the reply from the controller
before sending the next request.
All experiments are performed with cbench. Each cbench
run consists of 10 test loops with the duration of 10 secs.
We run each experiment three times and take the average
number as the result.
The scripts with the methodology realization can be found
in the repository: https://github.com/ARCCN/ctltest.
By reliability we understand the ability of the controller
to work for a long time under an average workload without
accidentally closing connections with switches or dropping
OpenFlow messages from the switches.
To evaluate the reliability, we measure the number of fail-
ures during long-term testing under a given workload proﬁle.
The traﬃc proﬁle is generated using hcprobe. We use ﬂow
setup rate recorded in Stanford campus in 2011 as a sample
workload proﬁle (Figure 3). This proﬁle stands for a typical
workload in a campus or oﬃce network during a 24-hour
period: the highest requests rate is seen in the middle of the
day, and by night the rate decreases.
In our test case we use ﬁve switches sending OpenFlow
PacketIn messages with the rate varying from 2000 to 18000
requests per second. We run the test for 24 hours and record
the number of errors during the test. By error we mean
either a session termination or a failure to receive a reply
from the controller.
Figure 3: The sample workload proﬁle
To identify any security issues, we have also examined how
the controllers manipulate malformed OpenFlow messages.
In our test cases we send OpenFlow messages with incor-
rect or inconsistent ﬁelds, which could lead to a controller
4.3.1 Malformed OpenFlow header.
Incorrect message length. PacketIn messages with modi-
ﬁed length ﬁeld in OpenFlow header have been generated.
According to the protocol speciﬁcation , this ﬁeld stands
for the whole message length (including the header). Its
value was set as follows: 1) length of OpenFlow header;
2) length of OpenFlow header plus ’a’, where ’a’ is smaller
than the header of the encapsulated OpenFlow message; 3)
a number larger than the length of the whole message. This
test case examines how the controller parses the queue of
Invalid OpenFlow version. All controllers under test sup-
port OpenFlow protocol v1.0. In this test case we start the
session with announcing OpenFlow v1.0 during the hand-
shake between the switch and the controller, then set an
invalid OpenFlow protocol version in PacketIn messages.
Incorrect OpenFlow message type. We examine how the
controller processes the OpenFlow messages, in which the
value of the ’type’ ﬁeld of the general OpenFlow header
does not correspond to the actual type of the encapsulated
OpenFlow message. As an example we send messages with
OFPT PACKET IN in the ’type’ ﬁeld, but containing Port-
4.3.2 Malformed OpenFlow message.
PacketIn message: incorrect protocol type in the encap-
sulated Ethernet frame. This test case shows how the con-
troller’s application processes an Ethernet frame with a mal-
formed header. We put the code of the ARP protocol
(0x0806) in the Ethernet header, though the frame contains
the IP packet.
Port status message: malformed ’name’ ﬁeld in the port
description. The ’name’ ﬁeld in the port description struc-
ture is an ASCII string, so a simple way to corrupt this ﬁeld
is not to put ’\0’ at the end of the string.
In this section we present the results of the experimen-
tal study of the controllers performed with cbench (perfor-
mance/scalability tests) and hcprobe (reliability and secu-
CPU cores scalability (see Figure 4). Python controllers
(POX and Ryu) do not support multi-threading, so they
show no scalability across CPU cores. Maestro’s scalabil-
ity is limited to 8 cores as the controller does not run with
more than 8 threads. The performance of Floodlight in-
creases steadily in line with the increase in the number of
cores, slightly diﬀering on 8-12 cores. Beacon shows the best
scalability, achieving the throughput of nearly 7 billion ﬂows
The diﬀerence in throughput scalability between multi-
threaded controllers depends on two major factors: the ﬁrst
one is the algorithm of distributing incoming messages be-
tween threads, and the second one is the mechanism or the
libraries used for network interaction.
NOX, MuL and Beacon pin each switch connection to a
certain thread, this strategy performs well when all switches
send approximately equal number of requests per second.
Maestro distributes incoming packets using round-robin al-
gorithm, so this approach is expected to show better results
with an unbalanced load. Floodlight relies on Netty library,
which also associates each new connection with a certain
Figure 4: The average throughput achieved with dif-
ferent number of threads (with 32 switches, 105hosts
Correlation with the number of connected switches (see
Figure 5). The performance of POX and Ryu does not de-
pend on the number of switches, as they do not support
multi-threading. For multi-threaded controllers adding more
switches leads to better utilization of available CPU cores,
so their throughput increases until the number of connected
switches becomes larger than the number of cores. Due
to the round-robin packets distribution algorithm, Maestro
shows better results on a small amount of switches, but the
drawback of this approach is that its performance decreases
when a large number of switches (256) is connected.
Correlation with the number of connected hosts (see Fig-
ure 6). The number of hosts in the network has immaterial
Figure 5: The average throughput achieved with dif-
ferent number of switches (with 8 threads, 105hosts
inﬂuence on the performance of most of the controllers under
test. Beacon’s throughput decreases from 7 to 4 billion ﬂows
per second with 107hosts, and the performance of Maestro
goes down signiﬁcantly when more than 106hosts are con-
nected. This is caused by speciﬁc details of Learning Switch
application implementation, namely, the implementation of
its lookup table.
Figure 6: The average throughput achieved with dif-
ferent number of hosts (with 8 threads, 32 switches)
The average response time of the controllers demonstrates
insigniﬁcant correlation with the number of connected hosts.
For the average response time with one connected switch and
105hosts see Table 1. The smallest latency has been demon-
strated by MuL and Beacon controllers, while the largest
latency is typical of python-based controller POX.
Table 1: The minimum response time (10−6secs/
The results of performance/scalability testing show that
Beacon controller achieves the best throughput (about 7 bil-
lion ﬂows/sec) and has a potential for further throughput
scalability if the number of available CPU cores is increased.
The scalability of other multi-threaded controllers is limited
by 4-8 CPU cores. Python-based controllers POX and Ryu
are more suitable for fast prototyping than for enterprise
The experiments have shown that most of the controllers
successfully cope with the test load, although two con-
trollers, MuL and Maestro, start to drop PacketIns after
several minutes of work: MuL has dropped 660,271,177 mes-
sages and closed 214 connections, and Maestro has dropped
463,012,511 messages without closing the connections. For
MuL, failures are caused by problems with Learning Switch
module that cannot add new entries to the table, which leads
to packet loss and the closing of connections with switches.
5.3.1 Malformed OpenFlow header
Incorrect message length (1). Most of the controllers will
not crash on receiving this type of malformed messages but
close the connection with the switch that sends these mes-
sages. The two controllers that crash upon receiving mes-
sages with incorrect value of length ﬁeld are Maestro and
NOX. Maestro crashes only when these messages are sent
without delay, however, if we set a delay between two mes-
sages to 0.01 second the controller replies with PacketOuts.
Ryu is the only controller whose work is not aﬀected by mal-
formed messages, it ignores the messages without closing the
Invalid OpenFlow version (2). POX, MuL and Ryu close
the connection with the switch upon receiving a message
with invalid protocol version. Other controllers process mes-
sages regardless of an invalid protocol version and reply with
PacketOuts with version set to 0x01.
Incorrect OpenFlow message type (3). MuL and Ryu ig-
nore these messages. MuL sends another FeaturesRequest
to the switch and after receiving the reply closes the con-
nection with the switch. Other controllers parse them as
PacketIn messages and reply with PacketOuts.
5.3.2 Malformed OpenFlow message
PacketIn message (4). MuL closes the connection with
the switch after sending an additional FeaturesRequest and
receiving the reply. Other controllers try to parse encapsu-
lated packets as ARP and reply with a PacketOut message.
POX is the only controller which detects invalid values of
ARP header ﬁelds.
Port status message (5). All controllers process messages
with malformed ASCII strings correctly, reading the number
of characters equal to the ﬁeld length.
The summary of the security tests results is shown in the
Table 2, where: red cell — the controller crashed, orange
cell — the controller closed the connection, green cell — the
controller successfully passed the test. If the cell is yellow,
the controller processed the message without crashing or
closing the connection, but the error was not detected, which
is a possible security vulnerability.
Thus, the experiments demonstrate that sending mal-
1 2 3 4 5
Table 2: Security tests results
formed OpenFlow messages to a controller can cause the
termination of a TCP session with the switch or even the
controller’s failure resulting in a failure of a network segment
or even of the whole network. The expected behaviour for
the OpenFlow controller on receiving a malformed message
is ignoring it and removing from the incoming queue with-
out aﬀecting other messages. If the controller proceeds the
malformed message without indicating an error, this could
possibly cause incorrect network functioning. So it is impor-
tant to verify not only the OpenFlow headers, but also the
correctness of the encapsulated network packets.
We present an impartial experimental analysis of the cur-
rently available open source OpenFlow controllers: NOX,
POX, Floodlight, Beacon, MuL, Maestro, and Ryu.
The key results of the work:
•The list of the controllers under the test has been ex-
tended as compared to previous studies.
•We present a detailed description of the experiment
setup and methodology.
•Controller performance indexes have been upgraded
using advanced hardware with a 12 CPU threads
conﬁguration (previous research used maximum 8
threads). Maximum throughput is demonstrated by
Beacon, with 7 billion ﬂows per second.
•Reliability analysis showed that most of the controllers
are capable of coping with an average workload during
long-term testing. Only MuL and Maestro controllers
are not ready for the 24/7 work.
•Security analysis highlighted a number of possible se-
curity vulnerabilities of tested controllers which can
make them targets for the malware attacks.
The results show that these aspects have to be taken into
account during development of new SDN/OpenFlow con-
trollers and should be improved for the current controllers.
The current controllers have bad scaling over the cores and
will not able to meet increasing demands in communication
for future data centers. The security and reliability have
highest priorities for enterprise networks.
We also present hcprobe — the framework, which per-
fectly ﬁts the demands of ﬁne-grained functional and secu-
rity testing of controllers. It is a good base for a full test
suite for testing SDN/OpenFlow controllers on compliance
to the OpenFlow speciﬁcation (like OFTest for OpenFlow
 M. Casado, T. Koponen, D. Moon, S. Shenker.
Rethinking Packet Forwarding Hardware. In Proc. of
 S. Jain, A. Kumar, S. Mandal. B4: experience with a
globally-deployed software deﬁned wan. In proceedings
of ACM SIGCOMM conference, ACM, 2013
 C. Hong, S. Kandula, R. Mahajan. Achieving high
utilization with software-driven WAN. In proceedings of
ACM SIGCOMM conference, ACM, 2013
 Open Networking Foundation. OpenFlow Switch
 D. Ferrary. Computer Systems Performance Evaluation.
N.J.:Prentice-Hall, Englewood Cliﬀts,1978
 D. Ferrary, M. Liu. A general purpose software
measurement tools. Software Practice and Experience,
 Amin Tootoonchian, Sergey Gorbunov, Martin Casado,
Rob Sherwood. On Controller Performance in
Software-Deﬁned Networks. In Proc. Of HotIce, April,
 Cbench. http://docs.projectﬂoodlight.org/display
 Hcprobe. https://github.com/ARCCN/hcprobe/
 N. Gude, T. Koponen, J. Pettit, B. Pfaﬀ, M. Casado,
N. McKeown, and S. Shenker. NOX: towards an
operating system for networks. SIGCOMM Computer
Communication Review 38, 3 (2008), 105-110.
 POX. http://www.noxrepo.org/pox/about-pox/
 Floodlight. http://Floodlight.openﬂowhub.org/
 Mul. http://sourceforge.net/p/mul/wiki/Home/
 Zheng Cai, Maestro: Achieving Scalability and
Coordination in Centralized Network Control Plane,
Ph.D. Thesis, Rice University, 2011
 Ryu. http://osrg.github.com/ryu/
 FlowER. https://github.com/travelping/ﬂower
 Trema. http://trema.github.com/trema/
 Nettle. http://haskell.cs.yale.edu/nettle/
 Big Switch Networks OFTest – Validating OpenFlow