Adaptive State Consistency
for Distributed ONOS Controllers
Fetia Bannour, Sami Souihi and Abdelhamid Mellouk
LISSI/TincNetwork Research Team
University of Paris-Est Cr´
eteil (UPEC), France
firstname.lastname@example.org, (sami.souihi, mellouk)@u-pec.fr
Abstract—Logically-centralized but physically-distributed
SDN controllers are mainly used in large-scale SDN networks for
scalability, performance and reliability reasons. These controllers
host various applications that have different requirements in
terms of performance, availability and consistency. Current
SDN controller platform designs employ conventional strong
consistency models so that the SDN applications running on
top of the distributed controllers can beneﬁt from strong
consistency guarantees for network state updates. However, in
large-scale deployments, ensuring strong consistency is usually
achieved at the cost of generating performance overheads and
limiting system availability. That makes weaker optimistic
consistency models such as the eventual consistency model
more attractive for SDN controller platform applications with
high-availability and scalability requirements. In this paper,
we argue that the use of the standard eventual consistency
models, though a necessity for efﬁcient scalability in modern
SDN systems, provides no bounds on the state inconsistencies
tolerated by the SDN applications. To remedy that, we propose
an adaptive consistency model for the distributed ONOS
controllers following the notion of continuous and compulsory
(per-controller) eventual consistency, where network application
states adapt their eventual consistency level dynamically at run-
time based on the observed state inconsistencies under changing
network conditions. When compared to the ONOS approach
to static eventual consistency, our approach proved efﬁcient
in minimizing state synchronization overheads while taking
into account application state consistency SLAs and without
compromising the application requirements of high-availability,
in the context of large-scale SDN networks.
Index Terms—Software-Deﬁned Networking (SDN), logically-
centralized control, physically-distributed control, scalability, re-
liability, adaptive consistency, eventual consistency, consistency-
based Service-Level Agreement (SLA), ONOS controller,
The emergence of Software-Deﬁned Networking (SDN) has
aroused great interest in rethinking traditional approaches to
network architecture, design and management. By decoupling
the control plane logic from the data forwarding hardware,
SDN aims for a centralized control (governed by a global
network view) and for network programmability, thereby
enabling network innovation, simplifying network operations,
and easing service deployment. Another core beneﬁt of the
SDN paradigm is that it drives network abstraction and
automation along with the incorporation of adaptive con-
trol aspects in network conﬁguration and management, thus
promising to bring agility and to address the exponential
growth in network infrastructure and services.
Beyond the hype, there have been serious concerns about
the widespread adoption of SDN. For instance, the physical
centralization of the control plane in a single programmable
software-based controller, shows important limitations in
terms of scalability, availability and reliability, especially in
the context of large-scale real-world network deployments .
As a matter of fact, recent research in SDN perceives the
SDN control plane as a distributed system , where mul-
tiple physically-distributed SDN controllers are responsible
for handling multiple network domains, while synchronizing
their local state information and coordinating to maintain a
logically-centralized network view.
However, when compared to physically centralized SDN de-
signs, distributed SDN controller platforms face an additional
set of challenges related to inter-controller communications
for achieving network state consistency. In fact, guaranteeing
an appropriate level of controller state consistency while
sustaining good performance in a fault-tolerant SDN network
is a challenging task as stated by the CAP theorem . In this
respect, different state distribution trade-offs, mainly between
consistency and performance properties, are involved when
designing a distributed SDN controller platform.
Current implementations of distributed SDN controller plat-
forms [3, 4] use strong and/or eventual consistency models for
their internal and external SDN control applications. These
models have different advantages and drawbacks.
The Strong Consistency control model ensures a strongly-
consistent network state view to ensure the correct behavior of
the applications running on top of the distributed controllers.
However, it introduces controller-to-controller synchronization
overhead, and hinders response-time and throughput, thereby
impacting availability and performance as the network scales.
On the other hand, the Eventual Consistency control model
allows for higher availability, especially in the context of
large-scale distributed SDNs supporting scalable applications.
However, it may present a temporarily inconsistent network
view that leads to incorrect application behaviors, but that
becomes eventually consistent throughout the network.
In this paper, we introduce the use of an adaptive multi-
level consistency model following the continuous consistency
concept  in the context of distributed SDN controllers. That
model presents many advantages when compared to the strong
consistency and optimistic (eventual) consistency extremes,
especially in large-scale deployments. Our consistency adapta-
tion strategy was implemented for the distributed open-source
ONOS controllers; It mainly consists in continuously tuning
the application’s consistency level (at run-time) based on the
observed state inconsistencies with respect to the application
requirements (SLAs) under changing network conditions.
The rest of this paper is organized as follows: In Section
II, we provide an overview of the consistency models used
by state-of-the-art SDN controller platforms. In Section III,
we review the consistency problem in SDN and investigate
the involved consistency trade-offs. In Section IV, we discuss
the consistency models implemented in the ONOS controller
platform. In Section V, we present our consistency adapta-
tion approach for the distributed ONOS controllers. Finally,
Section V shows and discusses the experimentation results.
II. RE LATE D WORK
The challenges related to consistency in distributed SDN
control have been recently addressed in the SDN literature.
Some works focused on the impact of switch-to-controller
state consistency between switches and controllers on net-
work application performance. Reitblatt et al.  studied
the consistency of controller-driven ﬂow updates in terms of
network policy conservation. They proposed a new type of
consistency abstractions to enforce a consistent forwarding
state at different levels (per-ﬂow consistency and per-packet
consistency). Recent approaches  focused on efﬁciently
updating the network data plane state while preventing for-
warding anomalies at the switches and maintaining desired
consistency properties (e.g. loop and black-hole freedom).
Another category of works, falling within the scope of
this paper, focused on achieving controller-to-controller state
consistency between the distributed controllers without com-
promising application performance.
Current implementations of distributed SDN controller plat-
forms offer different state consistency abstractions. They use
static mono-level  or multi-level  consistency models
such as the strong, eventual and weak state consistency levels.
Onix  offers two separate dissemination mechanisms for
synchronizing network state updates between the NIBs stored
at the controller instances. These mechanisms are based on
two implemented data-store options: A replicated transactional
database designed for ensuring strong consistency at the cost
of good performance for persistent but slowly-changing states,
and a high-performance memory-only distributed hash table
(DHT) for volatile states that are tolerant to inconsistency.
Similarly, ONOS  provides different state sharing mech-
anisms to achieve a consistent network state across the cluster
of ONOS controllers. More speciﬁcally, ONOS’s distributed
core eases the state management and coordination tasks for
application developers by providing them with an available
set of core building blocks for dealing with different types of
distributed control plane states, including a consistent primi-
tive for state requiring strong consistency and an eventually
consistent primitive for state tolerating relaxed consistency.
On the other hand, ODL  supports a strong consistency
model in its distributed datastore architecture. In fact, all the
data shared across the cluster of controllers for maintaining the
logically centralized network view is handled in a strongly-
consistent manner using the RAFT consensus algorithm .
Recent approaches to handling the issues of controller
state consistency [10, 11] recommended the use of adaptive
consistency for the distributed SDN controller platforms.
Aslan et al.  attempted to mitigate the impact of controller
state distribution on SDN application performance by propos-
ing an adaptive tunable consistency model following the delta
consistency model. In their model, the automatic control plane
adaptation module tunes the consistency level (the synchro-
nization period parameter) based on an application-speciﬁc
performance indicator that is measured given the current state
of the network. To assess their approach, the authors compared
the performance of the distributed load-balancing application
running on top of adaptive and non-adaptive controllers.
In the same spirit, the work described in  put forward
an adaptive consistency model for distributed SDN controllers
following the Eventual Consistency level. The main aim of
changing the controller consistency level on-the-ﬂy was to
maintain a scalable system that sacriﬁces application optimal-
ity for less synchronization overhead. Accordingly, the authors
propose a cost-based approach that bounds the correctness to a
tunable threshold, where the consistency level is adapted based
on the effort of state convergence after the expiration of a non-
synchronization period, and the application inefﬁciencies due
to operations with stale state. The performance of the proposed
model was evaluated based on a speciﬁc routing application.
III. THE CONSISTENCY PRO BL EM I N SDN
A. Consistency trade-offs in SDN
In distributed SDN architectures, the SDN control plane
supports the interaction between multiple controllers through
their ”east-west” interfaces. Inter-controller communications
are indeed needed to synchronize the controllers’ shared
data structures to maintain a consistent global network view,
and therefore ensure the correct behavior of the network
applications running on top of the distributed controllers. Such
control trafﬁc can be in-band or out-band.
However, distributing the network control state across the
SDN control plane affects the performance objectives of the
control applications. In fact, many state distribution trade-offs
arise as discussed by Levin et al. such as the trade-off between
application state consistency/staleness (state synchronization
overhead) and application performance (objective optimality),
and the trade-off between application logic complexity and
robustness to inconsistency.
More generally, Brewer’s CAP theorem applied to networks
 investigates the involved trade-offs between Consistency
(C), Availability (A), and Partition-tolerance (P). It states that,
in SDN networks, it is generally only possible to achieve two
out of the three desirable properties: CA, CP or AP.
However, in the context of modern and scalable distributed
database systems (DDBS), Abadi’s PACELC theorem 
is believed to be more relevant, as it combines in a single
complete formulation, the CAP theorem trade-offs, and in the
absence of partitions (E), the Latency (L)/Consistency (C)
trade-off. Many popular modern DDBSs do not by default
guarantee strong consistency, as stated by CAP. Conversely,
they come with trade-offs that are better represented by the
PACELC alternative. For example, Amazon’s Dynamo ,
Facebook’s Cassandra , and Riak are PA/EL systems,
MongoDB is PA/EC, yahoo’s PNUTS is PC/EL, and ﬁnally
BigTable and HBase are PC/EC systems.
In this context, we argue that SDN is bringing the network
design much closer to the design of distributed database
systems. In the same spirit, we argue that PACELC can apply
to large-scale SDN controller platforms, in the same way it
applies to modern and scalable NoSQL DDBSs.
B. Consistency Models in SDN
Many architectures have been proposed to support dis-
tributed SDN controllers, with the goal of improving the
scalability, reliability and performance of SDNs. Two main
consistency models are used by current controller platforms:
1) The Strong Consistency Model: In SDN, the strong
consistency model guarantees that all controller replicas in the
cluster have access to the most updated network information
at all times. That comes at the cost of increased state syn-
chronization delay and communication overhead, especially
in large-scale deployments. In fact, strong consistency relies
on a blocking synchronization process that keeps the switches
from reading the data, unless the controllers are fully updated,
thereby affecting network availability and scalability.
Strong consistency is a requirement for certain applications
that favor consistency and correctness properties over avail-
ability. In current controller platforms, strong consistency is
usually achieved using Paxos, RAFT  and similar protocols.
2) The Eventual Consistency Model: In SDN, the eventual
consistency model takes a relaxed approach to consistency
by assuming that all controller replicas will ”eventually”
converge and become consistent throughout the network. That
means that controllers may temporarily present an inconsistent
network view, allowing for some stale data to be read, and
potentially causing a transient incorrect application behavior.
Many applications opt for eventual consistency to guarantee
high-availability and performance at scale. Modern DDBSs,
including Dynamo  and Cassandra , support eventual
consistency settings by default in exchange for extremely high
availability (fast data access) and scalability.
3) Adaptive Consistency Models: Recent research in SDN
[10, 11] has introduced the concept of adaptive consistency
in the context of distributed SDN control. Unlike static
consistency approaches, adaptively-consistent controllers ad-
just their consistency level at run-time to reach the desired
application performance and consistency requirements. That
alternative offers many beneﬁts: It spares application designers
the task of developing complex applications that require im-
plementing multiple consistency models, it provides applica-
tions with robustness against sudden network conditions, and
it reduces the overhead of state distribution across controllers
without compromising application performance .
In the community of modern database systems, the need for
adaptable consistency where the consistency level is decided
dynamically over-time based on various factors has been
recognized. Many adaptive consistency models have been
proposed, such as the QUOROM-based consistency ,
RedBlue Consistency , Chameleon and Harmony , the
delta consistency , and the continuous consistency . In
addition, most of modern database systems such as Cassandra
 and Dynamo  are currently equipped with an adaptive
consistency feature, offering multiple consistency options with
tunable parameters for application developers.
In our opinion, all the above consistency models could
be leveraged by the SDN community to build adaptively-
consistent SDN controllers.
IV. CONSISTENCY MODELS IN ONOS
In this paper, we are particularly interested in the open-
source Java-based ONOS controller . In this section, we
describe in detail the ONOS approach to state consistency in
a distributed controller setting.
To achieve high-availability, scale-out and performance, the
ONOS controller platform supports a physically-distributed
cluster-based control plane architecture, where each controller
is responsible for handling the state of a subsection of the
network. To maintain the logically-centralized network view,
local controller state information is disseminated across the
cluster in the form of events that are shared via ONOS’s dis-
tributed core. The latter consists of core subsystems tracking
different types of network states being stored in distributed
data structures and requiring different coordination strategies.
Two main state consistency schemes are implemented in
ONOS’s subsystem stores to provide two different levels of
state consistency: strong consistency and eventual consistency.
A. Strong Consistency in ONOS
To ensure strong consistency among replicated network
states, ONOS uses (since version 1.4) the Atomix framework
that is based on the RAFT consensus protocol . For
instance, the store for switch-to-controller mastership (map-
ping) management is handled in a strongly consistent manner
using that framework. Besides, ONOS exposes a set of core
distributed state management primitives that can be leveraged
by application developers to implement their application-
speciﬁc stores. In this respect, applications whose state is
maintained in a strongly consistent fashion can leverage
the ConsistentMap distributed primitive, which guarantees
strong consistency for a distributed key-value store.
B. Eventual Consistency in ONOS
For eventually-consistent behaviors, ONOS employs an op-
timistic replication technique complemented by a background
gossip/anti-entropy protocol. For instance, the stores for de-
vices, links, and hosts are managed in an eventually-consistent
manner. The distributed topology store is also eventually con-
sistent since it relies on the distributed versions of the device,
link and host stores. For the eventual consistency option,
ONOS offers the EventuallyConsistentMap distributed
primitive for control programs and applications. The latter
can create different instances of these primitives for managing
their eventually-consistent application-speciﬁc states.
(a) Optimistic Replication:
Optimistic replication is a key technology that is used in
large-scale distributed data sharing systems, meeting the goal
of achieving higher availability and scalability as compared
to strongly-consistent systems. That strategy for replication
propagates changes in the background, and discovers conﬂicts
after they occur. It is based on the ”optimistic” assumption that
inconsistencies rarely occur and that replicas will converge af-
ter some time, thus providing eventual consistency guarantees.
In ONOS, optimistic replication is used in the distributed
maps. Whenever an update occurs in the store managed by
one controller, the associated EventuallyConsistentMap
replicates events immediately to the rest of the controllers.
That means that maps on each controller will get closely in
sync (apart from a small propagation delay) in case the con-
trollers are functioning properly. On each controller, updates
are added to an EventAccumulator as they are written.
(b) Gossip-based Anti-Entropy:
Controllers that purely rely on optimistic replication might
progressively get out of sync, especially in the event of node
failures and partitions or in the case updates get missed or
dropped. The anti-entropy protocol takes care of ensuring that
replicas are back in sync by resolving discrepancies, and that
the entire cluster converges fairly quickly to the same state.
In ONOS, the gossip-based anti-entropy mechanism is a
lightweight peer-to-peer background process that runs pe-
riodically: At ﬁxed intervals (3-5 seconds), each controller
randomly chooses another controller, they both exchange
information in order to compare the actual content (entries) of
their distributed stores (based on timestamps). After synchro-
nizing their respective topology views, the controllers become
mutually consistent. This reconciliation approach proves use-
ful in ﬁxing controllers when their state drifts slightly, and
also in quickly synchronizing a newly-joining controller with
the rest of the controllers in the cluster.
V. TH E PROPOSED ADAP TI VE CONSISTENCY FOR ONOS
In this section, we explain our approach which is mainly
aimed at optimizing the consistency management in ONOS.
A. A Continuous Consistency model for ONOS
As explained in III-B3, an adaptive consistency model
offers many beneﬁts for the distributed SDN controllers. In
particular, the ONOS controllers can beneﬁt from the contin-
uous consistency model proposed in . The latter is based
on a middle-ware framework (called TACT) for adaptively
tuning the consistency and availability requirements for repli-
cated online services, following the continuous consistency
concept. In contrast to the strong consistency model (which
imposes performance overheads and limits availability), and
to optimistic consistency models (which provide no bounds
on system inconsistencies), the continuous consistency model
explores the semantic space between these two types of
traditional models: It offers a continuum of intermediate
consistency models (multi-level consistency) with tunable pa-
rameters. These quantiﬁable degrees of consistency can be ex-
ploited by applications to explore, at runtime, their own trade-
offs between consistency and availability, while taking into
account the changing network and service conditions. More
speciﬁcally, TACT bounds the amount of inconsistency and
divergence among the replicas in an application-speciﬁc man-
ner. Basically, applications specify their consistency semantics
through conits; a set of metrics that capture the consistency
spectrum: Numerical Error,Order Error,Staleness.
Hence, for each conit, consistency is quantiﬁed continu-
ously along a three-dimensional vector:
Consistency = (N umericalE rror, OrderE rror, Staleness)
Numerical Order bounds the discrepancy between the value
delivered to the application client and the most consistent ”ﬁ-
nal” value. Order Error bounds inconsistency by the number
of tentative/unseen writes at any replica. Staleness places a
real-time bound on the delay for propagating the writes among
In our opinion, many features from the discussed contin-
uous consistency spectrum can indeed be incorporated when
rethinking the ONOS strategy to state consistency, especially
in the context of large-scale deployments.
Fig. 1: The proposed Adaptive Consistency Strategy
B. Our Consistency Adaptation Strategy for ONOS
We propose to maintain the strong consistency model imple-
mented in ONOS for applications requiring strict consistency
guarantees, but we suggest turning the eventual consistency
model into an adaptive tunable model following the concept
of continuous consistency in order to explore the availability,
consistency and scalability beneﬁts of such a model.
With this in mind, we adopt the following strategy when
reviewing the eventual consistency model:
- We keep the current implementation of the optimistic
replication technique used for replicating events and updates
among controllers in the cluster.
- We bring signiﬁcant changes to the anti-entropy pro-
tocol used for eventual consistency in ONOS: Instead of
running the anti-entropy process for each controller replica
periodically at ﬁxed intervals (each 3-5 seconds) (voluntary
anti-entropy) causing too much overhead and affecting sys-
tem scalability and performance, we argue that the anti-
entropy process should only be scheduled when the system
consistency is at risk (compulsory anti-entropy). In other
words, the choice of the anti-entropy reconciliation period
for each controller replica (per-replica consistency) should be
based on the correctness of the system with respect to the
consistency requirements expressed by the applications. That
way, at each controller replica and for each application state,
the consistency level is dynamically adapted based on the
computed values of the consistency metrics (see Equation (1))
capturing the application’s consistency semantics with respect
to the given thresholds set in advance by the application.
C. Our Implementation Approach
To implement our state consistency adaptation approach, we
consider a replicated source routing SDN application running
on top of a cluster of distributed ONOS controllers. The cost-
based source routing application operates on a distributed
topology graph for computing the shortest-path costs between
source and destination hosts. Since the topology graph state is
handled in an eventually-consistent manner, the application’s
state is also considered as eventually-consistent.
In our routing application f, the path between the source
host Aand the destination host B(see Figure 1) may be
deﬁned as a conit. Besides, we argue that an important
consistency requirement for our control application is the
result optimality of the instant path computation cost (in
terms of hop-count in our case) which is captured by the
Numerical Error metric. The numerical error of our conit
Ccan be deﬁned as relative difference between the value
of the ”shortest-path” cost xlocal as perceived by a local
replica, and its ”ﬁnal ”optimal” value xoptimal at a replica
that has reached some ”ﬁnal” consistent state. That error is
continuously bounded at run-time using an application-deﬁned
threshold T(f)(in percentage) as follows:
NumericalError(Cf) = ( |xlocal −xoptimal |
)< T (f)(2)
Finally, it is worth noting that other consistency semantics
for the source routing application might be expressed using
Staleness and Order Error.
To implement our approach, we introduce some modiﬁ-
cations to the ONOS Java source code. We start by de-
veloping our adaptive source routing application (similar to
the Intent Forwarding Application) for computing the
shortest-path cost between host Aand host B. Running on
each ONOS instance, our application gets information from
the in-memory topology cache maintained by each ONOS
instance (DistributedTopologyStore). Whenever an up-
date occurs in the topology graph (e.g. links/devices failing
or joining), our application detects that topology change
and updates the ”local” shortest-path cost between host A
and Baccordingly. We also modify the implementation of
the EventuallyConsistentMap distributed primitive, espe-
cially for the eventually-consistent stores that feed the topol-
ogy store (e.g. link and device stores). We indeed focus on the
BackgroundExecutor service of the eventually-consistent
maps, which runs the background anti-entropy tasks, and we
propose a new implementation of the Runnable interface used
for executing the scheduled anti-entropy thread.
In fact, instead of sending the anti-entropy advertisement
messages periodically each 3-5 seconds between the con-
trollers, as it is the case in ONOS, we propose to run, at each
replica, a periodic check on the consistency of other replicas
with respect to the application’s shortest-path cost state, by
computing at run-time the relative Numerical Error deﬁned
in Equation (2). In fact, in the event of a controller failure,
the rest of the controllers in the cluster that detect that failure,
keep a screen-shot of their own topology graph at the moment
of the failure. During the periodic consistency check, they use
that stored topology graph to estimate the inconsistency of
the failed controller; which is equal to the relative difference
between the ”local” shortest-path cost (computed based on
the current topology graph state) and the shortest-path cost as
perceived by the controller after recovery (computed based on
the stale topology graph state being stored).
When the failed controller recovers, the rest of the con-
trollers make an anti-entropy decision based on the checked
numerical error. If the error exceeds an ”alarming” consistency
threshold set in-advance by the application, then, an anti-
entropy process is launched to ﬁx the failed controller state.
That is achieved by synchronizing the controllers’ eventually-
consistent stores that feed the topology view. In the opposite
case, the inconsistency is considered as tolerated by the
application, and an anti-entropy session might be scheduled
afterwards in case the controller state signiﬁcantly drifts away.
VI. PE RF OR MA NC E EVALUATI ON
A. Experimental Setup
Our experiments are performed on an Ubuntu 16.04 LTS
server using ONOS 1.13. We also use Mininet 2.2.1 and an
ONOS-provided script (onos.py) to start an emulated ONOS
network in a single machine; including a logically-centralized
ONOS cluster, a modeled control network and a data network.
Wireshark is used as a sniffer to capture the inter-controller
trafﬁc which uses TCP port 9876.
To validate our proposed approach on ONOS, which we will
refer to as ONOS-WAC (ONOS-With Adaptive Consistency),
we have considered many test scenarios. In each scenario, we
run a cluster of NONOS controller instances, controlling a
Mininet network topology of Sswitches (see Table I).
Test N S F Controller
scenarios Controllers Switches Failure scenarios
no1 3 16 2
no2 5 36 3
no3 7 64 4
no4 9 100 4
no5 10 121 5
TABLE I: Test Scenarios
In order to create state inconsistencies among the controller
instances in the cluster with respect to the shortest-path cost
state of our source routing application, we create different
controller failure scenarios F. Shortly after a controller failure
(in F) in a speciﬁc scenario Si, we consider changing the
network topology by taking down network switches and links
along the shortest-path (computed by the application) between
source host Aand destination host B. That way, after recovery,
the controller will have an inconsistent network topology view
as compared to the rest of the controllers in the cluster.
According to our proposed approach, that inconsistency in the
network topology view affects the optimality of the shortest-
path cost computation performed by the source routing appli-
cation instance running on top of the recovered controller. The
induced Numerical Error is likely to trigger a synchronization
process achieved by anti-entropy tasks (see Section V-C).
In Scenario no1 (Scenario with three controllers as shown
in Table I), we adopt the methodology described in VI-A. In
Figure 2, we show the inter-controller trafﬁc captured during
the test scenario period in an ONOS cluster (Figure 2(a))
and in an ONOS-WAC cluster (Figure 2(b)). After running
the Mininet topology according to Scenario no1, the same
event sequence is performed for both ONOS and ONOS-WAC
clusters. For instance, the ﬁrst trafﬁc peak in both ﬁgures (at
t= 90s) corresponds to a ”pingall” Mininet CLI command
executed for topology discovery. At t= 150s, we simulate
a failure scenario by taking down one controller instance.
That action is followed by other topology changes (e.g. links
down) corresponding to the subsequent peaks in both ﬁgures.
At t= 180s, we bring back the failed controller, resulting
in a trafﬁc peak that appears to be more signiﬁcant in the
case of ONOS-WAC. That increase in trafﬁc is due to the
anti-entropy process which has been triggered by an inconsis-
tency (Numerical Error) value that exceeded the application
threshold. Conversely, in the ONOS network, the anti-entropy
trafﬁc is generated periodically over the test period regardless
of the observed inconsistencies. Likewise, at t= 280s, we
repeat the same scenario following the same event sequence,
but considering the failure of a different controller.
In the test scenario described above, the application in-
consistency threshold was set to 0%, triggering the start of
anti-entropy sessions for any observed inconsistencies in the
considered application state. In the following test experiments,
we repeat the same scenario (Scenario no1), but we con-
sider varying the source routing application’s inconsistency
Fig. 2: Scenario no1: Captured Inter-controller trafﬁc (in packets per
second) during the test scenario period (using Wireshark)
threshold. As shown in Figure 3, ONOS shows an aver-
age inter-controller overhead equal to 315kbps regardless of
the application inconsistency threshold. On the other hand,
ONOS-WAC shows relatively less inter-controller overhead
(due to low anti-entropy overhead). The latter is impacted
by the application’s consistency requirements. For example,
in the case of strict consistency requirements (application
threshold between 0% and 30%), inconsistencies occurring in
the application state are more likely to trigger the anti-entropy
reconciliation sessions causing much more overhead, when
compared to the case of less strict consistency requirements
(application threshold between 40% and 50%).
Fig. 3: Scenario no1: Inter-controller Overhead in ONOS and ONOS-
WAC according to the Application Threshold
In order to assess the gain in anti-entropy overhead of
our adaptive consistency model implemented on ONOS, as
compared to the eventual consistency model of ONOS, we
consider estimating the rate (R(Si)) of increase in anti-entropy
overhead of ONOS with respect to ONOS-WAC (see Equation
(3)) as a function of the number of controllers in the cluster
(following Scenarios no1, no2, no3, no4, no5) (see Figure 4).
Ri(Si) = 1 −[A(Si)−B(Si)
-A(Si): the inter-controller overhead generated by ONOS-
WAC after the event sequence (VI-B) following Scenario Si.
-B(Si): the inter-controller overhead generated by ONOS-
WAC before the event sequence (the overall inter-controller
trafﬁc without the anti-entropy trafﬁc).
-C(Si): the inter-controller overhead generated by ONOS
after the event sequence (VI-B) following Scenario Si.
As shown in Figure 4, the gain in anti-entropy overhead,
when adopting ONOS-WAC, grows almost linearly with the
number of controllers in the cluster. For example, in Scenario
no5 (corresponding to 10 controllers in the network cluster),
the gain in anti-entropy overhead has reached 25%.
Fig. 4: Gain in Anti-Entropy Overhead of ONOS-WAC with respect
to ONOS according to the number of Controllers in the cluster
In this paper, we investigated the use of adaptive
consistency for the distributed ONOS controllers. Our ap-
proach was aimed at turning the eventual consistency model
of ONOS into an adaptable multi-level consistency model
following the concept of continuous consistency. The latter
delivers the performance and availability beneﬁts of an even-
tual consistency model, but has the additional advantage of
controlling the state inconsistencies in an application-speciﬁc
manner. Our consistency adaptation strategy was implemented
for a source routing application on top of ONOS. Besides en-
suring the application’s state consistency requirements (speci-
ﬁed in the given SLA), our results showed a substantial reduc-
tion in the Anti-Entropy reconciliation overhead, especially
in the context of large-scale networks. As a future work,
we consider extending our adaptive consistency approach to
the optimistic replication technique used in ONOS’s eventual
consistency model by leveraging multiple replication degrees
as well as the geo-placement of the controller replicas.
Although the main focus of this work was placed at dy-
namically adjusting the consistency level of application states
(which use controller states), we plan to extend our approach
to the controller states (internal controller applications). In-
deed, the long-term goal of this work is to design adaptively-
consistent controllers that adjust both control and application
plane consistency levels under changing network conditions.
 F. Bannour, S. Souihi, and A. Mellouk, “Distributed
SDN control: Survey, taxonomy, and challenges,” IEEE
Communications Surveys Tutorials, vol. 20, no. 1, pp.
333–354, Firstquarter 2018.
 A. Panda, C. Scott, A. Ghodsi, T. Koponen, and
S. Shenker, “Cap for networks,” in Proceedings of the
Second ACM SIGCOMM Workshop on Hot Topics in
Software Deﬁned Networking, 2013, pp. 91–96.
 ONOS. [Online]. Available: https://onosproject.org/
 ODL. [Online]. Available: http://opendaylight.org/
 H. Yu and A. Vahdat, “Design and evaluation of a
continuous consistency model for replicated services,”
in Proceedings of the 4th Conference on Symposium on
Operating System Design & Implementation - Volume 4,
ser. OSDI’00, Berkeley, CA, USA, 2000.
 M. Reitblatt, N. Foster, J. Rexford, and D. Walker, “Con-
sistent updates for software-deﬁned networks: Change
you can believe in!” in Proceedings of the 10th ACM
Workshop on Hot Topics in Networks, 2011, pp. 7:1–7:6.
 T. D. Nguyen, M. Chiesa, and M. Canini, “Decentral-
ized consistent updates in sdn,” in Proceedings of the
Symposium on SDN Research, 2017, pp. 21–33.
 T. Koponen, M. Casado, N. Gude, J. Stribling,
L. Poutievski, M. Z. Google, R. Ramanathan, Y. I.
NEC, H. I. NEC, T. H. NEC, and S. Shenker, “Onix:
a distributed control platform for large-scale production
networks,” in 9th Conference on Operating Systems
Design and Implementation, 2010, pp. 351–364.
 D. Ongaro and J. Ousterhout, “In search of an under-
standable consensus algorithm,” in 2014 USENIX Annual
Technical Conference (USENIX ATC 14), 2014.
 M. Aslan and A. Matrawy, “Adaptive consistency for
distributed sdn controllers,” in 2016 17th International
Telecommunications Network Strategy and Planning
Symposium (Networks), Sept 2016, pp. 150–157.
 E. Sakic, F. Sardis, J. W. Guck, and W. Kellerer,
“Towards adaptive state consistency in distributed sdn
control plane,” in 2017 IEEE International Conference
on Communications (ICC), May 2017, pp. 1–7.
 D. Abadi, “Consistency tradeoffs in modern distributed
database system design: CAP is only part of the story,”
Computer, vol. 45, no. 2, pp. 37–42, Feb 2012.
 S. Sivasubramanian, “Amazon dynamodb: A seamlessly
scalable non-relational database service,” in Proceedings
of the 2012 ACM SIGMOD International Conference on
Management of Data, ser. SIGMOD ’12. New York,
NY, USA: ACM, 2012, pp. 729–730.
 A. Lakshman and P. Malik, “Cassandra: A decentralized
structured storage system,” SIGOPS Oper. Syst. Rev.,
vol. 44, no. 2, pp. 35–40, Apr. 2010.
 S. P. Kumar, “Adaptive Consistency Protocols for Repli-
cated Data in Modern Storage Systems with a High
Degree of Elasticity,” Theses, CNAM, Mar. 2016.
 C. Li, D. Porto, A. Clement, J. Gehrke, N. Preguic¸a, and
R. Rodrigues, “Making geo-replicated systems fast as
possible, consistent when necessary,” in Conference on
Operating Systems Design and Implementation, 2012.
 H.-E. Chihoub, M. P´
erez, G. Antoniu, and L. Boug´
“Chameleon:customized application-speciﬁc consistency
by means of behavior modeling,” Research Report, 2013.