Towards evaluating emergent behavior of the
Internet of Things using large scale simulation
Stig Bosmans1, Siegfried Mercelis1, Peter Hellinckx1and Joachim Denil2
University of Antwerp
February 27, 2018
With the increase in Internet of Things devices and more decentral-
ized architectures we see a new type of application gain importance,
a type where local interactions between individual entities lead to a
global emergent behavior, Emergent-based IoT (EBI) Systems. In this
position paper we explore techniques to evaluate this emergent behav-
ior in IoT applications. Because of the required scale and diversity this
is not an easy task. Therefore, we mainly focus on a distributed sim-
ulation approach and provide an overview of possible techniques that
could optimize the overall simulation performance. Our focus is both
on modeling and simulation technology.
arXiv:1802.09207v1 [cs.DC] 26 Feb 2018
In recent years, there has been a signiﬁcant increase in the number of internet-
connected devices. By 2020 it is expected that up to 50 billion devices will be
connected to the internet. This Internet of Things (IoT) will create novel ser-
vices that will highly impact our way of life in areas such as transportation,
shopping, health and safety. With IoT becoming more mainstream and with
the rise in the amount of devices getting interconnected, the complexity and
scale of the IoT landscape will largely increase. The interoperability between
IoT devices and actuators of all sorts will prove to be vital for future IoT
applications. As a result of the increasing scale and diversity and because
of new IoT architectures such as edge computing, the concept of emergent
behavior in IoT systems is gaining more attention within the IoT commu-
nity. Mataric et. al.  deﬁnes emergent behavior as a collection of actions
and patterns that result from local interactions between elements and their
environment which have not been explicitly programmed. It is loosely based
on the emergent behavior that is observed in bird ﬂocks where birds apply
three local rules which result in emergent ﬂocking behavior (e.g. remain x
distance to neighbor birds). D. Roca et. al.  argues that this emergent
behavior will lead to improved scalability, interoperability and cost eﬃciency
of ultra large scale IoT systems as opposed to traditional approaches that
heavily rely on extensive programming of explicit behaviors. This approach
particularly beneﬁts the IoT areas which require the interaction of an enor-
mous amount of devices where relying solely on a centralized architecture is
insuﬃcient, such as smart power grids, autonomous car ﬂocking and smart
traﬃc lights. We will refer to this type of systems as emergent based IoT
(EBI) systems in the remainder of this paper. Building such emergent-based
IoT systems is complex as the system is composed from diﬀerent heteroge-
neous and autonomous components. Examples of such components can range
from actual IoT sensors sharing observations, towards smart actuators such
as smart traﬃc lights that make decisions based on interactions with their
counterparts and by observing sensor data. However, a human actor is also
a component in an IoT system. From one perspective a human actor can be
seen as a simple data generator, e.g. by walking around with a GPS-sensor
embedded in their smartphone, they could broadcast location-data to an IoT
middleware. From another perspective they can play a very active role in
the IoT system by for example generating evolving traﬃc patterns which will
inﬂuence the behavior of smart traﬃc lights. All these various components
and their individual behavior, their interactions and their goals need to be
considered and evaluated when developing an EBI system. We believe that
EBI systems will be the standard for future decentralized large scale IoT
applications. Because of the complexity, the large-scale and decentralized
property of these EBI systems and their components, testing, validation and
calibration will be vital in successfully developing such systems. In practice,
IoT tests are either performed in small-scale/lab-based prototype setups or
based on software testing approaches. The limited scaling capabilities of
these testing approaches make it very hard to properly test EBI systems as a
whole. EBI systems introduce a new component to test, being the global vs
local behavior. Traditional testing techniques would not be able to test this
global behavior at scale. Furthermore, the testing of the emergent behavior
of IoT has been largely ignored in the literature. In this position paper, we
investigate the challenges in testing EBI and look for solutions, methods and
techniques mainly based on large-scale and distributed simulation based test-
ing of IoT. The rest of this paper is organized as follows. Section 2 positions
the challenges and needs in testing EBI systems, the next section provides an
overview of possible techniques to improve performance and overall simula-
tion scalability, in section 4 we further investigate state-of-the-art techniques
that could be used in the scope of simulating EBI systems. Section 6 presents
a discussion of the remaining challenges of real-time testing of IoT and how
the presented techniques could help to solve these. Finally, in section 7 we
conclude our work.
2 Challenges in evaluating emergent behav-
In the context of this paper, we are only interested in system-level evaluation
of the EBI. This means that we are interested in observing the overall, global
behavior posed by the application and we are unaware of the technical im-
plementation details such as network and the hardware and software of the
diﬀerent sensor nodes.
We distinguish three diﬀerent scenarios where evaluating the emergence
of behavior is necessary:
•Validation of emergent behavior: We want to make sure that the
local behavior and rules imposed by the application lead to the pre-
ferred emergent behavior.
•Calibration of emergent behavior: Calibration of the application
needs to be done by tuning the various parameters in the IoT applica-
tion, e.g. the value of an incentive. Therefore, an optimization of emer-
gent behavior has to be done by monitoring the impact the local rules
and incentives have until the global, emergent behavior converges to a
desired behavior. In other words, design-space exploration is needed to
obtain optimized parameters for the simulation.
•Integration testing and deployment: Integrating the EBI applica-
tions in the real-world has a signiﬁcant impact and therefore requires
an incremental means to accept and deploy the application. Ideally,
this translates to running the simulation models in parallel with the
IoT middleware and the actual nodes operating in the real world. The
number of nodes can be increased gradually to fully deploy the ap-
plication in the ﬁeld. For example, in a ﬁrst integration test, a user
could directly interact with a real prototype and with the simulated
environment. Further tests increase the number of users in a local
The testing of EBI systems for each of these scenarios poses signiﬁcant
challenges because of the scale and diversity of these systems. We diﬀerenti-
ate between three major methods that are relevant in this work:
1. Real-world evaluation: This method relies on deploying the system
in a real environment. Often the scale of such tests are limited to small
lab-based setups. This has as an advantage, that the tests are easy to
deploy. However in practice, this type of testing should only be used
for testing the behavior of a single IoT device. Formulating reliable
conclusions about the validity of the behavior of the IoT system as
a whole is unfeasible. Instead, there are a few large-scale, real-life
IoT testbeds which allow tests to be executed at a more appropriate
size. An example of such a testbed is the Smart Santander projects
. It is a city-wide, real IoT testbed. It oﬀers many thousands of
IoT devices and interconnected gateways. It oﬀers great opportunities
to eﬀectively test certain aspects of IoT devices deployed in a real
environment. However, many EBI systems, such as the smart traﬃc
light example, rely on custom devices and applications which would be
too costly and too diﬃcult to include in existing testbeds.
2. Simulation-based evaluation: A simulated environment consisting
of a wide range of virtual IoT devices interacting with each-other and
with the IoT middleware under test can be seen as a very eﬀective test-
ing solution. Although real testing is often desirable, simulation testing
is a more ﬂexible and cost-eﬃcient approach. It allows for a more con-
trolled and reusable environment that can be tweaked much easier.
Within the IoT domain simulation testing is most often used to test
technical aspects of the system such as network related features, power
consumption etc. The most well-known examples of such simulators
are NS-3  and Omnet++ . Various IoT operating systems such as
Contiki and TinyOs also oﬀer their dedicated simulator, Contiki 
and Tossim  respectively. Some simulators are focused on testing
more high-level IoT setups such as the iFogSim  which is used to test
IoT edge or fog architectures. Another example is the CupCarbon sim-
ulator  which is used to simulate Smart City environments. Finally,
D’Angelo et. al. demonstrate the use of the Gaia/Artis speciﬁcally to
run large-scale IoT simulations . However, most of the state-of-the-
art IoT simulators do not have the necessary scalability requirements
(more than 100.000 simulation entities) to simulate such large-scale IoT
environments and to speciﬁcally evaluate emergent behavior.
3. Hybrid evaluation: In order to facilitate the real-time testing ap-
proach explained above, a hybrid testing can be used which combines a
small-scale real life environment with a virtual simulated environment.
This way, actual IoT prototypes can interact with virtual devices, oper-
ating both as a single system able to generate emergent behavior. This
could be particularly interesting for pilot projects or proof of concepts,
where the scale is still too limited in order for emergent behavior to
arise. Furthermore, it combines the advantages of both real-life testing
and simulation testing.
Relying solely on real-world evaluation environments such as the Smart
Santander project to test emergent behavior is costly and unpractical. Al-
though these testbeds oﬀer a number of ready-to-use sensors and IoT devices,
deploying an emergent-based application on such an environment requires a
signiﬁcant amount of costly changes. Each EBI system will need a number
of specialized sensors and actuators that need to be deployed, installed and
linked to the available smart city infrastructure. Making such investments is
impractical at the early stages of a project. Therefore, simulation or hybrid
approach have a much smaller threshold to use.
However, our observations detailed above show that the major challenge
lies in the modeling of the global behavior and the scalability of the simulation
framework. Furthermore, in all the diﬀerent scenarios, there are diﬀerent
requirements on the accuracy of the simulation. On the one hand, validating
the behavior of the EBI requires accurate results on the global behavior
of the system. On the other hand, during calibration the parameter space
of the application is explored with less accurate simulations to ﬁnd good
conﬁgurations of the application which are validated afterwards with detailed
The main challenge in simulating emergent behavior is related to perfor-
mance. In order to obtain some level of emergent behavior a great amount
of interacting entities is required. Running such a simulation on a single
device wouldn’t be possible, instead distributing to multiple servers is a bet-
ter approach to cope with the necessary scale. Example implementations
are described in the Parallel and Distributed Simulation (PADS) methodol-
ogy  and in the IEEE 1516 standard. Facilitating the proper execution
of these simulation entities in a simulated environment still requires several
performance optimization techniques in the simulation kernel. Preferably,
these optimizations can be performed transparent to the simulation modeler.
However, this won’t be possible all the time. In this sections we provide an
overview of various optimization techniques that can be applied dynamically
or statically. A dynamic optimization can be applied at run-time and can be
context-dependent, while a static optimization needs to be applied up-front.
We made a break-out of major performance optimization techniques in ﬁgure
1 below. Note, that this list of techniques isn’t complete and is the current
result of on-going study.
4 Possible Solutions when Modeling EBI
An important aspect of generating emergent behavior is to properly describe
the behavior of the individual elements of the real-world within the simulation
Figure 1: Overview of simulation performance optimization techniques
model. Furthermore, once these systems are modeled we show which model-
ing techniques can help us in achieving a scalable EBI simulation framework.
4.1 Model Representation
In order to model emergent behavior the agent based modeling (ABM)
paradigm seems the most appropriate . With ABM, a bottom-up mod-
eling approach is applied. Instead of modeling the global expected behavior,
the modeler describes the behavior of the individuals. The ABM paradigm
allows these individuals to interact. Eventually, these interactions will lead
to a global behavior. As the EBI application are deployed in a similar mat-
ter, the simulation engineer has to model the diﬀerent entities involved in the
simulation. While this seems trivial, the individuals require detailed models
on their decision making processes.
Other formalisms can also be appropriate. Discrete-event formalisms such
as DEVS  model the behavior of a system using a timed sequence of
”events” either as input to a system or as a timeout within the system. These
events cause instantaneous changes to the state of the system. For example,
when simulating traﬃc behavior, agent based modeling might not be the
most appropriate formalism. Lots of research has been done in the domain
of queuing theory which might be a better, less computationally expensive
formalism. Atomic discrete event models can be combined together into a
couple model. Furthermore, extensions exist that couple these models in a
grid, e.g. Cell-DEVS . These types of cellular automata can have an
advantage to model geographical areas. E.g. the city is divided into a grid
of cells where each cell models a part of the city.
Finally, diﬀerential equations can help to model rates of changes of prop-
erties within the IoT setting. E.g. traﬃc ﬂows can be modeled with a
diﬀerential equation. Diﬀerent equations can also be coupled together us-
ing co-simulation techniques to allow for a divide-and-conquer approach to
model complex systems.
However, simulating such a large model of our IoT application is too com-
plex. Model abstraction techniques should decrease the computational cost
of individual models by simplifying the representation of such a model while
maintaining the original behavior as accurate as possible. As a result, the
overall computational complexity can be optimized. The main objective of
applying model abstraction is to make a proper trade-oﬀ between computa-
tional complexity and accuracy. Extensive taxonomies of model abstraction
techniques exist in literature [13, 5]. In this work we will only discuss the
most relevant techniques. As demonstrated in ﬁgure 1, we diﬀerentiate be-
tween two major categories, one based on multi-formalism model abstraction
and another based on single-formalism abstraction. We than discuss how we
can use these techniques in an IoT simulation environment.
4.2 Abstractions within a single formalism
These abstractions are applied on the model without changing the formalism
to describe it. We deﬁne three types of abstractions that can happen within
a single model. We will apply these types of abstractions on the individuals
of the ABM but they can also be applied to models in other formalisms as
State abstraction This technique abstracts the state of a single individ-
ual. As the decision processes of individuals can be very complex, certain
parts of the decision making process that an individual exhibits can be ig-
nored without compromising the global behavior of the simulation, e.g. the
individual also reasons about other properties that do not inﬂuence the global
behavior, or higher-order reasoning with little to no impact on the decision
of the agent.
The process of creating a more abstract individual from a detailed one can
be done manually. This requires insight into both the decision making pro-
cess of the individuals as into the application that generates the emergence of
behavior. Therefore, more automatic techniques might be more appropriate.
An example technique that can be used for this is metamodeling (as in surro-
gate modeling). Caughlin et. al. deﬁnes a metamodel as a projection of the
original, high-ﬁdelity model onto a subspace deﬁned by new constraints or
regions of interest . In practice, a metamodel is a mathematical approxi-
mation of a complex model. The original model is treated as a blackbox, and
the metamodel operates as a surrogate model that replaces the original. To
come to such a metamodel a detailed analysis of the input/output mapping
of the original model has to be performed. Based on this analysis a sur-
rogate metamodel should learn to represent a similar mapping. Preferably,
the surrogate model is more abstract and is less computationally complex.
Various methods have been used for the development of metamodels such as
polynomial regressions, radial basis functions (RBF) and others . Most
of these techniques try to approximate input functions or data. Since the
goal of metamodeling is to map an input value to a speciﬁc output value
we believe other supervised learning techniques could be used as well. For
example, P. Symonds et. al. demonstrate that neural networks (NN) could
perform up to 15% better compared to classic RBF approaches  . We
believe that evaluating various state-of-the-art deep neural networks could
lead to even better results. We consider this as a promising area for further
Entity Aggregation Another option to reduce the computational com-
plexity of simulation entities is Entity Aggregation. The idea of entity ag-
gregation is to combine multiple low-level simulation entities with a single
high-level entity while preserving the collective behavior. For example, when
simulating emergent behavior in a smart power grid application, a single
agent could represent a single household but depending on the required level
of accuracy a single agent could also represent an entire neighborhood. Ro-
driguez et. al. demonstrates that NN’s could be used to transform a collec-
tion of low-level entities to an aggregated model  . From an architectural
perspective, a mechanism in the simulation kernel can be created to detect
and analyze clusters of agents that interact a lot and pose homogeneous be-
havior. Using metamodeling techniques such a cluster of agents can then
be replaced by a single surrogate model. Conversely, the modeler can also
manually model an aggregation of individual simulation agents.
Temporal Abstraction Finally, temporal abstraction can be used to limit
the granularity of simulation entity state updates over time. Since, each state
update results in a cascade of simulation events the overall computational
cost could be signiﬁcantly reduced.
Engineers can themselves easily change the temporal granularity (time-
step) of their simulation. However, choosing a too large time-step can result
in an unstable simulation. Automatic techniques to adapt the time-step
also exist. E.g. for solving diﬀerential equations, several techniques are
available that automatically adapts the time-step of the simulation based on
the estimated error, e.g. Runge-Kutta 4-5 embeds a higher-order method in
its solver to estimate the error and adapt the time-step.
4.3 Multi-formalism Modeling
Changing formalisms during abstraction can help to achieve better perfor-
mance results as we can leverage the strengths of diﬀerent types of for-
malisms. For example, as explained in section 4.1, in the domain of traﬃc
simulation, queuing theory might be a more appropriate, less computation-
ally expensive formalism compared to agent based modeling. The formalism
change allows for an easier state and aggregation abstraction.The disadvan-
tage of applying multi-model abstraction is that this can’t be automated,
the modeler will be responsible to model the same behavior in diﬀerent for-
4.4 Multi-resolution Modeling
Multi-resolution modeling can be used to vary between model abstraction
levels (and possibly formalisms) within the same simulation. The advantage
of this is that we can increase abstraction levels in simulation areas where less
accuracy is required. This will consequently reduce the computational com-
plexity of the simulation. Oppositely we could also increase levels of detail
in certain simulation areas when an increased level of accuracy is required.
The multi-resolution model can be static or dynamic.
When applying a static approach, the most appropriate formalisms and
levels of detail have to be speciﬁed upfront and will be used during the entire
course of the simulation. Due to the dynamic nature of IoT applications,
the potential computational beneﬁt is limited. Conversely, with a dynamic
approach we can switch between various levels of abstraction during the
course of the simulation. This can be done by predeﬁning when and where
an abstraction switch should occur. Domain knowledge is necessary to detect
and abstract/reﬁne the (part of the) model.
Learning approaches are also possible to switch levels of abstraction, by
learning areas of opportunity and applying the necessary level of abstraction.
For this, an additional mechanisms in the simulation kernel are required that
detects areas of opportunity based on various parameters. For example, sim-
ulation entities that have a limited amount of interaction with the rest of
the simulation could be marked as candidates for an abstraction level in-
crease. The advantage of such an approach is that we can adaptively detect
opportunities to switch between abstraction levels in order to decrease nec-
essary computational resources or oppositely, use all available computational
Applying a dynamic multi-resolution modeling approach leads to addi-
tional challenges that need to be taken into account . For example, reini-
tializing a more abstract model from the current state of the original model
is possible as there is enough information available in the original. However,
when switching from a more abstract to a reﬁned model, extra information
is required. This should be mend by modeling additional domain-knowledge
to ﬁll this information gap.
5 Possible Solutions when Simulating EBI
In this section we will discuss techniques that could improve the performance
capabilities of the simulator itself. Most of these techniques have been studied
in various domains such as parallel and distributed simulation (PADS) and
distributed discrete event simulation.
5.1 Simulation Architecture
Most of the current IoT simulators have an underlying monolithic architec-
tures which limits its scalability capabilities. D’Angelo et. al. recognizes
this limitation in state-of-the-art IoT simulation in their work. Instead, they
propose their custom built large-scale simulator Gaia/Artis which is based
on the parallel and distributed computing (PADS) paradigm. PADS allows
a simulation to be executed among multiple distributed devices or Physical
Execution Units (PEUs) , which will signiﬁcantly improve scaling capabil-
ities. Each PEU consists of a collection of logical processes (LP). An LP
represents a part of the simulation and contains a collection of simulation
entities (SE). A simulation entity, as the name implies, represents an in- di-
vidual simulation model or agent (in the context of agent based simulation).
A disadvantage of PADS and distributed simulation is that it leads to ad-
ditional diﬃculties with regards to synchronization, simulation partitioning
and transparency. Extensive research has already been done to cope with
these challenges. Although the PADS methodology to our knowledge hasn’t
been applied to evaluate real-time, simulation-based IoT systems, many of
its challenges are comparable.
5.2 Model Partitioning
Model partitioning aims to optimally distribute agents across multiple servers
to decrease the overall communication cost. A signiﬁcant part of the com-
putational interaction cost is characterized by remote communication across
multiple PEU’s . This computational cost of remote communication is
much higher than the local communication cost between SE’s located in the
same region. Much eﬃciency can be gained by analyzing communication
patterns and optimizing the distribution of the simulation entities, so that
local communication is maximized and as a result the communication cost
will be lower.
Static Model Partitioning: We make a distinction between a static and
a dynamic approach. With a static partitioning approach it is assumed that
the interaction dynamics between SEs remain unchanged, as soon as these
dynamics change over the course of the simulation the beneﬁt of the par-
titioning decreases. For example, a ﬁxed partitioning could be conﬁgured
that speciﬁes the migration times of individual simulation models. Such a
schedule could be performing well at ﬁrst, but as soon as the communica-
tion dynamics change it will lead to suboptimal simulator performance .
Instead a ﬁxed execution schedule that speciﬁes exact migration times or
preferred clustering schemes of individual simulation entities. However, in
the domain of IoT we can assume the presence of stochastic dynamic nodes
and devices. This will inherently lead to unpredictable changes in interaction
dynamics. Furthermore, because of the required scale to simulate emergent
behavior we can conclude that static partitioning methods are impractical
and not preferable in order to simulate EBI applications .
Dynamic Model Partitioning using Heuristics: When compared to
static partitioning, the dynamic partitioning methods will be more adequate
for IoT use cases, as they are able to automatically migrate simulation entities
when their communication patterns evolve. In the context of PADS, many
research has been done in the area of dynamic simulation partitioning. Most
of these techniques rely on generic solutions based on heuristics. In most
cases these heuristics don’t have a complete view on the global simulation
state because this would be too computationally expensive. Instead most
heuristics rely on limited data directly available on the individual hosts.
D’Angelo proposes three heuristics , each one is slightly diﬀerent and can
be used for diﬀerent types of simulation use cases.
•Heuristic 1: The ﬁrst heuristic relies on a sliding window approach
of the last k timesteps. For each simulation entity the ratio between
external interactions versus internal interactions is evaluated. When
the ratio exceeds a predeﬁned threshold the SE can be migrated to the
LP it communicated with the most when at least a minimum amount
of timesteps have passed since the last migration of the SE.
•Heuristic 2: This heuristic is very similar to the latter, but in this case
the sliding window is based on the last k interactions instead of the
last k timesteps. As a result SE’s that have less interactions will be
marked sooner as migration candidates. This is because old interaction
events will still impact the migration decision, opposed to the timestep
windowing approach where these would possibly have been discarded
•Heuristic 3: The third heuristic is very similar to the second, but in-
stead of evaluating the interactions each timestep, which is very com-
putationally intensive, the evaluation will only occur when at least a
certain amount of interactions occurred since the last evaluation.
The heuristics discussed above prove to have very positive impact on the
overall performance and the scalability of the simulations. In other work,
various algorithms are proposed that focus more on the distribution of sim-
ulation workload instead of communication. For example, Boukerche et. al.
propose a solution to dynamically distribute the SE’s in the context of a
distributed simulation that leverages the Chandy-Misra null messages con-
servative synchronization algorithm  . As a result, a signiﬁcant decrease
in simulation runtime is achieved.
Dynamic Model Partioning using Domain Knowledge: A possible
disadvantage in the heuristics approach is that it doesn’t take domain knowl-
edge into account. This can lead to a number of undesired side eﬀects. For
example, migrations might occur that need to be undone in a later phase.
This could result in a higher computational cost (introduced by these con-
secutive migrations) than the obtained gain which was only temporal. The
heuristics presented above do try to prevent oscillating migrations by intro-
ducing a threshold that prevents immediate undoing a migration. However,
another solution would be to inject domain knowledge that could prevent
these unwanted migrations to occur in the ﬁrst place. Furthermore, it could
perform more optimal migrations that couldn’t have been achieved by solely
analyzing local communication patterns of individual SEs.
Van Tendeloo and Vangheluwe demonstrate in their work that signiﬁcant
performance improvements can be obtained by injecting domain knowledge in
their Python based distributed DEVS simulator PythonPDEVS . Their
work builds further on the abstract notion of activity introduced by Muzy
et. al. which represents a measure for a number of events in the context of
a Discrete Event Simulation system . This activity concept can refer to
various resources such as time, memory or energy which are relevant in a DES
system. In the context of performance optimization the time resource will be
most relevant. One can look at activity from various perspectives, either from
a computational load perspective or from a communicational perspective.
The heuristics discussed in the previous paragraph looks at activity from a
communicational perspective whereas activity-based on the computational
perspective. The concept of activity prediction allows simulation models to
provide hints to the simulator kernel about both their current and anticipated
activity and how they should be distributed. Consequently, these hints will be
exploited to decide when a SE migration should occur. This approach leads to
a more optimal distribution of computational load across servers. The idea of
activity prediction based on domain knowledge is also relevant when looking
at the activity concept from a communicational load perspective, in that
case, the goal would be to reduce the overall communicational load over the
network. A disadvantage of this approach is that a level of transparency has
to be sacriﬁced, breaking down the boundary between model and simulation
engine in order to obtain better performance.
Finally, another important decision that needs to be considered when devel-
oping an EBI simulator is synchronization. This section gives an overview of
the most relevant techniques described in state-of-the-art research.
Synchronization is an important topic in the ﬁeld of distributed simula-
tion. Its goal is to maintain validity of a distributed simulation system by
preventing causal inconsistencies. Such an inconsistency occurs when events
that depend on each-other are executed in the wrong order . For exam-
ple, imagine an event B that depends on the results of an event A, if event B
were to be processed before the execution of event A, a causal inconsistency
occurs and simulation results could become invalid. Various synchronization
techniques are discussed in literature to maintain this causal consistency, a
distinction is made between two major categories 1) optimistic synchroniza-
tion techniques and 2) conservative optimization techniques. Each technique
will lead to diﬀerent performance results.
Conservative Synchronization One way to maintain causal consistency
among various LP’s in a distributed simulation is to prevent a simulator to
move to the next event only when it’s sure that no LP’s will insert an earlier
event or a next event to be removed . This approach is called a conser-
vative synchronization approach. The best known conservative techniques
are based on the Chandy Misra Bryant algorithm . In their work each
LP should have an individual communication channel for all other LP’s it
communicates with. Each LP is assumed to post timestamped events in the
right order. An LP is only able to accept and process events with the lowest
time-stamp as soon as all messages in all channels are received. This could
lead to deadlocks when some LPs are not generating events, leading to some
empty channels and an LP consequently remaining in a waiting/blocking
state. As a solution, the idea of null messages is added, where null mes-
sages can be send to channels when no events are assumed for a particular
LP. The CMB technique allows the simulation to progress at various rates,
however it introduces high overhead because of the null messages in terms of
computational overhead and network load .
Optimistic Synchronization The conservative synchronization method
blocks the simulator and results in some LPs waiting in an idle state on
other, slower LPs, which is not always necessary. Instead, the optimistic
synchronization method introduces a non-blocking approach where each LP
continues on its own rate, not waiting for other LPs until an inconsistency
occurs, e.g. an event is received with a time-stamp earlier than the local time
of the LP. When such a inconsistency is found the simulator will turn back
its state to a checkpoint prior to the time-stamp of the received event before
continuing processing. As a result, the simulation remains valid. This non-
blocking optimistic synchronization mechanism as described above, is called
the time warp protocol, ﬁrst introduced in Jeﬀeron’s paper . In many
cases it could lead to better simulation performance and a reduced execution
Most of the performance optimizing techniques discussed in the above sec-
tions are focused on a full simulation-based testing approach of EBI systems.
However, our ambition is to eventually enable a hybrid simulation based test-
ing environment. The high-level architecture of such a system is depicted in
ﬁgure 2 below. A large-scale simulated environment would be combined with
an actual environment containing a prototype setup limited in scale. Both
environments are able to communicate with a supporting middleware sys-
tem. This way a full-scale EBI system can be deployed, tested, calibrated
and validated as a whole without the need to do a full deployment in an
actual environment. However, this will have a number of consequences to
the simulation technology being used. A distributed simulation environment
relying on a PADS-like architecture would be preferable. However, the meth-
ods that can be used are limited. An optimistic synchronization approach
is not realistic because we could not allow a simulation model to rewind its
state after an inconsistency is found as this would break the required real-
time property. As a result, only conservative synchronization methods are
feasible. Furthermore, we need to have a guarantee that all SE states are
updated in due time while maintaining consistencies, the blocking nature of
the conservative synchronization mechanism might result in state updates
not being processed at the proper rate.
Additionally, in the hybrid evaluation scenario a state synchronization
is required in the context of EBI applications. This mechanism needs to
ensure that actual prototype in the real-world perceives similar data as the
virtual, simulated devices. Therefore, the state of the simulation needs to be
synchronized with the state of the real-world For example, in the case of an
EBI systems related to air pollution, measurements of air pollution in the
virtual environment need to match those measured in the real world.
Figure 2: Hybrid evaluation scenario
Finally, from a model
we must make sure that
SE migrations between
LP’s maintain the avail-
ability and responsive-
ness of the SE, e.g. it
could not be allowed
that a SE remains unre-
sponsive during the pro-
cess of migrating. We
identiﬁed many existing techniques that could help to improve the perfor-
mance and overall scalability of large-scale EBI IoT systems. In further
research we will focus on adding IoT and EBI related domain knowledge to
the aforementioned techniques and to better understand in which cases which
techniques are most appropriate.
In this position paper, we identiﬁed the challenges that arise when trying
to evaluate emergent behavior in Internet of Tings applications. A type
of application, where decentralized behavior is key. Furthermore, we pro-
posed the idea of a hybrid simulation environment that combines real-world
and simulated nodes for testing and deployment of these emergent behavior
IoT systems. As a solution, we propose a distributed simulation-based ap-
proach. However, relying solely on a simulation architecture is not enough
to cope with the scalability challenges of simulating emergent behavior. In-
stead, we present an overview of state-of-the-art techniques that could help
to improve simulation performance and overall scaling capabilities. In further
research we model and leverage the IoT domain knowledge on the techniques
mentioned in this study to develop a methodology for evaluating emergent
behavior in IoT systems.
 R. R. Barton. Metamodeling: a state of the art review. In Simulation
Conference Proceedings, 1994. Winter, pages 237–244. IEEE, 1994.
 L. Bononi, M. Bracuto, G. D’Angelo, and L. Donatiello. Proximity
detection in distributed simulation of wireless mobile systems. In Pro-
ceedings of the 9th ACM international symposium on Modeling analysis
and simulation of wireless and mobile systems, pages 44–51. ACM, 2006.
 A. Boukerche and S. K. Das. Dynamic load balancing strategies for
conservative parallel simulations. In Parallel and Distributed Simulation,
1997., Proceedings. IEEE, 1997.
 G. Carneiro. Ns-3: Network simulator 3. In UTM Lab Meeting April,
volume 20, 2010.
 D. Caughlin and A. F. Sisti. Summary of model abstraction techniques.
In Enabling Technology for Simulation Science, volume 3083, pages 2–
14. International Society for Optics and Photonics, 1997.
 K. M. Chandy and J. Misra. Distributed simulation: A case study in
design and veriﬁcation of distributed programs. IEEE Transactions on
software engineering, (5):440–452, 1979.
 G. D’Angelo. Parallel and distributed simulation from many cores to the
public cloud. In High Performance Computing and Simulation (HPCS),
2011 International Conference on. IEEE, 2011.
 G. D’Angelo, S. Ferretti, and V. Ghini. Simulation of the internet of
things. In High Performance Computing & Simulation (HPCS), 2016
International Conference on, pages 1–8. IEEE, 2016.
 O. Developer. Omnet++ network simulation frame-work.
 A. Dunkels, B. Gronvall, and T. Voigt. Contiki-a lightweight and ﬂex-
ible operating system for tiny networked sensors. In Local Computer
Networks, 2004. 29th Annual IEEE International Conference on, pages
455–462. IEEE, 2004.
 G. DAngelo. The simulation model partitioning problem: An adaptive
solution based on self-clustering. Simulation Modelling Practice and
Theory, 70:1–20, 2017.
 G. DAngelo and M. Marzolla. New trends in parallel and distributed
simulation: From many-cores to cloud computing. Simulation Modelling
Practice and Theory, 49:320–335, 2014.
 F. K. Frantz. A taxonomy of model abstraction techniques. In Pro-
ceedings of the 27th conference on Winter simulation, pages 1413–1420.
IEEE Computer Society, 1995.
 R. M. Fujimoto. Parallel and distributed simulation systems, volume
300. Wiley New York, 2000.
 R. Gore, S. Diallo, C. Lynch, and J. Padilla. Augmenting bottom-up
metamodels with predicates. Journal of Artiﬁcial Societies and Social
Simulation, 20(1), 2017.
 H. Gupta, A. Vahid Dastjerdi, S. K. Ghosh, and R. Buyya. ifogsim:
A toolkit for modeling and simulation of resource management tech-
niques in the internet of things, edge and fog computing environments.
Software: Practice and Experience, 47(9):1275–1296, 2017.
 D. R. Jeﬀerson. Virtual time. ACM Transactions on Programming
Languages and Systems (TOPLAS), 7(3):404–425, 1985.
 L. Lamport. Time, clocks, and the ordering of events in a distributed
system. Communications of the ACM, 21(7):558–565, 1978.
 P. Levis, N. Lee, M. Welsh, and D. Culler. Tossim: Accurate and scal-
able simulation of entire tinyos applications. In Proceedings of the 1st
international conference on Embedded networked sensor systems, pages
126–137. ACM, 2003.
 C. M. Macal and M. J. North. Tutorial on agent-based modeling and
simulation. In Simulation Conference, 2005 Proceedings of the Winter,
pages 14–pp. IEEE, 2005.
 M. J. Mataric. Designing emergent behaviors: From local interactions
to collective intelligence. In Proceedings of the 2nd International Con-
ference on Simulation of Adaptive Behavior, 1993.
 K. Mehdi, M. Lounis, A. Bounceur, and T. Kechadi. Cupcarbon: A
multi-agent and discrete event wireless sensor network design and sim-
ulation tool. In Proceedings of the 7th International ICST Conference
on Simulation Tools and Techniques, pages 126–131, 2014.
 J. Misra. Distributed discrete-event simulation. ACM Computing Sur-
veys (CSUR), 18(1):39–65, 1986.
 A. Muzy, L. Touraille, H. Vangheluwe, O. Michel, M. K. Traor´e, and
D. R. Hill. Activity regions for the speciﬁcation of discrete event systems.
In Proceedings of the 2010 Spring Simulation Multiconference, page 136.
Society for Computer Simulation International, 2010.
 D. M. Nicol. Principles of conservative parallel simulation. In Proceed-
ings of the 28th conference on Winter simulation, pages 128–135. IEEE
Computer Society, 1996.
 D. Roca, D. Nemirovsky, M. Nemirovsky, R. Milito, and M. Valero.
Emergent behaviors in the internet of things: the ultimate ultra-large-
scale system. IEEE micro, 36(6):36–44, 2016.
 J. D. Rodriguez, K. W. Bauer Jr, J. O. Miller, and R. E. Neher Jr.
Building prediction models of large hierarchical simulation models with
artiﬁcial neural networks and other statistical techniques. In Visual
Information Processing, page 69780, 2008.
 L. Sanchez, L. Mu˜noz, J. A. Galache, P. Sotres, J. R. Santana, V. Gutier-
rez, R. Ramdhany, A. Gluhak, S. Krco, E. Theodoridis, et al. Smart-
santander: Iot experimentation over a smart city testbed. Computer
Networks, 61:217–238, 2014.
 V. Stojkovski. Multi-abstraction, multi-formalism modelling and sim-
ulation: Integrating forrester system dynamics, cellular automata and
agent based modelling. Master’s thesis, University of Antwerp, Belgium,
 P. Symonds, J. Taylor, Z. Chalabi, and M. Davies. Performance of neural
networks vs. rbf when forming a metamodel for residential buildings.
International Journal Of Civil, Environmental, Structural, Construction
And Architectural Engineering, 9(12):1446–1450, 2015.
 Y. Van Tendeloo and H. Vangheluwe. Activity in pythonpdevs. In ITM
Web of Conferences, volume 3, page 01002. EDP Sciences, 2014.
 G. Wainer. Cd++: a toolkit to develop devs models. Software: Practice
and Experience, 32(13):1261–1306, 2002.
 B. P. Zeigler, H. Praehofer, and T. G. Kim. Theory of modeling and
simulation: integrating discrete event and continuous complex dynamic
systems. Academic press, 2000.