Conference PaperPDF Available

Towards evaluating emergent behavior of the Internet of Things using large scale simulation techniques

Authors:

Abstract and Figures

With the increase in Internet of Things devices and more decentralized architectures we see a new type of application gain importance, a type where local interactions between individual entities lead to a global emergent behavior, Emergent-based IoT (EBI) Systems. In this position paper we explore techniques to evaluate this emergent behavior in IoT applications. Because of the required scale and diversity this is not an easy task. Therefore, we mainly focus on a distributed simulation approach and provide an overview of possible techniques that could optimize the overall simulation performance. Our focus is both on modeling and simulation technology.
No caption available
… 
Content may be subject to copyright.
Towards evaluating emergent behavior of the
Internet of Things using large scale simulation
techniques
Stig Bosmans1, Siegfried Mercelis1, Peter Hellinckx1and Joachim Denil2
University of Antwerp
imec, IDLab1
Flanders Make2
Groenenborgerlaan 173
Antwerp, Belgium
{stig.bosmans, siegfried.mercelis,
peter.hellinckx, joachim.denil}@uantwerpen.be
February 27, 2018
Abstract
With the increase in Internet of Things devices and more decentral-
ized architectures we see a new type of application gain importance,
a type where local interactions between individual entities lead to a
global emergent behavior, Emergent-based IoT (EBI) Systems. In this
position paper we explore techniques to evaluate this emergent behav-
ior in IoT applications. Because of the required scale and diversity this
is not an easy task. Therefore, we mainly focus on a distributed sim-
ulation approach and provide an overview of possible techniques that
could optimize the overall simulation performance. Our focus is both
on modeling and simulation technology.
1
arXiv:1802.09207v1 [cs.DC] 26 Feb 2018
1 Introduction
In recent years, there has been a significant increase in the number of internet-
connected devices. By 2020 it is expected that up to 50 billion devices will be
connected to the internet. This Internet of Things (IoT) will create novel ser-
vices that will highly impact our way of life in areas such as transportation,
shopping, health and safety. With IoT becoming more mainstream and with
the rise in the amount of devices getting interconnected, the complexity and
scale of the IoT landscape will largely increase. The interoperability between
IoT devices and actuators of all sorts will prove to be vital for future IoT
applications. As a result of the increasing scale and diversity and because
of new IoT architectures such as edge computing, the concept of emergent
behavior in IoT systems is gaining more attention within the IoT commu-
nity. Mataric et. al. [21] defines emergent behavior as a collection of actions
and patterns that result from local interactions between elements and their
environment which have not been explicitly programmed. It is loosely based
on the emergent behavior that is observed in bird flocks where birds apply
three local rules which result in emergent flocking behavior (e.g. remain x
distance to neighbor birds). D. Roca et. al. [26] argues that this emergent
behavior will lead to improved scalability, interoperability and cost efficiency
of ultra large scale IoT systems as opposed to traditional approaches that
heavily rely on extensive programming of explicit behaviors. This approach
particularly benefits the IoT areas which require the interaction of an enor-
mous amount of devices where relying solely on a centralized architecture is
insufficient, such as smart power grids, autonomous car flocking and smart
traffic lights. We will refer to this type of systems as emergent based IoT
(EBI) systems in the remainder of this paper. Building such emergent-based
IoT systems is complex as the system is composed from different heteroge-
neous and autonomous components. Examples of such components can range
from actual IoT sensors sharing observations, towards smart actuators such
as smart traffic lights that make decisions based on interactions with their
counterparts and by observing sensor data. However, a human actor is also
a component in an IoT system. From one perspective a human actor can be
seen as a simple data generator, e.g. by walking around with a GPS-sensor
embedded in their smartphone, they could broadcast location-data to an IoT
middleware. From another perspective they can play a very active role in
the IoT system by for example generating evolving traffic patterns which will
influence the behavior of smart traffic lights. All these various components
2
and their individual behavior, their interactions and their goals need to be
considered and evaluated when developing an EBI system. We believe that
EBI systems will be the standard for future decentralized large scale IoT
applications. Because of the complexity, the large-scale and decentralized
property of these EBI systems and their components, testing, validation and
calibration will be vital in successfully developing such systems. In practice,
IoT tests are either performed in small-scale/lab-based prototype setups or
based on software testing approaches. The limited scaling capabilities of
these testing approaches make it very hard to properly test EBI systems as a
whole. EBI systems introduce a new component to test, being the global vs
local behavior. Traditional testing techniques would not be able to test this
global behavior at scale. Furthermore, the testing of the emergent behavior
of IoT has been largely ignored in the literature. In this position paper, we
investigate the challenges in testing EBI and look for solutions, methods and
techniques mainly based on large-scale and distributed simulation based test-
ing of IoT. The rest of this paper is organized as follows. Section 2 positions
the challenges and needs in testing EBI systems, the next section provides an
overview of possible techniques to improve performance and overall simula-
tion scalability, in section 4 we further investigate state-of-the-art techniques
that could be used in the scope of simulating EBI systems. Section 6 presents
a discussion of the remaining challenges of real-time testing of IoT and how
the presented techniques could help to solve these. Finally, in section 7 we
conclude our work.
2 Challenges in evaluating emergent behav-
ior
In the context of this paper, we are only interested in system-level evaluation
of the EBI. This means that we are interested in observing the overall, global
behavior posed by the application and we are unaware of the technical im-
plementation details such as network and the hardware and software of the
different sensor nodes.
We distinguish three different scenarios where evaluating the emergence
of behavior is necessary:
Validation of emergent behavior: We want to make sure that the
local behavior and rules imposed by the application lead to the pre-
3
ferred emergent behavior.
Calibration of emergent behavior: Calibration of the application
needs to be done by tuning the various parameters in the IoT applica-
tion, e.g. the value of an incentive. Therefore, an optimization of emer-
gent behavior has to be done by monitoring the impact the local rules
and incentives have until the global, emergent behavior converges to a
desired behavior. In other words, design-space exploration is needed to
obtain optimized parameters for the simulation.
Integration testing and deployment: Integrating the EBI applica-
tions in the real-world has a significant impact and therefore requires
an incremental means to accept and deploy the application. Ideally,
this translates to running the simulation models in parallel with the
IoT middleware and the actual nodes operating in the real world. The
number of nodes can be increased gradually to fully deploy the ap-
plication in the field. For example, in a first integration test, a user
could directly interact with a real prototype and with the simulated
environment. Further tests increase the number of users in a local
environment.
The testing of EBI systems for each of these scenarios poses significant
challenges because of the scale and diversity of these systems. We differenti-
ate between three major methods that are relevant in this work:
1. Real-world evaluation: This method relies on deploying the system
in a real environment. Often the scale of such tests are limited to small
lab-based setups. This has as an advantage, that the tests are easy to
deploy. However in practice, this type of testing should only be used
for testing the behavior of a single IoT device. Formulating reliable
conclusions about the validity of the behavior of the IoT system as
a whole is unfeasible. Instead, there are a few large-scale, real-life
IoT testbeds which allow tests to be executed at a more appropriate
size. An example of such a testbed is the Smart Santander projects
[28]. It is a city-wide, real IoT testbed. It offers many thousands of
IoT devices and interconnected gateways. It offers great opportunities
to effectively test certain aspects of IoT devices deployed in a real
environment. However, many EBI systems, such as the smart traffic
light example, rely on custom devices and applications which would be
too costly and too difficult to include in existing testbeds.
4
2. Simulation-based evaluation: A simulated environment consisting
of a wide range of virtual IoT devices interacting with each-other and
with the IoT middleware under test can be seen as a very effective test-
ing solution. Although real testing is often desirable, simulation testing
is a more flexible and cost-efficient approach. It allows for a more con-
trolled and reusable environment that can be tweaked much easier.
Within the IoT domain simulation testing is most often used to test
technical aspects of the system such as network related features, power
consumption etc. The most well-known examples of such simulators
are NS-3 [4] and Omnet++ [9]. Various IoT operating systems such as
Contiki and TinyOs also offer their dedicated simulator, Contiki [10]
and Tossim [19] respectively. Some simulators are focused on testing
more high-level IoT setups such as the iFogSim [16] which is used to test
IoT edge or fog architectures. Another example is the CupCarbon sim-
ulator [22] which is used to simulate Smart City environments. Finally,
D’Angelo et. al. demonstrate the use of the Gaia/Artis specifically to
run large-scale IoT simulations [8]. However, most of the state-of-the-
art IoT simulators do not have the necessary scalability requirements
(more than 100.000 simulation entities) to simulate such large-scale IoT
environments and to specifically evaluate emergent behavior.
3. Hybrid evaluation: In order to facilitate the real-time testing ap-
proach explained above, a hybrid testing can be used which combines a
small-scale real life environment with a virtual simulated environment.
This way, actual IoT prototypes can interact with virtual devices, oper-
ating both as a single system able to generate emergent behavior. This
could be particularly interesting for pilot projects or proof of concepts,
where the scale is still too limited in order for emergent behavior to
arise. Furthermore, it combines the advantages of both real-life testing
and simulation testing.
Relying solely on real-world evaluation environments such as the Smart
Santander project to test emergent behavior is costly and unpractical. Al-
though these testbeds offer a number of ready-to-use sensors and IoT devices,
deploying an emergent-based application on such an environment requires a
significant amount of costly changes. Each EBI system will need a number
of specialized sensors and actuators that need to be deployed, installed and
linked to the available smart city infrastructure. Making such investments is
5
impractical at the early stages of a project. Therefore, simulation or hybrid
approach have a much smaller threshold to use.
However, our observations detailed above show that the major challenge
lies in the modeling of the global behavior and the scalability of the simulation
framework. Furthermore, in all the different scenarios, there are different
requirements on the accuracy of the simulation. On the one hand, validating
the behavior of the EBI requires accurate results on the global behavior
of the system. On the other hand, during calibration the parameter space
of the application is explored with less accurate simulations to find good
configurations of the application which are validated afterwards with detailed
simulations.
3 Overview
The main challenge in simulating emergent behavior is related to perfor-
mance. In order to obtain some level of emergent behavior a great amount
of interacting entities is required. Running such a simulation on a single
device wouldn’t be possible, instead distributing to multiple servers is a bet-
ter approach to cope with the necessary scale. Example implementations
are described in the Parallel and Distributed Simulation (PADS) methodol-
ogy [14] and in the IEEE 1516 standard. Facilitating the proper execution
of these simulation entities in a simulated environment still requires several
performance optimization techniques in the simulation kernel. Preferably,
these optimizations can be performed transparent to the simulation modeler.
However, this won’t be possible all the time. In this sections we provide an
overview of various optimization techniques that can be applied dynamically
or statically. A dynamic optimization can be applied at run-time and can be
context-dependent, while a static optimization needs to be applied up-front.
We made a break-out of major performance optimization techniques in figure
1 below. Note, that this list of techniques isn’t complete and is the current
result of on-going study.
4 Possible Solutions when Modeling EBI
An important aspect of generating emergent behavior is to properly describe
the behavior of the individual elements of the real-world within the simulation
6
Figure 1: Overview of simulation performance optimization techniques
model. Furthermore, once these systems are modeled we show which model-
ing techniques can help us in achieving a scalable EBI simulation framework.
4.1 Model Representation
In order to model emergent behavior the agent based modeling (ABM)
paradigm seems the most appropriate [20]. With ABM, a bottom-up mod-
eling approach is applied. Instead of modeling the global expected behavior,
the modeler describes the behavior of the individuals. The ABM paradigm
allows these individuals to interact. Eventually, these interactions will lead
to a global behavior. As the EBI application are deployed in a similar mat-
ter, the simulation engineer has to model the different entities involved in the
simulation. While this seems trivial, the individuals require detailed models
on their decision making processes.
Other formalisms can also be appropriate. Discrete-event formalisms such
as DEVS [33] model the behavior of a system using a timed sequence of
”events” either as input to a system or as a timeout within the system. These
events cause instantaneous changes to the state of the system. For example,
when simulating traffic behavior, agent based modeling might not be the
most appropriate formalism. Lots of research has been done in the domain
of queuing theory which might be a better, less computationally expensive
formalism. Atomic discrete event models can be combined together into a
couple model. Furthermore, extensions exist that couple these models in a
7
grid, e.g. Cell-DEVS [32]. These types of cellular automata can have an
advantage to model geographical areas. E.g. the city is divided into a grid
of cells where each cell models a part of the city.
Finally, differential equations can help to model rates of changes of prop-
erties within the IoT setting. E.g. traffic flows can be modeled with a
differential equation. Different equations can also be coupled together us-
ing co-simulation techniques to allow for a divide-and-conquer approach to
model complex systems.
However, simulating such a large model of our IoT application is too com-
plex. Model abstraction techniques should decrease the computational cost
of individual models by simplifying the representation of such a model while
maintaining the original behavior as accurate as possible. As a result, the
overall computational complexity can be optimized. The main objective of
applying model abstraction is to make a proper trade-off between computa-
tional complexity and accuracy. Extensive taxonomies of model abstraction
techniques exist in literature [13, 5]. In this work we will only discuss the
most relevant techniques. As demonstrated in figure 1, we differentiate be-
tween two major categories, one based on multi-formalism model abstraction
and another based on single-formalism abstraction. We than discuss how we
can use these techniques in an IoT simulation environment.
4.2 Abstractions within a single formalism
These abstractions are applied on the model without changing the formalism
to describe it. We define three types of abstractions that can happen within
a single model. We will apply these types of abstractions on the individuals
of the ABM but they can also be applied to models in other formalisms as
well.
State abstraction This technique abstracts the state of a single individ-
ual. As the decision processes of individuals can be very complex, certain
parts of the decision making process that an individual exhibits can be ig-
nored without compromising the global behavior of the simulation, e.g. the
individual also reasons about other properties that do not influence the global
behavior, or higher-order reasoning with little to no impact on the decision
of the agent.
The process of creating a more abstract individual from a detailed one can
be done manually. This requires insight into both the decision making pro-
8
cess of the individuals as into the application that generates the emergence of
behavior. Therefore, more automatic techniques might be more appropriate.
An example technique that can be used for this is metamodeling (as in surro-
gate modeling). Caughlin et. al. defines a metamodel as a projection of the
original, high-fidelity model onto a subspace defined by new constraints or
regions of interest [5]. In practice, a metamodel is a mathematical approxi-
mation of a complex model. The original model is treated as a blackbox, and
the metamodel operates as a surrogate model that replaces the original. To
come to such a metamodel a detailed analysis of the input/output mapping
of the original model has to be performed. Based on this analysis a sur-
rogate metamodel should learn to represent a similar mapping. Preferably,
the surrogate model is more abstract and is less computationally complex.
Various methods have been used for the development of metamodels such as
polynomial regressions, radial basis functions (RBF) and others [1]. Most
of these techniques try to approximate input functions or data. Since the
goal of metamodeling is to map an input value to a specific output value
we believe other supervised learning techniques could be used as well. For
example, P. Symonds et. al. demonstrate that neural networks (NN) could
perform up to 15% better compared to classic RBF approaches [30] [15]. We
believe that evaluating various state-of-the-art deep neural networks could
lead to even better results. We consider this as a promising area for further
research.
Entity Aggregation Another option to reduce the computational com-
plexity of simulation entities is Entity Aggregation. The idea of entity ag-
gregation is to combine multiple low-level simulation entities with a single
high-level entity while preserving the collective behavior. For example, when
simulating emergent behavior in a smart power grid application, a single
agent could represent a single household but depending on the required level
of accuracy a single agent could also represent an entire neighborhood. Ro-
driguez et. al. demonstrates that NN’s could be used to transform a collec-
tion of low-level entities to an aggregated model [27] . From an architectural
perspective, a mechanism in the simulation kernel can be created to detect
and analyze clusters of agents that interact a lot and pose homogeneous be-
havior. Using metamodeling techniques such a cluster of agents can then
be replaced by a single surrogate model. Conversely, the modeler can also
manually model an aggregation of individual simulation agents.
9
Temporal Abstraction Finally, temporal abstraction can be used to limit
the granularity of simulation entity state updates over time. Since, each state
update results in a cascade of simulation events the overall computational
cost could be significantly reduced.
Engineers can themselves easily change the temporal granularity (time-
step) of their simulation. However, choosing a too large time-step can result
in an unstable simulation. Automatic techniques to adapt the time-step
also exist. E.g. for solving differential equations, several techniques are
available that automatically adapts the time-step of the simulation based on
the estimated error, e.g. Runge-Kutta 4-5 embeds a higher-order method in
its solver to estimate the error and adapt the time-step.
4.3 Multi-formalism Modeling
Changing formalisms during abstraction can help to achieve better perfor-
mance results as we can leverage the strengths of different types of for-
malisms. For example, as explained in section 4.1, in the domain of traffic
simulation, queuing theory might be a more appropriate, less computation-
ally expensive formalism compared to agent based modeling. The formalism
change allows for an easier state and aggregation abstraction.The disadvan-
tage of applying multi-model abstraction is that this can’t be automated,
the modeler will be responsible to model the same behavior in different for-
malisms.
4.4 Multi-resolution Modeling
Multi-resolution modeling can be used to vary between model abstraction
levels (and possibly formalisms) within the same simulation. The advantage
of this is that we can increase abstraction levels in simulation areas where less
accuracy is required. This will consequently reduce the computational com-
plexity of the simulation. Oppositely we could also increase levels of detail
in certain simulation areas when an increased level of accuracy is required.
The multi-resolution model can be static or dynamic.
When applying a static approach, the most appropriate formalisms and
levels of detail have to be specified upfront and will be used during the entire
course of the simulation. Due to the dynamic nature of IoT applications,
the potential computational benefit is limited. Conversely, with a dynamic
approach we can switch between various levels of abstraction during the
10
course of the simulation. This can be done by predefining when and where
an abstraction switch should occur. Domain knowledge is necessary to detect
and abstract/refine the (part of the) model.
Learning approaches are also possible to switch levels of abstraction, by
learning areas of opportunity and applying the necessary level of abstraction.
For this, an additional mechanisms in the simulation kernel are required that
detects areas of opportunity based on various parameters. For example, sim-
ulation entities that have a limited amount of interaction with the rest of
the simulation could be marked as candidates for an abstraction level in-
crease. The advantage of such an approach is that we can adaptively detect
opportunities to switch between abstraction levels in order to decrease nec-
essary computational resources or oppositely, use all available computational
resources.
Applying a dynamic multi-resolution modeling approach leads to addi-
tional challenges that need to be taken into account [29]. For example, reini-
tializing a more abstract model from the current state of the original model
is possible as there is enough information available in the original. However,
when switching from a more abstract to a refined model, extra information
is required. This should be mend by modeling additional domain-knowledge
to fill this information gap.
5 Possible Solutions when Simulating EBI
In this section we will discuss techniques that could improve the performance
capabilities of the simulator itself. Most of these techniques have been studied
in various domains such as parallel and distributed simulation (PADS) and
distributed discrete event simulation.
5.1 Simulation Architecture
Most of the current IoT simulators have an underlying monolithic architec-
tures which limits its scalability capabilities. D’Angelo et. al. recognizes
this limitation in state-of-the-art IoT simulation in their work. Instead, they
propose their custom built large-scale simulator Gaia/Artis which is based
on the parallel and distributed computing (PADS) paradigm. PADS allows
a simulation to be executed among multiple distributed devices or Physical
Execution Units (PEUs) , which will significantly improve scaling capabil-
11
ities. Each PEU consists of a collection of logical processes (LP). An LP
represents a part of the simulation and contains a collection of simulation
entities (SE). A simulation entity, as the name implies, represents an in- di-
vidual simulation model or agent (in the context of agent based simulation).
A disadvantage of PADS and distributed simulation is that it leads to ad-
ditional difficulties with regards to synchronization, simulation partitioning
and transparency. Extensive research has already been done to cope with
these challenges. Although the PADS methodology to our knowledge hasn’t
been applied to evaluate real-time, simulation-based IoT systems, many of
its challenges are comparable.
5.2 Model Partitioning
Model partitioning aims to optimally distribute agents across multiple servers
to decrease the overall communication cost. A significant part of the com-
putational interaction cost is characterized by remote communication across
multiple PEU’s [2]. This computational cost of remote communication is
much higher than the local communication cost between SE’s located in the
same region. Much efficiency can be gained by analyzing communication
patterns and optimizing the distribution of the simulation entities, so that
local communication is maximized and as a result the communication cost
will be lower.
Static Model Partitioning: We make a distinction between a static and
a dynamic approach. With a static partitioning approach it is assumed that
the interaction dynamics between SEs remain unchanged, as soon as these
dynamics change over the course of the simulation the benefit of the par-
titioning decreases. For example, a fixed partitioning could be configured
that specifies the migration times of individual simulation models. Such a
schedule could be performing well at first, but as soon as the communica-
tion dynamics change it will lead to suboptimal simulator performance [31].
Instead a fixed execution schedule that specifies exact migration times or
preferred clustering schemes of individual simulation entities. However, in
the domain of IoT we can assume the presence of stochastic dynamic nodes
and devices. This will inherently lead to unpredictable changes in interaction
dynamics. Furthermore, because of the required scale to simulate emergent
behavior we can conclude that static partitioning methods are impractical
and not preferable in order to simulate EBI applications [7].
12
Dynamic Model Partitioning using Heuristics: When compared to
static partitioning, the dynamic partitioning methods will be more adequate
for IoT use cases, as they are able to automatically migrate simulation entities
when their communication patterns evolve. In the context of PADS, many
research has been done in the area of dynamic simulation partitioning. Most
of these techniques rely on generic solutions based on heuristics. In most
cases these heuristics don’t have a complete view on the global simulation
state because this would be too computationally expensive. Instead most
heuristics rely on limited data directly available on the individual hosts.
D’Angelo proposes three heuristics [11], each one is slightly different and can
be used for different types of simulation use cases.
Heuristic 1: The first heuristic relies on a sliding window approach
of the last k timesteps. For each simulation entity the ratio between
external interactions versus internal interactions is evaluated. When
the ratio exceeds a predefined threshold the SE can be migrated to the
LP it communicated with the most when at least a minimum amount
of timesteps have passed since the last migration of the SE.
Heuristic 2: This heuristic is very similar to the latter, but in this case
the sliding window is based on the last k interactions instead of the
last k timesteps. As a result SE’s that have less interactions will be
marked sooner as migration candidates. This is because old interaction
events will still impact the migration decision, opposed to the timestep
windowing approach where these would possibly have been discarded
long ago.
Heuristic 3: The third heuristic is very similar to the second, but in-
stead of evaluating the interactions each timestep, which is very com-
putationally intensive, the evaluation will only occur when at least a
certain amount of interactions occurred since the last evaluation.
The heuristics discussed above prove to have very positive impact on the
overall performance and the scalability of the simulations. In other work,
various algorithms are proposed that focus more on the distribution of sim-
ulation workload instead of communication. For example, Boukerche et. al.
propose a solution to dynamically distribute the SE’s in the context of a
distributed simulation that leverages the Chandy-Misra null messages con-
servative synchronization algorithm [6] [3]. As a result, a significant decrease
in simulation runtime is achieved.
13
Dynamic Model Partioning using Domain Knowledge: A possible
disadvantage in the heuristics approach is that it doesn’t take domain knowl-
edge into account. This can lead to a number of undesired side effects. For
example, migrations might occur that need to be undone in a later phase.
This could result in a higher computational cost (introduced by these con-
secutive migrations) than the obtained gain which was only temporal. The
heuristics presented above do try to prevent oscillating migrations by intro-
ducing a threshold that prevents immediate undoing a migration. However,
another solution would be to inject domain knowledge that could prevent
these unwanted migrations to occur in the first place. Furthermore, it could
perform more optimal migrations that couldn’t have been achieved by solely
analyzing local communication patterns of individual SEs.
Van Tendeloo and Vangheluwe demonstrate in their work that significant
performance improvements can be obtained by injecting domain knowledge in
their Python based distributed DEVS simulator PythonPDEVS [31]. Their
work builds further on the abstract notion of activity introduced by Muzy
et. al. which represents a measure for a number of events in the context of
a Discrete Event Simulation system [24]. This activity concept can refer to
various resources such as time, memory or energy which are relevant in a DES
system. In the context of performance optimization the time resource will be
most relevant. One can look at activity from various perspectives, either from
a computational load perspective or from a communicational perspective.
The heuristics discussed in the previous paragraph looks at activity from a
communicational perspective whereas activity-based on the computational
perspective. The concept of activity prediction allows simulation models to
provide hints to the simulator kernel about both their current and anticipated
activity and how they should be distributed. Consequently, these hints will be
exploited to decide when a SE migration should occur. This approach leads to
a more optimal distribution of computational load across servers. The idea of
activity prediction based on domain knowledge is also relevant when looking
at the activity concept from a communicational load perspective, in that
case, the goal would be to reduce the overall communicational load over the
network. A disadvantage of this approach is that a level of transparency has
to be sacrificed, breaking down the boundary between model and simulation
engine in order to obtain better performance.
14
5.3 Synchronization
Finally, another important decision that needs to be considered when devel-
oping an EBI simulator is synchronization. This section gives an overview of
the most relevant techniques described in state-of-the-art research.
Synchronization is an important topic in the field of distributed simula-
tion. Its goal is to maintain validity of a distributed simulation system by
preventing causal inconsistencies. Such an inconsistency occurs when events
that depend on each-other are executed in the wrong order [18]. For exam-
ple, imagine an event B that depends on the results of an event A, if event B
were to be processed before the execution of event A, a causal inconsistency
occurs and simulation results could become invalid. Various synchronization
techniques are discussed in literature to maintain this causal consistency, a
distinction is made between two major categories 1) optimistic synchroniza-
tion techniques and 2) conservative optimization techniques. Each technique
will lead to different performance results.
Conservative Synchronization One way to maintain causal consistency
among various LP’s in a distributed simulation is to prevent a simulator to
move to the next event only when it’s sure that no LP’s will insert an earlier
event or a next event to be removed [25]. This approach is called a conser-
vative synchronization approach. The best known conservative techniques
are based on the Chandy Misra Bryant algorithm [23]. In their work each
LP should have an individual communication channel for all other LP’s it
communicates with. Each LP is assumed to post timestamped events in the
right order. An LP is only able to accept and process events with the lowest
time-stamp as soon as all messages in all channels are received. This could
lead to deadlocks when some LPs are not generating events, leading to some
empty channels and an LP consequently remaining in a waiting/blocking
state. As a solution, the idea of null messages is added, where null mes-
sages can be send to channels when no events are assumed for a particular
LP. The CMB technique allows the simulation to progress at various rates,
however it introduces high overhead because of the null messages in terms of
computational overhead and network load [12].
Optimistic Synchronization The conservative synchronization method
blocks the simulator and results in some LPs waiting in an idle state on
other, slower LPs, which is not always necessary. Instead, the optimistic
15
synchronization method introduces a non-blocking approach where each LP
continues on its own rate, not waiting for other LPs until an inconsistency
occurs, e.g. an event is received with a time-stamp earlier than the local time
of the LP. When such a inconsistency is found the simulator will turn back
its state to a checkpoint prior to the time-stamp of the received event before
continuing processing. As a result, the simulation remains valid. This non-
blocking optimistic synchronization mechanism as described above, is called
the time warp protocol, first introduced in Jefferon’s paper [17]. In many
cases it could lead to better simulation performance and a reduced execution
time.
6 Discussion
Most of the performance optimizing techniques discussed in the above sec-
tions are focused on a full simulation-based testing approach of EBI systems.
However, our ambition is to eventually enable a hybrid simulation based test-
ing environment. The high-level architecture of such a system is depicted in
figure 2 below. A large-scale simulated environment would be combined with
an actual environment containing a prototype setup limited in scale. Both
environments are able to communicate with a supporting middleware sys-
tem. This way a full-scale EBI system can be deployed, tested, calibrated
and validated as a whole without the need to do a full deployment in an
actual environment. However, this will have a number of consequences to
the simulation technology being used. A distributed simulation environment
relying on a PADS-like architecture would be preferable. However, the meth-
ods that can be used are limited. An optimistic synchronization approach
is not realistic because we could not allow a simulation model to rewind its
state after an inconsistency is found as this would break the required real-
time property. As a result, only conservative synchronization methods are
feasible. Furthermore, we need to have a guarantee that all SE states are
updated in due time while maintaining consistencies, the blocking nature of
the conservative synchronization mechanism might result in state updates
not being processed at the proper rate.
Additionally, in the hybrid evaluation scenario a state synchronization
is required in the context of EBI applications. This mechanism needs to
ensure that actual prototype in the real-world perceives similar data as the
virtual, simulated devices. Therefore, the state of the simulation needs to be
16
synchronized with the state of the real-world For example, in the case of an
EBI systems related to air pollution, measurements of air pollution in the
virtual environment need to match those measured in the real world.
Figure 2: Hybrid evaluation scenario
Finally, from a model
partitioning perspective,
we must make sure that
SE migrations between
LP’s maintain the avail-
ability and responsive-
ness of the SE, e.g. it
could not be allowed
that a SE remains unre-
sponsive during the pro-
cess of migrating. We
identified many existing techniques that could help to improve the perfor-
mance and overall scalability of large-scale EBI IoT systems. In further
research we will focus on adding IoT and EBI related domain knowledge to
the aforementioned techniques and to better understand in which cases which
techniques are most appropriate.
7 Conclusion
In this position paper, we identified the challenges that arise when trying
to evaluate emergent behavior in Internet of Tings applications. A type
of application, where decentralized behavior is key. Furthermore, we pro-
posed the idea of a hybrid simulation environment that combines real-world
and simulated nodes for testing and deployment of these emergent behavior
IoT systems. As a solution, we propose a distributed simulation-based ap-
proach. However, relying solely on a simulation architecture is not enough
to cope with the scalability challenges of simulating emergent behavior. In-
stead, we present an overview of state-of-the-art techniques that could help
to improve simulation performance and overall scaling capabilities. In further
research we model and leverage the IoT domain knowledge on the techniques
mentioned in this study to develop a methodology for evaluating emergent
behavior in IoT systems.
17
References
[1] R. R. Barton. Metamodeling: a state of the art review. In Simulation
Conference Proceedings, 1994. Winter, pages 237–244. IEEE, 1994.
[2] L. Bononi, M. Bracuto, G. D’Angelo, and L. Donatiello. Proximity
detection in distributed simulation of wireless mobile systems. In Pro-
ceedings of the 9th ACM international symposium on Modeling analysis
and simulation of wireless and mobile systems, pages 44–51. ACM, 2006.
[3] A. Boukerche and S. K. Das. Dynamic load balancing strategies for
conservative parallel simulations. In Parallel and Distributed Simulation,
1997., Proceedings. IEEE, 1997.
[4] G. Carneiro. Ns-3: Network simulator 3. In UTM Lab Meeting April,
volume 20, 2010.
[5] D. Caughlin and A. F. Sisti. Summary of model abstraction techniques.
In Enabling Technology for Simulation Science, volume 3083, pages 2–
14. International Society for Optics and Photonics, 1997.
[6] K. M. Chandy and J. Misra. Distributed simulation: A case study in
design and verification of distributed programs. IEEE Transactions on
software engineering, (5):440–452, 1979.
[7] G. D’Angelo. Parallel and distributed simulation from many cores to the
public cloud. In High Performance Computing and Simulation (HPCS),
2011 International Conference on. IEEE, 2011.
[8] G. D’Angelo, S. Ferretti, and V. Ghini. Simulation of the internet of
things. In High Performance Computing & Simulation (HPCS), 2016
International Conference on, pages 1–8. IEEE, 2016.
[9] O. Developer. Omnet++ network simulation frame-work.
[10] A. Dunkels, B. Gronvall, and T. Voigt. Contiki-a lightweight and flex-
ible operating system for tiny networked sensors. In Local Computer
Networks, 2004. 29th Annual IEEE International Conference on, pages
455–462. IEEE, 2004.
18
[11] G. DAngelo. The simulation model partitioning problem: An adaptive
solution based on self-clustering. Simulation Modelling Practice and
Theory, 70:1–20, 2017.
[12] G. DAngelo and M. Marzolla. New trends in parallel and distributed
simulation: From many-cores to cloud computing. Simulation Modelling
Practice and Theory, 49:320–335, 2014.
[13] F. K. Frantz. A taxonomy of model abstraction techniques. In Pro-
ceedings of the 27th conference on Winter simulation, pages 1413–1420.
IEEE Computer Society, 1995.
[14] R. M. Fujimoto. Parallel and distributed simulation systems, volume
300. Wiley New York, 2000.
[15] R. Gore, S. Diallo, C. Lynch, and J. Padilla. Augmenting bottom-up
metamodels with predicates. Journal of Artificial Societies and Social
Simulation, 20(1), 2017.
[16] H. Gupta, A. Vahid Dastjerdi, S. K. Ghosh, and R. Buyya. ifogsim:
A toolkit for modeling and simulation of resource management tech-
niques in the internet of things, edge and fog computing environments.
Software: Practice and Experience, 47(9):1275–1296, 2017.
[17] D. R. Jefferson. Virtual time. ACM Transactions on Programming
Languages and Systems (TOPLAS), 7(3):404–425, 1985.
[18] L. Lamport. Time, clocks, and the ordering of events in a distributed
system. Communications of the ACM, 21(7):558–565, 1978.
[19] P. Levis, N. Lee, M. Welsh, and D. Culler. Tossim: Accurate and scal-
able simulation of entire tinyos applications. In Proceedings of the 1st
international conference on Embedded networked sensor systems, pages
126–137. ACM, 2003.
[20] C. M. Macal and M. J. North. Tutorial on agent-based modeling and
simulation. In Simulation Conference, 2005 Proceedings of the Winter,
pages 14–pp. IEEE, 2005.
[21] M. J. Mataric. Designing emergent behaviors: From local interactions
to collective intelligence. In Proceedings of the 2nd International Con-
ference on Simulation of Adaptive Behavior, 1993.
19
[22] K. Mehdi, M. Lounis, A. Bounceur, and T. Kechadi. Cupcarbon: A
multi-agent and discrete event wireless sensor network design and sim-
ulation tool. In Proceedings of the 7th International ICST Conference
on Simulation Tools and Techniques, pages 126–131, 2014.
[23] J. Misra. Distributed discrete-event simulation. ACM Computing Sur-
veys (CSUR), 18(1):39–65, 1986.
[24] A. Muzy, L. Touraille, H. Vangheluwe, O. Michel, M. K. Traor´e, and
D. R. Hill. Activity regions for the specification of discrete event systems.
In Proceedings of the 2010 Spring Simulation Multiconference, page 136.
Society for Computer Simulation International, 2010.
[25] D. M. Nicol. Principles of conservative parallel simulation. In Proceed-
ings of the 28th conference on Winter simulation, pages 128–135. IEEE
Computer Society, 1996.
[26] D. Roca, D. Nemirovsky, M. Nemirovsky, R. Milito, and M. Valero.
Emergent behaviors in the internet of things: the ultimate ultra-large-
scale system. IEEE micro, 36(6):36–44, 2016.
[27] J. D. Rodriguez, K. W. Bauer Jr, J. O. Miller, and R. E. Neher Jr.
Building prediction models of large hierarchical simulation models with
artificial neural networks and other statistical techniques. In Visual
Information Processing, page 69780, 2008.
[28] L. Sanchez, L. Mu˜noz, J. A. Galache, P. Sotres, J. R. Santana, V. Gutier-
rez, R. Ramdhany, A. Gluhak, S. Krco, E. Theodoridis, et al. Smart-
santander: Iot experimentation over a smart city testbed. Computer
Networks, 61:217–238, 2014.
[29] V. Stojkovski. Multi-abstraction, multi-formalism modelling and sim-
ulation: Integrating forrester system dynamics, cellular automata and
agent based modelling. Master’s thesis, University of Antwerp, Belgium,
2015.
[30] P. Symonds, J. Taylor, Z. Chalabi, and M. Davies. Performance of neural
networks vs. rbf when forming a metamodel for residential buildings.
International Journal Of Civil, Environmental, Structural, Construction
And Architectural Engineering, 9(12):1446–1450, 2015.
20
[31] Y. Van Tendeloo and H. Vangheluwe. Activity in pythonpdevs. In ITM
Web of Conferences, volume 3, page 01002. EDP Sciences, 2014.
[32] G. Wainer. Cd++: a toolkit to develop devs models. Software: Practice
and Experience, 32(13):1261–1306, 2002.
[33] B. P. Zeigler, H. Praehofer, and T. G. Kim. Theory of modeling and
simulation: integrating discrete event and continuous complex dynamic
systems. Academic press, 2000.
21
... The required cost and effort to deploy the vast amounts of IoT entities in the real world would otherwise be too high. Simulation techniques can be used to validate and verify if the demonstrated emergent behavior is preferable [6]. In this paper we will often refer to a Python IoT simulation framework that can be used to test both virtual and real-life, large-scale, complex adaptive IoT systems and allows for integration [5] with a real-life IoT environment. ...
Preprint
Full-text available
With the rise of complex Internet of Things systems we see an increasing need for testing and evaluating these systems. Especially, when we expect emergent complex adaptive behavior to arise. Agent based simulation is an often used technique to do this. However, the effectiveness of a simulation depends on the quality of individual models. In this work we look in depth what the characteristics are of Internet of Things devices, actors and environments. We look at how these characteristics can be used to find appropriate, performance optimized modeling techniques and formalisms. During the course of this work we will extensively refer to a custom-developed Internet of Things simulation framework and to relevant related literature.
... The required cost and effort to deploy the vast amounts of IoT entities in the real world would otherwise be too high. Simulation techniques can be used to validate and verify if the demonstrated emergent behavior is preferable [6]. In this paper we will often refer to a Python IoT simulation framework that can be used to test both virtual and real-life, large-scale, complex adaptive IoT systems and allows for integration [5] with a real-life IoT environment. ...
Chapter
With the rise of complex Internet of Things systems we see an increasing need for testing and evaluating these systems. Especially, when we expect emergent complex adaptive behavior to arise. Agent based simulation is an often used technique to do this. However, the effectiveness of a simulation depends on the quality of individual models. In this work we look in depth what the characteristics are of Internet of Things devices, actors and environments. We look at how these characteristics can be used to find appropriate, performance optimized modeling techniques and formalisms. During the course of this work we will extensively refer to a custom-developed Internet of Things simulation framework and to relevant related literature.
... Due to computational complexity and real-time requirements, we opt for a distributed approach. That is, multiple limited nodes control one vehicle each, based on the Agent Based Modelling (ABM) paradigm [4]. This means that each agent represents a vehicle with its own behaviour, while it is able to interact with the other present agents. ...
Article
Full-text available
To thoroughly test and validate algorithms and systems of autonomous vehicles, a large number of vehicles, many tests and a multitude of datasets are needed. This way of developing and testing is difficult, expensive and sometimes even dangerous. To combine the benefits of real world testing with the scalability and lower cost of simulation based testing, we present a novel methodology for a real-time hybrid simulator that is capable of handling real and simulated vehicles simultaneously with full interaction, in real time. We validated our methodology by assessing its overall performance and real-time capabilities using an F1/10 scale vehicle. The results effectively show the viability of this approach for validation of autonomous vehicles in a cost-efficient and safe manner.
Chapter
The DUST framework is a middleware platform developed to create software components that are distributable across heterogeneous networks. Developing and maintaining the component-based system and its configuration can become a challenge. In this paper, we present a web-based tool that allows engineers to visually model a DUST application graph in a CAD canvas. This graph is used by the back-end template generators to create the necessary configuration files and code for each module to implement in order to focus on the business logic. The current version of the tool is able to fully generate message objects and builder classes that are used for communication over DUST. Therefore, allowing the engineers to manage and update these objects through the GUI interface instead of error-prone coding.
Article
Full-text available
Metamodeling refers to modeling a model. There are two metamodeling approaches for ABMs: (1) top-down and (2) bottom-up. The top down approach enables users to decompose high-level mental models into behaviors and interactions of agents. In contrast, the bottom-up approach constructs a relatively small, simple model that approximates the structure and outcomes of a dataset gathered from the runs of an ABM. The bottom-up metamodel makes behavior of the ABM comprehensible and exploratory analyses feasible. For most users the construction of a bottom-up metamodel entails: (1) creating an experimental design, (2) running the simulation for all cases specified by the design, (3) collecting the inputs and output in a dataset and (4) applying first-order regression analysis to find a model that effectively estimates the output. Unfortunately, the sums of input variables employed by first-order regression analysis give the impression that one can compensate for one component of the system by improving some other component even if such substitution is inadequate or invalid. As a result the metamodel can be misleading. We address these deficiencies with an approach that: (1) automatically generates Boolean conditions that highlight when substitutions and tradeoffs among variables are valid and (2) augments the bottom-up metamodel with the conditions to improve validity and accuracy. We evaluate our approach using several established agent-based simulations.
Article
Full-text available
Recent advances in computing architectures and networking are bringing parallel computing systems to the masses so increasing the number of potential users of these kinds of systems. In particular, two important technological evolutions are happening at the ends of the computing spectrum: at the “small” scale, processors now include an increasing number of independent execution units (cores), at the point that a mere CPU can be considered a parallel shared-memory computer; at the “large” scale, the Cloud Computing paradigm allows applications to scale by offering resources from a large pool on a pay-as-you-go model. Multi-core processors and Clouds both require applications to be suitably modified to take advantage of the features they provide. Despite laying at the extreme of the computing architecture spectrum – multi-core processors being at the small scale, and Clouds being at the large scale – they share an important common trait: both are specific forms of parallel/distributed architectures. As such, they present to the developers well known problems of synchronization, communication, workload distribution, and so on. Is parallel and distributed simulation ready for these challenges? In this paper, we analyze the state of the art of parallel and distributed simulation techniques, and assess their applicability to multi-core architectures or Clouds. It turns out that most of the current approaches exhibit limitations in terms of usability and adaptivity which may hinder their application to these new computing architectures. We propose an adaptive simulation mechanism, based on the multi-agent system paradigm, to partially address some of those limitations. While it is unlikely that a single approach will work well on both settings above, we argue that the proposed adaptive mechanism has useful features which make it attractive both in a multi-core processor and in a Cloud system. These features include the ability to reduce communication costs by migrating simulation components, and the support for adding (or removing) nodes to the execution architecture at runtime. We will also show that, with the help of an additional support layer, parallel and distributed simulations can be executed on top of unreliable resources.
Conference Paper
Full-text available
In these days two main changes are revolutionizing the execution architectures used to run simulations: on the bottom level the processors (CPU) are gaining more and more cores while on the high level we need to cope with virtual resources: "everything as a service" in a Public Cloud infrastructure. Given the magnitude of these changes, what is going to happen to simulation? What are the (many) limits of current approaches, technologies and tools? Is it possible to finally find a solution to some of the many problems of PADS while broadening its scope? In this tutorial we aim to introduce all the basic aspects of these subjects and to discuss the main drawbacks of the current approaches. The main aim of the tutorial is to foster discussion and to increase the knowledge of some now undervalued technologies. The last part of the tutorial will be about our practical experience in the development of the ARTACE simulation middleware and in the proposal of a new paradigm for adaptive distributed simulation (called GAIA) that could be able to tackle with some of the issues described above. The tutorial will conclude with some examples derived from our experience in the performance evaluation of complex systems.
Article
To reach its potential, the Internet of Things (IoT) must break down the silos that limit applications' interoperability and hinder their manageability. Doing so leads to the building of ultra-large-scale systems (ULSS) in several areas, including autonomous vehicles, smart cities, and smart grids. The scope of ULSS is both large and complex. Thus, the authors propose Hierarchical Emergent Behaviors (HEB), a paradigm that builds on the concepts of emergent behavior and hierarchical organization. Rather than explicitly programming all possible decisions in the vast space of ULSS scenarios, HEB relies on the emergent behaviors induced by local rules at each level of the hierarchy. The authors discuss the modifications to classical IoT architectures required by HEB, as well as the new challenges. They also illustrate the HEB concepts in reference to autonomous vehicles. This use case paves the way to the discussion of new lines of research.
Article
Internet of Things (IoT) aims to bring every object (e.g. smart cameras, wearable, environmental sensors, home appliances, and vehicles) online, hence generating massive amounts of data that can overwhelm storage systems and data analytics applications. Cloud computing offers services at the infrastructure level that can scale to IoT storage and processing requirements. However, there are applications such as health monitoring and emergency response that require low latency, and delay caused by transferring data to the cloud and then back to the application can seriously impact their performances. To overcome this limitation, Fog computing paradigm has been proposed, where cloud services are extended to the edge of the network to decrease the latency and network congestion. To realize the full potential of Fog and IoT paradigms for real-time analytics, several challenges need to be addressed. The first and most critical problem is designing resource management techniques that determine which modules of analytics applications are pushed to each edge device to minimize the latency and maximize the throughput. To this end, we need a evaluation platform that enables the quantification of performance of resource management policies on an IoT or Fog computing infrastructure in a repeatable manner. In this paper we propose a simulator, called iFogSim, to model IoT and Fog environments and measure the impact of resource management techniques in terms of latency, network congestion, energy consumption, and cost. We describe two case studies to demonstrate modeling of an IoT environment and comparison of resource management policies. Moreover, scalability of the simulation toolkit in terms of RAM consumption and execution time is verified under different circumstances.
Article
This paper presents an overview of model abstraction methods. Model abstraction methods are techniques that derive simpler conceptual models while maintaining the validity of the simulation results. These methods include variable resolution modeling, combined modeling, multimodeling, and metamodeling. In addition, some taxonomies include approximation, aggregation, linear function interpolation, and look up tables as model abstraction methods. We discuss these methods in a general framework to assist in understanding the applicability of the various model abstraction methods. Bibtex entry for this abstract Preferred format for this abstract (see Preferences) Find Similar Abstracts: Use: Authors Title Abstract Text Return: Query Results Return items starting with number Query Form Database: Astronomy Physics arXiv e-prints
Article
Traditional discrete-event simulations employ an inherently sequential algorithm. In practice, simulations of large systems are limited by this sequentiality, because only a modest number of events can be simulated. Distributed discrete-event simulation (carried out on a network of processors with asynchronous message-communicating capabilities) is proposed as an alternative; it may provide better performance by partitioning the simulation among the component processors. The basic distributed simulation scheme, which uses time encoding, is described. Its major shortcoming is a possibility of deadlock. Several techniques for deadlock avoidance and deadlock detection are suggested. The focus of this work is on the theory of distributed discrete-event simulation.