Intelligent visual interface with the Internet of Things
University of the Aegean
University of the Aegean
University of the Aegean
Communication between users and physical objects and
sensors through the web within the Internet of Things
framework, requires by definition the capability to
perceive the sensors and the underlying information and
services. Visualization of the Things in IoT is thus a
requirement for natural interaction between users and IoT
instances in the upcoming but steadily established
computing paradigm. The immense quantity of sensors
and variety of usable information introduces the need to
intelligently filter and adapt the respective information
sources and layers. Current work proposes an architecture
that supports intelligent interaction between users and the
IoT addressing the intelligent perception requirement
described earlier. On the one hand, sensory visualization
is tackled via Augmented Reality layers of sensors and
information and on the other hand context and location
awareness enhance the system by providing usable in the
respective senses information.
Internet of Things; Natural interaction; Augmented
reality; Context Awareness; Markerless tracking
ACM CLASSIFICATION KEYWORDS
H.5.2 Human-centered computing: Systems and tools
for interaction design, Human-centered computing:
Ubiquitous and mobile computing systems and tools,
Human-centered computing: Visualization techniques
The emergence of the Internet of Things (IoT) promises a
wide range of new applications and services that will
shape our everyday life. The building blocks are smart
objects (SOs) around us that can interact with each other
and with the user, providing real-time personalization of
the system behavior according to the user’s preferences.
The IoT envisions interaction with billions of such SOs.
Valli  discusses the importance of «natural
interaction» and the need to design appropriate interfaces
and interaction models that will allow users to
communicate with the machine in ways that are more
natural to them than the use of a mouse or a keyboard. An
augmented representation of smart objects can act as a
natural interface that provides a better understanding of
the building blocks of the IoT infrastructure of the area,
giving users the ability to monitor and process the
operation of SOs from their own device. A marker is the
most common and simple way to track a device with an
augmented reality application . Even though it is a
widespread technique, these markers have considerable
limits when it comes to an IoT implementation, especially
when referring to non-dynamic markers.
Besides that, the precise localization provided by markers
is not important for an IoT application, as knowledge of
the presence of sensors is enough for the perception of a
smart system. So, even if accurate tracking fails to be
performed in many cases, the user will still benefit from
the procedure. However, there are cases in which precise
localization of an SO is crucial, for example when the
sensor’s data is meaningful only for its position like the
temperature of a specific object or when an SO needs
maintenance. But this may expose smart devices to
security dangers, especially in outdoor environments; so
in order to prevent that, access to precise localization
should be allowed only with special privileges.
Presenting the smart objects of the IoT as augmented
artifacts is one way to provide natural interaction to the
users. Let us consider the following scenario:
Lane is visiting a city that boasts an extensive use of
IoT services. Her initial experience is with a smart
transport system that plans her trips. Soon she starts
using more smart services but she still feels ignorant
of the numerous sensors and smart objects that
function all around her. She would like to have more
information about all those sensors, what they
measure and where they are located.
Lane started using the AR system on her smartphone.
Among other things she can now identify sensors
located around her, understand their function, enable
or disable them. As a result, she can use the services
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
for components of this work owned by others than ACM must be
honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from
SmartObjects'17, March 13 2017, Limassol, Cyprus
© 2017 ACM. ISBN 978-1-4503-4902-4/17/03…$15.00
provided more efficiently.
We propose a system architecture that combines the
sensory layer, the middleware and the user’s device,
adding context aware methods and smart object
management, discussing markerless tracking methods
which can be applied in our system and also focusing on
contextual matters . The rest of the paper is organized
as follows: Section 2 briefly reviews the related work. In
Section 3 we introduce our proposed framework,
analyzing issues like context awareness, security and
markerless tracking methods. Section 4 describes the
interaction methods in our system. Finally in Section 5 we
discuss our future plans and directions.
Several papers explore the use of AR systems in smart
spaces. Garcia et al  discuss the use of sentient
browsers for the Internet of Things by implementing a
prototype UbiVisor. Similar efforts can be found in  &
. All the efforts listed above introduce solutions limited
to a user device. Our system differs since it can be
integrated into the IoT infrastructure combining sensors,
middleware and AR components into a seamless platform.
Marker based techniques are being used in the majority of
visualization methods which integrate Augmented Reality
in the IoT framework. QR codes printed upon smart
devices or close to their position provide less complicated
detection methods but user’s devices must be a few
centimeters close to the marker for a successful
identification . Same issues appear in larger projects
which use thousands of QR codes to implement smart city
strategies . Bluetooth Low Energy is amongst the most
promising new hardware technologies for tagging
devices, opening new horizons for IoT applications .
PROPOSED SYSTEM ARCHITECTURE
Perera et al  reviewing the field, present a typical IoT
architecture that comprises of the middleware and the
Sensor Network, allowing Users to interact through
Applications. We enhance the depicted architecture as
shown in figure 1, in which the colored areas and arrows
show our additions to the original figure of the authors
The heterogeneity of smart objects deployed into a fully
realized Internet of Things needs to be addressed by the
use of middleware. The role of middleware is to provide
an abstraction layer between the sensory network and the
applications running in the environment. In our proposed
system, the middleware is enhanced with a Context
Aware Module and an SO Management component. Both
modules are necessary to provide a personalized list of the
SOs of the area to the AR System. Research for
middleware in IoT has been very active recently,
proposing many solutions .
The original architecture scheme did not illustrate the
usage of the User Device, which receives an upgraded
role in our enhanced version. The User Device module
consists of the User Profile component which is
responsible for transmitting the user access and
preferences and the AR component that is responsible for
the augmented visualization of the sensors.
The data flow starts at the sensor layer which
communicates directly with the middleware. The
transmitted data consists of both the sensed context
measured by the sensors (e.g. noise detection) and the
descriptive metadata of the sensors such as: ID,
description, capabilities, location (for the AR tracking).
We propose that the middleware should catalog and
manage the smart objects residing in their area of
influence, a daunting task that raises many challenges.
In particular, there is a need to define the area of control
attributed to each middleware. Although many spaces will
be constrained and easily attributed to a single
middleware (e.g. rooms in a Smart Home), some -mostly
public- spaces will be too large. A geospatial approach
could assign middleware to all SOs within a perimeter,
thus allowing overlapping areas, which may add
complexity but enhance the SO management. The
middleware system should be able to identify which smart
objects are in its area, their IDs and functions, their last
known location and other metadata. Also, as users moves
around, the user device should be able to hop from
middleware to middleware, sometimes even merging lists
from different sources, eventually presenting to the user
the actual SOs of interest around the area.
Responsible for the execution of the AR system, the user
device receives a list of smart objects, before visualizing
the SOs on the screen. In a typical scenario, a user enters
a smart space, with a smart device (smartphone). IoT
Fig. 1 Proposed IoT architecture
systems residing in that space track the entrance of a new
smart object and initiate communication protocols. The
middleware system responsible for the interaction with
the sensory network interacts with it, exchanging profiles.
Eventually, a list of all available SOs is transmitted to the
device, making it available to the AR program installed
The proposed system also tackles issues of multi-users.
What happens if two users antagonize for the same smart
object? How will the system respond to multiple
conflicting requests given at the same time frame? Part of
the answer to multiuser challenges lies to the correct
attributing of profiles to users, will be discussed in the
next part of this section. But eventually, equally
antagonizing users will need to be tackled with an
Garcia et al  argue that candidate visors browsing the
area for smart objects should satisfy the CA requirements
if they wish to be perceived as sentient. Adding context
aware computing, the list of SOs presented to the user is
filtered based on their preferences, privileges and the
general area context. As a result, the user is not confused
by the overexposure to irrelevant objects.
The Context Aware Module residing on the middleware
receives two context types:
a) sensed context
The sensor layer produces great amounts of sensed data,
all of which can be considered by the system as context.
Contextual variables like light and sound emissions or
traffic and people congestion have to be measured by the
sensors and transmitted to the CA reasoning engine of the
middleware. For example, based on sound filtering, the
engine may identify that privacy conditions are satisfied
and accordingly visualize SOs that the user may wish to
hide from public view.
b) user profile
The IoT infrastructure will cover many services and
procedures, some of which may have restricted access. A
public camera recording passengers for safety reasons
should not be available for all users but only those with
the appropriate privileges. User access data is thus
essential and will be transmitted from the user device to
the middleware when initiating their communication.
Apart from identity and privileges, the user profile can
provide user preferences that further personalize the
system behavior producing context information of higher
Fusing the sensed data with the user profile, the CA
module produces reasoned contextual information which
is meaningful and transmits it to the Smart Objects
Managing module which may decide to hide the smart
objects that are irrelevant in the specific context, or focus
on those that are better suited. E.g. a museum visitor will
probably not be interested in the exhibits displayed in an
open area when it is raining. Furthermore since the
procedure is executed by the middleware, it will be
responsible to capture changes in the environment, thus
providing an up-to-date perception of the situation.
Security and Privacy issues
As in all IoT applications, there are many security issues
that need to be answered. A new security risk added by
the proposed architecture is the danger of disclosing to
users the accurate position of public smart objects, which
may increase the risk of those objects being stolen. An
anti-theft technique could be applied: there can be two
types of SOs, those that are and those that are not critical
to being stolen. The first type of SOs may be decided to
transmit only a vague value of its exact location.
At the same time, privacy and trust issues are crucial for
the acceptance of IoT applications context from users.
Confidentiality, integrity and availability (known as the
CIA triad) need to be guaranteed while authorization and
authentication techniques must be established in order to
protect sensitive data and personal information. Our
proposed system comprehends many of the above
mechanisms so that it can handle shared data in
compliance with user needs and privileges .
Markerless tracking techniques
As we already discussed, markers are not an ideal solution
for the IoT applications so it is necessary to find
alternative techniques. For our proposed architecture, we
need to address the following challenges:
● Devices/sensors may change their position anytime.
● Users can access data of all devices/sensors inside a
certain area without being close to them.
● Devices/sensors in the area are displayed in the user’s
smart phone even if there is not a clear line of sight.
We examine two technologies which can address these
challenges. Firstly, the active RFID tag can be attached in
physical objects and communicates with an RFID-
middleware in charge through RF signals. As a result, the
real-time locations of all active RFID tags can be acquired
in a map provided by the middleware in charge that loads
in a tracking device which enters its area. Then, users can
interact with these tags through their camera view by
pointing their device to a direction according to the map.
Due to technology restrictions, we only have the
representation of each object in two coordinates but
cannot calculate their distance from the ground, so we
assume that all SOs are in a fixed height (1.5 m).
As a second technology we propose the Bluetooth Low
Energy (BLE) beacons. In this case, we present a different
approach for our architecture. Users no longer search
around for SOs in order to interact with, instead the IoT
devices are the ones which scan their area and give the
ability for any user who approaches them to use their
device and access their data. However, BLE beacons
expose many limitations regarding the IoT
implementation, especially on the interaction part, but
eventually new architectures will be introduced and tackle
these issues, revealing BLE’s potential in this field .
The interaction between humans and things is very crucial
for the successful deployment of the Internet of Things.
The Augmented Sensors system described in this paper
aims towards the direction of giving more control to the
user over the surrounding smart objects.
The challenge lies in the wide heterogeneity of existing
smart objects and sensors. If standardization efforts fail,
multiple protocols administered by various manufacturers
will be used. In such a scenario, the user’s device will
scan the area for smart objects and upon identification of
the desirable object will request access. The targeting
object will respond by transmitting the interaction
protocol it understands. The user’s device can then initiate
a communication based on the selected protocol which
may be known or not (can be downloaded from the Web).
Visualizing the smart objects as augmented sensors at the
screen of the user device is our proposal for more natural
interactions. The abstraction layer offered by the AR
system can hide the technical details of smart objects to
users not interested in those and allow them to interact
using the touch screen, which might not be possible due
to the size or positioning of those objects. Finally, the
wide range of system capabilities of user devices can also
let voice or gesture interfaces and other abstract ways of
communication fulfilling the user’s requirement for
In this paper we introduced an architecture that improves
the natural interaction between users and IoT
environments by visualizing the sensor layer through
augmented reality. It also embeds a context awareness
layer, which personalizes the AR experience while
depending on markerless tracking techniques, since
precise localization of smart objects only adds complexity
to the system without enhancing the perception of the IoT
experience. As a result, users are able to observe and
control smart objects through an augmented reality
representation by scanning the area around them with the
appropriate tracking device.
In the future, we plan to simulate our architecture in order
to review the middleware and software prototypes that
will have the ability to connect and share data between
SOs and tracking devices. Furthermore, we intend to
natural experience in a smart environment.
1. Ajanki, A., Billinghurst, M., Gamper, H., Järvenpää,
T., Kandemir, M., Kaski, S., Koskela, M., Kurimo,
M., Laaksonen, J., Puolamäki, K., Ruokolainen, T.
(2011). An augmented reality interface to contextual
information. Virtual reality, 15(2-3), 161-173.
2. García-Macías, J., Alvarez-Lozano, J., & Estrada-
Martinez, P. (2011). Browsing the internet of things
with sentient visors. Computer, 44(5), 46-52.
3. Gutiérrez, V., Galache, J. A., Sánchez, L., Muñoz, L.,
Hernández-Muñoz, J. M., Fernandes, J., & Presser, M.
(2013). Smartsantander: Internet of things research
and innovation through citizen participation. In The
Future Internet Assembly (pp. 173-186). Springer
4. Mohan, A., Woo, G., Hiura, S., Smithwick, Q., &
Raskar, R. (2009). Bokode: imperceptible visual tags
for camera based interaction from a distance. In ACM
Transactions on Graphics (TOG) (Vol. 28, No. 3, p.
5. Oh, S., & Woo, W. (2009). CAMAR: Context-aware
mobile augmented reality in smart space. In
Proceedings of International Workshop on Ubiquitous
Virtual Reality (pp. 15-18).
6. Perera, C., Zaslavsky, A., Christen, P., &
Georgakopoulos, D. (2014). Context aware computing
for the internet of things: A survey. IEEE
Communications Surveys & Tutorials, 16(1), 414-454.
7. Pointr, (2015 January 18), Beacons: Everything you
need to know, Retrieved from
8. Razzaque, M. A., Milojevic-Jevric, M., Palade, A., &
Clarke, S. (2016). Middleware for internet of things: a
survey. IEEE Internet of Things Journal, 3(1), 70-95.
9. Sato, K., Sakamoto, N., & Shimada, H. (2015).
Visualization and Management Platform with
Augmented Reality for Wireless Sensor Networks.
Wireless Sensor Network, 7(01), 1.
10. Sicari, S., Rizzardi, A., Grieco, L. A., & Coen-
Porisini, A. (2015). Security, privacy and trust in
Internet of Things: The road ahead. Computer
Networks, 76, 146-164.
11. Valli, A. (2008). The design of natural interaction.
Multimedia Tools and Applications, 38(3), 295-305
12. Want, R., Schilit, B. N., & Jenson, S. (2015). Enabling
the Internet of Things. IEEE Computer, 48(1), 28-35.