Internet of Everything: A Large-Scale Autonomic IoT Gateway

Full text

IEEE Proof
1Internet of Everything: A Large-Scale
2Autonomic IoT Gateway
3Byungseok Kang, Daecheon Kim, and Hyunseung Choo, Member, IEEE
4Abstract—Gateways are emerging as a key element of bringing legacy and next generation devices to the Internet of Things (IoT).
5They integrate protocols for networking, help manage storage and edge analytics on the data, and facilitate data flow securely between
6edge devices and the cloud. Current IoT gateways solve the communication gap between field control/sensor nodes and customer
7cloud, enabling field data to be harnessed for manufacturing process optimization, remote management, and preventive maintenance.
8However, these gateways do not support fully-automatic configuration of newly added IoT devices. In this paper, we proposed a
9self-configurable gateway featuring real time detection and configuration of smart things over the wireless networks. This novel
10 gateway’s main features are: dynamic discovery of home IoT device(s), automatic updates of hardware changes, connection
11 management of smart things connected over AllJoyn. We use the ‘option’ field for automatic configuration of IoT devices rather than
12 modify standard format of CoAP protocol. Proposed gateway functionality has been validated over the large-scale IoT testbed.
13 Index Terms—IoT, IoT gateway, self-configuration, large scale IoT
Ç
14 1INTRODUCTION
15 INTERNET of Things (IoT) is a new paradigm built up with
16 a continuum of uniquely addressable things which is able
17 to communicate with each other through a Internet, with
18 the bolster of approaches and protocols such as pervasive
19 computing, ubiquitous computing, sensing technology,
20 Constrained Application Protocol (CoAP) [1], IPV6 and
21 other protocols. It is a system that bridges physical and
22 cyber world is formed to enable symbiotic communication
23 seamlessly between the two parties [2]. It means that, IoT
24 gives a vision where a large network of uniquely identified
25 smart things with different end devices such as sensors and
26 actuators connected at any-time, any-place, any-thing,
27 working together to provide variety of services on demand
28 to the customers [3], [4].
29 An autonomic networking [5], [6] is the network that has
30 the capabilities of being self-defining, self-healing, self-con-
31 figuring, self-optimizing, etc. Started by IBM in 2001, this
32 initiative ultimately aims to develop network systems capa-
33 ble of self-management, to overcome the rapidly growing
34 complexity of network systems management, and to reduce
35 the barrier that complexity poses to further growth [7], [8].
36 The network makes decisions on its own, using high-level
37 policies; it will constantly check and optimize its status and
38 automatically adapt itself to changing network conditions.
39 An autonomic networking framework is composed of auto-
40 nomic components (AC) interacting with each other. An AC
41can be modeled in terms of two main control loops (local
42and global) with sensors (for self-monitoring), effectors (for
43self-adjustment), knowledge and planner/adapter for
44exploiting policies based on self- and network environment
45awareness [9], [10].
46The huge number and large-scale devices mandate the use
47of gateways for Internet services. There are various kinds of
48access technologies for this gateway design. While 4G/LTE
49could be a good solution for many applications, it suffers
50from increasing packet collisions with increasing amounts of
51downlink access. Furthermore, traditional cellular network is
52also not suitable for quality of service (QoS) support and con-
53sumes a fair amount of power [11], [12]. As the proprietary
54and unlicensed solutions are not optimized for spectral effi-
55ciency, with exponential increase in IoT deployment, these
56solutions are very likely to congest the unlicensed bands and
57trigger complaints from existing customers. At the same
58time, a variety of new wide area applications (e.g., smart cit-
59ies, traffic monitoring, and smart grids) for IoT are offering
60new markets to wireless operators for enhancing their reve-
61nues. As a result, the Third Generation Partnership Project
62(3GPP) has recently decided to include narrowband IoT (NB-
63IoT) in Release more than 10 standards [13].
64With this context, large-scale IoT gateway is one of the
65challenges in IoT industry. However, current IoT gateways
66operate with passive or semi-automatic modes. [14], [15] In
67other words, when the customer buys a new IoT device,
68they manually install the device based on the setup manual.
69Moreover, IoT gateway asks the user whether to register a
70new device or not. To counter these limitations, in this
71paper, we propose a self-configurable IoT gateway for the
72large-scale IoT environment. When the users bring a new
73IoT device into existing wireless networks, IoT gateway
74automatically detects and registers it. If the user throws out
75old devices from their place, proposed IoT gateway auto-
76matically delete that device form the device list (database).
The authors are with the Department of Computer Science and Engineering,
Sungkyunkwan University, Suwon 440-746, Korea.
E-mail: {byungseok, daecheon, choo}@skku.edu.
Manuscript received 14 Feb. 2017; revised 19 Apr. 2017; accepted 15 May
2017. Date of publication 0 . 0000; date of current version 0 . 0000.
(Corresponding author: Hyunseung Choo.)
Recommended for acceptance by S. Bhunia.
For information on obtaining reprints of this article, please send e-mail to:
reprints@ieee.org, and reference the Digital Object Identifier below.
Digital Object Identifier no. 10.1109/TMSCS.2017.2705683
IEEE TRANSACTIONS ON MULTI-SCALE COMPUTING SYSTEMS, VOL. 3, NO. X, XXXXX 2017 1
2332-7766 ß2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

IEEE Proof
77 Proposed system saves user from trouble of registering,
78 installing, and managing smart devices. The state-of-the-art
79 AllJoyn framework provides convenient user interface with-
80 out modifying standard source codes. To validate proposed
81 gateway, we developed large-scale IoT experimental testbed
82 and measured several QoS factors. The remainder of this
83 paper is organized as follows. Related work on IoT gateway
84 and platform technologies is presented in Section 2, Section 3
85 describes our proposed IoT gateway, experimental testbed
86 development and experimental results are provided in
87 Section 4, Section 5 describe conclusions and future work.
88 2RELATED WORK
89 IoT gateway supports a variety of communication protocols
90 and data types between various sensors, which can realize
91 the conversion of data format which communicated
92 between varieties of sensors to unify the uploaded data for-
93 mats (see Fig. 1). At the same time, the acquisition or control
94 command which reach at the perception network are
95 mapped to produce messages that meet specific device com-
96 munication protocol. In this section, we discuss the most
97 known contributions in the literature.
98 The goal of IoT gateway is to bridge various sensing
99 domain networks with public communication networks or
100 Internet, settle with the heterogeneity between these various
101 networks, strengthen the management of both IoT gateway
102 itself and terminal nodes. We mainly classify the IoT gate-
103 ways into three types: passive, semi-automatic, and fully-
104 automatic. The proposed gateway belongs to fully-
105 automatic IoT gateway (see Fig. 2).
106 2.1 Passive Gateways
107 Passive gateways require manual settings for the new IoT
108 devices. The users or IoT engineers have to manually add
109new devices and remove old devices. In [16], authors intro-
110duced a generic gateway-based framework for WSN-IP net-
111work interconnection. Their framework allow access to
112individual sensor nodes and enables network-based query
113for data-centric WSN. In addition, they provided transpar-
114ent access from one network to the other without modifying
115protocols running in either network. Work [21] proposed
116IoT Gateway system based on Zigbee and generalized radio
117packet service (GPRS) protocols according to the typical IoT
118application scenarios and requirements from telecom opera-
119tors. It presented the data transmission between wireless
120sensor networks and mobile communication networks.
121However, those two gateways are not flexible, because they
122cannot be customized for different applications.
123In [17], authors proposed a generic sensor network plat-
124form (SwissQM/SwissGate) that provided a high level
125interface for programming sensor networks and also pre-
126sented a multi-tier architecture for efficiently handling and
127optimizing the operation of the network. Research [18]
128introduced a smart gateway for specific kinds of IoT appli-
129cations. The gateway has full knowledge and control both
130over the sensor network and the Internet. They can act as
131performance-enhancing proxies and intelligent caches to
132preserve the limited resources of the sensor network.
133Authors [22], proposed sensor network middleware that
134can translate sensed data from sensor networks using
135extended markup language (XML) and provide translated
136data for various Web applications. However, most tasks are
137completed by different network-dependent sensing servers
138not a smart gateway. It is observed that the required hard-
139ware cost will become too high.
1402.2 Semi-Automatic Gateways
141Semi-automatic gateways generate a connection link
142between new device and gateway. However, they cannot
143support automatic link for device attributes (or functions).
144For instance, when a user buys a new TV in living room, the
145gateway creates a connection link to TV itself not its
146attribute. It means that the user cannot switch the TV chan-
147nel and adjust volume. Authors [19], designed a plug-
148configurable-play service-oriented gateway. It was aimed at
149making fast and easy to employ various external sensor net-
150work applications. Work of [23] proposed IoT gateway with
151multi-channel RS485. It ensures the real-time performance
152and reliability for IoT while transferring data between the
153application and the underlying layer. It resulted in improv-
154ing the organization and management efficiency of
155endpoint devices and solving the problems of data trans-
156mission among different protocols. However, the gateway
157is running on PC which demands for higher hardware envi-
158ronment. Thus, the advantages in the actual IoT applica-
159tions are not obvious.
Fig. 2. Classification of IoT gateways.
Fig. 1. General feature of IoT gateway.
2 IEEE TRANSACTIONS ON MULTI-SCALE COMPUTING SYSTEMS, VOL. 3, NO. X, XXXXX 2017

IEEE Proof
160 In [15], authors proposed a novel configurable smart IoT
161 gateway. This gateway has plug-able architecture, whose
162 modules with different communication protocols can be
163 customized and plugged in according to different networks.
164 In addition, the gateway has unified external interfaces in
165 accordance with flexible software development. Work [20],
166 presented a multi-interface gateway. It can be used in spe-
167 cific smart spaces to automatically control traditional TV,
168 air conditioner, smart meter, sphygmomanometer, and
169 smart phone. The work of [24] supported heterogeneous
170 sensor networks and access networks. moreover, it can sup-
171 port different types of sensor nodes and access methods,
172 and can provide a unified data format for middleware or
173 applications. However, all above work do not support auto-
174 nomic attribute generation of new devices.
175 3LARGE-SCALE AUTONOMIC IOTGATEWAY
176 To support full automatic IoT device registration and solve
177 the heterogeneous network transmission problems, this
178 paper aimed at designing and implementing a self-configu-
179 rable gateway. It plays the role of analyzer of different type
180 of IoT devices. The designed gateway is able to communi-
181 cate with a variety of smart objects using 3G/4G, LTE, Wi-
182 Fi, ZigBee, Bluetooth and IrDA standards. This section pro-
183 vides an overview of the large-scale IoT testbed, introducing
184 its key design considerations and the real testbed.
185 3.1 Design Considerations
186 The gateway is a key component of every IoT solution. As
187 we mentioned in Section 1, our main objective was to build
188 an experimental facility that allows large-scale experimenta-
189 tion with heterogeneous IoT technologies deployed in a
190real-world setting, where real-world data and feedback can
191be obtained from users and their environment under realis-
192tic experimental conditions. While our final ambition is to
193cover both indoor and outdoor environments on our
194research center (see Fig. 3).
195In addition to this, the testbed infrastructure should sup-
196port autonomic registration, installation, and experimenta-
197tion cycles for the different envisioned IoT techniques,
198smart objects and allow for the evaluation of those in an
199interdisciplinary context. We chose to implement the facility
200as a living lab in our research center called ICT convergence
201research center, where each employee can become part of
202the experiment during his or her daily activities. In order to
203increase the flexibility of the testbed to cater for demands of
204diverse experiments, we deployed in each room a mix of
205heterogeneous IoT devices, implementing a wide range of
206sensing modalities and common communication interfaces.
207This infrastructure is complemented with mobile experi-
208mentation nodes that can be carried around by end users
209and fixed interaction displays in the infrastructure provid-
210ing additional means for the communications.
2113.2 Architecture
212Fig. 4 provides an overview of the IoT architecture of our
213testbed. Three main devices can be identified: i) a Server
214system that hosts all the back-end functionalities of the
215testbed and provides the entry point for an experimenter to
216access the testbed, ii) an embedded Gateway (GW) tier
217which forms the testbed infrastructure and allows the
218iii) IoT tier to be connected and reachable to a backbone net-
219work through 4G/LTE or WiFi. Although all the devices
220can be involved in each experimentation phase, the IoT
221device represents the user-centric component of our testbed,
Fig. 3. Large-scale IoT testbed.
KANG ET AL.: INTERNET OF EVERYTHING: A LARGE-SCALE AUTONOMIC IOT GATEWAY 3

IEEE Proof
222 merging embedded IoT nodes with sensing capabilities
223 together with higher-end user-centric devices such as
224 Smartphones and Smart Displays. Each IoT node provides
225 two forms of connectivity: i) wireless communication capa-
226 bilities (IEEE 802.15.4, and through the GW devices WiFi
227 and Bluetooth) that can be exploited during an experiment
228 in order to form different kinds of networks, ii) a wired USB
229 connection to a dedicated GW for management purposes.
230 Differently, Smartphones and Smart Displays are connected
231 only through wireless links (Bluetooth or WiFi) either
232 through GWs or directly to the backbone network.
233 The typical architecture of IoT gateway is usually far more
234 complex than the architecture of most enterprise systems.
235 One of the main factors that increases the complexity of IoT
236 systems is that backend services residing in the data center,
237 which is the heart of most enterprise systems, are actually
238 just a piece of the bigger IoT picture. With IoT gateway, we
239 have to deal with a myriad of devices working in the field.
240 Because the nature of these devices is very different from
241 web, desktop, or even mobile clients, we need an intermedi-
242 ate architectural element that will act as a proxy between the
243 world of field devices and the enterprise data center.
244 For the IoT network testbed development, we use a All-
245 Joyn framework [25]. The AllJoyn framework runs on the
246 local network. It enables devices and apps to advertise and
247 discover each other. This section explains the network archi-
248 tecture and the relationship between various AllJoyn com-
249 ponents. In addition, this framework comprises AllJoyn
250 Apps and AllJoyn Routers, or Apps and Routers for short.
251 Apps communicate with Routers and Routers communicate
252 with Apps. Apps can only communicate with other Apps
253 by going through a Router. Fig. 5 shows the basic topology
254 of AllJoyn network architecture. Apps and Routers can live
255 on the same physical device, or on different devices. From
256 an AllJoyn perspective, it doesn’t matter. In reality, three
257 common topologies exist:
258 1) An App uses its own Router. In this case, the Router
259 is called a “Bundled Router” as it is bundled with
260 the App. AllJoyn Apps on mobile OSes like Android
261 and iOS and desktop OSes like Mac OS X and Win-
262 dows generally fall in this group.
263 2) Multiple Apps on the same device use one Router. In
264 this case, the Router is called a “Standalone Router”
265and it typically runs in a background/service pro-
266cess. This is common on Linux systems where the
267AllJoyn Router runs as a daemon process and other
268AllJoyn apps connect to the Standalone Router. By
269having multiple apps on the same device use the
270common AllJoyn Router, the device consumes less
271overall resources.
2723) An App uses a Router on a different device. Embed-
273ded devices (which use the Thin variant of the AllJoyn
274framework, more on this later) typically fall in this
275camp as the embedded device typically does not have
276enough CPU and memory to run the AllJoyn router.
2773.2.1 Gateway Software
278The software application is the heart of the IoT gateway.
279The gateway software is responsible for collecting messages
280from the sensors and storing them appropriately until they
281can be pre-processed and sent to the data center. The gate-
282way software decides if the data at a given stage of process-
283ing should be temporary, persistent, or kept in-memory.
284The gateway software should be designed with failure
285and disaster recovery in mind. Since gateway devices are
286often operated in the field, we should prepare for working
287conditions that are far from ideal. For example, the gateway
288software should be prepared for a power outage or other
289actions that may result in an interruption of gateway proc-
290essing. The gateway software should be bootstrapped and
291started automatically as soon as power returns to the device,
292and it should continue to work from the point where it was
293interrupted. Gateway software should also be autonomic
294enough to properly handle system logging. It has to find the
295right balance between the number of log entries stored
296on the various kinds of IoT devices and those sent to the
297data center.
298An IoT application comprises app code, service frame-
299works libraries, and core library. First, app code is the appli-
300cation logic of the application. It can be programmed to
301either the service frameworks libraries, which provide
302higher level functionality, or the core library, which pro-
303vides direct access to the core APIs. Second, service frame-
304works libraries implement a set of common services, like
305onboarding, notification, or control panel. By using the com-
306mon service frameworks, apps and devices can properly
307interoperate with each other to perform a specific function-
308ality. Finally, core library provides the lowest level set of
Fig. 5. Network architecture of AllJoyn framework.
Fig. 4. Three main devices.
4 IEEE TRANSACTIONS ON MULTI-SCALE COMPUTING SYSTEMS, VOL. 3, NO. X, XXXXX 2017

IEEE Proof
309 APIs to interact with the IoT network. It provides direct
310 access to advertisements and discovery, session creation,
311 interface definition, and object creation/handling. We use
312 these APIs to implement large-scale IoT service frameworks
313 and private interfaces.
314 3.2.2 Gateway Data Transfer
315 Usually gateways are connected to the Internet using GPS,
316 WiFi, or Ethernet. Some gateways can also work in both GPS
317 and WiFi modes (for example, gateways mounted in moving
318 vehicles). In general, non-GPS connectivity is preferred to
319 send data, as it doesnt require a subscription toa paid mobile
320 plan. Some gateways will be constantly connected to inex-
321 pensive local networks, but those using GPS connectivity
322 should be very conservative in terms of what data they send
323 to the data center. The gateway should apply service logic
324 against the data it collects to understand which messages
325 should be sent over expensive GPS networks, and which data
326 can be cached on the device for deferred offline processing.
327 The messages collected by the gateway from the IoT
328 device (sensors and actuators) are usually small in size. For
329 example, the current value of the temperature measured by
330 the sensor is just a decimal number. GPS coordinates are
331 two decimal numbers, which represent longitude and lati-
332 tude. This is an important thing to remember: the gateway
333 operates on a large number of small messages.
334 3.3 Real-Time Device Monitoring (RTDM)
335 Real-time data monitoring [26], [27] is a process through
336 which an administrator can review, evaluate and modify
337 the addition, deletion, modification and use of data on soft-
338 ware, a database or a system. It enables data administrators
339 to review the overall processes and functions performed on
340 the data in real time, or as it happens, through graphical
341 charts and bars on a central interface/dashboard.
342 It is critical to have a good knowledge about the condi-
343 tion of the current IoT devices in order to appropriately
344 maintain the device table. In order to be able to meet
345 demands and provide satisfactory QoS, device monitoring
346 mechanisms are required and can lead to collection and
347 processing of a certain amount of runtime data. To monitor
348 the IoT device and sensors continuously, we used CoAP [1]
349 to collect device and attributes information (e.g., Device ID,
350 Name, Join date).
351 CoAP is a specialized web transfer protocol for use with
352 constrained nodes and constrained networks in the Internet
353 of Things. This protocol is designed for M2M applications
354 such as smart home and building automation. A CoAP
355 packet is sent periodically on each IoT device to discover
356 and test connections. CoAP packets are broadcasting to
357enable dynamic discovery of every devices and sensors.
358The detail information of CoAP fields in the header are
359defined in IETF RFC document [1].
360As shown on Tables 1 and 2, proposed gateway uses
361device and attribute tables, respectively. Every intercon-
362nected IoT device information is periodically updated by
363CoAP packet through ACK packet that is being sent by the
364IoT device. The device Table contains ID, Name, Number of
365attributes, and joining date (see Table 1). The attribute table
366includes ID, attribute name, and attribute function, last
367update (see Table 2).
3683.4 Self-Configuration Scheme (SCN)
369A Self-Configuration Network [28], [29] is an automation
370technology designed to make the planning, configuration,
371management, optimization and healing of mobile radio
372access networks simpler and faster. SCN functionality and
373behavior has been defined and specified in generally
374accepted mobile industry recommendations produced by
375organizations such as 3rd Generation Partnership Project
376and the Next Generation Mobile Networks (NGMN).
377SCN has been codified within 3GPP Release 8 and subse-
378quent specifications in a series of standards including
37936.902, [30] as well as public white papers outlining use
380cases from the NGMN [31]. The first technology making use
381of SCN features will be LTE network, but the technology
382has also been retro-fitted to older radio access technologies
383such as Universal Mobile Telecommunications System
384(UMTS) [32]. The LTE specification inherently supports
385SCN features like Automatic Neighbor Relation (ANR)
386detection, which is the 3GPP LTE Rel. 8 flagship feature [33].
387In our case, we use a Centralized SCN (C-SCN). In C-
388SCN, function is more typically concentrated closer to
389higher-order network nodes or the network Operations sup-
390port systems (OSS), to allow a broader overview of more
391edge elements and coordination of, e.g., load across a wide
392geographic area. Due to the need to inter-work with cells
393supplied by different equipment vendors, C-SCN systems
394are more typically supplied by 3rd parties.
395Self-configuration strives towards the “plug-and-play”
396paradigm in the way that new base stations shall automati-
397cally be configured and integrated into the IoT network.
398This means both connectivity establishment, and download
399of configuration parameters are software. Self-configuration
400is typically supplied as part of the software delivery with
401each radio cell by equipment vendors. When a new IoT
402device is introduced into the network and powered on, it
403gets immediately recognized and registered by the network.
404The IoT gateway then automatically adjust their technical
405parameters in order to provide the required function and
406capacity, and, in the same time, avoid the device conflict.
TABLE 1
Informations of Device Table
ID Name Num. of attributes Joining date
D070410 TV 4 2014.08.23
D431831 HVAC control 2 2015.02.01
S235342 Smoke sensor 0 2016.04.04
S889001 Temp sensor 0 2012.11.20
D121110 Lamp 1 2012.11.20
S900010 Light sensor 0 2012.11.20
TABLE 2
Informations of Attribute Table
ID Attribute name Attribute function Last update
D070410 Channel up, down 2016.08.20
D070410 Volume up, down 2016.08.20
D070410 Power on, off 2016.08.20
D070410 Mute on, off 2016.08.20
D121110 Lamp on, off 2016.08.20
KANG ET AL.: INTERNET OF EVERYTHING: A LARGE-SCALE AUTONOMIC IOT GATEWAY 5

IEEE Proof
407 The IoT gateways currently available, allow the user(s) to
408 configure a few properties such as TV channel, light, refrig-
409 erator, etc. However, large-scale environment, such as those
410 based on software-defined networks, are likely to have very
411 rich configuration controls. The goal of this research is to
412 develop algorithms that enable IoT gateway autonomic con-
413 figuration. We propose an initial study of auto-configura-
414 tion that considers two properties: dynamic discovery and
415 auto-registration.
416 4EXPERIMENTAL RESULTS
417 In the following section, we provide a large-scale testbed
418 development for IoT experimentation and evaluation of
419 proposed IoT gateway. We concentrate measurement on
420 three main dimensions: device setup response time, energy
421 consumption and monitoring overhead. We further discuss
422 the extent to which the available testbeds fulfill the require-
423 ments for future IoT testbeds introduced in the Section 2.
424 4.1 Large-Scale Testbed
425 In [34], the authors survey relevant experimental wireless
426 sensor network testbeds available to the community today.
427 The most advanced and active testbed in that survey is the
428 “SmartSantander” project which offers an experimental
429 research facility at the scale of a city, and which can be used
430 to test smart city applications and services. The Smart-
431 Santander project targets a 20,OOO-node network. On the
432 other hand, loT-LAB [35] is a federation of platforms, and is
433 part of OneLab (Internet-overlaid, Broadband access, wire-
434 less loT). In the SmartSantander project, all nodes—called
435 “service nodes” (battery-constrained nodes)—only produce
436 data and can be configured only by the administrators of
437 the testbed. They are, however, not open to be reprog-
438 rammed by users. Some nodes (with fewer battery con-
439 straints) can be reprogrammed so users can test their own
440 protocols. Both approaches are complimentary: Smart-
441 Santander targets smart city applications and experimenta-
442 tion at the loT application level, loT-LAB give bare-metal
443 access to all nodes in all sites.
444 Development of prototypes for large-scale IoT systems is
445 challenging. A few hundred IoT nodes can be deployed on
446 the IoT platform, but it is not easy to get repeatable results
447 when using a shared and public infrastructure. To over-
448 come this critical hurdle, we offers a large-scale federated
449 experimental platform allowing researchers, loT designers,
450 developers and engineers to construct, benchmark and opti-
451 mize their protocols, applications and services. As a
452 state-of-the-art testbed, our goal is to answer the needs and
453 requirements of current and future loT technology. In par-
454 ticular, it offers: (i) a heterogeneous and rich environment
455 (e.g., IoT hardware, topologies, OS, platform, up-to-date
456 standardized protocol stacks and libraries) applicable to a
457 large spectrum of loT services; (ii) the ability to instrument
458 an experiment through virtualization and reproducibility
459 tools; (iii) the ability to manage, interact with and monitor
460 running experiments.
461 The large-scale IoT testbed (originally installed for moni-
462 toring building energy consumption) has been imple-
463 mented on the ICT research center, 2nd Engineering Bld.,
464 Sungkyunkwan University in Suwon, Korea. It aims to
465collect sensing data from a set of areas of offices (e.g., meet-
466ing area, relaxing area, and work area) and the lecture
467room. The deployed sensors (for measuring indoor climate,
468energy consumption of office utilities, peoples presence in
469offices, and parking lot status) collect information about the
470physical status of indoor and outdoor building environ-
471ment, and transfer it to the IoT server platform, AllJoyn
472standard-compatible server platform, which allows further
473processing and analysis.
474The testbed is composed of 40 compound sensors, each of
475them having four kind of raw sensors (temperature, humid-
476ity, illumination and presence sensor), 10 CO2 (Carbon
477dioxide) concentration detection sensors, 10 smart sockets
478for measuring the electrical power consumption, and 20
479parking lot sensors, with total of 200 sensors (i.e., 160 raw
480sensors + 10 CO2 + 10 sockets + 20 parking sensors). Table x
481summarizes IoT devices supported by the University
482testbed that will be available in the scope of the IITP-IoT
483federation.
4844.2 System Measurements
485One of the key issues of the application based on IoT is
486extracting the analysis result from huge amount of collected
487sensing data and recognizing users context from recent
488incoming sensing data to provide customized real-time ser-
489vice [36], [37]. Therefore, the response time is one of the
490most important criteria of user satisfaction on IoT service.
491We also consider dynamical change of IoT devices which
492comes from their resource constraint and mobility, as a
493result some devices leave from the network and some
494device join to the network for replacing broken-down or left
495devices. The average response time for device registration
496while increasing the number of IoT device gives a good
497indication of both the general performance and the dynam-
498ics of the system. We show the response time prediction
499and its dependence on bottleneck utilization in the system.
500Fig. 6 shows the service time, delay and response time in
501IoT server. We see that average device response time is less
502than 25 ms. In addition, average service time + delay is
503resulted around 12 ms.
504The definition for the lifetime of a IoT sensor node usu-
505ally depends upon battery lifetime [38], [39]. If the mote
506were to stop transmitting and never transmits again, the
507definition would be helpful. However, as the experiments
508show a new definition is required to account for motes that
509stop transmitting, but could potentially begin transmitting
510at a later time. Multiple tests were necessary to build an
511accurate model. Results measuring the direct and indirect
512energy consumption in each IoT sensor are presented and
513used to derive an basic energy consumption model and
514device [40], [41]. Fig. 7 shows the average energy consump-
515tion of temperature sensors while changing the indoor tem-
516perature. we can estimate the expected energy consumption
517and system overhead based on this result.
518Data that we send across an IoT (wireless) network is
519housed in a data envelope called a packet. Each transmis-
520sion includes additional information, called overhead, that
521is required to route the data to the proper location. We can
522calculate network overhead by sending a fixed-size data
523transmission across the network and observing the number
524of extra bytes of data transmitted for the action to be
6 IEEE TRANSACTIONS ON MULTI-SCALE COMPUTING SYSTEMS, VOL. 3, NO. X, XXXXX 2017

IEEE Proof
525 completed. Fig. 8 shows the endpoint device monitoring
526 overhead at the IoT server. First, we measured level of wire-
527 less network congestion while chaning the monitoring fre-
528 quency (top-left). In other words, IoT gateway sends the
529 monitoring packets to all endpoints with different fre-
530 quency. Second, bottom-left figure depicts a comparison of
531 the measured response time against our predictions during
532 50 minutes. The result shows that monitoring overhead has
533 increased continuously. Finally, left figure shows normal-
534 ized histogram of overhead during 10 minutes. The average
535 value of entire network overhead resulted around 1,000
536 control packets.
537 5CONCLUSION
538 The internet of things is being termed as the future of internet
539 communication. This work is an attempt to integrate sensors
540 and actuators with large-scale environment and wireless gate-
541 way that could fit in most common small devices today. We
542 have introduced a general IoT architecture to provide novel
543 Machine to Machine (M2M) services [42], [43], [44]. The
544 mobile clients interact with the M2M devices and endpoints
545 through the IoT gateway. The M2M services including device
546 discovery, local storage of resource configurations are
547structured around the gateway. The architecture is capable of
548interacting with a non-smart device over AllJoyn framework.
549Detail testbed development and system measurement are
Fig. 7. Average energy consumption of IoT devices (temperature
sensors).
Fig. 6. Average response time and CPU overhead at the IoT Server.
KANG ET AL.: INTERNET OF EVERYTHING: A LARGE-SCALE AUTONOMIC IOT GATEWAY 7

IEEE Proof
550 discussed. Such real testbed can be used in several real life sce-
551 narios including personal health monitoring, environmental
552 monitoring and semantics in IoT. The usefulness of the system
553 lies in the fact that it provides an incremental methodology to
554 add new services and is generic enough to be compliant with
555 future standards in M2M communications and IoT. The sys-
556 tem is scalable to handle huge amount of traffic as it is based
557 on RESTful paradigm and SenML is very lightweight which
558 can be processed very fast. As future direction, the gateway
559 interfaces and their functionalities are being implemented
560 using updated AllJoyn framework, access rights for multiple
561 users and security/privacy of IoT architecture are being
562 investigated.
563 ACKNOWLEDGMENTS
564 This work was supported by the G-ITRC Program under
565 Grant IITP-2015R6812-15-0001, the NRF Research Fellow
566 program under Grant NRF-2016R1A6A3A11934080, and
567 the NRF Korea under Grant 2010-0020210.
568 REFERENCES
569 [1] C. Bormann, A. P. Castellani, and Z. Shelby, “CoAP: An applica-
570 tion protocol for billions of tiny internet nodes,” IEEE Internet
571 Comput., vol. 16, no. 2, pp. 62–67, 2012.
572 [2] N. Jazdi, “Cyber physical systems in the context of industry 4.0,”
573 in Proc. IEEE Int. Conf. Autom. Quality Testing Robot., 2014, pp. 1–4.
574 [3] G. Fortino and P. Trunfio, Internet of Things Based on Smart Objects,
575 Fortino & P. Trunfio, Eds. Cham, Switzerland: Springer, 2014.
576 [4] M. Soliman, T. Abiodun, T. Hamouda, J. Zhou, and C.-H. Lung,
577 “Smart home: Integrating internet of things with Web services
578 and cloud computing,” in Proc. IEEE 5th Int. Conf. Cloud Comput.
579 Technol. Sci., 2013, vol. 2, pp. 317–320.
580[5] P. C. Vinh, “Toward formalized autonomic networking,” Mobile
581Netw. Appl., vol. 19, no. 5, pp. 598–607, 2014.
582[6] K. Tsagkaris, et al., “A survey of autonomic networking architec-
583tures: Towards a unified management framework,” Int. J. Netw.
584Manage., vol. 23, no. 6, pp. 402–423, 2013.
585[7] B. Kang, P. Nguyen, V. Zalyubouskiy, and H. Choo, “A distrib-
586uted delay-efficient data aggregation scheduling for duty-cycled
587WSNs,” IEEE Sensors J., vol. 17, no. 11, pp. 3422–3437, Jun. 2017.
588[8] B.-S. Kang and I.-Y. Ko, “Effective route maintenance and restora-
589tion schemes in mobile ad hoc networks,” Sensors, vol. 10, no. 1,
590pp. 808–821, 2010.
591[9] B. Kang and H. Choo, “An SDN-enhanced load-balancing tech-
592nique in the cloud system,” J. Supercomputing, vol. 72, pp. 1–24,
5932016.
594[10] B. Kang and H. Choo, “A cluster-based decentralized job dis-
595patching for the large-scale cloud,” EURASIP J. Wireless Commun.
596Netw., vol. 2016, no. 1, pp. 1–8, 2016.
597[11] B. Kang, N. Kwon, and H. Choo, “Developing route optimization-
598based PMIPv6 testbed for reliable packet transmission,” IEEE
599Access, vol. 4, no. 1, pp. 1039–1049, Mar. 2016.
600[12] B. Kang, S. Myoung, and H. Choo, “Distributed degree-based link
601scheduling for collision avoidance in wireless sensor networks,”
602IEEE Access, vol. 4, no. 3, pp. 7452–7468, Sep. 2016.
603[13] M. Lauridsen, I. Z. Kov
acs, P. Mogensen, M. Sørensen, and
604S. Holst, “Coverage and capacity analysis of LTE-M and NB-
605IoT in a rural area,” in Proc.IEEE8thVeh.Technol.Conf., 2016,
606pp. 1–5.
607[14] S. K. Datta, C. Bonnet, and N. Nikaein, “An IoT gateway centric
608architecture to provide novel M2M services,” in Proc. IEEE World
609Forum Internet Things, 2014, pp. 514–519.
610[15] S. Guoqiang, C. Yanming, Z. Chao, and Z. Yanxu, “Design and
611implementation of a smart IoT gateway,” in Proc. IEEE Int. Conf.
612Green Comput. Commun. IEEE Internet Things IEEE Cyber Phys.
613Social Comput., 2013, pp. 720–723.
614[16] K. A. Emara, M. Abdeen, and M. Hashem, “A gateway-based
615framework for transparent interconnection between WSN and IP
616network,” in Proc. IEEE Int. Conf. Comput. Tool, 2009, pp. 1775–
6171780.
Fig. 8. Monitoring overhead: (top-left) level of network congestion; (bottom-left) number of message overhead; (right) normalized histogram of
system overhead over 10 minutes.
8 IEEE TRANSACTIONS ON MULTI-SCALE COMPUTING SYSTEMS, VOL. 3, NO. X, XXXXX 2017

IEEE Proof
618 [17] R. Mueller, J. S. Rellermeyer, M. Duller, and G. Alonso, “Demo: A
619 generic platform for sensor network applications,” in Proc. IEEE
620 Int. Conf. Mobile Adhoc Sensor Syst., 2007, pp. 1–3.
621 [18] D. Bimschas, H. Hellbr€
uck, R. Mietz, D. Pfisterer, K. R€
omer, and
622 T. Teubler, “Middleware for smart gateways connecting sensor-
623 nets to the Internet,” in Proc. 5th Int. Workshop Middleware Tools
624 Services Run-Time Support Sensor Netw., 2010, pp. 8–14.
625 [19] L. Wu, Y. Xu, C. Xu, and F. Wang, “Plug-configure-play service-
626 oriented gateway-for fast and easy sensor network application
627 development,” in Proc. 2nd Int. Conf. Sensor Netw., 2013, pp. 53–58.
628 [20] C.-T. Chang, C.-Y. Chang, R. D. B. Martinez, P.-T. Chen, and
629 Y.-D. Chen, “An IoT multi-interface gateway for building a smart
630 space,” Open J. Social Sci., vol. 3, no. 7, 2015, Art. no. 56.
631 [21] Q. Zhu, R. Wang, Q. Chen, Y. Liu, and W. Qin, “IOT gateway:
632 Bridging wireless sensor networks into Internet of Things,” in
633 Proc. IEEE/IFIP 8th Int. Conf. Embedded Ubiquitous Comput., 2010,
634 pp. 347–352.
635 [22] J.-W. Yoon, Y.-K. Ku, C.-S. Nam, and D.-R. Shin, “Sensor network
636 middleware for distributed and heterogeneous environments,” in
637 Proc. Int. Conf. New Trends Inf. Service Sci., 2009, pp. 979–982.
638 [23] D. Min, Z. Xiao, B. Sheng, and G. Shiya, “Design and implementa-
639 tion of the multi-channel RS485 IOT gateway,” in Proc. Int. Conf.
640 Cyber Technol. Autom. Control Intell. Syst., 2012, pp. 366–370.
641 [24] D. Xu, L. Yang, and L. Jiang, “Research and design of IoT gate-
642 way,” in Proc. Int. Ind. Informat. Comput. Eng. Conf., 2015, pp. 837–
643 840.
644 [25] I. AllSeen Alliance, “AllJoyn framework,” 2016. [Online]. Avail-
645 able: https://allseenalliance.org/framework/
646 [26] D. Giannone, “Now-casting and the real-time data flow,” Hand-
647 book of Economic Forecasting, vol. 2, pp. 195–237, 2013.
648 [27] Q. Xin, P. Olofsson, Z. Zhu, B. Tan, and C. E. Woodcock, “Toward
649 near real-time monitoring of forest disturbance by fusion of modis
650 and landsat data,” Remote Sens. Environ., vol. 135, pp. 234–247,
651 2013.
652 [28] R. Wojciechowski, A. Siersze
n, and º. Sturgulewski, “Self-configu-
653 ration networks,” in Image Processing and Communications Chal-
654 lenges 7. Berlin, Germany: Springer, 2016, pp. 301–308.
655 [29] H. Hu, J. Zhang, X. Zheng, Y. Yang, and P. Wu, “Self-configura-
656 tion and self-optimization for LTE networks,” IEEE Commun.
657 Mag., vol. 48, no. 2, pp. 94–100, Feb. 2010.
658 [30] E. U. T. R. Access and Evolved Universal Terrestrial Radio Access
659 Network (E-UTRAN), Overall description, vol. 126, 2008.
660 [31] N. Alliance, “NGMN 5G white paper,” Next Generation Mobile
661 Networks, White paper, 2015.
662 [32] A. Samukic, “UMTS universal mobile telecommunications sys-
663 tem: Development of standards for the third generation,” in Proc.
664 Global Telecommun. Conf., 1998, vol. 4, pp. 1976–1983.
665 [33] N. A. Ali, A.-E. M. Taha, and H. S. Hassanein, “Quality of service
666 in 3GPP R12 LTE-advanced,” IEEE Commun. Mag., vol. 51, no. 8,
667 pp. 103–109, Aug. 2013.
668 [34] L. Sanchez, et al., “SmartSantander: IoT experimentation over a
669 smart city testbed,” Comput. Netw., vol. 61, pp. 217–238, 2014.
670 [35] C. Adjih, et al., “FIT IoT-LAB: A Large scale open experimental
671 IoT testbed,” in Proc. IEEE 2nd World Forum Internet Things, 2015,
672 pp. 459–464.
673 [36] B. Kang and H. Choo, “A deep-learning-based emergency alert
674 system,” ICT Express, vol. 2, no. 2, pp. 67–70, 2016.
675 [37] S.-G. Jung, B. Kang, S. Yeoum, and H. Choo, “Trail-using ant
676 behavior based energy-efficient routing protocol in wireless sen-
677 sor networks,” Int. J. Distrib. Sensor Netw., vol. 2016, 2016,
678 Art. no. 11.
679 [38] J. Gubbi, R. Buyya, S. Marusic, and M. Palaniswami, “Internet of
680 Things (IoT): A vision, architectural elements, and future
681 directions,” Future Generation Comput. Syst., vol. 29, no. 7,
682 pp. 1645–1660, 2013.
683 [39] P. Bellavista, G. Cardone, A. Corradi, and L. Foschini,
684 “Convergence of MANET and WSN in IoT urban scenarios,”
685 IEEE Sensors J., vol. 13, no. 10, pp. 3558–3567, Oct. 2013.
686 [40] Q. Wang and W. Yang, “Energy consumption model for power
687 management in wireless sensor networks,” in Proc. 4th Annu.
688 IEEE Commun. Soc. Conf. Sensor Mesh Ad Hoc Commun. Netw.,
689 2007, pp. 142–151.
690 [41] Q. Chi, H. Yan, C. Zhang, Z. Pang, and L. Da Xu, “A reconfigura-
691 ble smart sensor interface for industrial WSN in IoT environ-
692 ment,” IEEE Trans. Ind. Informat., vol. 10, no. 2, pp. 1417–1425,
693 May 2014.
694[42] J. Holler, V. Tsiatsis, C. Mulligan, S. Avesand, S. Karnouskos, and
695D. Boyle, From Machine-to-Machine to the Internet of Things:
696Introduction to a New Age of Intelligence. New York, NY, USA:
697Academic, 2014.
698[43] K.-C. Chen and S.-Y. Lien, “Machine-to-machine communications:
699Technologies and challenges,” Ad Hoc Netw., vol. 18, pp. 3–23,
7002014.
701[44] M. Hasan, E. Hossain, and D. Niyato, “Random access for
702machine-to-machine communication in LTE-advanced networks:
703Issues and approaches,” IEEE Commun. Mag., vol. 51, no. 6,
704pp. 86–93, Jun. 2013.
705Byungseok Kang received the BS degree in
706computer engineering from Sejong University,
707Korea, in 2006, the MS degree in electrical and
708electronics engineering from Korea University,
709Korea, in 2008, and the PhD degree in electron-
710ics and computer science from the University of
711Southampton, United Kingdom, in 2015. He is
712currently a research professor with Sungkyunk-
713wan University, in 2015. His research interests
714include wired/wireless networking, internet of
715things, sensor networking, mobile computing,
716network protocols, and simulations/numerical
717analysis.
718
719
Daecheon Kim received the BS degree in infor-
720mation and communication engineering from
721Sungkyul University, Korea, in 2016, He is cur-
722rently working toward the master’s degree in
723information and communication engineering at
724Sungkyunkwan University. His research interests
725include internet of things, sensor networking, and
726mobile computing.
727Hyunseung Choo received the BS degree in
728mathematics from Sungkyunkwan University,
729Korea, in 1988, the MS degree in computer sci-
730ence from the University of Texas at Dallas, in
7311990, and the PhD degree in computer science
732from the University of Texas at Arlington, in 1996.
733From 1997 to 1998, he was a patent examiner
734with Korean Industrial Property Office. Since
7351998, he has joined the College of Information
736and Communication Engineering, Sungkyunkwan
737University, and is an associate professor and
738director of Convergence Research Institute. Since 2005, he is director of
739Intelligent HCI Convergence Research Center (eight-year research pro-
740gram) supported by the Ministry of Knowledge Economy (Korea) under
741the Information Technology Research Center support program super-
742vised by the Institute of Information Technology Assessment. His
743research interests include wired/wireless/optical embedded networking,
744mobile computing, and grid computing. He is vice president of Korean
745Society for Internet Information (KSII). He has been editor-in-chief of the
746Journal of KSII for three years and journal editors of the Journal of Com-
747munications and Networks, the ACM Transactions on Internet Technol-
748ogy, the International Journal of Mobile Communication, Springer-
749Verlag Transactions on Computational Science Journal, and editor of
750the KSII Transactions on Internet and Information Systems since 2006.
751He has published more than 200 papers in international journals and ref-
752ereed conferences. He is a member of IEEE and the ACM.
753
"For more information on this or any other computing topic,
754please visit our Digital Library at www.computer.org/publications/dlib.
KANG ET AL.: INTERNET OF EVERYTHING: A LARGE-SCALE AUTONOMIC IOT GATEWAY 9