PresentationPDF Available

Ontology-Driven Automation of IoT-Based Human-Machine Interfaces Development

Authors:

Abstract

The paper is devoted to the development of high-level tools to automate tangible human-machine interfaces creation bringing together IoT technologies and ontology engineering methods. We propose using ontology-driven approach to enable automatic generation of firmware for the devices and middleware for the applications to design from scratch or transform the existing M2M ecosystem with respect to new human needs and, if necessary, to transform M2M systems into human-centric ones. Thanks to our previous research, we developed the firmware and middleware generator on top of SciVi scientific visualization system that was proven to be a handy tool to integrate different data sources, including software solvers and hardware data providers, for monitoring and steering purposes. The high-level graphical user SciVi interface enables to design human-machine communication in terms of data flow and ontological specifications. Thereby the SciVi platform capabilities are sufficient to automatically generate all the necessary components for IoT ecosystem software. We tested our approach tackling the real-world problems of creating hardware device turning human gestures into semantics of spatiotemporal deixis, which relates to the verbal behavior of people having different psychological types. The device firmware generated by means of SciVi tools enables researchers to understand complex matters and helps them analyze the linguistic behavior of users of social networks with different psychological characteristics, and identify patterns inherent in their communication in social networks.
Ontology-Driven Automation !
of IoT-Based Human-Machine Interfaces
Development
Faro – 2019
Perm State University
Bukireva Str. 15, 614990, Perm, Russia
Konstantin Ryabinin
e-mail: kostya.ryabinin@gmail.com
Svetlana Chuprina
e-mail: chuprinas@inbox.ru
Konstantin Belousov
e-mail: belousovki@gmail.com
Motivation 2 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Custom Hardware!
Human-Machine Interface?
Motivation 2 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Custom Hardware!
Human-Machine Interface?
Virtual Reality
Motivation 2 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Custom Hardware!
Human-Machine Interface?
Virtual Reality
Simulators
Motivation 2 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Custom Hardware!
Human-Machine Interface?
Virtual Reality
Simulators
Complex Data!
Analytics
Motivation 2 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Custom Hardware!
Human-Machine Interface?
Virtual Reality
Simulators
Complex Data!
Analytics
Human-Centric!
IoT
Motivation 2 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Custom Hardware!
Human-Machine Interface?
Virtual Reality
Simulators
Complex Data!
Analytics
Interactive!
Exhibits
Human-Centric!
IoT
Motivation 2 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Custom Hardware!
Human-Machine Interface?
Virtual Reality
Simulators
Complex Data!
Analytics
Interactive!
Exhibits
Human-Centric!
IoT
Motivation 3 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Motivation 3 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Enrich, not replace!
existing digital infrastructure
Objectives 4 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
1. Increase the level of HMI development tools
2. Propose the approach of smart systems building to automate:
2.2. IoT devices firmware generation
2.3. Middleware generation
2.4. Using IoT devices as HMI to steer third-party software
3. Create the smart system according to the proposed approach
4. Test smart system as a middleware to marry custom IoT-based
HMI with third-party software
Idea 5 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Idea 5 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Ontology – formal model of application domain!
(T.R. Gruber, 1993)
O = T, R, A
T – thesaurus of application domain concepts
R – set of relations between concepts
A – set of axioms
Idea 5 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Background: SciVi System 6 / 20
Multiplatform client-server adaptive scientific visualization system SciVi
Ryabinin, K., Chuprina, S.: Development of Ontology-Based Multiplatform !
Adaptive Scientific Visualization System. Journal of Computational Science 10, !
370–381 (2015). https://doi.org/10.1016/j.jocs.2015.03.003
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Device Integration Stage 7 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
Device Integration Stage 7 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
ONTOLIS!
Ontology Editor
Device Integration Stage 7 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
Device Integration Stage 7 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
ESP8266
MPU6050
Push Button
Data Link Control Link
Device Integration Stage 7 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
ESP8266
MPU6050
Push Button
Data Link Control Link
Device Integration Stage 7 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
8 / 20
Firmware Composing Stage
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
8 / 20
Firmware Composing Stage
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
MPU6050
DataSource
Code Block
MPU6050Lib
Type
C++
Vector3
Output
Gyro
is_a
use_for
instance_of
has
language
Accel
9 / 20
Firmware Generation Stage
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
9 / 20
Firmware Generation Stage
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
10 / 20
Ontological Profile of IoT Device
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
10 / 20
Ontological Profile of IoT Device
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Part of DFD!
to Generate
Device!
Ontological Profile
10 / 20
Ontological Profile of IoT Device
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Part of DFD!
to Generate
Device!
Ontological Profile
10 / 20
Ontological Profile of IoT Device
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Part of DFD!
to Generate
Device!
Ontological Prole
Glove
DataSource
ClientSideWorker
GloveWorker
Type
JavaScript
Quaternion
Output
Finger
is_a
use_for
instance_of
has
language
Orientation
Bool
11 / 20
Firmware Debug Stage
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
11 / 20
Firmware Debug Stage
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
11 / 20
Firmware Debug Stage
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
12 / 20
Firmware Debug Stage
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
12 / 20
Firmware Debug Stage
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
13 / 20
Software Integration Stage
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
13 / 20
Software Integration Stage
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
Keynote
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
13 / 20
Software Integration Stage
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
Keynote
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
13 / 20
Software Integration Stage
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
14 / 20
Middleware Composing Stage
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
15 / 20
Middleware Generation Stage
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
15 / 20
Middleware Generation Stage
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
16 / 20
HMI Usage
HMI Middleware!
generated by!
SciVi
3rd-Party!
Software
Data Link Control Link
Raw Data Commands
Feedback!
(optional)
Feedback!
(optional)
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
17 / 20
Use Case
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Task:
Discover relationships between psychological characteristics !
of social network users and their verbal behavior
Instruments:
1. Semograph computer linguistics system
2. SciVi visual analytics system
Data:
1. 18'000 posts from VKontakte social network made by 800 users
2. Big Five personality traits (psychological profiles) of these users:
2.1. Openness
2.2. Conscientiousness
2.3. Extraversion
2.4. Agreeableness
2.5. Neuroticism
Preprocessing:
Correlation analysis resulting in the graph where psychological
characteristics are connected with verbal patterns
18 / 20
Use Case
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Verbal behavior patterns are classified into semantic groups
HMI: Use gestures to activate groups with specific spatial semantics
Conclusion 19 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
What we have:
1.Approach to building ontology driven smart systems !
for automating hardware HMI creation
2.Single smart system with high-level graphical toolset to
2.1. Program
2.2. Debug
2.3. Monitor
2.4. Create middleware to steer 3rd-party systems !
(including legacy ones)
3. Perceptive-cognitive HMI for multimodal analytics
What we plan:
1.Tackle problems of transforming M2M IoT systems !
into Human-Centric ones
2.Create IoT-based healthcare monitoring systems
3.Create HMI for virtual reality applications
}hardware HMI
Thank You for Attention!
Faro – 2019
Perm State University
Bukireva Str. 15, 614990, Perm, Russia
Konstantin Ryabinin
e-mail: kostya.ryabinin@gmail.com
Svetlana Chuprina
e-mail: chuprinas@inbox.ru
Konstantin Belousov
e-mail: belousovki@gmail.com
... 2. Возникает острая нехватка высокоуровневых средств для развёртывания и последующего управ-ления CPS в условиях небольшого количества IT-специалистов в штате музейных сотрудников. В рамках предыдущих исследований нами были предложены методы и высокоуровневые средства авто-матизации создания аппаратных человеко-машинных интерфейсов на базе онтологического инжиниринга и технологий IoT [7], а также сформулирована концеп-ция применения технологий IoT и научной визуализа-ции для создания интерактивных музейных экспонатов [8]. Целью данной работы является синтез результа-тов предыдущих исследований и формулирование на их основе общей концепции создания музейных CPS, а также применение этой концепции на практике при со-здании киберфизических музейных экспонатов в Мос-ковском Государственном Дарвиновском музее и Му-зее пермских древностей. ...
... В качестве инструментария, удовлетворяющего указанным требованиям, предлагается использовать разработанную в рамках предыдущих исследований адаптивную мультиплатформенную систему научной визуализации SciVi [7]. Поведение этой системы управляется лежащей в её основе онтологической ба-зой знаний, что обеспечивает её адаптивность и мас-штабируемость [7]. ...
... В качестве инструментария, удовлетворяющего указанным требованиям, предлагается использовать разработанную в рамках предыдущих исследований адаптивную мультиплатформенную систему научной визуализации SciVi [7]. Поведение этой системы управляется лежащей в её основе онтологической ба-зой знаний, что обеспечивает её адаптивность и мас-штабируемость [7]. Возможность использования SciVi в качестве основы превращения существующих циф-ровых музейных инфраструктур в полноценные CPS обусловлена следующими функциями системы, до-ступными пользователям посредством высокоуровне-вого графического интерфейса: 1. Механизмы автоматизированной интеграции (включая прямую и обратную связь) со сторонни-ми источниками данных, включая статические хра-нилища и динамические генераторы (программно-аппаратные решатели) [9]. ...
Conference Paper
Full-text available
In this paper we propose an approach to create cyber-physical museum exhibits based on the methods and means of scientific visualization, Internet of things, additive technologies and ontology engineering. Cyber-physical exhibit consists of tightly interconnected virtual and real parts. Scientific visualization is used as methodological and technological basis for presenting the virtual part, which can contain relevant digital content and 3D-reconstructions. Internet of things is a core technology to create tangible interfaces for the corresponding visualization software. Additive technologies allow creating virtual reconstructions and precise copies of museum exhibits. Ontology engineering provides adaptive mechanisms for seamless integration of new cyber-physical exhibits into the existing digital infrastructure of museums. The proposed approach is used in practice to create cyber-physical exhibits in State Darwin Museum (Moscow) and Museum of Permian Antiquities (Perm).
... It can be treated as a guideline, which components fit together and how they should be interconnected. The firmware generation mechanism included with SciVi operates on the basis of the electronic components' ontology indicated above [23]. The logic of the PCI, as well as the visualization and analysis algorithms, are declared using a data flow diagram (DFD), composed by the user within built-in high-level editor. ...
... Входящий в состав SciVi механизм генерации про-шивок функционирует на основе указанной выше он-тологии электронных компонентов [15]. Логика работы PCI, а также алгоритм управления процессом визуали-зации и анализа, декларируются при помощи диаграм-мы потоков данных (англ. ...
Conference Paper
Full-text available
The paper is devoted to using Internet of Things technologies for hardware human-machine interfaces development. Thanks to these technologies, it may be possible to improve the capabilities of visual analytics systems with multiple modalities: movements, audio, etc. It can speed up semantic data filtering and interpretation, increasing the efficiency of analytics. We suggest using ontology engineering methods and tools to automate both the programming of custom hardware human-machine interfaces and connecting them to the third-party software. The proposed concept is tested by solving the real-world tasks of discovering the relationships between the psychological characteristics of the native speakers and their verbal behavior.
Article
In this study, a facile and scalable fabrication process of the flexible and transparent tactile sensor array where its operation is based on the triboelectric effect is proposed. The overall process is assisted by the solution based screen printing method, which is well-known as advantageous in the batch fabrication of the electronic devices. The introduction of the batch fabrication process to the tactile sensor array enables the realization of high functionality such as transparency and flexibility. The performance of the fabricated tactile sensor array is systematically investigated through the parametric studies and the discernibility to multiple external physical stimuli such as vertical contact and horizontal sliding is verified. It is confirmed that although the present self-triggered tactile sensor array could be easily fabricated and scaled-up without cumbersome processes, the functionalities such as sensitivity, resolution, transparency, flexibility and even peak power density, are comparable to previously developed tactile sensors. Consequently, the scalable and design-flexible fabrication methodology with cost-effect and high production speed in this study effectively broadens the applicability of the sensor array. As proof-of-concept demonstration of potential applications, a stand-alone sensory lighting system as well as a wearable transparent code generator are developed and their performances are investigated.
Chapter
The paper is devoted to new aspects of ontology-based approach to control the behavior of Edge Computing devices. Despite the ontology-driven solutions are widely used to develop adaptive mechanisms to the specifics of the Internet of Things (IoT) and ubiquitous computing ecosystems, the problem of creating withal full-fledged, easy to handle and efficient ontology-driven Edge Computing still remains unsolved. We propose the new approach to utilize ontology reasoning mechanism right on the extreme resource-constrained Edge devices, not in the Fog or Cloud. Thanks to this, on-the-fly modifying of device functions, as well as ad-hoc monitoring of intermediate data processed by the device and interoperability within the IoT are enabled and become more intelligent. Moreover, the smart leverage of on-demand automated transformation of Machine-to-Machine to Human-Centric IoT becomes possible. We demonstrate the practical usefulness of our solution by the implementation of ontology-driven Smart Home edge device that helps locating the lost things.
ResearchGate has not been able to resolve any references for this publication.