Content uploaded by Konstantin Ryabinin
Author content
All content in this area was uploaded by Konstantin Ryabinin on Feb 05, 2020
Content may be subject to copyright.
Ontology-Driven Automation !
of IoT-Based Human-Machine Interfaces
Development
Faro – 2019
Perm State University
Bukireva Str. 15, 614990, Perm, Russia
Konstantin Ryabinin
e-mail: kostya.ryabinin@gmail.com
Svetlana Chuprina
e-mail: chuprinas@inbox.ru
Konstantin Belousov
e-mail: belousovki@gmail.com
Motivation 2 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Custom Hardware!
Human-Machine Interface?
Motivation 2 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Custom Hardware!
Human-Machine Interface?
Virtual Reality
Motivation 2 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Custom Hardware!
Human-Machine Interface?
Virtual Reality
Simulators
Motivation 2 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Custom Hardware!
Human-Machine Interface?
Virtual Reality
Simulators
Complex Data!
Analytics
Motivation 2 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Custom Hardware!
Human-Machine Interface?
Virtual Reality
Simulators
Complex Data!
Analytics
Human-Centric!
IoT
Motivation 2 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Custom Hardware!
Human-Machine Interface?
Virtual Reality
Simulators
Complex Data!
Analytics
Interactive!
Exhibits
Human-Centric!
IoT
Motivation 2 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Custom Hardware!
Human-Machine Interface?
Virtual Reality
Simulators
Complex Data!
Analytics
Interactive!
Exhibits
Human-Centric!
IoT
Electronic
Components: !
cheap and versatile
Developers: !
high technical
qualification needed!
Motivation 3 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Motivation 3 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Enrich, not replace!
existing digital infrastructure
Objectives 4 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
1. Increase the level of HMI development tools
2. Propose the approach of smart systems building to automate:
2.2. IoT devices firmware generation
2.3. Middleware generation
2.4. Using IoT devices as HMI to steer third-party software
3. Create the smart system according to the proposed approach
4. Test smart system as a middleware to marry custom IoT-based
HMI with third-party software
Idea 5 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Idea 5 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Ontology – formal model of application domain!
(T.R. Gruber, 1993)
O = ⟨T, R, A⟩
T – thesaurus of application domain concepts
R – set of relations between concepts
A – set of axioms
Idea 5 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Background: SciVi System 6 / 20
Multiplatform client-server adaptive scientific visualization system SciVi
Ryabinin, K., Chuprina, S.: Development of Ontology-Based Multiplatform !
Adaptive Scientific Visualization System. Journal of Computational Science 10, !
370–381 (2015). https://doi.org/10.1016/j.jocs.2015.03.003
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Device Integration Stage 7 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
Device Integration Stage 7 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
ONTOLIS!
Ontology Editor
Device Integration Stage 7 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
Device Integration Stage 7 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
ESP8266
MPU6050
Push Button
Data Link Control Link
Device Integration Stage 7 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
ESP8266
MPU6050
Push Button
Data Link Control Link
Device Integration Stage 7 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
8 / 20
Firmware Composing Stage
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
8 / 20
Firmware Composing Stage
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
MPU6050
DataSource
Code Block
MPU6050Lib
Type
C++
Vector3
Output
Gyro
is_a
use_for
instance_of
has
language
Accel
9 / 20
Firmware Generation Stage
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
9 / 20
Firmware Generation Stage
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
10 / 20
Ontological Profile of IoT Device
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
10 / 20
Ontological Profile of IoT Device
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Part of DFD!
to Generate
Device!
Ontological Profile
10 / 20
Ontological Profile of IoT Device
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Part of DFD!
to Generate
Device!
Ontological Profile
10 / 20
Ontological Profile of IoT Device
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Part of DFD!
to Generate
Device!
Ontological Profile
Glove
DataSource
ClientSideWorker
GloveWorker
Type
JavaScript
Quaternion
Output
Finger
is_a
use_for
instance_of
has
language
Orientation
Bool
11 / 20
Firmware Debug Stage
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
11 / 20
Firmware Debug Stage
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
11 / 20
Firmware Debug Stage
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
12 / 20
Firmware Debug Stage
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
12 / 20
Firmware Debug Stage
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
13 / 20
Software Integration Stage
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
13 / 20
Software Integration Stage
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
Keynote
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
13 / 20
Software Integration Stage
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
Keynote
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
13 / 20
Software Integration Stage
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
14 / 20
Middleware Composing Stage
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
15 / 20
Middleware Generation Stage
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
15 / 20
Middleware Generation Stage
HMI Ontologies
Integration!
Module
Firmware!
Generator
Middleware!
Generator
DFD Editor
3rd-Party!
Software
Data Filtering!
Module
Visualization!
Module
Smart System
Data Link Control Link
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
16 / 20
HMI Usage
HMI Middleware!
generated by!
SciVi
3rd-Party!
Software
Data Link Control Link
Raw Data Commands
Feedback!
(optional)
Feedback!
(optional)
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
17 / 20
Use Case
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Task:
Discover relationships between psychological characteristics !
of social network users and their verbal behavior
Instruments:
1. Semograph computer linguistics system
2. SciVi visual analytics system
Data:
1. 18'000 posts from VKontakte social network made by 800 users
2. Big Five personality traits (psychological profiles) of these users:
2.1. Openness
2.2. Conscientiousness
2.3. Extraversion
2.4. Agreeableness
2.5. Neuroticism
Preprocessing:
Correlation analysis resulting in the graph where psychological
characteristics are connected with verbal patterns
18 / 20
Use Case
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
Verbal behavior patterns are classified into semantic groups
HMI: Use gestures to activate groups with specific spatial semantics
Conclusion 19 / 20
K. Ryabinin, S. Chuprina, K. Belousov
Perm State University
Ontology-Driven Automation of IoT-Based HMI Development
What we have:
1.Approach to building ontology driven smart systems !
for automating hardware HMI creation
2.Single smart system with high-level graphical toolset to
2.1. Program
2.2. Debug
2.3. Monitor
2.4. Create middleware to steer 3rd-party systems !
(including legacy ones)
3. Perceptive-cognitive HMI for multimodal analytics
What we plan:
1.Tackle problems of transforming M2M IoT systems !
into Human-Centric ones
2.Create IoT-based healthcare monitoring systems
3.Create HMI for virtual reality applications
}hardware HMI
Thank You for Attention!
Faro – 2019
Perm State University
Bukireva Str. 15, 614990, Perm, Russia
Konstantin Ryabinin
e-mail: kostya.ryabinin@gmail.com
Svetlana Chuprina
e-mail: chuprinas@inbox.ru
Konstantin Belousov
e-mail: belousovki@gmail.com