Content uploaded by Thanos G. Stavropoulos
Author content
All content in this area was uploaded by Thanos G. Stavropoulos on Feb 09, 2021
Content may be subject to copyright.
eHealth4MS: Problem Detection from Wearable Activity
Trackers to Support the Care of Multiple Sclerosis
Thanos G. Stavropoulos1[0000-0003-2389-4329], Georgios Meditskos1[0000-0003-4242-5245],
Sotirios Papagiannopoulos2[0000-0001-6743-6544] and Ioannis Kompatsiaris1[0000-0001-6447-9020]
1 Information Technologies Institute, Centre for Research & Technology Hellas,
6th Km Charilaou Thermi, 57001 Thessaloniki, Greece
2 Department of Neurology III, Medical School, Aristotle University of Thessaloniki,
Thessaloniki 54124, Greece
{athstavr, gmeditsk, ikom}@iti.gr, spapagia@auth.gr
Abstract. This paper presents eHealth4MS, an assistive technology system based
on wearable trackers to support the care of Multiple Sclerosis (MS). Initially, the
system integrates a tracker and a smartphone to collect and unanimously store
movement, sleep and heart rate (HR) data in an ontology-based knowledge base.
Then, ontology patterns are used to provide an initial approach to detect problems
and symptoms of interest, such as lack of movement, stress or pain, insomnia,
excessive sleep, lack of sleep and restlessness. Finally, the system visualizes data
trends and detected problems in dashboards and apps. This will allow patients to
self-manage and for clinicians to drive effective and timely interventions and to
monitor progress in future trials to evaluate the system’s accuracy and effective-
ness.
Keywords: Multiple Sclerosis, eHealth, IoT, Wearables, AAL, Ontologies.
1 Introduction
Multiple Sclerosis (MS) is an autoimmune disease that affects the brain and the spinal
cord (central nervous system), with serious implications in family, professional and
social life. MS disrupts regular communication between the brain and the rest of the
body, which may lead to vision impairment, muscle weakness, motor coordination and
balance issues, cognitive deficit and depression. It may appear at any age, although in
majority it does between 15 and 45 years of age. Out of two million suffering world-
wide, 65-70% are women [1]. The course of the disease follows a relapsing-remitting
cycle, but 60-70% of the cases develop a steady progression of symptoms with or with-
out remission periods, known as secondary-progressive MS.
Regular monitoring, effective support and personalized guidance are vital to the best
possible outcomes[2]. Holistic and tailored solutions help patients preserve independ-
ence, effective communication, social roles and productivity to the greatest extent pos-
sible. The specialized doctor’s role is also central to the therapeutic team, who is in
2
charge of designing a treatment plan. However, such constant monitoring is often fi-
nancially or practically impossible, while many times it relies on unreliable subjective
observation.
New assisted living technologies combined with intelligent applications may pro-
vide a reliable, remote monitoring solution, supporting the patients and their caregivers
and doctors alike, to ensure personalized and constant care, in order to hinder the dis-
ease’s progression. Towards this, the young age of MS patients has been shown to help
in the acceptance of technological aids [3].
However, holistic systems that can monitor multiple life aspects to aid in neuro-
degenerative diseases, such as MS, have yet to emerge. On the contrary, existing Am-
bient Assisted Living (AAL) systems can be exploited to support MS aid in other as-
pects, such as communication means [3], clinical information exchange between doc-
tors [4] and even diagnosis via videoconferencing [5]. Technology has also been used
to record rehabilitation exercises at home, with limitations in the way, place and time
where monitoring takes place [6]. Beyond MS, many eHealth monitoring systems focus
on a single aspect such as sleep [7]. Even systems that do combine information, are
lacking the smart, personalized inference mechanisms to assess the status of a patient
with MS [8].
The vision of a holistic monitoring eHealth platform for MS is introduced by the
EFPIA alliance
1
of the European Commission and the pharmaceutical industry includ-
ing Novartis, Johnson & Johnson, Bayer, Roche and Sanofi. RADAR-BASE
2
is a plat-
form developed by EFPIA and academia, where such a monitoring platform is being
developed with applications in multiple diseases, including MS [9].
This paper presents eHealth4MS; an intelligent system that collects, interlinks and
interprets data from wearable devices, in order to support remote and reliable monitor-
ing of people with MS at home, leading to more effective, efficient and accessible treat-
ment and self-management, increasing the patients’ Quality of Life (QoL). The system
is based on the Service-Oriented Architecture (SOA) principles to integrate heteroge-
neous Internet of Things (IoT) data sources to obtain QoL information, such as step
count, calories burnt, distance walked, heart rate and sleep, through wearable smart-
watches and smartphones. The data is processed and homogeneously stored in an on-
tology, and then interpreted, in order to extract clinically relevant information related
to the disease, such as physical activity and exercise, stress, sleep quality patterns and
problems, in relation to patient profile. The outcomes are presented to patients for self-
management, and to doctors for effective decision-making, through adaptive user inter-
faces on web and mobile applications.
In detail, this paper aims to address the following research questions:
• How can IoT, QoL data from heterogeneous IoT platforms, such as wearable
watches and smartphones be retrieved in a modular manner?
• How can IoT, QoL data be represented and stored unanimously, analyzed and inter-
preted in order to produce clinically relevant and valuable information for the care
of MS?
1
European Federation of Pharmaceutical Industries and Associations: https://www.efpia.eu
2
The RADAR-BASE Platform: https://radar-base.org/
3
• How can data be visualized for patients and clinicians to facilitate the care of MS?
In response, the paper provides and initial prototype approach towards the following
goals:
• To develop a heterogeneous data collection platform that integrates wearable
watches and smartphones in a modular manner.
• To semantically describe and unanimously store heterogeneous IoT/QoL data from
wearable watches and smartphones.
• To provide an initial approach to process, interpret and extract clinically relevant
problems and symptoms from physical activity, sleep and HR data.
• To visualize data in end-user dashboards and apps for patients and clinicians.
To do so, the system builds upon previous integrated systems, applied to assisted living
with dementia [10]. While those systems were far more complex, the system proposed
in this paper relies on simpler, more effective but also modern, comfortable and ac-
ceptable sensors in daily life, as well as analysis and interpretation tailored for MS.
Meanwhile, building upon previous experience in clinical trials with technology [11],
the clinical evaluation of eHealth4MS is carved out to include around thirty patients
and control groups, testing the system for several months in a future study.
2 Related Work
Technologies such as IoT, promise to deliver eHealth solutions even to chronic diseases
where pharmaceutical treatment is lacking, such as dementia, or even when a more
holistic lifestyle change is needed, such as cardiovascular disease. However, eHealth
solutions to date are characterized by their focus on a single aspect of life each, such as
physical activity or exercise, sleep quality, serious games, alerts etc. [7], while also
lacking the intelligent analysis and interpretation tailored to each field of disease [8].
Combined with market segmentation, their uptake is, as a result, limited.
Especially for MS, eHealth applications are limited to message exchange and com-
munication between doctors and patients [3] [4], telemedicine via videoconferencing
[5] and rehabilitation exercise support in a specific place and time [6]. At the same time,
the road for further developments in eHealth for MS is being paved. The young age of
those suffering from MS has already proved to favor acceptance of technology as an
aid on their behalf [3]. Meanwhile, the pharmaceutical industry, especially in alliance
with the EU in the EFPIA initiative, co-develop such research ideas of eHealth systems
providing so-called “digital biomarkers” for monitoring, so as to more efficiently and
objectively assess a participant’s status in a clinical trial and therefore, aid in drug de-
velopment. RADAR-CNS
3
is such a platform to collect heterogeneous data, applied in
the scenario of MS but has yet to leverage the potential for data analysis, interpretation
and personalization [9].
3
The RADAR-CNS Project: https://www.radar-cns.org/
4
Apart from IoT and digital biomarker data collection in eHealth, data models and
intelligent analysis are progressing as well. Ontologies, such as OWL 2 [12], have at-
tracted growing interest as means for modelling and reasoning over contextual infor-
mation and human activities in particular. Activity recognition is often augmented with
rules for representing richer relationships not supported by the standard ontology se-
mantics, like e.g. temporal reasoning and structured, composite activities [13].
The present paper, integrates and extends data collection, semantic web modelling
and analysis and tailored user interfaces to address the needs of MS. The system collects
data for multiple life aspects, such as physical activity and exercise, quality of sleep
and stress, through comfortable wearable smartwatches and smartphones. Conse-
quently, it homogeneously represents them, in novel models, and interprets them to
extract medically relevant information. After knowledge is stored, following the latest
advances in security and privacy in the cloud, they can be visualized through various
web and mobile apps. Low cost technology set up the platform for higher chances of
future uptake.
3 Integration and Data Collection
As in previous works [10], modular architectures based on SOA can integrate multiple
kinds of wearable smartwatches, wristbands and any type of smart home sensor with
extensibility. The same architecture is followed in this paper, however, adapted to mo-
bile and more modern wearable devices as shown on Fig. 1.
Most devices available in the market provide their data through their own well-de-
fined Cloud APIs or Software Development Kits (SDKs). Therefore, two modules, the
so-called “Adapters”, are used for the eHealth4MS platform for the two devices to be
integrated: 1) the smartwatch, Fitbit Charge 3 and 2) a standard Android Smartphone
Fig. 1. eHealth4MS System Architecture
5
(Xiaomi Redmi 6A/7A). The FitBit Adapter downloads FitBit Charge 3 clock data from
the FitBit Cloud through secure user authorization (OAuth protocol) and the
Smartphone Adapter retrieves smartphone usage and sensor data through the open An-
droid SDK. Both data streams end up on the eHealth4MS Cloud online platform, and
through it to the Data Integration module, which seamlessly stores them in the semantic
knowledge base (Cloud-based RDF triple store), described in the next section.
To ensure SOA extensibility and modularity, Adapters and the central Data Integra-
tion module are connected through REST APIs. Therefore, based on the principles of
Open API Initiative (OAI), new devices and data sources can be supported in the future
by implementing new Adapters or otherwise new web services, using universal stand-
ards. Platform data becomes available for further analysis and interpretation, as well as
for display in user applications through semantic search and consumption protocols
(SPARQL endpoints).
4 Knowledge Representation and Analysis
A significant challenge in remote monitoring solutions is the ability to identify and
recognize the context signifying the presence of complex activities and situations in
order to support intelligent behaviour interpretation. An important factor to take into
consideration is that contextual information is typically collected by multiple sensors
and complementary modalities. The goal is to recognize the behaviour of the person
with MS and discern traits that have been identified by the clinicians as relevant, for
diagnostic, status assessment, enablement and safety purposes, achieving medical am-
bient intelligence and situational awareness. For example, a long-duration movement
outdoors may indicate a walking activity and disrupted sleeping patterns, such as regu-
larly waking up throughout the night and short sleep durations, may be evidence of
sleep disorders and insomnia.
Βehaviour interpretation and situational awareness require the aggregation of col-
lected information and their infusion with clinical knowledge and user preferences (pro-
files, history, etc.). To this end, eHealth4MS aggregates individual pieces of infor-
mation provided by monitoring and clinical experts and then meaningfully fuse them
in order to derive high-level interpretations of the person behavior and achieve situa-
tional awareness. Two constituents are considered for supporting the underlying fusion
tasks, namely representation and reasoning support services. Representation provides
the vocabulary and infrastructure for capturing and storing information relevant to mon-
itoring, environment, clinical and profile knowledge. The reasoning services support
the integrated interpretation of the person behaviour and recognition of clinically rele-
vant activities and problems.
4.1 Representation
A common prerequisite in context-aware, sensor-driven systems, such as eHealth4MS,
is the ability to share and process information coming from heterogeneous devices and
services. This translates into a twofold requirement. First, there is a need for commonly
6
agreed vocabularies of consensual and precisely defined terms for the description of
data in an unambiguous manner. Second, there is a need for mechanisms to integrate,
correlate and semantically interpret these data. To achieve this, there is a need to model
context at different levels of granularity and abstraction, and support the derivation of
higher-level interpretations. Context refers to any information that can be used to char-
acterize the situation of a person or a computing entity; for example, the location of a
person and the room temperature are aspects of context.
The formalization of the ontological vocabulary follows the OWL 2 language and
extends the Semantic Sensor Network (SSN) ontology for capturing measurements,
following the horizontal and vertical modularization architecture of the standard,
through a lightweight but self-contained core ontology called SOSA (Sensor, Observa-
tion, Sample, and Actuator) [14]. The current version of the domain ontology supports:
• Atomic activities and measurements detected by the monitoring infrastructure (e.g.
steps and sleeping activities) and complex activities inferred through context inter-
pretation (e.g. having meal, walking sleeping).
• Problems and situations of significance that the monitored people and the clinicians
need to be informed about (e.g. sleep problems, correlations between heart rate and
location).
4.2 Reasoning and Interpretation
The interpretation framework analyses collectively the aggregated observations and de-
rives a higher-level understanding of behaviour, in terms of activities and situations the
person engages in, and the identification of clinically defined functional problems. To
this end, two components have been developed: the Online Event Detection (OED)
component and the Semantic Interpretation (SI) component. The developed compo-
nents support interpretation tasks at different levels of granularity. OED serves for un-
derstanding context in a real-time manner. SI on the other hand addresses situations
that require encapsulating pieces of information of higher abstraction.
More specifically, OED focuses primarily on the real-time detection of situations of
interest and their aggregation with clinical and profile information, in order to trigger
respective feedback and alerts. Such events include elementary states and activities us-
ing sensor measurements (e.g. high physical activity for more than 2 minutes). In turn,
the primary focus of SI is on the recognition of: i) complex situations and correlations
(e.g. movement at night after the person has gone to sleep), ii) functional problems as
defined by clinicians (e.g. interrupted sleep). SI espouses a hybrid approach that com-
bines ontology- and rule-based reasoning. OWL 2 is used to model the domain concepts
(activities, situations, problems, etc.); SPARQL rules are used to enhance typical on-
tology-based reasoning with complex activity and problem detection, temporal reason-
ing and incremental knowledge updates.
The collected observations and measurements from the sensor network are sent to
the OED module for real-time fusion with profile knowledge. OED uses the Drools
CEP (Complex Event Processing) engine that queries the KB to retrieve profile infor-
7
mation (e.g. behaviour patterns). The detected events are then sent to the alert and feed-
back services of the framework. Note that detected events by OED are also stored in
the KB for further offline processing and fusion with other observations by SI. Fig 2.
(left) presents the abstract representation and reasoning framework.
Fig 2 (right) depicts the ontology pattern we have defined for modelling restlessness
problems. The pattern captures the number of sleep interruptions observed for a single
day, which is further classified as a Restlessness problem by the reasoning and inter-
pretation layer. In the same way, patterns detect problems of:
• “Stress or Pain” (high HR for long periods of time without movement),
• “Lack of Movement” (low steps in a day),
• “Lack of Exercise” (HR entirely out of cardio zone for a day),
• “Insomnia” (sleep latency in night), “Lack of sleep” (Short total sleep in a day),
• “Too much sleep” (Long total sleep in a day) and
• “Increased Napping” (Too long or too many naps in a day).
5 End-User Applications
Graphical User Interfaces (GUIs) and web applications in eHealth4MS are designed
and developed to serve the user’s needs (caregivers, as well as their relatives and ther-
apists), based on their requirements. These applications, in addition to communicating
the results to healthcare professionals, carers and patients themselves, allow the com-
munication between stakeholders (e.g. by sending personal messages, questions, etc.).
The dashboard visualizations design takes into consideration user goals, behaviours,
needs and profiles, as well as performance, acceptance, clinical and therapeutic value
characteristics. Many design choices are based on previous works in other eHealth
fields [10]. MS differs in the sense that it can cause both physical and mental problems
and, therefore requires: i) holistic view of all aspects and ii) clear detection of problems.
Fig. 2. Logical architecture of the representation and reasoning framework (left);
Ontology pattern capturing the restlessness problem (right)
8
The implementation relies on open source frameworks and libraries and mainly the
well-established and modern Python - Django web framework. Responsive design en-
sures adaptation to: i) mobile phones and tablets that are mostly used by the patients;
and ii) large computer screens, mostly used by the physicians. In addition to responsive
design, user roles require adaptive views, with different permissions in the application.
The patient view displays only abstract and positive observation features, while the
clinician view provides complete, detailed information to physicians. Python-Django's
Model-View-Controller (MVC) approach appears effective in the way that it enables
to easily manage model data, controls and views. Communication with analysis sub-
systems and the knowledge base operates on top of the semantic interfaces of data col-
lection (SPARQL queries).
Fig. 3. presents a detailed view of the clinician dashboard, which visualizes trends
of sensor information such as intraday, hourly, daily, monthly or yearly aggregates of
Fig. 3. The eHealth4MS Dashboard showing daily steps, sleep duration per stage, HR and
detected problems in daily scale.
9
steps, sleep totals and segmentation in sleep stages and average heart rate. It also shows
the analysis and interpretation outcomes in the form of detected problems, which allows
them to investigate occurrences and context, suggest interventions and view progress.
In the future, the clinician view may allow them to modify rules and patient profiles to
detect problems and symptoms which are now modelled in the ontology.
6 Conclusions and Future Work
This paper presents eHealth4MS, an assistive technology system, based on wearable
trackers to support the care of Multiple Sclerosis (MS). The system integrates a tracker
and a smartphone to collect and unanimously store movement, sleep and hear rate (HR)
data in a knowledge base. Then, problem detection techniques extract symptoms and
behaviours clinically relevant to MS, such as lack of movement or exercise, stress or
pain, insomnia, excessive sleep or lack of sleep and restlessness. Finally, the system
visualizes trends and problems that may allow patients to self-manage and doctors to
drive interventions, central to care, and to monitor progress more effectively.
As future work, foremost we plan for a clinical study to evaluate the system’s usa-
bility and clinical value in a real-life environment for patients with MS. The technology
package for each participant will include a wearable tracker and an Android
smartphone, the eHealth4MS self-monitoring app and online access to the dashboard
for doctors and caregivers. 45 participants will be recruited and randomly placed in
three groups. The first group will be given the system and interventions from doctors
adjusted and supported by problem detection and progress monitoring from the system.
The second group will act as controls who will be given interventions without the sys-
tem’s support, and the third group will not be monitored or perform interventions at all.
Participants will be given a written consent form in order to collect personal infor-
mation and monitoring data for the sole purpose of contributing to the research. Pro-
tecting the participants’ privacy, the information can be anonymously disclosed so as
to contribute more broadly to future work, following guidelines from our previous stud-
ies [11].
After the period of six months, the evaluation will address both clinical aspects of
the system as well as technological. The clinical part will include neuropsychological
evaluation that will take place both at the beginning and following the end of the study
and for each group. The tests will include those widely used for MS [15].The usability
of applications will be assessed using the standard System Usability Scale (SUS) [16]
and the User Experience Questionnaire (UEQ) [17], as well as open-ended questions
for patients with MS, their relatives and the attending physicians, different for each
group of interest. Likewise, the acceptance of technology by patient users will be ex-
amined, taking into account earlier studies [18].
Acknowledgements
This research has been co-financed by the European Union and Greek national funds
through the Operational Program Human Resources Growth, Education and Lifelong
10
Learning, under the call for Support of Researchers with Emphasis on Young Research-
ers (Project: eHealth4MS).
References
1. Liguori, M., Marrosu, M.G., Pugliatti, M., Giuliani, F., De Robertis, F., Cocco, E., Zimatore, G.B.,
Livrea, P., Trojano, M.: Age at onset in multiple sclerosis. Neurol. Sci. 21, S825-9 (2000).
2. Doshi, A., Chataway, J.: Multiple sclerosis, a treatable disease, (2017).
https://doi.org/10.7861/clinmedicine.17-6-530.
3. Haase, R., Schultheiss, T., … R.K.-J. of medical, 2012, undefined: Use and acceptance of
electronic communication by patients with multiple sclerosis: a multicenter questionnaire study.
ncbi.nlm.nih.gov.
4. Marrie, R., Leung, S., Tyry, T., Cutter, G., … R.F.-M. sclerosis and, 2019, undefined: Use of
eHealth and mHealth technology by persons with multiple sclerosis. Elsevier.
5. Kane, R., Bever, C., … M.E.-… of telemedicine and, 2008, undefined: Teleneurology in patients
with multiple sclerosis: EDSS ratings derived remotely and from hands-on examination.
journals.sagepub.com.
6. Huijgen, B.C.H., Vollenbroek-Hutten, M.M.R., Zampolini, M., Opisso, E., Bernabeu, M., van
Nieuwenhoven, J., Ilsbroukx, S., Magni, R., Giacomozzi, C., Marcellari, V., Marchese, S.S.,
Hermens, H.J.: Feasibility of a home-based telerehabilitation system compared to usual care:
Arm/hand function in patients with stroke, traumatic brain injury and multiple sclerosis. J.
Telemed. Telecare. 14, 249–256 (2008). https://doi.org/10.1258/jtt.2008.080104.
7. Chang, Y.-J., Chen, C.-H., Lin, L.-F., Han, R.-P., Huang, W.-T., Lee, G.-C.: Wireless sensor
networks for vital signs monitoring: Application in a nursing home. Int. J. Distrib. Sens. Networks.
2012, (2012).
8. Cislo, N., Arbaoui, S., Becis-Aubry, Y., Aubry, D., Parmentier, Y., Doré, P., Guettari, T., Ramdani,
N.: A System for Monitoring Elderly and Dependent People in Nursing Homes: the E-monitor’age
Concept. Stud. Inform. Univ. 11, 30–33 (2013).
9. Radaelli, M., Martinis, M., Locafaro, G., Temussi, S., Mulero, P., Magyari, M., Buron, M.,
Montalban, X., Soerensen, S., Leocani, L., Kieseier, B., Comi, G.: A New Way to Monitor Multiple
Sclerosis. In: IMI 10th Anniversary Scientific Symposium (2018).
10. Stavropoulos, T.G., Meditskos, G., Kompatsiaris, I.: DemaWare2: Integrating sensors, multimedia
and semantic analysis for the ambient care of dementia. Pervasive Mob. Comput. (2016).
https://doi.org/10.1016/j.pmcj.2016.06.006.
11. Lazarou, I., Karakostas, A., Stavropoulos, T.G., Tsompanidis, T., Meditskos, G., Kompatsiaris, I.,
Tsolaki, M.: A Novel and Intelligent Home Monitoring System for Care Support of Elders with
Cognitive Impairment. J. Alzheimer’s Dis. 54, 1561–1591 (2016). https://doi.org/10.3233/JAD-
160348.
12. Motik, B., Grau, B.C., Horrocks, I., Wu, Z., Fokoue, A., Lutz, C.: OWL 2 Web Ontology Language
Profiles (Second Edition), (2012).
13. Eiter, T., Ianni, G., Krennwallner, T., Polleres, A.: Rules and Ontologies for the Semantic Web.
Springer-Verlag (2008). https://doi.org/10.1007/978-3-540-85658-0_1.
14. Janowicz, K., Haller, A., Cox, S.J.D., Le Phuoc, D., Lefrançois, M.: SOSA: A Lightweight
Ontology for Sensors, Observations, Samples, and Actuators.
15. Korakas, N., Tsolaki, M.: Cognitive Impairment in Multiple Sclerosis. Cogn. Behav. Neurol. 29,
55–67 (2016). https://doi.org/10.1097/WNN.0000000000000097.
16. Brooke, J.: SUS - A quick and dirty usability scale. Usability Eval. Ind. (1996).
https://doi.org/10.1002/hbm.20701.
17. Laugwitz, B., Held, T., Schrepp, M.: Construction and Evaluation of a User Experience
Questionnaire. HCI Usability Educ. Work. 63–76 (2008). https://doi.org/10.1007/978-3-540-
89350-9_6.
18. Stavropoulos, T.G., Meditskos, G., Andreadis, S., Kompatsiaris, I.: Real-time health monitoring
and contextualised alerts using wearables. In: Proceedings of 2015 International Conference on
Interactive Mobile Communication Technologies and Learning, IMCL 2015 (2015).
https://doi.org/10.1109/IMCTL.2015.7359619.
