Conference PaperPDF Available

Performance of a wrist-worn multimodal seizure detection system for more than a year in real-life settings

Authors:
  • Eni SpA - San Donato Milanese (MI)

Abstract

Purpose The Embrace smartwatch combines accelerometer and electrodermal activity sensors to alert to generalized tonic-clonic seizures (GTCSs). While a multi-site clinical study validated 94.5% sensitivity (Se) of the GTCS detector with a false alarm rate (FAR) typically less than 0.2/day (Onorati et al, Epilepsia 2017), longitudinal analysis of Embrace performance is needed to understand its reliability and robustness in outpatient settings. A first analysis on ~1 year of data from one user showed Se=98% and FAR per day worn=0.14 (Regalia et al, Mobil 2017, Copenhagen). Here, we report on new longitudinal data from three users. Method One male (P1, 29y) and two female (P2 and P3, 15y) patients used Embrace from 04/2016 to 12/2017 in real-life. Each Embrace alarm generated a call to patients’ caregivers, and it was logged on an online database. Patients and caregivers were instructed to carefully mark any false alarms and any missed seizures. Two seizure-data experts independently inspected all the recordings to verify the accuracy of users’ self-reports. Results Overall, 618 (P1), 437 (P2) and 554 (P3) recording days were collected (average hours per day=23, 19 and 21, respectively). For P1, all 201 GTCSs were detected (Se=100%), with FAR=0.11. For P2, all 64 GTCSs were detected (Se=100%), with FAR=0.32. For P3, 63 out 65 GTCSs were detected (Se=97%), with FAR=0.12. Nocturnal seizures represented 87%, 92% and 43%, respectively, of each patients’ GTCS and were all detected except for one GTCS experienced by P3. No false alarms happened during sleep. Conclusion For the first time, the real-life long-term performance of a wearable GTCSs detector, i.e. Embrace, was evaluated for multiple users, each collecting data for more than 1.5 years, while diligently cross-checking each alarm with self-reports. Embrace provided an excellent trade-off between Se and FAR for all the testers. Analyses on more patients are underway.
Performance of a wrist-worn multimodal seizure detection system
for more than a year in real-life settings
F. Onorati 1, C. Caborni 1,MF Guzman 1,G.Regalia 1, R.W. Picard 1,2
1. Empatica Srl, Milan, Italy
2. MIT Media Lab, Cambridge (MA)
Embrace
(made by Empatica) is an
FDA-approved
wrist-worn device using machine learning applied to 3-
axis
accelerometer (ACC)
and
electrodermal activity (EDA)
data to monitor and alert to
generalized
tonic-clonic seizures (GTCSs)
(Figure 1).
Here we report the performance of Embrace from 3 PWEs
wearing the device for
more than 1 year in real-life settings
.
Figure 1.
Embrace wristband detects an event and transmits an alert to a smartphone, which
generates a call to designated caregivers.
Methods
Two retrospective clinical studies
in EMU’s (single site
[1]
and multi-site
[2]
)assessed the
performance of
the detection system
on video-EEG labelled data, finding sensitivity higher than 94% with false
positives less than once every 4days (false alarm rate (FAR) of 0.2 per day).
Comparable performance
has been found
in real-life settings
using
diary annotations as ground truth [3]
[4],
as well as in a
longitudinal analysis
of Embrace data collected from one patient with epilepsy (PWE)
for almost
a full year [5]
.
Conclusions
1. 3PWE with GTCSs
were continuously monitored with Embrace
in real-life settings
for more than
1year
(04/2016 to 12/2017):
one male (
P1
,29 y) and two females (
P2
and
P3
, both 15 y).
2. PWEs and/or their caregivers were instructed
to carefully
annotate
false alarms and missed seizures using the dedicated
mobile app
.
Figure 2
shows some of the GTCSs annotated by
P1
.
3. Embrace analyzed sensor data in
real-time.
Embrace’s alerts
were logged on an
online database
.
4. 2GTCS
data experts
independently and retrospectively
inspected all the recordings to verify annotations.
5. Overall,
618
(
P1
),
437
(
P2
) and
554
(
P3
)
recording days
were
collected (average hours per day were 23,19 and 21 hrs,
respectively).
6. Sensitivity (Se)
was computed as the percentage of true
triggered alerts on the total of GTCSs.
False alarms (FA)
were
counted by subtracting the number of correctly recognized
GTCSs from the total of alerts.
For
P1
and
P2
,all GTCSs were detected (201 and 64, respectively), resulting in
Se=100%
, with
FAR=0.11
and
FAR=0.32
, respectively. For
P3
,63 out 65 GTCSs were detected (
Se=97%
), with
FAR=0.12
.
Figure 3
shows the performance for each PWE.
Nocturnal seizures
represented 87%, 92% and 43%of all GTCSs, respectively, and were
all detected
except for one GTCS experienced by
P3
(
Figure 4A
).
No false alarms
happened during
rest time
(
Figure 4B
).
The
FAs to GTCSs ratio
was
lower than 1
for most weeks (example for
P1
in
Figure 5
).
For the first time, the
real-life long-term
performance of awearable GTCSs
detector, i.e. Embrace, was evaluated for multiple users, each collecting
data for
as long as 1.5 years
, while diligently annotating each event.
For all the users, Embrace provided an
agreeable tradeoff
between
Sensitivity (>97%) and False alarm rate (<0.4), which is
in line with the best
case clinical settings performances [1][2]
.
Even if information from PWEs and caregivers are considered a standard
ground truth
for field studies
[6]
, the addition of
automatic labeling
(e.g.,
wearable EEG patches) will be considered for
future analyses
.
Purpose
Aim
Results
Figure 2.
Wrist accelerometry (ACC) raw signals
during 101 out of
P1
’s 201 GTCSs exhibited over
the monitoring period. The zero refers to the
times Embrace detected each GTCS.
Figure 4. (A)
Number of GTCSs for rest and active periods, for
the three PWEs over the monitoring period.
(B)
Number of
false alarms (FAs) for rest and active periods.
Figure 5. (A)
Weekly amount of GTCSs and FAs for
P1
.
(B)
Ratio between the number of FAs
and the number of GTCSs over weeks. The red dashed line represents an acceptable upper
limit
P1 P2
P3
90
92
94
96
98
100
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Sensitivity
False Alarm Rate (FAR)
Figure 3.
Sensitivity and FAR (number of false alarms per day) for the
three PWEs over the whole monitoring period.
P3
P2
P1
A B
A
B
[1]
Poh et al 2012, Convulsive seizure detection using a wrist-worn electrodermal activity and accelerometry biosensor, Epilepsia, 53(5):e93-7.
[2]
Onorati, Regalia et al 2017, Multicenter clinical assessment of improved wearable multimodal convulsive seizure detectors, Epilepsia, 58(11):1870-1879.
[3]
Picard et al 2016,Embrace, awearable convulsive seizure detection and alert system – First performance report of acase study in real-life settings, American Epilepsy Society
meeting 2016.
[4]
Caborni et al 2017, Tuning decision thresholds for active/ rest periods significantly improves seizure detection algorithm performance: an evaluation using Embrace smartwatch
on outpatient settings, 32° International Epilepsy Congress.
[5]
Regalia et al 2017,Real-time seizure detection performance with Embrace alert system: one year real-life setting case study, MobilHealth, Copenhagen 2017.
[6]
Benickzy and Ryvlin 2018, Standards for testing and clinical validation of seizure detection devices, Epilepsia, 59(1):9-13.
References
Francesco Onorati, PhD
fo@empatica.com
Rosalind Picard, ScD
rp@empatica.com
Giulia Regalia, PhD
gr@empatica.com
Contacts
... This is, however, a very challenging goal given the broad range of characteristics and lack of apparent ictal signal in non-EEG biomarkers for seizures without motor semiology. Furthermore, most prior validation studies have been performed in patients during in-hospital monitoring with limited mobility, and few studies exist assessing wearable sensors in an ambulatory setting (3). The limited movement in the hospital environment may lead to artificially low false-positive seizure detection rates with movement-sensitive devices. ...
... Recent studies of seizure detection using data from wearable devices have focused on motor seizures with tonic-clonic symptoms in which the primary signal is accelerometry (3,9) (10,11). Onorati et al. (9) used hand-annotated inpatient video-electroencephalographic (vEEG) seizure events collected from 22 patients and compared to electrodermal activity (EDA) and accelerometer (ACC) signals recorded for 55 convulsive seizures (six focal tonic-clonic seizures and 49 focal to bilateral tonic-clonic seizures). ...
Article
Objective: Detection of seizures using wearable devices would improve epilepsy management, but reliable detection of seizures in an ambulatory environment remains challenging, and current studies lack concurrent validation of seizures with EEG. Approach: An adaptively trained Long Short-Term Memory (LSTM) deep neural network was developed to be trained with a modest number of seizures from wrist-worn device data. Transfer learning was used to adapt the classifier initially trained on intracranial electroencephalography (iEEG) signals to facilitate classification of non-EEG physiological datasets comprised of accelerometry, blood volume pulse, skin electrodermal activity, heart rate and temperature signals, and the algorithm's performance was assessed with and without pre-training on iEEG and transfer learning. To assess the performance of the seizure detection classifier on long-term ambulatory data, wearable devices were used for multiple months with an implanted neurostimulator capable of recording iEEG, which provided independent electrographic seizure detections that were reviewed by a board-certified epileptologist. Main results: For 19 motor seizures from 10 in-hospital patients, the algorithm yielded mean AUC, sensitivity, and FAR/day of 0.98, 0.93, and 2.3, respectively. Additionally, for eight seizures with probable motor semiology from two ambulatory patients, the classifier achieved a mean AUC of 0.97, and FAR of 2.45 events/day at a sensitivity of 0.9. For all seizure types in the ambulatory setting the classifier had mean AUC of 0.82 with sensitivity of 0.47 and FAR of 7.2 events/day. Significance: The performance of the algorithm was evaluated on motor and non-motor seizures during in-hospital and ambulatory use. The classifier was able to detect multiple types of motor and non-motor seizures, but performed significantly better on motor seizures.
... al monitored a patient for more than a year and sent alarms when their device detected a seizure. The false alarm rate was lower than 0.4% and the sensitivity over 97% [26]. In another study by empatica's researches [8], it was explored how to detect non-convulsive seizures using heart rate, blood oxygen and electrodermal activity. ...
... In another conference presentation, data were presented from three patients who had collected Embrace data for more >1 year each. 58 In total, the three patients collected data for 1609 days, experiencing 330 GTCSs in total, a high seizure rate. Sensitivity was 100% in two patients and 97% in the third; false-alarm rate varied from 0.11 per day to 0.32 per day. ...
Article
Full-text available
In patients with epilepsy, the potential to prevent seizure‐related injuries and to improve the unreliability of seizure self‐report have fostered the development and marketing of numerous seizure detection devices for home use. Understanding the requirements of users (patients and caregivers) is essential to improve adherence and mitigate barriers to the long‐term use of such devices. Here we reviewed the evidence on the needs and preferences of users and provided an overview of currently marketed devices for seizure detection (medically approved or with published evidence for their performance). We then compared devices with known needs. Seizure‐detection devices are expected to improve safety and clinical and self‐management, and to provide reassurance to users. Key factors affecting a device’s usability relate to its design (attractive appearance, low visibility, low intrusiveness), comfort of use, confidentiality of recorded data, and timely support from both technical and clinical ends. High detection sensitivity and low false alarm rates are paramount. Currently marketed devices are focused primarily on the recording of non–electroencephalography (EEG) signals associated with tonic‐clonic seizures, whereas the detection of focal seizures without major motor features remains a clear evidence gap. Moreover, there is paucity of evidence coming from real‐life settings. A joint effort of clinical and nonclinical experts, patients, and caregivers is required to ensure an optimal level of acceptability and usability, which are key aspects for a successful continuous monitoring aimed at seizure detection at home.
... • DA1 -an online DA, which has showed promising ambulatory performances [2]. ...
... A real-life longitudinal analysis was made on one patient affected by Dravet syndrome, monitored for 113 days. In this outpatient, 22 out of 24 convulsive seizures were detected (Sens=92%), with FAR=0.35(Picard et al, 2016). Results from three users monitored for more than 1.5 years/each, with a total of 331 GTCS, achieved Sens=99.4% with FAR=0.18(Onorati et al, 2018). ...
Article
Wearable automated seizure detection devices offer a high potential to improve seizure management, through continuous ambulatory monitoring, accurate seizure counts, and real-time alerts for prompt intervention. More importantly, these devices can be a life-saving help for people with a higher risk of sudden unexpected death in epilepsy (SUDEP), especially in case of generalized tonic-clonic seizures (GTCS). The Embrace and E4 wristbands (Empatica) are the first commercially available multimodal wristbands that were designed to sense the physiological hallmarks of ongoing GTCS: while Embrace only embeds a machine learning-based detection algorithm, both E4 and Embrace devices are equipped with motion (accelerometers, ACC) and electrodermal activity (EDA) sensors and both the devices received medical clearance (E4 from EU CE, Embrace from EU CE and US FDA). The aim of this contribution is to provide updated evidence of the effectiveness of GTCS detection and monitoring relying on the combination of ACM and EDA sensors. A machine learning algorithm able to recognize ACC and EDA signatures of GTCS-like events has been developed on E4 data, labeled using gold-standard video-EEG examined by epileptologists in clinical centers, and has undergone continuous improvement. While keeping an elevated sensitivity to GTCS (92-100%), algorithm improvements and growing data availability led to lower false alarm rate (FAR) from the initial ˜2 down to 0.2-1 false alarms per day, as showed by retrospective and prospective analyses in inpatient settings. Algorithm adjustment to better discriminate real-life physical activities from GTCS, has brought the initial FAR of ˜6 on outpatient real life settings, down to values comparable to best-case clinical settings (FAR < 0.5), with comparable sensitivity. Moreover, using multimodal sensing, it has been possible not only to detect GTCS but also to quantify seizure-induced autonomic dysfunction, based on automatic features of abnormal motion and EDA. The latter biosignal correlates with the duration of post-ictal generalized EEG suppression, a biomarker observed in 100% of monitored SUDEP cases.
Article
Introduction Wearable devices for continuous seizure monitoring have drawn increasing attention in the field of epilepsy research. One of the parameters recorded by these devices is electrodermal activity (EDA). The aim of this study was to systematically review the literature to estimate the incidence of electrodermal response during seizures. Methods We searched all articles recording concurrent EDA and EEG activity during the pre-ictal, ictal, and postictal periods in children and adults with epilepsy. Studies reporting the total number of seizures and number of seizures with an EDA response were included for a random-effects meta-analysis. Results Nineteen studies, including 550 participants and 1115 seizures were reviewed. All studies demonstrated an EDA increase during the ictal and postictal periods, while only three reported pre-ictal EDA responses. The meta-analysis showed a pooled EDA response incidence of 82/100 seizures (95% CI 70–91). Tonic-clonic seizures (both generalized tonic-clonic seizures (GTCS) and focal to bilateral tonic-clonic seizures (FBTCS)) elicited a more pronounced (higher and longer-lasting) EDA response when compared with focal seizures (excluding FBTCS). Discussion Epileptic seizures produce an electrodermal response detectable by wearable devices during the pre-ictal, ictal, and postictal periods. Further research is needed to better understand EDA changes and to analyze factors which may influence the EDA response.
Article
Full-text available
Objective: New devices are needed for monitoring seizures, especially those associated with sudden unexpected death in epilepsy (SUDEP). They must be unobtrusive and automated, and provide false alarm rates (FARs) bearable in everyday life. This study quantifies the performance of new multimodal wrist-worn convulsive seizure detectors. Methods: Hand-annotated video-electroencephalographic seizure events were collected from 69 patients at six clinical sites. Three different wristbands were used to record electrodermal activity (EDA) and accelerometer (ACM) signals, obtaining 5,928 h of data, including 55 convulsive epileptic seizures (six focal tonic-clonic seizures and 49 focal to bilateral tonic-clonic seizures) from 22 patients. Recordings were analyzed offline to train and test two new machine learning classifiers and a published classifier based on EDA and ACM. Moreover, wristband data were analyzed to estimate seizure-motion duration and autonomic responses. Results: The two novel classifiers consistently outperformed the previous detector. The most efficient (Classifier III) yielded sensitivity of 94.55%, and an FAR of 0.2 events/day. No nocturnal seizures were missed. Most patients had <1 false alarm every 4 days, with an FAR below their seizure frequency. When increasing the sensitivity to 100% (no missed seizures), the FAR is up to 13 times lower than with the previous detector. Furthermore, all detections occurred before the seizure ended, providing reasonable latency (median = 29.3 s, range = 14.8-151 s). Automatically estimated seizure durations were correlated with true durations, enabling reliable annotations. Finally, EDA measurements confirmed the presence of postictal autonomic dysfunction, exhibiting a significant rise in 73% of the convulsive seizures. Significance: The proposed multimodal wrist-worn convulsive seizure detectors provide seizure counts that are more accurate than previous automated detectors and typical patient self-reports, while maintaining a tolerable FAR for ambulatory monitoring. Furthermore, the multimodal system provides an objective description of motor behavior and autonomic dysfunction, aimed at enriching seizure characterization, with potential utility for SUDEP warning.
Conference Paper
Full-text available
Purpose Embrace (Empatica, Inc., Boston, Massachusetts) is a wrist-worn device coupled with a smartphone-based alert system using accelerometer and electrodermal activity sensors. A machine learning classifier trained on inpatient and outpatient convulsive seizure data provides real-time alarms; its performances in real life settings have been previously shown (Onorati et al., 2016), with Se (sensitivity)=90% and FAR (false alarm rate)=1.05/24hrs. In this work, we show how applying different decision thresholds for rest and active periods can improve performance in outpatient settings. Methods The testing set consists of 27 patients who experienced 111 convulsive seizures (65 during rest, 46 during active periods) for a total of 95 days of recordings in outpatient settings. Patients and caregivers were instructed to accurately report seizures; these reports were used as ground truth. We examined three different thresholds: TH1 for active periods, to minimize false alarm rate (FAR); TH2 for rest periods, to maximize Sensitivity (Se); and TH3 as a cut-off between minimizing the FAR and maximizing the Se. Classifier I was tuned on TH3 only. Classifiers II and III were tuned on TH2 for rest periods, and on TH1 and TH3, respectively, for active periods. Rest periods were determined using a proprietary rest-detection algorithm. Results Classifier I detected 99 seizures (Se=89%) with 55 false alarms (FAR=0.58/24hrs), all during active periods. Classifiers II (Se=92%) and III (Se=93%) detected 4 more seizures during rest, but Classifier II missed one seizure during an active period. While Classifier III had the same FAR as Classifier I, Classifier II showed a 44% decrease in the FAR, i.e. FAR=0.33/24hrs. Conclusion Using different decision thresholds for rest and active periods results in both increased sensitivity and decreased FAR. Thus, performance improvements can be achieved by detecting rest and activity and optimizing discrete decision thresholds for these periods.
Conference Paper
Full-text available
RATIONALE: Empatica (www.empatica.com) is working on the development of an automated comfortably wearable convulsive seizure (CS) detection system relying on accelerometer (ACC) and electrodermal activity (EDA) data (Epilepsia 2012, 53, 93-7). Using machine learning algorithms trained on generalized tonic-clonic seizures (GTCS) gathered inside Epilepsy Monitoring Units (EMUs), the system has achieved sensitivity (Se) of 92-100% and false alarm (FA) rates ranging from 0.48 to 2.02 false alarms per day (Regalia et al. 2015, American Epilepsy Society Annual Meeting; Onorati et al. 2016, Epilepsy Pipeline Conference, Caborni et al. 2016, Partners Against Mortality in Epilepsy). However, EMU settings do not mimic the real-life environment where the system is intended for use. In real-life settings people are engaged in very different physical activities, such as sports and physical labor, which may result in higher FA rates. Moreover, the dynamic and semiology of CSs occurring outside EMUs might be different, influencing their detection. In this work we present a case study from a real-life setting. METHODS: Embrace is a wrist-worn device and smartphone-based alert system which analyses 3-axis acceleration and EDA data from the patient and provides an alert to designated caregivers when an unusual event is detected. The Embrace and alert system are being evaluated in an IRB-approved clinical trial. This case study is of a patient with Dravet Syndrome (14 y.) enrolled in the trial of Embrace in the outpatient setting. No data from this patient was used in training the system. In order to evaluate performance, the patient’s caregiver was asked to meticulously annotate the occurrence of each CS and any activity that generated an alert. The number of FA’s was obtained by subtracting the number of correctly recognized CSs from the total alerts fired by the device. The Se was the percentage of CSs that automatically triggered an alert. RESULTS: Over a period of 113 days, the patient wore the device for 82 days (i.e., 1973 hours, average hours per day: 17.2). The system detected 22 out of 24 CSs (Se=92%). The 2 missed seizures were characterized by a mild motor component and brief duration ( < 50 sec). Figure 1 depicts the distribution of the patient’s seizures according to the hour of the day. The total number of FA was 39, for a FA rate of 0.48 per day worn. FAs were generated by activities such as hands clapping/shaking, car transport and dancing. CONCLUSIONS: In this work, we have reported the performance of an unobtrusive CS detector used by a patient for a period of more than 3 months in a real-life setting, where none of the patient’s data had been used in training the system. The performance, both Se and FA rates, were in the same range as those for data gathered in best-case clinical settings. In ongoing research with more patients, we are seeing similar results. At AES, and in future publications, we will present more detailed evaluations with other patients and healthy subjects engaged in diverse real-life activities.
Article
To increase the quality of studies on seizure detection devices, we propose standards for testing and clinical validation of such devices. We identified 4 key features that are important for studies on seizure detection devices: subjects, recordings, data analysis and alarms, and reference standard. For each of these features, we list the specific aspects that need to be addressed in the studies, and depending on these, studies are classified into 5 phases (0-4). We propose a set of outcome measures that need to be reported, and we propose standards for reporting the results. These standards will help in designing and reporting studies on seizure detection devices, they will give readers clear information on the level of evidence provided by the studies, and they will help regulatory bodies in assessing the quality of the validation studies. These standards are flexible, allowing classification of the studies into one of the 5 phases. We propose actions that can facilitate development of novel methods and devices.