Conference PaperPDF Available

Tuning decision thresholds for active/rest periods significantly improves seizure detection algorithm performance: An evaluation using Embrace Smartwatch on outpatient settings

Authors:

Abstract

Purpose Embrace (Empatica, Inc., Boston, Massachusetts) is a wrist-worn device coupled with a smartphone-based alert system using accelerometer and electrodermal activity sensors. A machine learning classifier trained on inpatient and outpatient convulsive seizure data provides real-time alarms; its performances in real life settings have been previously shown (Onorati et al., 2016), with Se (sensitivity)=90% and FAR (false alarm rate)=1.05/24hrs. In this work, we show how applying different decision thresholds for rest and active periods can improve performance in outpatient settings. Methods The testing set consists of 27 patients who experienced 111 convulsive seizures (65 during rest, 46 during active periods) for a total of 95 days of recordings in outpatient settings. Patients and caregivers were instructed to accurately report seizures; these reports were used as ground truth. We examined three different thresholds: TH1 for active periods, to minimize false alarm rate (FAR); TH2 for rest periods, to maximize Sensitivity (Se); and TH3 as a cut-off between minimizing the FAR and maximizing the Se. Classifier I was tuned on TH3 only. Classifiers II and III were tuned on TH2 for rest periods, and on TH1 and TH3, respectively, for active periods. Rest periods were determined using a proprietary rest-detection algorithm. Results Classifier I detected 99 seizures (Se=89%) with 55 false alarms (FAR=0.58/24hrs), all during active periods. Classifiers II (Se=92%) and III (Se=93%) detected 4 more seizures during rest, but Classifier II missed one seizure during an active period. While Classifier III had the same FAR as Classifier I, Classifier II showed a 44% decrease in the FAR, i.e. FAR=0.33/24hrs. Conclusion Using different decision thresholds for rest and active periods results in both increased sensitivity and decreased FAR. Thus, performance improvements can be achieved by detecting rest and activity and optimizing discrete decision thresholds for these periods.
Convulsive Seizure detection improved by setting different
parameters for active/rest periods.
An evaluation using Embrace Smartwatch
C. Caborni ¹, M. Migliorini ¹, F. Onorati ¹, G. Regalia¹, R.W. Picard ¹ ²
1 Empatica Inc., Milan, Italy
2 MIT Media Lab, Massachuses Institute of Technology, Cambridge, Massachuses, U.S.A
Introduction
Aim
Embrace (Empatica Inc., Cambridge, MA) is a wrist-worn device that exploits electro-dermal activity and accelerometer data in
order to detect epileptic convulsive seizures patterns on signals. A machine learning classier trained on inpatient and outpatient
convulsive seizures data provides real-time alarms. In a dataset with both convulsive seizures data and real life activities which
display patterns that are challenging for the classier to discern from seizures, the classier reached Sensitivity (Se) of 90% and
a False Alarm Rate (FAR) of 1.05/24hrs (Onorati et al., 2016, Improving convulsive seizure detection by exploiting data from outpatient seings using
theEmbrace wristband, 12th European Congress on Epileptology, at Prague, Epilepsia, Volume 57 (Suppl. 2):226–233).
Figure 1
Schematics of Embrace CS detection and alert system, composed of a
wristband that detects an event, transmits an alert to his/her smartphone,
which generates a call via a cloud-based service to designated caregivers.
Methods
Embrace Seizure
detected
No Seizure
Accelerometer
Skin conduction
Convulsive
Seizure
Detector
On-wrist continuous
data recording
Alert generation
and trasmission
Alert delivery
to caregivers
1. The dataset consists of 2286 hours (95 days) of recordings
collected in outpatient settings from 27 subjects (17
females) with epilepsy, ranging from 10yo to 76yo (mean 25yo
+/- 15yo), who accurately reported seizures.
2. The rest periods were determined by a proprietary algorithm.
3. The users experienced 111 convulsive seizures (CSs), 65
during rest periods.
4. We examined three decision thresholds of the actual
classier embedded in Embrace (Onorati et al., 2016).
5. Three classiers were tuned with different decision
thresholds during Rest and Active periods:
Conclusion
Using different decision thresholds
for Rest and Active periods results
in both increased Sensitivity and
decreased FAR;
Further improvements can be
achieved by detecting rest and activity
and optimizing discrete decision
thresholds for these periods;
Real life testing of the thresholds
proposed in Classier III is ongoing.
Contacts
Chiara Caborni
cc@empatica.com
Rosalind Picard
rp@empatica.com
www.empatica.com/product-embrace
Figure 3
Performance of the three classifiers during Rest, Active periods and overall. Sensitivity on the top plot
and False Alarm Rate on the boom.
All the Classier didn’t generate false alarms during Rest periods.
Classier I detected 99 seizures (Se=89%) with 55 false alarms
(FAR=0.58/24hrs), all during Active periods.
Classiers II detected a total of 102 seizures (Se=92%): 4 more seizures
during Rest periods, one less during Active period.
Classiers III detected a 103 seizures (Se=93%), 4 more seizures during
Rest periods with respect to Classier I, while keeping the same FAR.
Results
Classier I
Classier II
Classier III
TH2
TH3
TH3
TH2
TH1
TH2
Rest Active Figure 2
The curve shows the performance of the seizures detector: each point of a
curve represents the number of false alarms per day and the percentage of
CSs correctly recognized at a specific value of the decision threshold.
TH3: detects more seizures
at a cost of more false alarms;
TH1: provides less false alarms
at a cost of less detected seizures;
TH2: cut-off between detecting
seizures and reducing the false alarms.
0051
In this work we explore how applying different decision
threshold for rest and active periods can improve the seizure
detection algorithm performance in outpatient settings.
... In 27 outpatients, a total of 2286 hours of Empatica sensor data were collected and compared with seizure diary data collected by the patient and/or caregivers. 57 Therefore, each patient was monitored for an average of 3.5 days. During this time, a total of 111 GTCSs were recorded in the patients' seizure diaries, a very high seizure rate. ...
Article
Full-text available
In patients with epilepsy, the potential to prevent seizure‐related injuries and to improve the unreliability of seizure self‐report have fostered the development and marketing of numerous seizure detection devices for home use. Understanding the requirements of users (patients and caregivers) is essential to improve adherence and mitigate barriers to the long‐term use of such devices. Here we reviewed the evidence on the needs and preferences of users and provided an overview of currently marketed devices for seizure detection (medically approved or with published evidence for their performance). We then compared devices with known needs. Seizure‐detection devices are expected to improve safety and clinical and self‐management, and to provide reassurance to users. Key factors affecting a device’s usability relate to its design (attractive appearance, low visibility, low intrusiveness), comfort of use, confidentiality of recorded data, and timely support from both technical and clinical ends. High detection sensitivity and low false alarm rates are paramount. Currently marketed devices are focused primarily on the recording of non–electroencephalography (EEG) signals associated with tonic‐clonic seizures, whereas the detection of focal seizures without major motor features remains a clear evidence gap. Moreover, there is paucity of evidence coming from real‐life settings. A joint effort of clinical and nonclinical experts, patients, and caregivers is required to ensure an optimal level of acceptability and usability, which are key aspects for a successful continuous monitoring aimed at seizure detection at home.
Conference Paper
Full-text available
Purpose The Embrace smartwatch combines accelerometer and electrodermal activity sensors to alert to generalized tonic-clonic seizures (GTCSs). While a multi-site clinical study validated 94.5% sensitivity (Se) of the GTCS detector with a false alarm rate (FAR) typically less than 0.2/day (Onorati et al, Epilepsia 2017), longitudinal analysis of Embrace performance is needed to understand its reliability and robustness in outpatient settings. A first analysis on ~1 year of data from one user showed Se=98% and FAR per day worn=0.14 (Regalia et al, Mobil 2017, Copenhagen). Here, we report on new longitudinal data from three users. Method One male (P1, 29y) and two female (P2 and P3, 15y) patients used Embrace from 04/2016 to 12/2017 in real-life. Each Embrace alarm generated a call to patients’ caregivers, and it was logged on an online database. Patients and caregivers were instructed to carefully mark any false alarms and any missed seizures. Two seizure-data experts independently inspected all the recordings to verify the accuracy of users’ self-reports. Results Overall, 618 (P1), 437 (P2) and 554 (P3) recording days were collected (average hours per day=23, 19 and 21, respectively). For P1, all 201 GTCSs were detected (Se=100%), with FAR=0.11. For P2, all 64 GTCSs were detected (Se=100%), with FAR=0.32. For P3, 63 out 65 GTCSs were detected (Se=97%), with FAR=0.12. Nocturnal seizures represented 87%, 92% and 43%, respectively, of each patients’ GTCS and were all detected except for one GTCS experienced by P3. No false alarms happened during sleep. Conclusion For the first time, the real-life long-term performance of a wearable GTCSs detector, i.e. Embrace, was evaluated for multiple users, each collecting data for more than 1.5 years, while diligently cross-checking each alarm with self-reports. Embrace provided an excellent trade-off between Se and FAR for all the testers. Analyses on more patients are underway.
ResearchGate has not been able to resolve any references for this publication.