Conference Paper

Smartwatch based respiratory rate estimation during sleep using CNN/LSTM neural network

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... Bidirectional-LSTM model has been applied to support the radiotherapy treatment for thoracic-abdominal tumors by analyzing respiratory motions (40). LSTM has been combined with CNN to determine respiratory actions during sleep (41). ...
... The key feature of CNN is convolution operation which provides a form of automated feature extraction (41). CNN's have been combined with LSTM networks (CNN-LSTM) to learn temporal features in tasks such as video image captioning (73), multi-view object recognition (74) and other fields such as medicine (41; 73; 75; 76; 77; 78). ...
... CNN is used for a multi-channel signal to extract deep features followed by LSTM for prediction. Havriushenko et al. (41) factorized photoplethysmogram signals into spatial and cross channels and applied depth-wise convolutions separately to each channel. CNN-LSTM has shown major improvement in classification over machine learning methods such as Dense-Net (132), extreme learning machine (78), and random forests (73). ...
Article
Full-text available
In the past decade, deep learning models have been applied to bio-sensors used in a body sensor network for prediction. Given recent innovations in this field, the prediction accuracy of novel models needs to be evaluated for bio-signals. In this paper, we evaluate the performance of deep learning models for respiratory rate prediction. We consider three datasets from bio-sensors which include electrocardiogram (ECG), photoplethysmogram (PPG) data, and surface electromyogram (sEMG) data. The deep learning models include Long short-term memory (LSTM) networks, Bidirectional LSTM (Bi-LSTM), attention-based variants of LSTM, CNN-LSTM and Convolutional-LSTM networks. The deep learning models are evaluated for two separate windows which are 32 s and 64 s window. The models’ performance is evaluated using mean absolute error (MAE). The 64 s window has more accurate prediction compared to the 32 s window. Our results indicate Bi-LSTM with Bahdanu Attention has the best performance for the bio-signals. LSTM performs best with one of the datasets, yielding an MAE of 0.70 ± 0.02. Bi-LSTM with Bahdanau attention showed best results with two of the three datasets with MAE of 0.51 ± 0.03 for sEMG based data and MAE of 0.24 ± 0.03 with PPG and ECG based data.
... In terms of mobile and wearable devices, extracted physiological data (for example, using the most popular HRV and activity information which can be simply measured by standard accelerometers, gyroscopes or plethysmography (PPG) sensors available almost in all wearable and mobile devices) can be effectively used to estimate a number of related vital signs and pathological states (such as blood oxygen saturation, sleep apnea and hypopnea, snore, blood pressure, etc.) by applying specialized algorithms [3,[7][8][9]. Recently, deep-learning based models have shown promising results in the field of biomedical engineering, in particular for the analysis of sensors data, recognition of specific medical patterns, identification of hidden models, and decision-making in the field of healthcare. ...
... As a basis of our current system the algorithms developed in our previous papers for sleep stages analysis, respiratory events classification and smart alarm, were used [3,[7][8][9]15]. ...
... We use the same dataset as in our recent studies [3,[7][8][9]15]. The full dataset consists of 263 logs from 176 different users and was prepared by Samsung Medical Center. ...
Conference Paper
Recently, mobile and wearable devices have become an increasingly integral part of our lives. They provide a possibility of detailed health monitoring using noninvasive and user-friendly techniques. However, lack of continuous monitoring, the need of specific sensors, and the limitations in memory and power consumption are only some of the potential drawbacks of such devices. In the current paper a system based on a deep recurrent neural network is developed for an automatic continuous monitoring of sleep-related physiological parameters by means of a wearable biosignal monitoring systems. Smartwatches based algorithm for non-invasive monitoring of sleep stages, respiratory events (including sleep apnea and hypopnea), snore and blood oxygen saturation is developed. Our experimental results demonstrate that proposed model constitutes a noninvasive and inexpensive screening system for sleep-related physiological parameters and pathological states. The model has shown a 77 % accuracy in sleep stages prediction, more than 80 % accuracy in epoch-by-epoch respiratory events classification, above 60 % accuracy in snore events classification and above 70 % accuracy in blood oxygen saturation (SpO2) level classification (for a two class problem with a SpO2 threshold of 95 %).
Conference Paper
The COVID-19 pandemic has induced dramatic effects in all areas of society worldwide. It has led to drastic restrictions on academic exchanges and caused research projects to be put on hold. The need to maintain social distancing has severely affected the research communities that rely on in-person studies. In particular, Quality of Experience (QoE) research relies heavily on user studies and in-person subjective tests as means of obtaining ground truths for system designs. In this position paper, we focus on the impact of COVID-19 on conducting subjective tests for digital media quality assessment. An overview of the related discussions that take place in associated research communities is provided. A number of challenges are posed in terms of hygiene, ethics, standardization, and research quality issues. Opportunities for consolidating QoE research under COVID-19 and beyond are put forward, including alternative experimental designs and adaptation of research methodologies. Numerous enablers are suggested and discussed, revealing various avenues that can be followed to assist in conducting subjective tests under pandemic conditions. We believe that this position paper will be helpful for researchers and practitioners relying on in- person studies to keep abreast about the potential measures that support coming out of the COVID-19 pandemic strong and being prepared for future endeavors.
Article
Full-text available
There is an ever-growing demand for measuring respiratory variables during a variety of applications, including monitoring in clinical and occupational settings, and during sporting activities and exercise. Special attention is devoted to the monitoring of respiratory rate because it is a vital sign, which responds to a variety of stressors. There are different methods for measuring respiratory rate, which can be classed as contact-based or contactless. The present paper provides an overview of the currently available contact-based methods for measuring respiratory rate. For these methods, the sensing element (or part of the instrument containing it) is attached to the subject’s body. Methods based upon the recording of respiratory airflow, sounds, air temperature, air humidity, air components, chest wall movements, and modulation of the cardiac activity are presented. Working principles, metrological characteristics, and applications in the respiratory monitoring field are presented to explore potential development and applicability for each method.
Conference Paper
Full-text available
Depthwise separable convolution has shown great efficiency in network design, but requires time-consuming training procedure with full training-set available. This paper first analyzes the mathematical relationship between regular convolutions and depth-wise separable convolutions, and proves that the former one could be approximated with the latter one in closed form. We show depthwise separable convolutions are principal components of regular convolutions. And then we propose network decoupling (ND), a training-free method to accelerate convolutional neural networks (CNNs) by transferring pre-trained CNN models into the MobileNet-like depthwise separable convolu-tion structure, with a promising speedup yet negligible accuracy loss. We further verify through experiments that the proposed method is orthogonal to other training-free methods like channel decomposition, spatial decomposition, etc. Combining the proposed method with them will bring even larger CNN speedup. For instance, ND itself achieves about 2× speedup for the widely used VGG16, and combined with other methods, it reaches 3.7× speedup with graceful accuracy degradation. We demonstrate that ND is widely applicable to classification networks like ResNet, and object detection network like SSD300.
Article
Full-text available
The human has a very complex perception system, including vision, auditory, olfactory, touch, and gustation. This paper will introduce the recent studies about providing a technical solution for image recognition, by applying a algorithm called Convolutional Neural Network (CNN) which is inspired by animal visual system. Convolution serves as a perfect realization of an optic nerve cell which merely responds to its receptive field and it performs well in image feature extraction. Being highly-hierarchical networks, CNN is structured with a series of different functional layers. The function blocks are separated and described clearly by each layer in this paper. Additionally, the recognition process and result of a pioneering CNN on MNIST database are presented.
Article
Full-text available
In this study, we proposed a novel method for extracting the instantaneous respiratory rate (IRR) from the pulse oximeter photoplethysmogram (PPG). The method was performed in three main steps: (1) a time-frequency transform called synchrosqueezing transform (SST) was used to extract the respiratory-induced intensity, amplitude and frequency variation signals from PPG, (2) the second SST was applied to each extracted respiratory-induced variation signal to estimate the corresponding IRR, and (3) the proposed peak-conditioned fusion method then combined the IRR estimates to calculate the final IRR. We validated the implemented method with capnography and nasal/oral airflow as the reference RR using the limits of agreement (LOA) approach. Compared to simple fusion and single respiratory-induced variation estimations, peak-conditioned fusion shows better performance. It provided a bias of 0.28 bpm with the 95% LOAs ranging from −3.62 to 4.17, validated against capnography and a bias of 0.04 bpm with the 95% LOAs ranging from −5.74 to 5.82, validated against nasal/oral airflow. This algorithm would expand the functionality of a conventional pulse oximetry beyond the measurement of heart rate and oxygen saturation to measure the respiratory rate continuously and instantly.
Conference Paper
Full-text available
Photoplethysmography (PPG) is one of the key technologies for unobtrusive physiological monitoring, with ongoing attempts to use it in several medical fields, ranging from night to night sleep analysis to continuous cardiac arrhythmia monitoring. However, the PPG signals are susceptible to be corrupted by noise and artifacts, caused, e.g., by limb or sensor movement. These artifacts affect the morphology of PPG waves and prevent the accurate detection and localization of beats and subsequent cardiovascular feature extraction. In this paper a new algorithm for beat detection and pulse quality assessment is described. The algorithm segments the PPG signal in pulses, localizes each beat and grades each segment with a quality index. The obtained index results from a comparison between each pulse and a template derived from the surrounding pulses, by mean of dynamic time warping barycenter averaging. The quality index is used to discard corrupted pulse beats. The algorithm is evaluated by comparing the detected beats with annotated PPG signals and the results are published over the same data. The described method achieves an improved sensitivity and a higher predictive value.
Article
Full-text available
Sleep apnea is highly prevalent in patients with cardiovascular disease. These disordered breathing events are associated with a profile of perturbations that include intermittent hypoxia, oxidative stress, sympathetic activation, and endothelial dysfunction, all of which are critical mediators of cardiovascular disease. Evidence supports a causal association of sleep apnea with the incidence and morbidity of hypertension, coronary heart disease, arrhythmia, heart failure, and stroke. Several discoveries in the pathogenesis, along with developments in the treatment of sleep apnea, have accumulated in recent years. In this review, we discuss the mechanisms of sleep apnea, the evidence that addresses the links between sleep apnea and cardiovascular disease, and research that has addressed the effect of sleep apnea treatment on cardiovascular disease and clinical endpoints. Finally, we review the recent development in sleep apnea treatment options, with special consideration of treating patients with heart disease. Future directions for selective areas are suggested.
Article
Full-text available
The conservation of sleep across all animal species suggests that sleep serves a vital function. We here report that sleep has a critical function in ensuring metabolic homeostasis. Using real-time assessments of tetramethylammonium diffusion and two-photon imaging in live mice, we show that natural sleep or anesthesia are associated with a 60% increase in the interstitial space, resulting in a striking increase in convective exchange of cerebrospinal fluid with interstitial fluid. In turn, convective fluxes of interstitial fluid increased the rate of β-amyloid clearance during sleep. Thus, the restorative function of sleep may be a consequence of the enhanced removal of potentially neurotoxic waste products that accumulate in the awake central nervous system.
Article
Full-text available
A cardiorespiratory-based automatic sleep staging system for subjects with sleep-disordered breathing is described. A simplified three-state system is used: Wakefulness (W), rapid eye movement (REM) sleep (R), and non-REM sleep (S). The system scores the sleep stages in standard 30-s epochs. A number of features associated with the epoch RR-intervals, an inductance plethysmography estimate of rib cage respiratory effort, and an electrocardiogram-derived respiration (EDR) signal were investigated. A subject-specific quadratic discriminant classifier was trained, randomly choosing 20% of the subject's epochs (in appropriate proportions of W, S and R) as the training data. The remaining 80% of epochs were presented to the classifier for testing. An estimated classification accuracy of 79% (Cohen's kappa value of 0.56) was achieved. When a similar subject-independent classifier was trained, using epochs from all other subjects as the training data, a drop in classification accuracy to 67% (kappa = 0.32) was observed. The subjects were further broken in groups of low apnoea-hypopnea index (AHI) and high AHI and the experiments repeated. The subject-specific classifier performed better on subjects with low AHI than high AHI; the performance of the subject-independent classifier is not correlated with AHI. For comparison an electroencephalograms (EEGs)-based classifier was trained utilizing several standard EEG features. The subject-specific classifier yielded an accuracy of 87% (kappa = 0.75), and an accuracy of 84% (kappa = 0.68) was obtained for the subject-independent classifier, indicating that EEG features are quite robust across subjects. We conclude that the cardiorespiratory signals provide moderate sleep-staging accuracy, however, features exhibit significant subject dependence which presents potential limits to the use of these signals in a general subject-independent sleep staging system.
Conference Paper
Full-text available
Sleep apnea is a sleep disorder with a high prevalence in the adult male population. Sleep apnea is regarded as an independent risk factor for cardiovascular sequelae such as ischemic heart attacks and stroke. The diagnosis of sleep apnea requires polysomnographic studies in sleep laboratories with expensive equipment and attending personnel. Sleep apnea can be treated effectively using nasal ventilation therapy (nCPAP). Early recognition and selection of patients with sleep related breathing disorders is an important task. Although it has been suggested that this can be done on the basis of the ECG, careful quantitative studies of the accuracy of such techniques are needed. An annotated database with 70 nighttime ECG recordings has been created to support such studies. The annotations were based on visual scoring of disordered breathing during sleep
Article
Wireless detection of respiration rates is crucial for many applications. Most of the state-of-art solutions estimate breathing rates with the prior knowledge of crowd numbers as well as assuming the distinct breathing rates of different users, which is neither natural nor realistic. However, few of them can leverage the estimated breathing rates to recognize human subjects (a.k.a, identity matching). In this work, using the channel state information (CSI) of a single pair of commercial WiFi devices, a novel system is proposed to continuously track the breathing rates of multiple persons without such impractical assumptions. The proposed solution includes an adaptive subcarrier combination method that boosts the signal to noise ratio (SNR) of breathing signals, and an iterative dynamic programming and a trace concatenating algorithm that continuously track the breathing rates of multiple users. By leveraging both the spectrum and time diversity of the CSI, our system can correctly extract the breathing rate traces even if some of them merge together for a short time period. Furthermore, by utilizing the breathing traces obtained, our system can do people counting and recognition simultaneously. Extensive experiments are conducted in two environments (an on-campus lab and a car). The results show that 86% of average accuracy can be achieved for people counting up to 4 people for both cases. For 97.9% out of all the testing cases, the absolute error of crowd number estimates is within 1. The system achieves an average accuracy of 85.78% for people recognition in a smart home case.
Conference Paper
This paper provides a novel best in the domain automatic classifier of sleep stages based on wrist photoplethysmography and 3D-accelerometer data obtained from smartwatches. The sleep is classified into rapid-eye-movement (REM), Light, Deep, or Wake stages. State of the art classifiers based on wearable sensors suffer from motion artifacts and apnea events. Proposed novel techniques of artifacts and apnea events elimination result in the high accuracy of sleep stages classification and robustness to apnea events. The model provides the Cohen's Kappa score of 0.65 and accuracy of 0.80 on 254 night-logs of 173 subjects with a broad distribution of apnea-hypopnea index until 90. The approach is applicable to unobtrusive sleep monitoring by wearables.
Article
There is a growing interest in designing models that can deal with images from different visual domains. If there exists a universal structure in different visual domains that can be captured via a common parameterization, then we can use a single model for all domains rather than one model per domain. A model aware of the relationships between different domains can also be trained to work on new domains with less resources. However, to identify the reusable structure in a model is not easy. In this paper, we propose a multi-domain learning architecture based on depthwise separable convolution. The proposed approach is based on the assumption that images from different domains share cross-channel correlations but have domain-specific spatial correlations. The proposed model is compact and has minimal overhead when being applied to new domains. Additionally, we introduce a gating mechanism to promote soft sharing between different domains. We evaluate our approach on Visual Decathlon Challenge, a benchmark for testing the ability of multi-domain models. The experiments show that our approach can achieve the highest score while only requiring 50% of the parameters compared with the state-of-the-art approaches.
Article
Sleep apnea is highly prevalent in patients with cardiovascular disease. These disordered breathing events are associated with a profile of perturbations that include intermittent hypoxia, oxidative stress, sympathetic activation, and endothelial dysfunction, all of which are critical mediators of cardiovascular disease. Evidence supports a causal association of sleep apnea with the incidence and morbidity of hypertension, coronary heart disease, arrhythmia, heart failure, and stroke. Several discoveries in the pathogenesis, along with developments in the treatment of sleep apnea, have accumulated in recent years. In this review, we discuss the mechanisms of sleep apnea, the evidence that addresses the links between sleep apnea and cardiovascular disease, and research that has addressed the effect of sleep apnea treatment on cardiovascular disease and clinical endpoints. Finally, we review the recent development in sleep apnea treatment options, with special consideration of treating patients with heart disease. Future directions for selective areas are suggested.
Article
This article derives an estimation procedure to evaluate how many Monte Carlo realizations need to be done in order to achieve prescribed accuracies in the estimated mean value and also in the cumulative probabilities of achieving values greater than, or less than, a particular value as the chosen particular value is allowed to vary. In addition, by inverting the argument and asking what the accuracies are that result for a prescribed number of Monte Carlo realizations, one can assess the computer time that would be involved should one choose to carry out the Monte Carlo realizations. These two complementary procedures are of great benefit in attempting to control the worth of undertaking an unknown number of Monte Carlo realizations, and of continuing to carry out the unknown number until the results have reached a level of accuracy that one deems acceptable. Such a procedure is not only computer intensive; however, is also very open-ended, a less than desirable trait when running a complex computer program that might take many hours or days to run through even once. The procedure presented here allows one to assess, ahead of performing a large number of Monte Carlo realizations, roughly how many are actually needed. Several illustrative numerical examples provide indications how one uses this novel procedure in practical situations.
Conference Paper
While the incidence of sleep disorders is continuously increasing in western societies, there is a clear demand for technologies to asses sleep-related parameters in ambulatory scenarios. The present study introduces a novel concept of accurate sensor to measure RR intervals via the analysis of photo-plethysmographic signals recorded at the wrist. In a cohort of 26 subjects undergoing full night polysomnography, the wrist device provided RR interval estimates in agreement with RR intervals as measured from standard electrocardiographic time series. The study showed an overall agreement between both approaches of 0.05 ± 18 ms. The novel wrist sensor opens the door towards a new generation of comfortable and easy-to-use sleep monitors.
Article
To record sleep, actigraph devices are worn on the wrist and record movements that can be used to estimate sleep parameters with specialized algorithms in computer software programs. With the recent establishment of a Current Procedural Terminology code for wrist actigraphy, this technology is being used increasingly in clinical settings as actigraphy has the advantage of providing objective information on sleep habits in the patient's natural sleep environment. Actigraphy has been well validated for the estimation of nighttime sleep parameters across age groups, but the validity of the estimation of sleep-onset latency and daytime sleeping is limited. Clinical guidelines and research suggest that wrist actigraphy is particularly useful in the documentation of sleep patterns prior to a multiple sleep latency test, in the evaluation of circadian rhythm sleep disorders, to evaluate treatment outcomes, and as an adjunct to home monitoring of sleep-disordered breathing. Actigraphy has also been well studied in the evaluation of sleep in the context of depression and dementia. Although actigraphy should not be viewed as a substitute for clinical interviews, sleep diaries, or overnight polysomnography when indicated, it can provide useful information about sleep in the natural sleep environment and/or when extended monitoring is clinically indicated.
Article
We have developed a real-time algorithm for detection of the QRS complexes of ECG signals. It reliably recognizes QRS complexes based upon digital analyses of slope, amplitude, and width. A special digital bandpass filter reduces false detections caused by the various types of interference present in ECG signals. This filtering permits use of low thresholds, thereby increasing detection sensitivity. The algorithm automatically adjusts thresholds and parameters periodically to adapt to such ECG changes as QRS morphology and heart rate. For the standard 24 h MIT/BIH arrhythmia database, this algorithm correctly detects 99.3 percent of the QRS complexes.
Design and development of real time respiratory rate monitor using non-invasive biosensor
  • B G Rao
  • Sudarshan
The AASM manual for the scoring of sleep and associated events: rules, terminology and technical specifications
  • R B Berry
LSTM knowledge transfer for HRV-based sleep
  • M Radha