No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Recognizing multi-user activities using wearable sensors in a smart home
Abstract
The advances of wearable sensors and wireless networks offer many opportunities to recognize human activities from sensor readings in pervasive computing. Existing work so far focuses mainly on recognizing activities of a single user in a home environment. However, there are typically multiple inhabitants in a real home and they often perform activities together. In this paper, we investigate the problem of recognizing multi-user activities using wearable sensors in a home setting. We develop a multi-modal, wearable sensor platform to collect sensor data for multiple users, and study two temporal probabilistic models—Coupled Hidden Markov Model (CHMM) and Factorial Conditional Random Field (FCRF)—to model interacting processes in a sensor-based, multi-user scenario. We conduct a real-world trace collection done by two subjects over two weeks, and evaluate these two models through our experimental studies. Our experimental results show that we achieve an accuracy of 96.41% with CHMM and an accuracy of 87.93% with FCRF, respectively, for recognizing multi-user activities.
... More sensors have been combined with wearables, such as audio, temperature, humidity, and light sensors in [40] to recognize multi-user activities. Ince et al. [41] have combined a wrist worn accelerometer and environmental sensors to recognize brushing teeth, washing face, and shaving activities. ...
... When wearable sensors are used, data is collected continuously at a pre-defined frequency depending on the sensors capabilities, the resolution needed, and the storage capacity or transfer speed of the device. Commonly used accelerometer frequencies range from 10 Hz [37] for simple activities such as standing, sitting, or lying down to 50 Hz for most smartphone based systems [42,43], to 90 Hz [33], 125 Hz [31], 128 Hz [40], or 150 Hz [36] for higher resolution systems. ...
... High-pass filters are commonly used to remove the unwanted DC component in accelerometer data, which corresponds to the contribution of gravity [33,36,37,41]. Low-pass filters have been used on accelerometer data in [40,43]. Nguyen et al. [54] have used a threshold based approach to eliminate noise from sound data. ...
This review presents the state of the art and a global overview of research challenges of real-time distributed activity recognition in the field of healthcare. Offline activity recognition is discussed as a starting point to establish the useful concepts of the field, such as sensor types, activity labeling and feature extraction, outlier detection, and machine learning. New challenges and obstacles brought on by real-time centralized activity recognition such as communication, real-time activity labeling, cloud and local approaches, and real-time machine learning in a streaming context are then discussed. Finally, real-time distributed activity recognition is covered through existing implementations in the scientific literature, and six main angles of optimization are defined: Processing, memory, communication, energy, time, and accuracy. This survey is addressed to any reader interested in the development of distributed artificial intelligence as well activity recognition, regardless of their level of expertise.
... Activity recognition has been categorized mainly into two approaches: Vision based [12][13][14] and pervasive sensing based [15][16][17]. Vision-based activity recognition can provide good results but have raised various privacy concerns among the residents due to required camera installations in their private spaces [18,19] whereas pervasive sensingbased activity recognition approaches use data from wearable sensors and non-intrusive environment sensors [20]. A significant amount of work has been performed on activity recognition using wearable sensors. ...
... Evidently, there is a need for a model that is capable of capturing the complex nature of both joint and independent activities. Previous works have addressed multiple resident activity recognition using wearable sensors such as RFID [17], accelerometer [27] and videos [28]. Machine learning approaches used previously for multi-resident activity recognition are naive Bayes, Markov Model classifier [29] and conditional random field (CRF) [30] on CASAS [31] dataset in which data association problem was investigated. ...
Recognizing multiple residents’ activities is a pivotal domain within active and assisted living technologies, where the diversity of actions in a multi-occupant home poses a challenge due to their uneven distribution. Frequent activities contrast with those occurring sporadically, necessitating adept handling of class imbalance to ensure the integrity of activity recognition systems based on raw sensor data. While deep learning has proven its merit in identifying activities for solitary residents within balanced datasets, its application to multi-resident scenarios requires careful consideration. This study provides a comprehensive survey on the issue of class imbalance and explores the efficacy of Long Short-Term Memory and Bidirectional Long Short-Term Memory networks in discerning activities of multiple residents, considering both individual and aggregate labeling of actions. Through rigorous experimentation with data-level and algorithmic strategies to address class imbalances, this research scrutinizes the explicability of deep learning models, enhancing their transparency and reliability. Performance metrics are drawn from a series of evaluations on three distinct, highly imbalanced smart home datasets, offering insights into the models’ behavior and contributing to the advancement of trustworthy multi-resident activity recognition systems.
... During the years, various smart environment systems have been proposed. They have been used in many scenarios, such as family houses [14], offices [15], shopping malls [16], and museums [17], and have been applied for a variety of purposes, including tracking people in buildings [18], counting people numbers [19], recognizing human behavior [20], etc. In addition, energy consumption can be monitored, and the indoor environment can be controlled automatically by using appropriate sensors and controllers [21]. ...
... Vera et al. [49] proposed a system to count people using depth cameras mounted in the zenithal position: people are detected in each camera and the tracklets that belong to the same person are determined. Even though vision-based methods are efficient and reliable [14,50,51], they are unsuitable for smart homes due to privacy reasons. Algorithms based on wearable devices are infeasible for detecting visitors who do not wear such devices. ...
Population aging requires innovative solutions to increase the quality of life and preserve autonomous and independent living at home. A need of particular significance is the identification of behavioral drifts. A relevant behavioral drift concerns sociality: older people tend to isolate themselves. There is therefore the need to find methodologies to identify if, when, and how long the person is in the company of other people (possibly, also considering the number). The challenge is to address this task in poorly sensorized apartments, with non-intrusive sensors that are typically wireless and can only provide local and simple information. The proposed method addresses technological issues, such as PIR (Passive InfraRed) blind times, topological issues, such as sensor interference due to the inability to separate detection areas, and algorithmic issues. The house is modeled as a graph to constrain transitions between adjacent rooms. Each room is associated with a set of values, for each identified person. These values decay over time and represent the probability that each person is still in the room. Because the used sensors cannot determine the number of people, the approach is based on a multi-branch inference that, over time, differentiates the movements in the apartment and estimates the number of people. The proposed algorithm has been validated with real data obtaining an accuracy of 86.8%.
... The advantages of a device that can understand the environment are evident [19,20]. Recent advances in pervasive computing and wearable devices frequently point at the location of the user as a valuable information source to design contextaware systems [14,13,21]. An intuitive way to find the location is to use Global Positioning Systems (GPS). ...
Wearable cameras stand out as one of the most promising devices for the upcoming years, and as a consequence, the demand of computer algorithms to automatically understand the videos recorded with them is increasing quickly. An automatic understanding of these videos is not an easy task, and its mobile nature implies important challenges to be faced, such as the changing light conditions and the unrestricted locations recorded. This paper proposes an unsupervised strategy based on global features and manifold learning to endow wearable cameras with contextual information regarding the light conditions and the location captured. Results show that non-linear manifold methods can capture contextual patterns from global features without compromising large computational resources. The proposed strategy is used, as an application case, as a switching mechanism to improve the hand-detection problem in egocentric videos.
... Other applications have also exploited the combination of different sensors. Activity recognition using the accelerometer and RFID in [42,71], accelerometer, compass, and gyroscope in [46], accelerometer and GPS in [18], accelerometer and wearable camera in [17], accelerometer and gyroscope in [19,20], accelerometer and webcam in [16], accelerometer and microphone in [72,73], accelerometer, gyroscope, and inertial sensor in [9], accelerometer, microphone, and RFID in [74], wireless sensors in [45,75], infrared sensors in [76], radio tomographic imaging in [77], accelerometer and other sensors in [22,24,35,43,45,75,78] are some other instances of these applications.. ...
Human activity recognition systems using wearable sensors is an important issue in pervasive computing, which applies to various domains related to healthcare, context aware and pervasive computing, sports, surveillance and monitoring, and the military. Three approaches can be considered for activity recognition: video sensor-based, physical sensor-based, and environmental sensor-based. This paper investigates the related work regarding the physical sensor-based approaches to motion processing. In this paper, a wide range of artificial intelligence models, from single classifications to methods based on deep learning, have been reviewed. The human activity detection accuracy of different methods, under natural and experimental conditions poses several challenges. These challenges cause problems regarding the accuracy and applicability of the proposed methods. This paper analyzes the methods, challenges, approaches, and future work. The goal of this paper is to establish a clear distinction in the field of motion detection using inertial sensors.
... Assembly and maintenance applications are the primary focus of the work cited in [35], which focuses on identifying actions denoted by a hand gesture and an associated sound. Refer to [36] for discussion on collaborative work. Inter-subject variation is the primary topic of [37], which examines a massive dataset. ...
This research analyzes the effectiveness of several methods for categorizing human actions captured by inertial and magnetic sensor units worn on the chest, arms, and legs. Each device has tri-axial sensors, including a gyroscope, accelerometer, and magnetometer. Voting ensemble classification models, where votes are weighted and optimized with a new optimization technique, are offered as a means to actualize this classification problem. The optimization technique is a combination of the sine cosine and particle swarm optimization algorithms, and the ensemble model is made up of three classifiers: support vector machines, decision trees, and multilayer perceptron. The classifiers are checked for accuracy using three distinct cross-validation strategies. Classifiers' proper differentiation rates and computational costs are compared to help you choose the best one for your needs. When it comes to body location, sensor devices worn on the legs provide the most valuable data. From a comparison of the various sensor modalities, we can deduce that magnetometers, followed by accelerometers and gyroscopes, provide the best classification results when only a single sensor type is employed. Furthermore, the study contrasts three machine learning models—support vector machines, decision trees, and multilayer perceptron —with respect to their usability, controllability, and classifier performance. Results reveal that the suggested method performs well in categorizing both typical daily activities and athletic endeavors.
... Lots of sensors such as microphones, contact sensors, infrared (IR) sensors, accelerometer and magnetometer are used to monitor the activities of older persons (Fleury et al., 2010). There are usually more than one member in a family, to identify each of the dwellers, Wang et al. presented a multiuser activity recognition system using wearable audio sensors, altimetry sensors and RFID tags, and high accuracy was obtained (Wang et al., 2011). To ensure they live in a safer environment, behavior monitoring, such as anomalous behavior detection, is urgently needed for elderly people, especially for those living alone (Alemdar and Ersoy, 2010;Rashidi and Mihailidis, 2013;Hoque et al., 2015;Chaccour et al., 2016;Stone and Skubic, 2017;Deep et al., 2020). ...
In recent years, a huge number of individuals all over the world, elderly people, in particular, have been suffering from Alzheimer’s disease (AD), which has had a significant negative impact on their quality of life. To intervene early in the progression of the disease, accurate, convenient, and low-cost detection technologies are gaining increased attention. As a result of their multiple merits in the detection and assessment of AD, biosensors are being frequently utilized in this field. Behavioral detection is a prospective way to diagnose AD at an early stage, which is a more objective and quantitative approach than conventional neuropsychological scales. Furthermore, it provides a safer and more comfortable environment than those invasive methods (such as blood and cerebrospinal fluid tests) and is more economical than neuroimaging tests. Behavior detection is gaining increasing attention in AD diagnosis. In this review, cutting-edge biosensor-based devices for AD diagnosis together with their measurement parameters and diagnostic effectiveness have been discussed in four application subtopics: body movement behavior detection, eye movement behavior detection, speech behavior detection, and multi-behavior detection. Finally, the characteristics of behavior detection sensors in various application scenarios are summarized and the prospects of their application in AD diagnostics are presented as well.
... Over the years, many researchers have tried to estimate daily life human activities, such as walking and sitting up and down from acceleration and/or gyro sensor data obtained from wearable devices and/or smartphones. In this paper, we refer to the research done respectively by Zhan et al. and Wang et al. [12] [13]. Only a few motions were given in this research; distinctions among activities were not recognized. ...
A system is needed for quantitatively evaluating the activity recovery level of functional disable people. Although functional recovery is administered to hemiplegic patients during rehabilitation, some patients who have recovered function in a rehabilitation facility are still unable to perform daily activities at home. Therefore, recovering activities of daily living (ADL) has become more important than functional recovery. Since existing ADL recovery level indices are based on responses to questionnaires, judgment of recovery level is easily affected by an evaluator's subject. We have developed a system for collecting and storing motion data on daily life activities for use in quantitatively evaluating ADL recovery levels. Evaluation of the system using data measured for a healthy participant with restricted movement and two actual hemiplegic patients demonstrated that slight differences in disability levels can be detected. This system is thus well suited for quantitative ADL assessment for patients with a disability.
... MICAR falls in this category. In [49], the residents wear a specifically designed RFID wristband reader in charge of detecting the interaction with tagged objects. However, the use of sophisticated wearable devices in realistic deployments is questionable, especially considering elderly subjects that should constantly wear them. ...
The sensor-based recognition of Activities of Daily Living (ADLs) in smart-home environments enables several important applications, including the continuous monitoring of fragile subjects in their homes for healthcare systems. The majority of the approaches in the literature assume that only one resident is living in the home. Multi-inhabitant ADLs recognition is significantly more challenging, and only a limited effort has been devoted to address this setting by the research community. One of the major open problems is called data association , which is correctly associating each environmental sensor event (e.g., the opening of a fridge door) with the inhabitant that actually triggered it. Moreover, existing multi-inhabitant approaches rely on supervised learning, assuming a high availability of labeled data. However, collecting a comprehensive training set of ADLs (especially in multiple-residents settings) is prohibitive. In this work, we propose MICAR: a novel multi-inhabitant ADLs recognition approach that combines semi-supervised learning and knowledge-based reasoning. Data association is performed by semantic reasoning, combining high-level context information (e.g., residents’ postures and semantic locations) with triggered sensor events. The personalized stream of sensor events is processed by an incremental classifier, that is initialized with a limited amount of labeled ADLs. A novel cache-based active learning strategy is adopted to continuously improve the classifier. Our results on a dataset where up to 4 subjects perform ADLs at the same time show that MICAR reliably recognizes individual and joint activities while triggering a significantly low number of active learning queries.
... The proposed method minimizes the amount of features required to detect and recognize movement [232]. In addition, a multiuser movement detection platform was proposed by Wang et al. [233]. The proposed platform used Coupled HMM and Factorial Conditional Random Field to model sensors' data transformation and communication. ...
Applications of machine learning (ML) methods have been used extensively to solve various complex challenges in recent years in various application areas, such as medical, financial, environmental, marketing, security, and industrial applications. ML methods are characterized by their ability to examine many data and discover exciting relationships, provide interpretation, and identify patterns. ML can help enhance the reliability, performance, predictability, and accuracy of diagnostic systems for many diseases. This survey provides a comprehensive review of the use of ML in the medical field highlighting standard technologies and how they affect medical diagnosis. Five major medical applications are deeply discussed, focusing on adapting the ML models to solve the problems in cancer, medical chemistry, brain, medical imaging, and wearable sensors. Finally, this survey provides valuable references and guidance for researchers, practitioners, and decision-makers framing future research and development directions.
... Indeed, SVM training requires solving a quadratic programming (QP) problem in a number of coefficients equal to the number of training examples, which, in turn, makes standard numerical techniques for QP infeasible for a such large dataset. To overcome this difficulty, some practical techniques decompose the problem into manageable sub-problems over part of the data Lester et al. (2005) or, perform component-wise optimization (Kim et al. 2013), or, to some extent, iterative pairwise comparison (Wang et al. 2011). Other researchers suggested to transform the batch SVMs to the incremental ones by adapting an incremental or online learning techniques (Cauwenberghs et al. 2001;Syed et al. 1999). ...
Smart homes are equipped with several sensor networks to keep an eye on both residents and their environment, to interpret the current situation and to react immediately. Handling large scale dataset of sensory events on real time to enable efficient interventions is challenging and very difficult. To deal with these data flows and challenges, traditional streaming data classification approaches can be boosted by use of incremental learning. In this paper, we presented two new Incremental SVM methods to improve the performance of SVM classification in the context of human activity recognition tasks. Two feature extraction methods elaborated by refining dependency sensor extraction feature and focusing on the last sensor event only have been suggested. On the other hand, a clustering based approach and a similarity based approach have been suggested to boost learning performance of the incremental SVM algorithms capitalizing on the relationship between data chunk and support vectors of previous chunk. We demonstrate through several simulations on two major publicly available data sets (Aruba and Tulum), the feasibility and improvements in learning and classification performances in real time achieved by our proposed methods over the state-of-the-art. For instance, we have shown that the introduced similarity-based incremental learning is 5 to 9 times faster than other methods in terms of training performances. Similarly, the introduced Last-state sensor feature method induces at least 5% improvement in terms of F1-score when using baseline SVM classifier.
... These methods may not be applicable for detecting multi-person activity in real-world scenarios. Previous research has shown that the difficulty of identifying human-human interactions (HHIs), which involve multiple interacting people (e.g., high five and pushing interactions), is more challenging than identifying single-human activities (e.g., running and sitting activities) [16]. A three-layer CNN [17] is proposed, which employs publicly available CSI data [18], converting it into a 2D grayscale image to recognize the HHIs. ...
In recent years, Wi-Fi infrastructures have become ubiquitous, providing device-free passive-sensing features. Wi-Fi signals can be affected by their reflection, refraction, and absorption by moving objects in their path. The channel state information (CSI), a signal property indicator, of the Wi-Fi signal can be analyzed for human activity recognition (HAR). Deep learning-based HAR models can enhance performance and accuracy without sacrificing computational efficiency. However, to save computational power, an inception network, which uses a variety of techniques to boost speed and accuracy, can be adopted. In contrast, the concept of spatial attention can be applied to obtain refined features. In this paper, we propose a human–human interaction (HHI) classifier, CSI-IANet, which uses a modified inception CNN with a spatial-attention mechanism. The CSI-IANet consists of three steps: i) data processing, ii) feature extraction, and iii) recognition. The data processing layer first uses the second-order Butterworth low-pass filter to denoise the CSI signal and then segment it before feeding it to the model. The feature extraction layer uses a multilayer modified inception CNN with an attention mechanism that uses spatial attention in an intense structure to extract features from captured CSI signals. Finally, the refined features are exploited by the recognition section to determine HHIs correctly. To validate the performance of the proposed CSI-IANet, a publicly available HHI CSI dataset with a total of 4800 trials of 12 interactions was used. The performance of the proposed model was compared to those of existing state-of-the-art methods. The experimental results show that CSI-IANet achieved an average accuracy of 91.30%, which is better than that of the existing best method by 5%.
... (307) dos(as) alunos(as) avaliaram as aulas como "muito bom"; 21% (83), como "bom"; 2% (10), como "regular"; e apenas 1% (4), como "ruim" ou "muito ruim". (BELLE et al., 2015;BISSONNETTE;BERGERON, 2017;GATZOULIS;IAKOVIDIS, 2007;GROSSGLAUSER;SANER, 2014;HANDEL, 2011;MARTIN, 2012;NICHOLAS et al., 2015;SOH et al., 2013;SRIVASTAVA et al., 2015;WEST et al., 2012). ...
Massive Open Online Courses (MOOCs) são cursos abertos e massivos ofertados em ambientes virtuais de aprendizagem, que permitem o compartilhamento de experiências, conhecimentos e informações. Este capítulo objetiva realizar um mapeamento da produção de MOOCs na saúde no Brasil, nas dissertações e teses e das iniciativas governamentais na área da saúde. Trata-se de um estudo qualitativo, com base na análise documental dos sites dos Ministérios da Saúde e da Educação brasileiros, do MOOC-List.com e da Biblioteca Digital Brasileira de Teses e Dissertações (BDTD). Embora não use a nomenclatura MOOC, o Brasil investe em iniciativas pedagógicas massivas, on-line e abertas. Os resultados indicam que a produção acadêmica sobre MOOCs na saúde é incipiente. Há um potencial a ser explorado, que inclui a educação permanente e continuada dos profissionais de saúde e a troca de experiências e conhecimentos entre pesquisadores, profissionais e usuários do sistema de saúde brasileiro.
... To meet these requirements, we are seeing a strong transition from traditional rigid printed circuit boards toward flexible devices and interconnects. Providing an easy substitute for wire harnesses and wired assemblies, flexible circuits have already been incorporated into keyboards, telephones, and laptops 1 and are showing great promise in a host of other applications, such as smart home systems [2][3][4][5] and medical devices. [6][7][8] Many of these devices contribute to the Internet of Things and can participate in networked functionality through flexible antennas and electrodes. ...
This work investigates the origins of electrical performance degradation under uniaxial stretching of a silver filled polyurethane ink (DuPont PE 874) screen printed onto a thermoplastic polyurethane substrate. The ink develops surface ruptures at strains of only a few percent yet remains conductive through continued elongation. We identify increasing sensitivity to surface damage beyond 10% applied strain, ɛapp, as the trace width, w, is reduced from 2 to 0.1 mm. This lowers the threshold strain for open circuit failure, from approximately 180% for w = 2 mm down to 25% for w = 0.1 mm. The damage progression remains largely consistent across trace widths: surface cracks coalesce to form longer channels, which grow perpendicular to the direction of elongation. These channels both deepen and widen with increasing ɛapp and some become laterally linked. The evolution of the network of interlinked channels is not width dependent, but a width effect manifests as a result of the channels constituting a larger fraction of specimen width for narrower traces. In addition, the narrower traces exhibit reduced cross sections due to an edge taper—an artifact of the screen printing process—which attenuates ink thickness by as much as 50% for w = 0.1 mm.
... A wide variety of sensors have been utilized to capture relevant contextual factors in smart homes, including (but not limited to) those that measure ambient temperature, record sound, motion, wifi signal characteristics, visual information (through, for example, cameras), and proximity information [2,7,61]. Furthermore, body worn sensors are also widely used for direct recording of an occupant's movements [60]. In the context of the smart home domain, activity recognition specifically serves to identify and log the occupant's activities of daily living (ADL) from temporal-ordered sensor data. ...
Smart home environments are designed to provide services that help improve the quality of life for the occupant via a variety of sensors and actuators installed throughout the space. Many automated actions taken by a smart home are governed by the output of an underlying activity recognition system. However, activity recognition systems may not be perfectly accurate and therefore inconsistencies in smart home operations can lead a user to wonder "why did the smart home do that?" In this work, we build on insights from Explainable Artificial Intelligence (XAI) techniques to contribute computational methods for explainable activity recognition. Specifically, we generate explanations for smart home activity recognition systems that explain what about an activity led to the given classification. To do so, we introduce four computational techniques for generating natural language explanations of smart home data and compare their effectiveness at generating meaningful explanations. Through a study with everyday users, we evaluate user preferences towards the four explanation types. Our results show that the leading approach, SHAP, has a 92% success rate in generating accurate explanations. Moreover, 84% of sampled scenarios users preferred natural language explanations over a simple activity label, underscoring the need for explainable activity recognition systems. Finally, we show that explanations generated by some XAI methods can lead users to lose confidence in the accuracy of the underlying activity recognition model, while others lead users to gain confidence. Taking all studied factors into consideration, we make a recommendation regarding which existing XAI method leads to the best performance in the domain of smart home automation, and discuss a range of topics for future work in this area.
... The results obtained from the study show that the proposed method can achieve high accuracy, but that forgetting to wear the tag is the main issue with this method that greatly reduces the accuracy. Likewise, Wang et al. (2011), investigated the challenge of recognising multi-occupancy activities utilising wearable sensors in a home. The idea of this research was to study two probabilistic temporal models; the Coupled Hidden Markov Model (CHMM) and factorial conditional random Field (FCRF), to model and classify multi-occupancy activities. ...
Human activity recognition (HAR) is used to support older adults to live independently in their own homes. Once activities of daily living (ADL) are recognised, gathered information will be used to identify abnormalities in comparison with the routine activities. Ambient sensors, including occupancy sensors and door entry sensors, are often used to monitor and identify different activities. Most of the current research in HAR focuses on a single-occupant environment when only one person is monitored, and their activities are categorised. The assumption that home environments are occupied by one person all the time is often not true. It is common for a resident to receive visits from family members or health care workers, representing a multi-occupancy environment. Entropy analysis is an established method for irregularity detection in many applications; however, it has been rarely applied in the context of ADL and HAR. In this paper, a novel method based on different entropy measures, including Shannon Entropy, Permutation Entropy, and Multiscale-Permutation Entropy, is employed to investigate the effectiveness of these entropy measures in identifying visitors in a home environment. This research aims to investigate whether entropy measures can be utilised to identify a visitor in a home environment, solely based on the information collected from motion detectors [e.g., passive infra-red] and door entry sensors. The entropy measures are tested and evaluated based on a dataset gathered from a real home environment. Experimental results are presented to show the effectiveness of entropy measures to identify visitors and the time of their visits without the need for employing extra wearable sensors to tag the visitors. The results obtained from the experiments show that the proposed entropy measures could be used to detect and identify a visitor in a home environment with a high degree of accuracy.
... For example, there are many systems in which radio waves are used. These include activity and gesture recognition systems using Wi-Fi [26][27][28] or radio frequency identification (RFID) systems [29][30][31][32] and a healthcare reporting generation support system that is designed for nursing homes that uses Bluetooth low energy (BLE) components [33,34]. However, because these systems are directly affected by radio wave reflections, they need to be reconfigured every time the environment or device location changes. ...
As aging populations continue to grow, primarily in developed countries, there are increasing demands for the system that monitors the activities of elderly people while continuing to allow them to pursue their individual, healthy, and independent lifestyles. Therefore, it is required to develop the activity of daily living (ADL) sensing systems that are based on high-performance sensors and information technologies. However, most of the systems that have been proposed to date have only been investigated and/or evaluated in experimental environments. When considering the spread of such systems to typical homes inhabited by elderly people, it is clear that such sensing systems will need to meet the following five requirements: (1) be inexpensive; (2) provide robustness; (3) protect privacy; (4) be maintenance-free; and, (5) work with a simple user interface. In this paper, we propose a novel senior-friendly ADL sensing system that can fulfill these requirements. More specifically, we achieve an easy collection of ADL data from elderly people while using a proposed system that consists of a small number of inexpensive energy harvesting sensors and simple annotation buttons, without the need for privacy-invasive cameras or microphones. In order to evaluate the practicality of our proposed system, we installed it in ten typical homes with elderly residents and collected the ADL data over a two-month period. We then visualized the collected data and performed activity recognition using a long short-term memory (LSTM) model. From the collected results, we confirmed that our proposed system, which is inexpensive and non-invasive, can correctly collect resident ADL data and could recognize activities from the collected data with a high recall rate of 72.3% on average. This result shows a high potential of our proposed system for application to services for elderly people.
In this chapter, data-science techniques are used to assess players’ ability. Both artificial intelligence and machine learning based viewpoints and mentoring alter team effectiveness. One might assume the opposing team’s strategy and build one’s own team to obtain the best outcomes. Data science provides another possibility for data engineers to gain more insight into sport. As a result, this chapter will provide information on the role of big data analytics in the fitness and health sector.
Conjugated double bonds generated during chemical or electrochemical oxidation of the monomer are the building blocks of conducting polymers [1]. Dopant or counterions’ insertion in the polymer backbone occurs as a result of oxidation at both the monomer and polymer levels; for example, chloride is inserted when iron chloride is used as an oxidant.
Smart home environments are designed to provide services that help improve the quality of life for the occupant via a variety of sensors and actuators installed throughout the space. Many automated actions taken by a smart home are governed by the output of an underlying activity recognition system. However, activity recognition systems may not be perfectly accurate and therefore inconsistencies in smart home operations can lead users reliant on smart home predictions to wonder “why did the smart home do that?” In this work, we build on insights from Explainable Artificial Intelligence (XAI) techniques and introduce an explainable activity recognition framework in which we leverage leading XAI methods (LIME, SHAP, Anchors) to generate natural language explanations that explain what about an activity led to the given classification. We evaluate our framework in the context of a commonly targeted smart home scenario: autonomous remote caregiver monitoring for individuals who are living alone or need assistance. Within the context of remote caregiver monitoring, we perform a two-step evaluation: (a) utilize ML experts to assess the sensibility of explanations, and (b) recruit non-experts in two user remote caregiver monitoring scenarios, synchronous and asynchronous, to assess the effectiveness of explanations generated via our framework. Our results show that the XAI approach, SHAP, has a 92% success rate in generating sensible explanations. Moreover, in 83% of sampled scenarios users preferred natural language explanations over a simple activity label, underscoring the need for explainable activity recognition systems. Finally, we show that explanations generated by some XAI methods can lead users to lose confidence in the accuracy of the underlying activity recognition model, while others lead users to gain confidence. Taking all studied factors into consideration, we make a recommendation regarding which existing XAI method leads to the best performance in the domain of smart home automation, and discuss a range of topics for future work to further improve explainable activity recognition.
In this paper, a Two-domain Joint Attention Mechanism based on Sensor Data (TJAMSD) for Group Activity Recognition (GAR) is proposed. We build two networks in the semantic domain and data domain as the teacher network and student network, respectively. In the data domain, a GAR network based on Graph Convolutional Network (GCN) with Group Relation Graph (GRG) is proposed. In this network, in order to reflect the relationship between individuals, the individual action feature correlation and position correlation in a group are calculated respectively to construct two relation graphs. Then, the two relation graphs are fused to obtain the final GRG. Finally, the GRG and the individual action features obtained by a hybrid CNN and BLSTM network are used as the input of the GCN to infer the group activity. Besides, a semantic-domain network is constructed by the known individual action semantics and the group activity semantics. A joint attention mechanism based on the data-domain network and semantic-domain network is proposed. The attention weights learned in the semantic-domain network are used to guide the learning of attention weights in the data-domain network, which allocates attention to different individuals. In this way, TJAMSD makes the networks pay more attention to the key individual actions in the group and overcome the interference caused by non-critical individual actions in GAR. Experiments are conducted on two constructed datasets, the Garsensors dataset and the UT-Data-gar dataset. Different group cases are considered in the experiments and the experimental results show that in all cases, GCN with GRG can better express the interaction features of groups and improve the recognition performance. Furthermore, the TJAMSD can effectively suppress the interference of non-critical actions to advance the model robustness.
User authentication nowadays has become an important support for not only security guarantees but also emerging novel applications. Although WiFi signal-based user authentication has achieved initial success, it works in single-user scenarios while multi-user authentication remains a challenging task. In this paper, we present
MultiAuth
, a multi-user authentication system that can authenticate multiple users with a single pair of commodity WiFi devices. The basic idea is to profile multipath components of WiFi signals, and leverage the multipath components to characterize each user individually for multi-user authentication.
MultiAuth
first profiles multipath components of WiFi signals through a proposed MUltipath Time-of-Arrival estimation algorithm (MUTA). Then, after matching corresponding multipath components to each user in complex multi-user scenarios,
MultiAuth
constructs individual CSI based on the multipath components to characterize each user individually. An AoA-based approach is exploited to further separate individual CSI constructed by the users with same ToA. To identify users through their activities,
MultiAuth
extracts user behavior profiles based on the individual CSI, and leverages a dual-task neural network for robust user authentication. Extensive experiments involving 3 simultaneously present users demonstrate that
MultiAuth
is effective in multi-user authentication with 86.2% average accuracy and 9.5% average false accept rate.
Today, all countries globally, including Iran, face the challenge of
spreading theCOVID-19 disease. Given the limited capacity of the health care system
and the risk of a recurrence of demand for testing and intensive care units (ICU),
some countries may again experience fluctuating behaviors or the return of viral
pandemics. Various reasons are resulting in this emerging virus in the world. Apart
from the tendency to cause severe disease with a higher death rate than previous
coronavirus diseases, it can spread so quickly. This study aimed to reach an optimal
capacity allocation in the health care centers by intervening in government decisions
and public holidays in Iran on disease control. This research is the first to analyze and
develop a healthcare capacity allocation strategy by considering the mutual effects
of disease outbreaks and government actions as decision aiding tools via a hybrid
simulation–optimization framework. Also, we calibrate the developed hybrid simulation–
optimization model for Iran on the improved SEIR framework to generate
reliable outputs. The simulation results show that it is necessary to allocate some
part of the health capacity to the mild symptomatic patients.
In recent years, technology-based health programs, including context-aware and user-adaptive smart home ecosystems, have shown tremendous potential to limit disparities in healthcare access and improve uptake in health programs, thereby improving health outcomes. In this chapter, we review the potential use and applications of context-aware and user-adaptive smart home ecosystems to support the diagnosis, monitoring and management of health conditions. Further, we also discuss the barriers and limitations of using wearable and smart home technologies, their facilitators, current gaps and future opportunities. The chapter provides evidence-based applications and a comprehensive understanding of the use of wearables and smart home ecosystems during and after COVID-19 pandemic for health care providers, researchers, students, and technology developers.
During the past years, there has been rapid development in Ambient Assisted Living technologies and Smart Home solutions for the growing aging population. A smart home system consists of many different components that interact with each other and can provide services that are context-aware and personalized. In order to develop a flexible, scalable, and acceptable smart home solution, certain improvements are required in different areas such as meeting end-user requirements, activity recognition, privacy aspect, and interaction with the system. In this thesis, we explored different ways to further improve the performances of the major components of a smart home system that include activity recognition, privacy-preserving, and dialogue systems. The research in this thesis has been conducted in five main areas. First, we proposed a privacy-enabled smart home framework in an open-source software platform. Next, we perform different user studies and interviews with the older people and end-users to understand their needs better, their perceptions and attitudes towards smart home applications such as activity monitoring, data sharing, privacy notions, and views regarding voice assistants. Third, we investigated deep learning algorithms on smart home sensor datasets for both single and multiple resident activity recognition. Using deep learning models, we achieved promising results on raw sensor datasets, in comparison to traditional machine learning methods. Furthermore, we investigated various class imbalance techniques on smart home datasets to address the class imbalance problem. Fourth, we proposed and implemented a privacy-preserving mechanism for data transformation and anonymization that allows sharing of encoded data instead of original data with third parties and thus, protects users’ sensitive information. In the last part of this thesis, we provided an overview of the dialogue system and presented a new human-annotated dialogue dataset with benchmark metrics using deep learning models to evaluate the quality of dialogue replies for the given context. Overall, the thesis addresses the challenges in existing research and components of smart home systems and the results obtained will potentially help developers and service providers in designing practical and adaptable smart home solutions.
The rapid development of flexible and stretchable materials has portrayed a bright future of flexible electronics in pursuing the concept of “Internet of Things.” Transistors, as one of the fundamental units in electronic devices, are being widely studied in exploring the possibility of increasing their flexibility and/or stretchability to fulfill the requirement in flexible electronics, from basic logic circuits to intelligent devices. However, the mechanical stimuli induced by bending and stretching activities can lead to unexpected performance degradations. Here, an in‐depth analysis of approaches to modulating the sensitivity of flexible and stretchable transistors, from a mechanically sensitive type to an insensitive type is demonstrated. Starting with the structural design, the mechanism of sensitivity modulation is explicitly analyzed, such as effects based on the modulation of dielectric capacitance, conductance of active channels, or contact resistance at interfaces in sensitive transistors and strain patterning technology applied to an insensitive type. The sensitivity modulation for transistors is further analyzed regarding material utilization based on different dimensional forms. Finally, the potential applications where flexible and stretchable transistors can be employed as an outlook are discussed. Deformation‐induced strain has become an inevitable problem for flexible and stretchable transistors and hinders the stable performance of flexible electronics in applications. A series of methods to optimize transistors’ performance under different strains have been proposed. This review on the state‐of‐the‐art approaches is presented to manipulate the strains inside transistors, from delicate structural designs to rational material utilizations.
Activity recognition is a fundamental way to support context-aware services for users in smart spaces. Data sources such as video or wearable devices are used in many recognition approaches, but there are challenges in utilizing them in the real world. Recent approaches propose deep learning-based methods on IoT sensor data streams to overcome the issues. Since they only describe single user-based spaces, they are vulnerable to complex sequences of events triggered by multiple users. When multiple users exist in a space, various overlapping events occur with longer correlations than a single user situation. Additionally, ambient sensor-based events appear far more than actuator-based events, making it difficult to extract actuator-based events as important features. We propose a transformer-based approach to derive long-term event correlations and important events as elements of activity patterns. We also develop a duration incorporated embedding method to differentiate between the same type but different duration events and add a sequential manner to the transformer approach. In the experiments section, we prove that our approach outperforms the existing approaches based on real datasets.
The smart home is one application of intelligent environments, where sensors are equipped to detect the status inside the domestic home. With the development of sensing technologies, more signals can be obtained with heterogenous statistical properties with faster processing speed. To make good use of the technical advantages, data-driven methods are becoming popular in intelligent environments. On the other hand, to recognize human activity is one essential target to understand the status inside a smart home. In this chapter, the authors focus on the human activity recognition (HAR) problem, which is the recognition of lower levels of activities, using data-driven models.
The multiuser behavior recognition task based on environmental sensors can provide reliable health monitoring, suspicious person identification and behavior correction. Compared with camera equipment and wearable sensors, the task can achieve acquisition of binary data from the environmental sensors without requiring wearable sensors. Therefore, privacy protection of users and use burden can be improved. However, there are still challenges in this behavior recognition scenario: First, the data consistency shown by the different behaviors of a single user in the same scenario need to be guaranteed. Second, the interactive behavior of multiusers may cause a data association problem. Therefore, the multiuser behavior recognition task based on environmental sensors has, apart from application value, important research challenges. In response, we propose the divide and conquer dynamic memory network model (DC-DMN). Based on the periodicity of user behavior, personal habits, time and spatial characteristics, the multiuser behavior recognition ability of the model can be enhanced. First, the GRU model is used to solve the consistency problem of different behaviors at the data level. Then, we expand the model memory based on the idea of a dynamic memory network. In addition, two sections of memory are designed to integrate and store data more effectively. In this way, the data association and support problem can be solved. Finally, we use three standard datasets to conduct experiments and compare them with the existing benchmark methods in two dimensions of accuracy and recall. Experiments show that DC-DMN performs well in three different datasets. It can effectively solve the problems of data consistency and data association, thereby improving the recognition accuracy.
Smart toilet provides a feasible platform for the long-term analysis of person’s health. Common solutions for identification are based on camera or radio-frequency identification (RFID) technologies, but it is doubted for privacy issues. Here, we demonstrate an artificial intelligence of toilet (AI-toilet) based on a triboelectric pressure sensor array offering a more private approach with low cost and easily deployable software. The pressure sensor array attached on the toilet seat is composed of 10 textile-based triboelectric sensors, which can leverage the different pressure distribution of individual users' seating manner to get the biometric information. 6 users can be correctly identified with more than 90% accuracy using deep learning. The signals from pressure sensors also can be used for recording the seating time on the toilet. The system integrates a camera sensor to analyze the simulated urine by comparing with urine chart and classify the types and quantities of objects using deep learning. All information including two-factor user identification and entire seating time using pressure sensor array, and data from the urinalysis and stool analysis were automatically transferred to a cloud system and were further shown in user's mobile devices for better tracking their health status.
Human activity plays a significant role in various fields, such as manufacturing, healthcare, and public safety; therefore, recognizing human activity is crucial to enable smart innovative services. The development of ubiquitous sensing and pervasive computing allows studying what humans perform in real time and mobility. Single- and multi-user activity recognition (AR) differ by the number of involved users. With recent developments of multi-sensor and multi-information fusion, multi-user activity recognition is gradually becoming an emerging and relevant research frontier. In this paper, we propose a software architecture which combines cloud and edge computing with collaborative body sensor networks (CBSNs) to support the development of CBSNs-enabled services and in particular we provide its case-study in the context of multi-user AR.
Emerging flexible artificial sensory systems using neuromorphic electronics have been considered as a promising solution for processing massive data with low power consumption. The construction of artificial sensory systems with synaptic devices and sensing elements to mimic complicated sensing and processing in biological systems is a prerequisite for the realization. To realize high-efficiency neuromorphic sensory systems, the development of artificial flexible synapses with low power consumption and high-density integration is essential. Furthermore, the realization of efficient coupling between the sensing element and the synaptic device is crucial. This Review presents recent progress in the area of neuromorphic electronics for flexible artificial sensory systems. We focus on both the recent advances of artificial synapses, including device structures, mechanisms, and functions, and the design of intelligent, flexible perception systems based on synaptic devices. Additionally, key challenges and opportunities related to flexible artificial perception systems are examined, and potential solutions and suggestions are provided.
Wearable systems have unlocked new sensing paradigms in various applications such as human activity recognition, which can enhance effectiveness of mobile health applications. Current systems using wearables are not capable of understanding their surroundings, which limits their sensing capabilities. For instance, distinguishing certain activities such as attending a meeting or class, which have similar motion patterns but happen in different contexts, is challenging by merely using wearable motion sensors. This article focuses on understanding user's surroundings, i.e., environmental context, to enhance capability of wearables, with focus on detecting complex activities of daily living (ADL). We develop a methodology to automatically detect the context using passively observable information broadcasted by devices in users’ locale. This system does not require specific infrastructure or additional hardware. We develop a pattern extraction algorithm and probabilistic mapping between the context and activities to reduce the set of probable outcomes. The proposed system contains a general ADL classifier working with motion sensors, learns personalized context, and uses that to reduce the search space of activities to those that occur within a certain context. We collected real-world data of complex ADLs and by narrowing the search space with context, we improve average F1-score from 0.72 to 0.80.
We address the problem of recognizing sequences of human interaction patterns in meetings, with the goal of structuring them in semantic terms. The investigated patterns are inherently group-based (defined by the individual activities of meeting participants, and their interplay), and multimodal (as captured by cameras and microphones). By defining a proper set of individual actions, group actions can be modeled as a two-layer process, one that models basic individual activities from low-level audio-visual features, and another one that models the interactions. We propose a two-layer Hidden Markov Model (HMM) framework that implements such concept in a principled manner, and that has advantages over previous works. First, by decomposing the problem hierarchically, learning is performed on low-dimensional observation spaces, which results in simpler models. Second, our framework is easier to interpret, as both individual and group actions have a clear meaning, and thus easier to improve. Third, different HMM models can be used in each layer, to better reflect the nature of each subproblem. Our framework is general and extensible, and we illustrate it with a set of eight group actions, using a public five-hour meeting corpus. Experiments and comparison with a single-layer HMM baseline system show its validity.
Recognizing patterns of human activities is an important enabling technology for building intelligent home environ- ments. Existing approaches to activity recognition often fo- cus on mutually exclusive activities only. In reality, peo- ple routinely carry out multiple concurrent activities. It is therefore necessary to model the co-temporal relationships among activities. In this paper, we propose using Factorial Conditional Random Fields (FCRFs) for recognition of mul- tiple concurrent activities. We designed experiments to com- pare our FCRFs model with Linear Chain Condition Random Fields (LCRFs) in learning and performing inference with the MIT House n data set, which contains annotated data col- lected from multiple sensors in a real living environment. The experimental results show that FCRFs can effectively im- prove the F-score in activity recognition for up to 8% in the presence of multiple concurrent activities.
Understanding and recognizing human activities from sensor readings is an important task in pervasive computing. Existing work on activity recognition mainly focuses on recognizing activities for a single user in a smart home environment. However, in real life, there are often multiple inhabitants live in such an environment. Recognizing activities of not only a single user, but also multiple users is essential to the development of practical context-aware applications in pervasive computing. In this paper, we investigate the fundamental problem of recognizing activities for multiple users from sensor readings in a home environment, and propose a novel pattern mining approach to recognize both single-user and multi-user activities in a unified solution. We exploit emerging pattern -a type of knowledge pattern that describes significant changes between classes of data - for constructing our activity models, and propose an emerging pattern based multi-user activity recognizer (epMAR) to recognize both single-user and multiuser activities. We conduct our empirical studies by collecting real-world activity traces done by two volunteers over a period of two weeks in a smart home environment, and analyze the performance in detail with respect to various activity cases in a multi-user scenario. Our experimental results demonstrate that our epMAR recognizer achieves an average accuracy of 89.72% for all the activity cases.
In this paper we consider a class of human activities—atomic activities—which can be represented as a set of measurements over a finite temporal window (e.g., the motion of human body parts during a walking cycle) and which has a relatively small space of variations in performance. A new approach for modeling and recognition of atomic activities that employs principal component analysis and analytical global transformations is proposed. The modeling of sets of exemplar instances of activities that are similar in duration and involve similar body part motions is achieved by parameterizing their representation using principal component analysis. The recognition of variants of modeled activities is achieved by searching the space of admissible parameterized transformations that these activities can undergo. This formulation iteratively refines the recognition of the class to which the observed activity belongs and the transformation parameters that relate it to the model in its class. We provide several experiments on recognition of articulated and deformable human motions from image motion parameters.
Spatial scaffolding is a naturally occurring human teaching behavior, in which teachers use their bodies to spatially structure the learning environment to direct the attention of the learner. Robotic systems can take advantage of simple, highly reliable ...
A sensor system capable of automatically recognizing activities would allow many potential ubiquitous applications. In this paper, we present an easy to install sensor network and an accurate but inexpensive annotation method. A recorded dataset consisting of 28 days of sensor data and its annotation is described and made available to the community. Through a number of experiments we show how the hidden Markov model and conditional random fields perform in recognizing activities. We achieve a timeslice accuracy of 95.6% and a class accuracy of 79.4%.
We study activity recognition using 104 hours of annotated data collected from a person living in an instrumented home. The home contained over 900 sensor inputs, including wired reed switches, current and water flow inputs, object and person motion detectors, and RFID tags. Our aim was to compare dierent sensor modalities on data that approached "real world" conditions, where the subject and annotator were unaliated with the authors. We found that 10 infra-red motion detectors outperformed the other sensors on many of the activities stud- ied, especially those that were typically performed in the same location. However, several activities, in particular "eating" and "reading" were dicult to detect, and we lacked data to study many fine-grained activi- ties. We characterize a number of issues important for designing activity detection systems that may not have been as evident in prior work when data was collected under more controlled conditions.
We describe a real-time computer vision and machine learning system for modeling and recognizing human behaviors in a visual surveillance task. The system is particularly concerned with detecting when interactions between people occur, and classifying the type of interaction. Examples of interesting interaction behaviors include following another person, altering one's path to meet another, and so forth. Our system combines top-down with bottom-up information in a closed feedback loop, with both components employing a statistical Bayesian approach. We propose and compare two different state-based learning architectures, namely HMMs and CHMMs, for modeling behaviors and interactions. The CHMM model is shown to work much more efficiently and accurately.
Finally, to deal with the problem of limited training data, a synthetic ‘Alife-style’ training system is used to develop flexible prior models for recognizing human interactions. We demonstrate the ability to use these a priori models to accurately classify real human behaviors and interactions with no additional tuning or training.
With the convergence of technologies in artificial intelligence, human- computer interfaces, and pervasive computing, the idea of a "smart environment" is becoming a reality. While we all would like the benefits of an environment that automates many of our daily tasks, a smart environment that makes the wrong decisions can quickly becoming annoying. In this paper, we describe a simulation tool that can be used to visualize activity data in a smart home, play through proposed automation schemes, and ultimately provide guidance to automating the smart environment. We describe how automation policies can adapt to resident feedback, and demonstrate the ideas in the context of the MavHome smart home.
Accurate recognition and tracking of human activities is an important goal of ubiquitous computing. Recent advances in the development of multi-modal wearable sensors enable us to gather rich datasets of human activities. However, the problem of automatically identifying the most useful features for modeling such activities remains largely unsolved. In this paper we present a hybrid approach to recognizing activities, which combines boosting to discriminatively select useful features and learn an ensemble of static classifiers to recognize different activities, with hidden Markov models (HMMs) to capture the temporal regularities and smoothness of activities. We tested the activity recognition system using over 12 hours of wearable-sensor data collected by volunteers in natural unconstrained environments. The models succeeded in identifying a small set of maximally informative features, and were able identify ten different human activities with an accuracy of 95%.
The pervasive sensing technologies found in smart homes offer unprecedented opportunities for providing health monitoring and assistance to individuals experiencing difficulties living independently at home. A primary challenge that needs to be tackled to meet this need is the ability to recognize and track functional activities that people perform in their own homes and everyday settings. In this paper, we look at approaches to perform real-time recognition of Activities of Daily Living. We enhance other related research efforts to develop approaches that are effective when activities are interrupted and interleaved. To evaluate the accuracy of our recognition algorithms we assess them using real data collected from participants performing activities in our on-campus smart apartment testbed.
In order to provide relevant information to mobile users, such as workers engaging in the manual tasks of maintenance and assembly, a wearable computer requires information about the user's specific activities. This work focuses on the recognition of activities that are characterized by a hand motion and an accompanying sound. Suitable activities can be found in assembly and maintenance work. Here, we provide an initial exploration into the problem domain of continuous activity recognition using on-body sensing. We use a mock "wood workshop" assembly task to ground our investigation. We describe a method for the continuous recognition of activities (sawing, hammering, filing, drilling, grinding, sanding, opening a drawer, tightening a vise, and turning a screwdriver) using microphones and three-axis accelerometers mounted at two positions on the user's arms. Potentially "interesting" activities are segmented from continuous streams of data using an analysis of the sound intensity detected at the two different locations. Activity classification is then performed on these detected segments using linear discriminant analysis (LDA) on the sound channel and hidden Markov models (HMMs) on the acceleration data. Four different methods at classifier fusion are compared for improving these classifications. Using user-dependent training, we obtain continuous average recall and precision rates (for positive activities) of 78 percent and 74 percent, respectively. Using user-independent training (leave-one-out across five users), we obtain recall rates of 66 percent and precision rates of 63 percent. In isolation, these activities were recognized with accuracies of 98 percent, 87 percent, and 95 percent for the user-dependent, user-independent, and user-adapted cases, respectively.
An important issue to be addressed in a smart home environment is how to provide appropriate services according to the preference of inhabitants. In this paper, we aim at developing a system to learn a multiple users' preference model that represents relationships among users as well as dependency between services and sensor observations. Thus, the service can be inferred based on the learnt model. To achieve this, we propose a three-layer model in our work. At the first layer, raw data from sensors are interpreted as context information after noise removal. The second layer is dynamic Bayesian networks which model the observation sequences including inhabitants' location and electrical appliance (EA) information. At the highest layer, we integrate second layer's, environment information and the relations between inhabitants to provide the service to inhabitants. Therefore, the system can infer appropriate services to inhabitants at right time and right place, and let them feel comfortable. In experiments, we show our model can provide reliable and appreciate services to inhabitants in a smart home environment.
In this paper we present results related to achieving finegrained activity recognition for context-aware computing applications. We examine the advantages and challenges of reasoning with globally unique object instances detected by an RFID glove. We present a sequence of increasingly powerful probabilistic graphical models for activity recognition. We show the advantages of adding additional complexity and conclude with a model that can reason tractably about aggregated object instances and gracefully generalizes from object instances to their classes by using abstraction smoothing. We apply these models to data collected from a morning household routine.
We address the problem of recognizing sequences of human interaction patterns in meetings, with the goal of structuring them in semantic terms. The investigated patterns are inherently group-based (defined by the individual activities of meeting participants, and their interplay), and multimodal (as captured by cameras and microphones). By defining a proper set of individual actions, group actions can be modeled as a two-layer process, one that models basic individual activities from low-level audio-visual features, and another one that models the interactions. We propose a two-layer Hidden Markov Model (HMM) framework that implements such concept in a principled manner, and that has advantages over previous works. First, by decomposing the problem hierarchically, learning is performed on low-dimensional observation spaces, which results in simpler models. Second, our framework is easier to interpret, as both individual and group actions have a clear meaning, and thus easier to improve. Third, different HMM models can be used in each layer, to better reflect the nature of each subproblem. Our framework is general and extensible, and we illustrate it with a set of eight group actions, using a public five-hour meeting corpus. Experiments and comparison with a single-layer HMM baseline system show its validity.
Dynamic Probabilistic Networks (DPNs) are exploited for modeling the temporal relationships among a set of different object temporal events in the scene for a coherent and robust scene-level behaviour interpretation. In particular, we develop a Dynamically Multi-Linked Hidden Markov Model (DML-HMM) to interpret group activities involving multiple objects captured in an outdoor scene. The model is based on the discovery of salient dynamic interlinks among multiple temporal events using DPNs. Object temporal events are detected and labeled using Gaussian Mixture Models with automatic model order selection. A DML-HMM is built using Schwarz's Bayesian Information Criterion based factorisation resulting in its topology being intrinsically determined by the underlying causality and temporal order among different object events. Our experiments demonstrate that its performance on modelling group activities in a noisy outdoor scene is superior compared to that of a Multi-Observation Hidden Markov Model (MOHMM), a Parallel Hidden Markov Model (PaHMM) and a Coupled Hidden Markov Model (CHMM).
Our goal is to exploit human motion and object context to perform action recognition and object classification. Towards this end, we introduce a framework for recognizing actions and objects by measuring image-, object- and action-based information from video. Hidden Markov models are combined with object context to classify hand actions, which are aggregated by a Bayesian classifier to summarize activities. We also use Bayesian methods to differentiate the class of unknown objects by evaluating detected actions along with low-level, extracted object features. Our approach is appropriate for locating and classifying objects under a variety of conditions including full occlusion. We show experiments where both familiar and previously unseen objects are recognized using action and context information
A key aspect of pervasive computing is using computers and sensor networks to effectively and unobtrusively infer users' behavior in their environment. This includes inferring which activity users are performing, how they're performing it, and its current stage. Recognizing and recording activities of daily living is a significant problem in elder care. A new paradigm for ADL inferencing leverages radio-frequency-identification technology, data mining, and a probabilistic inference engine to recognize ADLs, based on the objects people use. We propose an approach that addresses these challenges and shows promise in automating some types of ADL monitoring. Our key observation is that the sequence of objects a person uses while performing an ADL robustly characterizes both the ADL's identity and the quality of its execution. So, we have developed Proactive Activity Toolkit (PROACT).
We compare discriminative and generative learning as typified by logistic regression and naive Bayes. We show, contrary to a widely held belief that discriminative classifiers are almost always to be preferred, that there can often be two distinct regimes of performance as the training set size is increased, one in which each algorithm does better. This stems from the observation -- which is borne out in repeated experiments -- that while discriminative learning has lower asymptotic error, a generative classifier may also approach its (higher) asymptotic error much faster.
Our goal is to exploit human motion and object context to perform action recognition and object classification. Towards this end, we introduce a framework for recognizing actions and objects by measuring image-, object- and action-based information from video. Hidden Markov models are combined with object context to classify hand actions, which are aggregated by a Bayesian classifier to summarize activities. We also use Bayesian methods to differentiate the class of unknown objects by evaluating detected actions along with lowlevel, extracted object features. Our approach is appropriate for locating and classifying objects under a variety of conditions including full occlusion. We show experiments where both familiar and previously unseen objects are recognized using action and context information. 1. Introduction This paper proposes a novel approach to human activity recognition that uses context information of particular objects in the scene. We define classes that contain object-s...
In sequence modeling, we often wish to represent complex interaction between labels, such as when performing multiple, cascaded labeling tasks on the same sequence, or when long-range dependencies exist. We present dynamic conditional random fields (DCRFs), a generalization of linear-chain conditional random fields (CRFs) in which each time slice contains a set of state variables and edges---a distributed state representation as in dynamic Bayesian networks (DBNs)---and parameters are tied across slices. Since exact inference can be intractable in such models, we perform approximate inference using several schedules for belief propagation, including tree-based reparameterization (TRP). On a natural-language chunking task, we show that a DCRF performs better than a series of linear-chain CRFs, achieving comparable performance using only half the training data.
The Index of ADL was developed to study results of treatment and prognosis in the elderly and chronically ill. Grades of the Index summarize over-all performance in bathing, dressing, going to toilet, transferring, continence, and feeding. More than 2,000 evaluations of 1,001 individuals demonstrated use of the Index as a survey instrument, as an objective guide to the course of chronic illness, as a tool for studying the aging process, and as an aid in rehabilitation teaching. Of theoretical interest is the observation that the order of recovery of Index functions in disabled patients is remarkably similar to the order of development of primary functions in children. This parallelism, and similarity to the behavior of primitive peoples, suggests that the Index is based on primary biological and psychosocial function, reflecting the adequacy of organized neurological and locomotor response.
Comparison of generative and discriminative classifiers is an ever-lasting topic. As an important contribution to this topic,
based on their theoretical and empirical comparisons between the naïve Bayes classifier and linear logistic regression, Ng
and Jordan (NIPS 841–848, 2001) claimed that there exist two distinct regimes of performance between the generative and discriminative
classifiers with regard to the training-set size. In this paper, our empirical and simulation studies, as a complement of
their work, however, suggest that the existence of the two distinct regimes may not be so reliable. In addition, for real
world datasets, so far there is no theoretically correct, general criterion for choosing between the discriminative and the
generative approaches to classification of an observation x into a class y; the choice depends on the relative confidence we have in the correctness of the specification of either p(y|x) or p(x, y) for the data. This can be to some extent a demonstration of why Efron (J Am Stat Assoc 70(352):892–898, 1975) and O’Neill
(J Am Stat Assoc 75(369):154–160, 1980) prefer normal-based linear discriminant analysis (LDA) when no model mis-specification
occurs but other empirical studies may prefer linear logistic regression instead. Furthermore, we suggest that pairing of
either LDA assuming a common diagonal covariance matrix (LDA-Λ) or the naïve Bayes classifier and linear logistic regression
may not be perfect, and hence it may not be reliable for any claim that was derived from the comparison between LDA-Λ or the
naïve Bayes classifier and linear logistic regression to be generalised to all generative and discriminative classifiers.
Understanding and recognizing human activities from sensor readings is an important task in pervasive computing. In this paper, we investigate the fundamental problem of recognizing activities for multiple users from sensor readings in a home environment, and propose a novel pattern mining approach to recognize both single-user and multi-user activities in a unified solution. We exploit Emerging Pattern - a type of knowledge pattern that describes significant changes between classes of data - for constructing our activity models, and propose an Emerging Pattern based Multi-user Activity Recognizer (epMAR) to recognize both single-user and multi-user activities.
The world’s population is aging at a phenomenal rate. Certain types of cognitive decline, in particular some forms of memory impairment, occur much more frequently in the elderly. This paper describes Autominder, a cognitive orthotic system intended to help older adults adapt to cognitive decline and continue the satisfactory performance of routine activities, thereby potentially enabling them to remain in their own homes longer. Autominder achieves this goal by providing adaptive, personalized reminders of (basic, instrumental, and extended) activities of daily living. Cognitive orthotic systems on the market today mainly provide alarms for prescribed activities at fixed times that are specified in advance. In contrast, Autominder uses a range of AI techniques to model an individual’s daily plans, observe and reason about the execution of those plans, and make decisions about whether and when it is most appropriate to issue reminders. Autominder is currently deployed on a mobile robot, and is being developed as part of the Initiative on Personal Robotic Assistants for the Elderly (the Nursebot project).
Formalizing computational models for everyday human activities remains an open challenge. Many previous approaches towards this end assume prior knowledge about the structure of activities, using which explicitly defined models are learned in a completely supervised manner. For a majority of everyday environments however, the structure of the in situ activities is generally not known a priori. In this paper we investigate knowledge representations and manipulation techniques that facilitate learning of human activities in a minimally supervised manner. The key contribution of this work is the idea that global structural information of human activities can be encoded using a subset of their local event subsequences, and that this encoding is sufficient for activity-class discovery and classification.In particular, we investigate modeling activity sequences in terms of their constituent subsequences that we call event n-grams. Exploiting this representation, we propose a computational framework to automatically discover the various activity-classes taking place in an environment. We model these activity-classes as maximally similar activity-cliques in a completely connected graph of activities, and describe how to discover them efficiently. Moreover, we propose methods for finding characterizations of these discovered classes from a holistic as well as a by-parts perspective. Using such characterizations, we present a method to classify a new activity to one of the discovered activity-classes, and to automatically detect whether it is anomalous with respect to the general characteristics of its membership class. Our results show the efficacy of our approach in a variety of everyday environments.
We present the use of layered probabilistic representations for modeling human activities, and describe how we use the representation to do sensing, learning, and inference at multiple levels of temporal granularity and abstraction and from heterogeneous data sources. The approach centers on the use of a cascade of Hidden Markov Models named Layered Hidden Markov Models (LHMMs) to diagnose states of a user’s activity based on real-time streams of evidence from video, audio, and computer (keyboard and mouse) interactions. We couple these LHMMs with an expected utility analysis that considers the cost of misclassification. We describe the representation, present an implementation, and report on experiments with our layered architecture in a real-time office-awareness setting.
Activity recognition is a key component for creating intel- ligent, multi-agent systems. Intrinsically, activity recogni- tion is a temporal classification problem. In this paper, we compare two models for temporal classification: hidden Markov models (HMMs), which have long been applied to the activity recognition problem, and conditional random fields (CRFs). CRFs are discriminative models for label- ing sequences. They condition on the entire observation sequence, which avoids the need for independence assump- tions between observations. Conditioning on the observa- tions vastly expands the set of features that can be incorpo- rated into the model without violating its assumptions. Us- ing data from a simulated robot tag domain, chosen because it is multi-agent and produces complex interactions between observations, we explore the dierences in performance be- tween the discriminatively trained CRF and the generative HMM. Additionally, we examine the eect of incorporating features which violate independence assumptions between observations; such features are typically necessary for high classification accuracy. We find that the discriminatively trained CRF performs as well as or better than an HMM even when the model features do not violate the indepen- dence assumptions of the HMM. In cases where features de- pend on observations from many time steps, we confirm that CRFs are robust against any degradation in performance.
Recognising behaviours of multiple people, especially high-level behaviours, is an important task in surveillance systems. When the reliable assignment of people to the set of observations is unavailable, this task becomes compli- cated. To solve this task, we present an approach, in which the hierarchical hidden Markov model (HHMM) is used for modeling the behaviour of each person and the joint probabilistic data association filters (JPDAF) is applied for data association. The main contributions of this paper lie in the integration of multiple HHMMs for recognising high-level behaviours of multiple people and the construction of the Rao-Blackwellised particle filters (RBPF) for ap- proximate inference. Preliminary experimental results in a real environment show the robustness of our integrated method in behaviour recognition and its advantage over the use of Kalman filter in tracking people.
In this work, algorithms are developed and evaluated to de- tect physical activities from data acquired using five small biaxial ac- celerometers worn simultaneously on different parts of the body. Ac- celeration data was collected from 20 subjects without researcher su- pervision or observation. Subjects were asked to perform a sequence of everyday tasks but not told specifically where or how to do them. Mean, energy, frequency-domain entropy, and correlation of acceleration data was calculated and several classifiers using these features were tested. De- cision tree classifiers showed the best performance recognizing everyday activities with an overall accuracy rate of 84%. The results show that although some activities are recognized well with subject-independent training data, others appear to require subject-specific training data. The results suggest that multiple accelerometers aid in recognition because conjunctions in acceleration feature values can effectively discriminate many activities. With just two biaxial accelerometers - thigh and wrist - the recognition performance dropped only slightly. This is the first work to investigate performance of recognition algorithms with multiple, wire-free accelerometers on 20 activities using datasets annotated by the subjects themselves.
Activity recognition is significant in intelligent surveillance. In this paper, we present a novel approach to the recognition of interacting activities based on dynamic Bayesian network (DBN). In this approach the features representing the object motion are divided into two classes: global features and local features, which are at two different spatial scales. Global features describe object motion at a large spatial scale and relations between objects or between the object and environment, and local ones represent the motion details of objects of interest. We propose a new DBN model structure with state duration to model human interacting activities. This DBN model structure combines the global features with local ones harmoniously. The effectiveness of this novel approach is demonstrated by experiment
In this paper we introduce a new dynamic Bayesian network that separates the speakers and their speak- ing turns in a multi-person conversation. We pro- tect the speakers' privacy by using only features from which intelligible speech cannot be recon- structed. The model we present combines data from multiple audio streams, segments the streams into speech and silence, separates the different speakers, and detects when other nearby individu- als who are not wearing microphones are speaking. No pre-trained speaker specific models are used, so the system can be easily applied in new and dif- ferent environments. We show promising results in two very different datasets that vary in background noise, microphone placement and quality, and con- versational dynamics.
This paper presents a synergistic track- and body-level analysis framework for multi-person inter- action and activity analysis in the context of video sur- veillance. The proposed two-level analysis framework covers human activities both in wide and narrow fields of view with distributed camera sensors. The track-level analysis deals with the gross-level activity patterns of multiple tracks in various wide-area surveillance situa- tions. The body-level analysis focuses on detailed-level activity patterns of individuals in isolation or in groups. 'Spatio-temporal personal space' is introduced to model various patterns of grouping behavior between persons. 'Adaptive context switching' is proposed to mediate the track-level and body-level analysis depending on the interpersonal configuration and imaging fidelity. Our approach is based on the hierarchy of action concepts: static pose, dynamic gesture, body-part action, single- person activity, and group interaction. Event ontology with human activity hierarchy combines the multi-level analysis results to form a semantically meaningful event description. Experimental results with real-world data show the effectiveness of the proposed framework.
Recognizing human activities from sensor readings has recently attracted much research interest in pervasive computing due to its potential in many applications, such as assistive living and healthcare. This task is particularly challenging because human activities are often performed in not only a simple (i.e., sequential), but also a complex (i.e., interleaved or concurrent) manner in real life. Little work has been done in addressing complex issues in such a situation. The existing models of interleaved and concurrent activities are typically learning-based. Such models lack of flexibility in real life because activities can be interleaved and performed concurrently in many different ways. In this paper, we propose a novel pattern mining approach to recognize sequential, interleaved, and concurrent activities in a unified framework. We exploit Emerging Pattern—a discriminative pattern that describes significant changes between classes of data—to identify sensor features for classifying activities. Different from existing learning-based approaches which require different training data sets for building activity models, our activity models are built upon the sequential activity trace only and can be applied to recognize both simple and complex activities. We conduct our empirical studies by collecting real-world traces, evaluating the performance of our algorithm, and comparing our algorithm with static and temporal models. Our results demonstrate that, with a time slice of 15 seconds, we achieve an accuracy of 90.96 percent for sequential activity, 88.1 percent for interleaved activity, and 82.53 percent for concurrent activity. Index Terms—Human activity recognition, pattern analysis, emerging pattern, classifier design and evaluation.
We present methods for coupling hidden Markov models (hmms) to model systems of multiple interacting processes. The resulting models have multiple state variables that are temporally coupled via matrices of conditional probabilities. We introduce a deterministic O(T (CN) 2 ) approximation for maximum a posterior (MAP) state estimation which enables fast classification and parameter estimation via expectation maximization. An "N-heads" dynamic programming algorithm samples from the highest probability paths through a compact state trellis, minimizing an upper bound on the cross entropy with the full (combinatoric) dynamic programming problem. The complexity is O(T (CN) 2 ) for C chains of N states apiece observing T data points, compared with O(TN 2C ) for naive (Cartesian product), exact (state clustering), and stochastic (Monte Carlo) methods applied to the same inference problem. In several experiments examining training time, model likelihoods, classification accuracy, and ro...
We present conditional random fields, a framework for building probabilistic models to segment and label sequence data. Conditional random fields offer several advantages over hidden Markov models and stochastic grammars for such tasks, including the ability to relax strong independence assumptions made in those models. Conditional random fields also avoid a fundamental limitation of maximum entropy Markov models (MEMMs) and other discriminative Markov models based on directed graphical models, which can be biased towards states with few successor states. We present iterative parameter estimation algorithms for conditional random fields and compare the performance of the resulting models to HMMs and MEMMs on synthetic and natural-language data.
- J Nocedal
- S J Wright
J. Nocedal, S.J. Wright, Numerical Optimization, Springer, New York, 1999.
- E R Smith
- D M Mackie
E.R. Smith, D.M. Mackie, Social Psychology, Routledge, London, 1999.
Joint recognition of multiple concurrent activities using factorial conditional random fields
- T Y Wu
- C C Lian
- J Y Hsu
T.Y. Wu, C.C. Lian, J.Y. Hsu, Joint recognition of multiple concurrent activities using factorial conditional random fields, in: Proc. AAAI Workshop Plan,
Activity, and Intent Recognition, California, 2007.
Dynamic conditional random fields: factorized probabilistic models for labeling and segmenting sequence data
- Brand