Article

Recognizing interleaved and concurrent activities using qualitative and quantitative temporal relationships

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

The majority of approaches to activity recognition in sensor environments are either based on manually constructed rules for recognizing activities or lack the ability to incorporate complex temporal dependencies. Furthermore, in many cases, the rather unrealistic assumption is made that the subject carries out only one activity at a time. In this paper, we describe the use of Markov logic as a declarative framework for recognizing interleaved and concurrent activities incorporating both input from pervasive lightweight sensor technology and common-sense background knowledge. In particular, we assess its ability to learn statistical-temporal models from training data and to combine these models with background knowledge to improve the overall recognition accuracy. We also show the viability and the benefit of exploiting both qualitative and quantitative temporal relationships like the duration of the activities and their temporal order. To this end, we propose two Markov logic formulations for inferring the foreground activity as well as each activities’ start and end times. We evaluate the approach on an established dataset where it outperforms state-of-the-art algorithms for activity recognition.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... It is an established data set used by many researchers. It was used by Helaoui et al. [17] for testing their HMM based models for recognition of interleaved activities. The data set was collected by observing routine morning activities performed using common household objects and are commonly interleaved. ...
... It must be noted that the number of false positives and false negatives coincide. This is due to the fact that at each time step only one activity occurs [17]. The results obtained by the algorithm using 10 fold cross validation are given in Table 2 and in Figure 2. Since the number of false positives and false negatives coincide, the precision, recall and Fmeasure values remain Figure 2. Performance of OBCAR in predicting the foreground activity the same. ...
... The average F-measure value obtained is 0.96. This is more than the F-measure values 0.92 and 0.93 obtained by Helaoui et al. [17] for the same data set. Task 2 is about deriving all background activities at each time step. ...
Article
Interleaving or concurrently pursuing two or more activities is the nature of human beings. This is more so for activities of daily living(ADLs). So, an activity recognition system must be able to recognize concurrent and interleaved activities in real-life situations without imposing any restrictions on the user. Users must be allowed to freely interleave and switch over activities. In this paper a simple and novel algorithm, named Object Based Composite Activities Recognition(OBCAR) is proposed for recognizing composite activities from environmental sensor data sets. The algorithm makes use of an automatically constructed finite automaton to identify composite activities. The results achieved by the algorithm are highly promising when tested with a publicly available data set.
... It is an established data set used by many researchers. It was used by Helaoui et al. [17] for testing their HMM based models for recognition of interleaved activities. The data set was collected by observing routine morning activities performed using common household objects and are commonly interleaved. ...
... It must be noted that the number of false positives and false negatives coincide. This is due to the fact that at each time step only one activity occurs [17]. The results obtained by the algorithm using 10 fold cross validation are given in Table 2 and in Figure 2. Since the number of false positives and false negatives coincide, the precision, recall and Fmeasure values remain Figure 2. Performance of OBCAR in predicting the foreground activity the same. ...
... The average F-measure value obtained is 0.96. This is more than the F-measure values 0.92 and 0.93 obtained by Helaoui et al. [17] for the same data set. Task 2 is about deriving all background activities at each time step. ...
Article
Interleaving or concurrently pursuing two or more activities is the nature of human beings. This is more so for activities of daily living(ADLs). So, an activity recognition system must be able to recognize concurrent and interleaved activities in real-life situations without imposing any restrictions on the user. Users must be allowed to freely interleave and switch over activities. In this paper a simple and novel algorithm, named Object Based Composite Activities Recognition(OBCAR) is proposed for recognizing composite activities from environmental sensor data sets. The algorithm makes use of an automatically constructed finite automaton to identify composite activities. The results achieved by the algorithm are highly promising when tested with a publicly available data set.
... Activity recognition has been widely investigated using three categories of approaches, namely, data-driven (DD) [4][5][6][7][8], knowledge-driven (KD) [9][10][11][12][13], and hybrid [14][15][16] activity recognition approaches. In data-driven activity recognition, activity models are learnt from pre-existing datasets using existing well-developed machine learning techniques. ...
... Simple activity recognition has been widely explored in DD [6,[17][18][19][20], KD [9][10][11][21][22][23] , and hybrid [13,24,25] activity recognition. However, composite activity recognition is only investigated to a limited extent in DD [4,6,8,[26][27][28] and hybrid [14][15][16] activity recognition communities. In the KD activity recognition research community, the recognition of composite activities still remains largely unexplored. ...
... Hybrid approaches to activity recognition combine techniques from data-driven and knowledge-driven activity recognition. So far, only Markov logic networks (MLN) [15] and HMMs with Allen logic [16] have been used to support activity recognition of simple activities, and interleaved and concurrent activities. Both approaches encode and use temporal knowledge but rely on automatically extracting the relevant temporal patterns from data sets. ...
Article
Full-text available
Activity recognition is essential in providing activity assistance for users in smart homes. While significant progress has been made for single-user single-activity recognition, it still remains a challenge to carry out real-time progressive composite activity recognition. This paper introduces a hybrid ontological and temporal approach to composite activity modelling and recognition by extending existing ontology-based knowledge-driven approach. The compelling feature of the approach is that it combines ontological and temporal knowledge representation formalisms to provide powerful representation capabilities for activity modelling. The paper describes in details ontological activity modelling which establishes relationships between activities and their involved entities, and temporal activity modeling which defines relationships between constituent activities of a composite activity. As an essential part of the model, the paper also presents methods for developing temporal entailment rules to support the interpretation and inference of composite activities. In addition, this paper outlines an integrated architecture for composite activity recognition and elaborated a unified activity recognition algorithm which can support the recognition of simple and composite activities. The approach has been implemented in a feature-rich prototype system upon which testing and evaluation have been conducted. Initial experimental results have shown average recognition accuracy of 100% and 88.26% for simple and composite activities, respectively.
... In [61], [62],the authors applied a statistical relational framework for the recognition of composite activities. They use Markov Logic Network (MLN) to incorporate commonsense background knowledge to model qualitative temporal relationships. ...
... Strengths Weaknesses [61], [62]  Accurately recognize composite activities through combination of statistical and relational features.  Flexibility of Markov Logic by ability to integrate and evaluate more temporal relationships without redeveloping a novel model each time. ...
Article
Full-text available
Recognition of human activities is a challenging task due to human’s tendency to perform activities not only in a simple way, but also in a complex and multitasking way. Many research attempts address the recognition of simple activities, but little work targets the recognition of complex activities. Currently research on complex activity recognition using sensors is growing in many application domains. This paper provides an analysis of the most prominent complex sensor-based activity recognition. We analyze the structure and working methodology of the existing complex activities recognition systems, discuss their strengths and weaknesses. In addition, we evaluate existing proposals from three different perspectives including overall system evaluation, performance evaluation, and dataset evaluation.
... Previous studies have investigated interleaved activities, activity recognition using agents and using multi-agent systems, however, none have combined these, especially using a MAS for the purpose of ambient assisted living. A study by Helaoui et al. looked at recognizing interleaved activities using a Markov logic method [5]. They took into consideration the start and end times of activities and evaluated it against activity recognition algorithms. ...
... Thus, supporting that a MAS approach is successful at fulfilling the task of interleaved activity identification due to its ability to run the agents in parallel, all looking for their own activities. When results were compared to that of the study by Helaoui et al. [5] it was found that the proposed methods' results were comparable to their study with a precision, recall and F-measure of 0.69, 0.81 & 0.74 as to their results of 0.71, 0.99 & 0.82. Their results were based around the assumptions that if the first sensor event is missed then the activity is not registered as happening. ...
Conference Paper
This paper presents a Multi-agent approach to identifying interleaved activities in a smart environment. The use of binary contact sensors was explored to identify Activities of Daily Living with assistance from a system made up of agents. Activities were identified when an activity trigger event was detected. Upon detection, a time window would activate around the trigger event, prompting the activity agents to identify which of their events were present within the set time window, thus enabling them to calculate a percentage of likeliness that the activity was their own. As a result, the highest percentage of activity matches would be displayed as having occurred. To evaluate this approach, 36 interleaved activities were processed and compared with a single agent system in addition to 28 non-interleaved activities. As a benchmark, the results were compared to that of another study. Results presented a precision, recall and F-measure of 0.69, 0.81 and 0.74. This paper concluded that the Multi Agent System (MAS) is a promising approach for identifying interleaved activities when compared to methods that fail when presented with data that is not in a set order. However, several limitations are present which need to be overcome to make the results more accurate when compared to other approaches.
... In our research, we provide the possible practical benefits of such a perspective in Section V. One final stream of research regarding schedulebased systems is regarding the processing of the events data [17]. ...
... • While the analysis of serial events is fundamental, consideration can be made in future research for concurrent events (events that can independently take place at the same time) in the system. The consideration for concurrent events would require significant changes in the augmentation algorithm, as it would require complex event processing [17]. However, the applicability of the current framework, as well as the types of analysis and the insights obtained, would still be relevant and useful. ...
Article
Full-text available
A schedule-based system is a system that operates on or contains within a schedule of events and breaks at particular time intervals. Entities within the system show presence or absence in these events by entering or exiting the locations of the events. Given radio frequency identification (RFID) data from a schedule-based system, what can we learn about the system (the events and entities) through data mining? Which data mining methods can be applied so that one can obtain rich actionable insights regarding the system and the domain? The research goal of this paper is to answer these posed research questions, through the development of a framework that systematically produces actionable insights for a given schedule-based system. We show that through integrating appropriate data mining methodologies as a unified framework, one can obtain many insights from even a very simple RFID dataset, which contains only very few fields. The developed framework is general, and is applicable to any schedule-based system, as long as it operates under certain basic assumptions. The types of insights are also general, and are formulated in this paper in the most abstract way. The applicability of the developed framework is illustrated through a case study, where real world data from a schedule-based system is analyzed using the introduced framework. Insights obtained include the profiling of entities and events, the interactions between entity and events, and the relations between events.
... Combining both paradigms has been the motivation of some recent works, including [7] and [22]. However, the looselycoupled nature of most hybrid activity recognition systems proposed so far provides only a partial solution to the limitations of data-driven and knowledge based approaches. ...
... To the best of our knowledge, the publicly available log-linear DL reasoner Elog [20] is the only reasoner which responds to the need of finding the most probable consistent ontology instead of computing the a-posteriori probability of axioms such as in [26]. A further motivation of our choice is related to the successful application of Log-linear models to the problem of activity recognition in a previous work [7]. There, the activity recognition problem was addressed by employing Markov Logic Networks, a statistical relational paradigm combining log-linear models and first order logic [25]. ...
Article
Full-text available
A major challenge of ubiquitous computing resides in the acquisition and modelling of rich and heterogeneous context data, among which, ongoing human activities at different degrees of granularity. In a previous work, we advocated the use of probabilistic description logics (DLs) in a multilevel activity recognition framework. In this paper, we present an in-depth study of activity modeling and reasoning within that framework, as well as an experimental evaluation with a large real-world dataset. Our solution allows us to cope with the uncertain nature of ontological descriptions of activities, while exploiting the expressive power and inference tools of the OWL 2 language. Targeting a large dataset of real human activities, we developed a probabilistic ontology modeling nearly 150 activities and actions of daily living. Experiments with a prototype implementation of our framework confirm the viability of our solution.
... Hong et al. [2009] have used the Dempster-Shafer theory to incorporate contextual information from uncertain sensor data for activity recognition. In Helaoui et al. [2011b], Markov logic networks are used for complex activity recognition where hard and soft formulae are defined for recognizing activities. Such techniques can allow for expressive and flexible integration of background knowledge and context, but this has not been demonstrated. ...
... Such techniques can allow for expressive and flexible integration of background knowledge and context, but this has not been demonstrated. Our approach is fundamentally different from Hong et al. [2009] and Helaoui et al. [2011b], as we use a context-driven activity theory for recognizing complex activities and discover complex activity signatures. We incorporate situation recognition in the process of activity recognition. ...
Article
Full-text available
In pervasive and ubiquitous computing systems, human activity recognition has immense potential in a large number of application domains. Current activity recognition techniques (i) do not handle varia-tions in sequence, concurrency and interleaving of complex activities; (ii) do not incorporate context; and (iii) require large amounts of training data. There is a lack of a unifying theoretical framework which exploits both domain knowledge and data-driven observations to infer complex activities. In this article, we propose, develop and validate a novel Context-Driven Activity Theory (CDAT) for recognizing complex activities. We develop a mechanism using probabilistic and Markov chain analysis to discover complex activity signatures and generate complex activity definitions. We also develop a Complex Activity Recognition (CAR) algorithm. It achieves an overall accuracy of 95.73% using extensive experimentation with real-life test data. CDAT utilizes context and links complex activities to situations, which reduces inference time by 32.5% and also reduces training data by 66%.
... Knowledge representation-based approaches can also aid the visual analysis of network traffic [61] or the semantic fusion of data originating from different sources [60]. Moreover, ontology-based approaches can reason about the occurrence of composite activities [43,12,21,33,49]. ...
Article
The design and operation of modern software systems exhibit a shift towards virtualization, containerization and service-based orchestration. Performance capacity engineering and resource utilization tuning become priority requirements in such environments. Measurement-based performance evaluation is the cornerstone of capacity engineering and designing for performance. Moreover, the increasing complexity of systems necessitates rigorous performance analysis approaches. However, empirical performance analysis lacks sophisticated model-based support similar to the functional design of the system. The paper proposes an ontology-based approach for facilitating and guiding the empirical evaluation throughout its various steps. Hyperledger Fabric (HLF), an open-source blockchain platform by the Linux Foundation, is modelled and evaluated as a pilot example of the approach, using the standard TPC-C performance benchmark workload.
... Knowledge representation-based approaches can also aid the visual analysis of network traffic [51] or the semantic fusion of data originating from different sources [50]. Moreover, ontology-based approaches can reason about the occurrence of composite activities [33,10,18,26,38]. ...
Preprint
Full-text available
The design and operation of modern software systems exhibit a shift towards virtualization, containerization and service-based orchestration. Performance capacity engineering and resource utilization tuning become priority requirements in such environments. Measurement-based performance evaluation is the cornerstone of capacity engineering and designing for performance. Moreover, the increasing complexity of systems necessitates rigorous performance analysis approaches. However, empirical performance analysis lacks sophisticated model-based support similar to the functional design of the system. The paper proposes an ontology-based approach for facilitating and guiding the empirical evaluation throughout its various steps. Hyperledger Fabric (HLF), an open-source blockchain platform by the Linux Foundation, is modelled and evaluated as a pilot example of the approach, using the standard TPC-C performance benchmark workload.
... The combination of semantic and probability statistics algorithm is the promising method of the inference, especially for the complex representation and relationship of activities situation (Okeyo et al. 2014b;Riboni and Bettini 2011;Ordóñez et al. 2013;Meditskos et al. 2013). Markov Logic Network (MLN) is a combination solution which has been widely adopted (Gayathri et al. 2017;Helaoui et al. 2011). These studies are mainly handling the activity recognition in interleaved and concurrent scenes. ...
Article
Full-text available
Using Sensor-based approach in activity recognition usually requires the deployment of many ambient sensors to objects and environments. Each sensor can be triggered by more than one activity, e.g., a touch sensor of a cooker can be triggered by cooking, doing dish and so on. An activity consists of some sensor events. When the number of same sensors are in the majority of two activities, the two activities are defined as similar activities which are difficult to distinguish. To address the challenge of recognizing similar activities, this paper conceives a new activity recognition approach incorporating high-dimensional features of duration and time block characteristics to improve the inference performance. In a further step, we take advantage of these similar activities to build a hierarchical structure model which can improve capacities of expandability and standardization. We design experiments of similar activity in our daily life to evaluate this solution. The results show that high-dimensional temporal features improved similar activity inference accuracy on an average of 1.88 times, and the use of hierarchical structure can generalize specific rules to standard ones which decreases similar activity recognition computation time on an average of 0.36 times.
... Nevertheless, this method requires residents to wear devices and some residents such as elderly and children may forget to wear them in their houses. Hence such approaches that use low-cost infrastructurebased activity recognition [3], [4], [11] are more reasonable in the home. ...
Article
Location-based services in household enable not only estimating activities but also detecting the accident location of residents including children, parents and elderly people. Furthermore, home management systems make use of location information of residents for encouraging residents to live comfortably in their own homes. Passive infrared binary motion sensors are widely used in home tracking systems because of low energy consumption and flexibility on deployment. However, it's hard for non-specialists like residents to manage such location information of sensors which are very important for tracking systems. To mitigate human efforts and errors, we propose a sensor localization method to automatically identify the location of multiple binary motion sensors in a house from the observed motion detection event sequences by binary sensors. The method finds the movement patterns and characteristics of sojourn time of residents to identify the rooms where those sensors are located as well as the proximity relations among sensors assisted by prior knowledge such as from floorplan. The experimental results in actual houses show that our method can estimate the location of sensors placed close to the anchor locations within a one-day observation, and the accuracy of our approach is above 80% after three-day observation.
... Markov Logic Networks (MLN) [52] is a probabilistic logic that was successfully applied to sensor-based AR, due to its natural support for reasoning with uncertain information [53,37]. Formally, a MLN M is a finite set of pairs ...
Article
In several domains, including healthcare and home automation, it is important to unobtrusively monitor the activities of daily living (ADLs) carried out by people at home. A popular approach consists in the use of sensors attached to everyday objects to capture user interaction, and ADL models to recognize the current activity based on the temporal sequence of used objects. Often, ADL models are automatically extracted from labeled datasets of activities and sensor events, using supervised learning techniques. Unfortunately, acquiring such datasets in smart homes is expensive and violates users’ privacy. Hence, an alternative solution consists in manually defining ADL models based on common sense, exploiting logic languages such as description logics. However, manual specification of ADL ontologies is cumbersome, and rigid ontological definitions fail to capture the variability of activity execution. In this paper, we introduce a radically new approach enabled by the recent proliferation of tagged visual contents available on the Web. Indeed, thanks to the popularity of social network applications, people increasingly share pictures and videos taken during the execution of every kind of activity. Often, shared contents are tagged with metadata, manually specified by their owners, that concisely describe the depicted activity. Those metadata represent an implicit activity label of the picture or video. Moreover, today’s computer vision tools support accurate extraction of tags describing the situation and the objects that appear in the visual content. By reasoning with those tags and their corresponding activity labels, we can reconstruct accurate models of a comprehensive set of human activities executed in the most disparate situations. This approach overcomes the main shortcomings of existing techniques. Compared to supervised learning methods, it does not require the acquisition of training sets of sensor events and activities. Compared to knowledge-based methods, it does not involve any manual modeling effort, and it captures a comprehensive array of execution modalities. Through extensive experiments with large datasets of real-world ADLs, we show that this approach is practical and effective.
... Despite all the researches on HAR, still opened challenges such as overlapping or concurrent activities, have yet to be solved. Overlap is noted as the phenomenon that different activity classes activate the same set of sensor events which makes overlapping activities hard to discriminate only based on the types of sensor events that they have triggered (Wen et al. 2015;Helaoui et al. 2011). ...
Article
Full-text available
In the last few years there has been a growing interest in Human activity recognition (HAR) topic. Sensor-based HAR approaches, in particular, has been gaining more popularity owing to their privacy preserving nature. Furthermore, due to the widespread accessibility of the internet, a broad range of streaming-based applications such as online HAR, has emerged over the past decades. However, proposing sufficiently robust online activity recognition approach in smart environment setting is still considered as a remarkable challenge. This paper presents a novel online application of Hierarchical Hidden Markov Model in order to detect the current activity on the live streaming of sensor events. Our method consists of two phases. In the first phase, data stream is segmented based on the beginning and ending of the activity patterns. Also, on-going activity is reported with every receiving observation. This phase is implemented using Hierarchical Hidden Markov models. The second phase is devoted to the correction of the provided label for the segmented data stream based on statistical features. The proposed model can also discover the activities that happen during another activity - so-called interrupted activities. After detecting the activity pane, the predicted label will be corrected utilizing statistical features such as time of day at which the activity happened and the duration of the activity. We validated our proposed method by testing it against two different smart home datasets and demonstrated its effectiveness, which is competing with the state-of-the-art methods.
... Such temporal relations require a tertiary relation among instances of the concepts whereas ontologies typically hold a binary relation among the concepts. In order to accommodate these temporal relations [20,36], a time ontology [22] and the 4D extended fluent approach [48] have been used. In smart homes, activities are partially static and diachronic in nature. ...
Article
Full-text available
Activity recognition has a vital role in smart home operations. One of the major challenges in object-sensor-based activity recognition is to learn the complete activity model derived from a generic activity model for sequential and parallel activities. Such challenge exists due to erratic degrees of dissimilar activities in which inhabitants perform activities in sequential and interleaved fashion while interacting with different objects. The proposed work focuses on recognizing a complete set of actions (of activity) by exploiting different knowledge engineering techniques, ontology-based temporal formalisms and data driven techniques. Semantic Segmentation has been employed to establish the generic activity model. The spurious semantic segmentation produced by sensor noise or erratic behaviour is removed by Allen’s temporal formalism. Moreover, Tversky’s feature-based similarity has been used to remove the highly similar spurious activities produced as a result of mistaken interactions with wrong home objects. The duration to perform activities varies among inhabitants; such duration intervals are identified dynamically using the proposed model in order to have a complete activity model. A comprehensive set of experiments has been carried out for evaluating the proposed model where the results based upon different metrics assert its effectiveness especially when compared with other contemporary techniques. © 2018 Springer Science+Business Media, LLC, part of Springer Nature
... modelling of complex logical relations, sharing information coming from heterogeneous sources, availability of sound and complete reasoning engines), little focus has been given on the recognition of interleaved activities. In [11], the problem of detecting interleaved activities is approached by combining statisticaltemporal models obtained from training data and background knowledge in the form of temporal first-order rules. Although the combination of data-and knowledge-driven solutions seems promising, the definition of strict temporal rules often fails to incorporate the level of flexibility required in pervasive environments. ...
Conference Paper
Full-text available
Understanding human activities in pervasive environments is a key challenge that involves fusion and correlation of multimodal sensor information. Many research efforts have been recently focused on knowledge-driven solutions to human activity recognition, using ontologies for defining activity models and for capturing contextual information. In most cases, however, the unrealistic assumption is made that activities are performed in a sequential, non-interrupted manner, hampering their applicability in real-world scenarios. In this paper, we present a framework for detecting interleaved activities of daily living (ADL) using (a) OWL 2 for implementing the underlying model semantics capturing contextual dependencies among activities, and (b) defeasible reasoning for introducing a flexible conflict resolution mechanism. The proposed framework has been integrated in an existing context-aware ADL recognition framework, which is being used for supporting the diagnosis of the Alzheimer's disease in a controlled environment.
... The hidden predicates correspond to the activity boundaries: startActivity(a, t) indicates that activity a begins at time t and endActivity(a, t) indicates that activity a ends at time t. The approach used for boundary recognition, initially proposed in [17], is to write appropriate soft formulae to create a correlation between windows of n consecutive sensor events and start/end of activities. For example, in the case of n = 1 the following soft formulae can be used: ...
Conference Paper
Objective: In an ageing world population more citizens are at risk of cognitive impairment, with negative consequences on their ability of independent living, quality of life and sustainability of healthcare systems. Cognitive neuroscience researchers have identified behavioral anomalies that are significant indicators of cognitive decline. A general goal is the design of innovative methods and tools for continuously monitoring the functional abilities of the seniors at risk and reporting the behavioral anomalies to the clinicians. SmartFABER is a pervasive system targeting this objective. Methods: A non-intrusive sensor network continuously acquires data about the interaction of the senior with the home environment during daily activities. A novel hybrid statistical and knowledge-based technique is used to analyses this data and detect the behavioral anomalies, whose history is presented through a dashboard to the clinicians. Differently from related works, SmartFABER can detect abnormal behaviors at a fine-grained level. Results: We have fully implemented the system and evaluated it using real datasets, partly generated by performing activities in a smart home laboratory, and partly acquired during several months of monitoring of the instrumented home of a senior diagnosed with MCI. Experimental results, including comparisons with other activity recognition techniques, show the effectiveness of SmartFABER in terms of recognition rates.
... Despite their ability to cope with some of the limitations of data-driven approaches, rule-based systems suffer from many restrictions, including limited support for uncertainty and temporal reasoning. Combining both paradigms has been the motivation of our previous works (Helaoui, Niepert, and Stuckenschmidt 2011) and (Riboni and Bettini 2011). In the former, we applied Markov Logic Networks to unite both logical statements as well as probabilistic ones in one single framework. ...
Conference Paper
A major challenge of pervasive context-aware computing and intelligent environments resides in the acquisition and modelling of rich and heterogeneous context data. Decisive aspects of this information are the ongoing human activities at different degrees of granularity. We conjecture that ontology-based activity models are key to support interoperable multilevel activity representation and recognition. In this paper, we report on an initial investigation about the application of probabilistic description logics (DLs) to a framework for the recognition of multilevel activities in intelligent environments. In particular, being based on Log-linear DLs, our approach leverages the potential of highly expressive description logics with probabilistic reasoning in one unified framework. While we believe that this approach is very promising, our preliminary investigation suggests that challenging research issues remain open, including extensive support for temporal reasoning, and optimizations to reduce the computational cost. Copyright © 2012, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.
Article
Human activity recognition (HAR) is crucial for ubiquitous computing systems. While HAR systems are able to recognize a predefined set of activities established during the development process, they often fail to handle users' unique ways of completing these activities and changes in their behavior over time, as well as different activities. Knowledge-based HAR models have been proposed to help individuals create new activity definitions based on common-sense rules, but little research has been done to understand how users approach this task. To investigate this process, we developed and studied how people interact with an explainable knowledge-based HAR development tool called exHAR. Our tool empowers users to define their activities as a set of factual propositions. Users can debug these definitions by soliciting explanations for model predictions (why and why-not) and candidate corrections for faulty predictions (what-if and how-to). After conducting a study to evaluate the effectiveness of exHAR in helping users design accurate HAR systems, we conducted a think-aloud study to better understand people's approach to debugging and personalizing HAR systems and the challenges they may encounter. Our findings revealed why some participants had inaccurate mental models of knowledge-based HAR systems and inefficient approaches to the debugging process.
Chapter
Activity recognition for shepherding is a way for an artificial intelligence system to learn and understand shepherding behaviours. The problem we describe is one of recognising behaviours within a shepherding environment, where a cognitive agent (the shepherd) influences agents within the system (sheep) through a shepherding actuator (sheepdog), to achieve an intent. Shepherding is pervasive in everyday life with AI agents, collections of animals, and humans all partaking in different forms. Activity recognition in this context is the generation of a transformation from sensor stream data to the perceived behaviour of an agent under observation from the perspective of an external observer. We present a method of classifying behaviour through the use of spatial data and codify action, behaviour, and intent states through a multi-level classification mapping process.
Chapter
Activity recognition is essential in providing activity assistance for users in smart homes. While significant progress has been made for single-user single-activity recognition, it still remains a challenge to carry out real-time progressive composite activity recognition. This Chapter introduces a hybrid approach to composite activity modelling and recognition by extending existing ontology-based knowledge-driven approach with temporal modelling and reasoning methods. It combines and describes in details ontological activity modelling which establishes relationships between activities and their involved entities, and temporal activity modelling which defines relationships between constituent activities of a composite activity, thus providing powerful representation capabilities for composite activity modelling. The Chapter describes an integrated architecture for composite activity recognition, and elaborates a unified activity recognition algorithm for the recognition of simple and composite activities. As an essential part of the model, the Chapter also presents methods for developing temporal entailment rules to support the interpretation and inference of composite activities. An example case study has been undertaken using a number of experiments to evaluate and demonstrate the proposed approach in a feature-rich multi-agent prototype system.
Conference Paper
This study's aim was to create a modelisation, and a simulation of a wireless sensor network in conjunction with the use of sensitivity analysis, robust analysis, and multicriteria optimization. The idea behind this is to use this technology in the medical scope of home cardiac monitoring. After an initial phase of research to find the right network simulator, the definition of the simulation parameters has started the robust analysis and sensitivity analysis using HDMR method. Next stage was to implement this method into Matlab, and to define a communication protocol between Matlab and the simulator, so they can exchange parameters and results. At last, gathered data analysis will help to define a product with optimized characteristics.
Conference Paper
In this paper, we propose a zone-based living activity recognition method. The proposed method introduces a new concept called activity zone which represents the location and the area of an activity that can be done by a user. By using this activity zone concept, the proposed scheme uses Markov Logic Network (MLN) which integrates a common sense knowledge (i.e. area of each activity) with a probabilistic model. The proposed scheme can utilize only a positioning sensor attached to a resident with/without power meters attached to appliances of a smart environment. We target 10 different living activities which cover most of our daily lives at a smart environment and construct activity recognition models. Through experiments using sensor data collected by four participants in our smart home, the proposed scheme achieved average F-measure of recognizing 10 target activities starting from 84.14 % to 94.53 % by using only positioning sensor data.
Conference Paper
In this paper, we propose a method for logging micro-motion of in-home daily activity based on the skeleton recognition of the elderly in their daily life. We believe that in near future, many types of mobile robots will be spread to general household, and our idea is to let such a home robot be equipped with a 3D-depth camera such as Microsoft Kinect to enable tracking and observation of the elderly people at any location, any time, from any angle at home. There are lots of furniture and other items at home, which often make hard to set fixed-point observation, but robots are flexible to move to the best position to acquire the motion logging. The collected micro-motion data can be used for early detection of mild cognitive impairment (MCI) or depression, both of which often affect the physical body ability. Our robot moves in the vicinity of the elderly and performs a joint detection from 3D depth information. Through the experiment in the real home, we could recognize the in-home activities and micro-motions with high accuracy.
Conference Paper
Activity modelling is required to support activity recognition and further to provide activity assistance. In knowledge driven activity modelling, though the time and location is known, no inference can be made thus limiting to simple activities. In data driven approach, activity models are learnt from existing data sets using machine learning based techniques. Thus, conventional approaches for activity modelling do not work well with composite activities due to complexity and uncertainty in real scenarios. Hence, real time modelling of static and dynamic characteristics of activities is required to be done effectively. Activity recognition encompasses three important tasks, namely, activity sensing, activity modelling and activity inference. Due to vast needs from variety of application systems, video based human activity recognition is used in order to track the user activities. In order to bridge the gap between low level data and human understanding, sensor based composite activity recognition is required. In order to analyse composite user activities, hybrid approach is required to build computational models. Also, to determine the ongoing activity, the sensor data is to be processed effectively. This approach in the paper describes the sensor based real time activity modelling to monitor composite user activities for recognition.
Chapter
The "big data" explicitly produced by people through social applications, or implicitly gathered through sensors and transaction records, enables a new generation of mining and analysis tools to understand the trends and dynamics of today's interconnected society. While important steps have been made towards personal, urban, and social awareness, several research challenges still need to be addressed to fully realize the pervasive computing vision. On the one hand, the lack of standard languages and common semantic frameworks strongly limit the possibility to opportunistically acquire available context data, reason with it, and provide proactive services. On the other hand, existing techniques for identifying complex contextual situations are mainly restricted to the recognition of simple actions and activities. Most importantly, due to the unprecedented quantity of digital traces that people leave as they go about their everyday lives, formal privacy methods and trust models must be enforced to avoid the "big data" vision turning into a "big brother" nightmare. In this chapter, the authors discuss the above-mentioned research issues and highlight promising research directions.
Article
Activity recognition enables ambient assisted living applications to provide activityaware services to users in smart homes. Despite significant progress being made in activity recognition research, the focus has been on simple activity recognition leaving composite activity recognition an open problem. For instance, knowledge-driven activity recognition has recently attracted increasing attention but mainly focused on simple activities. This paper extends previous work by introducing a knowledge-driven approach to recognition of composite activities such as interleaved and concurrent activities. The approach combines the recognition of single and composite activities into a unified framework. To support composite activity modelling, it combines ontological and temporal knowledge modelling formalisms. In addition, it exploits ontological reasoning for simple activity recognition and qualitative temporal inference to support composite activity recognition. The approach is organized as a multi-agent system to enable multiple activities to be simultaneously monitored and tracked. The presented approach has been implemented in a prototype system and evaluated in a number of experiments. The experimental results have shown that average recognition accuracy for composite activities is 88.26%.
Article
Ambient intelligence (AmI) proposes pervasive information systems composed of autonomous agents embedded within the environment who, in orchestration, complement human activity in an intelligent manner. As such, it is an interesting and challenging application area for many computer science fields and approaches. A critical issue in such application scenarios is that the agents must be able to acquire, exchange, and evaluate knowledge about the environment, its users, and their activities. Knowledge populated between the agents in such systems may be contextually dependent, ambiguous, and incomplete. Conflicts may thus naturally arise, that need to be dealt with by the agents in an autonomous way. In this survey, we relate AmI to the area of knowledge representation and reasoning (KR), where conflict resolution has been studied for a long time. We take a look at a number of KR approaches that may be applied: context modelling, multi-context systems, belief revision, ontology evolution and debugging, argumentation, preferences, and paraconsistent reasoning. Our main goal is to describe the state of the art in these fields, and to draw attention of researchers to important theoretical issues and practical challenges that still need to be resolved, in order to reuse the results from KR in AmI systems or similar complex and demanding applications.
Conference Paper
In this work we have proposed to make a study of the optimal number of sensors, and their optimal placement on the human body for proper recognition of certain body positions. In our experiment we used three-axis accelerometer sensor built into the intelligent watch Chronos eZ430 product of Texas Instruments. Studying a large part of the published work addressing this issue [1-11], we placed the three sensors as follows: the first on the right hand, the second on the right thigh and the third one, on the chest. The Chronos devices transmit the data collected, wireless, to three access points connected to an Arduino ChipKIT MAX32 microcontroller module. Data are transmitted either to a PC through a serial port, or can be stored on the SD-card. WiFi Shield module provides the ability to connect and communicate through wireless IEEE 802.11 networks with portable equipment (e.g. smart phones, tablets, etc.). The data were imported into Matlab and after processing and analysis, we can draw conclusions on how the three sensors can be used in the activity recognition. To recognize the type of activity we used an artificial neural network modeled in Matlab. This network has been configured according to the number of sensors that acquired data and the number of activities that we want to recognize.
Conference Paper
Knowledge-driven activity recognition has recently attracted increasing attention but mainly focused on simple activities. This paper extends previous work to introduce a knowledge-driven approach to recognition of composite activities such as interleaved and concurrent activities. The approach combines ontological and temporal knowledge modelling formalisms for composite activity modelling. It exploits ontological reasoning for simple activity recognition and rule-based temporal inference to support composite activity recognition. The presented approach has been implemented in a prototype system and evaluated in a number of experiments. The initial experimental results have shown that average recognition accuracy for simple and composite activities is 100% and 88.26%, respectively.
Conference Paper
Activity modelling is required to support activity recognition and further to provide activity assistance for users in smart homes. Current research in knowledge-driven activity modelling has mainly focused on single activities with little attention being paid to the modelling of composite activities such as interleaved and concurrent activities. This paper presents a hybrid approach to composite activity modelling by combining ontological and temporal knowledge modelling formalisms. Ontological modelling constructors, i.e. concepts and properties for describing composite activities, have been developed and temporal modelling operators have been introduced. As such, the resulting approach is able to model both static and dynamic characteristics of activities. Several composite activity models have been created based on the proposed approach. In addition, a set of inference rules has been provided for use in composite activity recognition. A concurrent meal preparation scenario is used to illustrate both the proposed approach and associated reasoning mechanisms for composite activity recognition.
Article
Full-text available
This paper presents our system for the Open Track of the CoNLL 2008 Shared Task (Surdeanu et al., 2008) in Joint Dependency Parsing and Semantic Role Labelling. We use Markov Logic to define a joint SRL model and achieve a semantic F-score of 74.59%, the second best in the Open Track.
Article
Full-text available
In this work we present Cutting Plane Inference (CPI), a Maximum A Posteriori (MAP) inference method for Statistical Relational Learning. Framed in terms of Markov Logic and inspired by the Cutting Plane Method, it can be seen as a meta algorithm that instantiates small parts of a large and complex Markov Network and then solves these using a conventional MAP method. We evaluate CPI on two tasks, Semantic Role Labelling and Joint Entity Resolution, while plugging in two different MAP inference methods: the current method of choice for MAP inference in Markov Logic, MaxWalkSAT, and Integer Linear Programming. We observe that when used with CPI both methods are significantly faster than when used alone. In addition, CPI improves the accuracy of MaxWalkSAT and maintains the exactness of Integer Linear Programming.
Conference Paper
Full-text available
While most activity recognition systems rely on data-driven approaches, the use of knowledge-driven techniques is gaining increasing interest. Research in this field has mainly concentrated on the use of ontologies to specify the semantics of activities, and ontological reasoning to recognize them based on context information. However, at the time of writing, the experimental evaluation of these techniques is limited to computational aspects; their actual effectiveness is still unknown. As a first step to fill this gap, in this paper, we experimentally evaluate the effectiveness of the ontological approach, using an activity dataset collected in a smart-home setting. Preliminary results suggest that existing ontological techniques underperform data-driven ones, mainly because they lack support for reasoning with temporal information. Indeed, we show that, when ontological techniques are extended with even simple forms of temporal reasoning, their effectiveness is comparable to the one of a state-of-the-art technique based on Hidden Markov Models. Then, we indicate possible research directions to further improve the effectiveness of ontology-based activity recognition through temporal reasoning.
Conference Paper
Full-text available
Ontology matching is the problem of determining correspondences between concepts, properties, and individuals of different heterogeneous ontologies. With this paper we present a novel probabilistic-logical framework for ontology matching based on Markov logic. We define the syntax and semantics and provide a formalization of the ontology matching problem within the framework. The approach has several advantages over existing methods such as ease of experimentation, incoherence mitigation during the alignment process, and the incorporation of a-priori confidence values. We show empirically that the approach is efficient and more accurate than existing matchers on an established ontology alignment benchmark dataset. Copyright © 2010, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.
Conference Paper
Full-text available
In the last years, techniques for activity recognition have attracted increasing attention. Among many applications, a special interest is in the pervasive e-Health domain where automatic activity recognition is used in rehabilitation systems, chronic disease management, monitoring of the elderly, as well as in personal well being applications. Research in this field has mainly adopted techniques based on supervised learning algorithms to recognize activities based on contextual conditions (e.g., location, surrounding environment, used objects) and data retrieved from body-worn sensors. Since these systems rely on a sufficiently large amount of training data which is hard to collect, scalability with respect to the number of considered activities and contextual data is a major issue. In this paper, we propose the use of ontologies and ontological reasoning combined with statistical inferencing to address this problem. Our technique relies on the use of semantic relationships that express the feasibility of performing a given activity in a given context. The proposed technique neither increases the obtrusiveness of the statistical activity recognition system, nor introduces significant computational overhead to real-time activity recognition. The results of extensive experiments with data collected from sensors worn by a group of volunteers performing activities both indoor and outdoor show the superiority of the combined technique with respect to a solely statistical approach. To the best of our knowledge, this is the first work that systematically investigates the integration of statistical and ontological reasoning for activity recognition.
Conference Paper
Full-text available
We introduce Interleaved Hidden Markov Models for rec- ognizing multitasked activities. The model captures both inter-activity and intra-activity dynamics. Although the state space is intractably large, we describe an approximation that is both effective and efficient. This method significantly re- duces the error rate when compared with previously pro- posed methods. The algorithm is suitable for mobile plat- forms where computational resources may be limited.
Conference Paper
Full-text available
Intelligent environments are increasingly rich in ubiquitous sensing ca- pabilities that can be leveraged to know which actions a user is engaged in at any given moment in time. The ability of an intelligent environment to recognize a high-level plan of activities performed by the user in a smart home would allow to construct proactive services, such as reminding, forecasting and providing timely physical support. This article proposes an approach to human activity recognition based on temporal planning. The approach leverages on one hand the ubiquitous sensors provided by the PEIS-Home, a sensor-rich intelligent environment, and, on the other hand, the temporal representation and reasoning capabilities of OMPS, a constraint-based temporal planning and scheduling framework.
Conference Paper
Full-text available
Smart environments with ubiquitous sensing technologies are a promis- ing perspective for reliable and continuous healthcare systems with reduced costs. A primary challenge for such assisted living systems is the automated recognition of everyday activities carried out by humans in their own home. In this work, we investigate the use of Markov Logic Networks as a framework for activity recogni- tion withinintelligenthome-likeenvironmentsequippedwith pervasivelight-weight sensor technologies. In particular, we explore the ability of MLNs to capture tempo- ral relations and backgroundknowledgefor improvingthe recognitionperformance.
Conference Paper
Full-text available
Activity recognition has attracted increasing attention in recent years due to its potential to enable a number of compelling context-aware applications. As most approaches rely on supervised learning methods, obtaining substantial amounts of labeled data is often an important bottle-neck for these approaches. In this paper, we present and explore a novel method for activity recognition from sparsely labeled data. The method is based on multi-instance learning allowing to significantly reduce the required level of supervision. In particular we propose several novel extensions of multi-instance learning to support different annotation strategies. The validity of the approach is demonstrated on two public datasets for three different labeling scenarios.
Article
Full-text available
In principle, activity recognition can be exploited to great societal benefits, especially in real-life, human centric applications such as elder care and healthcare. This article focused on recognizing simple human activities. Recognizing complex activities remains a challenging and active area of research and the nature of human activities poses different challenges. Human activity understanding encompasses activity recognition and activity pattern discovery. The first focuses on accurate detection of human activities based on a predefined activity model. An activity pattern discovery researcher builds a pervasive system first and then analyzes the sensor data to discover activity patterns.
Conference Paper
Full-text available
In this paper we present results related to achieving finegrained activity recognition for context-aware computing applications. We examine the advantages and challenges of reasoning with globally unique object instances detected by an RFID glove. We present a sequence of increasingly powerful probabilistic graphical models for activity recognition. We show the advantages of adding additional complexity and conclude with a model that can reason tractably about aggregated object instances and gracefully generalizes from object instances to their classes by using abstraction smoothing. We apply these models to data collected from a morning household routine.
Conference Paper
Full-text available
This paper addresses the problem of learning and recognizing human activities of daily living (ADL), which is an important research issue in building a pervasive and smart environment. In dealing with ADL, we argue that it is beneficial to exploit both the inherent hierarchical organization of the activities and their typical duration. To this end, we introduce the switching hidden semi-markov model (S-HSMM), a two-layered extension of the hidden semi-Markov model (HSMM) for the modeling task. Activities are modeled in the S-HSMM in two ways: the bottom layer represents atomic activities and their duration using HSMMs; the top layer represents a sequence of high-level activities where each high-level activity is made of a sequence of atomic activities. We consider two methods for modeling duration: the classic explicit duration model using multinomial distribution, and the novel use of the discrete Coxian distribution. In addition, we propose an effective scheme to detect abnormality without the need for training on abnormal data. Experimental results show that the S-HSMM performs better than existing models including the flat HSMM and the hierarchical hidden Markov model in both classification and abnormality detection tasks, alleviating the need for presegmented training data. Furthermore, our discrete Coxian duration model yields better computation time and generalization error than the classic explicit duration model.
Article
Full-text available
We present a framework for statistical machine translation of natural languages based on direct maximum entropy models, which contains the widely used source -channel approach as a special case. All knowledge sources are treated as feature functions, which depend on the source language sentence, the target language sentence and possible hidden variables.
Article
Ontology matching is the problem of determining correspondences between concepts, properties, and individuals of different heterogeneous ontologies. With this paper we present a novel probabilistic-logical framework for ontology matching based on Markov logic. We define the syntax and semantics and provide a formalization of the ontology matching problem within the framework. The approach has several advantages over existing methods such as ease of experimentation, incoherence mitigation during the alignment process, and the incorporation of a-priori confidence values. We show empirically that the approach is efficient and more accurate than existing matchers on an established ontology alignment benchmark dataset.
Conference Paper
In this work we present Cutting Plane Inference (CPI), a Maximum A Posteriori (MAP) inference method for Statistical Relational Learning. Framed in terms of Markov Logic and inspired by the Cutting Plane Method, it can be seen as a meta algorithm that instantiates small parts of a large and complex Markov Network and then solves these using a conventional MAP method. We evaluate CPI on two tasks, Semantic Role Labelling and Joint Entity Resolution, while plugging in two different MAP inference methods: the current method of choice for MAP inference in Markov Logic, MaxWalkSAT, and Integer Linear Programming. We observe that when used with CPI both methods are significantly faster than when used alone. In addition, CPI improves the accuracy of MaxWalkSAT and maintains the exactness of Integer Linear Programming.
Chapter
Advanced statistical modeling and knowledge representation techniques for a newly emerging area of machine learning and probabilistic reasoning; includes introductory material, tutorials for different proposed approaches, and applications. Handling inherent uncertainty and exploiting compositional structure are fundamental to understanding and designing large-scale systems. Statistical relational learning builds on ideas from probability theory and statistics to address uncertainty while incorporating tools from logic, databases and programming languages to represent structure. In Introduction to Statistical Relational Learning, leading researchers in this emerging area of machine learning describe current formalisms, models, and algorithms that enable effective and robust reasoning about richly structured systems and data. The early chapters provide tutorials for material used in later chapters, offering introductions to representation, inference and learning in graphical models, and logic. The book then describes object-oriented approaches, including probabilistic relational models, relational Markov networks, and probabilistic entity-relationship models as well as logic-based formalisms including Bayesian logic programs, Markov logic, and stochastic logic programs. Later chapters discuss such topics as probabilistic models with unknown objects, relational dependency networks, reinforcement learning in relational domains, and information extraction. By presenting a variety of approaches, the book highlights commonalities and clarifies important differences among proposed approaches and, along the way, identifies important representational and algorithmic issues. Numerous applications are provided throughout.
Article
In recent years there has been an increased interest in the modeling and recognition of human activities involving highly structured and semantically rich behavior such as dance, aerobics, and sign language. A novel approach for automatically acquiring stochastic models of the high-level structure of an activity without the assumption of any prior knowledge is presented. The process involves temporal segmentation into plausible atomic behavior components and the use of variable-length Markov models for the efficient representation of behaviors. Experimental results that demonstrate the synthesis of realistic sample behaviors and the performance of models for long-term temporal prediction are presented.
Article
Human activity recognition is an important task which has many potential applications. In recent years, researchers from pervasive computing are interested in deploying on-body sensors to collect observations and applying machine learning techniques to model and recognize activities. Supervised machine learning techniques typically require an appropriate training process in which training data need to be labeled manually. In this paper, we propose an unsupervised approach based on object-use fingerprints to recognize activities without human labeling. We show how to build our activity models based on object-use fingerprints, which are sets of contrast patterns describing significant differences of object use between any two activity classes. We then propose a fingerprint-based algorithm to recognize activities. We also propose two heuristic algorithms based on object relevance to segment a trace and detect the boundary of any two adjacent activities. We develop a wearable RFID system and conduct a real-world trace collection done by seven volunteers in a smart home over a period of 2 weeks. We conduct comprehensive experimental evaluations and comparison study. The results show that our recognition algorithm achieves a precision of 91.4% and a recall 92.8%, and the segmentation algorithm achieves an accuracy of 93.1% on the dataset we collected.
Article
The paper introduces k-bounded MAP inference, a parameterization of MAP inference in Markov logic networks. k-Bounded MAP states are MAP states with at most k active ground atoms of hidden (non-evidence) predicates. We present a novel delayed column generation algorithm and provide empirical evidence that the algorithm efficiently computes k-bounded MAP states for meaningful real-world graph matching problems. The underlying idea is that, instead of solving one large optimization problem, it is often more efficient to tackle several small ones.
Conference Paper
In artificial intelligence and pervasive computing research, in- ferring users' high-level goals from activity sequences is an important task. A major challenge in goal recognition is that users often pursue several high-level goals in a concurrent and interleaving manner, where the pursuit of goals may spread over different parts of an activity sequence and may be pur- sued in parallel. Existing approaches to recognizing multi- ple goals often formulate this problem either as a single-goal recognition problem or in a deterministic way, ignoring un- certainty. In this paper, we propose CIGAR (Concurrent and Interleaving Goal and Activity Recognition) - a novel and simple two-level probabilistic framework for multiple-goal recognition where we can recognize both concurrent and in- terleaving goals. We use skip-chain conditional random fields (SCCRF) for modeling interleaving goals and we model con- current goals by adjusting inferred probabilities through a correlation graph, which is a major advantage in that we are able to reason about goal interactions explicitly through the correlation graph. The two-level framework also avoids the high training complexity when modeling concurrency and in- terleaving together in a unified CRF model. Experimental re- sults show that our method can effectively improve recogni- tion accuracies on several real-world datasets collected from various wireless and sensor networks.
Conference Paper
Recognizing and understanding the activities of people from sensor readings is an important task in ubiquitous computing. Activity recognition is also a particularly difficult task because of the inherent uncertainty and complexity of the data collected by the sensors. Many researchers have tackled this problem in an overly simplistic setting by assuming that users often carry out single activities one at a time or multiple activities consecutively, one after another. However, so far there has been no formal exploration on the degree in which humans perform concurrent or interleaving activities, and no thorough study on how to detect multiple goals in a real world scenario. In this article, we ask the fundamental questions of whether users often carry out multiple concurrent and interleaving activities or single activities in their daily life, and if so, whether such complex behavior can be detected accurately using sensors. We define several classes of complexity levels under a goal taxonomy that describe different granularities of activities, and relate the recognition accuracy with different complexity levels or granularities. We present a theoretical framework for recognizing multiple concurrent and interleaving activities, and evaluate the framework in several real-world ubiquitous computing environments.
Conference Paper
We explore a dense sensing approach that uses RFID sensor network technology to recognize human activities. In our setting, everyday objects are instrumented with UHF RFID tags called WISPs that are equipped with accelerometers. RFID readersdetect when the objects are used by examining this sensor data, and daily activities are then inferred from the traces of object use via a Hidden Markov Model. In a study of 10 participants performing 14 activities in a model apartment, our approach yielded recognition rates with pre- cision and recall both in the 90% range. This compares well to recognition with a more intrusive short-range RFID bracelet that detects objects in the proximity of the user; this approach saw roughly 95% precision and 60% recall in the same study. We conclude that RFID sensor networks are a promising approach for indoor activity monitoring.
Conference Paper
We consider large margin estimation in a broad range of prediction models where inference involves solving combinatorial optimization problems, for example, weighted graph-cuts or matchings. Our goal is to learn parameters such that inference using the model reproduces correct answers on the training data. Our method relies on the expressive power of convex optimization problems to compactly capture inference or solution optimality in structured prediction models. Directly embedding this structure within the learning formulation produces concise convex problems for efficient estimation of very complex and diverse models. We describe experimental results on a matching task, disulfide connectivity prediction, showing significant improvements over state-of-the-art methods.
Conference Paper
Inference in Conditional Random Fields and Hidden Markov Models is done using the Viterbi algorithm, an efficient dynamic programming algorithm. In many cases, general (non-local and non-sequential) constraints may exist over the output sequence, but cannot be incorporated and exploited in a natural way by this inference procedure. This paper proposes a novel inference procedure based on integer linear programming (ILP) and extends CRF models to naturally and efficiently support general constraint structures. For sequential constraints, this procedure reduces to simple linear programming as the inference process. Experimental evidence is supplied in the context of an important NLP problem, semantic role labeling.
Conference Paper
We address the problem of visual event recognition in surveillance where noise and missing observations are serious problems. Common sense domain knowledge is exploited to overcome them. The knowledge is represented as first-order logic production rules with associated weights to indicate their confidence. These rules are used in combination with a relaxed deduction algorithm to construct a network of grounded atoms, the Markov Logic Network. The network is used to perform probabilistic inference for input queries about events of interest. The system's performance is demonstrated on a number of videos from a parking lot domain that contains complex interactions of people and vehicles.
Conference Paper
In this paper, we introduce a first-order probabilistic model that combines multiple cues to classify human activities from video data accurately and robustly. Our system works in a realistic office setting with background clutter, natural illumination, different people, and par- tial occlusion. The model we present is compact, requires only fifteen sen- tences of first-order logic grouped as a Dynamic Markov Logic Network (DMLNs) to implement the probabilistic model and leverages existing state-of-the-art work in pose detection and object recognition.
Conference Paper
A majority of the approaches to activity recognition in sensor environments are either based on manually constructed rules for recognizing activities or lack the ability to incorporate complex temporal dependencies. Furthermore, in many cases, the rather unrealistic assumption is made that the subject carries out only one activity at a time. In this paper, we describe the use of Markov logic as a declarative framework for recognizing interleaved and concurrent activities incorporating both input from pervasive light-weight sensor technology and common-sense background knowledge. In particular, we assess its ability to learn statistical-temporal models from training data and to combine these models with background knowledge to improve the overall recognition accuracy. To this end, we propose two Markov logic formulations for inferring the foreground activity as well as each activities' start and end times. We evaluate the approach on an established dataset. where it outperforms state-of-the-art algorithms for activity recognition.
Article
We propose a simple approach to combining first-order logic and probabilistic graphical models in a single representation. A Markov logic network (MLN) is a first-order knowledge base with a weight attached to each formula (or clause). Together with a set of constants representing objects in the domain, it specifies a ground Markov network containing one feature for each possible grounding of a first-order formula in the KB, with the corresponding weight. Inference in MLNs is performed by MCMC over the minimal subset of the ground network required for answering the query. Weights are efficiently learned from relational databases by iteratively optimizing a pseudo-likelihood measure. Optionally, additional clauses are learned using inductive logic programming techniques. Experiments with a real-world database and knowledge base in a university domain illustrate the promise of this approach.
Conference Paper
To recognize user intention proactively and do a suitable action or service are one of important issues in intelligent robot. Even when a user acts the same behavior, its intention may be different according to the user's context. It means that user intention recognition involves the uncertainties, and by minimizing the uncertainties can improve the accuracy of the user intention recognition. This paper suggests a novel ontology-based approach for user intention recognition. We propose a method of minimizing the uncertainties that are the main obstacles against the precise recognition of user intention. This approach creates an ontology for user intention, makes a hierarchy and relationship among user intentions, and precisely recognizes user intention by using the gathered sensor data such as temperature, humidity, vision, and auditory. We developed a simulator that evaluates the performance of robot proactive planning mechanism.
  • N Creignou
  • S Khanna
  • M Sudan
N. Creignou, S. Khanna, M. Sudan, Complexity Classifications of Boolean Constraint Satisfaction Problems, SIAM, 2001.
Monitoring domestic activities with temporal constraints and components
  • M Cirillo
  • F Lazellotto
  • F Pecora
  • A Saffiotti