No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Transforming knowledge capture in healthcare: Opportunistic and Context-aware affect Sensing on Smartphones
Abstract
Opportunistic affect sensing offers unprecedented potential for capturing spontaneous affect, eliminating biases inherent in the controlled setting. Facial expression and voice are two major affective displays, however most affect sensing systems on smartphone avoid them due to extensive power requirements. Encouragingly, due to the recent advent of low-power DSP (Digital Signal Processing) co-processor and GPU (Graphics Processing Unit) technology, audio and video sensing are becoming more feasible on smartphone. To utilize opportunistically captured facial expression and voice, gathering contextual information about the dynamic audio-visual stimuli is also important. This paper discusses recent advances of affect sensing on the smartphone and identifies the key barriers and potential solutions for implementing opportunistic and context-aware affect sensing on smartphone platforms. In addition to exploring the technical challenges (privacy, battery life and robust algorithms), the challenges of recruiting and retention of mental health patients have also been considered; as experimentation with mental health patients is difficult but crucial to showcase the importance/effectiveness of the smartphone centred affect sensing technology
... Essentially, a large body of research has focused on responding from different perspectives regarding how the natural and built environment affect people's health, i.e., to understand the degree and ways in which the features and manifestations of the environment influence public health and aspects of our lifestyle in cities [15,16]. Computer scientists are recently gaining importance in the field as the increasing democratization and deployment of mobile computing, mobile devices and sensing technologies [17] provide new opportunities to deliver mobile mental health care [18][19][20] thus improving quality of life [21]. Various authors [20,22,23] have suggested that interventions with mobile phones designed to support evidence-based treatments could even reduce barriers to access and increase participation in treatment. ...
... Interface BinDhim et al. [48], Rana et al. [17], Haggerty [50], and Bohnsack et al. [51]. Gamification Seaborn and Fels [92], Nacke and Deterding [93], Bartle [94], and Tondello et al. [95]. ...
This work identifies major areas of knowledge and proposes a set of relevant dimensions by area that must be taken into account in the design and delivery of context-aware mobile applications for mental health interventions. We argue that much of the related research has focused only on a few dimensions, paying little or no attention to others and, most importantly, to potential relationships between them. Our belief is that the improvement of the effectiveness of mobile interventions to support mental health necessarily implies that developers and therapists comprehensively consider the interaction between the proposed dimensions. Taking as a starting point the three areas of knowledge (Technology, Context, and Mental Health), we re-examine each area to identify relevant dimensions, discuss the relationships between them and finally draw a series of considerations. The resulting considerations can help therapists and developers to devise, design, and generate custom mobile applications in a way that increases the motivation and engagement of patients and, therefore, the effectiveness of psychological treatments.
... Sensors 2020, 20, 6367 2 of 43 account selected emotional states, e.g., happiness [3], while some focus on selected input modalities, e.g., in [4], where the authors present a general point of view but also focus on the difficulties faced while sensing affect from facial expression and voice. An extensive and valuable review has been presented in [5], where the full spectrum of research studies published between 2010 and 2017 has been presented. ...
In recent years, emotion recognition algorithms have achieved high efficiency, allowing the development of various affective and affect-aware applications. This advancement has taken place mainly in the environment of personal computers offering the appropriate hardware and sufficient power to process complex data from video, audio, and other channels. However, the increase in computing and communication capabilities of smartphones, the variety of their built-in sensors, as well as the availability of cloud computing services have made them an environment in which the task of recognising emotions can be performed at least as effectively. This is possible and particularly important due to the fact that smartphones and other mobile devices have become the main computer devices used by most people. This article provides a systematic overview of publications from the last 10 years related to emotion recognition methods using smartphone sensors. The characteristics of the most important sensors in this respect are presented, and the methods applied to extract informative features on the basis of data read from these input channels. Then, various machine learning approaches implemented to recognise emotional states are described.
... Smartphones, which embed all these sensors, can be considered a type of wearable due to their pervasiveness. Furthermore, the low adoption barrier on healthcare applications [28] through application markets such as Google Play or AppStore makes them the best option to target the mass market. Some of them are focused on fall detection [29,30], but normally do not cover both ADL and falls [31], so a classification system must be designed to consider them. ...
Currently, the lifestyle of elderly people is regularly monitored in order to establish guidelines for rehabilitation processes or ensure the welfare of this segment of the population. In this sense, activity recognition is essential to detect an objective set of behaviours throughout the day. This paper describes an accurate, comfortable and efficient system, which monitors the physical activity carried out by the user. An extension to an awarded activity recognition system that participated in the EvAAL 2012 and EvAAL 2013 competitions is presented. This approach uses data retrieved from accelerometer sensors to generate discrete variables and it is tested in a non-controlled environment. In order to achieve the goal, the core of the algorithm Ameva is used to develop an innovative selection, discretization and classification technique for activity recognition. Moreover, with the purpose of reducing the cost and increase user acceptance and usability, the entire system uses only a smartphone to recover all the information required.
At the present time, 15% of the growing world population is estimated to have disabilities and special needs. Disabilities can seriously limit participation in regular life activities, such as controlling home facilities, using transportation services, joining social events, accessing educational contents, to name but a few. With the advancement in ubiquitous and pervasive computing, context-aware systems (CAS) are gaining much attention and demonstrating a stronger association with applications for people with disability. Modern CAS tend to minimize user interactions with the system and provide seamless services, automated awareness, and ambient intelligence and monitoring. CAS for people with disability can detect the surrounding environment, identify an appropriate user interface, interact, and service the user depending on the situation. Nevertheless, a large number of investigations on CAS for people with disability are presented in the literature, limited systems are practically available in the market. In this paper, we survey the literature to thoroughly analyze, evaluate, and critique state-of-the-art research in accessible CAS. Systems are classified according to the type of disability; besides, many interaction models are examined and strategies for making CAS accessible are identified. The investigation confirms the need for frameworks that enable improving security aspects, better exploiting modern hardware systems, performing reliable verification, and further supporting system customization and adaptation.
In large-scale context-aware applications, a central design concern is capturing, managing and acting upon location and context data. The ability to understand the collected data and define meaningful contextual events, based on one or more incoming (contextual) data streams, both for a single and multiple users, is hereby critical for applications to exhibit location- and context-aware behaviour. In this article, we describe a context-aware, data-intensive metrics platform —focusing primarily on its geospatial support—that allows exactly this: to define and execute metrics, which capture meaningful spatio-temporal and contextual events relevant for the application realm. The platform (1) supports metrics definition and execution; (2) provides facilities for real-time, in-application actions upon metrics execution results; (3) allows post-hoc analysis and visualisation of collected data and results. It hereby offers contextual and geospatial data management and analytics as a service, and allow context-aware application developers to focus on their core application logic. We explain the core platform and its ecosystem of supporting applications and tools, elaborate the most important conceptual features, and discuss implementation realised through a distributed, micro-service based cloud architecture. Finally, we highlight possible application fields, and present a real-world case study in the realm of psychological health.
The spontaneous recognition of emotional states and personality traits of individuals has been puzzling researchers for years whereas pertinent studies demonstrating the progress in the field, despite their diversity, are still encouraging. This work surveys the most well-known research studies and the state-of-the-art on affect recognition domain based on smartphone acquired data, namely smartphone embedded sensors and smartphone usage. Inevitably, supplementary modalities employed in many eminent studies are also reported here for the sake of completeness. Nevertheless, the intention of the survey is threefold; firstly to document all the to-date relevant literature on affect recognition through smartphone modalities, secondly to argue for the full potential of smartphone use in the inference of affect, and thirdly to demonstrate the current research trends towards mobile affective computing.
The digital revolution in medicine not only offers exciting new directions for the treatment of mental illness, but also presents challenges to patient privacy and security. Changes in medicine are part of the complex digital economy based on creating value from analysis of behavioral data acquired by the tracking of daily digital activities. Without an understanding of the digital economy, recommending the use of technology to patients with mental illness can inadvertently lead to harm. Behavioral data are sold in the secondary data market, combined with other data from many sources, and used in algorithms that automatically classify people. These classifications are used in commerce and government, may be discriminatory, and result in non-medical harm to patients with mental illness. There is also potential for medical harm related to poor quality online information, self-diagnosis and self-treatment, passive monitoring, and the use of unvalidated smartphone apps. The goal of this paper is to increase awareness and foster discussion of the new ethical issues. To maximize the potential of technology to help patients with mental illness, physicians need education about the digital economy, and patients need help understanding the appropriate use and limitations of online websites and smartphone apps.
In ambient assisted living, the home care treatment of elderly people is faced with the unreliability of correctly identifying all activities over the entire day. As the self-assessment is quite weak, the sensor-based tracking is researched intensively. In this research, we propose a minimal set of low cost and non-intrusive sensors in order to identify a set of prescribed patient activities. Prototypes of sensor systems are built and tested in real-life cases. The retrieved big data about patient activities are stored and processed comparing different methods for data analytics. Finally, sensor systems, although using low-cost sensors but providing high accuracy and reliability, are proposed and discussed to be used in home care treatment.
Major Depressive Disorder (MDD) is the most common mental health disorder and remains a leading cause of disability and lost productivity with huge costs for society. MDD has high rates of relapse and recurrence, and it has strong correlations with feelings of low social support and disrupted sleep. However, MDD is also commonly misdiagnosed by primary care providers, which leads to delayed treatment and unnecessary suffering. Changes in technology now make it possible to cheaply and effectively monitor social and sleep behaviors, offering the potential of early detection of the onset of MDD. We report on the design of Big Black Dog, a smartphone-based system for gathering data about social and sleep behaviors. We also report on the results of a pilot evaluation to understand the feasibility of gathering and using data from smartphones for inferring the onset of depression.
Much of the stress and strain of student life remains hidden. The StudentLife continuous sensing app assesses the day-to- day and week-by-week impact of workload on stress, sleep, activity, mood, sociability, mental well-being and academic performance of a single class of 48 students across a 10 week term at Dartmouth College using Android phones. Results from the StudentLife study show a number of significant cor- relations between the automatic objective sensor data from smartphones and mental health and educational outcomes of the student body. We also identify a Dartmouth term lifecycle in the data that shows students start the term with high pos- itive affect and conversation levels, low stress, and healthy sleep and daily activity patterns. As the term progresses and the workload increases, stress appreciably rises while posi- tive affect, sleep, conversation and activity drops off. The StudentLife dataset is publicly available on the web.
Driving is known to be a daily stressor. Measurement of driver's stress in real-time can enable better stress management by increasing self-awareness. Recent advances in sensing technology has made it feasible to continuously assess driver's stress in real-time, but it requires equipping the driver with these sensors and/or instrumenting the car. In this paper, we present "GStress", a model to estimate driver's stress using only smartphone GPS traces. The GStress model is developed and evaluated from data collected in a mobile health user study where 10 participants wore physiological sensors for 7 days ( for an average of 10.45 hours/day) in their natural environment. Each participant engaged in 10 or more driving episodes, resulting in a total of 37 hours of driving data. We find that major driving events such as stops, turns, and braking increase stress of the driver. We quantify their impact on stress and thus construct our GStress model by training a Generalized Linear Mixed Model (GLMM) on our data. We evaluate the applicability of GStress in predicting stress from GPS traces, and obtain a correlation of 0.72. By obviating any burden on the driver or the car, we believe, GStress can make driver's stress assessment ubiquitous.
Active Appearance Model (AAM) [1, 4] has been favored among computer vision researchers, since it is useful for tracking human face or diverse objects. Moreover, recent development of the inverse composition method allows us to relieve computational burden in tracking human face with the real-time basis. The progress of fast algorithm development makes it possible that we may able to track a human face within video image feeding from a smartphone. However, such mobile device gives us some limitation in computing power, but also illumination environment around it is not favorable to vision algorithm designers. This paper presents new method by which such illumination variation occurred in the mobile environment can be overcome. We have found that the Difference of Gaussian (DoG) kernel preceded the AAM stage is very effective in tracking human face including important facial features such as eyes, nose and mouth, despite a strong directional illumination. Performance of the proposed system is evaluated for diverse illumination conditions, and result suggests that the DoG kernel, that has been identified as biologically plausible, can plays an important role against such odd illumination environment. This algorithm has been implemented on a smartphone for the purpose of human facial expression recognition and it works well with a camera as an application program.
A key challenge for mobile health is to develop new technology that can assist individuals in maintaining a healthy lifestyle by keeping track of their everyday behaviors. Smartphones embedded with a wide variety of sensors are enabling a new generation of personal health applications that can actively monitor, model and promote wellbeing. Automated wellbeing tracking systems available so far have focused on physical fitness and sleep and often require external non-phone based sensors. In this work, we take a step towards a more comprehensive smartphone based system that can track activities that impact physical, social, and mental wellbeing namely, sleep, physical activity, and social interactions and provides intelligent feedback to promote better health. We present the design, implementation and evaluation of BeWell, an automated wellbeing app for the Android smartphones and demonstrate its feasibility in monitoring multi-dimensional wellbeing. By providing a more complete picture of health, BeWell has the potential to empower individuals to improve their overall wellbeing and identify any early signs of decline.
The rapidly growing adoption of sensor-enabled smartphones has greatly fueled the proliferation of applications that use phone sensors to monitor user behavior. A central sensor among these is the microphone which enables, for instance, the detection of valence in speech, or the identification of speakers. Deploying multiple of these applications on a mobile device to continuously monitor the audio environment allows for the acquisition of a diverse range of sound-related contextual inferences. However, the cumulative processing burden critically impacts the phone battery.
To address this problem, we propose DSP.Ear -- an integrated sensing system that takes advantage of the latest low-power DSP co-processor technology in commodity mobile devices to enable the continuous and simultaneous operation of multiple established algorithms that perform complex audio inferences. The system extracts emotions from voice, estimates the number of people in a room, identifies the speakers, and detects commonly found ambient sounds, while critically incurring little overhead to the device battery. This is achieved through a series of pipeline optimizations that allow the computation to remain largely on the DSP. Through detailed evaluation of our prototype implementation we show that, by exploiting a smartphone's co-processor, DSP.Ear achieves a 3 to 7 times increase in the battery lifetime compared to a solution that uses only the phone's main processor. In addition, DSP.Ear is 2 to 3 times more power efficient than a naïve DSP solution without optimizations. We further analyze a large-scale dataset from 1320 Android users to show that in about 80-90% of the daily usage instances DSP.Ear is able to sustain a full day of operation (even in the presence of other smartphone workloads) with a single battery charge.
Today's health care is difficult to imagine without the possibility to objectively measure various physiological parameters related to patients symptoms (from temperature through blood pressure to complex tomographic procedures). Psychiatric care remains a notable exception that heavily relies on patient interviews and self assessment. This is due to the fact that mental illnesses manifest themselves mainly in the way patients behave throughout their daily life and, until recently there were no "behavior measurement devices". This is now changing with the progress in wearable activity recognition and sensor enabled smartphones. In this article we introduce a system, which, based on smartphone-sensing is able to recognize depressive and manic states and detect state changes of patients suffering from bipolar disorder. Drawing upon a real-life dataset of 10 patients, recorded over a time-period of 12 weeks (in total over 800 days of data tracing 17 state changes) by 4 different sensing modalities we could extract features corresponding to all disease-relevant aspects in behavior. Using these features we gain recognition accuracies of 76% by fusing all sensor modalities and state change detection precision and recall of over 97%. This article furthermore outlines the applicability of this system in the physician-patient relations in order to facilitate the life and treatment of bipolar patients.
Stress can have long term adverse effects on individuals' physical and mental well-being. Changes in the speech production process is one of many physiological changes that happen during stress. Microphones, embedded in mobile phones and carried ubiquitously by people, provide the opportunity to continuously and non-invasively monitor stress in real-life situations. We propose StressSense for unobtrusively recognizing stress from human voice using smartphones. We investigate methods for adapting a one-size-fits-all stress model to individual speakers and scenarios. We demonstrate that the StressSense classifier can robustly identify stress across multiple individuals in diverse acoustic environments: using model adaptation StressSense achieves 81% and 76% accuracy for indoor and outdoor environments, respectively. We show that StressSense can be implemented on commodity Android phones and run in real-time. To the best of our knowledge, StressSense represents the first system to consider voice based stress detection and model adaptation in diverse real-life conversational situations using smartphones.
In this paper we provide the first evidence that daily happiness of individuals can be automatically recognized using an extensive set of indicators obtained from the mobile phone usage data (call log, sms and Bluetooth proximity data) and "background noise" indicators coming from the weather factor and personality traits. Our final machine learning model, based on the Random Forest classifier, obtains an accuracy score of 80.81% for a 3-class daily happiness recognition problem. Moreover, we identify and discuss the indicators, which have strong predictive power in the source and the feature spaces, discuss different approaches, machine learning models and provide an insight for future research.
The paper describes the development of a mobile solution based on smartphones and sensors for the early recognition of stress. The solution is based on real-time capture and analysis of vital data such as heart rate variability as well as activity and contextual data such as location and time of day. Individual recognition patterns for stress are derived from combining vital and contextual data by using subjective stress assessments via mood maps as additional input during an initial learning phase. The reliability of stress alerts and therapeutic impact will be tested in a clinic specialised on the treatment of alcoholics since stress tends to cause craving and therefore trigger relapses
There are many potential pitfalls in the identification and enlistment of suitable candidates for psychiatric research. The challenges of recruitment are highlighted, detailing impact of study design, characteristics of participants, including demographics and personal preferences, investigator characteristics and collaboration with clinicians. Techniques used in recruitment are discussed, including financial incentives, assertive tracking and communication methods. Ethical issues, methods of data collection, and control participants are also considered. Key issues are: early consideration of the impact of study design on the recruitment process; the participant's perspective; close collaboration with colleagues; the investigator's good interpersonal, communication and organisational skills; and feedback to collaborators, associated clinicians and participants.
Work should be a source of health, pride, and happiness, in the sense of enhancing motivation and strengthening personal development. Healthy and motivated employees perform better and remain loyal to the company for a longer time. But, when the person constantly experiences high workload over a longer period of time and is not able to recover, then work may lead to prolonged negative effects and might cause serious illnesses like chronic stress disease. In this work, we present a solution for assessing the stress experience of people, using features derived from smartphones and wearable chest belts. In particular, we use information from audio, physical activity, and communication data collected during workday and heart rate variability data collected at night during sleep to build multinomial logistic regression models. We evaluate our system in a real work environment and in daily-routine scenarios of 35 employees over a period of 4 months and apply the leave-one-day-out cross-validation method for each user individually to estimate the prediction accuracy. Using only smartphone features, we get an accuracy of 55 %, and using only heart rate variability features, we get an accuracy of 59 %. The combination of all features leads to a rate of 61 % for a three-stress level (low, moderate, and high perceived stress) classification problem.
The increasing number of people playing games on touch-screen mobile phones raises the question of whether touch behaviors reflect players’ emotional states. This prospect would not only be a valuable evaluation indicator for game designers, but also for real-time personalization of the game experience. Psychology studies on acted touch behavior show the existence of discriminative affective profiles. In this article, finger-stroke features during gameplay on an iPod were extracted and their discriminative power analyzed. Machine learning algorithms were used to build systems for automatically discriminating between four emotional states (Excited, Relaxed, Frustrated, Bored), two levels of arousal and two levels of valence. Accuracy reached between 69% and 77% for the four emotional states, and higher results (~89%) were obtained for discriminating between two levels of arousal and two levels of valence. We conclude by discussing the factors relevant to the generalization of the results to applications other than games.
Sparse approximation has now become a buzzword for classification in numerous research domains. We propose a distributed sparse approximation method based on l1 minimization for frog sound classification, which is tailored to the resource constrained wireless sensor networks. Our pilot study demonstrates that l1 minimization can run on wireless sensor nodes producing satisfactory classification accuracy.
We report a first-of-its-kind smartphone software system, MoodScope, which infers the mood of its user based on how the smartphone is used. Compared to smartphone sensors that measure acceleration, light, and other physical properties, MoodScope is a "sensor" that measures the mental state of the user and provides mood as an important input to context-aware computing. We run a formative statistical mood study with smartphone-logged data collected from 32 participants over two months. Through the study, we find that by analyzing communication histo-ry and application usage patterns, we can statistically infer a user's daily mood average with an initial accuracy of 66%, which gradu-ally improves to an accuracy of 93% after a two-month personal-ized training period. Motivated by these results, we build a service, MoodScope, which analyzes usage history to act as a sensor of the user's mood. We provide a MoodScope API for developers to use our system to create mood-enabled applications. We further create and deploy a mood-sharing social application.
The combination of smart phones, wearable sensor devices and social media offer new ways of monitoring and promoting mental and physical wellbeing. In this contribution, we describe recent developments in the field of mobile healthcare (or mHealth), by focusing in particular on mobile mental health applications. First, we examine the potential benefits associated with this approach, providing examples from existing projects. Next, we identify and explain possible differences in focus between mobile mental health and mobile wellbeing applications. Finally, we discuss some open challenges associated with the implementation of this vision, ranging from the lack of evidence-based validation to privacy, security and ethical concerns.
Factor-analytic evidence has led most psychologists to describe affect as a set of dimensions, such as displeasure, distress, depression, excitement, and so on, with each dimension varying independently of the others. However, there is other evidence that rather than being independent, these affective dimensions are interrelated in a highly systematic fashion. The evidence suggests that these interrelationships can be represented by a spatial model in which affective concepts fall in a circle in the following order: pleasure (0), excitement (45), arousal (90), distress (135), displeasure (180), depression (225), sleepiness (270), and relaxation (315). This model was offered both as a way psychologists can represent the structure of affective experience, as assessed through self-report, and as a representation of the cognitive structure that laymen utilize in conceptualizing affect. Supportive evidence was obtained by scaling 28 emotion-denoting adjectives in 4 different ways: R. T. Ross's (1938) technique for a circular ordering of variables, a multidimensional scaling procedure based on perceived similarity among the terms, a unidimensional scaling on hypothesized pleasure–displeasure and degree-of-arousal dimensions, and a principal-components analysis of 343 Ss' self-reports of their current affective states. (70 ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Ubiquitous computing technologies offer exciting new possibilities for monitoring and analyzing user’s experience in real time. In this paper, we describe the design and development of Psychlog, a mobile phone platform designed to collect users’ psychological, physiological, and activity information for mental health research. The tool allows administering self-report questionnaires at specific times or randomly within a day. The system also permits to collect heart rate and activity information from a wireless electrocardiogram equipped with a three-axial accelerometer. By combining self-reports with heart rate and activity data, the application makes it possible to investigate the relationship between psychological, physiological, and behavioral variables, as well as to monitor their fluctuations over time. The software runs on Windows mobile operative system and is available as open source (http://sourceforge.net/projects/psychlog/).
The vast majority of advances in sensor network research over the last five years have focused on the development of a series of small-scale (100s of nodes) testbeds and special-ized applications (e.g., environmental monitoring, etc.) that are built on low-powered sensor devices that self-organize to form application-specific multihop wireless networks. We believe that sensor networks have reached an important cross-roads in their development. The question we address in this paper is how to propel sensor networks from their small-scale application-specific network origins, into the commer-cial mainstream of people's every day lives; the challenge being: how do we develop large-scale general-purpose sen-sor networks for the general public (e.g., consumers) capable of supporting a wide variety of applications in urban set-tings (e.g., enterprises, hospitals, recreational areas, towns, cities, and the metropolis). We propose MetroSense, a new people-centric paradigm for urban sensing at the edge of the Internet, at very large scale. We discuss a number of challenges, interactions and characteristics in urban sens-ing applications, and then present the MetroSense architec-ture which is based fundamentally on three design princi-ples: network symbiosis, asymmetric design, and localized interaction. The ability of MetroSense to scale to very large areas is based on the use of an opportunistic sensor network-ing approach. Opportunistic sensor networking leverages mobility-enabled interactions and provides coordination be-tween people-centric mobile sensors, static sensors and edge wireless access nodes in support of opportunistic sensing, op-portunistic tasking, and opportunistic data collection. We discuss architectural challenges including providing sensing coverage with sparse mobile sensors, how to hand off roles and responsibilities between sensors, improving network per-formance and connectivity using adaptive multihop, and im-portantly, providing security and privacy for people-centric sensors and data.
Mental health has long been a neglected problem in global healthcare. The social and economic impacts of conditions affecting the mind are still underestimated. However, in recent years it is becoming more apparent that mental disorders are a growing global concern and there is a necessity of developing novel services and researching effective means of providing interventions to sufferers. Such novel services could include technology-based solutions already used in other healthcare applications but are yet to make their way into standard psychiatric practice.
This manuscript proposes a system where sensors are utilised to devise an "early warning" system for patients with bipolar disorder. The system, containing wearable and environmental sensors, would collect behavioural data independent from the patient's self-report. To test the feasibility of the concept, a prototype system was devised, which was followed by trials including four healthy volunteers as well as a bipolar patient.
The sensors utilised in the study yielded behavioural data which may be of significant use in detecting early effects of a bipolar episode. Basic processing performed on particular data inputs provided information about activity patterns in areas, which are usually strongly influenced by the course of Bipolar Disorder.
The manuscript discusses the basic usage issues and other barriers which are to be tackled before technology-based approaches to mental care can be successfully rolled out and their true value appraised.
As GPU becomes an integrated component in handheld devices like smartphones, we have been investigating the opportunities and limitations of utilizing the ultra-low-power GPU in a mobile platform as a general-purpose accelerator, similar to its role in desktop and server platforms. The special focus of our investigation has been on mobile GPU's role for energy-optimized real-time applications running on battery-powered handheld devices. In this work, we use face recognition as an application driver for our study. Our implementations on a smartphone reveals that, utilizing the mobile GPU as a co-processor can achieve significant speedup in performance as well as substantial reduction in total energy consumption, in comparison with a mobile-CPU-only implementation on the same platform.
Mobile phones or smartphones are rapidly becoming the central computer and communication device in people's lives. Application delivery channels such as the Apple AppStore are transforming mobile phones into App Phones, capable of downloading a myriad of applications in an instant. Importantly, today's smartphones are programmable and come with a growing set of cheap powerful embedded sensors, such as an accelerometer, digital compass, gyroscope, GPS, microphone, and camera, which are enabling the emergence of personal, group, and communityscale sensing applications. We believe that sensor-equipped mobile phones will revolutionize many sectors of our economy, including business, healthcare, social networks, environmental monitoring, and transportation. In this article we survey existing mobile phone sensing algorithms, applications, and systems. We discuss the emerging sensing paradigms, and formulate an architectural framework for discussing a number of the open issues and challenges emerging in the new area of mobile phone sensing research.
Sensor convergence on the mobile phone is spawning a broad base of new and interesting mobile applications. As applications grow in sophistication, raw sensor readings often require classification into more useful application-specific high-level data. For example, GPS readings can be classified as running, walking or biking. Unfortunately, traditional classifiers are not built for the challenges of mobile systems: energy, latency, and the dynamics of mobile.
Kobe is a tool that aids mobile classifier development. With the help of a SQL-like programming interface, Kobe performs profiling and optimization of classifiers to achieve an optimal energy-latency-accuracy tradeoff. We show through experimentation on five real scenarios, classifiers on Kobe exhibit tight utilization of available resources. For comparable levels of accuracy traditional classifiers, which do not account for resources, suffer between 66% and 176% longer latencies and use between 31% and 330% more energy. From the experience of using Kobe to prototype two new applications, we observe that Kobe enables easier development of mobile sensing and classification apps.
Mobile phone sensors can be used to develop context-aware systems that automatically detect when patients require assistance. Mobile phones can also provide ecological momentary interventions that deliver tailored assistance during problematic situations. However, such approaches have not yet been used to treat major depressive disorder.
The purpose of this study was to investigate the technical feasibility, functional reliability, and patient satisfaction with Mobilyze!, a mobile phone- and Internet-based intervention including ecological momentary intervention and context sensing.
We developed a mobile phone application and supporting architecture, in which machine learning models (ie, learners) predicted patients' mood, emotions, cognitive/motivational states, activities, environmental context, and social context based on at least 38 concurrent phone sensor values (eg, global positioning system, ambient light, recent calls). The website included feedback graphs illustrating correlations between patients' self-reported states, as well as didactics and tools teaching patients behavioral activation concepts. Brief telephone calls and emails with a clinician were used to promote adherence. We enrolled 8 adults with major depressive disorder in a single-arm pilot study to receive Mobilyze! and complete clinical assessments for 8 weeks.
Promising accuracy rates (60% to 91%) were achieved by learners predicting categorical contextual states (eg, location). For states rated on scales (eg, mood), predictive capability was poor. Participants were satisfied with the phone application and improved significantly on self-reported depressive symptoms (beta(week) = -.82, P < .001, per-protocol Cohen d = 3.43) and interview measures of depressive symptoms (beta(week) = -.81, P < .001, per-protocol Cohen d = 3.55). Participants also became less likely to meet criteria for major depressive disorder diagnosis (b(week) = -.65, P = .03, per-protocol remission rate = 85.71%). Comorbid anxiety symptoms also decreased (beta(week) = -.71, P < .001, per-protocol Cohen d = 2.58).
Mobilyze! is a scalable, feasible intervention with preliminary evidence of efficacy. To our knowledge, it is the first ecological momentary intervention for unipolar depression, as well as one of the first attempts to use context sensing to identify mental health-related states. Several lessons learned regarding technical functionality, data mining, and software development process are discussed.
Clinicaltrials.gov NCT01107041; http://clinicaltrials.gov/ct2/show/NCT01107041 (Archived by WebCite at http://www.webcitation.org/60CVjPH0n).
Automated analysis of human affective behavior has attracted increasing attention from researchers in psychology, computer science, linguistics, neuroscience, and related disciplines. However, the existing methods typically handle only deliberately displayed and exaggerated expressions of prototypical emotions despite the fact that deliberate behaviour differs in visual appearance, audio profile, and timing from spontaneously occurring behaviour. To address this problem, efforts to develop algorithms that can process naturally occurring human affective behaviour have recently emerged. Moreover, an increasing number of efforts are reported toward multimodal fusion for human affect analysis including audiovisual fusion, linguistic and paralinguistic fusion, and multi-cue visual fusion based on facial expressions, head movements, and body gestures. This paper introduces and surveys these recent advances. We first discuss human emotion perception from a psychological perspective. Next we examine available approaches to solving the problem of machine understanding of human affective behavior, and discuss important issues like the collection and availability of training and test data. We finally outline some of the scientific and engineering challenges to advancing human affect sensing technology.
Research examining adolescent mood, stresses, and coping has tended to use retrospective questionnaires which are affected by recall biases. The aim of this study was to develop, pilot, and evaluate a youth-friendly mobile phone program to monitor, in real-time, young people's everyday experiences of mood, stress, and their coping behaviours.
A momentary sampling program was designed for mobile phones, and ran for 7 days, administering a brief questionnaire four random times each day, capturing information on current activity, mood, responses to negative mood, stresses, alcohol and cannabis use. Eleven high school students reviewed the program in focus groups, and 18 students completed 7 days of monitoring.
Engagement with the mobiletype program was high with 76% of 504 possible entries completed and 94% (17/18) of the participants reporting that the program adequately captured their moods, thoughts, and activities. The mobiletype program captured meaningful and analyzable data on the way young people's moods, stresses, coping strategies, and alcohol and cannabis use, vary both between and within individuals.
The mobiletype program captured a range of detailed and interesting qualitative and quantitative data about young people's everyday mood, stresses, responses, and general functioning.
This paper describes SmartMood, a mood tracking and analysis system designed for patients with mania. By analyzing the voice data captured from a smartphone while the user is having a conversation, statistics are generated for each behavioral factor to quantitatively describe his/her mood status. By comparing the newly generated statistics with those under normal mood, SmartMood tries to identify any new manic episodes so that appropriate consultation and medication actions can be taken. The daily behavioral statistics may serve as important references for psychiatrists to show the effectiveness of treatments. To reduce the probability of false alarms, we propose an adaptive running range method to estimate the normal mood range for each behavioral factor, and study methods to minimize the effects of background noise on the generated statistics. The preliminary experimental results on SmartMood show that a method using the pitch of a voice data sample to identify silent periods can better differentiate the voice of a normal or manic user in a call session than other methods. The results from the limited proof of concept testing indicate that moving to clinical testing is warranted.
Background:
Postpartum depression (PPD) is a disorder that often goes undiagnosed. The development of a screening program requires considerable and careful effort, where evidence-based decisions have to be taken in order to obtain an effective test with a high level of sensitivity and an acceptable specificity that is quick to perform, easy to interpret, culturally sensitive, and cost-effective. The purpose of this article is twofold: first, to develop classification models for detecting the risk of PPD during the first week after childbirth, thus enabling early intervention; and second, to develop a mobile health (m-health) application (app) for the Android(®) (Google, Mountain View, CA) platform based on the model with best performance for both mothers who have just given birth and clinicians who want to monitor their patient's test.
Materials and methods:
A set of predictive models for estimating the risk of PPD was trained using machine learning techniques and data about postpartum women collected from seven Spanish hospitals. An internal evaluation was carried out using a hold-out strategy. An easy flowchart and architecture for designing the graphical user interface of the m-health app was followed.
Results:
Naive Bayes showed the best balance between sensitivity and specificity as a predictive model for PPD during the first week after delivery. It was integrated into the clinical decision support system for Android mobile apps.
Conclusions:
This approach can enable the early prediction and detection of PPD because it fulfills the conditions of an effective screening test with a high level of sensitivity and specificity that is quick to perform, easy to interpret, culturally sensitive, and cost-effective.
The paper describes the approach based on using standard input devices, such as keyboard and mouse, as sources of data for the recognition of users' emotional states. A number of systems applying this idea have been presented focusing on three categories of research problems, i.e. collecting and labeling training data, extracting features and training classifiers of emotions. Moreover the advantages and examples of combining standard input devices with other sources of information on human emotions have been also described. Eventually some conclusions from the review have been drawn.
In this study, we aim to find physiological or behavioral markers for stress. We collected 5 days of data for 18 participants: a wrist sensor (accelerometer and skin conductance), mobile phone usage (call, short message service, location and screen on/off) and surveys (stress, mood, sleep, tiredness, general health, alcohol or caffeinated beverage intake and electronics usage). We applied correlation analysis to find statistically significant features associated with stress and used machine learning to classify whether the participants were stressed or not. In comparison to a baseline 87.5% accuracy using the surveys, our results showed over 75% accuracy in a binary classification using screen on, mobility, call or activity level information (some showed higher accuracy than the baseline). The correlation analysis showed that the higher-reported stress level was related to activity level, SMS and screen on/off patterns.
We describe initial results from an ongoing project to use mobile phone sensors to detect stress related situations1. The questions that we address in this stage of our work in progress is whether differences between stressful and non stressful periods can be detected in information readily available on a smartphone such as location traces, BlueTooth devices seen during the day and phone call patterns. We present an experiment with 7 students who were monitored during a two week exam session (stressful situation) and the two following weeks (non stressful period). The results show that a behaviour modification can clearly be seen, although the exact interpretation and generalization requires further work and larger scale experiments.
Awareness of the emotion of those who communicate with others is a fundamental challenge in building affective intelligent systems. Emotion is a complex state of the mind influenced by external events, physiological changes, or relationships with others. Because emotions can represent a user's internal context or intention, researchers suggested various methods to measure the user's emotions from analysis of physiological signals, facial expressions, or voice. However, existing methods have practical limitations to be used with consumer devices, such as smartphones; they may cause inconvenience to users and require special equipment such as a skin conductance sensor. Our approach is to recognize emotions of the user by inconspicuously collecting and analyzing user-generated data from different types of sensors on the smartphone. To achieve this, we adopted a machine learning approach to gather, analyze and classify device usage patterns, and developed a social network service client for Android smartphones which unobtrusively find various behavioral patterns and the current context of users. Also, we conducted a pilot study to gather real-world data which imply various behaviors and situations of a participant in her/his everyday life. From these data, we extracted 10 features and applied them to build a Bayesian Network classifier for emotion recognition. Experimental results show that our system can classify user emotions into 7 classes such as happiness, surprise, anger, disgust, sadness, fear, and neutral with a surprisingly high accuracy. The proposed system applied to a smartphone demonstrated the feasibility of an unobtrusive emotion recognition approach and a user scenario for emotion-oriented social communication between users.
A meta-analysis was conducted on the accuracy of predictions of various objective outcomes in the areas of clinical and social psychology from short observations of expressive behavior (under 5 min). The overall effect size for the accuracy of predictions for 38 different results was .39. Studies using longer periods of behavioral observation did not yield greater predictive accuracy; predictions based on observations under 0.5 min in length did not differ significantly from predictions based on 4- and 5-min observations. The type of behavioral channel (such as the face, speech, the body, tone of voice) on which the ratings were based was not related to the accuracy of predictions. Accuracy did not vary significantly between behaviors manipulated in a laboratory and more naturally occurring behavior. Last, effect sizes did not differ significantly for predictions in the areas of clinical psychology, social psychology, and the accuracy of detecting deception. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
This paper presents an implementation of the Active Appearance Model that is able to track a face on a mobile device in real-time.
We achieve this performance by discarding an explicit texture model, using fixed-point arithmetic for much of the computation,
applying a sequence of models with increasing complexity, and exploiting a sparse basis projection via Haar-like features.
Our results show that the Haar-like feature basis achieves similar performance to more traditional approaches while being
more suitable for a mobile device. Finally, we discuss mobile applications of the system such as face verification, teleconferencing
and human-computer interaction.
KeywordsFacial feature tracking–Active Appearance Model
We consider the problem of automatically recognizing human faces from frontal views with varying expression and illumination, as well as occlusion and disguise. We cast the recognition problem as one of classifying among multiple linear regression models and argue that new theory from sparse signal representation offers the key to addressing this problem. Based on a sparse representation computed by C
<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
-minimization, we propose a general classification algorithm for (image-based) object recognition. This new framework provides new insights into two crucial issues in face recognition: feature extraction and robustness to occlusion. For feature extraction, we show that if sparsity in the recognition problem is properly harnessed, the choice of features is no longer critical. What is critical, however, is whether the number of features is sufficiently large and whether the sparse representation is correctly computed. Unconventional features such as downsampled images and random projections perform just as well as conventional features such as eigenfaces and Laplacianfaces, as long as the dimension of the feature space surpasses certain threshold, predicted by the theory of sparse representation. This framework can handle errors due to occlusion and corruption uniformly by exploiting the fact that these errors are often sparse with respect to the standard (pixel) basis. The theory of sparse representation helps predict how much occlusion the recognition algorithm can handle and how to choose the training images to maximize robustness to occlusion. We conduct extensive experiments on publicly available databases to verify the efficacy of the proposed algorithm and corroborate the above claims.
As the ubiquitous communication environment advances, there is a demand for services that deliver the type of content appropriate for the preferences of the user. Services are already available that identify the user's preferences by analyzing his/her log information, and recommend items of content suitable for him/her. Methods of inferring the user's mood by partially analyzing pictograms frequently used in his/her emails have been proposed. In this paper, we extend such methods in order to enhance the accuracy of mood inference. Specifically, we propose a method of inferring the user's mood from emoticons, a set letters to form a face that graphically expresses the user's emotion, used in his/her multiple emails, and a method of identifying the user's preferences by analyzing pictograms that are used in his/her single email. We extract mood elements and preference elements and their intensities from pictograms and emoticons, and register them in respective databases. We developed a prototype system for evaluation. The study of the sample emails collected using this system confirmed the effectiveness of the proposed methods.
This paper describes the challenges of getting gro und truth affective labels for spontaneous video, and presents implicat ions for systems such as virtual agents that have automated facial analysis capabilities. We first present a dataset from an intelligent tutoring application an d describe the most prevalent approach to labeling such data. We then present an alternative labeling approach, which closely models how the majority of automated facial analysis systems are designed. We show that while participan ts, peers and trained judges report high inter-rater agreement on expressions of delight, confusion, flow, frustration, boredom, surprise, and neutral when sh own the entire 30 minutes of video for each participant, inter-rater agreement d rops below chance when human coders are asked to watch and label short 8 s econd clips for the same set of labels. We also perform discriminative analysis for facial action units for each affective state represented in the clips. The results emphasize that human coders heavily rely on factors such as familiarity of the person and context of the interaction to correctly infer a person's affec tive state; without this information, the reliability of humans as well as m achines attributing affective labels to spontaneous facial-head movements drops s ignificantly.
Today's mobile phones are essentially off-the-shelf, mass-market wearable computers. In this paper, we describe mobile social software that uses tone of voice, location and proximity information to create a sophisticated understanding of people's social lives, by automatically mining their face-to-face and phone interactions. We describe several applications of our system - automatic characterization of social and workplace interactions, a courtesy reminder for phone conversations, and a personal trainer for dating encounters.
Automatically identifying the person you are talking with using continuous audio sensing has the potential to enable many
pervasive computing applications from memory assistance to annotating life logging data. However, a number of challenges,
including energy efficiency and training data acquisition, must be addressed before unobtrusive audio sensing is practical
on mobile devices. We built SpeakerSense, a speaker identification prototype that uses a heterogeneous multi-processor hardware
architecture that splits computation between a low power processor and the phone’s application processor to enable continuous
background sensing with minimal power requirements. Using SpeakerSense, we benchmarked several system parameters (sampling
rate, GMM complexity, smoothing window size, and amount of training data needed) to identify thresholds that balance computation
cost with performance. We also investigated channel compensation methods that make it feasible to acquire training data from
phone calls and an automatic segmentation method for training speaker models based on one-to-one conversations.
The presence of multimodal sensors on current mobile phones enables a broad range of novel mobile applications. Environmental and user-centric sensor data of unprecedented quantity and quality can be captured and reported by a possible user base of billions of mobile phone subscribers worldwide. The strong focus on the collection of detailed sensor data may however compromise user privacy in various regards, e.g., by tracking a user's current location. In this survey, we identify the sensing modalities used in current participatory sensing applications, and assess the threats to user privacy when personal information is sensed and disclosed. We outline how privacy aspects are addressed in existing sensing applications, and determine the adequacy of the solutions under real-world conditions. Finally, we present countermeasures from related research fields, and discuss their applicability in participatory sensing scenarios. Based on our findings, we identify open issues and outline possible solutions to guarantee user privacy in participatory sensing.
This article, based on the available literature, attempts to discuss the importance of recruitment and retention of research participants, the associated barriers and challenges, and various strategies to overcome these barriers.
The inability to recruit and retain the required participants in a research project poses serious threats to both the internal and the external validity of a research study. Despite serious implications, the issues of recruitment and retention do not receive due attention in research and publications. Literature suggests a lack of coordinated efforts to collect information on the outcomes of recruitment experiences in clinical trials and population studies. Studies often mention the number of participants who refuse to participate; however, the majority of the studies often fail to mention the specific reasons insufficient recruitment or retention of the participants.
A methodological paper.
Various participant-, context-, environment- and research-related factors are examined that affect the phenomenon of recruitment and retention of the participants in a study.
Delayed or inefficient recruitment also has financial and ethical implications. Although there are many pieces of information scattered throughout academic journals on recruitment and retention of participants in research, few authors have dealt with the issue holistically. It is imperative for researchers to understand the importance of recruitment and retention of research participants, the associated barriers and challenges, and various strategies to overcome these barriers.
Appropriate recording and reporting of the problems faced while recruiting and retaining the participants in research studies can help not only in understating the challenge, but will also help in devising the strategies to overcome this problem. This article was an attempt to synthesise and review the available literature on recruitment and retention issues, which demand extensive theoretical and conceptual thinking as part of the research design.
An understanding of the challenges and issues related to recruitment and retention can help researchers to think ahead about the strategies to overcome these issues and consequently save the time and energy of the participants, researchers and funding agencies.
The five-factor model of personality is a hierarchical organization of personality traits in terms of five basic dimensions: Extraversion, Agreeableness, Conscientiousness, Neuroticism, and Openness to Experience. Research using both natural language adjectives and theoretically based personality questionnaires supports the comprehensiveness of the model and its applicability across observers and cultures. This article summarizes the history of the model and its supporting evidence; discusses conceptions of the nature of the factors; and outlines an agenda for theorizing about the origins and operation of the factors. We argue that the model should prove useful both for individual assessment and for the elucidation of a number of topics of interest to personality psychologists.