PosterPDF Available

Abstract

Diabetes mellitus Type II (T2DM) is a common chronic disease of the endocrine system in which the pancreas no longer produces enough insulin to metabolize blood glucose or the body becomes less sensitive to insulin. To manage blood glucose levels and to reduce the risk of diabetes-related complications (e.g., cardiovascular diseases, vision loss, amputations), patients need to follow medical recommendations for healthy eating, physical activity, and medication adherence in their everyday life. Evidence suggests that for married adults, illness management is mainly shared with their spouses. Spousal support is associated with healthier habits among diabetes patients. Additionally, spousal support has been shown to have beneficial effects on well-being or affect (feelings). Given that there is some relationship between spousal support and affect, through affect detection, we may have a proxy for received spousal support. Considering the health benefits of spousal support especially for chronic disease management, affect detection could be used to inform just-in-time adaptive interventions. In this ongoing work to develop an affect detection system for couples’ chronic illness management, we describe the specific research questions we want answer, our potential technical contributions, our data collection plan and finally our data analysis plan.
Multimodal Affect Detection among Couples for Diabetes Management
George Boateng1, Prabhakaran Santhanam1, Janina Lüscher2, Urte Scholz2, Tobias Kowatsch3
1ETH Zurich, Switzerland, 2University of Zurich, Switzerland, 3University of St. Gallen, Switzerland
Introduction
Diabetes mellitus Type II (T2DM) is a common chronic disease of the endocrine system in which the
pancreas no longer produces enough insulin to metabolize blood glucose or the body becomes less sensitive
to insulin (SDG, 2015). Over one in four of the 65 years and older adults in the U.S. population are estimated
to have T2DM resulting in 9.4% of the U.S. population (CDC, 2017). In Switzerland, almost 500,000 people
suffer from T2DM, which is approximately 4.9% of the male Swiss population and 4.2% of the female Swiss
population (SDG, 2015). To manage blood glucose levels and to reduce the risk of diabetes-related
complications (e.g., cardiovascular diseases, vision loss, amputations), patients need to follow medical
recommendations for healthy eating, physical activity, and medication adherence in their everyday life
(CDC, 2017). Evidence suggests that for married adults, illness management is mainly shared with their
spouses (Seidel et al., 2012; Rintala et al., 2013). Social support among spouses is associated with healthier
habits among diabetes patients (Miller et al, 2005). Additionally, spousal support has been shown to have
beneficial effects on well-being or affect (feelings) (Prati and Pietrantoni, 2010; Iida et al., 2010). Given that
there is some relationship between social support and affect, through affect detection, we may have a proxy
for received social support from spouses. Considering the health benefits of social support especially for
chronic disease management, affect detection could be used to inform just-in-time adaptive interventions
through for example a digital coach. Also, this digital coach could adapt its communication style based on
the detected affect.
Related work
Currently, psychologists measure affect through various self-reports such as the PANAS (Watson et al.,
1988). These self-reports are however not practical for continuous affect measurement in the wild because of
their obtrusive nature. On the other hand, a lot of work has been done in the area of affect detection.
However, a lot of these work use data from controlled settings such as having actors make specific facial
expressions mimicking certain emotions, or read text in a specific emotional tone (Poria et al., 2017). It is not
clear whether the algorithms developed using these data will work well in the naturalistic context of
couples’ interactions in everyday life. Additionally, there are well developed systems such as those by
Affectiva that use data from the face to recognize affect (Affectiva, 2018). These systems however use only a
unimodal source of data and hence will not work well in the context of couple’s affect recognition in
everyday life when for example facial data is not available.
Research Questions
It is not clear how well the affect of couples in everyday life can be detected, despite the potential for its
usage in improving couples’ chronic disease management. In our ongoing work, we plan on addressing the
following research questions:
RQ1: How accurately can affect be predicted using multimodal real-world sensor data from couples? There
are several challenges to address such as the kind of sensor data that should be collected, how the data should
be fused together, what features to extract for regular machine learning models, what deep learning
approaches to use, what algorithms will produce the best results, among others.
RQ2: How accurately can the affect of couples be detected in real-time in everyday life? There are several
challenges to address such as how well the algorithm will work when certain sensor data such as voice is not
available, how to ensure that there is little latency in prediction, whether to do the prediction on a remote
server considering various privacy issues or on-device, which will imply the machine learning model will
need to be compact, potentially reducing the prediction accuracy, among others.
1
There are three technical contributions should these research questions be answered:
1) A novel machine learning algorithm that predicts affect to a high degree of accuracy using multimodal
real-world sensor data from couples
2) A mobile system that predicts affect of couples to a high degree of accuracy in real-time in the wild
3) A novel module for the open source assessment and intervention platform MobileCoach
(www.mobile-coach.eu) to be used to predict affect of individuals
Methods: DyMand Study (Dyadic Management of Diabetes)
To answer our research questions, we will be running a user study starting in January 2019, DyMand funded
by the Swiss National Science Foundation (CR12I1_166348/1), through which we will collect various sensor
data in the wild along with corresponding self-reports with which to develop our affect detection algorithm.
The goal of this study are to understand the relationship between social support and the health behavior and
wellbeing (affect) of couples in which one partner has T2DM diabetes. In this study, we will have 180
couples (N=180; n=360), with one partner having T2DM diabetes. We will collect sensor and self-report data
from them for 7 days during which data will be collected in the mornings and evenings during the weekdays,
and the whole day during the weekends when they spend time together.
Data Collection
Each partner of the participating couples will receive a smartwatch (Polar M600), a smartphone (Nokia 6.1)
and an accelerometer (GT3X+ monitor devices; ActiGraph, Pensacola, FL). The smartwatch will collect the
following sensor data: audio, heart rate, accelerometer, gyroscope, ambient light, physical activity and BLE
signal strength between each couple’s smartwatches. The smartphone (Nokia 6.1) will collect video, audio
and ambient light for 3 seconds when the subjects are completing the self-report on the phone. The
ActiGraph will record physical activity information all day. Various studies have used these sensor data for
affect detection (Poria et al., 2017; Timmons et al., 2017; Boateng and Kotz, 2017). We will collect the
smartwatch sensor data for 5 minutes once per hour within the morning and evening hours set by the couples,
after which a self-report is triggered for each partner to rate their affect over the past 5 minutes. We ensure
that there is at least 20 minutes between subsequent data collection to reduce burden of completing the
self-reports. To optimize the quality of data collected within that hour, we collect data when the couple is
close together and when they are speaking. We will determine closeness using the BLE signal strength of the
smartwatches and we will determine speaking using a voice activity detection algorithm. In the case in which
this condition is not met in the hour, we record the last 5 minutes in the hour. After the 5 minutes of
recording, we trigger the Affective Slider, a digital affect measuring tool which measures the arousal and
pleasure dimensions of affect (Betella and Verschure, 2016). Additionally, at the end of the day, we trigger
the Affective Slider, and also a short form of the PANAS self-report (Mackinnon et al., 1999) for the couples
to report their affect over the whole day.
Data Analysis
In order to assess the relationship between the sensor data and self-reports, and predict affect, we plan on
exploring regular machine learning and deep learning approaches. For regular machine learning, we will use
the pipeline of preprocessing, feature extraction and selection, and cross validation. We will do this for
various algorithms such as random forest, support vector machines, etc. For deep learning, we will explore
using convolutional neural networks and recurrent neural networks along with different architectures that
may combine both of them. Additionally, because of the multimodal nature of the data, we will explore
fusion at the feature level i.e., feed the different sensor modalities into the same machine learning algorithm
or at the decision level i.e., have a different algorithm for each sensor modality and then combine the
individual algorithm predictions using for example majority voting, or some hybrid (Poria et al., 2017). We
will then pick the best performing algorithm and then optimize it to run in real-time.
2
References
Affectiva. Accessed on August 2018. Retrieved from: https://www.affectiva.com/
Betella, A., and Verschure, P.F. (2016). The Affective Slider: A Digital Self-Assessment Scale for the
Measurement of Human Emotions. PLoS One 11, e0148037. doi: 10.1371/journal.pone.0148037.
Boateng, G., and Kotz, D. (2016). StressAware: An app for real-time stress monitoring on the amulet
wearable platform. In MIT Undergraduate Research Technology Conference (URTC), 2016 IEEE (pp. 1-4).
IEEE.
Centers for Disease Control and Prevention (2017). National Diabetes Statistics Report, 2017
742 [Online]. Available: https://www.cdc.gov/diabetes/pdfs/data/statistics/national-diabetes743
statistics-report.pdf [Accessed 25/07/2018].
Iida, M., Parris Stephens, M.A., Rook, K.S., Franks, M.M., and Salem, J.K. (2010). When the going
839 gets tough, does support get going? Determinants of spousal support provision to type 2
840 diabetic patients. Personality and Social Psychology Bulletin 36, 780-791. doi:
841 10.1177/0146167210369897.
Mackinnon, A., Jorm, A.F., Christensen, H., Korten, A.E., Jacomb, P.A., and Rodgers, B. (1999). A short
form of the Positive and Negative Affect Schedule: Evaluation of factorial validity and invariance across
demographic variables in a community sample. Personality and Individual Differences 27,405-416. doi:
10.1016/S0191-8869(98)00251-7.
Miller, D., and Brown, J.L. (2005). Marital interactions in the process of dietary change for type 2 diabetes.
Journal of Nutrition Education and Behavior 37(5), 226-234.
Prati, G., and Pietrantoni, L. (2010). The relation of perceived and received social support to mental
949 health among first responders: A meta-analytic review. Journal of Community Psychology 38,
950 403-417. doi: 10.1002/jcop.20371.
Poria, S., Cambria, E., Bajpai, R., and Hussain, A. (2017). A review of affective computing: From unimodal
analysis to multimodal fusion. Information Fusion 37, 98-125. Doi: 10.1016/j.inffus.2017.02.003.
SDG (2015). Diabetes mellitus. Das Wichtigste in Kürze. [Online]. Schweizerische Diabetes-Gesellschaft.
Available: http://www.sdgshop.ch/media/pdf/197-de.pdf [Accessed 25/09/2015].
Timmons, A.C., Chaspari, T., Han, S.C., Perrone, L., Narayanan, S.S., and Margolin, G. (2017). Using
Multimodal Wearable Technology to Detect Conflict among Couples. Computer 50(3), 50-59.
Watson, D., Clark, L. A., & Tellegen, A. (1988). Development and validation of brief measures of positive
and negative affect: the PANAS scales. Journal of personality and social psychology, 54(6), 1063.
3
www.c4dhi.org
George Boateng1, Prabhakaran Santhanam1, Janina Lüscher2, Urte Scholz2, Tobi as Kowatsch3
1ETH Zurich, Switzerland, 2University of Zurich, Switzerland, 3University of St. Gallen, Switzerland
Multimodal Affect Detection among
Couples for Diabetes Management
2. Problem
It is not clear how well the affect of couples in everyday life can
be detected, despite the potential for its usage in improving
couples’ chronic disease management
3. Research Questions
RQ1: How accurately can affect be predicted using
multimodal real-world sensor data from couples?
RQ2: How accurately can the affect of couples be detected
in real-time in everyday life?
1. Background 4. Method
5. Expected Results
§Anovel machine learning algorithm that
predicts affect accurately using multimodal
real-world sensor data
§Amobile system that predicts affect of
couples accurately in real-time in everyday life
References
Boateng, G. , and Kotz, D. (2016). StressAware: An app for real-time stress monitoring on the amulet wearable platform.
In MIT Undergraduate Research Technology Conference (URTC), 2016 IEEE (pp. 1-4). IEEE.
Bolger, N., and Amarel, D. (2007). Effects of social support visibility on adjustment to stress: Experimental evidence.
Journal of Personality and Social Psychology 92, 458-475. doi: 10.1037/0022-3514.92.3.458.
Miller, D., and Brown, J.L . (2005). Marital interactions in the process of dietary change for type 2 diabe tes. Journal of
Nutrition Education and Behavior 37(5), 226-234.
Poria, S., Cambria, E., Bajpai, R., and Hussain, A. (2017). A review of affective computing: From unimodal analysis to
multimodal fusi on. Information Fusion 37, 98-125. Doi: 10.1016/j.inffus.2017.02.003.
Timmons, A.C., Chaspari, T., Han, S.C., Perrone, L., Narayanan, S.S., and Margolin, G. (2017). Using Multimodal
Wearable T echnology to Detec t Conflict amo ng Couples. Compute r 50(3), 50-59.
Montréal, Canada | December 7 | 2018Black in AI Workshop | NeurIPS
FundingPartner
Social Support from Spouses
Healthier habits among
diabetes patients
Affect Detection
Beneficial effects on well-
being or affect
May serve as proxy for received spousal support
Inform just-in-time adaptive interventions
Digital coach can adapt communication style
Digital Coach
DyMand Study: Collect data from 180 couples in which one
partner has Type II diabetes
Data Collection: 7 days, 5 minutes of data, collected once per
hour when the couple is close and speaking
Sensor Data: Audio, Video, Accelerometer, Gyroscope, Physical
Activity, Heart Rate, Ambient light, BLE signal strength (from
smartwatch, smartphone and ActiGraph)
Self Reports: Affective Slider and PANAS (short form)
Data Analysis: Predict affect using machine learning and deep
learning algorithms + multimodal approaches
Unimodal data source
Self Reports Data from controlled settings
... Our secondary motivation for developing VADLite is for it to be used in the development of a real-time smartwatch-based app for recognizing emotions of couples using speech prosody and the semantics of speech [27]. Emotion recognition from speech will then be used in combination with other sensor modalities from the smartwatch to perform real-time multimodal emotion recognition among couples [6]. Prior work has shown that social support among couples results in better health behavior when one partner has diabetes and also affect the emotions of the couple [15,23,28]. ...
Conference Paper
Full-text available
Smartwatches provide a unique opportunity to collect more speech data because they are always with the user and also have a more exposed microphone compared to smartphones. Speech data could be used to infer various indicators of mental well being such as emotions, stress and social activity. Hence, real-time voice activity detection (VAD) on smartwatches could enable the development of applications for mental health monitoring. In this work, we present VADLite, an open-source, lightweight, system that performs real-time VAD on smartwatches. It extracts mel-frequency cepstral coefficients and classifies speech versus non-speech audio samples using a linear Support Vector Machine. The real-time implementation is done on the Wear OS Polar M600 smartwatch. An offline and online evaluation of VADLite using real-world data showed better performance than WebRTC's open-source VAD system. VADLite can be easily integrated into Wear OS projects that need a lightweight VAD module running on a smartwatch.
Article
Full-text available
By monitoring human behavior unobtrusively, mobile sensing technologies have the potential to improve our daily lives. Initial results from a field study demonstrate that such passive technologies can detect a complex psychological state in an uncontrolled, real-life environment. In the web extra at https://youtu.be/n8Ap3Z44ojQ, guest editor Katarzyna Wac interviews authors Adela Timmons and Theodora Chaspari, quality-of-life technology researchers at the University of Southern California.
Article
Full-text available
Self-assessment methods are broadly employed in emotion research for the collection of subjective affective ratings. The Self-Assessment Manikin (SAM), a pictorial scale developed in the eighties for the measurement of pleasure, arousal, and dominance, is still among the most popular self-reporting tools, despite having been conceived upon design principles which are today obsolete. By leveraging on state-of-the-art user interfaces and metacommunicative pictorial representations, we developed the Affective Slider (AS), a digital self-reporting tool composed of two slider controls for the quick assessment of pleasure and arousal. To empirically validate the AS, we conducted a systematic comparison between AS and SAM in a task involving the emotional assessment of a series of images taken from the International Affective Picture System (IAPS), a database composed of pictures representing a wide range of semantic categories often used as a benchmark in psychological studies. Our results show that the AS is equivalent to SAM in the self-assessment of pleasure and arousal, with two added advantages: the AS does not require written instructions and it can be easily reproduced in latest-generation digital devices, including smartphones and tablets. Moreover, we compared new and normative IAPS ratings and found a general drop in reported arousal of pictorial stimuli. Not only do our results demonstrate that legacy scales for the self-report of affect can be replaced with new measurement tools developed in accordance to modern design principles, but also that standardized sets of stimuli which are widely adopted in research on human emotion are not as effective as they were in the past due to a general desensitization towards highly arousing content.
Article
Full-text available
Previous fieldwork has suggested that visible social support can entail an emotional cost and that a supportive act is most effective when it is accomplished either (a) outside of recipients' awareness or (b) within their awareness but with sufficient subtlety that they do not interpret it as support. To investigate the latter phenomenon, the authors conducted 3 experiments in which female participants were led to expect a stressful speech task and a confederate peer provided support in such a way that it was either visible or invisible (N=257). Invisible support (practical and emotional) reduced emotional reactivity relative to visible and no support. Visible support was either ineffective or it exacerbated reactivity. Explanatory analyses indicated that support was effective when it avoided communicating a sense of inefficacy to recipients.
Article
Affective computing is an emerging interdisciplinary research field bringing together researchers and practitioners from various fields, ranging from artificial intelligence, natural language processing, to cognitive and social sciences. With the proliferation of videos posted online (e.g., on YouTube, Facebook, Twitter) for product reviews, movie reviews, political views, and more, affective computing research has increasingly evolved from conventional unimodal analysis to more complex forms of multimodal analysis. This is the primary motivation behind our first of its kind, comprehensive literature review of the diverse field of affective computing. Furthermore, existing literature surveys lack a detailed discussion of state of the art in multimodal affect analysis frameworks, which this review aims to address. Multimodality is defined by the presence of more than one modality or channel, e.g., visual, audio, text, gestures, and eye gage. In this paper, we focus mainly on the use of audio, visual and text information for multimodal affect analysis, since around 90% of the relevant literature appears to cover these three modalities. Following an overview of different techniques for unimodal affect analysis, we outline existing methods for fusing information from different modalities. As part of this review, we carry out an extensive study of different categories of state-of-the-art fusion techniques, followed by a critical analysis of potential performance improvements with multimodal analysis compared to unimodal analysis. A comprehensive overview of these two complementary fields aims to form the building blocks for readers, to better understand this challenging and exciting research field.
Article
In recent studies of the structure of affect, positive and negative affect have consistently emerged as two dominant and relatively independent dimensions. A number of mood scales have been created to measure these factors; however, many existing measures are inadequate, showing low reliability or poor convergent or discriminant validity. To fill the need for reliable and valid Positive Affect and Negative Affect scales that are also brief and easy to administer, we developed two 10-item mood scales that comprise the Positive and Negative Affect Schedule (PANAS). The scales are shown to be highly internally consistent, largely uncorrelated, and stable at appropriate levels over a 2-month time period. Normative data and factorial and external evidence of convergent and discriminant validity for the scales are also presented. (PsycINFO Database Record (c) 2010 APA, all rights reserved)
Article
There are plenty of theories that may support the protective role of social support in the aftermath of potentially traumatic events. This meta-analytic review examined the role of received and perceived social support in promoting mental health among first responders (e.g., firefighters, police officers, and paramedics or emergency medical services personnel). Results showed that the effect sizes derived from 37 empirical studies ranged from 0 to 0.46. The overall weighted mean effect size was of medium magnitude (r=0.27). Moreover, moderator analysis showed that the effect size of perceived social support (r=0.31) was significantly higher than the effect size of received social support (r=0.22). Participants' age and gender or research design did not exert an influence on the relationship between social support and mental health. Theoretical implications and limitations are discussed. © 2010 Wiley Periodicals, Inc.
Article
To explore how couples adjust to dietary management of type 2 diabetes. Couples were interviewed, first together and then separately, during the first year after diagnosis and 1 year later. Qualitative interviews conducted in hospital classrooms using a semistructured interview guide. Couples (N = 20) with a recently diagnosed spouse who met the study criteria were purposefully selected from volunteers solicited from hospital-based diabetes classes. Processes used by the couple to address the prescribed diet. Thematic analysis of interview transcripts using grounded theory to identify patterns of adaptation processes used over time. Three couple categories emerged (cohesive, enmeshed, and disengaged), representing adaptation to the diabetic diet. Initially, 5 couples were cohesive (teamwork approach), 7 were enmeshed (nondiabetic spouse responsible for the diet; spouse was dependent), and 8 were disengaged (spouses functionally separate; spouse was solely responsible for the diet management). A year later, the majority of couples were disengaged (n = 14), 1 couple remained cohesive, and 4 couples remained enmeshed. Themes of flexibility, roles, rules, and communication varied across categories. Understanding categories of marital adjustment to the diabetic diet may improve nutrition-based diabetes interventions. Further study is needed to verify these findings in larger and more diverse populations.
When the going 839 gets tough, does support get going? Determinants of spousal support provision to type 2 840 diabetic patients
  • M Iida
  • M A Parris Stephens
  • K S Rook
  • M M Franks
  • J K Salem
Iida, M., Parris Stephens, M.A., Rook, K.S., Franks, M.M., and Salem, J.K. (2010). When the going 839 gets tough, does support get going? Determinants of spousal support provision to type 2 840 diabetic patients. Personality and Social Psychology Bulletin 36, 780-791. doi: 841 10.1177/0146167210369897.
A short form of the Positive and Negative Affect Schedule: Evaluation of factorial validity and invariance across demographic variables in a community sample
  • A Mackinnon
  • A F Jorm
  • H Christensen
  • A E Korten
  • P A Jacomb
  • B Rodgers
Mackinnon, A., Jorm, A.F., Christensen, H., Korten, A.E., Jacomb, P.A., and Rodgers, B. (1999). A short form of the Positive and Negative Affect Schedule: Evaluation of factorial validity and invariance across demographic variables in a community sample. Personality and Individual Differences 27,405-416. doi: 10.1016/S0191-8869(98)00251-7.