Conference PaperPDF Available

Exploring User Experience with a Conversational Agent to Treat Depression in Youth: A Think-Aloud Study

Authors:

Abstract and Figures

Conversational agents are a promising digital health intervention that can mitigate help-seeking barriers for youth with depression to receive treatment. Although studies have shown sufficient acceptance, feasibility, and promising effectiveness for adults, not much is known about how youth experience interacting with conversational agents to improve mental health. Therefore, we conducted an exploratory study with 15 youth with to collect data on their interaction with a conversational agent prototype using the think-aloud protocol. We coded the material from the think-aloud sessions using an inductive approach. Our findings provide insights into how youth with depression interacted with the prototype. Participants frequently and controversially discussed the conversational agent's (1) personality and interaction style, (2) its functionality, and (3) the dialogue content with implications for the design of conversational agents to treat depression and future research.
Content may be subject to copyright.
This is the author’s version of a work that was published in the following source:
Kuhlmeier, F.O., Gnewuch, U., Bauch, L., Metelmann, L. F., Lüttke, S. (2022). Exploring
User Experience with a Conversational Agent to Treat Depression in Youth: A Think-Aloud Study.
SIGHCI 2022 Proceedings.
https://h-lab.iism.kit.edu/
Please note: Copyright is owned by the author and / or the publisher. Commercial
use is not allowed.
© 2017. This manuscript version is made available under
the CC-BY-NC-ND 4.0 license
http://creativecommons.org/licenses/by-nc-nd/4.0/
Kuhlmeier et al. Exploring User Experience with a Conversational Agent for Depression
Proceedings of the Twenty-First Annual Pre-ICIS Workshop on HCI Research in MIS, Copenhagen, Denmark, December 11, 2022 1
Exploring User Experience with a Conversational Agent to
Treat Depression in Youth: A Think-Aloud Study
Florian Onur Kuhlmeier
University of Greifswald, Karlsruhe
Institute of Technology
florian.kuhlmeier@kit.edu
Ulrich Gnewuch
Karlsruhe Institute of
Technology
ulrich.gnewuch@kit.edu
Luise Bauch
University of Greifswald
luise.bauch@stud.uni-
greifswald.de
Lilli Feline Metelmann
University of Tübingen
lilli-feline.metelmann@student.uni-
tuebingen.de
Stefan Lüttke
University of Greifswald
stefan.luettke@uni-
greifswald.de
ABSTRACT
Conversational agents are a promising digital health
intervention that can mitigate help-seeking barriers for
youth with depression to receive treatment. Although
studies have shown sufficient acceptance, feasibility, and
promising effectiveness for adults, not much is known
about how youth experience interacting with
conversational agents to improve mental health. Therefore,
we conducted an exploratory study with 15 youth with to
collect data on their interaction with a conversational agent
prototype using the think-aloud protocol. We coded the
material from the think-aloud sessions using an inductive
approach. Our findings provide insights into how youth
with depression interacted with the prototype. Participants
frequently and controversially discussed the conversational
agent’s (1) personality and interaction style, (2) its
functionality, and (3) the dialogue content with
implications for the design of conversational agents to treat
depression and future research.
Keywords
conversational agent, depression, user experience, think-
aloud
INTRODUCTION
Depression in youth is a large-scale problem with
significant individual and socioeconomic costs
(Clayborne, Varin, & Colman, 2019). Yet, youth often do
not receive treatment due to stigma, uncertainty, and the
desire to solve problems on their own (Gulliver, Griffiths,
& Christensen, 2010). Conversational agents are
considered a promising digital mental health intervention
that may mitigate these barriers. Mimicking human
conversations may be the key to building a digital working
alliance, which is crucial for effective psychotherapy
(Darcy, Daniels, Salinger, Wicks, & Robinson, 2021).
Studies on conversational agents have shown sufficient
user experience and promising efficacy (Abd-Alrazaq,
Rababeh, Alajlani, Bewick, & Househ, 2020). However, it
remains largely unknown how youth with depression
experience interactions with conversational agents to treat
depression. The applicability of related research with adult
samples is limited as (1) depression in youth differs from
depression in adulthood (Rice et al., 2019) and (2) youth
interact with smartphones and conversational agents in
different ways than adults (Andone et al., 2016; Huffman,
2014). These differences should be considered when
designing conversational agents to treat depression.
Therefore, the purpose of this study was to investigate how
youth with depression experience interacting with a
conversational agent to treat depressive symptoms. The
results of this research-in-progress will be used to design a
conversational agent to treat depression in youth.
RELATED WORK
Conversational agents are digital applications designed to
mimic human conversational behavior (Dale, 2016). They
are popular in sectors such as customer service, education,
and healthcare (Følstad & Brandtzæg, 2017). Using
conversational agents to diagnose and treat mental
disorders has also been explored to expand current
capacities and intervention types, as they are scalable,
always available, and easily accessible (Vaidyam,
Linggonegoro, & Torous, 2020). Most conversational
agents to improve mental health are based on cognitive
behavioral therapy an effective and first-line
psychotherapy approach for depression (Oud et al., 2019).
The goal of cognitive behavioral therapy is to change
dysfunctional thoughts and behaviors (Auerbach, Webb, &
Stewart, 2016). Studies on conversational agents to
improve mental health have shown sufficient acceptance
and feasibility (Abd-Alrazaq et al., 2020; Vaidyam et al.,
2020). However, conversational agents have primarily
been designed for adults (Vaidyam et al., 2020) and
preventing rather than treating mental disorders (Chan et
al., 2022; Høiland, Følstad, & Karahasanovic, 2020).
Kuhlmeier et al. Exploring User Experience with a Conversational Agent for Depression
Proceedings of the Twenty-First Annual Pre-ICIS Workshop on HCI Research in MIS, Copenhagen, Denmark, December 11, 2022 2
METHODOLOGY
Study Design
The work presented in this article was part of an
exploratory study to investigate the needs and preferences
of youth for a conversational agent to treat depressive
symptoms. First, we conducted a semi-structured interview
on (1) problems and coping strategies for depression, (2)
attitudes toward conversational agents to treat depression,
and (3) design preferences. Second, we collected data on
how users experienced interacting with a conversational
agent prototype - which is the focus of this paper. We used
the think-aloud method (Jaspers, Steen, Bos, & Geenen,
2004), where we asked users to narrate their thoughts and
feelings while interacting with the prototype. At the
beginning of the trial, the interviewer briefly introduced
our conversational agent ‘Cady’, the content of the
dialogue, and explained how the think-aloud method
works. The users then interacted with the prototype via a
laptop placed in front of them. We recorded the screen of
the laptop and the user’s audio with an internal function of
the laptop. On average, the participants interacted with
Cady for 15:38 minutes (range = 11:54 20:38). After the
study, the recordings were transcribed for further analysis.
Participants
Participants qualified to take part in the study if they were
between the ages of 14 and 17 years, owned a smartphone,
and had previously been diagnosed with depression. They
were excluded if they had suicidal ideation, psychosis, or
low cognitive functioning. We advertised our study at local
psychotherapists. 15 youth (14-17 years old; M = 16, SD =
1.14) with symptoms of depression participated in the
study at the University of Tübingen. 13 participants
identified as female and 2 as non-binary.
Conversational Agent: Cady
Participants were asked to interact with a conversational
agent, developed with the prototyping software botsociety,
that guides users through a behavioral activation exercise.
Figure 1 shows a screenshot demonstrating the interaction
between the conversational agent and the user. Behavioral
activation is a core component of cognitive behavioral
therapy to treat depression in youth (Oud et al., 2019). It
aims to increase the engagement of patients with
pleasurable activities. The conversation consisted of the
following sections: (1) introduction, (2) mood check with
adaptive responses, (3) psychoeducation on the
relationship between behavior, thoughts, and feelings, (4)
finding and planning pleasant activities, (6) advice on how
to overcome barriers to performing activities, and (7)
feedback and goodbye. The conversation mainly consisted
of buttons with predefined answer alternatives and a few
free text input fields. We named the conversational agent
‘Cady’ and did not specify age, gender, or other
demographic characteristics to prevent specific
demographic characteristics from influencing the results of
the study.
Data Analysis
Two coders analyzed the think-aloud sessions using
content analysis according to Mayring (2004) with the
software QCAmap. We chose inductive coding to achieve
the most unbiased and thorough description of the data,
which we considered important due to the exploratory
nature of the study. Both coders first coded the material
independently and then agreed on overarching categories.
When codes differ due to the inductive approach, we report
the codes of the first coder.
RESULTS
Cady’s (1) personality and interaction style, (2)
functionality, and (3) dialogue content emerged as the main
categories.
Personality and Interaction Style
The personality and interaction style of Cady were
frequently discussed. Most participants characterized the
friendly and personal interaction style as one of its main
strengths. One participant said: I like that you
communicate in a friendly way, like an internet friend.’
Another participant was pleasantly surprised by Cady’s
reaction after finding three pleasant activities: That's
sweet! It is good that you can tell that Cady is happy.
Kuhlmeier et al. Exploring User Experience with a Conversational Agent for Depression
Proceedings of the Twenty-First Annual Pre-ICIS Workshop on HCI Research in MIS, Copenhagen, Denmark, December 11, 2022 3
Several participants also appreciated Cady’s use of emoji.
Others, however, criticized too many or inadequate emojis.
A participant explained their criticism: ‘Here, for example,
is a rocket emoji. And I would not use a rocket emoji
myself. Consequently, this participant would have
preferred fewer emojis. Another participant commented on
the language as too therapeutic: It was relatively
authentic, but you also notice that it is not written for my
age. But otherwise, if you see it as a therapy conversation,
it was good.’ The participant wished for more age-specific
language, such as short sentences: So, this was
therapeutic, but if it is supposed to be like a friend, I think
it is better if it is written like my age. Short sentences, not
completed. Furthermore, three participants appreciated
that Cady led the conversation, which was perceived as less
effort: I think it's good that the computer leads the
conversation, and you don't have to do too much yourself.
Two participants were pleased that Cady asked personal
questions. For example, a participant approved that Cady
had asked if she had experienced becoming less active
herself: I just think it's good that she is asking if I know
something like that or haven't experienced it yet.
However, one participant concluded that Cady asked too
many questions. Although two participants appreciated
Cady’s motivational style and humor, another participant
thought that Cady was too optimistic. One participant was
pleased that Cady asked for feedback after the
conversation. Lastly, one participant appreciated that Cady
disclosed personal information such as enjoying chats with
nice people.
Functionality
The participants pointed out several aspects of Cady's
functionality. Seven participants enjoyed interacting
through predefined responses. One participant stated: I
liked that there are predefined responses and I don’t need
to think of a response every time. Six participants
appreciated optional reminders to carry out the planned
activities, while two would have liked to personalize the
reminders. Another participant suggested automatic rather
than optional reminders. Four participants appreciated free
text responses, while one participant suggested including
more free text input. Furthermore, one participant liked
how Cady was able to understand their responses: ‘I think
it's really impressive that Cady understands what I write
and then responds to it.Yet, three other participants were
frustrated by inadequate responses. One participant said:
The message I sent does not even make sense. But Cady
does not realize that.
Another participant pointed out that the conversation felt
impersonal/ not tailored to them and their needs. Four
participants pointed out that Cady sent its messages too
fast. Two participants suggested lowering Cady’s typing
speed and including short breaks between messages. On
the contrary, one participant was happy with the speed of
the messages, while another participant felt that the pause
between messages was too long. One participant suggested
a typing indicator to make pauses more like human-to-
human chats. Four participants liked the simple, easy to
use, and familiar user chat interface, while one participant
suggested an option to personalize the user interface.
Kuhlmeier et al. Exploring User Experience with a Conversational Agent for Depression
Proceedings of the Twenty-First Annual Pre-ICIS Workshop on HCI Research in MIS, Copenhagen, Denmark, December 11, 2022 4
Dialogue Content
Participants frequently commented on the content of the
dialogue, primarily its selection and presentation. The
participants noted how useful and easy to implement the
content was. One participant said: I think the prototype has
already helped me, so that I can become more active
again’. The participants also appreciated the mood check-
in. One participant stated: ‘I think it's good that Cady asks
me how I’m doing.’ Another person was disappointed that
there was no option to just talk about something that was
on their mind and would have preferred such a feature: ‘If
you want to get something off your chest, it is good to have
the opportunity right at the beginning, during the
conversation or at the end, so that you read and
participate, but can still say something afterwards.’
Four participants suggested personalized content. For
example, they would like Cady to ask if the user wants to
learn about specific a topic (e.g., behavioral activation) or
if the user is affected by the presented content to determine
its relevance. Two participants appreciated that Cady
chooses and plans activities with them and that Cady
provides tips on how to carry out the activities. Four
participants enjoyed the images and examples, which
helped to understand the content. The participants also
noted that it was helpful that they had to text Cady three
activities they find pleasant. However, a participant
pointed out that Cady made choosing, planning, and doing
a pleasant activity seem easier than it really is: This is also
the case with my therapist, they simplify a lot. Like, it is
easy to do something pleasant. But sometimes it is difficult
to do anything at all. In addition, one participant stated
that breaking the content down into several messages rather
than a single text made it easier to understand and more
interactive. However, three participants said that there was
too much information. One participant suggested that Cady
should encourage users to read the content again to better
memorize it. Two participants proposed to improve the
content by shortening the text, receiving fewer consecutive
messages, and including emojis and text in every
predefined response.
DISCUSSION
In our exploratory study with 15 youth with symptoms of
depression, we found that conversational agents may serve
as an appropriate digital health intervention, as indicated
by the various strengths that the participants pointed out
during the think-aloud sessions. Our results also revealed
several challenges that provide implications for the design
of conversational agents for this target group.
First, our results show contradictory and controversial
discussions about several design aspects. For example,
language and emoji use was frequently and contradictorily
discussed, which emphasizes that personality and
interaction style are crucial. This result underscores the
need to thoroughly design and evaluate the character of the
conversational agent. However, it remains largely
unknown, how to provide a conversational agent with
personality traits through language use, emojis, and gifs.
Despite promising first attempts to guide and systematize
this process (Ahmad, Siemon, Gnewuch, & Robra-
Bissantz, 2022; Nißen et al., 2022), comprehensive and
specific guidelines are needed. Our results also show that
the success of conversational agents is not only based on
technological advances in natural language processing,
consistent with previous research on chatbots in other
domains (Schuetzler et al., 2020); success also depends on
sufficiently meeting user’s requirements and preferences
beyond conversational abilities. Contradictions regarding
the personality and interaction style indicate the difficulty
in satisfying the needs of all users, suggesting the use of
personalization (Kocaballi et al., 2019). For example,
designers could provide users with the option to disable the
use of emojis.
Second, a counterintuitive finding was that several
participants liked to interact with Cady by selecting
predefined responses. Intuitively, we would have assumed
that users prefer more human-like communication, which
has also been demonstrated in customer service (Diederich,
Brendel, Lichtenberg, & Kolbe, 2019). Some participants
explained their preference because the predefined
responses were more efficient. It seems that implementing
predefined responses is advantageous for highly structured
dialogues such as teaching behavioral activation. On the
other hand, more flexible dialogues to address individual
issues are complex to implement in a highly structured
way. Future research should address how the advantages of
predefined responses and advanced natural language
processing can be combined to deliver the best user
experience. Third, a high degree of anthropomorphism was
salient in the think-aloud sessions. Participants repeatedly
used pronouns (e.g., he/she) to refer to Cady even though
it was specifically designed to be gender neutral (i.e., a
gender-neutral name and a robot avatar). In addition,
Kuhlmeier et al. Exploring User Experience with a Conversational Agent for Depression
Proceedings of the Twenty-First Annual Pre-ICIS Workshop on HCI Research in MIS, Copenhagen, Denmark, December 11, 2022 5
different personality traits were attributed to Cady. Most
importantly, one participant had the impression that their
message made Cady happy. Interestingly, the diverse
statements indicate that there are different ways in which
anthropomorphism functions. Personalization appears to
be an appropriate strategy to address these different ways
of increasing anthropomorphism.
Third, our participants showed different perceptions of
conversational capabilities. Although one participant was
impressed that Cady understood what they were saying,
others criticized inadequate responses. Human
characteristics, such as expectations and previous
experience with conversational agents, may influence the
user experience. In line with the agenda for conversational
agent research developed by Diederich et al. (2022), future
research should investigate how user characteristics
influence the interaction with conversational agents.
LIMITATIONS
Our study has three main limitations. First, our sample
consists predominantly of female participants. Although
depression is substantially more prevalent among women
(Seedat et al., 2009), future studies need to include male
participants to provide a more holistic understanding of
youth’s user experience with conversational agents to treat
depression. Second, our conversational agent prototype
focused on behavioral activation, which is only one part of
cognitive behavioral therapy. Although most results appear
generalizable, think-aloud data on other therapeutic
exercises is needed to test generalizability. Lastly, Think-
aloud has proven valuable in providing rich and detailed
data, but does provide an overall assessment. Future
research should therefore complement a think-aloud
approach with open-ended questions, established user
experience questionnaires and interaction data.
CONCLUSION
This article provides insight into how youth with symptoms
of depression interact with a conversational agent designed
to deliver therapeutic content. Our results suggest that the
participants enjoyed the interaction and discussed (1)
Cady’s personality and interaction style, (2) its
functionality, and (3) the dialogue content. Our next step
will be to fully analyze and publish our study on needs and
preferences for a conversational agent to treat depression
in youth. In our future research, we will extend the
prototype to several components of cognitive behavioral
therapy and complement the think-aloud with semi-
structured interviews and user experience surveys to follow
a mixed methods approach.
REFERENCES
1. Abd-Alrazaq, A. A., Rababeh, A., Alajlani, M.,
Bewick, B. M., & Househ, M. (2020). Effectiveness
and Safety of Using Chatbots to Improve Mental
Health: Systematic Review and Meta-Analysis.
Journal of Medical Internet Research, 22(7).
2. Ahmad, R., Siemon, D., Gnewuch, U., & Robra-
Bissantz, S. (2022). Designing Personality-Adaptive
Conversational Agents for Mental Health Care.
Information Systems Frontiers.
3. Andone, I., Blaszkiewicz, K., Eibes, M., Trendafilov,
B., Montag, C., & Markowetz, A. (2016). How age
and gender affect smartphone usage. In Proceedings
of the 2016 ACM international joint conference on
pervasive and ubiquitous computing: adjunct (pp. 9
12).
4. Auerbach, R. P., Webb, C. A., & Stewart, J. G. (2016).
Cognitive behavior therapy for depressed adolescents:
A practical guide to management and treatment.
Routledge.
5. Chan, W. W., Fitzsimmons-Craft, E. E., Arielle C.
Smith, Firebaugh, M. L., Fowler, L. A., Bianca
DePietro, … Jacobson, N. C. (2022). The Challenges
in Designing a Prevention Chatbot for Eating
Disorders: Observational Study.
6. Clayborne, Z. M., Varin, M., & Colman, I. (2019).
Systematic Review and Meta-Analysis: Adolescent
Depression and Long-Term Psychosocial Outcomes.
Journal of the American Academy of Child &
Adolescent Psychiatry, 58(1), 7279.
7. Dale, R. (2016). The return of the chatbots. Natural
Language Engineering, 22(5), 811817.
8. Darcy, A., Daniels, J., Salinger, D., Wicks, P., &
Robinson, A. (2021). Evidence of Human-Level
Bonds Established With a Digital Conversational
Agent: Cross-sectional, Retrospective Observational
Study. JMIR Formative Research, 5(5), e27868.
9. Diederich, S., Brendel, A. B., Lichtenberg, S., &
Kolbe, L. (2019). Design for fast request fulfillment or
natural interaction? Insights from an experiment with
a conversational agent.
10. Diederich, S., Brendel, A., Morana, S., & Kolbe, L.
(2022). On the Design of and Interaction with
Conversational Agents: An Organizing and Assessing
Review of Human-Computer Interaction Research.
Journal of the Association for Information Systems,
23(1), 96138.
11. Følstad, A., & Brandtzæg, P. B. (2017). Chatbots and
the new world of HCI. Interactions, 24(4), 3842.
12. Gulliver, A., Griffiths, K. M., & Christensen, H.
(2010). Perceived barriers and facilitators to mental
health help-seeking in young people: a systematic
review. BMC Psychiatry, 10(1), 113.
13. Høiland, C. G., Følstad, A., & Karahasanovic, A.
(2020). Hi, Can I Help? Exploring How to Design a
Mental Health Chatbot for Youths. Human
Technology, 16(2), 139169.
14. Huffman, S. (2014). OMG! Mobile voice survey
reveals teens love to talk. Retrieved October, 14.
Kuhlmeier et al. Exploring User Experience with a Conversational Agent for Depression
Proceedings of the Twenty-First Annual Pre-ICIS Workshop on HCI Research in MIS, Copenhagen, Denmark, December 11, 2022 6
15. Jaspers, M. W. M., Steen, T., Bos, C. van den, &
Geenen, M. (2004). The think aloud method: a guide
to user interface design. International Journal of
Medical Informatics, 73(11), 781795.
16. Kocaballi, A. B., Berkovsky, S., Quiroz, J. C.,
Laranjo, L., Tong, H. L., Rezazadegan, D., … Coiera,
E. (2019). The Personalization of Conversational
Agents in Health Care: Systematic Review. Journal of
Medical Internet Research, 21(11), e15360.
17. Mayring, P. (2004). Qualitative content analysis. A
companion to qualitative research, 1(2), 159176.
18. Nißen, M., Rüegger, D., Stieger, M., Flückiger, C.,
Allemand, M., Wangenheim, F. v, & Kowatsch, T.
(2022). The Effects of Health Care Chatbot Personas
With Different Social Roles on the Client-Chatbot
Bond and Usage Intentions: Development of a Design
Codebook and Web-Based Study. Journal of Medical
Internet Research, 24(4), e32630.
19. Oud, M., de Winter, L., Vermeulen-Smit, E., Bodden,
D., Nauta, M., Stone, L., Stikkelbroek, Y. (2019).
Effectiveness of CBT for children and adolescents
with depression: A systematic review and meta-
regression analysis. European Psychiatry: The
Journal of the Association of European Psychiatrists,
57, 3345.
20. Rice, F., Riglin, L., Lomax, T., Souter, E., Potter, R.,
Smith, D. J., Thapar, A. (2019). Adolescent and
adult differences in major depression symptom
profiles. Journal of Affective Disorders, 243, 175181.
21. Schuetzler, R. M., Grimes, G. M., & Scott Giboney, J.
(2020). The impact of chatbot conversational skill on
engagement and perceived humanness. Journal of
Management Information Systems, 37(3), 875900.
22. Seedat, S., Scott, K. M., Angermeyer, M. C.,
Berglund, P., Bromet, E. J., Brugha, T. S., … Kessler,
R. C. (2009). Cross-national associations between
gender and mental disorders in the World Health
Organization World Mental Health Surveys. Archives
of General Psychiatry, 66(7), 785795.
23. Vaidyam, A. N., Linggonegoro, D., & Torous, J.
(2020). Changes to the Psychiatric Chatbot
Landscape: A Systematic Review of Conversational
Agents in Serious Mental Illness: Changements du
paysage psychiatrique des chatbots: une revue
systématique des agents conversationnels dans la
maladie mentale sérieuse. The Canadian Journal of
Psychiatry, 0706743720966429.
ResearchGate has not been able to resolve any citations for this publication.
Article
Full-text available
Background: The working alliance refers to an important relationship quality between health professionals and clients that robustly links to treatment success. Recent research shows that clients can develop an affective bond with chatbots. However, few research studies have investigated whether this perceived relationship is affected by the social roles of differing closeness a chatbot can impersonate and by allowing users to choose the social role of a chatbot. Objective: This study aimed at understanding how the social role of a chatbot can be expressed using a set of interpersonal closeness cues and examining how these social roles affect clients' experiences and the development of an affective bond with the chatbot, depending on clients' characteristics (ie, age and gender) and whether they can freely choose a chatbot's social role. Methods: Informed by the social role theory and the social response theory, we developed a design codebook for chatbots with different social roles along an interpersonal closeness continuum. Based on this codebook, we manipulated a fictitious health care chatbot to impersonate one of four distinct social roles common in health care settings-institution, expert, peer, and dialogical self-and examined effects on perceived affective bond and usage intentions in a web-based lab study. The study included a total of 251 participants, whose mean age was 41.15 (SD 13.87) years; 57.0% (143/251) of the participants were female. Participants were either randomly assigned to one of the chatbot conditions (no choice: n=202, 80.5%) or could freely choose to interact with one of these chatbot personas (free choice: n=49, 19.5%). Separate multivariate analyses of variance were performed to analyze differences (1) between the chatbot personas within the no-choice group and (2) between the no-choice and the free-choice groups. Results: While the main effect of the chatbot persona on affective bond and usage intentions was insignificant (P=.87), we found differences based on participants' demographic profiles: main effects for gender (P=.04, ηp2=0.115) and age (P<.001, ηp2=0.192) and a significant interaction effect of persona and age (P=.01, ηp2=0.102). Participants younger than 40 years reported higher scores for affective bond and usage intentions for the interpersonally more distant expert and institution chatbots; participants 40 years or older reported higher outcomes for the closer peer and dialogical-self chatbots. The option to freely choose a persona significantly benefited perceptions of the peer chatbot further (eg, free-choice group affective bond: mean 5.28, SD 0.89; no-choice group affective bond: mean 4.54, SD 1.10; P=.003, ηp2=0.117). Conclusions: Manipulating a chatbot's social role is a possible avenue for health care chatbot designers to tailor clients' chatbot experiences using user-specific demographic factors and to improve clients' perceptions and behavioral intentions toward the chatbot. Our results also emphasize the benefits of letting clients freely choose between chatbots.
Article
Full-text available
Millions of people experience mental health issues each year, increasing the necessity for health-related services. One emerging technology with the potential to help address the resulting shortage in health care providers and other barriers to treatment access are conversational agents (CAs). CAs are software-based systems designed to interact with humans through natural language. However, CAs do not live up to their full potential yet because they are unable to capture dynamic human behavior to an adequate extent to provide responses tailored to users’ personalities. To address this problem, we conducted a design science research (DSR) project to design personality-adaptive conversational agents (PACAs). Following an iterative and multi-step approach, we derive and formulate six design principles for PACAs for the domain of mental health care. The results of our evaluation with psychologists and psychiatrists suggest that PACAs can be a promising source of mental health support. With our design principles, we contribute to the body of design knowledge for CAs and provide guidance for practitioners who intend to design PACAs. Instantiating the principles may improve interaction with users who seek support for mental health issues.
Article
Full-text available
Background There are far more patients in mental distress than there is time available for mental health professionals to support them. Although digital tools may help mitigate this issue, critics have suggested that technological solutions that lack human empathy will prevent a bond or therapeutic alliance from being formed, thereby narrowing these solutions’ efficacy. Objective We aimed to investigate whether users of a cognitive behavioral therapy (CBT)–based conversational agent would report therapeutic bond levels that are similar to those in literature about other CBT modalities, including face-to-face therapy, group CBT, and other digital interventions that do not use a conversational agent. Methods A cross-sectional, retrospective study design was used to analyze aggregate, deidentified data from adult users who self-referred to a CBT-based, fully automated conversational agent (Woebot) between November 2019 and August 2020. Working alliance was measured with the Working Alliance Inventory-Short Revised (WAI-SR), and depression symptom status was assessed by using the 2-item Patient Health Questionnaire (PHQ-2). All measures were administered by the conversational agent in the mobile app. WAI-SR scores were compared to those in scientific literature abstracted from recent reviews. Results Data from 36,070 Woebot users were included in the analysis. Participants ranged in age from 18 to 78 years, and 57.48% (20,734/36,070) of participants reported that they were female. The mean PHQ-2 score was 3.03 (SD 1.79), and 54.67% (19,719/36,070) of users scored over the cutoff score of 3 for depression screening. Within 5 days of initial app use, the mean WAI-SR score was 3.36 (SD 0.8) and the mean bond subscale score was 3.8 (SD 1.0), which was comparable to those in recent studies from the literature on traditional, outpatient, individual CBT and group CBT (mean bond subscale scores of 4 and 3.8, respectively). PHQ-2 scores at baseline weakly correlated with bond scores (r=−0.04; P<.001); however, users with depression and those without depression had high bond scores of 3.45. Conclusions Although bonds are often presumed to be the exclusive domain of human therapeutic relationships, our findings challenge the notion that digital therapeutics are incapable of establishing a therapeutic bond with users. Future research might investigate the role of bonds as mediators of clinical outcomes, since boosting the engagement and efficacy of digital therapeutics could have major public health benefits.
Article
Full-text available
Conversational agents (CAs), described as software with which humans interact through natural language, have increasingly attracted interest in both academia and practice, due to improved capabilities driven by advances in artificial intelligence and, specifically, natural language processing. CAs are used in contexts like people's private life, education, and healthcare, as well as in organizations, to innovate and automate tasks, for example in marketing and sales or customer service. In addition to these application contexts, such agents take on different forms concerning their embodiment, the communication mode, and their (often human-like) design. Despite their popularity, many CAs are not able to fulfill expectations and to foster a positive user experience is a challenging endeavor. To better understand how CAs can be designed to fulfill their intended purpose, and how humans interact with them, a multitude of studies focusing on human-computer interaction have been carried out. These have contributed to our understanding of this technology. However, currently a structured overview of this research is missing, which impedes the systematic identification of research gaps and knowledge on which to build on in future studies. To address this issue, we have conducted an organizing and assessing review of 262 studies, applying a socio-technical lens to analyze CA research regarding the user interaction, context, agent design, as well as perception and outcome. We contribute an overview of the status quo of CA research, identify four research streams through a cluster analysis, and propose a research agenda comprising six avenues and sixteen directions to move the field forward.
Article
Full-text available
Chatbots represent new opportunities for low-threshold preventive mental health support to youths. To provide needed knowledge regarding how to design chatbots for this purpose, we present an exploratory design study where we designed and evaluated a prototype chatbot to complement the work of school nurses in the school health service. The prototype was designed with particular regard for preventive mental health support. The design process involved school nurses, digital health workers, and youths. Through user insight activities, we identified four types of support to be provided through the chatbot: informational, relational, processual, and referral. We explored these four types of support through concept development and prototyping. These results are discussed as a potential basis for a framework for understanding youths’ needs regarding chatbots for preventive mental health support. When discussing the study findings, we point out how the study contributes to theory and practice and suggest avenues for future research.
Article
Full-text available
Background: The global shortage of mental health workers has prompted the utilization of technological advancements, such as chatbots, to meet the needs of people with mental health conditions. Chatbots are systems that are able to converse and interact with human users using spoken, written, and visual language. While numerous studies have assessed the effectiveness and safety of using chatbots in mental health, no reviews have pooled the results of those studies. Objective: This study aimed to assess the effectiveness and safety of using chatbots to improve mental health through summarizing and pooling the results of previous studies. Methods: A systematic review was carried out to achieve this objective. The search sources were 7 bibliographic databases (eg, MEDLINE, EMBASE, PsycINFO), the search engine “Google Scholar,” and backward and forward reference list checking of the included studies and relevant reviews. Two reviewers independently selected the studies, extracted data from the included studies, and assessed the risk of bias. Data extracted from studies were synthesized using narrative and statistical methods, as appropriate. Results: Of 1048 citations retrieved, we identified 12 studies examining the effect of using chatbots on 8 outcomes. Weak evidence demonstrated that chatbots were effective in improving depression, distress, stress, and acrophobia. In contrast, according to similar evidence, there was no statistically significant effect of using chatbots on subjective psychological wellbeing. Results were conflicting regarding the effect of chatbots on the severity of anxiety and positive and negative affect. Only two studies assessed the safety of chatbots and concluded that they are safe in mental health, as no adverse events or harms were reported. Conclusions: Chatbots have the potential to improve mental health. However, the evidence in this review was not sufficient to definitely conclude this due to lack of evidence that their effect is clinically important, a lack of studies assessing each outcome, high risk of bias in those studies, and conflicting results for some outcomes. Further studies are required to draw solid conclusions about the effectiveness and safety of chatbots. Trial Registration: PROSPERO International Prospective Register of Systematic Reviews CRD42019141219; https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42019141219
Article
Full-text available
Background: The personalization of conversational agents with natural language user interfaces is seeing increasing use in health care applications, shaping the content, structure, or purpose of the dialogue between humans and conversational agents. Objective: The goal of this systematic review was to understand the ways in which personalization has been used with conversational agents in health care and characterize the methods of its implementation. Methods: We searched on PubMed, Embase, CINAHL, PsycInfo, and ACM Digital Library using a predefined search strategy. The studies were included if they: (1) were primary research studies that focused on consumers, caregivers, or health care professionals; (2) involved a conversational agent with an unconstrained natural language interface; (3) tested the system with human subjects; and (4) implemented personalization features. Results: The search found 1958 publications. After abstract and full-text screening, 13 studies were included in the review. Common examples of personalized content included feedback, daily health reports, alerts, warnings, and recommendations. The personalization features were implemented without a theoretical framework of customization and with limited evaluation of its impact. While conversational agents with personalization features were reported to improve user satisfaction, user engagement and dialogue quality, the role of personalization in improving health outcomes was not assessed directly. Conclusions: Most of the studies in our review implemented the personalization features without theoretical or evidence-based support for them and did not leverage the recent developments in other domains of personalization. Future research could incorporate personalization as a distinct design factor with a more careful consideration of its impact on health outcomes and its implications on patient safety, privacy, and decision-making.
Conference Paper
Full-text available
Conversational agents continue to permeate our lives in different forms, such as virtual assistants on mobile devices or chatbots on websites and social media. The interaction with users through natural language offers various aspects for researchers to study as well as application domains for practitioners to explore. In particular their design represents an interesting phenomenon to investigate as humans show social responses to these agents and successful design remains a challenge in practice. Compared to digital human-to-human communication, text-based conversational agents can provide complementary, preset answer options with which users can conveniently and quickly respond in the interaction. However, their use might also decrease the perceived human likeness and social presence of the agent as the user does not respond naturally by thinking of and formulating a reply. In this study, we conducted an experiment with N=80 participants in a customer service context to explore the impact of such elements on agent anthropomorphism and user satisfaction. The results show that their use reduces perceived humanness and social presence yet does not significantly increase service satisfaction. On the contrary, our findings indicate that preset answer options might even be detrimental to service satisfaction as they diminish the natural feel of human-CA interaction.
Article
Conversational agents (CAs)—frequently operationalized as chatbots—are computer systems that leverage natural language processing to engage in conversations with human users. CAs are often operationalized as chatbots which are used for many applications including technical support, customer service, and digital personal assistants. Despite their widespread use, little research to date has investigated how improving the conversational skill of a CA impacts user perceptions of the agent. To elucidate this relationship, this research uses Social Presence Theory to describe how conversational skill influences perceived social presence and ultimately anthropomorphism of a chatbot. We conducted a series of studies in which 450 participants interacted with CAs exhibiting varying levels of conversational skill. We show that people perceive a more skilled CA to be more socially present and anthropomorphic than a less skilled CA. This research advances the knowledge of computer-human interface in information systems, as CA research to date has largely focused on the technical challenges rather than the behavioral questions of how users interact with CAs.
Article
Objective The need for digital tools in mental health is clear, with insufficient access to mental health services. Conversational agents, also known as chatbots or voice assistants, are digital tools capable of holding natural language conversations. Since our last review in 2018, many new conversational agents and research have emerged, and we aimed to reassess the conversational agent landscape in this updated systematic review. Methods A systematic literature search was conducted in January 2020 using the PubMed, Embase, PsychINFO, and Cochrane databases. Studies included were those that involved a conversational agent assessing serious mental illness: major depressive disorder, schizophrenia spectrum disorders, bipolar disorder, or anxiety disorder. Results Of the 247 references identified from selected databases, 7 studies met inclusion criteria. Overall, there were generally positive experiences with conversational agents in regard to diagnostic quality, therapeutic efficacy, or acceptability. There continues to be, however, a lack of standard measures that allow ease of comparison of studies in this space. There were several populations that lacked representation such as the pediatric population and those with schizophrenia or bipolar disorder. While comparing 2018 to 2020 research offers useful insight into changes and growth, the high degree of heterogeneity between all studies in this space makes direct comparison challenging. Conclusions This review revealed few but generally positive outcomes regarding conversational agents’ diagnostic quality, therapeutic efficacy, and acceptability, which may augment mental health care. Despite this increase in research activity, there continues to be a lack of standard measures for evaluating conversational agents as well as several neglected populations. We recommend that the standardization of conversational agent studies should include patient adherence and engagement, therapeutic efficacy, and clinician perspectives.