Conference PaperPDF Available

Designing a Chatbot Social Cue Configuration System

Authors:

Abstract and Figures

Social cues (e.g., gender, age) are important design features of chatbots. However, choosing a social cue design is challenging. Although much research has empirically investigated social cues, chatbot engineers have difficulties to access this knowledge. Descriptive knowledge is usually embedded in research articles and difficult to apply as prescriptive knowledge. To address this challenge, we propose a chatbot social cue configuration system that supports chatbot engineers to access descriptive knowledge in order to make justified social cue design decisions (i.e., grounded in empirical research). We derive two design principles that describe how to extract and transform descriptive knowledge into a prescriptive and machine-executable representation. In addition, we evaluate the prototypical instantiations in an exploratory focus group and at two practitioner symposia. Our research addresses a contemporary problem and contributes with a generalizable concept to support researchers as well as practitioners to leverage existing descriptive knowledge in the design of artifacts.
Content may be subject to copyright.
This is the author’s version of a work that was published in the following source
Feine, J., Morana S. and Maedche A. (2019). “Designing a Chatbot Social Cue Configuration
System”. In Proceedings of the 40th International Conference on Information Systems (ICIS).
Munich: AISel.
Please note: Copyright is owned by the author and / or the publisher.
Commercial use is not allowed.
Institute of Information Systems and Marketing (IISM)
Fritz-Erler-Strasse 23
76133 Karlsruhe - Germany
http://iism.kit.edu
Karlsruhe Service Research Institute (KSRI)
Kaiserstraße 89
76133 Karlsruhe Germany
http://ksri.kit.edu
© 2017. This manuscript version is made available under the CC-
BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-
nc-nd/4.0/
Designing a Chatbot Social Cue Configuration System
Fortieth International Conference on Information Systems, Munich 2019 1
Designing a Chatbot Social Cue Configuration
System
Completed Research Paper
Jasper Feine
Karlsruhe Institute of Technology (KIT)
Karlsruhe, Germany
jasper.feine@kit.edu
Stefan Morana
Karlsruhe Institute of Technology (KIT)
Karlsruhe, Germany
stefan.morana@kit.edu
Alexander Maedche
Karlsruhe Institute of Technology (KIT)
Karlsruhe, Germany
alexander.maedche@kit.edu
Abstract
Social cues (e.g., gender, age) are important design features of chatbots. However,
choosing a social cue design is challenging. Although much research has empirically
investigated social cues, chatbot engineers have difficulties to access this knowledge.
Descriptive knowledge is usually embedded in research articles and difficult to apply as
prescriptive knowledge. To address this challenge, we propose a chatbot social cue
configuration system that supports chatbot engineers to access descriptive knowledge in
order to make justified social cue design decisions (i.e., grounded in empirical research).
We derive two design principles that describe how to extract and transform descriptive
knowledge into a prescriptive and machine-executable representation. In addition, we
evaluate the prototypical instantiations in an exploratory focus group and at two
practitioner symposia. Our research addresses a contemporary problem and contributes
with a generalizable concept to support researchers as well as practitioners to leverage
existing descriptive knowledge in the design of artifacts.
Keywords: configuration system, chatbot, social cue, descriptive knowledge, prescriptive
knowledge, design science research
Introduction
Conversational agents (CAs) are currently a popular technology in research and practice (Gartner 2018).
CAs are software-based systems that enable users to interact with information and communications
technologies through the use of text- and speech-based natural language (Dale 2016). In addition to popular
speech-based CAs such as Amazon’s Alexa or Apple’s Siri, many companies deployed text-based CAs on
their websites, in social media, or enterprise applications. Text-based CAs are often referred to as chatbots
as they communicate via text messages (i.e., chat) in an autonomous manner (i.e., bot) (Dale 2016; McTear
2017). Chatbots are available 24 hours a day, 7 days a week, run cost effectively, and are used in various
domains such as in medical-advisory, customer service, or work collaboration (Følstad and Brandtzæg
2017; Gartner 2018). However, the acceptance of chatbots is growing more slowly than expected
(Brandtzaeg and Følstad 2018). Many human-chatbot interactions do not feel natural and many chatbots
have already disappeared from the market (Brandtzaeg and Følstad 2018; Mimoun et al. 2012). The reasons
for these failures are manifold. The design of natural dialogues remains a major challenge, as many
unforeseen variables influence natural language interactions. However, not only poor dialog capabilities
are reasons for chatbot failures. Various verbal (e.g., small talk), visual (e.g., avatar), or invisible (e.g.
Designing a Chatbot Social Cue Configuration System
Fortieth International Conference on Information Systems, Munich 2019 2
response time) cues of chatbots can lead to either positive or negative user reactions (Feine et al. 2019a).
Since these cues trigger similar social reactions as in interpersonal communication, scholars often refer to
these cues as social cues (Feine et al. 2019a).
Choosing an appropriate social cue design for a chatbot is a difficult and complex challenge (Følstad and
Brandtzæg 2017). For example, the famous IKEA chatbot called Anna integrated many social cues such as
smiling, small talk, and an attractive female visual appearance. The chatbot engineers made Anna appear
very human-like (Brandtzaeg and Følstad 2018). However, the extensive implementation of social cues in
this specific way influenced several users to primarily talk about the visual appearance of Anna instead of
asking product related questions. As this was not the main objective, IKEA retired Anna in 2016
(Brandtzaeg and Følstad 2018). Despite this failure, it is not always wrong to implement multiple social
cues. For a fitness coach chatbot, for example, research has shown that conversational strategies such as
small talk, praise, and self-disclosure as well as the implementation of several nonverbal social cues (i.e.,
hand gestures, eyebrow raises, gaze cues, posture shifts, head nods) had a positive impact on the chatbot’s
long-term success (Bickmore and Picard 2005). Thus, users are generally more satisfied if a chatbot’s social
cue design fits to the context and task of an interaction (Brandtzaeg and Følstad 2018; Fogg 2002).
Moreover, different user groups (e.g., differences in gender, age, cultural origin) have different preferences
for a specific chatbot social cue design (Følstad and Brandtzæg 2017). Existing chatbots, however, are
typically set up following a one-size-fits-all approach” (Følstad and Brandtzæg 2017, p. 41). Regardless of
the individual situation and characteristics of the user, everyone receives a similar message and interacts
with the same chatbot (from a design perspective). Consequently, there is a high likelihood that a certain
social cue design is not optimally fitting to the user, task, and context of a human-chatbot interaction (Fogg
2002). This misfit is a challenge for organizations that intend to implement a chatbot for their customer
service or sales.
To address this challenge, chatbot engineers can rely on various support tools such as chatbot mockup tools,
chatbot UI design tools, but also on several high-level design guidelines (Kharkovyna 2018; McTear 2017).
Moreover, considerable research investigated social cues of spoken dialog systems, voice user interfaces,
embodied conversational agents, as well as chatbots since the 1960s (McTear 2017). However, current (tool)
support to access and use descriptive knowledge (i.e., published in the form of experimental studies) in the
design of an artifact is rather limited. As a consequence, chatbot engineers often do not leverage the existing
body of knowledge and thus, are in danger of re-inventing the wheel(McTear 2017, p. 46). Therefore,
our objective is to design a chatbot social cue configuration system that supports chatbot engineers to build
on descriptive knowledge in order to make justified social cue design decisions (i.e., grounded in existing
empirical research). However, it is not well understood how to extract and store existing descriptive
knowledge about social cues as a foundation to design a social cue configuration system. Henceforth, we
address the following research question:
How to design a chatbot social cue configuration system in order to support chatbot engineers in making
justified social cue design decisions?
To answer this research question, we follow a design science research (DSR) approach, because it is useful
to address a real-world challenge (i.e. support chatbot designers in making justified chatbot social cue
decisions) through the iterative creation and evaluation of the proposed software artifact (i.e. chatbot social
cue configuration system) with the respective users (i.e. chatbot engineers). Our research addresses a
contemporary problem of chatbot engineers and contributes with a concept to support researchers as well
as practitioners to access and leverage existing descriptive knowledge in the design of artifacts. More
specifically, we derive two design principles that describe how to extract and transform descriptive
knowledge from empirical research into a prescriptive and machine-executable representation. In addition,
we demonstrate the instantiation of the proposed design in the form of a chatbot social cue configuration
system and evaluate the design in an exploratory focus group as well as at two practitioner symposia.
Conceptual Foundations
Chatbots are Social Actors
Chatbots are software-based systems designed to communicate via natural language (Dale 2016). Although
the naming of these agents is under constant discussion (e.g., CA, chatbot, chatterbot, digital assistant), the
Designing a Chatbot Social Cue Configuration System
Fortieth International Conference on Information Systems, Munich 2019 3
main purpose stays the same, namely chatting with a human in order to achieve some purpose (e.g., retrieve
information, use a service) (Dale 2016). Many companies have recently identified chatbots as a key interface
to enhance user experience (Dale 2016). Examples range from booking chatbots of Domino’s and Taco Bell
(Følstad and Brandtzæg 2017) to various chatbots hosted on Facebook Messenger and to several chatbots
that are used and investigated in customer service (Feine et al. 2019b; Gnewuch et al. 2017). Gartner even
forecasts that 25% of all customer service operations will integrate chatbots by 2020 (Gartner 2018).
When designing chatbots, it becomes important to mind that users react socially towards computers that
use natural language interfaces (i.e., treat the computer as a relevant social entity) (Nass and Moon 2000).
For example, studies have shown that users apply gender stereotypes towards a chatbot based on its
appearance (Brahnam and Angeli 2012) and voice (Nass and Moon 2000). Others showed that humans
judge a chatbots politeness based on its choice of words (Mayer et al. 2006) and predict a chatbot’s
personality based on its language strength, interaction order, and expressed confidence level (Nass et al.
1995). Furthermore, users are more satisfied if a chatbot delays its response in a natural manner (Gnewuch
et al. 2018) and rate the chatbot differently depending on its naming and linguistic style (Araujo 2018). To
describe these phenomena, Nass and colleagues have introduced the Computer are Social Actors (CASA)
paradigm which states that human-computer interaction is fundamentally social and natural (Nass et al.
1994; Nass and Moon 2000). Social reactions are always triggered whenever the computer exhibits a certain
amount of cues that are associated with cues that are used in everyday interpersonal communication (e.g.,
use of natural language, smiling, small talk) (Nass and Moon 2000). Since these cues trigger emotional,
cognitive, or behavioral reactions similar to those reactions normally experienced during interpersonal
communication (Krämer 2005), scholar refer to these cues often as social cues (Feine et al. 2019a). Since
chatbots by definition inhere a key characteristic of interpersonal communication (i.e., use of natural
language), they always elicit some social behavior. As a consequence, chatbots should not only be designed
to be functional, but also to act social (Feine et al. 2019a). This is essential as an inadequate social cue design
can lead to irritated or angry users (Fogg 2002), can diminish the satisfaction of a chatbot (Diederich et al.
2019a), and can finally result in its rejection (Mimoun et al. 2012).
To create an appropriate chatbot social cue design, chatbot engineers must consider several influencing
factors that determine how users react to social cues of a chatbot (see Table 1 for a non-exhaustive overview)
(Nass and Moon 2000). For example, research has shown that users perceive small talk of a chatbot
differently according to their cultural background (Endrass et al. 2010), prefer visual appearances of
chatbots that correspond to task related gender stereotypes (Forlizzi et al. 2007), and prefer interactions
with a formal looking chatbot in a banking context but not in an entertainment context (McBreen 2002).
Therefore, it is highly important to consider several influences in the design of a chatbot in order to trigger
a desired user reaction and to avoid possible backfiring effects.
Social Cue (Design Characteristics)
Influencing Factor (Characteristics)
User Reactions Towards
Social Cues
Reference
Clothing (formal/ informal)
Interaction context (banking/
entertainment/ travel)
Impact on attitude
towards chatbot
McBreen (2002)
Clothing (casual dress/ conservative
dress),
Degree of human-likeness (cartoon/
animal),
Age (young/ old)
Product type (professional product/
casual product)
Impact on preferences
Keeling et al. (2004)
Politeness of statements (guarded
suggestions/ commands)
Computing experience of user
(novice/ expert)
Impact on perceived
politeness
Mayer et al. (2006)
Small talk behavior (many topic shifts/
sequential topics)
Cultural origin of user (Asian/
Western)
Impact on chatbot
evaluation
Endrass et al. (2010)
Ethnicity (Caucasian, African
American, Asian)
Ethnicity of user (same as chatbot/
different than chatbot)
Impact on preferences
Cowell and Stanney
(2005)
Ethnicity (Caucasian, Asian)
Ethnicity of user (same as chatbot/
different than chatbot)
Impact on user’s social
experience
Qiu and Benbasat (2010)
Degree of human-likeness (human-like/
disembodied)
Age of user (18-26/ >45 years)
Impact on trust
Pak et al. (2012)
Gender (male/ female)
Topic of conversation (stereotypically
female topic/ stereotypically male topic)
Impact on perceived
competence
Nass and Moon (2000)
Gender (male/ female)
Context (information/ entertainment/
education/ finance/ healthcare)
Impact on user
satisfaction score
Forlizzi et al. (2007)
Designing a Chatbot Social Cue Configuration System
Fortieth International Conference on Information Systems, Munich 2019 4
Gender (male/ female)
Product gender (male/ female)
Impact on credibility and
competence
Beldad et al. (2016)
Gender (male/ female)
Gender of user (male/ female)
Impact on leaning
performance and effort
Kraemer et al. (2016)
Jokes (sexual humor/ ‘nonsense’ humor/
general jokes)
Language proficiency of user (high/
low)
Impact on joke evaluation
Braslavski et al. (2018)
Strength of language (assertive/
submissive),
Interaction order (first/ last),
Confidence (self-confident/ self-doubting)
Dominance of user (dominant/
submissive)
Impact on user
satisfaction
Nass et al. (1995)
Strength of language (confident
assertions/ questions and suggestions),
Postures (wide/close body postures)
Extroversion of user (extroverted/
introverted)
Impact on perceived fun
and likeability
Isbister and Nass (2000)
Chatbot Development Platforms
Chatbot engineers can use several platforms to develop and host chatbot solutions (Diederich et al. 2019b).
These chatbot development platforms vary according to their technical complexity, analytical features, and
degree of standardization (Diederich et al. 2019b). Major technology providers such as IBM, Microsoft, or
Google offer various functionalities to develop chatbots (e.g., natural language understanding solutions,
speech- and text-based user interfaces, and multi-platform support). Beside the big technological players,
many start-ups offer solutions to develop domain specific chatbots (e.g., ManyChat) as well as provide
libraries for natural language understanding (e.g., Rasa). In addition, chatbot engineers can leverage crowd
sourcing approaches for authoring chatbot scripts (Huang et al. 2017), text generators for creating chatbot
dialogs (Klopfenstein et al. 2019), as well as frameworks to define test cases (Treml 2018).
Beside the technical support to develop and host chatbot solutions, some chatbot platforms also support
the design of a limited set of social cues. For example, Google’s platform Dialogflow enables chatbot
engineers to add small talk capabilities to a chatbot including courtesy, emotions, and greetings. In
addition, Microsoft integrated the personality chat module into their Azure bot service which enables
chatbot engineers to include small talk capabilities in accordance with a distinct personality (i.e.,
professional, friendly, humorous). In addition, many platforms enable chatbot engineers to alter social cues
such as avatars, naming, or the font of a chatbot, as well as to implement and modify buttons and hero cards
(Diederich et al. 2019a). However, there is limited tool support that helps chatbot engineers to make chatbot
social cue design decision for a specific interaction context (McTear 2017). Instead, many tools enable
chatbot engineers to create chatbot mockups in order to test and revise early versions of a chatbot without
writing one line of code. Popular mockup tools are Botmock, BotFrame, or Botsociety (Kharkovyna 2018).
These mockup tools enable the user to design interaction flows, revise dialogs, and modify several social
cues such as avatars, names, response times, or backgrounds. Moreover, some mockup tools are specialized
only on the chatbot’s user interface. For example, Layer provides a conversation design kit that includes a
collection of conversational user interface screens for Sketch (Layer 2019). However, none of these support
tools provide explicit chatbot social cue design recommendations for specific types of chatbots and account
for different influencing factors of a human-chatbot interaction such as the context and task. Similarly,
many online articles are published that provide high-level social cue design advices (e.g., don't sound like
a robot) (McTear 2017). However, these guidelines are unfortunately not easy to operationalize and often
do not leverage existing empirical research findings.
Design Science Research Project
Our research aims to design and evaluate a chatbot social cue configuration system that supports chatbot
engineers to access descriptive knowledge in order to make justified social cue design decisions. In this
paper, we report a design cycle following the DSR framework of Kuechler and Vaishnavi (2008).
Awareness of problem: we started the design cycle by investigating the underlying needs and problems
of the relevant stakeholders (i.e., chatbot engineers) in the chatbot social cue design process (Maedche et
al. 2019). To reveal contemporary real-world situations from people in organizations, qualitative interview
studies are an established research method (Myers 2009). We selected chatbot engineers that either
develop chatbots as well as product owners that are responsible for the development of chatbots within a
company. We conducted a series of ten semi-structured expert interviews with chatbot engineers from eight
companies in Europe which exceeds the suitable number of eight experts suggested by Myers (2009).
Designing a Chatbot Social Cue Configuration System
Fortieth International Conference on Information Systems, Munich 2019 5
Demographics, company domains, and the interviewees roles are outlined in Table 2 (note that F.1/F.2 and
H.1/H.2 are working at the same company in different departments).
Table 2. Interviewed Chatbot Engineers
ID
Profession
Company Domain
Working Experience With Chatbots
Age
A.1
Chatbot Business Development
Chatbot Service Provider
3 years
35
B.1
Chatbot Developer
Chatbot Service Provider
3 years
39
C.1
Chatbot Developer
Chatbot Service Provider
1.5 years
23
D.1
Chatbot Developer
IT Service Provider
1 year
24
E.1
Digital Service Consultant
IT Service Provider
2 years
39
F.1
Chatbot Project Manager
Telecommunication
2 years
47
F.2
Digital Service Manager
Telecommunication
2 years
27
G.1
Chatbot Product Owner
Energy Provider
1.5 years
50
H.1
Chatbot Developer
Manufacturer
0.75 year
22
H.2
Chatbot Developer
Manufacturer
1 year
26
Before conducing the interviews, we developed an interview guideline including opening and several open-
ended questions (Myers 2009). The opening questions asked for consent, demographics, profession, and
experience in the development of chatbots. The open-ended questions addressed the challenges that emerge
throughout the design process of a chatbot. Each interview session was conducted over the phone, was
audio recorded, and lasted on average 29 minutes (SD=9.861). After we conducted the interviews, all audio
recordings were imported in MAXQDA and transcribed into text (in total 67 pages). Subsequently, we
followed a previously agreed two-step coding approach (Zhang 2017). First, statements of the interviews
were coded into first order-concepts and subsequently aggregated into second-order concepts (Zhang
2017). The first order concepts relate to all statements that address challenges in the chatbot social cue
design process. Two example statements are it really takes much time to search and collect design
knowledgeand I need to collect the knowledge for designing the chatbot from many different sources”.
In the second step of the coding approach, we grouped all statements into second-order themes describing
the same overarching challenge (Zhang 2017). For example, both previous statements refer to the challenge
of distributed chatbot social cue design knowledge. Therefore, we combined them under the labelscattered
social cue design knowledge”. After labeling and aggregating all statements, we were able to identify
relevant challenges in the chatbot social cue design process.
Suggestion: subsequently, we address the identified challenges and proposed requirements that describe
the goals, scope, and boundaries of our to be designed software artifact (Gregor and Jones 2007). In order
to derive requirements, we followed the requirement articulation template of Rupp (2014). Based on the
requirements, we referred to established literature and derived design principles for the design of our
software artifact following the suggestions of Chandra et al. (2015).
Development: in the development phase, we instantiated the proposed design principles. Subsequently,
we developed a prototypical chatbot social cue configuration system using the prototyping tool Axure RP
8.1.3 in order to evaluate the proposed concept with the respective users.
Evaluation: in the evaluation phase, we pursued a human risk and effectiveness evaluation strategy in
order to evaluate our prototype (Venable et al. 2016). We chose this evaluation strategy, as the major design
risk of the chatbot social cue configuration system is user-oriented (Venable et al. 2016). Therefore, we
assessed whether the system really supports chatbot engineers in making justified social cue design
decisions and whether it reduces the risk of making inappropriate design decisions (Venable et al. 2016).
We evaluated the prototype by conducting an exploratory focus group workshop with chatbot engineers
(Tremblay et al. 2010). The focus group included five participants from a Swiss-German IT service provider
that develops chatbot solutions. The selected company is suitable to validate the prototype as they do both,
develop chatbot solutions based on commercial technologies as well as consult companies during their
chatbot development projects. In total, three female and two male participants with an average age of 35.2
years (SD=11.39), an average working experience of 9.7 years (SD=11.61), and an average chatbot
development experience of 1.5 years (SD=0.447) participated in the workshops. We demonstrated the
prototype and asked the participants to evaluate the design by using a SWOT analysis (i.e., strength,
weaknesses, opportunities, threats of our proposed design). The complete workshop was audio recorded,
one of the authors moderated, and another one took notes.
Validate research relevance to practice: finally, we extended the evaluation phase to further validate
the research relevance to practice (Rosemann and Vessey 2008). By doing so, we can assess whether
Designing a Chatbot Social Cue Configuration System
Fortieth International Conference on Information Systems, Munich 2019 6
potential users of the system would actually use and internalize the research findings. Therefore, we
followed the approach of Rosemann and Vessey (2008) and assessed the applicability of our research. In
addition, we followed the approach of Ho and Lim (2018) and assessed the applicability of our research at
two practitioner symposia in order to receive feedback from a variety of potential users. Therefore, we
presented a revised web version of our chatbot social cue configuration system prototype to participants
from practice and academia at two yearly practitioner symposia in Germany. Both symposia focus on
usability, user experience, and on new and innovative user interfaces such as chatbots. At both symposia,
we demonstrated how the system can be used to make justified chatbot social cue design decisions for a
particular scenario and then assessed the applicability of the system (Rosemann and Vessey 2008). To
collect the demographics (N1=71, N2=74) and all quantitative data during the symposia, we used
Mentimeter (www.mentimeter.com) which is an online audience response system that allows participants
to share their opinions and thoughts conveniently via their smartphones. We assessed the three dimensions
of relevance that are critical for practitioners to internalize research findings, namely importance,
accessibility, and suitability of the artifact (Rosemann and Vessey 2008). Therefore, we used established
scales following Klein et al. (2006) on a five-point sematic differential (i.e., (1) not at all (5) without any
doubt). In addition, we assessed the practitioner’s intention to use the system (“I intent to use the chatbot
social cue configuration system to design chatbots”) on a ten-point semantic differential (i.e., (1) not agree
(10) agree) (Ajzen 2006) and we also assessed the net promoter score (NPS) (“how likely is it that you
recommend the system to a colleague”) on a ten-point semantic differential (i.e., (1) not at all likely - (10)
extremely likely) (Reichheld 2004). Moreover, we engaged in an open discussion during the presentations
and in personal discussions after the presentations (Ho and Lim 2018).
Designing a Chatbot Social Cue Configuration System
Awareness of Problem
After we had analyzed the ten semi-structured expert interviews, we identified four key challenges (C1-C4)
that chatbot engineers face when making chatbot social cue design decisions. First, several chatbot
engineers reported that they have to select within a large range of different social cues which makes design
decisions very complex (C1). One product owner mentioned, “You have to select from the whole complexity
of the human communication system and you have to make many decisions.” Another chatbot engineer
said, Everything that exists in the UI can be personalized: you can select the colors, change the fonts,
logos, avatars, and much more, so basically you can personalize almost everything. Second, many
stakeholders are involved in the chatbot social cue design process leading to diverging expectations and
various competing ideas on how to make the correct design decision (C2). However, these decisions are
often not grounded on research findings or user testing. Instead, they only adapt to the corporate identity.
One chatbot developer mentioned, “There are many ideas how a design can work or how it has to look
like. But this is always biased as we are chatbot developers and there are always some design ideas which
are irrelevant or wrong for the real user.” One chatbot solution provider stated, “Companies often face the
challenge that they do not know how to convey their content in a conversational interaction, but they give
us many strict rules and guidelines that we must follow. Third, chatbot engineers have to account for
several influencing factors in order to decide on the final social cue design (C3). One interviewee stated,
“Chatbots can be young, dynamic, funny, friendly, but many chatbots do not need these features.” Another
one stated, “There are people who find it creepy when they talk to a computer and the avatar looks and
talks like a human. But there are others who find it very interesting when a computer seems very human-
like. How the decision should be made? You probably can't say in general; it really depends.” Fourth,
chatbot social cue design knowledge is scattered across various sources and difficult to access (C4). Chatbot
engineers inform their design based on best practices, workshops, guidelines from the internet, and insights
from scientific literature. One interviewee mentioned, “It really takes much time to search and collect
design knowledge.” Another one stated, “I need to collect the knowledge for designing the chatbot from
many different sources,” and thus, one interviewee mentioned, “A more collated platform would have
helped me especially in the beginning of my job.”
Suggestion
To address the identified challenges, we suggested to design a system that reduces decision complexity
within the large selection of potential social cue design characteristics (C1) and reduces deviations in design
Designing a Chatbot Social Cue Configuration System
Fortieth International Conference on Information Systems, Munich 2019 7
expectations (C2). In this context, research on decision aids and decision support systems (DSS) showed
that these systems improve decision quality and reduce decision complexity (Al-Natour and Benbasat 2009;
Limayem and DeSanctis 2000). Moreover, decisional guidance (i.e., enrichment of decision models that
direct decision makers towards successful structuring and execution of model components) has shown to
support group decisions making (Limayem and DeSanctis 2000). Consequently, a chatbot social cue
configuration system should provide chatbot engineers with decisional guidance to recommend social cue
designs that trigger a desired user reaction (R1). Additionally, a chatbot social cue configuration system
should simplify access to existing knowledge (C4). This is from utmost importance, as decision quality
strongly depends on the quality of the available information (Sauter 2014). To ensure that decisions are
justified (i.e., grounded in existing empirical research), a chatbot social cue configuration system should
store descriptive social cue knowledge from scientific publications (R2). Moreover, the system should help
chatbot engineers to account for various influencing factors that impact the user reactions towards a chatbot
social cue design (C3). In this context, Catrambone et al. (2004) propose to account for eleven features of
the user and additional eleven features of the context and task, which makes a chatbot social cue design
decision highly complex. To reduce the decision complexity, a chatbot social cue configuration system
should account for all potential influencing factors and then provide a suitable social cue design
recommendation (R3). Moreover, a chatbot social cue configuration system should collate scattered chatbot
social cue design knowledge (C4). However, a typical knowledge management problem is the difficulty to
access and extract knowledge that is captured in unstructured text-based data such as in scientific
publications (C4) (Davenport et al. 1998). In this context, various communities (e.g., knowledge
engineering, description logics, logic-based databases) investigated approaches to structure and capture
knowledge in a machine-executable representation for decades (Berners-Lee et al. 2001). As a consequence,
a chatbot social cue configuration system should store the existing knowledge in a machine-executable
representation in order to reduce difficulties in applying it in the design of a chatbot (R4).
Based on the articulated requirements, we propose two design principles. First, a system should be capable
to support chatbot engineers in making justified social cue design decision (R1) that are based on existing
empirically-grounded knowledge (R2) and account for different influencing factors (i.e., variations in user,
task, and context) of a human-chatbot interaction (R3). Therefore, we propose that a configuration system
should store all relevant knowledge pieces from the descriptive knowledge base that are necessary to make
prescriptive social cue design recommendations. To identify these knowledge pieces, we follow the
suggestions of Chandra et al. (2015) who propose to include three knowledge pieces (KPs) to articulate
purposeful prescriptive knowledge. These include (KP1) information about the properties making an action
possible, (KP2) boundary conditions for when it applies, and (KP3) the actions made possible through the
artifact. By extracting these information, we can define actionable prescriptive design knowledge in the
form of prescriptive design rules (Baskerville and Pries-Heje 2014). These prescriptive design rules can
capture relevant knowledge in order to achieve a desired outcome under some uncertainty (e.g., to achieve
X and you believe you are in Y, then design the social cue like Z) (Baskerville and Pries-Heje 2014). Thus,
we derive the following DP: (DP1) a chatbot social cue configuration system should provide theory grounded
social cue design recommendations based on prescriptive design rules in order to support chatbot engineers
to make justified social cue design decisions. To address R2 and R4, we propose to simplify access to the
existing knowledge by leveraging computing resources (Berners-Lee et al. 2001). Therefore, we need to
store descriptive knowledge in a machine-executable expression of meanings and concepts (Berners-Lee et
al. 2001). In this context, ontologies are considered as an important enabler to establish a shared
understanding as well as to enable sharing, interoperability, and (re-)use of knowledge (Staab and Studer
2009). Ontologies are conceptual models that provide a controlled vocabulary to describe a set of concepts
and relations of a domain (Staab and Studer 2009). Based on the ontological conceptualization, a
knowledge base is a repository that links classes in the ontology to individual instances. Since, ontological
approaches have been successfully applied in many domains, we derive the following DP: (DP2) a chatbot
social cue configuration system should store prescriptive design rules in a machine-executable
representation using ontologies and a knowledge base in order to enable sharing, interoperability, and
(re-)use of descriptive social cue knowledge.
Designing a Chatbot Social Cue Configuration System
Fortieth International Conference on Information Systems, Munich 2019 8
Development
To instantiate both design principles in the form of a software artifact that supports chatbot engineers in
making justified social cue design decisions, we followed a three-step development approach (Feine et al.
2019c). Both steps are outlined in more detail below.
In the first development step, we transformed descriptive knowledge from scientific publications into
prescriptive design rules (implementing DP1). To do so, we identified an initial set of descriptive social cue
knowledge by applying a predefined search term (i.e., chatbots synonyms AND social cue synonyms) to
three databases (i.e., EBSCOhost, Web of Science, IEEEexplore). We revealed an initial list of 92 relevant
publications that investigate user reactions towards specific social cue designs (see Feine et al. 2019a).
Subsequently, we extracted relevant knowledge pieces that are relevant in order to define prescriptive social
cue design rules following the suggestions of Chandra et al. (2015). Therefore, we extracted three relevant
knowledge pieces, namely (KP1) social cue design characteristics, (KP2) influencing factors, and (KP3) the
corresponding user reactions. Two examples that demonstrate how to extract the relevant knowledge pieces
in order to derive prescriptive design rules are displayed in Figure 1 (Feine et al. 2019c). In the first example,
McBreen (2002) investigates (KP1) one social cue design (i.e., clothing) influenced by (KP2) three different
interaction contexts (e.g., banking, entertainment, travel) in order to (KP3) investigate the corresponding
user reactions towards the chatbot (i.e., attitude towards chatbot). Based on this research, we were able to
derive prescriptive design rules on how to select the clothing of a chatbot depending on the interaction
context (e.g., in a banking context design clothing of chatbot formal in order to trigger a positive attitude
towards the chatbot). In the second example, Nass et al. (1995) showed that a (KP1) specific set of social
cues (i.e., interaction order, confidence level, strength of language) lead users to assume that the chatbot
possesses a specific personality (i.e., extroverted, introverted). Depending on the (KP2) personality of the
users (i.e., extroverted, introverted), (KP3) they are more or less satisfied with a chatbot. Thus, we derived
the prescriptive design rules that the design of the interaction order, confidence level, and strength of
language should match the personality of the user to maximize satisfaction.
By following this approach, we iteratively derived a list of (KP1) 48 distinct social cues, (KP2) 18 influencing
factors, and (KP3) 192 investigated user reactions towards specific social cue designs. To create an
impression about the complexity of manually making a social cue design decision, Table 3 displays the
prescriptive design rules that result in a positive user reaction for selecting the design of only one social cue,
namely the gender of a chatbot.
Social Cue: Design
Characteristic (KP1)
Influencing Factor: Characteristic
(KP2)
User Reaction Towards Social Cue (KP3)
Reference
Gender: female
Conversation topic: stereotypical female
topic
Positive impact on informative rating
Nass and Moon (2000)
Gender: male
Conversation topic: stereotypical male topic
Positive impact on informative rating
Nass and Moon (2000)
Gender: male
Feedback type: positive and negative
evaluation
Positive impact on perceived friendliness
Nass and Moon (2000)
Gender: female
Frustration of user: high due to errors
Increases impact of affective excuses to
reduce frustration
Hone (2006)
Gender: male
Chatbot role: stereotypical male jobs
Positive impact on satisfaction score
Forlizzi et al. (2007)
Gender: female
Chatbot role: stereotypical female jobs
Positive impact on satisfaction score
Forlizzi et al. (2007)
Gender: female
Chatbot role: Q&A agent
Positive impact on comfort, confidence,
and enjoyment
Niculescu et al. (2010)
Social Cue (KP1) Influencing Factor (KP2) User Reactio n (KP3)Design C haracteristic (KP1)Ref.
Context:Tra vel
Context: Entertain.
Clothing of c hatbot
Formal clot hing
Informa l clothing
Context: Banking
Positive attitute toward s chatbot
Negative a ttitude towa rds chatbo t
McBreen
(2002)
User: Int roverte d
Interaction order First / High / Str ong
Last / Low / Weak
User: Extrov erted
More satisfied
Less satisfied
Language strength
Confidence
Nass et a l.
(1995)
Designing a Chatbot Social Cue Configuration System
Fortieth International Conference on Information Systems, Munich 2019 9
Gender: male
Chatbot role: job interview
Higher perceived power, trust, and
expertise
Nunamaker et al. (2011)
Gender: female
Chatbot role: job interview
Higher perceived likeability
Nunamaker et al. (2011)
Gender: male
Platform: website
Less attribution of negative stereotypes
Brahnam and Angeli
(2012)
Gender: male
User Gender: female
Chatbot role: tutor bot
Positive impact on learning performance
and effort
Kraemer et al. (2016)
Gender: female
User Gender: male
Chatbot role: tutor bot
Positive impact on learning performance
and effort
Kraemer et al. (2016)
Gender: male
Chatbot role: tutor bot;
Feedback type: negative
Positive impact on learning performance
Hayashi (2016)
Gender: female
Chatbot role: tutor bot;
Feedback type: positive
Positive impact on learning performance
Hayashi (2016)
Gender: female
Context: sales;
Product gender: female
Positive belief in the credibility of advice
and competence of agent
Beldad et al. (2016)
Gender: male
Context: sales;
Product gender: male
Positive belief in the credibility of advice
and competence of agent
Beldad et al. (2016)
Gender: female
Chatbot role: job interview
User Personality: agreeable & trusting users
Higher willingness to listen to chatbot
persona
Li et al. (2017)
Gender: male
Chatbot role: job interview
Personality: extraverted & neurotic users
Reduces the probability that the user
inflates himself
Li et al. (2017)
In the second development step, we stored the prescriptive design rules in a machine-executable
representation using ontologies and a knowledge base in order to enable sharing, interoperability, and
(re-)use of knowledge (implementing DP2). We followed established guidelines (Ostrowski et al. 2014) and
defined the classes of the domain (i.e., (KP1) social cue, (KP2) influencing factor, (KP3) user reaction). Next,
we defined subclasses for each identified social cue design (e.g., chatbot clothing, chatbot gender),
influencing factor (e.g., user age, task complexity, context domain), and user reaction (e.g., attitude towards
the chatbot, user satisfaction). Finally, we defined properties that describe attributes of instances of th e
classes and the relations to other instances (i.e., has social cue, has influencing factor, and has user
reaction). In the next step, we developed a knowledge base by creating several instances that capture the
empirical study findings. Therefore, we reviewed each prescriptive design rules and identified instances of
each rule by reviewing the defined knowledge pieces (e.g., clothing: formal; context: banking, attitude
towards the chatbot: positive). Next, we defined instances of the prescriptive design rule class, each of which
holds specific properties concerning a particular rule (e.g., rule 1 has clothing formal, has context banking,
and has a positive attitude towards the chatbot). Based on this, we were able to query the instantiated
knowledge base in order to list all prescriptive design rules that fulfill a specific configuration condition.
For example, select all social cue design characteristics from the knowledge base where the context
characteristic is banking and the attitude towards the chatbot is positive. The resulting output includes all
social cue design characteristics that meet the query conditions (e.g., chatbot clothing should be formal
according to rule 1 and chatbot age should be old according to rule 3). However, it must be noted that the
resulting design recommendations can contain conflicts as they might violate disjoint assumptions (e.g.,
rule 4 proposes the chatbot gender to be male whereas rule 5 proposes a female chatbot name). In such a
case, researchers need to resolve the conflicts manually by reviewing the publications from which the
prescriptive design rules originated.
Subsequently, we used the machine-executable representation of prescriptive design rules to develop a social
cue configuration system that supports chatbot engineers in making justified social cue design decisions. The
configuration system enables chatbot engineers to configure all influencing factors of a possible human-
chatbot interaction and subsequently provides prescriptive social cue design recommendations. Screenshots
of the chatbot configuration system are displayed in Figure 2. The chatbot social cue configuration system
follows a four-step configuration process and in the fifth step recommends social cue designs (see process
steps in Figure 2). In the first step, the system enables the chatbot engineers to configure a desired goal of the
chatbot social cue design. In the next three steps, the chatbot engineers configure all relevant influencing
factors of the human-chatbot interaction regarding the user, task, and context. All potential configuration
options are based on the instantiated knowledge base (see configuration options). In the fifth step, the
configuration system queries the knowledge base and provides prescriptive design recommendations (see
social cue design recommendations). In addition, the system displays conflicts in the social cue design
recommendations in case different studies found contradictory results or exclude each other due to disjoint
assumptions (see references and conflicts). Moreover, the system is able to display the references of a rule and
to show design examples for each social cue design characteristic.
Designing a Chatbot Social Cue Configuration System
Fortieth International Conference on Information Systems, Munich 2019 10
Focus Group Evaluation
The exploratory focus group evaluation lasted in total 90 minutes. The moderator launched the chatbot
social cue configuration prototype and used the system to retrieve social cue design recommendations for
two scenarios. In both scenarios, a chatbot engineer was supposed to design a chatbot for a particular
stakeholder. Next, we asked the participants to evaluate the design of the prototype by conducting a SWOT
analysis. We handed out colored cards, gave time to formulate thoughts and ideas, and then had a n open
discussion for each category of the SWOT analysis. After the focus group, we analyzed the feedback using
the audio recording, written notes, and colored cards on the whiteboard. The feedback from the SWOT
analysis is summarized in Figure 3 and explained in more detailed below.
Strengths
Weaknesses
Provides guidance and decision support in designing social
cues of a chatbot
Helps to understand the interaction mechanisms,
influencing factors, and dependencies of social cue design
decisions
Enables and supports the development of personalized
chatbots
No transparency in the social cue design recommendation
process
Conflicting social cue design recommendations are difficult
to understand and resolve
Opportunities
Threats
Enhances customer experience due to a more grounded
social cue design
Forces you to make targeted social cue design decisions for
a specific user, task, and context
Engages other stakeholders in the chatbot social cue design
process
Relevant and important social cues are missing
System proposes wrong social cue design
Chatbot gets over-customized and excludes potential user
groups
High maintenance effort to update knowledge base and
decision rules
Strengths: Overall, participants had a positive impression and stated that such a social cue configuration
system is useful. One participant mentioned, “It provides some kind of guidance. It gives you an idea about
all criteria you should think about.” Moreover, one participant stated, You simply get the design elements
which makes it more easily, and definitely faster to get a decision. Therefore, “if you know your user and
your goal, it is not about playing with different social cue designs. It is more about finding the right ones
faster”. In addition, participants argued that it helps to understand mechanisms and dependencies of social
cue design decisions. One participant mentioned, “From a developer perspective it gives you a better idea
to understand how things interact and depend on each other as you have the opportunity to change single
dimensions and see the outcome.” Thus, “I know which parameters impact my bot design and determine
the user experience.” Finally, a participant mentioned, “I like that there were a lot of different options to
Process steps References and
conflicts
Social cue design
recommendations
Configuration
options
Designing a Chatbot Social Cue Configuration System
Fortieth International Conference on Information Systems, Munich 2019 11
craft the image of the user and define the outcome of the bot.” Therefore, it enables you to “develop
customized bots that you wouldn't normally have thought of”.
Opportunities: Participants argued that the greatest opportunity of the system is to create social cue
design decisions that are based on existing empirically-grounded knowledge. One participant mentioned,
“The biggest opportunity is to hopefully create a better user experience. And then you get better KPIs and
better results.” Moreover, another participant liked that “this system gives credibility” and another added,
“it makes clear that maybe you should think about having two chatbots for different target groups or
personas. And it is not because I am saying this, but it is the result of the tool”. Moreover, participants
stated that the system encourages you to think about relevant influencing factors as it “it forces you to be
targeted, it forces you to think about the interaction environments […], it forces you to find the right cue
combinations”. Lastly, participants acknowledged that the prototype provides a clear design guideline that
also helps chatbot engineers to engage other stakeholders in the design process.
Weaknesses: The participants explained that one of the biggest weaknesses of the system is the lack of
transparency in the social cue design recommendations. One participant mentioned, “At the moment it is
some kind of a black box. You do not know why it chose this and not that design.” Consequently, one
participant complaint, “I cannot be sure if all cues are covered, because it is not transparent and another
argued, “I do not know when all these studies were done. Is it five years ago? Are they still up to date?”
Because of the missing transparency, participants had difficulties to understand “why there were some
design conflicts”. Therefore, one participant recommended, “If I have conflicts in my social cue design, I
do not know which cue I should choose in the current design. It would be great to have some advice, that
you better use this social cue than this one based on some studies.”
Threats: The major threat of a chatbot social cue configuration system is that some relevant and important
social cues might not be covered but “you have the feeling that the system covered everything”. Moreover,
participants warned that the system could suggest wrong social cue designs. In this case, “no cues are better
than the wrong ones, […] because what is when your assumptions are wrong, or your goal is wrong?
Then you have a bot that is really not successful”. In addition, one participant warned, “there might be a
threat of over customization” as a chatbot social cue design might be too specific for a certain user group.
In this case, “you do not cover the other 30% which are not your core target group”. Finally, participants
mentioned that the configuration system is only useful when the knowledge base is constantly up -to-date.
However, “it is a long-term project to manage the whole tool, because you constantly need to update the
references, since science and technologies are moving forward very fast”.
Validate Research Relevance to Practice
Subsequently, we evaluated the research relevance to practice in order to retrieve feedback of a more diverse
set of potential users at two practitioner symposia. To present our system, we incorporated the feedback
from the focus group and developed a revised social cue configuration system that builds on the same
knowledge base (see Figure 4). We developed the revised system as a Java web application and deployed it
on a Tomcat web server.
During the development of the web application, we accounted for the lack of transparency in the provided
design recommendations. Therefore, we followed research that reports similar user reactions to system-
based recommendations (Al-Natour and Benbasat 2009; Limayem and DeSanctis 2000). Although
decisional guidance positively affects outcomes such as the decision quality and decision confidence
(Limayem and DeSanctis 2000), system-based recommendations are generally met with some resistance
(Giboney et al. 2015). To address this, researchers showed that explanations improve performance and
learning (Gregor and Benbasat 1999) and improve the usability of such as system in group interactions
(Limayem and DeSanctis 2000). Adapting these findings and concepts to our research context, we
identified the need to provide more explanations for each design recommendation. Consequently, we
addressed this shortcoming in the web application. The web application outlines which influencing factor
and study have led to a specific social cue design recommendation. In addition, the web application provides
explanations by offering a social cue description view (i.e., detailed information about a social cue) and a
publication view (i.e., full text link and links to other investigated social cues in the paper).
Designing a Chatbot Social Cue Configuration System
Fortieth International Conference on Information Systems, Munich 2019 12
After presenting and discussing the revised social cue configuration system at two practitioner symposia,
we collected real-time feedback using Mentimeter. 71 participants shared their demographic information
during the presentations and 74 at the second symposium (i.e., participation in the complete survey was
voluntary). From the 71 (74) participants, 61% (77%) were professional practitioners, 15% (0%) were
researchers, and 24% (23%) university students. 21% (70%) reported that they have no experience in the
design of chatbots, 56% (19%) reported that they have intermediate chatbot design experience, and 23%
(11%) reported that they are experienced chatbot designers. The results are displayed in Table 4. The mean
results of the three evaluation criteria relevant for the applicability assessment (Rosemann and Vessey
2008) were all higher than the mean of the scales (i.e., 3.0) (Klein et al. 2006). Finally, the mean NPS was
above 7.2, meaning that the respondents at both symposia can be classified to be at least passively satisfied
(Reichheld 2004).
Symposium
Importance [0,5]
Applicability [0,5]
Accessibility [0,5]
NPS [0,10]
Intention to use [0,10]
1 (N=71)
4.13 (SD=0.91)
4.1 (SD=0.89)
4.38 (SD= 0.87)
8.14 (SD= 2.67)
7.23 (SD=2.93)
2 (N=74)
4.17 (SD=1.01)
3.87 (SD=1.08)
3.37 (SD=1.15)
8.82 (SD=1.64)
8.42 (SD=1.96)
In the open discussion, several participants shared the importance of making more justified chatbot social
cue design decisions. Moreover, some participants stated that such a system will help them to discuss design
decisions with different company departments and explicitly addressed the need for such a system.
Consequently, the applicability assessment revealed that potential users of the system judge the social cue
configuration system as relevant to practice.
Discussion
In this paper, we followed a DSR approach and proposed a concept that supports chatbot engineers to make
justified social cue design decisions (i.e., grounded in empirical research). In particular, we proposed and
instantiated two design principles to evaluate the proposed concept as well as the underlying design. The
exploratory focus group evaluation of the first prototype indicated that the proposed design is useful and
supportive. Subsequently, we addressed weaknesses and threats of the first evaluation and developed a
Publication View
Configuration of Influencing
Factors Prescriptive Design Rules
Social Cue Description View
Designing a Chatbot Social Cue Configuration System
Fortieth International Conference on Information Systems, Munich 2019 13
second revised prototype as a web application. Therefore, we followed the suggestion of several scholars
(Gregor and Benbasat 1999; Limayem and DeSanctis 2000) and increased the transparency of a social cue
design recommendation in order to reduce resistance against recommendations of knowledge based
systems. In the subsequent applicability assessment at two practitioner symposia, we demonstrated the
functional feasibility of the (revised) system. We retrieved positive feedback regarding the importance,
accessibility, and suitability of the research to the needs of practice (Rosemann and Vessey 2008). Overall,
our DSR project contributes with an evaluated concept describing how to support chatbot engineers in
making justified chatbot social cue design decisions. We contribute with two design principles in order to
extract and transform descriptive knowledge from empirical research into a machine-executable
prescriptive representation. Therefore, we support chatbot engineers to avoid re-inventing the wheel (p.
46) when making chatbot social cue design decisions (McTear 2017).
Moreover, our proposed concept cannot only be applied in the chatbot social cue design but can be further
generalized to address a broader class of problem that is persistent in several research domains, namely the
difficulty to access and extract descriptive knowledge (which is hidden in unstructured text-based data) and
apply it in the design of artifacts (Feine et al. 2019c). To apply the proposed concept in other research
contexts, researchers and practitioners need to extract the knowledge pieces (KP1-3) relevant to define
purposeful prescriptive design rules (DP1) and subsequently need to transform them into a machine-
executable representation (DP2). Then they can use the configuration system and access the newly
developed knowledge base. For example, Reinecke and Bernstein (2013) showed that certain levels of
information density on a website (KP1) are perceived differently depending on the user’s cultural origin
(KP2) and result in diverse user preferences (KP3). After identifying these three knowledge pieces, it is
possible to derive prescriptive design rules (DP1) (i.e. how to adapt the information density of a website to
the cultural origin of a user). After conceptualizing the prescriptive design rules in the ontology and
instantiating the knowledge base of the configuration system (DP2), users of the system (e.g., website
designers) can query the system (e.g., show all website design characteristics that match the user
preferences of a specific cultural origin) and receive design recommendations (e.g., information density of
website should be high). This enables researchers as well as practitioners to efficiently access, query, and
(re-)use existing descriptive knowledge for the design of adaptive websites. Thus, both design principles are
generalizable to other contexts and thus, can be used to re(-use) existing descriptive knowledge. Figure 5
illustrates the system architecture of such a generic configuration system in order to leverage existing
descriptive knowledge in the design of an artifact.
Finally, there are some limitations that should be considered. First, future research is mandatory to
demonstrate the proof-of-value and the actual proof-of-use of such a system (Nunamaker et al. 2015).
Therefore, we already released the chatbot social cue configuration web application to the public in order
to retrieve feedback from real-world users. Second, we only evaluated the design of our system in one focus
group workshop with participants from one company. Consequently, the results may be biased as
employees of one company often behave similarly. To counteract this bias, we already assessed the design
with a larger and more diverse sample at two practitioner symposia. However, both evaluations rely on
qualitative studies which are highly contextual and thus, lack replicability and generalizability
(Bhattacherjee 2012). In addition, the applied methods may be affected by response biases such as demand
User can access descriptive knowledge via a configuration system in order to
retrieve theory grounded design recommendations
4
DP2: store prescriptive design rules in a machine-executable representation
3
Desig n Rules
[KP1: de sign charac teristi c,
KP2: influen cing factor,
KP3: user re actio n]
DP1: transform descriptive knowledge into prescriptive design rules
2
Unstructured descriptive knowledge in the form of empirical studies
1
Designing a Chatbot Social Cue Configuration System
Fortieth International Conference on Information Systems, Munich 2019 14
characteristics or social desirability bias (i.e., participants adjusted their responses in relation to what they
perceived to be our expectations) (Dell et al. 2012). As a consequence, future research should re-evaluate
the system also using experiments in order to increase the replicability and generalizability of the proposed
design (Venkatesh et al. 2013). Third, the proposed design (formulated in the two DPs) does not account
for causality. The proposed design only aggregates empirical cause-and-effect relationships without
questioning the underlying theoretical assumptions (Durand and Vaara 2009). Thus, the design does not
“identify the causal claims upon which proposed design principles or theories are founded” (Hovorka and
Gregor 2010 ,p. 8). As a consequence, a revised design should automatically account for construct identities
in order to capture the underlying theoretical foundations of a publication (Larsen and Bong 2016). Fourth,
the exiting knowledge base in the form of prescriptive social cue design rules is static and only mirrors a
current snapshot of existing descriptive knowledge. Since research is constantly evolving, it is from
importance to constantly update the knowledge base. Therefore, future research is needed to automatically
populate the knowledge base. A promising solution might be to leverage the historical interaction data
between users and chatbots by leveraging machine learning algorithms in order to enhance the design rules
on an individual level. This can further lead to more personalized chatbot social cue designs. Fifth, the social
cue design recommendations do not account for ethical as well as legal implications. However, it is
important that a chatbot social cue configuration system outlines whether a social cue design is ethical
justifiable and/or legally permitted in order to support chatbot engineers to mind ethical and legal concerns.
Conclusion
In this research project, we address a contemporary problem of chatbot engineers and contributes with a
concept to support chatbot engineers in making justified chatbot social cue design decisions (i.e., grounded
in existing empirical research). More specifically, we derive two DPs that describe how to extract and
transform descriptive knowledge from empirical research into a machine-executable prescriptive
representation. We instantiate both design principles and develop two prototypes in order to evaluate the
proposed concept. The prototypes are based on descriptive social cue knowledge and provide chatbot
engineers with prescriptive social cue design recommendations. We evaluated the first prototype by
conducting a focus group workshop with chatbot engineers and subsequently validated the second revised
prototype at two practitioner symposia regarding its relevance to practice. Summed up, our research
supports chatbot engineers and researchers to access the valuable descriptive knowledge base and leverage
it in the design of artifacts.
References
Ajzen, I. 2006. Constructing a theory of planned behavior questionnaire: Amherst, MA.
Al-Natour, S., and Benbasat, I. 2009. “The Adoption and Use of IT Artifacts: A New Interaction-Centric
Model for the Study of User-Artifact Relationships,Journal of the Association for Information Systems
(10:9), pp. 661685.
Araujo, T. 2018. “Living up to the chatbot hype: The influence of anthropomorphic design cues and
communicative agency framing on conversational agent and company perceptions,” Computers in
Human Behavior (85), pp. 183189.
Baskerville, R., and Pries-Heje, J. 2014. “Design Theory Projectability,” in Information Systems and Global
Assemblages. (Re)Configuring Actors, Artefacts, Organizations, B. Doolin, E. Lamprou, N. Mitev and
L. McLeod (eds.), Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 219232.
Beldad, A., Hegner, S., and Hoppen, J. 2016. “The effect of virtual sales agent (VSA) gender – product
gender congruence on product advice credibility, trust in VSA and online vendor, and purchase
intention,” Computers in Human Behavior (60), pp. 6272.
Berners-Lee, T., Hendler, J., and Lassila, O. 2001. “The semantic web,Scientific American (284:5), pp. 34
43.
Bhattacherjee, A. 2012. Social science research: Principles, methods, and practices, Tampa, Minneapolis:
Anol Bhattacherjee; Scholar Commons, University of South Florida; Open Textbook Library.
Bickmore, T. W., and Picard, R. W. 2005. “Establishing and Maintaining Long-Term Human-Computer
Relationships,” ACM Transactions on Computer-Human Interaction (12:2), pp. 293327.
Brahnam, S., and Angeli, A. de 2012. “Gender affordances of conversational agents,” Interacting with
Computers (24:3), pp. 139153.
Designing a Chatbot Social Cue Configuration System
Fortieth International Conference on Information Systems, Munich 2019 15
Brandtzaeg, P. B., and Følstad, A. 2018. “Chatbots: Changing User Needs and Motivations,Interactions
(25:5), pp. 3843.
Braslavski, P., Blinov, V., Bolotova, V., and Pertsova, K. 2018. “How to Evaluate Humorous Response
Generation, Seriously?” in Proceedings of the 2018 Conference on Human Information
Interaction&Retrieval - CHIIR '18, C. Shah, N. J. Belkin, K. Byström, J. Huang and F. Scholer (eds.),
New Brunswick, NJ, USA. 11.03.2018 - 15.03.2018, New York, New York, USA: ACM Press, pp. 225
228.
Catrambone, R., Stasko, J., and Xiao, J. 2004. “ECA as User Interface Paradigm,” in From Brows to Trust:
Evaluating Embodied Conversational Agents, Z. Ruttkay and C. Pelachaud (eds.), Dordrecht: Springer
Netherlands, pp. 239267.
Chandra, L., Seidel, S., and Gregor, S. 2015. “Prescriptive Knowledge in IS Research: Conceptualizing Design
Principles in Terms of Materiality, Action, and Boundary Conditions,” in 48th Hawaii International
Conference on System Sciences, pp. 40394048.
Cowell, A. J., and Stanney, K. M. 2005. “Manipulation of non-verbal interaction style and demographic
embodiment to increase anthropomorphic computer character credibility,International Journal of
Human-Computer Studies (62:2), pp. 281306.
Dale, R. 2016. “The return of the chatbots,Natural Language Engineering (22:5), pp. 811817.
Davenport, T. H., Long, D. W. de, and Beers, M. C. 1998. “Successful knowledge management projects,”
Sloan management review (39:2), pp. 4357.
Dell, N., Vaidyanathan, V., Medhi, I., Cutrell, E., and Thies, W. 2012. “"Yours is better!",” in Proceedings of
the SIGCHI Conference on Human Factors in Computing Systems, J. A. Konstan, E. H. Chi and K.
Höök (eds.), Austin, Texas, USA. 5/5/2012 - 5/10/2012, New York: ACM, p. 1321.
Diederich, S., Brendel, A., Lichtenberg, S., and M. Kolbe, L. 2019a. Design for Fast Request Fulfillment or
Natural Interaction? Insights from an Experiment with a Conversational Agent.
Diederich, S., Brendel, A., and M Kolbe, L. 2019b. “Towards a Taxonomy of Platforms for Conversational
Agent Design,” in 14. International Conference on Wirtschaftsinformatik (WI2019).
Durand, R., and Vaara, E. 2009. “Causation, counterfactuals, and competitive advantage,Strategic
Management Journal (30:12), pp. 12451264.
Endrass, B., Rehm, M., and André, E. 2010. “Planning Small Talk behavior with cultural influences for
multiagent systems,” Computer Speech & Language (25:2), pp. 158174.
Feine, J., Gnewuch, U., Morana, S., and Maedche, A. 2019a. “A Taxonomy of Social Cues for Conversational
Agents,International Journal of Human-Computer Studies.
Feine, J., Morana, S., and Gnewuch, U. 2019b. “Measuring Service Encounter Satisfaction with Customer
Service Chatbots using Sentiment Analysis,” in 14. Internationale Tagung Wirtschaftsinformatik
(WI2019).
Feine, J., Morana, S., and Maedche, A. 2019c. “Leveraging Machine-Executable Descriptive Knowledge in
Design Science Research The Case of Designing Socially-Adaptive Chatbots, in Extending the
Boundaries of Design Science Theory and Practice, B. Tulu, S. Djamasbi and G. Leroy (eds.), Cham:
Springer International Publishing, pp. 7691.
Fogg, B. J. 2002. “Computers as Persuasive Social Actors,” in Persuasive Technology: Using Computers to
Change What We Think and Do, San Francisco, CA, USA: Morgan Kaufmann Publishers, pp. 89120.
Følstad, A., and Brandtzæg, P. B. 2017. “Chatbots and the New World of HCI,” Interactions (24:4), pp. 38
42.
Forlizzi, J., Zimmerman, J., Mancuso, V., and Kwak, S. 2007. “How Interface Agents Affect Interaction
Between Humans and Computers,” in Proceedings of the 2007 Conference on Designing Pleasurable
Products and Interfaces, New York, NY, USA: ACM, pp. 209221.
Gartner 2018. Gartner Says 25 Percent of Customer Service Operations Will Use Virtual Customer
Assistants by 2020. https://www.gartner.com/newsroom/id/3858564. Accessed 22 November 2018.
Giboney, J. S., Brown, S. A., Lowry, P. B., and Nunamaker, J. F. 2015. “User acceptance of knowledge-based
system recommendations: Explanations, arguments, and fit,” Decision Support Systems (72), pp. 110.
Gnewuch, U., Morana, S., Adam, M., and Maedche, A. 2018. “Faster Is Not Always Better: Understanding
the Effect of Dynamic Response Delays in Human-Chatbot Interaction,” in Proceedings of the 26th
European Conference on Information Systems (ECIS), Portsmouth, United Kingdom, June 23-28.
Gnewuch, U., Morana, S., and Maedche, A. 2017. “Towards Designing Cooperative and Social
Conversational Agents for Customer Service,” in Proceedings of the 38th International Conference on
Information Systems (ICIS), Seoul: AISel.
Designing a Chatbot Social Cue Configuration System
Fortieth International Conference on Information Systems, Munich 2019 16
Gregor, S., and Benbasat, I. 1999. “Explanations from intelligent systems: Theoretical foundations and
implications for practice,” MIS Quarterly (23), pp. 497530.
Gregor, S., and Jones, D. 2007. “The Anatomy of a Design Theory,” Journal of the Association for
Information Systems (8:5), pp. 312335.
Hayashi, Y. 2016. “Lexical Network Analysis on an Online Explanation Task: Effects of Affect and
Embodiment of a Pedagogical Agent,IEICE Transactions on Information and Systems (99:6), pp.
14551461.
Ho, S. Y., and Lim, K. H. 2018. “Nudging Moods to Induce Unplanned Purchases in Imperfect Mobile
Personalization Contexts,” MIS Quarterly (42), pp. 757778.
Hone, K. 2006. “Empathic agents to reduce user frustration: The effects of varying agent characteristics,”
Interacting with Computers (18:2), pp. 227245.
Hovorka, D. S., and Gregor, S. 2010. “Untangling Causality in Design Science Theorizing,” in 5th biennial
ANU workshop on information systems foundations, Canberra, Australia, pp. 116.
Huang, T.-H. K., Chang, J. C., Swaminathan, S., and Bigham, J. P. 2017. “Evorus,” in Adjunct Publication of
the 30th Annual ACM Symposium on User Interface Software and Technology, K. Gajos (ed.),
Québec City, QC, Canada. 10/22/2017 - 10/25/2017, New York, NY: ACM, pp. 155157.
Isbister, K., and Nass, C. 2000. “Consistency of personality in interactive characters: Verbal cues, non-verbal
cues, and user characteristics,” International Journal of Human-Computer Studies (53:2), pp. 251267.
Keeling, K., Beatty, S., McGoldrick, P., and Macaulay, L. 2004. “Face value? Customer views of appropriate
formats for embodied conversational agents (ECAs) in online retailing,” in Proceedings of the 37th
Annual Hawaii International Conference on System Sciences., pp. 110.
Kharkovyna, O. 2018. Chatbot Prototyping: 7 Tools to Shape A Bot With No Coding.
https://chatbotsmagazine.com/chatbot-prototyping-7-tools-to-shape-a-bot-with-no-coding-
f2bfac3c94e3. Accessed 23 April 2019.
Klein, G., Jiang, J., and Saunders, C. 2006. “Leading the horse to water,” Communications of the
Association for Information Systems (18:1), p. 13.
Klopfenstein, L. C., Delpriori, S., and Ricci, A. 2019. “Adapting a Conversational Text Generator for Online
Chatbot Messaging,” in Internet Science, S. S. Bodrunova, O. Koltsova, A. Følstad, H. Halpin, P.
Kolozaridi, L. Yuldashev, A. Smoliarova and H. Niedermayer (eds.), Cham: Springer International
Publishing, pp. 8799.
Kraemer, N. C., Karacora, B., Lucas, G., Dehghani, M., Ruether, G., and Gratch, J. 2016. “Closing the gender
gap in STEM with friendly male instructors? On the effects of rapport behavior and gender of a virtual
agent in an instructional interaction,” Computers & Education (99), pp. 113.
Krämer, N. C. 2005. “Social Communicative Effects of a Virtual Program Guide,” in Intelligent Virtual
Agents, T. Panayiotopoulos, J. Gratch, R. Aylett, D. Ballin, P. Olivier and T. Rist (eds.), Berlin,
Heidelberg: Springer Berlin Heidelberg, pp. 442453.
Kuechler, B., and Vaishnavi, V. 2008. “On theory development in design science research: Anatomy of a
research project,European Journal of Information Systems (17:5), pp. 489504.
Larsen, K. R., and Bong, C. H. 2016. “A Tool for Addressing Construct Identity in Literature Reviews and
Meta-Analyses,” MIS Quarterly (40:3), pp. 529551.
Layer 2019. Conversation Design Kit for Sketch. https://layer.com/resources/conversation-design-kit/.
Accessed 28 April 2019.
Li, J., Zhou, M. X., Yang, H., and Mark, G. 2017. “Confiding in and Listening to Virtual Agents,” in
Proceedings of the 22nd International Conference on Intelligent User Interfaces - IUI, Limassol,
Cyprus. 13.03.2017 - 16.03.2017, ACM Press, pp. 275286.
Limayem, M., and DeSanctis, G. 2000. “Providing Decisional Guidance for Multicriteria Decision Making in
Groups,Information Systems Research (11:4), pp. 386401.
Maedche, A., Gregor, S., Morana, S., and Feine, J. 2019. “Conceptualization of the Problem Space in Design
Science Research,” in Extending the Boundaries of Design Science Theory and Practice, B. Tulu, S.
Djamasbi and G. Leroy (eds.), Cham: Springer International Publishing, pp. 1831.
Mayer, R. E., Johnson, W. L., Shaw, E., and Sandhu, S. 2006. “Constructing computer-based tutors that are
socially sensitive: Politeness in educational software,” International Journal of Human-Computer
Studies (64:1), pp. 3642.
McBreen, H. 2002. “Embodied Conversational Agents in E-Commerce Applications,” in Socially Intelligent
Agents: Creating Relationships with Computers and Robots, Boston, MA: Springer US, pp. 267274.
Designing a Chatbot Social Cue Configuration System
Fortieth International Conference on Information Systems, Munich 2019 17
McTear, M. F. 2017. “The Rise of the Conversational Interface: A New Kid on the Block?” in Future and
Emerging Trends in Language Technology. Machine Learning and Big Data, J. F. Quesada, F.-J.
Martín Mateos and T. López Soto (eds.), Cham: Springer International Publishing, pp. 3849.
Mimoun, M. S. B., Poncin, I., and Garnier, M. 2012. “Case study—Embodied virtual agents: An analysis on
reasons for failure,” Journal of Retailing and Consumer Services (19:6), pp. 605612.
Myers, M. D. 2009. Qualitative research in business & management, Thousand Oaks, CA: SAGE
Publications Ltd.
Nass, C., and Moon, Y. 2000. “Machines and Mindlessness: Social Responses to Computers,Journal of
Social Issues (56:1), pp. 81103.
Nass, C., Moon, Y., Fogg, B. J., Reeves, B., and Dryer, D.C. 1995. “Can computer personalities be human
personalities?” International Journal of Human-Computer Studies (43:2), pp. 223239.
Nass, C., Steuer, J., and Tauber, E. R. 1994. “Computers Are Social Actors,” in Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems, New York, NY, USA: ACM, pp. 7278.
Niculescu, A., Hofs, D., van Dijk, B., and Nijholt, A. 2010. “How the agent's gender influence users'
evaluation of a QA system,” in International Conference on User Science and Engineering (i-USEr).
Nunamaker, J. E., Derrick, D. C., Elkins, A. C., Burgoon, J. K., and Patton, M. W. 2011. “Embodied
Conversational Agent-Based Kiosk for Automated Interviewing,Journal of Management Information
Systems (28:1), pp. 1748.
Nunamaker, J. F., Briggs, R. O., Derrick, D. C., and Schwabe, G. 2015. “The Last Research Mile: Achieving
Both Rigor and Relevance in Information Systems Research,” Journal of Management Information
Systems (32:3), pp. 1047.
Ostrowski, L., Helfert, M., and Gama, N. 2014. “Ontology engineering step in design science research
methodology: a technique to gather and reuse knowledge,” Behaviour & Information Technology
(33:5), pp. 443451.
Pak, R., Fink, N., Price, M., Bass, B., and Sturre, L. 2012. “Decision support aids with anthropomorphic
characteristics influence trust and performance in younger and older adults,Ergonomics (55:9), pp.
10591072.
Qiu, L., and Benbasat, I. 2010. “A study of demographic embodiments of product recommendation agents in
electronic commerce,” International Journal of Human-Computer Studies (68:10), pp. 669688.
Reichheld, F. 2004. “The One Number you Need to Grow,” Harvard Business Review (81), pp. 111.
Reinecke, K., and Bernstein, A. 2013. “Knowing What a User Likes: A Design Science Approach to Interfaces
that Automatically Adapt to Culture,MIS Quarterly (37:2), pp. 427453.
Rosemann, M., and Vessey, I. 2008. “Toward improving the relevance of information systems research to
practice: the role of applicability checks,MIS Quarterly, pp. 122.
Rupp, C. 2014. Requirements-Engineering und -Management, München: Hanser.
Sauter, V. L. 2014. Decision Support Systems for Business Intelligence: Wiley.
Staab, S., and Studer, R. 2009. Handbook on ontologies: Springer Science & Business Media.
Tremblay, M. C., Hevner, A. R., and Berndt, D. J. 2010. “The Use of Focus Groups in Design Science
Research,” in Design Research in Information Systems: Theory and Practice, Boston, MA, USA:
Springer, pp. 121143.
Treml, F. 2018. How to Test a ChatbotPart 3: The Quest for Test Cases.
https://chatbotsmagazine.com/how-to-test-a-chatbot-part-3-the-quest-for-test-cases-cb7a90238a40.
Accessed 23 April 2019.
Venable, J., Pries-Heje, J., and Baskerville, R. 2016. “FEDS: a Framework for Evaluation in Design Science
Research,” European Journal of Information Systems (25:1), pp. 7789.
Venkatesh, V., Brown, S. A., and Bala, H. 2013. “Bridging the Qualitative-Quantitative Divide: Guidelines for
Conducting Mixed Methods Research in Information Systems,” MIS Quarterly (37:1), pp. 2154.
Zhang, X. 2017. “Knowledge Management System Use and Job Performance: A Multilevel Contingency
Model,MIS Quarterly (41:3), pp. 811840.
... Social cues are signals depicted by humans when interacting with each other [5]. A chatbot humanness is determined by various social cues which are verbal, visual, and invisible. ...
... Chatbots respond to user queries instantaneously. However, a study reported that chatbots which use delayed responses resulted in more user satisfaction compared to chatbots that respond immediately [5]. Normally, a human takes some time to read and respond to a message which increases the user's perception of humanness and social presence, as well as, fulfilling the user's social expectations. ...
Chapter
Full-text available
High school students start to apply to universities during their senior year. This marks the beginning of a frustrating and stressful period for the students as well as the admission staff. Students have countless queries and issues that need to be repeatedly answered and solved by the admission staff. However, with the rise of modern technologies such as conversational agents, the admission process can be automated to decrease the load of the admission staff. Therefore, we investigated the usefulness of a chatbot for university admission that is designed for a specific target demographic. A chatbot was developed named “Jooka” for a university to improve their admission process. Since our target audience is predominately bilingual, we added bilingualism support to Jooka. Jooka understands queries written in English and Arabic, and responds according to the query language. A survey was conducted to evaluate the performance of Jooka with parents and high school participants. Overall, the results showed great acceptance and willingness from users to adopt the new technology.
... To ensure that no design principle is without purpose, each design principle must address at least one meta-requirement (Goldkuhl 2004). Eliciting meta-requirements from interviews and generating responsive design principles that address them is a typical avenue for design principle generation (Möller et al. 2020b) (e.g., see Niemöller et al. 2019or Feine et al. 2019. Following the advice of Koppenhagen et al. (2012), we applied logical content aggregation to synthesize meta-requirements to key-requirements to ensure that the resulting design principles actually address issues of high importance rather than a broad spectrum of particular problems (see Fig. 4). ...
Article
Full-text available
Digital Twins offer considerable potential for cross-company networks. Recent research primarily focu-ses on using Digital Twins within the limits of a single organization. However, Shared Digital Twins extend application boundaries to cross-company utilization through their ability to act as a hub to share data. This results in the need to consider additional design dimensions which help practitioners design Digital Twins tailored for inter-company use. The article addresses precisely that issue as it investigates how Shared Digital Twins should be designed to achieve business success. For this purpose, the article proposes a set of design principles for Shared Digital Twins stemming from a qualitative interview study with 18 industry experts. The interview study is the primary data source for formulating and evaluating the design principles.
... Next, we evaluated the derived DPs and DFs by conducting exploratory focus groups according to Tremblay, et al. [24]. This evaluation approach was already successfully applied in other DSR studies [e.g., 54,55]. In general, we follow the evaluation framework proposed by Venable, et al. [56] for our DSR project. ...
... The area problem and goal differentiate in the problem space between building blocks for a specific problem and the more abstract problem class and, from a solutionoriented view, between solution objective (i.e., the goal the artifact is supposed to fulfill [13,22]) and design requirement. We can observe substantial heterogeneity in the terminology used to describe elements of design principles in our sample, for instance, key challenge [37], user requirement [38], design requirement [39], and meta-requirement [40]. That differentiation is usually associated with a dichotomy between theorydriven and practice-driven research (e.g., through the point of artifact design [13]). ...
Chapter
Full-text available
Researchers and practitioners often face challenges in structuring larger design projects and, therefore, struggle to capture, discuss, and reflect on essential components that should be considered. These first steps are, however, of great importance because decisions such as in terms of selecting an underpinning (kernel) theory, following certain development approaches, or specifying knowledge sources impact the resulting design solution. To provide a frame for developing one of the dominant forms of prescriptive knowledge in information systems (IS), we present the ‘Principle Constructor’ that seeks to support the iterative endeavor of formulating design principles. This so-called visual inquiry tool is grounded in previous research on design knowledge and an empirical analysis of IS articles that present principles, built according to available guidance for this class of tools, and evaluated through several workshops. Doing this, we provide an underlying structure with building blocks for creating design principles and complement research on their anatomy and development procedures.
... From a customer service perspective, one can design conversational agents with social cues to unconsciously influence customers' behavior (Feine, Gnewuch, Morana, & Maedche, 2019a) or to discriminate against individuals (Feine, Gnewuch, Morana, & Maedche, 2020b). In turn, social cues can increase user satisfaction (Gnewuch, Morana, Adam, & Maedche, 2018;Rietz, Benke, & Maedche, 2019) and help conversational agents to adapt to specific users, contexts, and tasks (Feine, Morana, & Maedche, 2019b). From an enterprise perspective, conversational agents might be a threat to rationalize human workers (Feine, Morana, & Maedche, 2020c, 2020d but can also provide valuable support in employees' work routines (Benke, Knierim, & Maedche, 2020;Feine, Adam, Benke, Maedche, & Benlian, 2020a). ...
Chapter
Full-text available
Technological innovations raise axiological questions such as what is right or wrong, good and bad, and so on (i.e., ethical considerations). These considerations have particular importance in design science research (DSR) projects since the developed artifacts often actively intervene into human affairs and, thus, cannot be free from value. To account for this fact, Myers and Venable (2014) proposed six ethical principles for DSR in order to support researchers to conduct ethical DSR. However, ethical principles per se — and the ethical DSR principles that Myers and Venable propose — have an abstract nature so that they can apply to a broad range of contexts. As a consequence, they do not necessarily apply to specific research projects, which means researchers need to contextualize them for each specific DSR project. Because doing so involves much challenge, we explore how contemporary DSR publications have dealt with this contextualization task and how they implemented the six ethical principles for DSR. Our results reveal that DSR publications have not discussed ethical principles in sufficient depth. To further promote ethical considerations in DSR, we argue that both DSR researchers and reviewers should be supported in implementing ethical principles. Therefore, we outline two pathways toward ethical DSR. First, we propose that researchers need to articulate the next generation of ethical principles for DSR using prescriptive knowledge structures from DSR. Second, we propose extending established DSR conceptualizations with an ethical dimension and specifically introduce the concept of ethical DSR process models. With this work, we contribute to the IS literature by reviewing ethical principles and their implementation in DSR, identifying potential challenges hindering efforts to implement ethics in DSR, and providing two pathways towards ethical DSR.
... Sixth, the design of the system is currently restricted to the improvement of chatbot responses only. However, chatbot development support should not only support the creation of meaningful responses but should further help developers to design all important design features in an effective manner (Feine et al. 2019b(Feine et al. , 2019c. For example, the gender of the chatbot has shown to be a powerful social cue that has a strong impact on user perception (Feine et al. 2020b). ...
Conference Paper
Full-text available
Domain experts with their special knowledge and understanding of a specific field are critical in the development of chatbots. However, engaging domain experts in the chatbot development is time-consuming and cumbersome because developers lack adequate systems. We address this problem by proposing three design principles for interactive chatbot development systems grounded in the interactivity effects model. We instantiate the proposed design and evaluate the resulting artifact in an online experiment. The results of the online experiment (N=70) show that the proposed design significantly increases subjective and objective engagement and that perceived interactivity mediates these effects. Our study contributes with prescriptive knowledge for designing interactive systems that increase engagement and with a novel artifact in the form of an interactive chatbot development system.
Research Proposal
Full-text available
Digital dialogue systems, known as “chatbots,” are an application sub-area in automation which is increasingly used in both work and business contexts. The growing use, functionality and benefits of chatbots are therefore also of interest for university teaching in economics. Students at the University of Bremenʼs Department of Economics are developing their own transdisciplinary chatbot project. Working together with partners from different companies within the framework of the HumanRoboLab (HRL), they explore human–machine interaction from an applicationoriented perspective. The main objective of the HRL is to test a didactic concept for transdisciplinary human–machine interaction projects in order to subsequently integrate this concept into business studies and business psychology degree programs. The purpose of such a module is to provide students without prior experience in programming with the knowledge, tools and competences to research, apply and question AI technologies in a self-effective manner; in other words, the HRL aims to help students develop expertise with regard to digitization, benefiting them in both social and economic contexts. The concept of the HRL, as well as some examples of the projects and the role of AI, are outlined in this chapter in order to provide a template for university teachers who might wish to adapt these concepts.
Article
Full-text available
Personal virtual assistants (PVAs) based on artificial intelligence are frequently used in private contexts but have yet to find their way into the workplace. Regardless of their potential value for organizations, the relentless implementation of PVAs at the workplace is likely to run into employee resistance. To understand what motivates such resistance, it is necessary to investigate the primary motivators of human behavior, namely emotions. This paper uncovers emotions related to organizational PVA use, primarily focusing on threat emotions. To achieve our goal, we conducted an in-depth qualitative study, collecting data from 45 employees in focus-group discussions and individual interviews. We identified and categorized emotions according to the framework for classifying emotions Beaudry and Pinsonneault (2010) designed. Our results show that loss emotions, such as dissatisfaction and frustration, as well as deterrence emotions, such as fear and worry, constitute valuable cornerstones for the boundaries of organizational PVA use.
Conference Paper
Research on Artificial Intelligence (AI) robots has surged as an important topic in Information Systems (IS) domain. Although much research is empirically conducted, the available literature is fragmented and there is a lack of comprehensive understanding of current literature on service robots. Based on a review of 55 selected articles published in the top 10 IS journals and the four leading international conferences in IS field, this study identified the main research themes on robot research in IS field from the views of individuals, organizations, and design. Some suggestions for future research are also raised based on the findings in the review.
Chapter
Chatbots have attracted tremendous interest in recent years and are increasingly employed in form of enterprise chatbots (ECBs) (i.e., chatbots used in the explicit context of enterprise systems). Although ECBs substantially differ in their design requirements from, for example, more common and widely deployed customer service chatbots, only few studies exist that specifically investigate and provide guidance for the design of ECBs. To address this emerging gap, we accumulated existing design knowledge from previous studies and created a list of 26 design features (DFs) which we integrated into 6 design principles (DPs). Subsequently, 36 practitioners from an IT consulting company which are experienced in using ECBs evaluated the importance of the DPs and DFs following the Analytic Hierarchy Process method. Our results provide evidence that DPs and DFs promoting usability and flexibility are ranked more important than DPs and DFs promoting socialness and human likeness. These findings provide valuable insights, as they are partially contrary to some existing studies investigating the importance of social cues of chatbots in other domains. Overall, the identified lists of DPs and DFs and their importance rankings provide guidance for the design of ECBs and can serve as a basis for future research projects.
Article
Full-text available
Conversational agents (CAs) are software-based systems designed to interact with humans using natural language and have attracted considerable research interest in recent years. Following the Computers Are Social Actors paradigm, many studies have shown that humans react socially to CAs when they display social cues such as small talk, gender, age, gestures, or facial expressions. However, research on social cues for CAs is scattered across different fields, often using their specific terminology, which makes it challenging to identify, classify, and accumulate existing knowledge. To address this problem, we conducted a systematic literature review to identify an initial set of social cues of CAs from existing research. Building on classifications from interpersonal communication theory, we developed a taxonomy that classifies the identified social cues into four major categories (i.e., verbal, visual, auditory, invisible) and ten subcategories. Subsequently, we evaluated the mapping between the identified social cues and the categories using a card sorting approach in order to verify that the taxonomy is natural, simple, and parsimonious. Finally, we demonstrate the usefulness of the taxonomy by classifying a broader and more generic set of social cues of CAs from existing research and practice. Our main contribution is a comprehensive taxonomy of social cues for CAs. For researchers, the taxonomy helps to systematically classify research about social cues into one of the taxonomy's categories and corresponding subcategories. Therefore, it builds a bridge between different research fields and provides a starting point for interdisciplinary research and knowledge accumulation. For practitioners, the taxonomy provides a systematic overview of relevant categories of social cues in order to identify, implement, and test their effects in the design of a CA.
Chapter
Full-text available
Design science research (DSR) aims to deliver innovative solutions for real-world problems. DSR produces Information Systems (IS) artifacts and design knowledge describing means-end relationships between problem and solution spaces. A key success factor of any DSR research endeavor is an appropriate understanding and description of the underlying problem space. However, existing DSR literature lacks a solid conceptualization of the problem space in DSR. This paper addresses this gap and suggests a conceptualization of the problem space in DSR that builds on the four key concepts of stakeholders, needs, goals, and requirements. We showcase the application of our conceptualization in two published DSR projects. Our work contributes methodologically to the field of DSR as it helps DSR scholars to explore and describe the problem space in terms of a set of key concepts and their relationships.
Chapter
Full-text available
In Design Science Research (DSR) it is important to build on descriptive (Ω) and prescriptive (Λ) state-of-the-art knowledge in order to provide a solid grounding. However, existing knowledge is typically made available via scientific publications. This leads to two challenges: first, scholars have to manually extract relevant knowledge pieces from the data-wise unstructured textual nature of scientific publications. Second, different research results can interact and exclude each other, which makes an aggregation, combination, and application of extracted knowledge pieces quite complex. In this paper, we present how we addressed both issues in a DSR project that focuses on the design of socially-adaptive chatbots. Therefore, we outline a two-step approach to transform phenomena and relationships described in the Ω-knowledge base in a machine-executable form using ontologies and a knowledge base. Following this new approach, we can design a system that is able to aggregate and combine existing Ω-knowledge in the field of chatbots. Hence, our work contributes to DSR methodology by suggesting a new approach for theory-guided DSR projects that facilitates the application and sharing of state-of-the-art Ω-knowledge.
Chapter
Full-text available
Conversational interfaces and chatbots have a long history, but have only recently been hyped as a disruptive technology ready to replace mobile apps and Web sites. Many online messaging platforms have introduced support to third-party chatbots, which can be procedurally programmed, but usually rely on a retrieval-based specification language (such as AIML), natural language processing to detect the user’s intent, or on machine learning. In this work we present a work-in-progress integration of a widely-used system for story generation, the Tracery grammar, a conversational agent design tool, the Bottery system, and online messaging platforms. The proposed system provides a complete and easy-to-use system that allows the creation of chatbots with a graph-based dialogue structure, a contextual memory, pattern-based text matching, and advanced text generation capabilities, that aims for being well-suited for experts and technically unskilled authors alike. Features of the system and future additions are discussed and compared to existing solutions.
Conference Paper
Full-text available
Conversational agents continue to permeate our lives in different forms, such as virtual assistants on mobile devices or chatbots on websites and social media. The interaction with users through natural language offers various aspects for researchers to study as well as application domains for practitioners to explore. In particular their design represents an interesting phenomenon to investigate as humans show social responses to these agents and successful design remains a challenge in practice. Compared to digital human-to-human communication, text-based conversational agents can provide complementary, preset answer options with which users can conveniently and quickly respond in the interaction. However, their use might also decrease the perceived human likeness and social presence of the agent as the user does not respond naturally by thinking of and formulating a reply. In this study, we conducted an experiment with N=80 participants in a customer service context to explore the impact of such elements on agent anthropomorphism and user satisfaction. The results show that their use reduces perceived humanness and social presence yet does not significantly increase service satisfaction. On the contrary, our findings indicate that preset answer options might even be detrimental to service satisfaction as they diminish the natural feel of human-CA interaction.
Conference Paper
Full-text available
Chatbots are software-based systems designed to interact with humans using text-based natural language and have attracted considerable interest in online service encounters. In this context, service providers face the challenge of measuring chatbot service encounter satisfaction (CSES), as most approaches are limited to post-interaction surveys that are rarely answered and often biased. As a result, service providers cannot react quickly to service failures and dissatisfied customers. To address this challenge, we investigate the application of automated sentiment analysis methods as a proxy to measure CSES. Therefore, we first compare different sentiment analysis methods. Second, we investigate the relationship between objectively computed sentiment scores of dialogs and subjectively measured CSES values. Third, we evaluate whether this relationship also exists for utterance sequences throughout the dialog. The paper contributes by proposing and applying an automatic and objective approach to use sentiment scores as a proxy to measure CSES.
Conference Paper
Full-text available
Software that interacts with its users through natural language, so-called conversational agents (CAs), is permeating our lives with improving capabilities driven by advances in machine learning and natural language processing. For organizations, CAs have the potential to innovate and automate a variety of tasks and processes, for example in customer service or marketing and sales, yet successful design remains a major challenge. Over the last few years, a variety of platforms that offer different approaches and functionality for designing CAs have emerged. In this paper, we analyze 51 CA platforms to develop a taxonomy and empirically identify archetypes of platforms by means of a cluster analysis. Based on our analysis, we propose an extended taxonomy with eleven dimensions and three archetypes that contribute to existing work on CA design and can guide practitioners in the design of CA for their organizations.
Conference Paper
Full-text available
A key challenge in designing conversational user interfaces is to make the conversation between the user and the system feel natural and human-like. In order to increase perceived humanness, many systems with conversational user interfaces (e.g., chatbots) use response delays to simu-late the time it would take humans to respond to a message. However, delayed responses may also negatively impact user satisfaction, particularly in situations where fast response times are expected, such as in customer service. This paper reports the findings of an online experiment in a customer service context that investigates how user perceptions differ when interacting with a chatbot that sends dynamically delayed responses compared to a chatbot that sends near-instant responses. The dynamic delay length was calculated based on the complexity of the re-sponse and complexity of the previous message. Our results indicate that dynamic response de-lays not only increase users’ perception of humanness and social presence, but also lead to greater satisfaction with the overall chatbot interaction. Building on social response theory, we provide evidence that a chatbot’s response time represents a social cue that triggers social re-sponses shaped by social expectations. Our findings support researchers and practitioners in understanding and designing more natural human-chatbot interactions.
Article
By tracking consumers' browsing and purchase history, web personalization generates taste-matched recommendations for each consumer to stimulate purchases. In addition to taste-matching, mobile personalization matches recommendations to a consumer's physiological need and current location. These two additional features, referred to as need-matching and location-matching, are believed to be enablers of unplanned purchases. However, mobile advertisers may not be able to generate recommendations that meet all personalization criteria. Hence, mobile recommendations may be imperfect. We examine two questions in relation to imperfect recommendations. First, how do we use a descriptor to promote such recommendations? Second, what personalization criterion should be downplayed to induce unplanned purchases? Drawing upon the theory of mood congruence, we theorize that the effect of imperfect recommendation on consumers' unplanned purchases depends on their mood. We conducted three field experiments to test our hypotheses. Our findings indicate that (1) consumers in positive moods are more likely to form an urge to buy than those in negative moods, and this difference is larger when the descriptor is partial than when it is complete (Experiment 1); (2) need-matching is more influential on urge to buy for consumers in negative moods than for those in positive moods (Experiment 2); and (3) for taste-and-need-matched recommendations, location-matching exerts a stronger effect on the urge to buy for consumers in negative moods than for those in positive moods (Experiment 3). We validated the relevance of our research findings to practice through interviews with senior executives in personalization solution providers. Pathways for enhancing practical impacts of this line of research are recommended.