No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Exploring Design Principles for Enterprise Chatbots: An Analytic Hierarchy Process Study
Abstract
Chatbots have attracted tremendous interest in recent years and are increasingly employed in form of enterprise chatbots (ECBs) (i.e., chatbots used in the explicit context of enterprise systems). Although ECBs substantially differ in their design requirements from, for example, more common and widely deployed customer service chatbots, only few studies exist that specifically investigate and provide guidance for the design of ECBs. To address this emerging gap, we accumulated existing design knowledge from previous studies and created a list of 26 design features (DFs) which we integrated into 6 design principles (DPs). Subsequently, 36 practitioners from an IT consulting company which are experienced in using ECBs evaluated the importance of the DPs and DFs following the Analytic Hierarchy Process method. Our results provide evidence that DPs and DFs promoting usability and flexibility are ranked more important than DPs and DFs promoting socialness and human likeness. These findings provide valuable insights, as they are partially contrary to some existing studies investigating the importance of social cues of chatbots in other domains. Overall, the identified lists of DPs and DFs and their importance rankings provide guidance for the design of ECBs and can serve as a basis for future research projects.
... They follow from flexibility and transparency principles identified in previous research. However, when transparency was described as an important goal of CAs (Diederich, Brendel, & Kolbe, 2020;Feine, Adam, Benke, Maedche, & Benlian, 2020), the authors meant that the agent needs to be clearly labelled as artificial, and users need to be able to understand what functionalities it offers (functional transparency). Through our interviews, it additionally became clear how crucial it is that the information offered by the agent is transparent, for example that its source is mentioned and that it is accurately dated (informational transparency, DP2). ...
... A similarly superficial parallel that, under closer scrutiny, reveals important distinctions can be found in the descriptions of the desired communication style. Previous research identified the requirements that the CA is proactive in its communication, for example that it notifies users about information that is relevant for them instead of only taking input from them (Feine et al., 2020). The requirements identified by our interviewees go one step further. ...
... In summary, this study uncovered five actionable design principles representing concrete but demanding requirements for EMAs. These go far beyond the previously known requirements for general-purpose CAs used in organizations (Feine et al., 2020), but they are necessary to ensure a satisfactory crisis response (Mirbabaie et al., 2021). Arguably, these requirements also go far beyond what current technology can offer. ...
Widespread mis- and disinformation during the COVID-19 social media “infodemic” challenge the effective response of Emergency Management Agencies (EMAs). Conversational Agents (CAs) have the potential to amplify and distribute trustworthy information from EMAs to the general public in times of uncertainty. However, the structure and responsibilities of such EMAs are different in comparison to traditional commercial organizations. Consequently, Information Systems (IS) design approaches for CAs are not directly transferable to this different type of organization. Based on semi-structured interviews with practitioners from EMAs in Germany and Australia, twelve meta-requirements and five design principles for CAs for EMAs were developed. In contrast to the traditional view of CA design, social cues should be minimized. The study provides a basis to design robust CAs for EMAs.
... The former dimensions have been described in the literature, however they could not be confirmed in the empirical review. For example, socio-technical features as the presence of conversational memory in chatbots has been described in the literature (see e.g., [19][20][21]), but was not present in any of the chatbots examined. Furthermore, we added to the initial taxonomy 4 empirically identified dimensions of chatbots in B2B customer service, composed in the following manner: service/product information = {no, yes}; success stories = {no, yes}; book/show a demo = {no, yes}; and career information = {no, yes}. ...
... See Table 5. [5,26] D 3 Access to business data Describes whether the chatbot has access to non-public business data and uses it to enhance its responses [19] D 9 Content related service Describes whether the chatbot provides only commercial content on products or services, or enable the user to acquire them [13] D 10 Account authentication Describes whether the chatbot requires the authentication of the user by means of a business email address or username and password to begin the interaction [13] D 11 Question personalization Describes the degree of response customization of the chatbot, e.g., the capacity of the chatbot to tailored highly personalized questions require information obtained through the interaction with the user [8,20,25] D 12 Customer service orientation Describes whether a chatbot is primarily oriented to provide information or to perform a task [24] D 16 Action request Describes the functional actions related to customer service that the chatbot is able to perform [25] D 17 Service request Describes the functional service inquires related to customer service elements present in the chatbots [25] ...
Chatbots are becoming increasingly important in the customer service sector due to their service automation, cost saving opportunities and broad customer satisfaction. Similarly, in the business-to-business (B2B) sector, more and more companies use chatbots on their websites and social media channels, to establish sales team contact, to provide information about their products and services or to help customers with their requests and claims. Customer relations in the B2B environment are especially characterized by a high level of personal contact service and support through expert explanations due to the complexity of the products and service offerings. In order to support these efforts, chatbots can be used to assist buying centers along the purchase decision process. However, B2B chatbots have so far only been marginally addressed in the scientific human-computer interaction and information systems literature. To provide both researchers and practitioners with knowledge about the characteristics and archetypal patterns of chatbots currently existing in B2B customer services, we develop and discuss a 17-dimensional chatbot taxonomy for B2B customer services based on Nickerson et al. [1]. By classifying 40 chatbots in a cluster analysis, this study has identified three archetypal structures prevailing in B2B customer service chatbot usage.
... This subjectivity, for example, may underlie to a certain extent in the construction of our sample(s). While the final taxonomy relies on the classification of 120 chatbots from three different samples and sampling strategies which ensures wide coverage of available chatbots, further analyzes of archetypes could investigate boundary conditions that relate to interactions with situational design factors (RD1) other than the temporal profile, for example, across different domains (Feine et al., 2020) or purposes (Scarpellini & Lim, 2020). Notwithstanding, we applied a systematic empirical evaluation process to analyze the final sample under the same structure and attributes, assuring homogeneity in quality and data format. ...
Users interact with chatbots for various purposes and motivations – and for different periods of time. However, since chatbots are considered social actors and given that time is an essential component of social interactions, the question arises as to how chatbots need to be designed depending on whether they aim to help individuals achieve short-, medium- or long-term goals. Following a taxonomy development approach, we compile 22 empirically and conceptually grounded design dimensions contingent on chatbots’ temporal profiles. Based upon the classification and analysis of 120 chatbots therein, we abstract three time-dependent chatbot design archetypes: Ad-hoc Supporters, Temporary Assistants, and Persistent Companions. While the taxonomy serves as a blueprint for chatbot researchers and designers developing and evaluating chatbots in general, our archetypes also offer practitioners and academics alike a shared understanding and naming convention to study and design chatbots with different temporal profiles.
Theory is an essential part of design research and helps us to explain what we see or guide what we design. In the paper, we shed light on how kernel theories are used in developing design principles in Design Science Research (DSR). We do this by reporting on a systematic literature review, from which we have extracted a set of six mechanisms to operationalize kernel theory. Each mechanism consists of an activity (e.g., "transform to" or "derive from") and an application point (e.g., meta-requirements or design principles) representing wherein the chain of concepts the kernel theory was used. The paper reflects on what we have learned about the use of kernel theories and translates this into recommendations and issues for further research. We provide researchers with guidance to use kernel theories more efficiently and give a big picture of the possibilities of kernel theory operationalization.
Many organizations are pursuing the implementation of chatbots to enable automation of service processes. However, previous research has highlighted the existence of practical setbacks in the implementation of chatbots in corporate environments. To gain practical insights on the issues related to the implementation processes from several perspectives and stages of deployment, we conducted semi-structured interviews with developers and experts of chatbot development. Using qualitative content analysis and based on a review of literature on human computer interaction (HCI), information systems (IS), and chatbots, we present an implementation framework that supports the successful deployment of chatbots and discuss the implementation of chatbots through a user-oriented lens. The proposed framework contains 101 guiding questions to support chatbot implementation in an eight-step process. The questions are structured according to the people, activity, context, and technology (PACT) framework. The adapted PACT framework is evaluated through expert interviews and a focus group discussion (FGD) and is further applied in a case study. The framework can be seen as a bridge between science and practice that serves as a notional structure for practitioners to introduce a chatbot in a structured and user-oriented manner.
Communicating with customers through live chat interfaces has become an increasingly popular means to provide real-time customer service in many e-commerce settings. Today, human chat service agents are frequently replaced by conversational software agents or chatbots, which are systems designed to communicate with human users by means of natural language often based on artificial intelligence (AI). Though cost- and time-saving opportunities triggered a widespread implementation of AI-based chatbots, they still frequently fail to meet customer expectations, potentially resulting in users being less inclined to comply with requests made by the chatbot. Drawing on social response and commitment-consistency theory, we empirically examine through a randomized online experiment how verbal anthropomorphic design cues and the foot-in-the-door technique affect user request compliance. Our results demonstrate that both anthropomorphism as well as the need to stay consistent significantly increase the likelihood that users comply with a chatbot’s request for service feedback. Moreover, the results show that social presence mediates the effect of anthropomorphic design cues on user compliance.
A recent UNESCO report reveals that most popular voice-based conversational agents are designed to be female. In addition, it outlines the potentially harmful effects this can have on society. However, the report focuses primarily on voice-based conversational agents and the analysis did not include chatbots (i.e., text-based conversational agents). Since chatbots can also be gendered in their design, we used an automated gender analysis approach to investigate three gender-specific cues in the design of 1,375 chatbots listed on the platform chatbots.org. We leveraged two gender APIs to identify the gender of the name, a face recognition API to identify the gender of the avatar, and a text mining approach to analyze gender-specific pronouns in the chatbot’s description. Our results suggest that gender-specific cues are commonly used in the design of chatbots and that most chatbots are – explicitly or implicitly – designed to convey a specific gender. More specifically, most of the chatbots have female names, female-looking avatars, and are described as female chatbots. This is particularly evident in three application domains (i.e., branded conversations, customer service, and sales). Therefore, we find evidence that there is a tendency to prefer one gender (i.e., female) over another (i.e., male). Thus, we argue that there is a gender bias in the design of chatbots in the wild. Based on these findings, we formulate propositions as a starting point for future discussions and research to mitigate the gender bias in the design of chatbots.
Social cues (e.g., gender, age) are important design features of chatbots. However, choosing a social cue design is challenging. Although much research has empirically investigated social cues, chatbot engineers have difficulties to access this knowledge. Descriptive knowledge is usually embedded in research articles and difficult to apply as prescriptive knowledge. To address this challenge, we propose a chatbot social cue configuration system that supports chatbot engineers to access descriptive knowledge in order to make justified social cue design decisions (i.e., grounded in empirical research). We derive two design principles that describe how to extract and transform descriptive knowledge into a prescriptive and machine-executable representation. In addition, we evaluate the prototypical instantiations in an exploratory focus group and at two practitioner symposia. Our research addresses a contemporary problem and contributes with a generalizable concept to support researchers as well as practitioners to leverage existing descriptive knowledge in the design of artifacts.
Digital and agile companies widely use chatbots in the form of integrations into enterprise messengers such as Slack and Microsoft Teams. However, there is a lack of empirical evidence about their action possibilities (i.e., affordances), for example, to link social interactions with third-party systems and processes. Therefore, we adopt a three-stage process. Grounded in a preliminary study and a qualitative study with 29 interviews from 17 organizations, we inductively derive rich contextual insights of 14 affordances and constraints, which serve as input for a Q-Methodology study that highlights five perceptional differences. We find that actualizing these affordances leads to higher-level affordances of chatbots that augment social information systems with affordances of traditional enterprise systems. Crossing the chasm between these, so far, detached systems contributes a novel perspective on how to balance novel digital with traditional systems, flexibility and malleability with stability and control, exploration with exploitation, and agility with discipline.
Conversational agents (CAs) are software-based systems designed to interact with humans using natural language and have attracted considerable research interest in recent years. Following the Computers Are Social Actors paradigm, many studies have shown that humans react socially to CAs when they display social cues such as small talk, gender, age, gestures, or facial expressions. However, research on social cues for CAs is scattered across different fields, often using their specific terminology, which makes it challenging to identify, classify, and accumulate existing knowledge. To address this problem, we conducted a systematic literature review to identify an initial set of social cues of CAs from existing research. Building on classifications from interpersonal communication theory, we developed a taxonomy that classifies the identified social cues into four major categories (i.e., verbal, visual, auditory, invisible) and ten subcategories. Subsequently, we evaluated the mapping between the identified social cues and the categories using a card sorting approach in order to verify that the taxonomy is natural, simple, and parsimonious. Finally, we demonstrate the usefulness of the taxonomy by classifying a broader and more generic set of social cues of CAs from existing research and practice. Our main contribution is a comprehensive taxonomy of social cues for CAs. For researchers, the taxonomy helps to systematically classify research about social cues into one of the taxonomy's categories and corresponding subcategories. Therefore, it builds a bridge between different research fields and provides a starting point for interdisciplinary research and knowledge accumulation. For practitioners, the taxonomy provides a systematic overview of relevant categories of social cues in order to identify, implement, and test their effects in the design of a CA.
This article summarizes the panel discussion at the International Conference on Wirtschafts-informatik in March 2019 in Siegen (WI 2019) and presents different perspectives on AI-based digital assistants. It sheds light on (1) application areas, opportunities, and threats as well as (2) the BISE community’s roles in the field of AI-based digital assistants. The different authors’ contributions emphasize that BISE, as a socio-technical discipline, must address the designs and the behaviors of AI-based digital assistants as well as their interconnections. They have identified multiple research opportunities to deliver descriptive and prescriptive knowledge, thereby actively shaping future interactions between users and AI-based digital assistants. We trust that these inputs will lead BISE researchers to take active roles and to contribute an IS perspective to the academic and the political discourse about AI-based digital assistants.
Enterprises are forecasted to spend more on chatbots than on mobile app development by 2021. Up to today little is known on the roles chatbots play in facilitating feedback exchange. However, digitization and automation put pressure on companies to setup digital work environments that enable reskilling of employees. Therefore, a structured analysis of feedback-related chatbots for Slack was conducted. Our results propose six archetypes that reveal the roles of chatbots in facilitating feedback exchange on performance, culture and ideas. We show that chatbots do not only consist of conversational agents integrated into instant messenger but are tightly linked to complementary front-end systems such as mobile and web apps. Like the upper part of an iceberg, the conversational agent is above water and visible within the chat, whereas many user interactions of feedback-related chatbots are only possible outside of the instant messenger. Further, we extract six design principles for chatbots as digital feedback systems. We do this by analyzing chatbots and linking empirically observed design features to (meta-)requirements derived from explanatory theory on feedback, self-determination and persuasive systems. The results suggest that chatbots benefit the social environment of conversation agents and the richness of the graphical user interface of external applications.
Design science research (DSR) aims to deliver innovative solutions for real-world problems. DSR produces Information Systems (IS) artifacts and design knowledge describing means-end relationships between problem and solution spaces. A key success factor of any DSR research endeavor is an appropriate understanding and description of the underlying problem space. However, existing DSR literature lacks a solid conceptualization of the problem space in DSR. This paper addresses this gap and suggests a conceptualization of the problem space in DSR that builds on the four key concepts of stakeholders, needs, goals, and requirements. We showcase the application of our conceptualization in two published DSR projects. Our work contributes methodologically to the field of DSR as it helps DSR scholars to explore and describe the problem space in terms of a set of key concepts and their relationships.
In Design Science Research (DSR) it is important to build on descriptive (Ω) and prescriptive (Λ) state-of-the-art knowledge in order to provide a solid grounding. However, existing knowledge is typically made available via scientific publications. This leads to two challenges: first, scholars have to manually extract relevant knowledge pieces from the data-wise unstructured textual nature of scientific publications. Second, different research results can interact and exclude each other, which makes an aggregation, combination, and application of extracted knowledge pieces quite complex. In this paper, we present how we addressed both issues in a DSR project that focuses on the design of socially-adaptive chatbots. Therefore, we outline a two-step approach to transform phenomena and relationships described in the Ω-knowledge base in a machine-executable form using ontologies and a knowledge base. Following this new approach, we can design a system that is able to aggregate and combine existing Ω-knowledge in the field of chatbots. Hence, our work contributes to DSR methodology by suggesting a new approach for theory-guided DSR projects that facilitates the application and sharing of state-of-the-art Ω-knowledge.
Text-based conversational systems, also referred to as chatbots, have grown widely popular. Current natural language understanding technologies are not yet ready to tackle the complexities in conversational interactions. Breakdowns are common, leading to negative user experiences. Guided by communication theories, we explore user preferences for eight repair strategies, including ones that are common in commercially-deployed chatbots (e.g., confirmation, providing options), as well as novel strategies that explain characteristics of the underlying machine learning algorithms. We conducted a scenario-based study to compare repair strategies with Mechanical Turk workers (N=203). We found that providing options and explanations were generally favored, as they manifest initiative from the chatbot and are actionable to recover from breakdowns. Through detailed analysis of participants' responses, we provide a nuanced understanding on the strengths and weaknesses of each repair strategy.
Information technology is rapidly changing the way how people
collaborate in enterprises. Chatbots integrated into enterprise collaboration
systems can strengthen collaboration culture and help reduce work overload. In
light of a growing usage of chatbots in enterprise collaboration systems, we
examine the influence of anthropomorphic and functional chatbot design features
on user acceptance. We conducted a survey with professionals familiar with
interacting with chatbots in a work environment. The results show a significant
effect of anthropomorphic design features on perceived usefulness, with a
strength four times the size of the effect of functional chatbot features. We
suggest that researchers and practitioners alike dedicate priorities to
anthropomorphic design features with the same magnitude as common for
functional design features in chatbot design and research.
Chatbots are software-based systems designed to interact with humans using text-based natural language and have attracted considerable interest in online service encounters. In this context, service providers face the challenge of measuring chatbot service encounter satisfaction (CSES), as most approaches are limited to post-interaction surveys that are rarely answered and often biased. As a result, service providers cannot react quickly to service failures and dissatisfied customers. To address this challenge, we investigate the application of automated sentiment analysis methods as a proxy to measure CSES. Therefore, we first compare different sentiment analysis methods. Second, we investigate the relationship between objectively computed sentiment scores of dialogs and subjectively measured CSES values. Third, we evaluate whether this relationship also exists for utterance sequences throughout the dialog. The paper contributes by proposing and applying an automatic and objective approach to use sentiment scores as a proxy to measure CSES.
In this research–in–progress paper, we employ design science research to articulate design knowledge for chatbots in higher education practice. We conducted a literature review to factor previous research into the design process. In adition, we performed a content analysis of student e-mails and forum posts from four instances of a basic Java programming course. Drawing from literature and data, we present a conceptual architecture for chatbots in higher education, discuss its rationale, and provide a proof-of-concept implementation. We conclude with a discussion including tentative design recommendations and a plan for continued research.
Conversational agents stand to play an important role in supporting behavior change and well-being in many domains. With users able to interact with conversational agents through both text and voice, understanding how designing for these channels supports behavior change is important. To begin answering this question, we designed a conversational agent for the workplace that supports workers' activity journaling and self-learning through reflection. Our agent, named Robota, combines chat-based communication as a Slack Bot and voice interaction through a personal device using a custom Amazon Alexa Skill. Through a 3-week controlled deployment, we examine how voice-based and chat-based interaction affect workers' reflection and support self-learning. We demonstrate that, while many current technical limitations exist, adding dedicated mobile voice interaction separate from the already busy chat modality may further enable users to step back and reflect on their work. We conclude with discussion of the implications of our findings to design of workplace self-tracking systems specifically and to behavior-change systems in general.
Text messaging-based conversational agents (CAs), popularly called chatbots, received significant attention in the last two years. However, chatbots are still in their nascent stage: They have a low penetration rate as 84% of the Internet users have not used a chatbot yet. Hence, understanding the usage patterns of first-time users can potentially inform and guide the design of future chatbots. In this paper, we report the findings of a study with 16 first-time chatbot users interacting with eight chatbots over multiple sessions on the Facebook Messenger platform. Analysis of chat logs and user interviews revealed that users preferred chatbots that provided either a 'human-like' natural language conversation ability, or an engaging experience that exploited the benefits of the familiar turn-based messaging interface. We conclude with implications to evolve the design of chatbots, such as: clarify chatbot capabilities, sustain conversation context, handle dialog failures, and end conversations gracefully.
From its earliest days, research in business and information
systems engineering (BISE) has been dedicated to envisioning
how information technology will change the way
we work and live. Today, technological innovation happens
at a faster pace and reaches users more quickly than
ever before. For example, while it took 75 years for the
telephone to reach 100 million users, it was 16 years for
mobile phones, 7 years for the World Wide Web, four and
a half years for Facebook (Dreischmeier et al. 2015), and
only a few weeks for Poke´mon GO (Moon 2016).
The rapid acceleration of technological diffusion confronts
BISE researchers, who usually study technological
innovations from the perspective of socio-technical systems
(Bostrom and Heinen 1977). Work systems are conceptualized
as an interplay of tasks, technologies, and
people (vom Brocke and Rosemann 2014), systems ‘‘in
which human participants and/or machines perform work
(processes and activities) using information, technology,
and other resources to produce specific products/services
for specific internal and/or external customers’’ (Alter
2013, p. 75).
Against this background, much of the current discourse
about future work systems addresses automation, as work
is increasingly performed by machines. For example,
blockchain and smart contracts can automate large parts of
the supply chain (Mendling et al. 2018), and machine
learning now facilitates automation in business areas that
were once too unstructured for automation (Willcocks et al.
2015). In such settings, people are likely to contribute to
work systems by means of creative work and exploration
(as opposed to exploitation), a distinction that O’Reilly and
Tushman (2013) referred to as organizational ambidexterity.
Therefore, from the perspective of BISE research, the
future of work poses questions about the interplay of
people and machines, as Lehrer et al. (2018) outlined in
their work on digital service innovation.
In this discussion, we differentiate between the social
intensity and the technical intensity of work and define four
basic types of work systems, as shown in Fig. 1.
A key challenge in designing conversational user interfaces is to make the conversation between the user and the system feel natural and human-like. In order to increase perceived humanness, many systems with conversational user interfaces (e.g., chatbots) use response delays to simu-late the time it would take humans to respond to a message. However, delayed responses may also negatively impact user satisfaction, particularly in situations where fast response times are expected, such as in customer service. This paper reports the findings of an online experiment in a customer service context that investigates how user perceptions differ when interacting with a chatbot that sends dynamically delayed responses compared to a chatbot that sends near-instant responses. The dynamic delay length was calculated based on the complexity of the re-sponse and complexity of the previous message. Our results indicate that dynamic response de-lays not only increase users’ perception of humanness and social presence, but also lead to greater satisfaction with the overall chatbot interaction. Building on social response theory, we provide evidence that a chatbot’s response time represents a social cue that triggers social re-sponses shaped by social expectations. Our findings support researchers and practitioners in understanding and designing more natural human-chatbot interactions.
Many conversational agents (CAs) are developed to answer users' questions in a specialized domain. In everyday use of CAs, user experience may extend beyond satisfying information needs to the enjoyment of conversations with CAs, some of which represent playful interactions. By studying a field deployment of a Human Resource chatbot, we report on users' interest areas in conversational interactions to inform the development of CAs. Through the lens of statistical modeling, we also highlight rich signals in conversational interactions for inferring user satisfaction with the instrumental usage and playful interactions with the agent. These signals can be utilized to develop agents that adapt functionality and interaction styles. By contrasting these signals, we shed light on the varying functions of conversational interactions. We discuss design implications for CAs, and directions for developing adaptive agents based on users' conversational behaviors.
Popular messaging platforms such as Slack have given rise to hundreds of chatbots that users can engage with individually or as a group. We present a Wizard of Oz study on the use of searchbots (i.e., chatbots that perform specific types of searches) during collaborative information-seeking tasks. Specifically, we study searchbots that intervene dynamically and compare between two intervention types: (1) the searchbot presents questions to users to gather the information it needs to produce results, and (2) the searchbot monitors the conversation among the collaborators, infers the necessary information, and then displays search results with no additional input from the users. We investigate three research questions: (RQ1) What is the effect of a searchbot (and its intervention type) on participants» collaborative experience' (RQ2) What is the effect of a searchbot»s intervention type on participants» perceptions about the searchbot and level of engagement with the searchbot' and (RQ3) What are participants» impressions of a dynamic searchbot? Our results suggest that dynamic searchbots can enhance users» collaborative experience and that the intervention type does not greatly affect users» perceptions and level of engagement. Participants» impressions of the searchbot suggest unique opportunities and challenges for future work.
Effective task management is essential to successful team collaboration. While the past decade has seen considerable innovation in systems that track and manage group tasks, these innovations have typically been outside of the principal communication channels: email, instant messenger, and group chat. Teams formulate, discuss, refine, assign, and track the progress of their collaborative tasks over electronic communication channels, yet they must leave these channels to update their task-tracking tools, creating a source of friction and inefficiency. To address this problem, we explore how bots might be used to mediate task management for individuals and teams. We deploy a prototype bot to eight different teams of information workers to help them create, assign, and keep track of tasks, all within their main communication channel. We derived seven insights for the design of future bots for coordinating work.
The idea of interacting with computers through natural language dates back to the 1960s, but recent technological advances have led to a renewed interest in conversational agents such as chatbots or digital assistants. In the customer service context, conversational agents promise to create a fast, convenient, and cost-effective channel for communicating with customers. Although numerous agents have been implemented in the past, most of them could not meet the expectations and disappeared. In this paper, we present our design science research project on how to design cooperative and social conversational agents to increase service quality in customer service. We discuss several issues that hinder the success of current conversational agents in customer service. Drawing on the cooperative principle of conversation and social response theory, we propose preliminary meta-requirements and design principles for cooperative and social conversational agents. Next, we will develop a prototype based on these design principles.
Every 10 years there has been a significant evolution in computer-based support for decision making. The next cycle, or generation, is due in the early 2020s and is starting to emerge. While this new cognitive generation has several important characteristics, the most significant will be the widespread use of artificial intelligence. This article describes the cognitive generation and provides recommendations for how companies should prepare for it.
Personal assistants using a command-dialogue model of speech recognition, such as Siri and Cortana, have become increasingly powerful and popular for individual use. In this paper we explore whether similar techniques could be used to create a speech-based agent system which, in a group meeting setting, would similarly monitor spoken dialogue, pro-actively detect useful actions, and carry out those actions without specific commands being spoken. Using a low-fi technical probe, we investigated how such a system might perform in the collaborative work setting and how users might respond to it. We recorded and transcribed a varied set of nine meetings from which we generated simulated lists of automated 'action items', which we then asked the meeting participants to review retrospectively. The low rankings given on these discovered items are suggestive of the difficulty in applying personal assistant technology to the group setting, and we document the issues emerging from the study. Through observations, we explored the nature of meetings and the challenges they present for speech agents.
By all accounts, 2016 is the year of the chatbot. Some commentators take the view that chatbot technology will be so disruptive that it will eliminate the need for websites and apps. But chatbots have a long history. So what's new, and what's different this time? And is there an opportunity here to improve how our industry does technology transfer?
Personal agent software is now in daily use in personal devices and in some organizational settings. While many advocate an agent sociality design paradigm that incorporates human-like features and social dialogues, it is unclear whether this is a good match for professionals who seek productivity instead of leisurely use. We conducted a 17-day field study of a prototype of a personal AI agent that helps employees find work-related information. Using log data, surveys, and interviews, we found individual differences in the preference for humanized social interactions (social-agent orientation), which led to different user needs and requirements for agent design. We also explored the effect of agent proactive interactions and found that they carried the risk of interruption, especially for users who were generally averse to interruptions at work. Further, we found that user differences in social-agent orientation and aversion to agent proactive interactions can be inferred from behavioral signals. Our results inform research into social agent design, proactive agent interaction, and personalization of AI agents.
The paper motivates, presents, demonstrates in use, and evaluates a methodology for conducting design science (DS) research in information systems (IS). DS is of importance in a discipline oriented to the creation of successful artifacts. Several researchers have pioneered DS research in IS, yet over the past 15 years, little DS research has been done within the discipline. The lack of a methodology to serve as a commonly accepted framework for DS research and of a template for its presentation may have contributed to its slow adoption. The design science research methodology (DSRM) presented here incorporates principles, practices, and procedures required to carry out such research and meets three objectives: it is consistent with prior literature, it provides a nominal process model for doing DS research, and it provides a mental model for presenting and evaluating DS research in IS. The DS process includes six steps: problem identification and motivation, definition of the objectives for a solution, design and development, demonstration, evaluation, and communication. We demonstrate and evaluate the methodology by presenting four case studies in terms of the DSRM, including cases that present the design of a database to support health assessment methods, a software reuse measure, an Internet video telephony application, and an IS planning method. The designed methodology effectively satisfies the three objectives and has the potential to help aid the acceptance of DS research in the IS discipline.
In the IS discipline, the formulation of design principles is an important vehicle to convey design knowledge that contributes beyond instantiations applicable in a limited context of use. However, their formulation still varies in terms of orientation, clarity, and precision. In this paper, we focus on the design of artifacts that are oriented towards human use, and we identify and analyze three orientations in the formulation of such design principles in IS journals - action oriented, materiality oriented, and both action and materiality oriented. We propose an effective and actionable formulation of design principles that is both clear and precise.
The success of information systems (IS) development strongly depends on the accuracy of the requirements gathered from users and other stakeholders. When developing a new IS, about 80 percent of these requirements are recorded in informal requirements documents (e.g., interview transcripts or discussion forums) using natural language. However, processing the resultant natural language requirements resources is inherently complex and often error prone due to ambiguity, inconsistency, and incompleteness. Thus, even highly qualified requirements engineers often struggle to process large amounts of natural language requirements resources efficiently and effectively. In this paper, we propose a design theory for requirement mining systems (RMSs) based on two design principles: (1) semi-automatic requirement mining and (2) usage of imported and retrieved knowledge. As part of an extensive design project, which led to these principles, we also implemented a prototype based on this design theory (REMINER). It supports requirements engineers in identifying and classifying requirements documented in natural language and allows us to evaluate the artifact's viability and the conceptual soundness of our design. The results of our evaluation suggest that an RMS based on our proposed design principles can significantly improve recall while maintaining precision levels. © 2015, Association for Information Systems. All rights reserved.
Design science research (DSR) has staked its rightful ground as an important and legitimate Information Systems (IS) research paradigm. We contend that DSR has yet to attain its full potential impact on the development and use of information systems due to gaps in the understanding and application of DSR concepts and methods. This essay aims to help researchers (1) appreciate the levels of artifact abstractions that may be DSR contributions, (2) identify appropriate ways of consuming and producing knowledge when they are preparing journal articles or other scholarly works, (3) understand and position the knowledge contributions of their research projects, and (4) structure a DSR article so that it emphasizes significant contributions to the knowledge base. Our focal contribution is the DSR knowledge contribution framework with two dimensions based on the existing state of knowledge in both the problem and solution domains for the research opportunity under study. In addition, we propose a DSR communication schema with similarities to more conventional publication patterns, but which substitutes the description of the DSR artifact in place of a traditional results section. We evaluate the DSR contribution framework and the DSR communication schema via examinations of DSR exemplar publications.
Focus groups to investigate new ideas are widely used in many research fields. The use of focus groups in design research poses interesting opportunities and challenges. Traditional focus group methods must be adapted to meet two specific goals of design research. For the refinement of an artifact design, exploratory focus groups (EFGs) study the artifact to propose improvements in the design. The cycle of build and evaluate using EFGs continues until the artifact is released for field test in the application environment. Then, the field test of the design artifact may employ confirmatory focus groups (CFGs) to establish the utility of the artifact in field use. Rigorous investigation of the artifact requires multiple CFGs to be run with opportunities for quantitative and qualitative data collection and analyses across the multiple CFGs. In this paper, we discuss the adaptation of focus groups to design research projects. We demonstrate the use of both EFGs and CFGs in a design research project in the health care field.
Until recently, organizations planning to acquire application software (AS) have had no choice but to adopt proprietary, on-premises software. With the advent of open-source and on-demand solutions, new models for developing and distributing software have entered the stage providing IS managers with more options in AS selection. On the basis of an Analytic Hierarchy Process model, we propose a framework including software package and implementation attributes on different hierarchy levels to examine how IS managers evaluate the relative fulfilment of key selection criteria by traditional, open-source, and on-demand office suites. By testing the framework with a random sample of 254 IS managers of 166 smaller and 88 mid-sized/larger firms, we validate its validity and usefulness in evaluating different software delivery models. Our empirical results show that open-source office suites were consistently perceived to be superior in meeting ease of customization (i.e., extensibility and adaptability) and cost criteria (i.e., acquisition and maintenance cost), whereas traditional software models were superior in fulfilling functionality, ease of use, and support requirements. On-demand office suites excelled in the fulfilment of time-to-value and data recoverability. We discuss further results of the comparison of the three software models, and derive practical and research implications for office suite selection.
This research investigates the meaning of “human-computer relationship” and presents techniques for constructing, maintaining, and evaluating such relationships, based on research in social psychology, sociolinguistics, communication and other social sciences. Contexts in which relationships are particularly important are described, together with specific benefits (like trust) and task outcomes (like improved learning) known to be associated with relationship quality. We especially consider the problem of designing for long-term interaction, and define relational agents as computational artifacts designed to establish and maintain long-term social-emotional relationships with their users. We construct the first such agent, and evaluate it in a controlled experiment with 101 users who were asked to interact daily with an exercise adoption system for a month. Compared to an equivalent task-oriented agent without any deliberate social-emotional or relationship-building skills, the relational agent was respected more, liked more, and trusted more, even after four weeks of interaction. Additionally, users expressed a significantly greater desire to continue working with the relational agent after the termination of the study. We conclude by discussing future directions for this research together with ethical and other ramifications of this work for HCI designers.
Design research (DR) positions information technology artifacts at the core of the Information Systems discipline. However, dominant DR thinking takes a technological view of the IT artifact, paying scant attention to its shaping by the organizational context. Consequently, existing DR methods focus on building the artifact and relegate evaluation to a subsequent and separate phase. They value technological rigor at the cost of organizational relevance, and fail to recognize that the artifact emerges from interaction with the organizational context even when its initial design is guided by the researchers' intent. We propose action design research (ADR) as a new DR method to address this problem. ADR reflects the premise that IT artifacts are ensembles shaped by the organizational context during development and use. The method conceptualizes the research process as containing the inseparable and inherently interwoven activities of building the IT artifact, intervening in the organization, and evaluating it concurrently. The essay describes the stages of ADR and associated principles that encapsulate its underlying beliefs and values. We illustrate ADR through a case of competence management at Volvo IT.
The paper motivates, presents, demonstrates in use, and evaluates
a methodology for conducting design science (DS) research in information
systems (IS). DS is of importance in a discipline oriented to the
creation of successful artifacts. Several researchers have pioneered
DS research in IS, yet over the past 15 years, little DS research
has been done within the discipline. The lack of a methodology to
serve as a commonly accepted framework for DS research and of a template
for its presentation may have contributed to its slow adoption. The
design science research methodology (DSRM) presented here incorporates
principles, practices, and procedures required to carry out such
research and meets three objectives: it is consistent with prior
literature, it provides a nominal process model for doing DS research,
and it provides a mental model for presenting and evaluating DS research
in IS. The DS process includes six steps: problem identification
and motivation, definition of the objectives for a solution, design
and development, demonstration, evaluation, and communication. We
demonstrate and evaluate the methodology by presenting four case
studies in terms of the DSRM, including cases that present the design
of a database to support health assessment methods, a software reuse
measure, an Internet video telephony application, and an IS planning
method. The designed methodology effectively satisfies the three
objectives and has the potential to help aid the acceptance of DS
research in the IS discipline.
While group chat is becoming increasingly popular for team collaboration, these systems generate long streams of unstructured back-and-forth discussion that are difficult to comprehend. In this work, we investigate ways to enrich the representation of chat conversations, using techniques such as tagging and summarization, to enable users to better make sense of chat. Through needfinding interviews with 15 active group chat users, who were shown mock-up alternative chat designs, we found the importance of structured representations, including signals such as discourse acts. We then developed Tilda, a prototype system that enables people to collaboratively enrich their chat conversation while conversing. From lab evaluations, we examined the ease of marking up chat using Tilda as well as the effectiveness of Tilda-enabled summaries for getting an overview. From a field deployment, we found that teams actively engaged with Tilda both for marking up their chat as well as catching up on chat.
This study investigates whether social- versus task-oriented interaction of virtual shopping assistants differentially benefits low versus high Internet competency older consumers with respect to social (perceived interactivity, trust), cognitive (perceived information load), functional (self-efficacy, perceived ease of use, perceived usefulness), and behavioral intent (website patronage intent) outcomes in an online shopping task. A total of 121 older adults (61–89 years) participated in a laboratory experiment with a 2 (digital assistant interaction style: (social-vs. task-oriented) × 2 (user Internet competency: low vs. high) × 2 (user exchange modality: text vs. voice) between-subjects design. The results revealed that users' Internet competency and the digital assistant's conversational style had significant interaction effects on social, functional, and behavioral intent outcomes. Social-oriented digital assistants lead to superior social outcomes (enhanced perceptions of two-way interactivity and trust in the integrity of the site) for older users with high Internet competency, who need less task-related assistance. On the other hand, low-competency older users showed significantly superior cognitive (lower perceived information load) and functional outcomes (greater perceived ease and self-efficacy of using the site) when the digital assistant employed a task-oriented interaction style. Theoretical and agent design implications are discussed.
With the rising interest in design science research (DSR), it is crucial to engage in the ongoing debate on what constitutes an acceptable contribution for publishing DSR—the design artifact, the design theory, or both. In this editorial, we provide some constructive guidance across different positioning statements with actionable recommendations for DSR authors and reviewers. We expect this editorial to serve as a foundational step toward clarifying misconceptions about DSR contributions and to pave the way for the acceptance of more DSR papers to top IS journals.
The conversational interface has become a hot topic in the past year or so, providing the primary means of interaction with chatbots, messaging apps, and virtual personal assistants. Major tech companies have been making huge investments in the supporting technologies of artificial intelligence, such as deep learning and natural language processing, with the aim of creating systems that will enable users of smartphones and other devices to obtain information and access services in a natural, conversational way. Yet the vision of the conversational interface is not new, and indeed there is a history of research in dialogue systems, voice user interfaces, embodied conversational agents, and chatbots that goes back more than fifty years. This chapter explores what has changed to make the conversational interface particularly relevant today, examines some key issues from earlier work that could inform the next generation of conversational systems, and highlights some challenges for future work.
In this chapter, we revisit the main theories of human emotion and personality and their implications for the development of affective conversational agents. We focus on the role that emotion plays for adapting the agents' behaviour and how this emotional responsivity can be conveniently modified by rendering a consistent artificial personality. The multiple applications of affective CAs are addressed by describing recent experiences in domains such as pedagogy, computer games, and computer-mediated therapy.
Technological knowledge has been characterized as having a scope that is specific to a particular problem. However, the information systems community is exploring forms of design science research that provide a promising avenue to technological knowledge with broader scope: design theories. Because design science research is materially prescriptive, it requires a different perspective in developing the breadth of applications of design theories. In this paper we propose different concepts that embody forms of general technological knowledge The concept of projectability, developed originally as a means of distinguishing realized generalizations from unrealized generalizations, helps explain how design theories, being prescriptive, possess a different form of applicability. The concept of entrenchment describes the use of a theory in many projections. Together these concepts provide a means for comparative discussions of the importance of design theories. Projectable design theories guide designers in the design of artifacts similar in principle, but different in context. These can also help design researchers understand interrelationships between design theories. © IFIP International Federation for Information Processing 2014
Evaluation of design artefacts and design theories is a key activity in Design Science
Research (DSR), as it provides feedback for further development and (if done
correctly) assures the rigour of the research. However, the extant DSR literature
provides insufficient guidance on evaluation to enable Design Science Researchers
to effectively design and incorporate evaluation activities into a DSR project that can
achieve DSR goals and objectives. To address this research gap, this research paper
develops, explicates, and provides evidence for the utility of a Framework for
Evaluation in Design Science (FEDS) together with a process to guide design science
researchers in developing a strategy for evaluating the artefacts they develop within
a DSR project. A FEDS strategy considers why, when, how, and what to evaluate.
FEDS includes a two-dimensional characterisation of DSR evaluation episodes
(particular evaluations), with one dimension being the functional purpose of the
evaluation (formative or summative) and the other dimension being the paradigm
of the evaluation (artificial or naturalistic). The FEDS evaluation design process is
comprised of four steps: (1) explicate the goals of the evaluation, (2) choose the
evaluation strategy or strategies, (3) determine the properties to evaluate, and
(4) design the individual evaluation episode(s). The paper illustrates the framework
with two examples and provides evidence of its utility via a naturalistic, summative
evaluation through its use on an actual DSR project.
European Journal of Information Systems advance online publication,
11 November 2014; doi:10.1057/ejis.2014.36
Open Access
Following Langer (1992), this article reviews a series of experimental studies that demonstrate that individuals mindlessly apply social rules and expectations to computers. The first set of studies illustrates how individuals overuse human social categories, applying gender stereotypes to computers and ethnically identifying with computer agents. The second set demonstrates thatpeople exhibit overlearned social behaviors such as politeness and reciprocity toward computers. In the third set of studies, premature cognitive commitments are demonstrated: A specialist television set is perceived as providing better content than a generalist television set. A final series of studies demonstrates the depth of social responses with respect to computer ‘personality.’ Alternative explanations for these findings, such asanthropomorphism and intentional social responses, cannot explain the results. We conclude with an agenda for future research.
Much of the literature on enterprise systems implementation suggests that ERP systems should support the strategic objectives of the organization. In fact some ERP vendors tend to assume that implementing their products is a straightforward translation from strategy to IT-enabled business processes. But the strategic management literature reveals that corporate strategy is often contested terrain. Conflict over strategy has the potential to delay or derail ERP implementation efforts, leading to cost overruns, failures, and lack of expected benefits.This paper seeks to understand and explain how conflicts over strategy can affect the success of ERP implementations. We studied one firm's attempt to pursue information technology-enabled enterprise integration using critical ethnography. The firm is a small-to-medium sized enterprise within a large conglomerate in the Asia-Pacific region. Our findings show that, as the dominant actors and political agendas changed, so too did the company's enterprise integration strategy. The ERP system was designed to support a future corporate world and work environment that never arrived.
This article describes an evaluation of six different methods for prioritizing software requirements. Based on the quality requirements for a telephony system, the authors individually used all six methods on separate occasions to prioritize the requirements. The methods were then characterized according to a number of criteria from a user's perspective. We found the analytic hierarchy process to be the most promising method, although it may be problematic to scale-up. In an industrial follow-up study we used the analytic hierarchy process to further investigate its applicability. We found that the process is demanding but worth the effort because of its ability to provide reliable results, promote knowledge transfer and create consensus among project members.
In online shopping environments, the product-advising function originally performed by salespeople is being increasingly taken over by software-based product recommendation agents (PRAs). However, the literature has mostly focused on the functionality design and utilitarian value of such decision support systems, mostly ignoring the potential social influence they could exert on their users. The objective of this study is to apply a social relationship perspective to the design of interfaces for PRAs. We investigate the effects of applying anthropomorphic interfaces-namely, humanoid embodiment and voice output-on users' perceived social relationship with a technological and software-based artifact designed for electronic commerce contexts. The findings from a laboratory experiment indicate that using humanoid embodiment and human voice-based communication significantly influences users' perceptions of social presence, which in turn enhances users' trusting beliefs, perceptions of enjoyment, and ultimately, their intentions to use the agent as a decision aid. These results extend the applicability of theories concerning traditional shopper-salesperson relationships to customers' interactions with technological artifacts residing on Web sites-that is, the recommendation agent software-and provide practitioners with guidelines on how to design Internet stores with the goal of building social relationships with online shoppers and enhancing their overall shopping experiences.
Humane anthropomorphic agents: the quest for the outcome measure
- E André
André, E., et al.: Humane anthropomorphic agents: the quest for the outcome measure;
position paper. In: AIS SIGPrag, Munich, 15-18 December 2019, 2019 pre-ICIS Workshop
Proceedings "Values and Ethics in the Digital Age", Munich, 14 December 2019 (2019)