No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Communicating Service Offers in a Conversational User Interface: An Exploratory Study of User Preferences in Chatbot Interaction
... However, three journal articles were duplicates of papers identified by our previous SMS. Finally, two HCI conference papers from these resources were added to our primary studies as [24] and [25]. To give readers a better understanding of the SMS, we report the procedure in the supplementary material, available online, and describe the validity threats to the SMS in Section 6. ...
... These experiments measured usability characteristics referred to effectiveness, efficiency and satisfaction as shown in Table 1. Some of the primary studies ( [19], [26], [27], [28], [29], [30], [31], [32], [33]) consider all three of these aspects, others ( [24], [25], [34], [35], [36], [37], [38], [39], [40]) investigate efficiency and satisfaction, others again ( [41], [42], [43], [44], [45]) consider only satisfaction, whereas only one ( [46]) evaluated both effectiveness and satisfaction. ...
... Within the primary studies, chatbots are dedicated to various domains. Most chatbots are used as personal assistants [25], [26], [29], [30], [34], [37], [38], [39], [40], [42], [43], [44], [45], [46], [47], especially in the healthcare domain [32], [35], [38], [44], some act as recommenders [27], [28], [31], emotion-aware conversational agents [41] and e-commerce chatbots [33], [36]. Nevertheless, none of the above chatbots is applied as a modelling tool like the SOCIO chatbot. ...
Context: Recent developments in natural language processing have facilitated the adoption of chatbots in typically collaborative software engineering tasks (such as diagram modelling). Families of experiments can assess the performance of tools and processes and, at the same time, alleviate some of the typical shortcomings of individual experiments (e.g., inaccurate and potentially biased results due to a small number of participants). Objective: Compare the usability of a chatbot for collaborative modelling (i.e., SOCIO) and an online web tool (i.e., Creately). Method: We conducted a family of three experiments to evaluate the usability of SOCIO against the Creately online collaborative tool in academic settings. Results: The student participants were faster at building class diagrams using the chatbot than with the online collaborative tool and more satisfied with SOCIO. Besides, the class diagrams built using the chatbot tended to be more concise -albeit slightly less complete. Conclusion: Chatbots appear to be helpful for building class diagrams. In fact, our study has helped us to shed light on the future direction for experimentation in this field and lays the groundwork for researching the applicability of chatbots in diagramming.
... However, three journal articles were duplicates of papers identified by our previous SMS. Finally, two HCI conference papers from these resources were added to our primary studies as [24] and [25]. To give readers a better understanding of the SMS, we report the procedure in the supplementary material and describe the validity threats to the SMS in Section 6. ...
... These experiments measured usability characteristics referred to effectiveness, efficiency and satisfaction as shown in Table 1. Some of the primary studies ( [19], [26], [27], [28], [29], [30], [31], [32], [33]) consider all three of these aspects, others ( [24], [25], [34], [35], [36], [37], [38], [39], [40]) investigate efficiency and satisfaction, others again ( [41], [42], [43], [44], [45]) consider only satisfaction, whereas only one ( [46]) evaluated both effectiveness and satisfaction. Satisfaction is again the usability characteristic of most concern to researchers since it is evaluated most often. ...
... Within the primary studies, chatbots are dedicated to various domains. Most chatbots are used as personal assistants [25], [26], [29], [30], [34], [37], [38], [39], [40], [42], [43], [44], [45], [46], [47], especially in the healthcare domain [32], [35], [38], [44], some act as recommenders [27], [28], [31], emotion-aware conversational agents [41] and e-commerce chatbots [33], [36]. Nevertheless, none of the above chatbots is applied as a modelling tool like the SOCIO chatbot. ...
Context: Recent developments in natural language processing have facilitated the adoption of chatbots in typically collaborative software engineering tasks (such as diagram modelling). Families of experiments can assess the performance of tools and processes and, at the same time, alleviate some of the typical shortcomings of individual experiments (e.g., inaccurate and potentially biased results due to a small number of participants). Objective: Compare the usability of a chatbot for collaborative modelling (i.e., SOCIO) and an online web tool (i.e., Creately). Method: We conduct a family of three experiments to evaluate the usability of SOCIO against the Creately online collaborative tool in academic settings. Results: The student participants were faster at building class diagrams using the chatbot than with the online collaborative tool and more satisfied with SOCIO. Besides, the class diagrams built using the chatbot tended to be more concise albeit slightly less complete. Conclusion: Chatbots appear to be helpful for building class diagrams. In fact, our study has helped us to shed light on the future direction for experimentation in this field and lays the groundwork for researching the applicability of chatbots in diagramming.
... Conversational agents belong to the systems designed to enable Human-Computer Interaction [1]. These systems represent a new form of interaction between humans and machines, allowing the user to interact using the tool most used by humans: natural language [2]. ...
... In contrast, Closed domains have limited knowledge about a specific domain and are designed to have conversations focused on one or a few specific topics [29]. Based on the length of the conversation, two other types of chatbots can be distinguished: systems based on Short-Term and Long-term relations [1]. A short-term relation is characterized by a one-shot interaction, also called single-turn [30], in which the response is generated solely based on a single message, without collecting the user's information. ...
Conversational agents are systems with great potential to enhance human-computer interaction in industrial settings. Although the number of applications of conversational agents in many fields is growing, there is no shared view of the elements to design and implement for chatbots in the industrial field. The paper presents the combination of many research contributions into an integrated conceptual architecture, for developing industrial conversational agents using Nickerson's methodology. The conceptual architecture consists of five core modules; every module consists of specific elements and approaches. Furthermore, the paper defines a taxonomy from the study of empirical applications of manufacturing conversational agents. Indeed, some applications of chatbots in manufacturing are available but those have never been collected in single research. The paper fills this gap by analyzing the empirical cases and presenting a qualitative analysis, with verification of the proposed taxonomy. The contribution of the article is mainly to illustrate the elements needed for the development of a conversational agent in manufacturing: researchers and practitioners can use the proposed conceptual architecture and taxonomy to more easily investigate, define, and develop all the elements for chatbot implementation.
... Articles related to this theme are presented in Table 10, where Ng et al. [14] stated that fintech chatbots are affected by privacy, social presence, and trust factors. Aslam et al. [16] reported a positive correlation between privacy and chatbot acceptance in financial industry, to which Folstad & Halvorsrud [49] added by Human factor is still needed to input, train, and help chatbots, resulting in more chatbot trust. ...
The Fintech industry, particularly banks, has witnessed a profound transformation with the integration of Artificial Intelligence chatbots, redefining customer experience and engagement. As Fintech firms increasingly integrate AI chatbots into their platforms, understanding customer perceptions becomes paramount for strategic decision-making and sustained success. To unravel the complexities of this convergence, a holistic examination is needed, encompassing not only the technological aspects but also the strategic dimensions that underpin competitive advantage. In this context, the role of intellectual property, particularly patents, emerges as a critical factor shaping the innovation landscape. This research aims to comprehensively investigate customers' perceptions towards AI chatbots in the Fintech industry, with a specific focus on technological convergence. The study seeks to analyze the impact of cutting- edge AI chatbot technologies, including those protected by patents, on user attitudes and overall customer experience within the dynamic fintech landscape. This study provides a comprehensive review of 40 empirical studies on AI chatbots in the fintech industry, particularly the banking sector, featuring patented innovations using the PRISMA methodology. Study outcomes illustrate emerging themes related to consumer behavior and response to financial chatbots in terms of acceptance and adoption intention. Additionally, four key factors that influence how people perceive, anticipate, and engage with fintech chatbots, namely satisfaction, trust, anthropomorphism, and privacy are explored. In conclusion, the finance industry's effective integration and broad use of AI chatbots is dependent on the convergence of four factors: satisfaction, privacy, trust, and anthropomorphism. Current study offers a strong basis for analysing and resolving the obstacles to AI chatbot acceptance and deployment in the financial sector by addressing all these elements extensively. This exploration of technological convergence in fintech industry by analyzing customers' behavior and response to financial chatbots not only contributes to a comprehensive understanding of its intricacies but also serves as a foundation for development and deployment of user-centric fintech chatbots.
CRM chatbots are widely used as an artificial intelligence tool to improve company-client based interactions. In this study, we examine users' preference for CRM chatbots. We conducted a conjoint analysis in the context of cellular providers to examine the effect of various CRM attributes on user preferences. We developed and assessed twelve conjoint profiles by varying levels of attributes derived from previous studies: appearance, conversation style, response behavior, and user input methods. The heterogeneities of user are considered: demographic, personality, and experience. Using 302 participants, we obtained seven user preference segments. Furthermore, two chatbot prototypes were developed: considering the user's preference structure and the other disregarding it. Prototype testing to twenty participants showed that the chatbot designed based on user preferences scored higher in preference level, usability, user experience, and intention to use compared to the one that did not.
For chatbots to be broadly adopted by users, it is critical that they are experienced as useful and pleasurable. While there is an emerging body of research concerning user uptake and use of chatbots, there is a lack of theoretically grounded studies detailing what constitutes good or poor chatbot user experiences. In this paper, we present findings from a questionnaire study involving more than 200 chatbot users who reported on episodes of chatbot use that they found particularly satisfactory or frustrating. The user reports were analysed with basis in theory on user experience, with particular concern for pragmatic and hedonic attributes. We found that pragmatic attributes such as efficient assistance (positive) and problems with interpretation (negative) were important elements in user reports of satisfactory and frustrating episodes. Hedonic attributes such as entertainment value (positive) and strange and rude responses (negative) were also frequently mentioned. Older participants tended to report on pragmatic attributes more often, whereas younger participants tended to report on hedonic attributes more often. Drawing on the findings, we propose four high-level lessons learnt that may benefit chatbot service providers, and we suggest relevant future research.
Text-based conversational systems, also referred to as chatbots, have grown widely popular. Current natural language understanding technologies are not yet ready to tackle the complexities in conversational interactions. Breakdowns are common, leading to negative user experiences. Guided by communication theories, we explore user preferences for eight repair strategies, including ones that are common in commercially-deployed chatbots (e.g., confirmation, providing options), as well as novel strategies that explain characteristics of the underlying machine learning algorithms. We conducted a scenario-based study to compare repair strategies with Mechanical Turk workers (N=203). We found that providing options and explanations were generally favored, as they manifest initiative from the chatbot and are actionable to recover from breakdowns. Through detailed analysis of participants' responses, we provide a nuanced understanding on the strengths and weaknesses of each repair strategy.
Text messaging-based conversational agents (CAs), popularly called chatbots, received significant attention in the last two years. However, chatbots are still in their nascent stage: They have a low penetration rate as 84% of the Internet users have not used a chatbot yet. Hence, understanding the usage patterns of first-time users can potentially inform and guide the design of future chatbots. In this paper, we report the findings of a study with 16 first-time chatbot users interacting with eight chatbots over multiple sessions on the Facebook Messenger platform. Analysis of chat logs and user interviews revealed that users preferred chatbots that provided either a 'human-like' natural language conversation ability, or an engaging experience that exploited the benefits of the familiar turn-based messaging interface. We conclude with implications to evolve the design of chatbots, such as: clarify chatbot capabilities, sustain conversation context, handle dialog failures, and end conversations gracefully.
Chatbots have been around since the 1960's, but recently they have risen in popularity especially due to new compatibility with social networks and messenger applications. Chatbots are different from traditional user interfaces, for they unveil themselves to the user one sentence at a time. Because of that, users may struggle to interact with them and to understand what they can do. Hence, it is important to support designers in deciding how to convey chatbots' features to users, as this might determine whether the user continues to chat or not. As a first step in this direction, in this paper our goal is to analyze the communicative strategies that have been used by popular chatbots to convey their features to users. To perform this analysis we use the Semiotic Inspection Method (SIM). As a result we identify and discuss the different strategies used by the analyzed chatbots to present their features to users. We also discuss the challenges and limitations of using SIM on such interfaces.
Chatbots focusing on a narrow domain of expertise are in great rise. As several tasks require multiple expertise, a designer may integrate multiple chatbots in the background or include them as interlocutors in a conversation. We investigated both scenarios by means of a Wizard of Oz experiment, in which participants talked to chatbots about visiting a destination. We analyzed the conversation content, users' speech, and reported impressions. We found no significant difference between single-and multi-chatbots scenarios. However, even with equivalent conversation structures, users reported more confusion in multi-chatbots interactions and adopted strategies to organize turn-taking. Our findings indicate that implementing a meta-chatbot may not be necessary, since similar conversation structures occur when interacting to multiple chatbots, but different interactional aspects must be considered for each scenario.
There is a growing interest in chatbots, which are machine agents serving as natural language user interfaces for data and service providers. However, no studies have empirically investigated people’s motivations for using chatbots. In this study, an online questionnaire asked chatbot users (N = 146, aged 16–55 years) from the US to report their reasons for using chatbots. The study identifies key motivational factors driving chatbot use. The most frequently reported motivational factor is “productivity”; chatbots help users to obtain timely and efficient assistance or information. Chatbot users also reported motivations pertaining to entertainment, social and relational factors, and curiosity about what they view as a novel phenomenon. The findings are discussed in terms of the uses and gratifications theory, and they provide insight into why people choose to interact with automated agents online. The findings can help developers facilitate better human–chatbot interaction experiences in the future. Possible design guidelines are suggested, reflecting different chatbot user motivations.
This paper discusses methods with which one can simultaneously counterbalance immediate sequential effects and pairing of conditions and stimuli in a within-subjects design using pairs of Latin squares. Within-subjects (repeated measures) experiments are common in human factors research. The designer of such an experiment must develop a scheme to ensure that the conditions and stimuli are not confounded, or randomly order stimuli and conditions. While randomization ensures balance in the long run, it is possible that a specific random sequence may not be acceptable. An alternative to randomization is to use Latin squares. The usual Latin square design ensures that each condition appears an equal number of times in each column of the square. Latin squares have been described which have the effect of counterbalancing immediate sequential effects. The objective of this work was to extend these earlier efforts by developing procedures for designing pairs of Latin squares which ensure complete counterbalancing of immediate sequential effects for both conditions and stimuli, and also ensure that conditions and stimuli are paired in the squares an equal number of times.
Spoken dialogue systems allow users to interact with computer-based applications such as databases and expert systems by using natural spoken language. The origins of spoken dialogue systems can be traced back to Artificial Intelligence research in the 1950s concerned with developing conversational interfaces. However, it is only within the last decade or so, with major advances in speech technology, that large-scale working systems have been developed and, in some cases, introduced into commercial environments. As a result many major telecommunications and software companies have become aware of the potential for spoken dialogue technology to provide solutions in newly developing areas such as computer-telephony integration. Voice portals, which provide a speech-based interface between a telephone user and Web-based services, are the most recent application of spoken dialogue technology. This article describes the main components of the technology---speech recognition, language understanding, dialogue management, communication with an external source such as a database, language generation, speech synthesis---and shows how these component technologies can be integrated into a spoken dialogue system. The article describes in detail the methods that have been adopted in some well-known dialogue systems, explores different system architectures, considers issues of specification, design, and evaluation, reviews some currently available dialogue development toolkits, and outlines prospects for future development.
In recent years, service providers have identified the proactive postsales service (PPS) as a viable measure for preempting service failures and their negative consequences. Due to the high costs associated with PPSs, companies are looking for ways to increase their efficiency. To understand how companies can increase their revenues and lower their costs, this study investigates how cross-selling activities and different media types affect the impact of a PPS on inbound service calls and customer churn. Based on a large-scale field experiment in the telecommunications industry, as well as a controlled lab experiment, the results demonstrate the overall effectiveness of the PPS and indicate two mediating effects. While the effect of cross-selling on customer churn and service calls is mediated by the customers’ uncertainty regarding the company’s motives, it is the customers’ perception of privacy invasion that mediates the influence of the contact medium on the effectiveness of the PPS. Our finding that PPS contacts have to be clear in their message and should not be perceived as invasive is an indication of the importance of service-(post)sales ambidexterity.
With recent advances in natural language understanding techniques and farfield microphone arrays, natural language interfaces, such as voice assistants and chatbots, are emerging as a popular new way to interact with computers. They have made their way out of the industry research labs and into the pockets, desktops, cars and living rooms of the general public. But although such interfaces recognize bits of natural language, and even voice input, they generally lack conversational competence, or the ability to engage in natural conversation. Today's platforms provide sophisticated tools for analyzing language and retrieving knowledge, but they fail to provide adequate support for modeling interaction. The user experience (UX) designer or software developer must figure out how a human conversation is organized, usually relying on commonsense rather than on formal knowledge. Fortunately, practitioners can rely on conversation science.
This book adapts formal knowledge from the field of Conversation Analysis (CA) to the design of natural language interfaces. It outlines the Natural Conversation Framework (NCF), developed at IBM Research, a systematic framework for designing interfaces that work like natural conversation. The NCF consists of four main components: 1) an interaction model of "expandable sequences," 2) a corresponding content format, 3) a pattern language with 100 generic UX patterns and 4) a navigation method of six basic user actions. The authors introduce UX designers to a new way of thinking about user experience design in the context of conversational interfaces, including a new vocabulary, new principles and new interaction patterns. User experience designers and graduate students in the HCI field as well as developers and conversation analysis students should find this book of interest.
Abstract
Chatbots on social networking sites are a recent innovation in computer-mediated marketing communication. In this study, 245 Facebook users between 18 and 35 years of age (Mage = 25.97, SD = 4.92) were asked to order tickets for the movies through Cinebot, a Facebook chatbot specifically built for the study. Afterwards, they were asked to evaluate their experiences via an online survey. The first purpose of this article was to investigate whether and how perceived helpfulness and usefulness of a chatbot consulted on the Facebook Messenger platform affected perceived intrusiveness of chatbot-initiated advertising in a later stage. In a second analysis, the relation between perceived intrusiveness and patronage intentions (i.e. purchase and recommendation intention of the product) was investigated. In addition, the role of message acceptance as a mediator and perceived message relevance as a moderator in this latter model were explored. As, to the best of our knowledge, our study is the first to investigate chatbot advertising, our research findings may hold important managerial implications.
This study was undertaken to analyze whether luxury fashion retail brands can adhere to their core essence of providing personalized care through e-services rather than through traditional face-to-face interactions, particularly through Chatbot, an emerging digital tool offering convenient, personal, and unique customer assistance. The authors use customer data to test a five-dimension model measuring Chatbot for customer perceptions of interaction, entertainment, trendiness, customization, and problem-solving. The study reveals that Chatbot e-service provides interactive and engaging brand/customer service encounters. Marketers and managers in the luxury context can adopt the instrument to measure whether e-service agents provide desired outcomes and to determine whether they should adopt Chatbot virtual assistance.
Two paradigms characterize much of the research in the Information Systems discipline: behavioral science and design science. The behavioral-science paradigm seeks to develop and verify theories that explain or predict human or organizational behavior. The design-science paradigm seeks to extend the boundaries of human and organizational capabilities by creating new and innovative artifacts. Both paradigms are foundational to the IS discipline, positioned as it is at the confluence of people, organizations, and technology. Our objective is to describe the performance of design-science research in Information Systems via a concise conceptual framework and clear guidelines for understanding, executing, and evaluating the research. In the design-science paradigm, knowledge and understanding of a problem domain and its solution are achieved in the building and application of the designed artifact. Three recent exemplars in the research literature are used to demonstrate the application of these guidelines. We conclude with an analysis of the challenges of performing high-quality design-science research in the context of the broader IS community.
A strong case has been made for iterative design, that is, progressing through several versions of a user interface design using feedback from users to improve each prototype. One obstacle to wider adoption of this approach is the perceived difficulty of obtaining useful data from users. This paper argues that quantitative experimental methods may not be practical at early stages of design, but a behavioural record used in conjunction with think-aloud protocols can provide a designer with the information needed to evaluate an early prototype in a cost-effective manner. Further, it is proposed that a method for obtaining this data can be specified which is straightforward enough to be used by people with little or no training in human factors.Two studies are reported in which trainee designers evaluated a user interface by observing a user working through some set tasks. These users were instructed to think aloud as they worked in a procedure described as “cooperative evaluation”. The instruction received by the designers took the form of a brief how-to-do-it manual. Study 1 examines the effectiveness of the trainee designers as evaluators of an existing bibliographic database. The problems detected by each team were compared with the complete set of problems detected by all the teams and the problems detected by the authors in a previous and more extensive evaluation. Study 2 examined the question of whether being the designer of a system makes one better or worse at evaluating it and whether designers can predict the problems users will experience in advance of user testing.
Conversational Design. A Book Apart
- Erika Hall
- Hall Erika
Designing Bots: Creating Conversational Experiences
- Amir Shevat
- Reilly
- Ma Amir Boston
- Shevat
- Shevat Amir