ArticlePDF Available

Chatbots for Brand Representation in Comparison with Traditional Websites

Authors:

Abstract and Figures

Chatbots have gained enormous popularity in recent years. IT giants such as Microsoft, Google and Facebook have taken an interest in automated conversations. Messaging apps like WhatsApp and Facebook Messenger are playing an increasingly important role in smartphone usage and communication in general perfect conditions for chatbots. This paper provides an introduction of recent chatbot development in general and in customer environments. As part of this work, a chatbot was developed for the Austrian IT company CodeFluegel GmbH. The chatbot, named Theodore, provides information about CodeFluegel via Facebook Messenger and webchat, like the existing company website. The design process and implementation of the chatbot as well as architectural considerations are explained throughout this document. In a user study, participants perform typical tasks with the website and the chatbot. The usage of both platforms is evaluated in order to identify advantages and disadvantages as well as differences compared to the other technology and to draw conclusions. Findings to this study indicate promising prospects for chatbots as alternative touchpoints for customers and others and a way to replace and enhance traditional websites.
Content may be subject to copyright.
PaperChatbots for Brand Representation in Comparison with Traditional Websites
Chatbots for Brand Representation in Comparison with
Traditional Websites
https://doi.org/10.3991/ijim.v14i18.13433
Johannes Kühnel, Markus Ebner, Martin Ebner ()
Graz University of Technology, Graz, Austria
martin.ebner@tugraz.at
AbstractChatbots have gained enormous popularity in recent years. IT gi-
ants such as Microsoft, Google and Facebook have taken an interest in automated
conversations. Messaging apps like WhatsApp and Facebook Messenger are
playing an increasingly important role in smartphone usage and communication
in general perfect conditions for chatbots. This paper provides an introduction of
recent chatbot development in general and in customer environments. As part of
this work, a chatbot was developed for the Austrian IT company CodeFluegel
GmbH. The chatbot, named Theodore, provides information about CodeFluegel
via Facebook Messenger and webchat, like the existing company website. The
design process and implementation of the chatbot as well as architectural consid-
erations are explained throughout this document. In a user study, participants per-
form typical tasks with the website and the chatbot. The usage of both platforms
is evaluated in order to identify advantages and disadvantages as well as differ-
ences compared to the other technology and to draw conclusions. Findings to this
study indicate promising prospects for chatbots as alternative touchpoints for cus-
tomers and others and a way to replace and enhance traditional websites.
KeywordsChatbots, messenger, websites, natural language understanding,
brand representation
1 Introduction
Chatbots and conversational interfaces have become increasingly popular in recent
years. At the developer conference Build 2016, Microsoft’s CEO Satya Nadella boldly
proclaimed that “Bots are the new apps” [1]. Several factors have nurtured this trend.
On the one hand, messaging applications like WhatsApp and Facebook Messenger are
attracting billions of users (see Table 1) and occupying a large portion of a user’s screen
time [2]. On the other hand, major technology companies like IBM, Microsoft, Google,
Facebook and Amazon have taken an interest in providing development tools and nat-
ural language processing centered around chatbots, resulting in easier development and
wider acceptance and adaption of conversational interfaces. [3]
Since the invention of the Internet, companies have been using websites as means to
get in touch with potential customers, employees and generally interested parties. This
paper aims to evaluate the feasibility and usability of chatbots in such a context, given
18
http://www.i-jim.org
PaperChatbots for Brand Representation in Comparison with Traditional Websites
the immense popularity of previously mentioned messenger and social media platforms.
For this evaluation, a chatbot was developed for the Austrian IT company CodeFlügel
GmbH1. The chatbot provides information about CodeFlügel via Facebook Messenger
and webchat, similar to the existing company website. The design process as well as
architectural considerations and details of the implementation are explained throughout
this document.
For the evaluation itself, a user study is conducted where participants perform typical
tasks with the website and the chatbot. The usage of both platforms is evaluated in order
to identify advantages and disadvantages as well as differences compared to the other
technology. Based on these evaluations, conclusions will further be drawn concerning,
for example, differences in speed and user satisfaction. Findings to this study indicate
promising prospects for chatbots as alternative touchpoints for customers and others
and a way to replace and enhance traditional websites.
Table 1. Monthly active users of popular messaging platforms with bot support according to
[4], [5] and Telegram2.
Messaging Platform
Monthly Active Users (in millions)
WhatsApp
1,500
Facebook Messenger
1,300
Weixin / Wechat
1,083
Twitter
326
Skype
300
Telegram
200
LINE
194
2 Related Work
Chatbots provide an advantage compared to traditional mobile apps. Apps are prone
to app fatigue, which describes the tiredness of users to install new apps. Other driv-
ing factors of bots are the growth of messaging apps and social media, as well as the
support of large companies like Facebook and Microsoft in the recent years [3] [6].
With these factors in mind, this chapter focuses on some case studies around the devel-
opment and research of chatbots and conversational interfaces.
Shawar et al. showed a chatbot answering Frequently Asked Questions (FAQ) for
the School of Computing3 (University of Leeds). The bot used pre-processed data of the
school's online FAQ on their website and pattern matching to map user input to ques-
tions and retrieve the correct answers. Users found it a novel and interesting way to
access the FAQ using natural language questions. Compared to a Google search, users
were able to find a relevant answer more often and the majority of them preferred using
the chatbot. Driving factors were the chatbots ability to provide direct answers to the
1 https://www.codefluegel.com, visited on 2020-01-02
2 https://telegram.org/blog/200-million, visited on 2019-02-13
3 https://engineering.leeds.ac.uk/computing, visited on 2019-03-14
iJIM Vol. 14, No. 18, 2020
19
PaperChatbots for Brand Representation in Comparison with Traditional Websites
question if there was only one matching answer and fewer links in general, resulting in
less search time and better overview. [7]
Almost 50% of Internet users in the U.S. use social media to contact customer ser-
vice. According to recent studies, more than two thirds of users expect an answer within
an hour, but it usually takes over six times as long to get a response. Scaling can be a
problem with high numbers of requests, which chatbots could solve. Xu et al. built a
chatbot for customer service on Twitter, as a feasibility study on such software. They
used deep learning algorithms and trained their bot with nearly a million Twitter con-
versations from more than 60 different companies and brands. One of their key findings
was that more than 40% of customer requests are only emotional and not looking for
any kind of information or help, possibly due to the nature of social media. Their cus-
tomer service bot showed results close to a human when handling such emotional re-
quests, while informational requests seem to require more complex training. [8]
Fig. 1. Chart showing potential annual savings of US salaries in billion US dollars by using
chatbots. Source: Business Insider4
Another example for bots in customer interaction is WAH Nails5. Sharmadean Reid,
the founder of the company, said that 30% of reservations were already handled by their
chatbot only two months after its launch and plans to enhance it further using AI were
already underway [9]. While this is only a single example, chatbots have the potential
for significant savings in such applications, where human agents could be replaced, or
new services created. According to an infographic (see Figure 1) by business magazine
4https://www.businessinsider.de/80-of-businesses-want-chatbots-by-2020-2016-12,
visited on 2019-03-20
5 https://wah-london.com/, visited on 2019-03-14
20
http://www.i-jim.org
PaperChatbots for Brand Representation in Comparison with Traditional Websites
Business Insider6, potential annual savings in customer service in the United States of
America alone amount to $23 billion US dollars7.
Beriault-Poirier, Tep, and Sénécal conducted an exploratory user study similar to the
one shown in this paper. The goal of the study was to learn more about the user experi-
ence provided by chatbots from different brands on Facebook Messenger, namely food
and beverage company Whole Foods8, clothing line Tommy Hilfiger9 and travel search
site Skyscanner10. Users were asked to complete two tasks for each brand (six in total)
one on their website and one with their Facebook Messenger chatbot. Chatbots and
websites were then evaluated from 1 to 7 in terms of usefulness, ease of use, ease of
learning and satisfaction. User experience scores and abandonment rates were favour-
ing the companies' websites. Analysis of the participants' facial expressions, however,
showed that chatbots generated more positive emotions. Participants liked the chatbots'
ability to deliver quick and precise answers and also showed a willingness to use chat-
bots in the future. [10]
Chatbots for customer interaction could also impose new problems. Heckmann and
Kraus mention that depending on the specific use case, personal data must be handled
carefully, and usage of such data has to be disclosed. Commercial usage could also
come with the requirement to inform users about those commercial intentions for
example, disclosing advertisements. Especially with the recent introduction of the Gen-
eral Data Protection Regulation (GDPR)11, companies must take special care of pro-
tecting personal data in and outside of the European Union (EU). [11]
While bots can help in building a brand and customer relation, a bot's personality is
also able to do harm. As seen with Microsoft's Twitter-bot Tay (see [12]), machine
learning and AI could lead to unforeseen behaviour, which in turn could potentially
damage a brand or company [13]. Brands and developers also have to keep in mind that
a chatbot's presentation and persona can result in unwanted perception and emotions,
especially in regard to the uncanny valley effect12 [14] [15].
6 https://www.businessinsider.de, visited on 2019-03-20
7https://www.businessinsider.de/80-of-businesses-want-chatbots-by-2020-2016-12,
visited on 2019-03-20
8 https://www.wholefoodsmarket.com, visited on 2019-03-19
9 http://www.tommy.com/, visited on 2019-03-19
10 https://www.skyscanner.com/, visited on 2019-03-19
11 https://ec.europa.eu/info/law/law-topic/data-protection_en, visited on 2019-03-17
12 Described as the feeling of eeriness and discomfort towards a given medium or
technology that frequently appears in various kinds of humanmachine interactions by
[17].
iJIM Vol. 14, No. 18, 2020
21
PaperChatbots for Brand Representation in Comparison with Traditional Websites
3 Theodore, A Chatbot
As part of this case study, a chatbot called Theodore was created. The chatbot's main
use is to represent a company, namely the Austrian-based software developer
CodeFlügel GmbH. CodeFlügel is located in Graz, Austria and specializes in develop-
ing augmented reality mobile applications. Virtual reality, traditional apps, web apps
and custom projects are also part of their offered services. CodeFlügel has about twenty
employees consisting of software developers, sales staff and other people from the op-
erative business such as a human resources manager, marketing personnel and the chief
executive officer.
Like most companies, CodeFlügel uses a website13 to represent themselves on the
Internet and inform potential interested parties about their services and other kinds of
information. Content and features of the website include, but are not limited to, infor-
mation about what CodeFlügel offers, a blog and a list of open positions.
CodeFlügel also engages in social media platforms, one of which is Facebook14. Fa-
cebook offers the possibility to directly contact a company via their own Facebook
Messenger15. Messenger is a chat and messaging platform providing APIs to build
third-party tools and more importantly chatbots.
Theodore was designed to reproduce most of the website's features and to work with
Facebook Messenger as well as a custom webchat, which could be embedded on a web-
site. Existing APIs and content should be (re-)used as much as possible. For example,
if a user asks the chatbot “What does your company do?”, it should answer with a proper
explanation about the company’s products and services.
3.1 Target audience
In order to design a company chatbot, it is imperative to recognize and understand
the target audience of CodeFlügel. The website users can be divided into three groups:
Clients: A client or potential client is someone interested in purchasing a product or
service from the company typically a mobile application or a website. Some of them
might have a very specific request or idea, while others are interested in consultation
and guidance. They most likely want to know what the company is offering, their level
of expertise in specific technologies and how to contact them. The technological exper-
tise of the customers varies greatly and ranges from beginners to experts. For clients,
the chatbot would not only offer information, but also function as a demonstration of
the company's previous work on chatbots.
13 https://www.codefluegel.com, visited on 2018-10-28
14 https://www.facebook.com, visited on 2019-01-20
15 https://developers.facebook.com/docs/messenger-platform, visited on 2019-01-20
22
http://www.i-jim.org
PaperChatbots for Brand Representation in Comparison with Traditional Websites
Potential employees: Potential employees are usually students with little job expe-
rience, but a technological background or less likely people interested in marketing,
sales or human resources. They could use the website or chatbot for general information
about the company or to inform themselves about job openings or projects the company
has worked on.
Blog readers and everyone else: This group of users is probably not looking for
anything specific about the company, but rather for one of the varying blog posts or
general information about the technology used at the company. They most likely found
the website / chatbot via a search engine such as Google16 or a social media channel
such as Facebook.
The chatbot provides little benefit for such users, besides interacting with the user
and possibly generating interest and publicity.
3.2 Dialog design
With such a target audience in mind, the chatbot's requirements and a feature set
were defined in consultation with CodeFlügel. The chatbot should feature almost all of
the website's content. Since the interaction is different from browsing a website, some
additional design decisions have to be made. For example, a help function and a fallback
dialog help to enhance the user experience. User input is limited to text input and quick
reply buttons, while the output can incorporate media and specific layouts as well.
Greeting: Theodore must be able to greet users in order to help them understand that
they are chatting with the company chatbot of CodeFlügel. Thus, the chatbot first in-
troduces himself and then shows some examples about what it can help the user with.
About the company: The chatbot should be able to provide a short explanation
about what CodeFlügel does and which services they offer, given an input like “What
does CodeFlügel do?” (German: “Was macht CodeFlügel eigentlich?”).
About products and service: Products and services should be explained by Theo-
dore. Furthermore, the chatbot should provide information about completed projects.
Contact information: The most important contact information should be displayed
by the chatbot. This includes the office address, phone number and email address. A
link to Google Maps17 is provided to help users in locating the office.
Open positions: Theodore should inform users about open positions and provide a
way to contact CodeFlügel. In case of no vacancies, an information about speculative
applications and where to apply is shown.
Newsletter subscription: CodeFlügel uses Mailchimp18 as a service to handle their
newsletter. The chatbot should provide a way to subscribe to the newsletter by access-
ing the Mailchimp API19.
16 https://www.google.com, visited on 2019-01-20
17 https://maps.google.com, visited on 2019-01-20
18 https://mailchimp.com, visited on 2019-01-20
19 https://developer.mailchimp.com, visited on 2019-07-03
iJIM Vol. 14, No. 18, 2020
23
PaperChatbots for Brand Representation in Comparison with Traditional Websites
Blog: Blog posts are published regularly by CodeFlügel. The website uses Word-
press20 as its content management system. Theodore should output the most recent blog
posts by accessing the website's REST API21.
Social media links: CodeFlügel's social media handles should be shown and linked
to their respective platforms, if a user asks for this information. This includes Facebook,
Instagram22 and Twitter23.
Help: Another important aspect in designing a chatbot is to provide help, if a user
gets stuck or does not know what to do. Consequently, if a user asks for help, a short
description about the chatbot's abilities should be shown.
Fallback: The chatbot is made for a specific domain which defines the capabilities.
If a user input is outside of this domain or the Natural Language Understanding (NLU)
engine is unable to detect the user's intent, some kind of fallback should be triggered.
For example, Theodore is not designed to announce the weather. Thus, if asked whether
it is going to rain today, the chatbot should tell the user that he could not understand
what the user wants or that it does not know an answer to the specific request. If this
happens several times, the chatbot should present a way to get in contact with a real
person.
Theodore should feature informal language and a friendly attitude. The chatbot
should make clear that it is a program and not pretend to be a real person, which should
help to avoid the uncanny valley effect mentioned in Related Work.
3.3 Architecture and implementation
Since CodeFlügel maintains a Facebook page as well as a website, the chatbot should
be usable on both platforms. Facebook Messenger is used for the first, while a custom
webchat is used for the second. Messenger already provides an existing messaging plat-
form and chat experience. Only the backend needs to be developed for this platform. A
webchat, however, also needs its own User Interface (UI) and ways to communicate
with the backend.
The same backend is used for both platforms. Facebook provides an API to com-
municate with Messenger as well as webhooks to handle incoming messages and ac-
tions. The webchat uses web-sockets to offer a fast and responsive chat experience.
In order to streamline the development process and to increase the maintainability
of the chatbot, Facebook Messenger's message format (in JavaScript Object Notation,
short JSON) is used for both platforms and its UI components are recreated for the
webchat. These components include simple text, Messenger's Generic, List, Button and
Media Templates.
20 https://wordpress.org, visited on 2019-01-20
21 https://developer.wordpress.org/rest-api, visited on 2018-04-16
22 https://www.instagram.com, visited on 2019-01-20
23 https://twitter.com, visited on 2019-01-20
24
http://www.i-jim.org
PaperChatbots for Brand Representation in Comparison with Traditional Websites
Typing indicators to simulate the behaviour of typing messages and quick reply but-
tons (see Figure 2) were implemented as well. This also makes the different platforms
more consistent and increases usability as users are presented with a familiar chat ex-
perience.
Instead of using a bot development framework to start the development, a custom
implementation was chosen. While such frameworks provide functions and templates
for quickly achieving first results, they tend to only support subsets of the features of-
fered by some bot platforms.
Fig. 2. Implementations of Quick Reply buttons (labelled “Ja, abonnieren”, “Nicht abonni-
eren”) on the webchat (left) and Facebook Messenger (right).
Google's Dialogflow24 (previously known as API.AI) was chosen as the NLU ser-
vice because it shows promising development with features such as versioning and en-
vironments. It also offers a concept which is easy to use and understand and, therefore,
enables fast prototyping. Another important aspect working in Dialogflow's favor is
their free Standard Edition, which comes with a sufficient quota of text requests.
24 https://dialogflow.com/, visited on 2019-03-31
iJIM Vol. 14, No. 18, 2020
25
PaperChatbots for Brand Representation in Comparison with Traditional Websites
Dialogflow's Messenger and webchat integrations, however, provide only little room
for customization and lack support for advanced features such as message receipts (Fa-
cebook Messenger) and UI templates (webchat).
Therefore, the chatbot uses a dedicated backend to handle incoming messages. On
the user side, a client application (webchat or Facebook Messenger) is used. In the case
of the webchat, communication happens directly between the server and the client.
When using Facebook Messenger, client requests are first sent to Facebook's server
before reaching Theodore's backend and vice versa.
Fig. 3. Communication structure between a user and the chatbot on a webchat.
Figure 3 shows the chronological structure of the communication between a user and
the chatbot on the webchat. First, the client sends the user's input to the backend. The
backend then sends the request to a NLU service, which extracts an intent and possible
parameter to get to know what the user wants. If necessary, additional APIs are visited.
For example, the latest blog posts are retrieved from the website using the Wordpress
REST API. Subsequently, a proper response is created. After all this processing, the
response is sent back to the client and presented to the user.
The chatbot has to access different APIs in order to provide all required features.
Wordpress's REST API is used to retrieve blog posts as well as open positions from the
website. Mailchimp's REST API is used to subscribe users to the newsletter, if they
want to.
Other content and responses such as product and service descriptions are stored in
files or in Dialogflow's intent handling.
26
http://www.i-jim.org
PaperChatbots for Brand Representation in Comparison with Traditional Websites
Theodore is written from scratch using Node.js25 (version 8), Socket.IO26 and Di-
alogflow. The backend handles connections from and to both Facebook Messenger and
a custom webchat, written in HTML and TypeScript with the popular frontend fram-
work Angular27. Communication is done with normal HTTP requests and websockets.
The chatbot was built with modularity in mind, in order to easily adapt to different
needs and provide an extendable base for other projects. Adapters for other bot plat-
forms, NLU services and analytics like Chatbase28 as well as API connectors and addi-
tional features can be added via extensions. Currently, Theodore features scripts for
Facebook Messenger, a webchat via Socket.IO and a complete adapter for Dialogflow.
4 Case Study
In order to see how the chatbot performs and whether users accept it as an alternative
to the website of CodeFlügel29, a user study was conducted. Tests took place from No-
vember 6th to December 22nd, 2018 with individual participants. Users were chosen
based on a specific user classification. The goal was to only introduce participants
which are relevant to the company’s website (and chatbot respectively) for example,
potential employees. Those users were asked to perform several common tasks on the
company website and with a corresponding chatbot. Timings of those tasks and user
feedback were recorded to get quantitative (speed) and qualitative (acceptance, dis- &
advantages) results and see what tasks are better suited for the chatbot compared to the
website.
4.1 Setup
User selection: Participants were chosen to reflect the user base of the company's
website. The relevant user groups have been explained in 3.1 Target Audience.
A total of twenty (20) users were selected based on the previously outlined classifi-
cation. This approach makes it possible to better adapt the selection of test subjects to
the actual users of the website and to create more appropriate tasks. At least five par-
ticipants per user type were among them. The age ranged from 20 to 33, while the
average user was 27.7 years old. 60% of the users had a bachelor or master's degree
while another 25% were still working on their bachelor (with 20% working on their
master's degree). 70% had a technical background (mostly IT), either through work or
education.
25 https://nodejs.org/, visited on 2019-03-31
26 https://socket.io/, visited on 2019-03-31
27 https://angular.io/, visited on 2019-04-04
28 https://chatbase.com/, visited on 2019-03-31
29 https://www.codefluegel.com, visited on 2018-10-28
iJIM Vol. 14, No. 18, 2020
27
PaperChatbots for Brand Representation in Comparison with Traditional Websites
Preparation: At the beginning of the test session, users were greeted and given a
summary about the project as well as about the user study and the upcoming steps. Then
they were asked to fill out a consent form and a questionnaire about their background,
knowledge in different technologies such as chat applications, augmented reality and
programming as well as their expectations of a company's website and chatbot. The
goal was to collect enough background information about the users to be able to find
possible connections between the use of the chatbot or the website and the different
backgrounds.
Setup and tasks: The test setup consisted of a workspace with a computer, two
monitors, a keyboard and a mouse. The participants were placed in front of the com-
puter while the facilitator was sitting to their left, a little further back, to observe the
test person's actions and measure the time of each task. The right monitor showed in-
structions for the tasks, while the left one was used to perform the tasks. The actions on
the left monitor were captured with the screen capturing software OBS Studio30.
Table 2. This table shows the tasks users have to complete with both the website
and the chatbot.
Task #
Description and Goal
1
Task: Find out what CodeFlügel does or which services they provide.
Goal: “Developer of Mobile Apps, Web and Augmented/Virtual Reality.”
2
Task: Find out where CodeFlügel’s office is and what phone number you can call.
Goal: User finds address and phone number.
3
Task: Find out which companies CodeFlügel has already implemented projects for.
Goal: User finds reference page/list.
4
Task: Find and open the latest blog entry.
Goal: User opens the latest blog entry.
5
Task: Sign up for the newsletter with the e-mail address <firstname>.<surname>@code
fluegel.com.
Goal: User signs up for the newsletter.
6
Task: Find out if and which jobs are currently available.
Goal: User finds job page/list.
7
Task: Find a way to try Augmented Reality (AR) for yourself.
Goal: User finds demo app.
8
Task: Find at least one Augmented Reality (AR) project created by CodeFlügel.
Goal: User names at least one (1) Augmented Reality (AR) project.
9
Task: Find out what Augmented Reality (AR) actually is.
Goal: User opens AR explanation.
A list of nine typical tasks was defined (see Table 2), taking into account the various
user groups and the target audience of the company's website. The list included things
such as looking for the latest blog post, searching for job openings and finding at least
one completed augmented reality project of the company. First, the tasks had to be per-
formed on the company's website. In order to increase the quality of feedback and ob-
servations, users were asked to verbalize their thoughts, similar to so called thinking-
aloud-tests.
30 https://obsproject.com, visited on 2019-01-04
28
http://www.i-jim.org
PaperChatbots for Brand Representation in Comparison with Traditional Websites
Each task instruction had to be read out loud by the participants. After that, a timer
was started by the facilitator. The user had to clearly state the solution to the given task
and, if it was the right one, the timer was stopped. When all nine tasks were completed,
the same tasks had to be performed using the chatbot instead of the website.
Feedback and interviews: After the completion of the tasks, another questionnaire
had to be filled out. The questionnaire was there to gather the participant's feedback,
recognize potential flaws in the chatbot design and measure the acceptability of the
company chatbot. An interview was then conducted to clear up any potential ambigui-
ties and obtain better-quality feedback.
4.2 Results
When asked what users expect from a chatbot and website of a company, the results
were pretty close. Information about Products & Services and Contact Details were
expected by all users from the website, while 70% (Contact Details) to 85% (Products
& Services) expected the same from the chatbot. References, Jobs and Blog & Articles
only amounted to 20% 55% of the users' expectations.
In terms of speed of the task completion, 80% of participants were faster using the
chatbot. The average difference in time needed for all tasks was 2 minutes and 54 sec-
onds in favor of the chatbot. The average completion time of all tasks was 3m 38s with
the chatbot and 6m 33s with the website, while the averages for one task were 24 sec-
onds (chatbot) and 43 seconds (website).
Fig. 4. Average completion time per task for both platforms
Six out of the nine tasks were completed faster with the chatbot, although two of
those six were only completed about six seconds faster. Tasks involving the website
iJIM Vol. 14, No. 18, 2020
29
PaperChatbots for Brand Representation in Comparison with Traditional Websites
were faster or equally fast when a corresponding and self-explanatory (e.g. Blog) menu
entry existed, while tasks seeking specific information not immediately accessible by
the main menu (tasks 1, 7, 8 and 9) where significantly faster achieved with the chatbot.
This is similar to what [10] concluded in their research.
In terms of perceived completion time, 80% of test users thought they were faster
using the chatbot. 75% correctly assessed the chatbot as being faster, while one of the
participants thought completion time was faster, but the opposite was true. Another one
thought their website tasks were completed faster, but in reality, the chatbot run was
the quicker one. 15% of the test users believed to have been equally quick with both
platform tasks while actually being faster with the website.
Although the chatbot matched every participant's expectations, seven out of the 20
users said the input-to-intent matching could be improved, as the chatbot did not un-
derstand every message and reformulating the request was necessary. 80% of the users
found the chatbot more entertaining than the website vice versa, only 10% were more
entertained using the website. 40% deemed the chatbot more modern, while none said
anything similar about the website. The chatbot was also described as fast by 50%,
while only 10% used this adjective for the website.
60% of the participants said they preferred a chatbot with informal communication.
The other 40% did not care about whether the chatbot approached the conversation
formally or informally. This pretty much confirms what [16] published in their study.
[17] showed that humans tend to use shorter messages when conversing with chat-
bots. Our study revealed that 50% of users relied on a combination of whole sentences
and simple keywords to communicate with the chatbot. 30% used only sentences, while
20% based their messages solely on keywords.
Half the users thought the website and chatbot were equally appealing to them. The
website was favored by 25% of the participants, partially due to being used to websites
or due to being able to explore the content more freely. [10] suspected similar reasons
for favoring websites over chatbots in their study. The other 25% found the chatbot
more appealing. Reasons to favor the chatbot were instantaneous answers to specific
questions without the hassle of clicking through menus and searching through pages,
as well as its interactive nature, which resulted in more fun for some of the participants.
Overall, the company chatbot was received positively and 85% of participants could
see themselves using more chatbots in the future.
Despite extensive background checks of the participants, no correlation between the
time needed to complete the tasks and the level of IT knowledge was observed.
5 Conclusion
In order to compare the usage of chatbots to traditional websites, a specific company
chatbot was created and a case study was conducted. Typical use cases of a company
chatbot were worked out in order to compare such a chatbot with the existing company
website and measure its performance.
The user study showed that chatbots are accepted by users. The users provided pos-
itive feedback about the chatbot, only some had minor issues with the matching of their
30
http://www.i-jim.org
PaperChatbots for Brand Representation in Comparison with Traditional Websites
queries and the size of the chat window. However, this could be improved easily by
slightly changing the web chat design and providing more training data to the chatbot
and the respective NLU engine. The study also revealed that it is important to train for
whole sentences as well as key words, as both are commonly used by users. This was
also discovered by [17]. The chatbot was faster than the website, especially when it
comes to finding specific information about the company, and the users perceived it as
more entertaining. Thus, it could potentially help companies in reaching new or
younger audiences and in improving specific parts of their websites, such as the FAQs.
6 Future Perspectives
Building on this study, further testing of the impact of UI improvements can be done.
Natural Language Processing and matching of user inputs to intents also bear potential
to increase the acceptance and enhance the user experience. In general, more chatbots
could be tested and more users from different backgrounds and age groups could pro-
vide more valuable insight.
Another aspect to investigate further is the effect of different chatbot personas on the
user experience and the influence of brand perception. One could also evaluate the pos-
sibilities of distributing and marketing bots. Do chatbot "marketplaces" provide a sim-
ilar exposure to mobile app stores such as Google Play and Apple App Store?
Conversion rates and other Key Performance Indicators (KPIs) are things that could
be explored in more detail as well. These analytics could show whether chatbots are
economically feasible and here to stay.
7 References
[1] USA Today (Mar. 30, 2016). Microsoft CEO Nadella: “Bots are the new apps”. Ed. By
Theresa Chong. USA Today. [Online], Available: https://eu.usatoday.com/story/tech/news/
2016/03/30/microsof-ceo-nadella-bots-new-apps/82431672/ [Accessed Jan. 28, 2019]
[2] Comscore (Sept. 22, 2015). The 2015 U.S. Mobile App Report. Tech. rep. Comscore.
[Online], Available: https://www.comscore.com/Insights/Presentations-and-Whitepapers/
2015/The-2015-US-Mobile-App-Report [Accessed Jan. 24, 2019]
[3] Klopfenstein, Lorenz Cuno et al. (June 2017). The Rise of Bots: A Survey of Conversa-
tional Interfaces, Patterns, and Paradigms. In: Proceedings of the 2017 Conference on De-
signing Interactive Systems. DIS ’17, pp. 555–565. ISBN: 978-1-4503-4922-2.doi:
https://doi.org/10.1145/3064663.3064672
[4] Hootsuite and We Are Social (2019). Digital 2019. Global Digital Overview. Tech. rep.
Mindbowser. [Online], Available: https://datareportal.com/reports/digital-2019-global-digi-
tal-overview [Accessed Feb. 2, 2019]
[5] Nadella, Satya (Mar. 25, 2016a). Build 2016. Keynote Presentation. Microsoft.[Online],
Available: https://channel9.msdn.com/Events/Build/2016/KEY01#time=1h41m11s [Ac-
cessed Feb. 13, 2019]
[6] Chatbots Magazine and Anatoly Khorozov (Mar. 1, 2017). Trends Driving the Chatbot
Growth. [Online], Available: https://chatbotsmagazine.com/trends-driving-the-chatbot-
growth-77b78145bac [Accessed Mar 14, 2019]
iJIM Vol. 14, No. 18, 2020
31
PaperChatbots for Brand Representation in Comparison with Traditional Websites
[7] Shawar, Bayan Abu and Eric Atwell (Sept. 2015). “A chatbot as a Question Answering
Tool.” In: 2015 International Conference on Advances in Software, Control and Mechanical
Engineering. 2015 International Conference on Advances in Software, Control and Mechan-
ical Engineering, pp. 16. ISBN: 978-93-84422-37-0. https://doi.org/10.17758/ur.u09
15120
[8] Xu, Anbang et al. (May 2017). “A New Chatbot for Customer Service on Social Media.” In:
Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 2017
CHI Conference on Human Factors in Computing Systems, pp. 35063510. ISBN: 978-1-
4503-4655-9. https://doi.org/10.1145/3025453.3025496
[9] Puscher, Frank (2018). “Holpriger Start. Chatbots für Unternehmen.” In: iX 6/2018, pp. 58–
62
[10] Beriault-Poirier, Amélie, Sandrine Prom Tep, and Sylvain Sénécal (2019). “Putting Chatbots
to the Test: Does the User Experience Score Higher with Chatbots Than Websites?” In:
Human Systems Engineering and Design. Springer International Publishing, pp. 204212.
ISBN: 978-3-030-02053-8. https://doi.org/10.1007/978-3-030-02053-8_32
[11] Heckmann, Jörn and Michael Kraus (2018). “You, Robot. Rechtliche Risiken beim Einsatz
von Chatbots.” In: iX 6/2018, pp. 70–72
[12] The Guardian (Mar. 24, 2016). “Tay, Microsoft’s AI chatbot, gets a crash course in racism
from Twitter. Ed. by Elle Hunt. The Guardian. [Online], Available: https://www.thegua
rdian.com/technology/2016/mar/24/tay-microsofts-ai-chatbot-gets-a-crash-course-in-rac-
ism-from-twitter. [Accessed Feb 22, 2019]. https://doi.org/10.1145/3008665.3008674
[13] Lebeuf, Carlene, Margaret-Anne Storey, and Alexey Zagalsky (Dec. 25, 2017). “Software
Bots.” In: IEEE Software vol. 35 (1), pp. 1823. https://doi.org/10.1109/ms.2017.4541027
[14] Araujo, Theo (Aug. 2018). “Living up to the chatbot hype: The influence of anthropo-
morphic design cues and communicative agency framing on conversational agent and com-
pany perceptions.” In: Computers in Human Behavior. Vol. 85, pp. 183189 https://doi.
org/10.1016/j.chb.2018.03.051
[15] Ciechanowski, Leon et al. (Mar. 2019). “In the shades of the uncanny valley: An experi-
mental study of human–chatbot interaction.” In: Future Generation Computer Systems. Vol.
92, pp. 539548. https://doi.org/10.1016/j.future.2018.01.055
[16] Survata and LivePerson (Apr. 2017). How Consumers View Bots In Customer Care. Tech.
rep. LivePerson. [Online], Available: https://www.liveperson.com/resources/reports/bots-
in-customer-care/ [Accessed Feb 20, 2019]
[17] Hill, Jennifer, W. Randolph Ford, and Ingrid G. Farreras (2015). “Real conversations with
artificial intelligence: A comparison between humanhuman online conversations and hu-
man–chatbot conversations.” In: Computers in Human Behavior. Vol. 49, pp. 245250.
https://doi.org/10.1016/j.chb.2015.02.026
8 Authors
Johannes Kühnel is currently working as a backend developer at CodeFluegel
GmbH in Graz, Austria. He is mostly responsible for developing REST APIs and
DevOps related work (e-mail: johannes@kuehnel.co.at).
Markus Ebner is currently working as a Researcher in the Department Educational
Technology at Graz University of Technology (e-mail: markus.ebner@tugraz.at). He
deals with e-learning, mobile learning, technology enhanced learning and Open Educa-
tional Resources. His focus is on Learning Analytics at K-12 level. In addition, several
32
http://www.i-jim.org
PaperChatbots for Brand Representation in Comparison with Traditional Websites
publications in the area of Learning Analytics were published and workshops on the
topic were held.
Martin Ebner is with the Department Educational Technology at Graz University
of Technology, Graz, Austria. (e-mail: martin.ebner@tugraz.at). As head of the Depart-
ment, he is responsible for all university wide e-learning activities. He holds an Assoc.
Prof. on media informatics and works at the Institute of Interactive Systems and Data
Science as senior researcher. For publications as well as further research activities,
please visit: http://martinebner.at.
Article submitted 2020-01-26. Resubmitted 2020-07-24. Final acceptance 2020-07-29. Final version pub-
lished as submitted by the authors.
iJIM Vol. 14, No. 18, 2020
33
Article
Purpose This study aims to examine the antecedents of privacy concerns in the era of artificial intelligence. Specifically, it focuses on the impact of various factors related to interactions with a chatbot (creepiness and perceived risk) and individual traits (familiarity with chatbots and need for privacy) in relation to privacy when interacting with a chatbot in the context of financial services. The moderating effect of gender on these relationships was also examined. Design/methodology/approach A total of 430 Canadians responded to an online questionnaire after interacting with a chatbot in the context of a simulated auto insurance quote. A structural equation model was used to test the hypotheses. Findings The results showed that privacy concerns are influenced primarily by creepiness, followed by perceived risk and the need for privacy. The last two relationships are moderated by gender. Conversely, familiarity with chatbots does not affect privacy concerns in this context. Originality/value This study is the first to consider the influence of creepiness as an antecedent of privacy concerns arising from interactions with AI tools and highlight its key impacts. It also shows how gender moderates specific relationships in this context.
Chapter
Full-text available
Chatbots are robots that simulate conversation with human users through instant messaging services. Though the technology is innovative and trendy, what kind of user experience do they provide? To answer this, we conducted an exploratory study with chatbots from different sectors of activity on Facebook Messenger. We invited ten participants to complete six precise search tasks using these chatbots and the corresponding websites from each brand. Based on observation and interviews according to a qualitative approach, our study shows that the user experience scores higher with websites than chatbots, while abandonment rate is higher with chatbots, even though they generate more positive emotions. As a conclusion, until artificial intelligence improves the technology, chatbots have some catching up to do as they are do not score higher than websites to fulfill users’ expectations.
Article
Full-text available
This project has been carried out in the context of recent major developments in botics and more widespread usage of virtual agents in personal and professional sphere. The general purpose of the experiment was to thoroughly examine the character of the human–non-human interaction process. Thus, in the paper, we present a study of human–chatbot interaction, focusing on the affective responses of users to different types of interfaces with which they interact. The experiment consisted of two parts: measurement of psychophysiological reactions of chatbot users and a detailed questionnaire that focused on assessing interactions and willingness to collaborate with a bot. In the first quantitative stage, participants interacted with a chatbot, either with a simple text chatbot (control group) or an avatar reading its responses in addition to only presenting them on the screen (experimental group. We gathered the following psychophysiological data from participants: electromyography (EMG), respirometer (RSP), electrocardiography (ECG), and electrodermal activity (EDA). In the last, declarative stage, participants filled out a series of questionnaires related to the experience of interacting with (chat)bots and to the overall human–(chat)bot collaboration assessment. The theory of planned behavior survey investigated attitude towards cooperation with chatbots in the future. The social presence survey checked how much the chatbot was considered to be a “real” person. The anthropomorphism scale measured the extent to which the chatbot seems humanlike. Our particular focus was on the so-called uncanny valley effect, consisting of the feeling of eeriness and discomfort towards a given medium or technology that frequently appears in various kinds of human–machine interactions. Our results show that participants were experiencing lesser uncanny effects and less negative affect in cooperation with a simpler text chatbot than with the more complex, animated avatar chatbot. The simple chatbot have also induced less intense psychophysiological reactions. Despite major developments in botics, the user’s affective responses towards bots have frequently been neglected. In our view, understanding the user’s side may be crucial for designing better chatbots in the future and, thus, can contribute to advancing the field of human–computer interaction. *L. Ciechanowski, A. Przegalinska, and M. Magnuski equally contributed to the article.
Conference Paper
Full-text available
This work documents the recent rise in popularity of messaging bots: chatterbot-like agents with simple, textual interfaces that allow users to access information, make use of services, or provide entertainment through online messaging platforms. Conversational interfaces have been often studied in their many facets, including natural language processing, artificial intelligence, human-computer interaction, and usability. In this work we analyze the recent trends in chatterbots and provide a survey of major messaging platforms, reviewing their support for bots and their distinguishing features. We then argue for what we call "Botplication", a bot interface paradigm that makes use of context, history, and structured conversation elements for input and output in order to provide a conversational user experience while overcoming the limitations of text-only interfaces.
Conference Paper
Full-text available
Users are rapidly turning to social media to request and receive customer service; however, a majority of these requests were not addressed timely or even not addressed at all. To overcome the problem, we create a new conversational system to automatically generate responses for users requests on social media. Our system is integrated with state-of-the-art deep learning techniques and is trained by nearly 1M Twitter conversations between users and agents from over 60 brands. The evaluation reveals that over 40% of the requests are emotional, and the system is about as good as human agents in showing empathy to help users cope with emotional situations. Results also show our system outperforms information retrieval system based on both human judgments and an automatic evaluation metric.
Article
Disembodied conversational agents in the form of chatbots are increasingly becoming a reality on social media and messaging applications, and are a particularly pressing topic for service encounters with companies. Adopting an experimental design with actual chatbots powered with current technology, this study explores the extent to which human-like cues such as language style and name, and the framing used to introduce the chatbot to the consumer can influence perceptions about social presence as well as mindful and mindless anthropomorphism. Moreover, this study investigates the relevance of anthropomorphism and social presence to important company-related outcomes, such as attitudes, satisfaction and the emotional connection that consumers feel with the company after interacting with the chatbot.
Build 2016. Keynote Presentation. Microsoft
  • Satya Nadella
Nadella, Satya (Mar. 25, 2016a). Build 2016. Keynote Presentation. Microsoft.[Online], Available: https://channel9.msdn.com/Events/Build/2016/KEY01#time=1h41m11s [Accessed Feb. 13, 2019]
Trends Driving the Chatbot Growth
  • Chatbots Magazine
  • Anatoly Khorozov
Chatbots Magazine and Anatoly Khorozov (Mar. 1, 2017). Trends Driving the Chatbot Growth. [Online], Available: https://chatbotsmagazine.com/trends-driving-the-chatbotgrowth-77b78145bac [Accessed Mar 14, 2019]
A chatbot as a Question Answering Tool
  • Bayan Shawar
  • Eric Abu
  • Atwell
Shawar, Bayan Abu and Eric Atwell (Sept. 2015). "A chatbot as a Question Answering Tool." In: 2015 International Conference on Advances in Software, Control and Mechanical Engineering. 2015 International Conference on Advances in Software, Control and Mechanical Engineering, pp. 1-6. ISBN: 978-93-84422-37-0. https://doi.org/10.17758/ur.u09 15120
Holpriger Start. Chatbots für Unternehmen
  • Frank Puscher
Puscher, Frank (2018). "Holpriger Start. Chatbots für Unternehmen." In: iX 6/2018, pp. 58-62