Conference PaperPDF Available

The First Impression Counts! The Importance of Onboarding for Educational Chatbots

Authors:

Abstract

Educational chatbots promise to be valuable tools for educational learning scenarios. However their use is not common yet. Therefore, an onboarding process is required. We compare two onboarding processes for our self-developed educational chatbot system which supports students in knowledge acquisition and review using lecture slides as content input: In one onboarding scenario users can create an account via a form without interacting with the chatbot and in the other solely by chatting with the chatbot. After creating an account, users test our system. We fond that users who are onboarded through a conversation find the chatbot more intuitive to use, conversations more natural, the navigation easier and conversation quality better than the control group. This can be attributed to the chatbot being perceived as a conversation partner right from the beginning – otherwise it was observed that the system is rather used like a search engine. We therefore conclude that how the onboarding process is designed has a significant impact on the general perception of the educational chatbot system and one should focus on presenting the chatbot as a conversation partner right from the beginning.
P. A. Henning, M. Striewe, M. Wölfel (Hrsg.): 20. Fachtagung Bildungstechnologien (DELFI),
Lecture Notes in Informatics (LNI), Gesellschaft für Informatik, Bonn 2022 239
doi: 10.18420/delfi2022-050
The First Impression Counts!
The Importance of Onboarding for Educational Chatbots
Trong Nghia Hoang
1
, Andreas Reich
2
, Matthias Wölfel
3
1 Introduction, problem, and motivation
Chatbots are computer programs that simulate natural, human-like conversation with
humans via text interactions. Despite their great potential for educational scenarios, their
presence in education is relatively small [WI18]. A limiting factor is that developing a
chatbot requires a lot of expertise and effort. To address this, Wölfel presented the
PEdagogical conversational Tutor (PET) chatbot system, which automatically trains a
chatbot from PowerPoint slides [WO21]. The system can not only answer lecture-specific
questions but also offer automatically generated tests and rate the replies.
According to [MC19] onboarding is very important for chatbots as most users are not
aware of the features. However, onboarding for chatbots is not common. We noticed that
many users perceive and use the PET more like a search engine than a conversational
partner. According to [SO17], chatbot users can learn how to text with chatbots and adapt
their language and behavior. Users mindlessly transfer human social rules and
expectations to chatbots [NA00], but only if they assume they are talking to a system
capable of conversation. To investigate how the onboarding process can influence the
perception of the chatbot, we designed two onboarding processes for the PET system.
2 Study design and results
To investigate how onboarding new users into the PET system affects chatbot perception,
we recruited 18 students and randomly divided them into two groups to perform an AB
test. The "chatbot" group could only use the PET chatbot window to register and log in,
while the "form" group was prompted to fill out a form. Both groups were able to text with
the chatbot during the onboarding process, but the group chatbot had to text with the
chatbot to continue, while texting was optional for the group form. After completing one
1
Karlsruhe University of Applied Sciences, Faculty of Computer Science and Business Information Systems,
Moltkestraße 30, 76133 Karlsruhe, - hotr1011@h-ka.de
2
University of Hohenheim, Schloß Hohenheim 1, 70599 Stuttgart, andreas.reich@uni-hohenheim.de,
https://orcid.org/0000-0002-2426-6490
3
Karlsruhe University of Applied Sciences, Faculty of Computer Science and Business Information Systems,
Moltkestraße 30, 76133 Karlsruhe matthias.woelfel@h-ka.de; University of Hohenheim,
Schloß Hohenheim 1, 70599 Stuttgart, https://orcid.org/0000-0003-1601-5146
240 Trong Nghia Hoang et al.
of the onboarding processes, students used the PET system. Students then rated the system
on a 5-point Likert scale (the higher the score, the better).
Group
Group Form
Group Chatbot
p
Interface
3.00 (1.25)
4.11 (1.11)
0.045
Conversation
2.67 (2.00)
4.11 (0.86)
0.023
Controls
3.33 (1.00)
4.67 (0.25)
0.004
Clarity
3.33 (1.50)
4.11 (0.61)
0.131
Conversation Quality
2.78 (0.69)
3.89 (0.61)
0.010
Search Function
4.00 (1.00)
4.11 (1.11)
0.821
Test Function
3.89 (0.36)
4.00 (0.25)
0.676
Visual Content
3.50 (0.86)
4.11 (0.61)
0.166
Tab. 1: Mean values of our questions; numbers in the brackets represent variances. Italic indicates
statistical significance (two-sided t-test, p<0.05).
Table 1 shows that the group chatbot generally perceives the PET as more sophisticated
and positive than the group form. During onboarding, most group form users did not
interact with the chatbot. Our results indicate that the group chatbot finds controlling the
system easier and the conversation quality better than users of group form. Moreover, we
observed that the group chatbot formulated longer and more natural sentences. We assume
that the users in the group chatbot are better primed to text naturally as they only could
text during onboarding, since the chatbot was the only interactable element.
We found no statistically significant difference in the perception of the test and search
function, the clarity, and visual content. This can be attributed to tests and searches being
relatively unrelated to the conversation. Furthermore, visual content and clarity are more
related to design than texting. Our study shows that the design of an onboarding process
can influence the perception of educational chatbots and is leading to the use of the chatbot
that is closer to a conversation rather than a search query.
Bibliography
[MC19] McAllister, Patrick; Kerr, James; McTear, Michael; Mulvenna, Maurice; Bond,
Raymond; Kirby, Karen et al. (2019): Towards Chatbots to Support Bibliotherapy
Preparation and Delivery.
[NA00] Nass, Clifford; Moon, Youngme (2000): Machines and mindlessness: Social responses
to computers.
[SO17] Sörensen, Ingrid (2017): Expectations on chatbots among novice users during the
onboarding process.
[WI18] Winkler, R., Söllner, M. (2018): Unleashing the Potential of Chatbots in Education: A
State-Of-The-Art Analysis.
[WO21] Wölfel, Matthias (2021): Towards the Automatic Generation of Pedagogical
Conversational Agents from Lecture Slides. In:. International Conference on
Multimedia Technology and Enhanced Learning: Springer, Cham, S. 216229.
... The system is capable of providing answers to given content-specific questions and evaluating the student's knowledge fully automatically. We found that students utilize the system differently depending on the onboarding process [5]. The onboarding provided two interfaces, a graphical user interface and a chat widget. ...
Article
Full-text available
The emergence of generative language models (GLMs), such as OpenAI’s ChatGPT, is changing the way we communicate with computers and has a major impact on the educational landscape. While GLMs have great potential to support education, their use is not unproblematic, as they suffer from hallucinations and misinformation. In this paper, we investigate how a very limited amount of domain-specific data, from lecture slides and transcripts, can be used to build knowledge-based and generative educational chatbots. We found that knowledge-based chatbots allow full control over the system’s response but lack the verbosity and flexibility of GLMs. The answers provided by GLMs are more trustworthy and offer greater flexibility, but their correctness cannot be guaranteed. Adapting GLMs to domain-specific data trades flexibility for correctness.
Chapter
Full-text available
Chatbots have become an increasingly popular choice for organisations in delivering services to users. Chatbots are beginning to become popular in mental health applications and are being seen as an accessible strategy in delivering mental health support alongside clinical/therapy treatment. The aim of this exploratory study was to investigate initial perceptions of the use and acceptance/adoption of text based chatbots by bibliotherapy facilitators and to also investigate perceptions of bibliotherapy facilitators using a bibliotherapy support chatbot through usability testing. Interviews were conducted to explore the relationship of bibliotherapy facilitators with chatbots, facilitators were then asked to complete a usability study using an early prototype chatbot. Post interviews were also conducted after usability study to discuss further improvements and requirements. Analysis of the interview transcripts reveal that bibliotherapy facilitators were keen on using chatbots to guide them in preparing for their bibliotherapy session for preparation and delivery. Facilitators stated that the reliability of the chatbot was a concern in relation to chatbot content and it is important to ensure that meaningful and quick conversational exchanges are designed. Analysis of the interview transcripts also stressed the importance of encoding a personality into the chatbot along with appropriate content to effectively guide facilitators. Facilitators stressed that appropriate onboarding and affordance measures should be integrated into the system to ensure that users are able to correctly interact chatbot and to understand the purpose of the chatbot as well as using personalisation for meaningful conversational exchanges.
Conference Paper
Full-text available
Chatbots are becoming a ubiquitous trend in many fields such as medicine, product and service industry, and education. Chatbots are computer programs used to conduct auditory or textual conversations. A growing body of evidence suggests that these programs have the potential to change the way students learn and search for information. Especially in large-scale learning scenarios with more than 100 students per lecturer, chatbots are able to solve the problem of individual student support. However, until now, there has been no systematic, structured overview of their use in education. The aim of this paper is therefore to conduct a systematic literature review based on a multi-perspective framework, from which we have derived initial search questions, synthesized past research, and highlighted future research directions. We reviewed titles and abstracts of 1405 articles drawn from management, education, information systems, and psychology literature before examining and individually coding a relevant subset of 80 articles. The results show that chatbots are in the very beginning of entering education. Few studies suggest the potential of chatbots for improving learning processes and outcomes. Nevertheless, past research has revealed that the effectiveness of chatbots in education is complex and depends on a variety of factors. With our literature review, we make two principal contributions: first, we structure and synthesize past research by using an input-process-output framework, and secondly, we use the framework to highlight research gaps for guiding future research in that area.
Expectations on chatbots among novice users during the onboarding process
  • Ingrid Sörensen
Sörensen, Ingrid (2017): Expectations on chatbots among novice users during the onboarding process.
Towards the Automatic Generation of Pedagogical Conversational Agents from Lecture Slides
  • Matthias Wölfel
Wölfel, Matthias (2021): Towards the Automatic Generation of Pedagogical Conversational Agents from Lecture Slides. In:. International Conference on Multimedia Technology and Enhanced Learning: Springer, Cham, S. 216-229.