Content uploaded by Andreas Reich
Author content
All content in this area was uploaded by Andreas Reich on Sep 16, 2022
Content may be subject to copyright.
P. A. Henning, M. Striewe, M. Wölfel (Hrsg.): 20. Fachtagung Bildungstechnologien (DELFI),
Lecture Notes in Informatics (LNI), Gesellschaft für Informatik, Bonn 2022 239
doi: 10.18420/delfi2022-050
The First Impression Counts!
The Importance of Onboarding for Educational Chatbots
Trong Nghia Hoang
1
, Andreas Reich
2
, Matthias Wölfel
3
1 Introduction, problem, and motivation
Chatbots are computer programs that simulate natural, human-like conversation with
humans via text interactions. Despite their great potential for educational scenarios, their
presence in education is relatively small [WI18]. A limiting factor is that developing a
chatbot requires a lot of expertise and effort. To address this, Wölfel presented the
PEdagogical conversational Tutor (PET) chatbot system, which automatically trains a
chatbot from PowerPoint slides [WO21]. The system can not only answer lecture-specific
questions but also offer automatically generated tests and rate the replies.
According to [MC19] onboarding is very important for chatbots as most users are not
aware of the features. However, onboarding for chatbots is not common. We noticed that
many users perceive and use the PET more like a search engine than a conversational
partner. According to [SO17], chatbot users can learn how to text with chatbots and adapt
their language and behavior. Users mindlessly transfer human social rules and
expectations to chatbots [NA00], but only if they assume they are talking to a system
capable of conversation. To investigate how the onboarding process can influence the
perception of the chatbot, we designed two onboarding processes for the PET system.
2 Study design and results
To investigate how onboarding new users into the PET system affects chatbot perception,
we recruited 18 students and randomly divided them into two groups to perform an AB
test. The "chatbot" group could only use the PET chatbot window to register and log in,
while the "form" group was prompted to fill out a form. Both groups were able to text with
the chatbot during the onboarding process, but the group chatbot had to text with the
chatbot to continue, while texting was optional for the group form. After completing one
1
Karlsruhe University of Applied Sciences, Faculty of Computer Science and Business Information Systems,
Moltkestraße 30, 76133 Karlsruhe, - hotr1011@h-ka.de
2
University of Hohenheim, Schloß Hohenheim 1, 70599 Stuttgart, andreas.reich@uni-hohenheim.de,
https://orcid.org/0000-0002-2426-6490
3
Karlsruhe University of Applied Sciences, Faculty of Computer Science and Business Information Systems,
Moltkestraße 30, 76133 Karlsruhe matthias.woelfel@h-ka.de; University of Hohenheim,
Schloß Hohenheim 1, 70599 Stuttgart, https://orcid.org/0000-0003-1601-5146
240 Trong Nghia Hoang et al.
of the onboarding processes, students used the PET system. Students then rated the system
on a 5-point Likert scale (the higher the score, the better).
Group
Group Form
Group Chatbot
p
Interface
3.00 (1.25)
4.11 (1.11)
0.045
Conversation
2.67 (2.00)
4.11 (0.86)
0.023
Controls
3.33 (1.00)
4.67 (0.25)
0.004
Clarity
3.33 (1.50)
4.11 (0.61)
0.131
Conversation Quality
2.78 (0.69)
3.89 (0.61)
0.010
Search Function
4.00 (1.00)
4.11 (1.11)
0.821
Test Function
3.89 (0.36)
4.00 (0.25)
0.676
Visual Content
3.50 (0.86)
4.11 (0.61)
0.166
Tab. 1: Mean values of our questions; numbers in the brackets represent variances. Italic indicates
statistical significance (two-sided t-test, p<0.05).
Table 1 shows that the group chatbot generally perceives the PET as more sophisticated
and positive than the group form. During onboarding, most group form users did not
interact with the chatbot. Our results indicate that the group chatbot finds controlling the
system easier and the conversation quality better than users of group form. Moreover, we
observed that the group chatbot formulated longer and more natural sentences. We assume
that the users in the group chatbot are better primed to text naturally as they only could
text during onboarding, since the chatbot was the only interactable element.
We found no statistically significant difference in the perception of the test and search
function, the clarity, and visual content. This can be attributed to tests and searches being
relatively unrelated to the conversation. Furthermore, visual content and clarity are more
related to design than texting. Our study shows that the design of an onboarding process
can influence the perception of educational chatbots and is leading to the use of the chatbot
that is closer to a conversation rather than a search query.
Bibliography
[MC19] McAllister, Patrick; Kerr, James; McTear, Michael; Mulvenna, Maurice; Bond,
Raymond; Kirby, Karen et al. (2019): Towards Chatbots to Support Bibliotherapy
Preparation and Delivery.
[NA00] Nass, Clifford; Moon, Youngme (2000): Machines and mindlessness: Social responses
to computers.
[SO17] Sörensen, Ingrid (2017): Expectations on chatbots among novice users during the
onboarding process.
[WI18] Winkler, R., Söllner, M. (2018): Unleashing the Potential of Chatbots in Education: A
State-Of-The-Art Analysis.
[WO21] Wölfel, Matthias (2021): Towards the Automatic Generation of Pedagogical
Conversational Agents from Lecture Slides. In:. International Conference on
Multimedia Technology and Enhanced Learning: Springer, Cham, S. 216–229.