Conference PaperPDF Available

Introducing Vicky: A Pedagogical Conversational Agent for the Classification of Learning Styles

Authors:

Abstract

Learners are faced with the challenge of processing a large amount of knowledge. However, they often lack individual support, and teaching is not tailored to their learning styles. Conversational Agents (CAs) could be a way to identify personal learning styles through a dialog between the CA and the learner, and to support him/her accordingly. This paper investigates whether learning styles can be determined through a dialog with a CA and how the conversation is perceived by users. For this purpose, a CA called Vicky was developed using the platform Rasa and the intent classifier DIET. Vicky determines the user's learning styles through a questionnaire as well as a quiz and acts human-like to be perceived as a virtual companion. The prototype was evaluated in an experiment with 25 participants. They predominantly perceived themselves to be correctly classified and treated kindly by the CA. Overall, we contribute to science and practice by showing that CAs acting as virtual companions can be used to better understand learner preferences.
Introducing Vicky: A Pedagogical Conversational Agent
for the Classification of Learning Styles
Bijan Khosrawi-Rad1[0000-0002-9074-5385], Paul Keller1[0000-0002-7786-6406], Linda
Grogorick1[0000-0003-0270-2666], and Susanne Robra-Bissantz1[0000-0001-9972-3935]
1 Technische Universität Braunschweig, Braunschweig, Germany
{b.khosrawi-rad, p.keller, l.grogorick, s.robra-bissantz}
@tu-braunschweig.de
Abstract. Learners are faced with the challenge of processing a large amount of
knowledge. However, they often lack individual support, and teaching is not tai-
lored to their learning styles. Conversational Agents (CAs) could be a way to
identify personal learning styles through a dialog between the CA and the learner,
and to support him/her accordingly. This paper investigates whether learning
styles can be determined through a dialog with a CA and how the conversation is
perceived by users. For this purpose, a CA called Vicky was developed using the
platform Rasa and the intent classifier DIET. Vicky determines the user’s learn-
ing styles through a questionnaire as well as a quiz and acts human-like to be
perceived as a virtual companion. The prototype was evaluated in an experiment
with 25 participants. They predominantly perceived themselves to be correctly
classified and treated kindly by the CA. Overall, we contribute to science and
practice by showing that CAs acting as virtual companions can be used to better
understand learner preferences.
Keywords: Conversational Agent, Virtual Companion, Education, Learning
Style Classification
1 Introduction and Motivation
School and university students have to process and remember a large amount of
knowledge during their academic life. However, in many educational institutions, there
is a lack of personalized support for learners, and teaching is not tailored to their indi-
vidual preferences [13]. Research suggests that learners have different learning styles,
which characterize learner preferences (e.g., in how knowledge is presented) and can
be assessed through questionnaires [1, 4]. The technology-assisted adaptation of the
instruction of learning content to learning styles could be a way to tailor teaching to the
individual needs of learners and thus contribute to solving the problems outlined [1, 5].
Conversational Agents (CAs) can be used to identify learners' needs and provide them
with individual support [6, 7]. CAs are intelligent dialog systems (e.g., chatbots) that
can communicate with their users using natural language [8]. One example is the virtual
learning tutor Oscar, which determines learning styles in a tutoring session and
teaches computer science knowledge adapted to the identified learning styles [1]. The
2
research relevance of pedagogical CAs has increased significantly in recent years [7,
9]. In addition, the CA design in general has evolved as well. Whereas classical ap-
proaches such as Sirifulfilled especially assistance functions, novel CAs like Rep-
likacan establish a friendly, companion-like relationship with users, interacting with
them emotionally, and thus fostering their trust [10, 11]. Virtual companions are char-
acterized by the fact that they interact with their users as co-equal partners [11], rather
than as tutors as in the case of Oscar [1]. Despite the great potential of CAs, many
interactions with them are not motivating [12]. Gamifying the dialog by incorporating
game elements could be one way to stimulate the conversations, although gamified CAs
are still underrepresented in research [ibid.]. While there are existing approaches for
learning style detection by CAs [1, 13], there is a lack of corresponding solutions in-
corporating companionship properties [9]. To address the identified issues (lack of per-
sonalized teaching based on learning styles; existing CAs are less stimulating), we de-
veloped and evaluated a CA called Vickyas part of a Design Science Research (DSR)
project. Vicky is a level 1 artifact [14] that determines learning styles via a companion-
like conversation consisting of a questionnaire as well as a quiz game.
2 Design of the Artifact
We developed the artifact usingRasa, an open-source platform for designing CAs,
which enables the quick implementation of a CA as well as its integration with external
services. Furthermore, we used DIET” (Dual Intent and Entity Transformer) as the
entity extractor and intent classifier. Rasa contains further intelligent functions for in-
teraction design, e.g., an entity synonym mapper to enable the use of synonyms, or a
fallback classifier to handle messages that cannot be unambiguously assigned to an in-
tent. The dialog with the CAs is entirely in English and text-based. Therefore, we inte-
grated the CA into the Telegramplatform to create an interface to a well-used mes-
senger service. Established design knowledge for CAs, especially from the Information
Systems domain, was used to enable a rigorous design of the artifact [14]. E.g., the CA
exhibits human-like traits (such as name or communication behavior) and addresses the
user personally to build a personal bond [15]. Vicky interacts in an overall friendly and
at the same time transparent manner to build trust by explaining the aims of the inter-
action (classification of learning styles) [3, 16]. When the user wonders about the rele-
vance of the dialog or whether he/she is communicating with a human or a machine,
the CA sets out the aim of the dialog in a kindly, motivating, and transparent manner
[16]. We also added a chitchat with Vicky telling jokes to keep the dialog interesting
[3]. However, to fulfill the objective of the dialog, the CA leads the user back to the
classification of learning styles as soon as the chitchat gets too in-depth (see Fig. 1).
The CA uses the FS model of Felder and Silverman [17] to identify learning styles.
We chose this model because it has already been used in technology-enhanced learning
[13] and is scientifically validated [1]. The FS model indicates that learning styles lie
on a continuum of the following dimensions: sensing/intuitive (preference for intaking
information), visual/verbal (preference for presenting information), active/reflective
(preference for processing information), and sequential/global (preference for
3
understanding information) [17]. The recognition of learning styles is done with a re-
duced version (17 items) of the Index of Learning Styles(ILS) questionnaire, which
was also used as well as validated by Latham (2011) [1]. During the interaction, the
learner has to choose one answer option for each question. If he/she is uncertain regard-
ing the preference of an option, Vicky explains to choose the option that is most likely
to apply. The CA remembers the selected answers through natural language understand-
ing and calculates the corresponding set of learning styles at the end of the conversation
based on logical rules [17]. Vicky also informs the user about the result of the question-
naire, explains the identified learning styles step by step and advises the learner on how
to optimize his/her learning process accordingly. Based on the FS model, it is also pos-
sible that the learner might exhibit a balanced learning style for some dimensions [1,
17]. To stimulate the interaction [12], questions are embedded in a storyline of an initial
small talk as well as questions about the learner's personality and everyday study life.
Furthermore, the CA builds common ground during the dialog to promote the percep-
tion as a virtual companion [11], e. g.: ‘You are a pretty cool person’. In doing so, the
CA shows sympathy and exhibits active listening [11, 15], e.g.: ‘You said at the begin-
ning that you remember your activity from yesterday as a picture’.
To refine the classification of learning styles, as well as to gamify the dialog [12],
we integrated a quiz game in which the learner has to solve certain tasks step by step
(also called scaffolding) [6] Based on the behavior in finding the solution, learning
styles are classified again [1]. Learning styles can be described by a small number of
characteristics, e.g., visual learners prefer to have content presented visually through
pictures or videos [1, 17]. If learners tend to answer questions correctly after watching
a video, this indicates a preference for visual learning content [1]. The identification of
learning styles is also based on logical rules. For this purpose, four different question
types were chosen, which are based on the FS model [1, 17]: practical questions (e.g.,
math problems), theoretical questions (to test knowledge and understanding), process
questions (to check whether the learner prefers to be guided through a process while
working on the solution), and trick questions (to test attention). If learners respond to
practical questions directly, this promotes an active and sensory learning style. If learn-
ers wish to be guided through the process when asked process questions, this indicates
a sequential and reflective learning style. Correctly answering theory questions favors
a reflective and intuitive learning style. Responding properly to trick questions indicates
that learners are good at focusing their attention, which favors a sensory and verbal
learning style. Learners can answer quiz questions directly or request help through a
button. If the answer is correct, Vicky informs the user about it and congratulates
him/her to foster motivation. In case of a wrong answer, different aids (e.g., pictures,
videos, texts) are offered, and depending on what the learner chooses, this influences
the learning style classification along with the logical rules. The concrete quiz questions
were developed in a joint discussion among the authors to design generally applicable
exercises.
Fig. 1 exemplifies the design of the CA, consisting of the ILS questionnaire, the quiz
game, and the chitchat. In addition, the following demo video illustrates the artifact:
https://youtu.be/BykICWZ_M2o.
4
Fig. 1. Excerpts of the Conversation with Vicky
3 Evaluation of the Artifact
We evaluated the artifact through a laboratory experiment so that participants could ask
questions in case of technical challenges. The 25 participants (female: 8, male: 17, avg.
age: 26) tested the CA (questionnaire and quiz game) and answered an online survey.
We assessed the general interaction with Vicky as well as the comprehensibility and
the subjective correctness of the learning style classification [4]. In addition, we asked
open-ended questions to assess the users’ overall impression.
In general, interaction and comprehensibility were evaluated positively. All partici-
pants perceived Vicky's questions as understandable. On average, the quiz questions
were considered to be difficult, but this was necessary for Vicky to capture the reaction
to the offered aids. Most participants had no problems in understanding the quiz ques-
tions, but some lacked background knowledge in solving the math tasks. E.g., they did
not know some terms like the median, but Vicky explained them through scaffolding.
The majority of the participants (n = 21) did not feel the need to abort the conversation.
Four participants wanted to quit because they felt misunderstood by the CA due to the
quiz length and the answers given. Slightly less than half of the participants (n = 11)
found both interactions equally entertaining. Almost a third preferred the dialog based
on the ILS questionnaire (n = 8). The remaining participants (n =6) enjoyed the quiz
the most. In general, the results show that Vicky is mostly correct in assessing learning
styles. 24 out of 25 participants felt that they had been correctly assessed by the CA
and were satisfied with the identified classification, as it was perceived as realistic.
However, classification via questionnaire was not only more fun but also evaluated
better, i.e., participants felt that they were better classified (MVQuestionnaire = 5,24 vs.
MVQuiz = 4,74 with a seven-point Likert scale from 1 = extremely disagree to 7 =
5
extremely agree). Finally, participants could mention positive aspects and improvement
suggestions regarding our artifact. The CA was associated with the following positive
aspects: fast, understandable, friendly, helpful, motivating, fun, and authentic. How-
ever, a few participants still perceived the interaction to be time-intensive, not yet hu-
man-like enough, and inflexible, so we could gain feedback for future improvement.
4 Conclusion
Significance to Research and Practice: The use of learning style classification is not
new to individualize teaching so that learners are optimally supported in studying [e.g.,
1, 13]. However, to the best of our knowledge, our proposed CA is the first artifact
incorporating companionship properties as well as two different options for classifica-
tion (questionnaire and quiz) to enable a stimulating dialog. Therefore, our research is
a first step towards exploring the design of learning style classification with a CA to
really provide an added value in learning environments while motivating learners. The
evaluation results indicate that users perceived their classification as correct, with more
agreement on the classification via questionnaire.
Technological progress is accelerating the development and distribution of CAs in a
variety of contexts, such as education [3, 7, 9]. As adaptive and personalized learning
can be seen as the future of online education [3, 7], our research contributes to further
evidence to the notion that classification of learning styles via CAs is possible as a way
to learn more about the learners and their preferences. This could also be extended to
other forms of needs assessments, such as surveying personality types to make learning
truly adaptive, personalized, and oriented towards achieving learning success.
Limitations and Outlook: While this paper already provides initial findings, it also
has some limitations. Only a small number of participants (n = 25) engaged in the ex-
periment, and they were asked to subjectively assess whether their learning style clas-
sification by Vicky was correct. However, this was necessary to gain training data in
an experimental setting following the recommendation of Latham (2011) [1]. An eval-
uation with a larger sample is already planned. In some cases, participants' intentions
were not well understood, so that they wanted to quit the interaction (n = 4). Therefore,
we integrated more buttons to provide choices and increase comprehension [18]. How-
ever, it is also important to collect more training data. Currently, the interactions with
Vicky are a bit inflexible, so an interruption could hinder a complete classification of
the learning styles. As a next step, we plan to refine Vicky’s language comprehension
as well as the quiz questions. Overall, additional design cycles are necessary to provide
rigorous knowledge [14]. For this purpose, we plan an iterative development with the
integration of further training data that involves an evaluation with a larger sample.
In summary, our novel artifact as well as the findings from its evaluation provide
insights into how CAs can be used for technology-enhanced learning.
Acknowledgements. This paper results from the project StuBu, funded by the German
Federal Ministry of Education and Research (BMBF); Grant #21INVI06.
6
References
1. Latham, A.M.: Personalising Learning with Dynamic Prediction and Adaptation to Learning
Styles in a Conversational Intelligent Tutoring System. (2011).
2. Seaman, J.E., Allen, I.E., Seaman, J.: Grade increase: Tracking distance education in the
United States. Babson Survey Research Group. (2018).
3. Wambsganss, T., Söllner, M., Leimeister, J.M.: Design and evaluation of an adaptive dialog-
based tutoring system for argumentation skills. In: Proceedings of the 41st International
Conference on Information Systems (ICIS). Hyderabad, India (2020).
4. Felder, R.M., Soloman, B.A.: Index of learning styles. (1991).
5. Luan, H., Tsai, C.-C.: A review of using machine learning approaches for precision educa-
tion. Educational Technology & Society. 24, 250266 (2021).
6. Winkler, R., Hobert, S., Salovaara, A., Söllner, M., Leimeister, J.M.: Sara, the Lecturer:
Improving Learning in Online Education with a Scaffolding-Based Conversational Agent.
In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. pp.
1–14. Association for Computing Machinery, New York, USA (2020).
7. Karrenbauer, C., König, C.M., Breitner, M.H.: Individual Digital Study Assistant for Higher
Education Institutions: Status Quo Analysis and Further Research Agenda. Presented at the
Wirtschaftsinformatik 2021 Proceedings. Essen, Germany (2021).
8. Gnewuch, U., Morana, S., Maedche, A.: Towards Designing Cooperative and Social Con-
versational Agents for Customer Service. In: Proceedings of the 38th International Confer-
ence on Information Systems (ICIS). Seoul, South Korea (2017).
9. Khosrawi-Rad, B., Rinn, H., Schlimbach, R., Gebbing, P., Yang, X., Lattemann, C., Robra-
Bissantz, S.: Conversational Agents in Education A Systematic Literature Review. In:
Proceedings of the 30th European Conference on Information Systems (ECIS). Timișoara,
Romania (forthcoming).
10. Xie, T., Pentina, I.: Attachment Theory as a Framework to Understand Relationships with
Social Chatbots: A Case Study of Replika. In: Proceedings of the 55th Hawaii International
Conference on System Sciences (HICSS). Maui, USA (2022).
11. Strohmann, T.: From Assistance to Companionship-Designing Virtual Companions. (2021).
12. Benner, D., Schöbel, S., Süess, C., Baechle, V., Janson, A.: Level-Up your Learning In-
troducing a Framework for Gamified Educational Conversational Agents. In:
Wirtschaftsinformatik 2022 Proceedings. Nürnberg, Germany (2022).
13. Dağ, F., Geçer, A.: Relations between online learning and learning styles. Procedia - Social
and Behavioral Sciences. 1, 862871 (2009).
14. Gregor, S., Hevner, A.R.: Positioning and presenting design science research for maximum
impact. MIS quarterly. 37, 337355 (2013).
15. Liebrecht, C., van Hooijdonk, C.: Creating Humanlike Chatbots: What Chatbot Developers
Could Learn from Webcare Employees in Adopting a Conversational Human Voice. In:
Følstad, A., Araujo, T., Papadopoulos, S., Law, E.L.-C., Granmo, O.-C., Luger, E., and
Brandtzaeg, P.B. (eds.) Chatbot Research and Design. pp. 5164. Springer, Cham (2020).
16. Wambsganss, T., Höch, A., Zierau, N., Söllne, M.: Ethical Design of Conversational Agents:
Towards Principles for a Value-Sensitive Design. In: Wirtschaftsinformatik 2021 Proceed-
ings. Essen, Germany (2021).
17. Felder, R.M., Silverman, L.K.: Learning and teaching styles in engineering education. En-
gineering education. 78, 674681 (1988).
18. Han, E., Yin, D., Zhang, H.: Interruptions during a service encounter: Dealing with imper-
fect chatbots. In: Proceedings of the 42nd International Conference on Information Systems
(ICIS) (2021).
ResearchGate has not been able to resolve any citations for this publication.
Conference Paper
Full-text available
Driven by circumstances like the global pandemic many learners and educators realize the importance and value of self-regulated digital learning. To better support self-regulated learning, conversational agents have become more relevant. Conversational agents can act as tutor or as learning mate for learners. Although conversational agents have potential to better support self-regulated learning processes, challenges exist requiring implications about how to make these interactions more engaging and supportive. This study discusses the value of gamified conversational learning chatbots that use game elements to engage learners to guide researchers and practitioners to design conversational agents that effectively motivate learners and provide self-regulated learning at the same time. Therefore, we propose a systematically developed framework for gamifying educational conversational agents and contribute to theory by consolidating several theories about games, digital learning, and conversational agents and support practitioners by providing implications about what to care about when gamifying conversational agents.
Conference Paper
Full-text available
Conversational Agents (CAs) are widely spread in a variety of domains, such as health and customer service. There is a recent trend of increasing publications and implementations of CAs in education. We conduct a systematic literature review to identify common methodologies, pedagogical CA roles, addressed target groups, the technologies and theories behind, as well as human-like design aspects. The initially found 3329 records were systematically reduced to 252 fully coded articles. Based on the analysis of the codings, we derive further research streams. Our results reveal a research gap for long-term studies on the use of CAs in education, and there is insufficient holistic design knowledge for pedagogical CAs. Moreover, target groups other than academic students are rarely considered. We condense our findings in a morphological box and conclude that pedagogical CAs have not yet reached their full potential of long-term practical application in education.
Conference Paper
Full-text available
With increasing adoption of AI social chatbots, especially during the pandemic-related lockdowns, when people lack social companionship, there emerges a need for in-depth understanding and theorizing of relationship formation with digital conversational agents. Following the grounded theory approach, we analyzed in-depth interview transcripts obtained from 14 existing users of AI companion chatbot Replika. The emerging themes were interpreted through the lens of the attachment theory. Our results show that under conditions of distress and lack of human companionship, individuals can develop an attachment to social chatbots if they perceive the chatbots’ responses to offer emotional support, encouragement, and psychological security. These findings suggest that social chatbots can be used for mental health and therapeutic purposes but have the potential to cause addiction and harm real-life intimate relationships.
Conference Paper
Full-text available
Conversational Agents (CAs) have become a new paradigm for human-computer interaction. Despite the potential benefits, there are ethical challenges to the widespread use of these agents that may inhibit their use for individual and social goals. However, besides a multitude of behavioral and design-oriented studies on CAs, a distinct ethical perspective falls rather short in the current literature. In this paper, we present the first steps of our design science research project on principles for a value-sensitive design of CAs. Based on theoretical insights from 87 papers and eleven user interviews, we propose preliminary requirements and design principles for a value-sensitive design of CAs. Moreover, we evaluate the preliminary principles with an expert-based evaluation. The evaluation confirms that an ethical approach for design CAs might be promising for certain scenarios.
Conference Paper
Full-text available
Recent advances in Natural Language Processing not only bear the opportunity to design new dialog-based forms of human-computer interaction but also to analyze the argumentation quality of texts. Both can be leveraged to provide students with individual and adaptive tutoring in their personal learning journey to develop argumentation skills. Therefore, we present the results of our design science research project on how to design an adaptive dialog-based tutoring system to help students to learn how to argue. Our results indicate the usefulness of an adaptive dialog-based tutoring system to support students individually, independent of a human instructor, time and place. In addition to providing our embedded software artifact, we document our evaluated design knowledge as a design theory. Thus, we provide the first step toward a nascent design theory for adaptive conversational tutoring systems to individual support metacognition skill education of students in traditional learning scenarios.
Conference Paper
Full-text available
The idea of interacting with computers through natural language dates back to the 1960s, but recent technological advances have led to a renewed interest in conversational agents such as chatbots or digital assistants. In the customer service context, conversational agents promise to create a fast, convenient, and cost-effective channel for communicating with customers. Although numerous agents have been implemented in the past, most of them could not meet the expectations and disappeared. In this paper, we present our design science research project on how to design cooperative and social conversational agents to increase service quality in customer service. We discuss several issues that hinder the success of current conversational agents in customer service. Drawing on the cooperative principle of conversation and social response theory, we propose preliminary meta-requirements and design principles for cooperative and social conversational agents. Next, we will develop a prototype based on these design principles.
Article
In recent years, in the field of education, there has been a clear progressive trend toward precision education. As a rapidly evolving AI technique, machine learning is viewed as an important means to realize it. In this paper, we systematically review 40 empirical studies regarding machine-learning-based precision education. The results showed that the majority of studies focused on the prediction of learning performance or dropouts, and were carried out in online or blended learning environments among university students majoring in computer science or STEM, whereas the data sources were divergent. The commonly used machine learning algorithms, evaluation methods, and validation approaches are presented. The emerging issues and future directions are discussed accordingly.
Chapter
Currently, conversations with chatbots are perceived as unnatural and impersonal. One way to enhance the feeling of humanlike responses is by implementing an engaging communication style (i.e., Conversational Human Voice (CHV); Kelleher 2009) which positively affects people’s perceptions of the organization. This communication style contributes to the effectiveness of online communication between organizations and customers (i.e., webcare), and is of high relevance to chatbot design and development. This project aimed to investigate how insights on the use of CHV in organizations’ messages and the perceptions of CHV can be implemented in customer service automation. A corpus study was conducted to investigate which linguistic elements are used in organizations’ messages. Subsequently, an experiment was conducted to assess to what extent linguistic elements contribute to the perception of CHV. Based on these two studies, we investigated whether the amount of CHV can be identified automatically. These findings could be used to design humanlike chatbots that use a natural and personal communication style like their human conversation partner.