Content uploaded by Regina Kaplan-Rakowski
Author content
All content in this area was uploaded by Regina Kaplan-Rakowski on Feb 02, 2024
Content may be subject to copyright.
“I Can Talk to Spanish Speakers in Illinois!”:
Student Perspectives on AI-Avatar Role Plays in Virtual Reality
Tricia Thrasher, Immerse, United States
Regina Kaplan-Rakowski, University of North Texas, United States
Uliana Ovsiannikova, University of Illinois at Urbana-Champaign, United States
Justine Meyr, WASC Senior College and University Commission, United States
Ye Yuan, University of California–Santa Barbara, United States
WorldCALL 2023 – CALL in Critical Times
Conference Proceedings
Abstract
This paper reports on a study exploring students’ perceptions of AI-avatar interactions within
a virtual reality (VR) language learning environment, Immerse. As part of a large-scale
project, this research explores the impact of Immerse’s AI-powered avatars on the
educational experience of 54 high school participants. Based on students’ attitudes and
opinions expressed in post-use surveys, findings reveal a positive experience. The majority of
students expressed enjoyment, being able to understand the AI chatbots, and being focused
on the VR activities. Challenges, such as technical and acoustic difficulties, underscore the
need for refinement in VR technology. This study contributes insights into the ongoing
integration of AI-avatar role plays in VR-assisted language education.
Keywords: Virtual Reality-Assisted Language Learning, Artificial Intelligence, AI-Powered
Avatars, Student Perspectives
Introduction
This paper reports on the early stages of a large-scale project examining student attitudes and
perceptions of learning with AI-avatars in the VR language learning platform, Immerse
(Thrasher et al., 2023). With grant-funding received in collaboration with Immerse, Inc. from
Facebook/Meta, we are conducting a large-scale project that involves distributing 500 VR
headsets across 10 high schools located throughout Illinois, California, and Texas, and
evaluating their impact on language education. As of December 2023, approximately 487
students are actively involved in the project.
This paper will focus on a subset of 54 participants (n1 = 54) who have been using Immerse’s
AI-powered avatars for L2 speaking practice in Spanish or French. After an overview of the
project and language learning platform, students’ attitudes regarding the impact of conversing
with AI-powered avatars on their language learning will be presented. Understanding
students’ opinions towards interacting with AI-powered avatars in VR is critical, as AI and
VR technologies are continually being refined and becoming readily available to educators.
Conversational AI in Second Language Acquisition
Conversational AI is designed to simulate real-life conversation thanks to natural language
processing (NLP) and natural language generation (NLG). With its recent surge in
development, many CALL researchers have focused on using conversational AI in second
language education. While several studies have explored the potential of this technology to
improve students’ willingness to communicate (WTC) and foreign language anxiety (FLA;
Ayedoun et al., 2019; Lee & Lee, 2020; Tai & Chen, 2020), most studies have focused on
students’ perceptions of conversational AI due to the relative novelty of the technology in
SLA (Fryer et al., 2019). The connection between students’ positive perceptions of
conversational AI and students’ engagement in foreign language classes has been
documented (Dizon & Tang, 2020; Yang et al., 2022). As positive perceptions of the
technology are closely tied to the human-likeness of conversational AI partners (Ebadi &
Amini, 2022; Fryer et al., 2019, Wang et al., 2022), immersive conversational AI, which
combines personalization of AI-powered bots with immersive experience in VR, warrants
further exploration in foreign language classrooms.
Several short-term studies on immersive conversational AI in SLA have been conducted in
non-naturalistic settings. For example, Hassani et al. (2016) reported significant learning
gains in the speaking skills of 10 university-level English as a Second Language (ESL)
students. Divekar et al. (2021) reported gains in vocabulary acquisition, increased WTC, and
positive perceptions of the technology among Chinese as a Foreign Language learners.
Nevertheless, more research needs to be conducted in a naturalistic setting with high-school
students focusing on other languages to contribute to the ecological validity of studies on
immersive conversational AI for SLA.
Methods
Participants in the study were 54 high school students enrolled in either French or Spanish
language classes. Their ages ranged from 13-19 years old (M = 15.46, SD = 1.11). All
students came from central Illinois, USA.
Students completed all VR activities in Immerse (www.immerse.com) using a Meta Quest 2
VR headset. In Immerse, students engaged in AI-powered avatar role plays which allowed
them to practice different communicative tasks one-on-one with an avatar in the target
language. Immerse has dozens of AI-role plays for pre-A1 – B2 (i.e., beginner-intermediate)
level learners, based on the Common European Framework of Reference for Languages
(CEFR). Scenarios ranged from ordering at a restaurant to talking about ailments at the
doctor’s office (Figure 1). Students were guided through each role play by a checklist that
told them what they needed to accomplish (Figure 2). Immerse offers built-in features that
allow students to ask for hints and translate words into English or their L2 when at a loss for
words. The AI avatars, powered by ChatGPT 4.0, can “understand” input that is not entirely
correct grammatically and can also reply to requests from the learners, for example, to speak
more slowly (see Figure 2).
Figure 1: AI practice scenes in Immerse
Figure 2: Screenshot of AI-powered role play asking about directions
For this study, students completed role plays in Immerse’s shopping center, fast-food
restaurant, and home scene. All role plays were selected by teachers to ensure that the task
aligned with what students were learning in their language classes.
Following each VR activity, participants completed a 5-point Likert-scale post-activity
questionnaire where they gave feedback on their experience. In this survey, participants rated
21 statements that targeted constructs such as enjoyment, FLA, and cognitive load. This
survey was kept brief to allow students to quickly respond and not detract from class time.
Students were also asked to provide qualitative feedback about what they liked and disliked
(if anything) about the AI-powered role-play scenarios.
Findings and Discussion
Table 1 presents participants’ ratings to a subset of 10 questionnaire statements, particularly
those pertinent to students’ interaction with the AI avatars.
Statement
Strongly
Disagree
Disagree
Neutral
Agree
Strongly
Agree
I enjoyed using VR today.
7.69%
15.28%
12.82%
33.33%
30.77%
VR helped me communicate
easily in the foreign
language.
8.11%
21.62%
21.62%
37.84%
10.81%
I understood most of what
the AI bot said.
5.56%
8.33%
8.33%
44.44%
33.33%
Chatting with the AI bot
helped me learn phrases and
sentences.
5.56%
19.44%
13.89%
41.46%
19.44%
Chatting with the AI bot felt
like a real conversation.
9.09%
15.15%
33.33%
33.33%
9.09%
I felt relaxed when
completing the activity
today.
5.56%
16.67%
30.56%
33.33%
13.89%
I felt confident during this
activity.
2.86%
17.14%
25.71%
40.00%
14.29%
I was more nervous during
this activity than during
normal class time.
25.71%
45.71%
20.00%
8.57%
0%
I was fully focused on the
activities in VR today.
3.03%
6.06%
21.21%
57.58%
12.12%
I was distracted by too many
things in VR today.
15.15%
48.48%
18.18%
15.15%
3.03%
Table 1: Students’ post-activity survey responses
Students’ responses indicate that they had a positive experience using VR. Most students
(64.1%) reported enjoying using VR. Moreover, about half (48.65%) of participants said that
it helped them communicate easily in a foreign language. Regarding interacting with AI-bots,
the majority (77.8%) expressed being able to understand most of what the AI-bot said, and
60.9% of the students said that chatting with AI helped them learn new phrases and
sentences. Claiming that chatting with AI felt like a real conversation was expressed by
42.4% of the participants.
Regarding students’ confidence and focus in VR, almost half (47.22%) of the students
reported being relaxed, with a little over half (54.29%) being confident during VR activities.
Only 8.57% of students said that they were more nervous during VR activities than during
normal class time. The majority of participants (69.7%) said that they were able to be fully
focused on the VR activities and only 18.18% reported being distracted by things in the VR
environment.
Lastly, students provided both positive and negative comments about their AI-VR
experiences (Table 2).
What students liked:
What students disliked:
Exploration
• “Exploring new locations.”
• “It was fun to take a break from a
normal class setting.”
Interaction
• “Talking to others.”
• “Got to have an advanced
conversation.”
• “The natural conversation with the
task list is nice.”
• “That I can talk to Spanish speakers
in Illinois.”
Learner Autonomy
• “I like just being on my own and
being able to learn the meaning of
words I didn’t know easily.”
• “Everyone was able to work on their
own activity and could choose the
difficulty at which best fit their skill
level.”
Technical Issues
• “The AI didn’t often work.”
• “Microphone didn’t pick up my
speech even when it was loud.”
Communication Issues
• “I was unable to communicate with
the AI bot in any capacity.”
Classroom Issues
• “Talking with everyone in the room
is challenging.”
Lack of Student Training
• “I felt like I didn’t know enough
about the headset in order to fix
problems myself.”
Table 2: Students’ qualitative feedback
Although students were provided with headphones that could be used with the headsets, they
still struggled at times to communicate effectively with the AI-bots due to background noise
from their classmates who were located in the same physical space. This is a limitation of
using VR in a classroom setting, as most VR experiences have been designed for users to
connect while being physically located in different locations.
Conclusion
This study examined students’ perceptions regarding interaction with AI chatbots in Immerse
within the context of foreign language education. The findings reveal a generally positive
reception to VR as a language learning tool. Most students enjoyed the VR experience,
feeling capable of understanding the AI bot’s speech, and learning new phrases and
sentences. These findings are aligned with previous studies reporting positive perceptions
towards conversational AI (Dizon & Tang, 2020; Yang et al., 2022) and new vocabulary
acquisition facilitation (Divekar et al. 2021). Interestingly, the minority of participants felt
that chatting with the AI bot was like a real conversation. This finding warrants further
investigation. Perhaps integrating more VR actions or interactions with the AI conversations
could be considered. Nonetheless, in the open-ended comments, students appreciated being
able to talk to others (e.g., Spanish speakers in Illinois) and having “advanced” and “natural”
conversations with the AI chatbots.
Most participants expressed high confidence. Importantly, they believed they were able to
fully focus on activities in VR which highlights VR’s ability to immerse learners in the
virtual environment. However, challenges, such as communication difficulties with AI-
powered avatars and technical issues, underline the need for further refinement. To address
these challenges, we suggest the manufacturers enhance microphones to eliminate
background noise while recording. In addition, we suggest that students receive sufficient
training on how to better navigate hand controllers in VR. Pedagogically, we suggest that AI
interactions with chatbots are better suited as individual assignments or small-group
activities, but not as whole-class activities. As VR technology advances, addressing these
challenges will be crucial to maximizing the benefits of AI-avatar role plays in language
education.
Acknowledgements
We would like to thank Meta and Immerse for their support and funding of this project. We
would also like to acknowledge the members of our larger research team who were not able
to be included on this publication, including: Dr. Dorothy Chun (Professor Emeritus at the
University of California, Santa Barbara) and Dr. Randall Sadler (Full Professor at the
University of Illinois at Urbana-Champaign).
References
Ayedoun, E., Hayashi, Y., & Seta, K. (2019). Adding Communicative and Affective
Strategies to an Embodied Conversational Agent to Enhance Second Language
Learners’ Willingness to Communicate. International Journal of Artificial
Intelligence in Education, 29(1), 29–57. https://doi.org/10.1007/s40593-018-0171-6
Divekar, R. R., Drozdal, J., Chabot, S., Zhou, Y., Su, H., Chen, Y., Zhu, H., Hendler, J. A., &
Braasch, J. (2022). Foreign language acquisition via artificial intelligence and
extended reality: Design and evaluation. Computer Assisted Language Learning,
35(9), 2332–2360. https://doi.org/10.1080/09588221.2021.1879162
Dizon, G., & Tang, D. (2020). Intelligent personal assistants for autonomous second language
learning: An investigation of Alexa. The JALT CALL Journal, 16(2), 107–120.
https://doi.org/10.29140/jaltcall.v16n2.273
Ebadi, S., & Amini, A. (2022). Examining the roles of social presence and human-likeness on
Iranian EFL learners’ motivation using artificial intelligence technology: A case of
CSIEC chatbot. Interactive Learning Environments, 0(0), 1–19.
https://doi.org/10.1080/10494820.2022.2096638
Fryer, L. K., Nakao, K., & Thompson, A. (2019). Chatbot learning partners: Connecting
learning experiences, interest and competence. Computers in Human Behavior, 93,
279–289. https://doi.org/10.1016/j.chb.2018.12.023
Hsu, H.-L., Chen, H. H.-J., & Todd, A. G. (2021). Investigating the impact of the Amazon
Alexa on the development of L2 listening and speaking skills. Interactive Learning
Environments, 1–14. https://doi.org/10.1080/10494820.2021.2016864
Huang, X., Zou, D., Cheng, G., Chen, X., & Xie, H. (2023). Trends, Research Issues and
Applications of Artificial Intelligence in Language Education. Educational
Technology & Society, 26(1), 112–131.
https://doi.org/10.30191/ETS.202301_26(1).0009
Katsarou, E., Wild, F., Sougari, A.-M., & Chatzipanagiotou, P. (2023). A Systematic Review
of Voice-based Intelligent Virtual Agents in EFL Education. International Journal of
Emerging Technologies in Learning, 18(10), 65–85.
https://doi.org/10.3991/ijet.v18i10.37723
Lee, J. S., & Lee, K. (2020). Role of L2 Motivational Self System on Willingness to
Communicate of Korean EFL University and Secondary Students. Journal of
Psycholinguist Research, 49, 147–161. https://doi.org/10.1007/s10936-019-09675-6
Moussalli, S., & Cardoso, W. (2020). Intelligent personal assistants: Can they understand and
be understood by accented L2 learners? Computer Assisted Language Learning,
33(8), 865–890. https://doi.org/10.1080/09588221.2019.1595664
Tai, T. Y. (2022). Effects of intelligent personal assistants on EFL learners’ oral proficiency
outside the classroom. Computer Assisted Language Learning.
https://doi.org/10.1080/09588221.2022.2075013
Tai, T. Y., & Chen, H. H. J. (2020). The impact of Google Assistant on adolescent EFL
learners’ willingness to communicate. Interactive Learning Environments.
https://doi.org/10.1080/10494820.2020.1841801
Thrasher, T., Kaplan-Rakowski, R., Chun, D., Sadler, R. (2023). Virtual reality: “Awesome”,
“OK”, or “Not so good” for language learning? In B. Bédi, Y. Choubsaz, K.
Friðriksdóttir, A. Gimeno-Sanz, S. Björg Vilhjálmsdóttir. & S. Zahova (Eds.), CALL
for all Languages - EUROCALL 2023 Short Papers.
https://doi.org/10.4995/EuroCALL2023.2023.16948
Wang, X., Pang, H., Wallace, M. P., Wang, Q., & Chen, W. (2022). Learners’ perceived AI
presences in AI-supported language learning: A study of AI as a humanized agent
from community of inquiry. Computer Assisted Language Learning.
https://doi.org/10.1080/09588221.2022.2056203
Contact email: thrasher.tricia@gmail.com
