Pedagogical Agents in Educational VR: An in the Wild Study

Abstract

Pedagogical agents are theorized to increase humans’ effort to
understand computerized instructions. Despite the pedagogical
promises of VR, the usefulness of pedagogical agents in VR remains
uncertain. Based on this gap, and inspired by global efforts
to advance remote learning during the COVID-19 pandemic, we
conducted an educational VR study in-the-wild (𝑁 = 161). With a
2 × 2 + 1 between subjects design, we manipulated the appearance
and behavior of a virtual museum guide in an exhibition about
viruses. Factual and conceptual learning outcomes as well as subjective learning experience measures were collected. In general,
participants reported high enjoyment and had significant knowledge
acquisition. We found that the agent’s appearance and behavior
impacted factual knowledge gain. We also report an interaction
effect between behavioral and visual realism for conceptual
knowledge gain. Our findings nuance classical multimedia learning
theories and provide directions for employing agents in immersive
learning environments.

Full text

Pedagogical Agents in Educational VR: An in the Wild Study
Gustav Bøg Petersen
gbp@psy.ku.dk
Department of Psychology
University of Copenhagen
Copenhagen, Denmark
Aske Mottelson
amot@psy.ku.dk
Department of Psychology
University of Copenhagen
Copenhagen, Denmark
Guido Makransky
gm@psy.ku.dk
Department of Psychology
University of Copenhagen
Copenhagen, Denmark
ABSTRACT
Pedagogical agents are theorized to increase humans’ eort to
understand computerized instructions. Despite the pedagogical
promises of VR, the usefulness of pedagogical agents in VR re-
mains uncertain. Based on this gap, and inspired by global eorts
to advance remote learning during the COVID-19 pandemic, we
conducted an educational VR study in-the-wild (
𝑁=
161). With a
2
×
2
+
1between subjects design, we manipulated the appearance
and behavior of a virtual museum guide in an exhibition about
viruses. Factual and conceptual learning outcomes as well as sub-
jective learning experience measures were collected. In general,
participants reported high enjoyment and had signicant knowl-
edge acquisition. We found that the agent’s appearance and behav-
ior impacted factual knowledge gain. We also report an interac-
tion eect between behavioral and visual realism for conceptual
knowledge gain. Our ndings nuance classical multimedia learning
theories and provide directions for employing agents in immersive
learning environments.
CCS CONCEPTS
•Human-centered computing →Virtual reality
; Empirical
studies in HCI;
•Applied computing →
Interactive learning envi-
ronments.
KEYWORDS
Immersive Virtual Reality; Educational Technology; Learning; Cog-
nitive Load, Pedagogical Agents
ACM Reference Format:
Gustav Bøg Petersen, Aske Mottelson, and Guido Makransky. 2021. Peda-
gogical Agents in Educational VR: An in the Wild Study. In CHI Conference
on Human Factors in Computing Systems (CHI ’21), May 8–13, 2021, Yoko-
hama, Japan. ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/
3411764.3445760
1 INTRODUCTION
Imagine visiting your favorite museum in the connes of your own
home through the technology of virtual reality (VR); not having to
stand in line to explore exhilarating exhibitions, and at the same
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for prot or commercial advantage and that copies bear this notice and the full citation
on the rst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specic permission and/or a
fee. Request permissions from permissions@acm.org.
CHI ’21, May 8–13, 2021, Yokohama, Japan
©2021 Association for Computing Machinery.
ACM ISBN 978-1-4503-8096-6/21/05. . . $15.00
https://doi.org/10.1145/3411764.3445760
time having your own private museum guide at hand. The goal of
this study is to examine the role of a virtual museum guide (hence-
forth, a pedagogical agent) and its design in producing learning
during a tour in an educational VR museum.
VR has recently received increased attention as an educational
tool [
15
]. Theoretically, the contents of VR experiences are only
bounded by the limits of our imagination [
46
]. In practice, however,
educational VR often takes the shape of simulations or depictions of
reality, and VR is typically combined with face-to-face instruction
when used in the classroom (e.g., [
42
]). Such use of VR is in line
with the blended learning systems approach [11].
Blended learning systems are in certain situations not ideal, be-
cause of cost or practicalities of mixing digital and classical learning.
There are situations where real-life circumstances force us to rely
solely on remote technology for instruction. This, for instance, is
evident during the coronavirus disease 2019 (COVID-19) pandemic,
where lock down has forced many teachers and students to tran-
sition into a remote learning style [
7
]. New approaches to remote
teaching, including use of pedagogical agents and VR for remote
learning, are in such situations desired.
Unlike avatars, that are controlled by humans [
9
], pedagogical
agents are anthropomorphous computer-controlled virtual charac-
ters that are used in online learning environments to serve various
instructional goals [
49
]. The purpose of using pedagogical agents
is to mimic the social processes that usually take place in real-life
teaching. According to multimedia learning theories, people are
inclined to treat computerized agents as social partners if they ex-
hibit social cues [
33
]. This, in turn, should motivate the learner
to make sense of the presented material [
33
]. However, there is
disagreement with regards to the usefulness of pedagogical agents.
Opponents claim that the visual appearance of a pedagogical agent
adds unnecessary distraction to the learning experience [
45
]. This
view is consistent with the notion that the human working memory
is limited and that embellished materials therefore impede learn-
ing [
18
]. For VR, where it is possible to render pedagogical agents in
realistic 3D with higher behavioral realism compared to traditional
media, their eect on learning is even less supported.
Inspired by the eorts to reorganize teaching into online formats
during the COVID-19 pandemic, we created a virtual museum on
the topic of viral diseases. The museum was accessed by participants
‘in the wild’ using their own Oculus Quest head-mounted display
(HMD), an approach which has been proven feasible in previous VR
research [
23
,
37
,
47
]. We were interested in how the addition of a
social entity to the museum impacted participants’ learning outputs
and subjective ratings of the learning experience. Specically, we
investigated the role of the pedagogical agent’s appearance and
behavior. The hypothesis behind this was that a more realistic

CHI ’21, May 8–13, 2021, Yokohama, Japan Petersen et al.
pedagogical agent would induce a stronger sense of interacting
with a social partner, and in turn enhance learning.
Our results show a large eect on learning (Cohen’s
𝑑=
1
.
4).
They also show that participants enjoyed the learning (
𝑀=
3
.
9
/
5).
We report that the presence of a pedagogical agent diminished fac-
tual knowledge acquisition; yet, for learning conceptual knowledge,
an interaction eect between visual and behavioral realism shows
that the presence of pedagogical agents in educational VR might
be worthwhile.
2 BACKGROUND AND RELATED WORK
2.1 VR in Education
Recent work in educational psychology and HCI emphasize the
potentials of learning in immersive environments (e.g., [
3
,
20
,
29
,
31
,
42
]); these works together show how instructional and visual design
of virtual environments are imperative for learning outcomes.
Jensen and Konradsen [
15
] recently conducted a review of re-
search on education and training with the current generation of
HMDs. They argue that VR does not automatically cause learning
but rather provides a possibility for accessing simulations that might
induce learning. Furthermore, with regard to cognitive skills acqui-
sition, immersive experiences risk overwhelming learners unless
they are designed specically for the purpose and not overly inter-
active [
15
]. According to Jensen and Konradsen [
15
], the biggest
barriers to adopting VR in education and training are a lack of con-
tent as well as the technical skills necessary, which challenge many
instructors. Radianti et al. [
43
] similarly reviewed research on VR in
higher education, showing how VR is used in many dierent elds,
but for teaching engineering and computer science in particular.
Furthermore, VR is used for teaching various types of knowledge,
including procedural, practical, and declarative knowledge [43].
Studies that compare the relative eectiveness of media in pro-
moting educational outcomes can be used to illuminate the advan-
tages of VR in education compared to traditional media. Findings
show that VR is an eective medium for promoting quality expe-
riences, yet with mixed results for objective learning outcomes
compared to traditional media [25, 29, 35].
Comparisons of the learning eects of immersive and non-immersive
learning environments, respectively, show that immersion is asso-
ciated with higher self-ecacy, enjoyment, and interest [
28
,
35
].
Parong and Mayer [
39
] compared the eects of administering a
biology lesson in VR or as a slideshow on a PC. They found that
the VR group reported signicantly higher ratings of motivation,
interest, engagement, and aect than the group who used a PC.
The VR group, however, scored worse on a post-test of factual
knowledge [
39
]. Similarly, Makransky et al. [
29
] found that learn-
ers gained more knowledge from a lesson when the material was
presented via a PC than via VR. Parong and Mayer [
40
] showed that
lower retention scores when learning via VR, as opposed to desktop,
is related to extraneous cognitive load. These ndings indicate that
VR may tax the cognitive resources of learners heavily. Careful
attention to the instructional design of educational VR lessons is
therefore necessary.
2.2 Social agency theory
Social agency theory is a frequently used theoretical framework
from multimedia learning that explains the use of pedagogical
agents [
32
]. According to social agency theory, social cues in mul-
timedia lessons can prime a feeling of social presence in learners,
which leads to deeper cognitive processing and more learning [
33
].
Social presence refers to a psychological state in which virtual so-
cial actors are experienced as actual social actors in either sensory
or non-sensory ways [
22
]. Social agency theory states that people
are attentive to social cues when interacting with computerized
agents, and that these may induce a feeling of interacting with
another social being. This will activate social rules such as the co-
operation principle, meaning that the learner will try to make sense
of the instructional message; consequently, with a social agent, the
learner will make a deeper eort to understand and process the
computerized instructional message [33].
According to Mayer [
33
], several kinds of social cues can in-
duce a feeling of social presence in the learner. Two such cues are
specically relevant to educational VR: image cues (i.e., displaying a
pedagogical agent who narrates the material) and embodiment cues
(i.e., making the pedagogical agent display human-like behavior
such as gesturing, movement, eye contact, etc.). Taken together, em-
bodied pedagogical agents should, theoretically, give rise to social
presence and, therefore, deeper processing and better learning [
33
].
Pedagogical agents have also been criticized for adding unnecessary
complexity to learning environments. According to this view, the
visual presence of a pedagogical agent is merely a seductive detail,
and what really matters is the narration it provides [45].
In reviewing the eect of adding image and embodiment cues to
multimedia lessons, Mayer [
33
] concluded that there is moderate
evidence that embodiment cues improve learning (
𝑑=
0
.
36), but
less support for the eect of adding a speaker’s image to the screen
(
𝑑=
0
.
20). This can be formulated as the image principle: people
do not necessarily learn more deeply from a multimedia lesson
when the speaker’s image is on the screen compared to not on
the screen [
33
]; and the embodiment principle: people learn more
deeply when pedagogical agents display human-like gesturing,
movement, eye contact, and facial expressions [
33
]. Importantly,
however, the principles are based on ndings in classic multimedia
learning environments (such as desktop computers), often with non-
humanoid pedagogical agents or humanoid pedagogical agents of
low realism. Consequently, there is a need to revisit the the image
and embodiment principles with 3D and immersive media such
as VR [
34
]. This is relevant since VR is known to induce feelings
of social presence in learners [
30
] and thereby potentially deep
cognitive processing. Here follows a description of the current state
of research regarding agents in VR and other media, as well as
related work on social presence in virtual environments.
2.3 Agents in less immersive media
The literature contains a number of relevant HCI studies using
non HMD-based technology (e.g., projection systems or PC) that
can inform the design of pedagogical agents in VR; this section
provides a short description of a select few. Kartiko et al. [
16
] used
a projection system to test the impact of virtual actors’ visual com-
plexity (i.e., amount of visual information) on science learning, and

Pedagogical Agents in Educational VR CHI ’21, May 8–13, 2021, Yokohama, Japan
Figure 1: Participants in this study were immersed in a virtual museum with an exhibition on the topic of viruses (left). Partici-
pants could freely walk or teleport around the museum that would feature 3D objects relevant to the topic of the exhibition. A
museum tour guide would lead the participant and use white boards with animated presentations to explain the topic. Before
and after experiencing the museum, participants responded to a knowledge test on the subject of the exhibition (right).
found no eect of manipulation with regard to learning outcomes.
Wang et al. [
50
] explored the impact of dierent virtual agents
presented via augmented reality (AR) on a simple object nding
task. Although there were no eect of manipulation on completion
time, participants gazed more often at human-like agents compared
to non-human. Kim et al. [
17
] investigated users’ perceptions of
AR agents acting as lab assistants, and reported that participants
displayed most trust and social presence when an agent had a hu-
man body and was capable of speech, gestures, and locomotion.
During a problem-solving task on PC, Groom et al. [
12
] found that
participants liked a human virtual agent the most when it displayed
inconsistent behavioral realism compared to agents either consis-
tently low or high in behavioral realism (however, scores were
generally low). The same pattern was reported with regard to par-
ticipants’ levels of comfort. Kizilcec et al. [
19
] compared the impact
of presenting the instructor’s face strategically (i.e., when learners
should focus on spoken text independent of lecture slides) vs. con-
stantly during video instruction. Results indicated that strategic
presentation induced higher social presence relative to constant
presentation but no dierence in achieved course grade. Taking
a more practice-oriented approach, Veletsianos et al. [
48
] provide
a framework, ‘EnALI’, to enhance agent-learner interactions en-
compassing 15 guidelines, including suggestions regarding agent
characteristics such as designing them to communicate in a polite
and positive manner.
2.4 Pedagogical agents in VR
Only a few studies have been conducted concerning pedagogical
agents in VR and their eect on learning outcomes and experiences
(e.g., [
30
,
44
]). Typically, these do not include a no-agent condition,
making it dicult to be conclusive about the image and embod-
iment principles in VR; specically if the presence of the agent
has an eect on the learned subject. One study investigated the
eect of realism of agents on learning in VR exhibitions [
44
]. In-
terestingly, participants rated the absence of an agent as higher in
‘humanness’ compared to a realistic agent; the study, however, did
not nd a signicant eect on learning, possibly because of low
power. Makransky et al. [30] designed two pedagogical agents for
VR and used them to teach middle school students about laboratory
safety. A robot-like drone was intended to be more appealing to
boys; a young female scientist was posited to be more inviting to
girls. They demonstrated that boys learned better with the drone,
and that girls learned better with the female scientist. This suggests
that gender-specic design of pedagogical agents could be impor-
tant in educational VR. The general lack of empirical studies on
pedagogical agents in immersive learning environments makes it
dicult to reason about the impact of virtual agents’ appearance
and behavior on learning outcomes.
2.5 Social presence in virtual environments
Social agency theory proposes that social presence during multi-
media learning is a central mechanism that leads to deeper cogni-
tive processing and consequently better learning outcomes. Oh et
al. [
38
] recently conducted a systematic review of the predictors of
social presence in virtual environments. Their ndings emphasize
the importance of visual representation of virtual communication
partners. Specically, they found that (i) people feel higher levels
of social presence when a visual representation is available rather
than not; (ii) behavioral realism is a powerful predictor of social
presence; (iii) there are mixed results with regard to the eect of
delity and human-likeness on social presence (some studies show
an eect, others none); and (iv) that a ‘consistency eect’ possibly
exists – level of behavioral realism should be consistent with level
of visual delity to maintain high levels of social presence [
38
].
When applying these ndings to pedagogical agents and social

CHI ’21, May 8–13, 2021, Yokohama, Japan Petersen et al.
agency theory, there are some similarities. First, there is empiri-
cal evidence for the eect of image cues as people generally feel
higher social presence when the speaker is visible. Second, there is
empirical evidence for the inuence of embodiment cues, as people
generally feel higher social presence when the speaker displays
realistic behavior. Furthermore, Oh et al. [
38
] show the potential
for delity (i.e., visual realism) and consistency between behavioral
and visual realism in pedagogical agents to be important sources
of social presence and thereby possibly learning.
3 EXPERIMENT
Based on social agency theory and research on social presence
in virtual environments, four dierent pedagogical agents for the
VR Museum were constructed. These agents varied by a combi-
nation of two levels of visual and behavioral realism, for a total
of four agent conditions. Additionally, a control condition with-
out a pedagogical agent (hence, only narration) was included. The
participants signed up for the experiment online, and installed the
experimental application on their Oculus Quest device. Self-report
inside the VR application collected variables related to learning (i.e.,
knowledge gain and enjoyment), in addition to variables related to
the pedagogical agent (i.e., humanness, attractiveness, and social
presence). Lastly, free text feedback was collected after study com-
pletion in participants’ web browser. See Table 1 for a full list of
the included variables. All procedures performed during the study
were approved by the institutional ethical committee.
3.1 Preregistration: Hypotheses and analyses
We preregistered the experimental study alongside hypotheses,
study plan, and a statistical analyses plan (see https://osf.io/7wsya).
The data collection, study design, and statistical analyses followed
the preregistration, only with a minor deviation, as we, due to an
unexpected high participation interest collected slightly more data
than intended (162 participants instead of 150).
We preregistered the below ve hypotheses. In summary, these
speculate that pedagogical agents, and their realism, cause higher
social presence which will lead to more learning.
H1
Participants in conditions with pedagogical agents of high
visual realism, compared to low visual realism, will report
higher social presence.
H2
Participants in conditions with pedagogical agents of high
behavioral realism, compared to low behavioral realism, will
report higher social presence.
H3
Participants who report higher social presence will have
higher knowledge acquisition.
H4
Learning with a virtual pedagogical agent leads to higher
knowledge acquisition compared to only learning with a
voice.
H5
An interaction eect between visual and behavioral real-
ism of the pedagogical agent exists, such that consistency
(high/high or low/low) leads to more learning than inconsis-
tency (high/low or low/high).
3.2 Participants
Following recommendations on conducting unsupervised VR stud-
ies [
23
,
37
], participants were recruited to install our experimental
application onto their own devices, and conduct the study at their
discretion. A total of 162 participants, recruited on social media,
participated in the experiment using their own VR headset over the
course of 11 days. Most of the participants found our advertisement
on Reddit (132), but some found it on Facebook (11) or Twitter
(6). Participants were reimbursed with a gift certicate worth $15
USD (or the equivalent in their preferred currency). All of the
demographics answers are nominal as participants answered the
questions within VR by pointing (see Figure 1, right). Participants
were mostly male (134 male, 24 female, 4 non-binary). Roughly half
of the participants were between 18-29 (88), the rest were: 30-39
(41), 40-49 (22), 50-59 (10), and 60+ (3). Based on IP, participants
were identied to be located in 23 dierent countries, among the
most common: United States (78), United Kingdom (22), Canada
(11), Dominican Republic (11), and Mexico (7). The participants’
educational level ranged from ‘High school or less’ (73), ‘Bachelor’
(57), ‘Master’ (28), and ‘PhD’ (6). The resulting sample is, as the
above shows, rather diverse. However, the majority were from a
cohort with expert VR familiarity. This follows from limiting partic-
ipation to only participants who own an Oculus Quest themselves,
and who have the ability to install custom applications onto their
device. This is also evident from self reports of VR experience, as
the majority of participants had extensive VR experience, having
been immersed more than 50 times (94); the remaining had mostly
some experience (10-50 times,
𝑁=
35), or little experience (1-10
times, 𝑁=29).
One participant was excluded for taking too long, as dened in
the preregistration (
𝑀+
3
𝑆𝐷
). Eight participants were recorded
with a negative knowledge acquisition; these were kept in the
sample as an exclusion criteria based on learning outcomes was not
established prior to data collection, and because of the relatively
limited amount of participants who did not learn. The analyses
presented are therefore conducted on 161 participants.
𝜒2
tests were conducted to assess the equivalence of conditions
on demographic variables. These were all non-signicant: age (
𝑝=
.
93), gender (
𝑝=.
60), education (
𝑝=.
27), and English prociency
(
𝑝=.
92). Hence, the assigned groups did not signicantly dier
based on demography.
3.3 Apparatus
The virtual environment was developed using Unity 2020. The
application was targeted Oculus Quest only. The environment was
an exhibition hall equipped with animated 3D models related to
the topic of viruses (see Figure 1, left). Most of the 3D models were
found on the Unity Asset store. The pedagogical agent was taken
from the Microsoft Rocketbox repository [
10
]. An American female
voice actor was employed for recording the manuscript. Ambient
museum background sounds were present during the simulation.
3.4 Design
A2
×
2between subjects design was employed, with an additional
control group for a total of ve conditions. Condition was assigned
randomly at run time on the device. The independent variables
manipulated were (i) behavioral realism, with the levels high and
low and (ii) visual realism, also with the levels high and low. For
high behavioral realism, the pedagogical agent featured gesturing,

Pedagogical Agents in Educational VR CHI ’21, May 8–13, 2021, Yokohama, Japan
Figure 2: Experimental design of the study. High behavioral realism entailed eye contact, gesturing, lip sync, and natural
movements (left). High visual realism entailed rendering the agent as a human (top) rather than in monochrome (bottom).
The control group experienced the simulation without a pedagogical agent (far right).
eye contact, idle animations, speech and lip synchronization, and
movement by walking (see Figure 2 left). Conversely, low behavioral
realism entailed neither of these, and instead featured a static agent
that would move by gliding over the oor (see Figure 2, right).
For high visual realism the pedagogical agent would look like a
human female museum tour guide (see Figure 2, top); for low visual
realism the same humanoid agent employed a black monochrome
mesh (see Figure 2, bottom). The control condition did not feature a
pedagogical agent, yet with the narration intact. This experimental
design allowed an investigation of the importance of both behavior
and appearance of pedagogical agents in virtual learning environ-
ments, also, it made comparisons between having an agent and
no agent possible. This way, the design enabled verication of the
image principle (concerning the appearance vs. absence of pedagog-
ical agents), the embodiment principle (concerning the presence
vs. absence of human-like behavior in pedagogical agents), as well
as potential new principles derived from Oh et al. [
38
] concern-
ing the eect of high vs. low visual realism in pedagogical agents
and, nally, the eect of consistency vs. inconsistency between
behavioral and visual realism in pedagogical agents. On a broader
level, the experimental design enabled an assessment of the two
opposing views in the eld: that pedagogical agents facilitate vs.
impede learning.
3.5 Dependent measures
Eight variables were measured from a total of 40 questions; two of
these (factual and conceptual knowledge [2]) were objective ques-
tions about the learning topic. Three variables relating to the inter-
action with the pedagogical agent were measured. Two measures,
humanness and attractiveness, were from Ho and MacDorman’s
measures of the Uncanny Valley Eect [
14
] (‘eeriness’ was omitted
to reduce the length of the within-VR questionnaire). Social pres-
ence [
27
] was included, which measures the subjective experience
of being present with a ‘real’ person. Additionally, enjoyment and
cognitive load were measured. Validated subjective scales were
employed for all psychological variables. The knowledge questions
were administered, in the same order, both before and after the
study to study pre-to-post changes on learning. The subjective mea-
sures were only administered after the study. All questions were
answered on a virtual screen within the VR application by pointing

CHI ’21, May 8–13, 2021, Yokohama, Japan Petersen et al.
Variable Category Questions Type Min/max Reference
Factual knowledge Learning outcome 10 Multiple choice 0-10 [2]
Conceptual knowledge Learning outcome 10 Multiple choice 0-10 [2]
Perceived enjoyment Experience 3 5-point Likert 1-5 [26]
Perceived humanness Uncanny valley 5 Semantic dierential 1-5 [14]
Attractiveness Uncanny valley 5 Semantic dierential 1-5 [14]
Social presence Pedagogical agent 5 5-point Likert 1-5 [27]
Intrinsic cognitive load Cognitive load 1 5-point Likert 1-5 [5]
Extraneous cognitive load Cognitive load 1 5-point Likert 1-5 [5]
Table 1: Dependent measures used in the study.
and pulling the trigger on the controller (see Figure 1, right). For
each knowledge question four possible answers were provided.
The collected variables were analyzed using analysis of variance
(ANOVA), specically the results presented are computed using the
Rfunction car::Anova.
3.6 Developing the learning material and
outcome test
To underline the potentials of using home VR as a commodity
educational tool, especially during the global health crisis, a virtual
museum exhibition about viruses was chosen. In addition to a
brief introduction to general virology, the exhibition progressed
as a learning tour through three viral diseases: measles, Zika virus
disease, and COVID-19.
The narration that accompanied the exhibition was developed
with inspiration from a national biology teaching repository about
epidemics and pandemics targeted 13-15 year-olds, as well as other
relevant information sources such as the World Health Organiza-
tion. The target group for such simulations is therefore potentially
large. The environment features slides on virtual screens to supple-
ment the narrations with relevant visuals such as a depiction of a
baby suering from microcephaly when learning about Zika virus
disease.
To have a direct measure of participants’ knowledge acquisition
as a result of experiencing the simulation, a multiple choice test
was developed, that contained questions about the information pre-
sented during the simulation. The test was developed with experts
in educational psychology and psychometrics, and measured both
factual and conceptual knowledge; that is, cognitive objectives of
recalling and understanding, respectively [
2
]. The initial version of
the test had a total of 10 questions. To estimate the diculty of the
test, 112 participants were recruited on Amazon Mechanical Turk
(AMT) to conduct the test without any preparation (10 minutes; 1$
pay). The resulting median score was 7 out of 10, with roughly 10%
of the participants answering all questions correctly. As a result of
this test, the number and the diculty of questions were increased.
A next iteration of the test had 20 questions, and was also tested
on AMT. This time, with 75 participants, the median score was
11/20. In this iteration, participants scored between 6 and 15 points,
which attested that it was neither too easy or too dicult. The
nal learning outcome test therefore held 20 questions. It included
ten questions on factual knowledge, such as How many deaths has
measles vaccination prevented?, and ten questions on conceptual
knowledge, such as How do vaccines help the body develop immunity
to diseases?.
3.7 Procedure
For an overview of the study procedure, please consult Figure 3.
Participants signed up to our study using an online survey. Upon
giving informed consent to data collection and study participation,
a brief guide to installing our experimental application followed.
The open app store SideQuest
1
was utilized for the purpose of
easing the installation burden. After completing the installation
1http://sidequestvr.com
Install VR
application
Sign up
online
Confirm ID
Reim-
bursement
Pre test Intro-
duction
Measles
COVID-19 Post test
Subjective
measures
ID code
provided
Demo-
graphics
Zika
IN VIRTUAL REALITY
Figure 3: Visual overview of the study procedure. Yellow boxes denote PC-based user activity, gray are within-VR question-
naires, blue are core learning material, and the red shows meta activity. The entire procedure took about one hour to complete.

Pedagogical Agents in Educational VR CHI ’21, May 8–13, 2021, Yokohama, Japan
instructions, participants were instructed to launch the application,
take the headset on, and follow within-VR guidelines. As such,
guidelines were not provided before immersion. It was not possible
to skip parts of the experience.
The virtual environment began with a brief introduction of the
controls, the purpose, and the content. This was provided both
in audio and on information displays blended into the museum
environment. Participants could freely walk around or teleport
themselves by pointing and clicking ‘A’ on the controller. The exhi-
bition progressed as the participant followed the tour guide around
to the dierent displays constituting the core learning material. A
knowledge test was conducted, before beginning the guided tour.
A general introduction to the topic of virology was followed by
presentations of three viruses: Measles, Zika, and COVID-19. After
completing the museum tour, participants conducted an identical
knowledge test, in addition to a survey about subjective measures
and demographics. Upon completion, a unique code emerged on
a screen; this code had to be entered into the online form where
participants signed up to ensure valid participation (and to qualify
for reimbursement).
The mean duration of the immersion was 20.0 minutes (
𝑆𝐷 =
5.4).
4 RESULTS
Here, quantitative ndings are reported. They relate to the knowl-
edge acquisition and the eect of manipulations on subjective and
objective measures. A visual inspection of the collected variables
showed that data followed normal distributions (e.g., see Figure 4).
The analyses are therefore based on parametric tests. Furthermore,
the study collected subjective measures from scales that have pre-
viously been validated for parametric testing.
4.1 How much did they learn?
See Table 2 for an overview of dierences between pre- and post
scores. Out of a combined maximum of 20 points, the mean pre-
score was 11.0 (
𝑆𝐷 =
2
.
8). The mean pre-score for factual knowl-
edge was 3.5 (
𝑆𝐷 =
1
.
5); it was 7.6 (
𝑆𝐷 =
2
.
1
)
for conceptual
knowledge. For the post test, the combined mean score was 15.1
(
𝑆𝐷 =
2
.
8). The mean post score for factual knowledge was 6.2
(
𝑆𝐷 =
1
.
8); for conceptual knowledge the mean was 8.9 (
𝑆𝐷 =
1
.
5
)
.
That shows that participants, on average, increased their tests scores
with 4.0 points (
𝑆𝐷 =
2
.
9) after experiencing the virtual exhibition
(see Figure 4); this dierence was also signicant, shown with a
repeated measures ANOVA: 𝐹(1,160)=305.9, 𝑝 <.0001, 𝑑 =1.4.
Factual Conceptual
Pre test, M (SD) 3.5 (1.5) 7.6 (2.1)
Post test, M (SD) 6.2 (1.8) 8.9 (1.5)
Dierence +2.7 +1.3
Welch’s t-test, p <.0001 <.0001
Cohen’s d1.6 0.7
Table 2: Scores obtained in the pre- and post tests, respec-
tively.
0.00
0.05
0.10
0.15
0.20
0 5 10 15 20
Test score
Density
Post score
Pre score
Figure 4: Pre- to post test scores: participants performed bet-
ter on the knowledge tests after experiencing the virtual ex-
hibition.
4.2 The eect of agent on subjective measures
Figure 5 shows the mean reported scores on the subjective measures
relating to interaction with the agent, divided by experimental
manipulation.
For humanness (Figure 5, A), the absence of a pedagogical agent
yields comparable humanness to an agent with high behavioral
realism. Also, high behavioral realism resulted in higher humanness
compared to low behavioral realism;
𝐹(
1
,
156
)=
8
.
2
, 𝑝 =.
005. In
other words, behavior, but not appearance, of the virtual agent
aected whether participants experienced the agent as human.
For attractiveness (Figure 5, B), the experimental manipulations
of agent had less of an impact, and the eect of manipulation was
not signicant.
The high behavioral realism conditions showed signicantly
higher social presence (Figure 5, C);
𝐹(
1
,
156
)=
5
.
5
, 𝑝 =.
02. A
comparable dierence for appearance was not found.
In summary, our ndings suggest that behavior of pedagogical
agents (gesturing, eye contact, natural movements) impact sub-
jective social accounts of the agent, while appearance to a lesser
degree does. It should be noted that the absence of an agent results
in comparable, or even higher, reports of humanness and attractive-
ness (but not social presence). This counter intuitive nding, that
no agent leads to high reports of attractiveness and humanness,
was also reported by Rzayev et al. [44].
4.3 The eect of agent on learning
Figure 6 shows the mean dierence between pre- and post-test
knowledge scores for each condition. Two learning outcomes were
measured; factual and conceptual knowledge gain [
2
]. Factual knowl-
edge relates to recalling (e.g., numbers, places, years) while concep-
tual knowledge relates to understanding (e.g., explaining, connect-
ing, transferring).
Both the appearance and behavior of a pedagogical agent had an ef-
fect on factual knowledge gain (see Figure 6, A). One-way ANOVAs
showed signicant eects:
𝐹(
2
,
158
)=[
5
.
5; 3
.
8
], 𝑝 =[.
005;
.
03
]
.
Post hoc Tukey’s HSDs showed that high visual realism as well
as high behavioral realism signicantly diered with the control
condition. This shows that, for learning facts, the presence of a
pedagogical agent is not ideal; rather, the agent, and its visual and
behavioral delity impede factual retention.

CHI ’21, May 8–13, 2021, Yokohama, Japan Petersen et al.
1
2
3
4
5
Control Low appearance High appearance
Mean humanness
A
1
2
3
4
5
Control Low appearance High appearance
Mean attractiveness
B
1
2
3
4
5
Control Low appearance High appearance
Mean social presence
C
Control High behavior Low behavior
Figure 5: Means of subjective measures by experimental
manipulation: humanness (A), attractiveness (B), and social
presence (C). Error bars show 95% condence intervals.
For conceptual knowledge, a two-way ANOVA showed a sig-
nicant interaction eect between appearance and behavior (see
Figure 6, B);
𝐹(
1
,
155
)=
9
.
2
, 𝑝 =.
003. Oh et al. [
38
] reported a
‘consistency eect’; they state that, for learning, consistency of an
agent is preferred (i.e., that delity of behavioral and visual realism
ideally match). Our ndings contradict this nding, as we observe a
higher conceptual knowledge gain when behavior and appearance
are incongruent.
4.4 Enjoyment
Participants generally reported high enjoyment rates for the learn-
ing experience, with a mean score of
𝑀=
3
.
9
(𝑆𝐷 =
0
.
8
)
of a
maximum 5. These enjoyment rates were consistent across exper-
imental manipulations, see Figure 7. Together with the general
positive knowledge acquisition, this tells us, that the virtual mu-
seum was received positively as a new form of remote learning
during the global health crisis. This nding is consistent with previ-
ous research that associated VR with higher enjoyment compared
to less immersive media [24].
0
1
2
3
4
5
Control Low
appearance High
appearance
∆score, factual
A
0
1
2
3
4
5
Control Low
appearance High
appearance
∆score, conceptual
B
Control High behavior Low behavior
Figure 6: Mean dierences between pre- and post test
scores. For factual knowledge (A), absence of a pedagogi-
cal agent leads to higher pre- to post scores. For concep-
tual learning (B), we nd a signicant interaction eect be-
tween behavior and appearance, namely that incongruence
(low/high or low/high) leads to better learning than congru-
ence (high/high or low/low). Error bars show 95% CI.
4.5 Cognitive load
As previous ndings in educational VR suggest that specic in-
structional designs in VR learning environments lead to increased
cognitive load [
3
,
29
], a subjective measure for both intrinsic (di-
culty of subject) and extraneous (diculty of instruction) cognitive
load [5] were collected.
Participants reported comparable intrinsic and extraneous cogni-
tive load for all conditions. As such, medians for all ve conditions
for both cognitive load measures were 2 out of 5. There were no
signicant dierences between conditions. It should be noted that
cognitive load was assessed via single items. Use of full scales could
have provided further insights (e.g. [1]).

Pedagogical Agents in Educational VR CHI ’21, May 8–13, 2021, Yokohama, Japan
1
2
3
4
5
Control Low appearance High appearance
Mean enjoyment
Control High behavior Low behavior
Figure 7: Means of enjoyment ratings: participants consis-
tently experienced the virtual exhibition as enjoyable. Error
bars show 95% condence intervals.
4.6 Brief Summary of hypotheses and ndings
We preregistered ve hypotheses related to the realism of peda-
gogical agents, their eect on social presence, and in turn learning.
We hypothesized that high visual realism would lead to high social
presence (H1); that high behavioral realism would lead to high
social presence (H2); that high social presence would lead to better
learning (H3); that agents would be better than no agents for learn-
ing (H4); and that there would be an interaction eect between
visual and behavioral realism, where consistency would be better
than inconsistency for learning (H5). In relation to our ndings,
specically, only H2 was conrmed, namely that behavior of an
avatar signicantly impacts social presence. On the contrary, we did
not nd support for H1; that is, that appearance of an agent impacts
social presence. We did not nd a signicant correlation between
social presence and factual learning (rather slightly negative, Pear-
son’s
𝜌=−
0
.
11
, 𝑝 =.
15). Yet, for conceptual learning we do nd
it signicantly correlated to social presence (
𝜌=
0
.
25
, 𝑝 =.
001).
Consequently, H3 has a more nuanced answer. Importantly, for H4,
we nd an opposite eect for factual learning. For conceptual learn-
ing we did nd an interaction eect, but rather the reverse than
hypothesized in H5. Consequently we nd partial support for the
opposite eect for H4 (for factual learning) and H5 (for conceptual
learning).
5 DISCUSSION
We conducted a VR experiment ‘in the wild’ during the COVID-19
pandemic. In general, our ndings show that unsupervised, remote
learning in VR is feasible as all participants enjoyed the experience
and improved on a knowledge test. Furthermore, we show that the
design of pedagogical agents in educational VR impacts learning
depending on the type of learning considered. Including a pedagog-
ical agent leads to lower factual knowledge acquisition compared
to only including a narration. Looking at conceptual information
acquisition, a pedagogical agent may aid learning. These ndings
expand classical multimedia learning theory in the context of VR.
Our nding that pedagogical agents are useful for learning about
concepts but not facts could be explained by the diering nature
of the two types of information in combination with the human
capacity for selectively attending to certain stimuli [
8
]. During the
lesson, factual information, such as specic dates and numbers,
was presented very quickly and therefore imposed large demands
on attention at specic moments in time. In contrast, much of the
conceptual information had broader explanations and therefore al-
lowed short diversions in attention. Thus, the addition of a detailed
agent would specically have a negative eect on factual learning
by stealing attention at critical moments. Although we did not use
eye-tracking, other research corroborates people’s tendency to gaze
at human agents [
50
]. This echoes the issue raised by Veletsianos et
al. [
48
] that pedagogical agents may be mesmerizing and misdirect
attention from the task.
Our participants were expert VR users, hence a novelty eect
most likely did not interfere with the results. The novelty eect
refers to a heightened motivation to use something simply on ac-
count of its newness [
21
]. Novelty may confound the results of
media studies as the increased eort and attention could result in
achievement gains that would not occur if the learner was familiar
with the medium [
6
]. In that light, and in combination with the
relatively high participation count, our results are a reliable source
of evidence on the ecacy of pedagogical agents in immersive
learning environments.
5.1 Theoretical implications
The ndings of this study has a number of implications for social
agency theory and its derived learning principles in the context of
educational VR.
5.1.1 The image principle. The image principle suggests that using
visible pedagogical agents has a small eect on learning compared
to only using narration. As some previous studies found negligi-
ble or even negative eects of presenting an image of the speaker,
however, it led Mayer to conclude that people do not necessar-
ily learn more from lessons when the speaker’s image is on the
screen compared to when it is not [
33
]. Importantly, the theory is
based on relatively dated empirical evidence, some of which date
back 20 years ago, where learning environments and pedagogical
agents were less sophisticated. We examined the image principle
with highly sophisticated pedagogical agents rendered using state
of the art consumer VR technology. Our ndings show that for
factual learning, including a pedagogical agent of high visual or
behavioral realism leads to less learning compared to using an ‘in-
visible’ speaker. Realistically looking agents presumably distract
the learner, yet, we did not record a change in subjectively mea-
sured cognitive load. In contrast, learning about concepts was not
hampered by the inclusion of pedagogical agents.
To sum up, our ndings suggest a renement of the image princi-
ple when applied to educational VR. When compared to only a nar-
ration, pedagogical agents do not lead to higher factual knowledge
gain. On the contrary, realistic pedagogical agents may actually
hamper learning of factual information. This was not the case when
learning about conceptual information.
5.1.2 The embodiment principle. The embodiment principle fo-
cuses on the absence vs. presence of behavioral cues in pedagogical
agents. Previous research points to the benets of embodiment,
which led Mayer to conclude that when pedagogical agents display

CHI ’21, May 8–13, 2021, Yokohama, Japan Petersen et al.
human-like gesturing, movement, facial expressions, etc. as op-
posed to appearing static, it leads to better learning. In the present
study we did not nd a positive eect of behavioral realism on
factual knowledge gain. In fact, participants who learned from
behaviorally realistic pedagogical agents (i.e., agents exhibiting
gesturing, eye contact, speech and lip synchronization) increased
their scores slightly less on the factual knowledge test than partici-
pants who learned from static pedagogical agents (although this
warrants further investigation). Consequently, our ndings do not
corroborate the embodiment principle when learning about facts.
When learning about concepts, however, our ndings indicate that
the embodiment principle exists if the agents are of low but not
high visual realism. In the latter scenario, a reversed embodiment
principle is actually found. This is an important addition to the
embodiment principle in educational VR.
5.1.3 Visual realism and consistency. Oh et al. [
38
] reviewed deter-
minants of social presence in virtual environments and found that
agents’ visual realism and consistency in appearance and behavior
could be important sources of social presence. We built on these
ndings and tested if visual realism and consistency inuenced
learning of factual and conceptual information. Our results indicate
that visual realism impacts factual learning negatively. One possible
explanation for this could be the uncanny valley eect, which refers
to the relation between the human-likeness of an entity and the per-
ceiver’s anity for it [
36
]. The theory posits that anity increases
as a function of the human-likeness of articial humans until it
reaches a valley where anity suddenly drops. This corresponds to
a point where there is a relatively high degree of human-likeness in
an entity combined with evidence that it is articial, and this is ac-
companied by a creepy sensation [
36
]. The visually realistic agents
might have caused a creepy sensation in the participants, lowering
their motivation to understand the learning material. This would be
consistent with the feedback reported by some of the participants
learning with visually realistic agents: “You could replace the guide
with a robot. A clearly non-human robot guide wouldn’t be so o
putting [sic]” or “the guide looks creepy”. In terms of consistency, our
results, again, indicate an eect in the opposite direction of what
we had hypothesized. There was a signicant interaction eect
between visual and behavioral realism for conceptual knowledge
gain; knowledge gain was higher when these were incongruent.
5.1.4 Social presence as a learning mechanism. Social presence
during multimedia learning is posited to be an important construct
that leads to better learning outcomes [
33
]. Consistent with social
agency theory, we found that behavioral realism had a signicant
eect on social presence. However, an increase in subjective social
presence was not unequivocally associated with more learning.
This shows that social presence does not necessarily lead to more
learning, and that other constructs than those provided by social
agency theory are important for learning with virtual agents.
5.2 Limitations and future research directions
Pedagogical agents can take on a variety of instructor roles dur-
ing learning [
13
]. Similar to Baylor and Kim [
4
], the agents in the
present study performed the role of an expert as their primary
function was to provide accurate and concise information. How-
ever, agents can also inhabit other roles, such as motivators whose
primary function is to provide encouragement [
4
]. Future research
should investigate learning with pedagogical agents inhabiting
dierent roles.
The participants in this study primarily consisted of expert VR
users. This was an advantage in terms of limiting novelty eects, but
future research should investigate the ecacy of remote learning
with pedagogical agents in a non-expert sample.
We investigated factual and conceptual learning as pre- to post-
test changes on a multiple choice question test administered in
VR before and after the lesson. However, we did not investigate
transfer of learning (i.e., when learning in one context improves
performance in another context [
41
]). Transfer is a key goal in edu-
cation, and future research should therefore consider investigating
transfer eects of learning with agents in VR.
Another limitation concerns the relevancy of the pedagogical
agents and their behavior in terms of teaching the content of the
VR simulation. For instance, it could be argued that an agent would
be more relevant for teaching specic procedures compared to
teaching facts, as its behavior could then be tied more easily to the
learning material (e.g., if demonstrating how to perform a certain
procedure). Theoretically, this would map onto classical psycho-
logical theories of model learning as put forward by scholars such
as Albert Bandura. Future research could therefore examine pro-
cedural learning from pedagogical agents in VR. One promising
direction could be to examine whether watching an agent perform
a procedure produces stronger procedural learning compared to
hearing about it.
While out-of-lab VR experimentation allows for a larger and
more heterogeneous sample which gives higher statistical power
and ecological ndings at a lower cost [
37
], it comes with the cost of
reduced internal validity. Specically, we cannot control the size and
constraints of the participants’ surroundings or a strict adherence
to the study protocol. Qualitative ndings are also hard to collect
using this paradigm. Some of the aspects discussed in this paper,
concerning for instance creepiness of agents, might be benecial
to investigate further using traditional laboratory protocols.
Based on inquiry by some of the participants in regards to invit-
ing lock downed family members and friends to participate in the
study, we allowed multiple participation from the same IP (a total of
17 recurrent IP addresses were recorded). This resulted in a slightly
more diverse participant pool than otherwise expected. We note
that this makes it technically possible for a single participant to
complete the study multiple times, we however, do not suspect this
have inuenced the reported data.
Furthermore we collected free form text answers as part of the de-
brieng by asking for any feedback. Although many wrote lengthy
comments of mostly feature requests, the focus of the study was
quantitative, and an extensive analysis of qualitative data was there-
fore neither signicant nor the scope. If the aim is qualitative, a
recommendation for future online user studies is therefore to be
specic about any desired type of qualitative data and to formulate
open-ended questions accordingly.
The general low number of non-males in the study was a lim-
itation along with the fact that we did not manipulate the sex of

Pedagogical Agents in Educational VR CHI ’21, May 8–13, 2021, Yokohama, Japan
the agent. Participants’ gender did not reveal considerable learning
dierences at post-test: M 15.4, 15.0, and 16.0 (F/M/X).
6 CONCLUSION
Remote learning in the connes of your own home through VR
is an enjoyable and eective alternative to the real-life classroom,
especially during global health crises. Caution should be taken,
however, when it comes to incorporating a pedagogical agent into
the virtual learning experience. Our ndings expand upon classical
multimedia learning theory in the context of VR, and provide new
insights into the scholarly discussion about whether pedagogical
agents are facilitators of learning or merely unnecessary distrac-
tions; our ndings suggest that the answer depends on the type of
knowledge in question. Behavior of a pedagogical agent did in fact
increase social presence and humanness, yet the eects on learning
were mixed. If the material to be learned is factual, a pedagogical
agent may impede learning. Contrary, if the material to be learned
is conceptual, a pedagogical agent may be worthwhile.
7 ACKNOWLEDGEMENTS
We would like to thank Martin Kampmann for designing and de-
veloping the virtual environment used in the study. We would also
like to thank all the people who participated in the experiment.
REFERENCES
[1]
Martin S. Andersen and Guido Makransky. 2021. The validation and further
development of a multidimensional cognitive load scale for virtual environments.
Journal of Computer Assisted Learning 37, 1 (2021), 183–196. https://doi.org/10.
1111/jcal.12478
[2]
Lorin W. Anderson, David R. Krathwohl, Peter W. Airasian, Kathleen A. Cruik-
shank, Richard E. Mayer, Paul R. Pintrich, James Raths, and Merlin C. Wittrock.
2001. A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s
Taxonomy of Educational Objectives. Longman.
[3]
Sarune Baceviciute, Aske Mottelson, Thomas Terkildsen, and Guido Makransky.
2020. Investigating Representation of Text and Audio in Educational VR Using
Learning Outcomes and EEG. In Proceedings of the 2020 CHI Conference on Human
Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for
Computing Machinery, New York, NY, USA, 1–13. https://doi.org/3313831.
3376872
[4]
Amy L. Baylor and Yanghee Kim. 2004. Pedagogical agent design: The impact of
agent realism, gender, ethnicity, and instructional role. In International conference
on intelligent tutoring systems. Springer, 592–603.
[5]
Gabriele Cierniak, Katharina Scheiter, and Peter Gerjets. 2009. Explaining the
split-attention eect: Is the reduction of extraneous cognitive load accompanied
by an increase in germane cognitive load? Computers in Human Behavior 25, 2
(2009), 315–324. https://doi.org/10.1016/j.chb.2008.12.020
[6]
Richard E. Clark and Terrance G. Craig. 1992. Research and theory on multi-
media learning eects. In Interactive multimedia learning environments. Springer,
19–30.
[7]
Joseph Crawford, Kerryn Butler-Henderson, Jürgen Rudolph, Matthias Glowatz,
Rob Burton, Paola A. Magni, and Sophia Lam. 2020. COVID-19: 20 Countries’
Higher Education Intra-Period Digital Pedagogy Responses. Journal of Applied
Teaching and Learning (JALT) 3, 1 (2020). https://doi.org/10.37074/jalt.2020.3.1.7
[8]
Peter Dayan, Sham Kakade, and P. Read Montague. 2000. Learning and selective
attention. Nature neuroscience 3, 11 (2000), 1218–1223. https://doi.org/10.37074/
jalt.2020.3.1.7
[9]
Jesse Fox, Sun Joo (Grace) Ahn, Joris H. Janssen, Leo Yeykelis, Kathryn Y. Segovia,
and Jeremy N. Bailenson. 2015. Avatars Versus Agents: A Meta-Analysis Quanti-
fying the Eect of Agency on Social Inuence. Human–Computer Interaction 30,
5 (2015), 401–432. https://doi.org/10.1080/07370024.2014.921494
[10]
Mar Gonzalez-Franco, Eyal Ofek, Ye Pan, Angus Antley, Anthony Steed, Bernhard
Spanlang, Antonella Maselli, Domna Banakou, Nuria Pelechano, Sergio Orts
Escolano, Veronica Orvahlo, Laura Trutoiu, Markus Wojcik, Maria V. Sanchez-
Vives, Jeremy Bailenson, Mel Slater, and Jaron Lanier. 2020. Importance of
rigging for procedural avatars. Microsoft Rocketbox a public library. https:
//github.com/microsoft/Microsoft-Rocketbox. [Online; accessed April 1, 2020].
[11]
Charles R. Graham. 2006. Blended learning systems: Denition, current trends, and
future directions. John Wiley & Sons.
[12]
Victoria Groom, Cliord Nass, Tina Chen, Alexia Nielsen, James K. Scarborough,
and Erica Robles. 2009. Evaluating the eects of behavioral realism in embodied
agents. International Journal of Human-Computer Studies 67, 10 (2009), 842–849.
https://doi.org/10.1016/j.ijhcs.2009.07.001
[13]
Ste Heidig and Geraldine Clarebout. 2011. Do pedagogical agents make a
dierence to student motivation and learning? Educational Research Review 6, 1
(2011), 27–54. https://doi.org/10.1016/j.edurev.2010.07.004
[14]
Chin-Chang Ho and Karl MacDorman. 2017. Measuring the Uncanny Valley
Eect. International Journal of Social Robotics 9, 1 (2017), 129–139. https://doi.
org/10.1007/s12369-016- 0380-9
[15]
Lasse Jensen and Flemming Konradsen. 2018. A review of the use of virtual reality
head-mounted displays in education and training. Education and Information
Technologies 23, 4 (2018), 1515–1529. https://doi.org/10.1007/s10639-017- 9676-0
[16]
Iwan Kartiko, Manolya Kavakli, and Ken Cheng. 2010. Learning science in a vir-
tual reality application: The impacts of animated-virtual actors visual complexity.
Computers & Education 55, 2 (2010), 881–891. https://doi.org/10.1016/j.compedu.
2010.03.019
[17]
Kangsoo Kim, Luke Boelling, Steen Haesler, Jeremy N. Bailenson, Gerd Bruder,
and Gregory F. Welch. 2018. Does a Digital Assistant Need a Body? The Inuence
of Visual Embodiment and Social Behavior on the Perception of Intelligent Virtual
Agents in AR. In 2018 IEEE International Symposium on Mixed and Augmented
Reality (ISMAR). 105–114. https://doi.org/10.1109/ISMAR.2018.00039
[18]
Paul A. Kirschner. 2002. Cognitive load theory: Implications of cognitive load
theory on the design of learning. Learning and Instruction 12, 1 (2002), 1–10.
https://doi.org/10.1016/S0959-4752(01)00014- 7
[19]
René F. Kizilcec, Jeremy N. Bailenson, and Charles J. Gomez. 2015. The in-
structor’s face in video instruction: Evidence from two large-scale eld studies.
Journal of Educational Psychology 107, 3 (2015), 724–739. https://doi.org/10.1037/
edu0000013
[20]
Sara Klingenberg, Maria L. M. Jørgensen, Gert Dandanell, Karen Skriver,
Aske Mottelson , and Guido Makransky. 2020. Investigating the eect of teaching
as a generative learning strategy when learning through desktop and immersive
VR: A media and methods experiment. British Journal of Educational Technology
51, 6 (2020), 2115–2138. https://doi.org/10.1111/bjet.13029
[21]
Michael Koch, Kai von Luck, Jan Schwarzer, and Susanne Draheim. 2018. The
Novelty Eect in Large Display Deployments - Experiences and Lessons-Learned
for Evaluating Prototypes. In ECSCW 2018 Exploratory Papers. European Society
for Socially Embedded Technologies (EUSSET). https://dl.eusset.eu/handle/20.
500.12015/3115
[22]
Kwan Min Lee. 2006. Presence, Explicated. Communication Theory 14, 1 (2006),
27–50. https://doi.org/10.1111/j.1468-2885.2004.tb00302.x
[23]
Xiao Ma, Megan Cackett, Leslie Park, Eric Chien, and Mor Naaman. 2018. Web-
Based VR Experiments Powered by the Crowd. In Proceedings of the 2018 World
Wide Web Conference (Lyon, France) (WWW ’18). International World Wide Web
Conferences Steering Committee, Republic and Canton of Geneva, CHE, 33–43.
https://doi.org/10.1145/3178876.3186034
[24]
Guido Makransky, Niels Koch Andreasen, Sarune Baceviciute, and Richard E.
Mayer. 2020. Immersive Virtual Reality Increases Liking but Not Learning with
a Science Simulation and Generative Learning Strategies Promote Learning in
Immersive Virtual Reality. Journal of Educational Psychology (2020). https:
//doi.org/10.1037/edu0000473
[25]
Guido Makransky, Stefan Borre-Gude, and Richard E. Mayer. 2019. Motivational
and cognitive benets of training in immersive virtual reality based on multiple
assessments. Journal of Computer Assisted Learning 35, 6 (2019), 691–707. https:
//doi.org/10.1111/jcal.12375
[26]
Guido Makransky and Lau Lilleholt. 2018. A Structural Equation Modeling
Investigation of the Emotional Value of Immersive Virtual Reality in Education.
Educational Technology Research and Development (2018), 1141–1164. https:
//doi.org/10.1007/s11423-018- 9581-2
[27]
Guido Makransky, Lau Lilleholt, and Anders Aaby. 2017. Development and
validation of the Multimodal Presence Scale for virtual reality environments: A
conrmatory factor analysis and item response theory approach. Computers in
Human Behavior 72 (2017), 276–285. https://doi.org/10.1016/j.chb.2017.02.066
[28]
Guido Makransky, Gustav B. Petersen, and Sara Klingenberg. 2020. Can an
immersive virtual reality simulation increase students’ interest and career aspira-
tions in science? British Journal of Educational Technology 51, 6 (2020), 2079–2097.
https://doi.org/10.1111/bjet.12954
[29]
Guido Makransky, Thomas S. Terkildsen, and Richard E. Mayer. 2019. Adding
immersive virtual reality to a science lab simulation causes more presence but
less learning. Learning and Instruction 60 (2019), 225 – 236. https://doi.org/10.
1016/j.learninstruc.2017.12.007
[30]
Guido Makransky, Philip Wismer,and Richard E. Mayer. 2019. A gender matching
eect in learning with pedagogical agents in an immersive virtual reality science
simulation. Journal of Computer Assisted Learning 35, 3 (2019), 349–358. https:
//doi.org/10.1111/jcal.12335
[31]
David M. Markowitz, Rob Laha, Brian P. Perone, Roy D. Pea, and Jeremy N.
Bailenson. 2018. Immersive Virtual Reality Field Trips Facilitate Learning About
Climate Change. Frontiers in Psychology 9, 2364 (2018), 1–20. https://doi.org/10.

CHI ’21, May 8–13, 2021, Yokohama, Japan Petersen et al.
3389/fpsyg.2018.02364
[32]
Ati Suci Dian Martha and Harry B. Santoso. 2019. The Design and Impact of the
Pedagogical Agent: A Systematic Literature Review. Journal of Educators Online
16, 1 (2019). https://eric.ed.gov/?id=EJ1204376
[33]
Richard E. Mayer. 2014. Principles based on social cues in multimedia learning:
Personalization, voice, image, and embodiment principles. In The Cambridge
handbook of multimedia learning. Cambridge University Press, 345–370.
[34]
Richard E. Mayer. 2019. Thirty years of research on online learning. Applied
Cognitive Psychology 33, 2 (2019), 152–159. https://doi.org/10.1002/acp.3482
[35]
Oliver A. Meyer, Magnus K. Omdahl, and Guido Makransky. 2019. Investigating
the eect of pre-training when learning through immersive virtual reality and
video: A media and methods experiment. Computers & Education 140 (2019).
https://doi.org/10.1016/j.compedu.2019.103603
[36]
Masahiro Mori, Karl F. MacDorman, and Norri Kageki. 2012. The Uncanny
Valley [From the Field]. IEEE Robotics Automation Magazine 19, 2 (2012), 98–100.
https://doi.org/10.1109/MRA.2012.2192811
[37]
Aske Mottelson and Kasper Hornbæk. 2017. Virtual Reality Studies Outside the
Laboratory (VRST ’17). Association for Computing Machinery, New York, NY,
USA, 1–10. https://doi.org/10.1145/3139131.3139141
[38]
Catherine S. Oh, Jeremy N. Bailenson, and Gregory F. Welch. 2018. A Systematic
Review of Social Presence: Denition, Antecedents, and Implications. Frontiers
in Robotics and AI 5, 114 (2018), 1–35. https://doi.org/10.3389/frobt.2018.00114
[39]
Jocelyn Parong and Richard E. Mayer. 2018. Learning science in immersive
virtual reality. Journal of Educational Psychology 110, 6 (2018), 785–797. https:
//doi.org/10.1037/edu0000241
[40]
Jocelyn Parong and Richard E. Mayer. 2021. Cognitive and aective processes
for learning science in immersive virtual reality. Journal of Computer Assisted
Learning 37, 1 (2021), 226–241. https://doi.org/10.1111/jcal.12482
[41]
David N. Perkins and Gavriel Salomon. 1992. Transfer of learning. In International
encyclopedia of education. Pergamon Press.
[42] Gustav B. Petersen, Sara Klingenberg, Richard E. Mayer, and Guido Makransky.
2020. The virtual eld trip: Investigating how to optimize immersive virtual
learning in climate change education. British Journal of Educational Technology
51, 6 (2020), 2098–2114. https://doi.org/10.1111/bjet.12991
[43]
Jaziar Radianti, Tim A. Majchrzak, Jennifer Fromm, and Isabell Wohlgenannt.
2020. A systematic review of immersive virtual reality applications for higher
education: Design elements, lessons learned, and research agenda. Computers &
Education 147 (2020). https://doi.org/10.1016/j.compedu.2019.103778
[44]
Rufat Rzayev, Gürkan Karaman, Katrin Wolf, Niels Henze, and Valentin Schwind.
2019. The Eect of Presence and Appearance of Guides in Virtual Reality Ex-
hibitions. In Proceedings of Mensch Und Computer 2019 (Hamburg, Germany)
(MuC’19). Association for Computing Machinery, New York, NY, USA, 11–20.
https://doi.org/10.1145/3340764.3340802
[45]
Noah L. Schroeder, Olusola O. Adesope, and Rachel Barouch Gilbert. 2013. How
Eective are Pedagogical Agents for Learning? A Meta-Analytic Review. Journal
of Educational Computing Research 49, 1 (2013), 1–39. https://doi.org/10.2190/EC.
49.1.a
[46]
Mel Slater. 2009. Place Illusion and Plausibility Can Lead to Realistic Behaviour in
Immersive Virtual Environments. Philosophical transactions of the Royal Society
B364 (2009), 3549–3557. https://doi.org/10.1098/rstb.2009.0138
[47]
Anthony Steed, Sebastian Frlston, Maria Murcia Lopez, Jason Drummond, Ye Pan,
and David Swapp. 2016. An In the Wild Experiment on Presence and Embodiment
using Consumer Virtual Reality Equipment. IEEE Transactions on Visualization &
Computer Graphics 22, 04 (2016), 1406–1414. https://doi.org/10.1109/TVCG.2016.
2518135
[48]
George Veletsianos, Charles Miller, and Aaron Doering. 2009. Enali: A Research
and Design Framework for Virtual Characters and Pedagogical Agents. Journal
of Educational Computing Research 41, 2 (2009), 171–194. https://doi.org/10.2190/
EC.41.2.c
[49]
George Veletsianosand Gregory S. Russell. 2014. Pedagogical agents. In Handbook
of research on educational communications and technology. Springer, 759–769.
[50]
Isaac Wang, Jesse Smith, and Jaime Ruiz. 2019. Exploring Virtual Agents for
Augmented Reality. In Proceedings of the 2019 CHI Conference on Human Factors in
Computing Systems (Glasgow, Scotland UK) (CHI ’19). Association for Computing
Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300511