Content uploaded by Michael Holly
Author content
All content in this area was uploaded by Michael Holly on Mar 30, 2021
Content may be subject to copyright.
Holly, M., Pirker, J., Resch, S., Brettschuh, S., & Gütl, C. (2021). Designing VR Experiences – Expectations for Teaching
and Learning in VR. Educational Technology & Society, 24 (2), 107–119.
107
ISSN 1436-4522 (online) and 1176-3647 (print). This article of the journal of Educational Technology & Society is available under Creative Commons CC-BY-NC-ND
3.0 license (https://creativecommons.org/licenses/by-nc-nd/3.0/). For further queries, please contact Journal Editors at ets.editors@gmail.com.
Designing VR Experiences – Expectations for Teaching and Learning in VR
Michael Holly*, Johanna Pirker, Sebastian Resch, Sandra Brettschuh and Christian
Gütl
Graz University of Technology, Austria // michael.holly@tugraz.at // johanna.pirker@tugraz.at //
sebastian.resch@tugraz.at // sandra.brettschuh@tugraz.at // c.guetl@tugraz.at
*Corresponding author
ABSTRACT: Skills in science, technology, engineering, and mathematics (STEM) are increasingly in demand.
Theoretical knowledge and formulas alone are frequently not sufficient to understand complex phenomena.
Simulations are a valuable tool to support the conceptual understanding by visualizing invisible processes. The
constant interaction with the learning material is an essential factor when learning with simulations and virtual
worlds. Virtual reality (VR) technologies enable interaction with the virtual environment with a high intensity of
immersion. Maroon is a VR platform for teaching physics and has been in development for over five years.
Previous results with Maroon have already demonstrated the potential of virtual reality for learners and teachers,
but also highlighted a list of potential challenges in terms of VR experience design, usability, and pedagogical
concepts. Over the past six months, we have conducted user studies with a total of 85 participants, both student
teachers (n = 26) and pupils (n = 59) at high schools and teacher training institutions. In this paper, we want to
facilitate the difficult task of designing educational VR platforms by describing the expectations of educators and
pupils.
Keywords: STEM education, Virtual Reality, Interactive simulations, Immersive learning
1. Introduction
Innovative technologies and high-quality scientific research have a significant impact on our daily lives. Topics
in the field of science, technology, engineering, and mathematics (STEM) are becoming more and more relevant
and are a significant driver of innovation. Consequently, there is a growing demand for experts with experience,
know-how, and skills in these fields (Zeidler, 2016). Olson and Riordan (2012) have already pointed out the
need to increase the number of students with a degree in STEM disciplines. Lack of interest and enthusiasm are
among the reasons for high failures in these fields. Students describe it as boring, complicated and uninteresting.
It is not clear to many students why they must study natural sciences. It is therefore necessary to promote the
interest and motivation of students in the STEM disciplines (Reeve, Jang, Carrell, Jeon, & Barch, 2004).
As Freedman (1996) shows, teaching science is a challenging task, especially in the field of physics. Traditional
teaching methods present solutions and concepts, but they fail to teach how to solve problems. Hake (1998)
confirms this observation by showing that students have difficulties understanding conceptual aspects while
memorizing formulas. An old Chinese saying supports the concepts of learning through experience: “I hear, and
I forget. I see and I remember. I do and I understand.” The integration of educational activities which involves
learners in the learning process has shown to be a successful teaching method. Students learn by doing things
and think about what they are doing (Freeman et al., 2014). Learning tools with interactive and engaging
activities can help students to better understand such conceptual aspects. Sanders (2008) shows that in physics,
the combination of hands-on experiments, interactive simulations and active participation are valuable tools that
support learners during their educational process. Interactive simulations such as PhET (see
https://phet.colorado.edu/) allow students to take the ownership of the learning experience and support their
conceptual understanding by making connections to everyday life. They can be integrated into teacher demos,
interactive discussions, classroom activities, labs and homework to support teachers in illustrating concepts
(Moore, Chamberlain, Parson, & Perkins, 2014). This active learning concept can be extended by various
modern technologies to meet the needs of a new generation of learners in a flexible and digital way. Simulations,
visualizations, virtual and remote laboratories support students in self-directed, active, and group-based learning.
De Jong et al. (2013) showed that any lab form has its advantages for certain use cases. While real laboratories
are more suitable for acquiring hands-on experience, virtual laboratories enable expandable experiments,
multiple access and visual representation of unseen phenomena with minimal potential for the occurrence of
dangerous situations. In virtual laboratories designed for active learning , learners become part of the simulated
environment by interacting with the virtual world, which helps them to learn complex concepts. The combination
of simulations and visualizations in a lab-like environment offers schools and universities a cost-effective way to
provide learning experiences similar to those in real-world labs (Asıksoy & Islek, 2017). The use of immersive
108
and interactive technologies such as virtual reality opens new possibilities for creating engaging learning
experiences.
In previous works (Pirker et al., 2017a; Pirker et al., 2017b; Pirker, Lesjak, Parger, & Gütl, 2018) we introduced
Maroon, an interactive physics laboratory and experiment environment designed to teach physics in a more
engaging and immersive way. In small-scale studies we found that various VR technologies can be used to
support and engage learners in understanding physics. Maroon supports immersive VR, where a stereoscopic
head-mounted display (HMD) that is tracked by sensors is used. With the HMD and controllers it is possible to
achieve spatial immersion and to conduct hands-on experiments. Additionally, Maroon can be used on a PC
(desktop VR), where a three-dimensional world is simulated on-screen. In a recent study (Pirker et al., 2019a) we
collected qualitative data from secondary school teachers to identify use cases, design goals and issues. In this
paper we intend to extend our previous work (Pirker, Holly, & Gütl, 2020) to facilitate the difficult task of
designing educational VR platforms by describing the challenges and the expectations of educators and pupils.
The main research goals are defined as the following:
• Identification of educator and learner expectations for teaching and learning in VR.
• Providing a first guideline for the design of a VR learning environment for the classroom.
• Discussion of challenges and recommendations for the design and development of learning and teaching
activities in VR.
Contribution: In this paper, we present a study with 26 student teachers and 59 pupils, discussing the room-
scale VR version of the learning environment Maroon for physics education in classroom situations. The focus is
on identifying and discussing expectations for teaching and learning in VR and challenges by combining the
study results with a literature review.
The following section gives an overview of the background and the related work in the field of STEM education
and of the challenges in designing educational VR platforms. In Section 3, we introduce the virtual physics
laboratory Maroon and the conceptual design of the experiments and simulations. Section 4 describes the study
design, and Section 5 presents the results. In Section 6, we discuss the challenges and recommendations based on
the literature review and our findings. Section 7 closes with a discussion about implications, potential and gives
some ideas for further studies.
2. Related work
STEM subjects have high drop-out and failure rates. This can be related to difficulties students have in
understanding theoretical concepts (Olson & Riordan, 2012). Therefore, researchers such as Olson and Riordan
(2012) or Dori, Hult, Breslow, and Belcher (2007) combined traditional classroom experiences with interactive
and engaging digital learning experiences such as virtual simulations or animations to help students better
understand the underlying concepts and phenomena. Conducting experiments in virtual or remote laboratories
that would otherwise be dangerous, hard to conduct and/or too expensive is another way for effectively support
learners (Corter et al., 2007). Wieman and Perkins (2005) showed that digital resources are a cost-effective, safe,
and fast alternative to traditional learning methods and experiment setups. Furthermore, these digital resources
have been shown to help users to achieve a better understanding.
VR tools can support the students in their learning process. Slavova and Mu (2018) showed that when using VR
as a complementary tool to traditional learning methods, students showed higher performance in understanding
and recognizing concepts. Furthermore, learners using VR technologies remain more motivated during their
learning process (Liou & Chang, 2018). Bogusevschi, Muntean, and Muntean (2020) have published a study on
the effect of a virtual 3D physics learning environment on 12-13-year old students that concluded that over 74%
of the participants found the simulation helpful for gaining a better understanding.
Due to the decreasing prices of HMDs, we could observe that an increasing number of studies on the use of VR,
especially for educational use, have been published. Furthermore, HMDs have become lighter and even wireless
alternatives, such as the Oculus Quest, are now available, factors which are helping to increase their broad
distribution and use. Due to the lower costs of HMDs, the increased research and the technical process, VR is
becoming both more immersive and more widely available to a wider user base (Dempsey, 2016), making it
easier to use VR learning experiences as an addition to the traditional classroom setups.
Nevertheless, researchers are still investigating the challenges that arise from the use of VR as a learning tool.
Velev and Zlateva (2017) point out that a highly immersive experience must be created to achieve a high
109
learning rate. According to Stark (1995) VR does not depend on highly realistic surroundings, as long as the
virtual space offers enough cues for the perceptual system, immersion can be achieved. It is of greater
importance when creating an immersive experience that the motion sickness is reduced to a minimum and the
interface design of the VR simulation is appealing (Callaghan, Eguíluz, McLaughlin, & McShane, 2015).
Especially in the field of physics, immersive learning experiences help to achieve a better understanding,
particularly if the design places a special emphasis on interactivity (Pirker, Gütl, Belcher, & Bailey, 2013; Pirker,
Berger, Guetl, Belcher, & Bailey, 2012). Abulrub, Attridge, and Williams (2011) confirm that for a high learning
rate, the VR experience must be designed so that the user can actively interact with the virtual world.
Furthermore, not only the setup time of HMDs and virtual simulations for the use in real classrooms could
become problematic, but also the time needed for students to become familiar with the new technology (Velev &
Zlateva, 2017). They need a guided tutorial or someone to help them and guide them through their first
experiences (Abdelaziz, Alaa El Din, & Senousy, 2014; Safikhani, Holly, & Pirker, 2020). De Jong and Van
Joolingen (1998) describe that it has a positive influence on the learning effect when theoretical information is
available during the simulation. Additionally, they explain that tasks help the users to focus on the desired
outcome and guide them through the experience. Most teachers have limited time and classroom settings usually
provide limited hardware resources, factors which reduce the time each of their students has in the virtual
laboratory. Most VR learning experiences allow only limited personal interactions, which makes collaborative
learning and teamwork difficult (Velev & Zlateva, 2017).
One of the most challenging, but also most promising parts of VR learning experiences is the optimization of the
knowledge acquisition process. Mainly Callaghan, Eguíluz, McLaughlin, and McShane (2015), Velev and
Zlateva (2017) and Liou and Chang (2018) discovered various guidelines for designing VR learning experiences.
Creating a good virtual experience requires that users are not initially overwhelmend with the virutal world,
which can be achieved by reducing dizziness and motion sickness. Furthermore, VR laboratories offer many
possibilities that conventional ones cannot provide. For example, invisible phenomena can be visualized and
made interactable and students can perform dangerous and expensive experiments. Students are offered a fully
controllable and safe learning environment that reduces health risks, in case of performing dangerous
experiments, and ensures that the same input will always lead to the same output. It also gives learners the
possibility to apply their theoretical knowledge in practice, which motivates them and boosts their creativity.
Given the large number of studies focusing on VR learning environments and laboratories, Ip and Li (2015)
criticize that most of them failed to demonstrate an increase in long-term knowledge acquisition and that more
long-term observations should be made.
3. Maroon – Virtual learning application
Maroon (see https://maroon.tugraz.at; to put someone ashore and abandon them on an island) is an interactive
virtual physics laboratory that allows students to explore various experiments and phenomena in an immersive
and engaging way. It is implemented in Unity (see https://unity.com) and supports different platforms with
different levels of immersion such as virtual reality, mobile devices, or web-based applications. The learning
activities and experiments are designed for active learning to involve students in the learning process. The main
laboratory room (Figure 1) consists of different experiment stations and functions as a three-dimensional menu
where the user can select one of the experiments by navigating to the specific station. Currently, the laboratory
contains a whiteboard scene with different learning lessons, and eight experiments in the field of
electromagnetism, electrostatics, oscillation, and waves (Pirker et al., 2019b). The experiments support several
virtual learning experiences with different forms of engagement and immersion through diverse activities and
interactions. In the following subsections the application concepts of Maroon VR and the different experiment
setups are presented.
110
Figure 1. Overview of Maroon’s laboratory with different learning stations
3.1. Maroon VR
The VR version of Maroon extends the core functionalities of Maroon with room-scale VR support for the HTC
Vive (see https://www.vive.com) and the Oculus Rift (see https://www.oculus.com/rift). This version of Maroon
offers a virtual reality experience with a high degree of immersion and focus on the learning content. Users can
walk freely in the classroom within a predefined play area, allowing natural movement within the virtual
environment. Due to real-world restrictions and the limitation of the tracking area, users need to become familiar
with a different form of movement to cover greater distances in the virtual world. In VR applications,
teleportation has become a standard and allows the user to move to a certain position while minimizing the
feeling of motion sickness. For teleportation, the user can press a button on the controller to activate a colored
arc, by moving the controller the user can point to the desired destination. After releasing the button, the user is
placed in that position. Teleport markers in front of each experiment station help to navigate to a specific
experiment by capturing the teleport beam. Each experiment or activity can be started from an entry-point that
acts as a portal into the simulation room. Each experiment has a customized user interface and offers various
virtual control elements to control specific experiment parameters and visualizations.
3.2. Experiment setups in VR
Maroon VR supports two different types of experiment setups each with an individual design. The first
experiment setup contains only the elements necessary for the simulation (see Figure 2a). The experiment is
placed in the middle of a simple and compact room and can be controlled via virtual control panels on the right
and left side. This allows the user to observe the experiment results while setting the experiment parameters. To
keep the interaction as simple as possible, all elements are arranged in such a way that they are accessible
without use of teleporting. In contrast, the second setup contains additional gamified elements in a retro-
futuristic laboratory room with a simplified design to improve student’s engagement and motivation (see Figure
2b). The predefined quest list gives clear instructions by displaying different tasks, the progress, and additional
information about the simulation.
(a)
(b)
Figure 2. Different design setups in VR: a) simple experiment room with virtual control elements and b) retro-
futuristic laboratory room with quest management system
111
3.3. Experiments and simulations
In this section the two interactive experiments Faraday’s Law and Huygens’s Principle are presented, which
were used to evaluate the room-scale VR version of Maroon. The goal of these simulations is to simulate and
visualize the concept of induction and diffraction. Each of them implements the general physical phenomenon
and extends it with an appropriate user interface. The Faraday’s Law Experiment shows the principle of
induction when interacting with a permanent magnet and a conductive non-magnetic ring. Whenever the user
moves the magnet, it causes a change in the magnetic flux and induces an electric current that generates another
magnetic field. Through different visualizations the invisible magnetic field becomes visible and helps the user
to better understand the underlying concepts. The virtual controls allow the user to influence the result of the
experiment by changing the parameters of the magnet and the coil. The special feature of this experiment is the
haptic feedback from the controllers, which allows the user to feel the acting forces. The controller vibrates as
soon as a physical force acts on the magnet, where low vibration means a weak force, and high vibration means a
heavy force. This allows users to have a real feeling of the acting forces. The Huygens’s Principle Experiment
uses water waves in a basin to demonstrate the physical concept of diffraction. It is a phenomenon that occurs
when a wave hits an obstacle or a slit. To show the effect of diffraction, a slit plate is placed into the basin. When
a wave hits this plate, the points on the wave act as a new source of secondary waves that propagate. This results
in an interference pattern behind the plate. To obtain different interference patterns, the user can replace the plate
with three types of slit plates. The experiment is influenced by the user by grabbing and moving the plates and
changing physical parameters such as frequency, amplitude, wavelength, or the propagation mode. To make the
wave peaks and wave trough more visible, the wave color can be changed using a color wheel.
4. Evaluation
In previous studies, we focused on different learning experiences with room-scale VR, mobile VR, and
traditional screen-based technologies as well as on engagement, usability, and user experience with room-scale
VR as the most engaging and most immersive form. However, we also found that teachers and students have
very different opinions and expectations about experience design in VR. Therefore, in this paper, we focus in
this paper on identifying teacher and pupil expectations for teaching and learning in VR. Since there are known
issues convincing experienced teachers to try new things in educational technology, we conducted a user study
with 26 student teachers and 59 pupils, which are open for new technologies. The study was organized in
cooperation with two local schools and two universities with a focus on the following:
• Experience and engagement,
• Usability,
• Learning value from learner’s perspective,
• Learning value from teacher’s perspective.
4.1. Setup
We used two portable setups, including a gaming notebook, an HTC Vive HMD, two controllers, two
lighthouses, and two tripods for the lighthouses. The two setups were placed in a single classroom with a
minimum size of 2m x 2m for each VR station. All participants were in the same classroom and worked in pairs.
Half of the participants tested the simple design of the experimental room, and the other half tested the retro-
futuristic laboratory room. While the persons who tested the experiments in VR were given instructions from
their partners outside (Figure 3a), the others tried the same experiments in the desktop version (Figure 3b). After
20 minutes, the groups were swapped, so that the participants using the VR version were now using the PC
version and vice versa. To get a real classroom situation the setup was conducted within the context of a physics
class and in a university classroom where the student teachers were role-playing as pupils. During the tests
participants were not wearing earphones so we could talk to them and guide them through the experiments.
112
(a)
(b)
Figure 3. Classroom setup: a) VR workstation and b) desktop workstation
4.2. Material and procedure
For this study, we worked with two high schools and two universities to test Maroon from the perspective of
students and prospective teachers. At the beginning of each test run the participants were asked to fill out a pre-
questionnaire to obtain information about their ages, gender, experience with games, VR platforms, and e-
learning tools. They were then given a short introduction how to work and interact with the environment. We
introduced them to the controls and gave them instructions for the different tasks to be completed during the test
run.
Pupils and student teachers were asked to perform the following tasks:
• Introductory session: Have a look at the lab environment and get a first impression.
• Go to the Faradays’ Law experiment and start the simulation by moving the magnet. Try to identify the
relationship between the electrical current and the acting force. Find out how the parameters of the magnet
and the coil affect the experiment outcome.
• Go to the Huygens’ Principle experiment and start the simulation. Try to understand the concept of
diffraction and describe the interference pattern behind the slit plate.
• Take time to try other experiments (optional).
We provided an additional exercise sheet for the pupils in which they could work on different questions about
the experiments. After conducting the experiments, the participants were asked to fill out a post-questionnaire in
which they had to answer open-ended questions about their overall experience; 22 questions on a Likert scale
between 1 (fully disagree) and 5 (fully agree) about their sentiment towards the physics lab, and 10 questions
regarding usability. We used the System Usability Scale (SUS) (Brooke, 1996) to measure the system usability
and the Computer Emotion Scale (Kay & Loverock, 2008) to assess users’ experiences of interacting and
learning with the virtual environment. To gain a deeper understanding of the user experience and use cases,
pupils and student teachers were interviewed about the experiments and application scenarios in school classes.
Student teachers were also asked about pedagogical models and cooperative scenarios.
4.3. Participants
In total 40 high school pupils, 19 pupils from an engineering secondary school (51 male, 8 female) and 26
student teachers (15 male, 11 female) took part in the user study. All 26 student teachers were attending the
teacher education program for physics. The pupils were aged 13 to 19 (AVG = 14.41, SD = 1.87) and the student
teachers from 21 to 31 (AVG = 23.69, SD = 2.65). 14 pupils and 11 student teachers had a visual impairment
(including people wearing glasses).
We asked each of them to rate their experience with computers, video games and VR on a Likert scale from 1
(low) to 5 (high). Most pupils rated themselves as experienced with computers (AVG = 3.22, SD = 0.91) and
video games (AVG = 3.59, SD = 1.16). Student teachers stated that they were also experienced in using
computers (AVG = 3.12, SD = 0.91) but rated their experience with video games lower (AVG = 2.54, SD = 1.65).
Pupils and student teachers indicated that they often play video games (pupils: AVG = 2.68, SD = 1.41; teachers:
AVG = 2.27, SD = 1.46). 52 of the pupils and 16 of the student teachers liked playing video games but had little
113
experience with VR (pupils: AVG = 1.69, SD = 0.93; teachers: AVG = 1.50, SD = 0.86). 52 pupils and 19 student
teachers had heard about VR devices previously, but only 22 pupils and 7 student teachers had tried one. 45
pupils and 20 student teachers have already used an e-learning tool. Both pupils and student teachers consider
the use of virtual reality in physics lessons to be a good idea.
5. Results
In this section, we describe the results obtained from the open-ended answers and the answers based on a Likert
scale. Since the learning outcome depends on the user experience and the acceptance of the system, we focus on
usability, engagement, immersion, and the learning experience as well as on application scenarios and
pedagogical models.
5.1. Usability and user experience
Most participants were able to handle the VR controllers without any problems. Only a few of them had some
initial problems when they tried to move in the virtual world or wanted to interact. They teleported themselves
against walls or used the wrong button for interaction. By contrast, all participants were able to interact with the
PC version without additional instructions. The users described the desktop application as more familiar but
would prefer the interaction in VR as it is more realistic and natural. We used the Computer Emotion Scale to
evaluate the emotions anger, anxiety, happiness, and sadness when learning with the VR environment. As shown
in Table 1, pupils and student teachers rated the emotion happiness (e.g., satisfied, excited, curious) as high and
the emotions of sadness, anger, and anxiety as very low. In total 44 pupils and 23 student teachers have
completed the SUS questionnaire. It shows the degree of usability from 0 (poor) to 100 (excellent) and was rated
with 76.5 by pupils and with 73.5 by student teachers. The resulting scores indicate an above average usability
compared to other VR learning tools. When we asked participants whether they felt nauseous or dizzy while
using VR, only two pupils and one student teacher reported cyber sickness.
Table 1. Results of the 12-item Computer Emotion Scale on a Likert Scale between 0 (never) and 3 (always)
Pupils
Student Teachers
AVG
SD
AVG
SD
Satisfied
2.63
0.48
2.58
0.69
Excited
2.78
0.41
2.73
0.44
Curious
2.73
0.48
2.73
0.44
Happy
2.46
0.56
2.42
0.74
Depressed
0.15
0.48
0.04
0.19
Discouraged
0.36
0.73
0.04
0.19
Scared
0.22
0.49
0.08
0.27
Insecure
0.54
0.62
0.69
0.72
Helpless
0.41
0.49
0.46
0.57
Nervous
0.51
0.74
0.38
0.56
Frustrated
0.03
0.18
0.08
0.27
Angry
0.05
0.39
0.04
0.19
5.2. Immersion and engagement
Pupils rated the immersion of the experience on a Likert scale from 1 (not immersive at all) to 10 (fully
immersive) as an average of 9.00 (SD = 0.98). Some pupils mentioned that learning with Maroon was easier and
more engaging than traditional learning methods. Only two pupils experienced dizziness, which could hinder a
convincing user experience.
Prospective student teachers rated the immersion of the experience on the Likert scale from 1 to 10 slightly
lower than the pupils with an average of 8.50 (SD = 1.24). Among other things, it was noted that Maroon VR has
a high degree of interaction that allows the inclusion of multiple senses. Furthermore, some of the student
teachers mentioned the feeling of forgetting about being in a virtual world after spending some time within
Maroon, which was induced by the high degree of interaction.
114
5.3. Learning value from the learner’s perspective
To evaluate the learning experience, pupils were asked to rate their learning experience on a Likert scale
between 1 (not agree) and 5 (fully agree). Table 2 gives an overview of the pupils’ results with a focus on
learning experiences. The pupils indicated that the experience was more engaging (AVG = 4.17, SD = 0.91) and
fun (AVG = 4.68, SD = 0.63). In general, learners found that the lab made the content more interesting and easier
to understand. Most of them said they would like to learn with Maroon (AVG = 4.39, SD = 0.89).
They also reported that they liked the immersive laboratory and outlined that concepts are easier to understand
using three-dimensional visualizations and interactions than with traditional methods. Being able to see field-
lines that are usually not visible contributed to this. They also mentioned that compared to real life experiments it
was easier to conduct the experiments in VR and that Maroon was a fun experience and was a welcome
relaxation from their typical school routine. Suggestions for improvements made the experiments look even
more realistic, further improving the visualizations. A larger collection of experiments was also mentioned to
improve Maroon.
5.4. Learning value from the teacher’s perspective
Besides pupils, student teachers were also asked to rate the learning experience with Maroon on a Likert scale
between 1 (not agree) and 5 (fully agree). Most student teachers mentioned that the VR setup is a good
supplement to regular learning (AVG = 4.15, SD = 1.08). They also reported that Maroon makes the learning
content more interesting (AVG = 3.88, SD = 1.18) and easier to understand (AVG = 3.65, SD = 1.13). In general,
student teachers found that learning with Maroon was more motivating than ordinary exercises and more fun, as
can also be seen from Table 2.
The student teachers reported that VR can be a good way to extend the variation of teaching methods in the
classroom. The fact that Maroon appeals to multiple senses makes the system attractive for several types of
learners, while the novelty of the technology is an attractive way to motivate students who normally show little
interest in a subject. It was also mentioned that virtual experiments are valuable when the real experiment is
expensive or dangerous, since students can carry these out with no health risks. Also mentioned was the
possibility of visualizing unseen phenomena, learning in a playful way and the ability to change the experiments
very quickly. Some student teachers criticized Maroon as being too close to regular games in its current state,
and this makes it more difficult to keep students focused. To allow students to use it without close monitoring
and guidance, and to prevent students from being distracted, the system should include detailed tasks with clear
instructions. It was also found that obtaining enough VR headsets for all students is currently too costly and for
this reason would be best to use the technology in the form of projects days. The setup time for the system can
be regarded as reasonable in the context of such a project day.
In addition, student teachers recommended the use of videos in Maroon to provide additional support in the form
of introductory film material that revisits the learning topics discussed, or videos showing the experiment and
additional information to support the process of learning while experimenting in VR.
Table 2. Learning Experience rated by pupils and student teachers on a Likert Scale between 1 (not agree) and 5
(fully agree)
Pupils
Student Teachers
AVG
SD
AVG
SD
I would like to learn with Maroon.
4.39
0.89
3.54
1.27
It is a good idea to use Maroon for learning.
4.51
0.86
3.92
0.98
Maroon is a good supplement to regular learning.
4.56
0.79
4.15
1.08
I learned something with Maroon.
4.08
0.92
2.69
1.23
Maroon makes the content more interesting.
4.63
0.64
3.88
1.18
Maroon makes the content easier to understand.
4.27
0.89
3.65
1.13
Maroon makes learning more engaging.
4.17
0.91
3.19
0.85
Maroon makes learning more fun.
4.68
0.63
4.08
1.09
Maroon makes learning more interesting.
4.46
0.73
3.96
1.08
The experience with Maroon inspired me to learn more about physics.
3.49
1.22
3.12
1.21
Learning with Maroon was more motivating than ordinary exercises.
4.51
0.70
4.23
0.95
It makes course content more interesting to learn about.
4.44
0.77
3.62
1.10
I would rather like to learn Physics with Maroon than with traditional
4.14
0.96
2.81
1.27
115
methods.
I find regular physics classes boring.
3.19
1.25
1.54
0.58
Seeing the simulations with the VR glasses was engaging.
4.17
0.81
3.81
0.98
Seeing the simulations with the VR glasses was interesting.
4.42
0.65
4.54
0.58
Seeing the simulations with the VR glasses was more engaging than
without.
4.37
0.87
3.96
1.04
I would rather use Maroon on my phone (+ VR glasses).
3.34
1.36
2.85
1.22
I would rather use Maroon on my own PC.
3.81
1.07
3.12
1.24
I would like to learn with Maroon in the classroom.
4.41
0.87
3.77
0.99
I would buy the VR glasses and download Maroon at home.
3.81
1.12
3.04
1.46
It was interesting to use Maroon.
4.81
0.43
4.73
0.53
5.5. Use cases and pedagogical models
The perspectives of pupils and teachers are often quite different. To consider their ideas and suggestions on how
to use VR in schools and what can be learned in VR into consideration, we asked the pupils and student teachers
in open questions to describe their ideas and scenarios. Based on these answers, we provide a list of use cases
and subjects that can be applied in schools.
5.5.1. How to use VR in schools
To identify different use cases, we asked pupils (P) and prospective student teachers (T) how they would use VR
in school. Student teachers were also asked to give collaborative scenarios as well as pedagogical models.
The following use cases were mentioned:
• Dedicated VR room (P, T) – Pupils mentioned that they would prefer a dedicated classroom for learning
with VR. This includes learning as an integral part of classroom lessons as well as after-school
(extracurricular) activities to repeat and review the learned materials. The student teachers also suggested a
dedicated room with pre-installed VR setups to save time.
• Mobile VR experience (P, T) – Due to the financial constraints, pupils and student teachers mentioned the
usage of smartphones in combination with a mobile VR headset. That allows pupils to run experiments on
their devices at school and at home. But are limited in their use due to the processing power of these
devices.
• Weekly classes (P, T) – Some pupils suggested using VR on a weekly basis to supplement the learning
material with simulations and visualizations. Student teachers also mentioned the potential of regular VR
sessions in classes to increase the learning outcomes of pupils.
• Optional Subject (P) – Pupils also recommended an optional subject or an afternoon class where they can
use VR for learning and increasing their skills.
• Project/Group Work (T) – Several student teachers reported the potential of blocked VR classes in the form
of project days during which pupils can work in groups on different experiments.
• Autonomous Learning (T) – The student teachers mentioned that VR as an effective educational method,
could be a valuable tool to support students in autonomous learning. They also mentioned that using
worksheets might be useful to give pupils clear instructions and help them to focus on the experiment.
• Collaborative Learning (T) - Since pupils benefit from each other’s resources and skills when learning
together, student teachers suggested working in groups, where one pupil performs the experiment and the
others give hints and take notes.
5.5.2. Teaching subjects
To identify additional subjects and experiments, we asked pupils and student teachers about phenomena they
would like to see, learn, or teach in VR.
The following subjects were mentioned:
• Astronomy
• Physics
• Chemistry
• Biology
116
• History
• Engineering
While pupils were interested in different STEM subjects, the student teachers were mainly interested in
experiments in their own field – physics. All of them also highlighted that VR would be an opportunity for
performing expensive and dangerous experiments within class and for visualizing and experimenting with
unseen phenomena.
6. Discussion
In this study, we examined the challenges and recommendations for learning and teaching in VR, with an
exemplary focus on the physics domain. In previous studies (Pirker et al., 2017a; Pirker, Lesjak, Parger, & Gütl,
2018), we have evaluated different VR experiences versus a classic desktop experience with a small group of
students and teachers. We showed the potential of VR in the field of STEM and discussed the advantages and
disadvantages. In this paper, we aim to facilitate the challenging task of designing VR learning platforms by
describing the challenges and giving some recommendations to overcome these challenges.
6.1. Challenges and recommendations
Summarizing the findings of our research and study, we concluded that we can distinguish between the
following main challenge categories: (1) immersion, (2) costs, (3), time restrictions, (4) knowledge gaining
process.
To achieve an increased learning effect, it is crucial to offer a highly immersive experience, since then the user
becomes part of the virtual world. Moreover, motion sickness should be restricted to a minimum. We
recommend using HMDs combined with motion tracking and input devices that enable the interaction with the
virtual world, as this already ensures a high level of immersion. Furthermore, the user interface should be
tailored to the VR world, meaning that we create an appealing interface design and give the learners the
possibility to interact directly with their learning content. As a result, users show more motivation for the
learning content and simulations as we enable them to learn and explore the phenomena in a playful way (Velev
& Zlateva, 2017; Callaghan, Eguíluz, McLaughlin, & McShane, 2015). We observed that students could ask
questions regarding controls and experiment specifics, because the headsets were not equipped with earphones.
This was especially helpful for first-time VR users, because they could show where a problem was in VR and we
could see what they were trying to accomplish on-screen and point them in the right direction.
Another factor for using VR experiences complementary to the traditional classroom learning methods are the
included costs. VR equipment prices already decreased in the previous years and HMDs are now available to a
broader user base. Nevertheless, it is still a significant factor for most schools as they cannot afford to buy many
HMDs and equip whole classes (Abulrub, Attridge, & Williams, 2011). We thus recommend using VR setups
within project days, so that less VR equipment will be needed. Using VR laboratories instead of traditional ones
reduces the costs of laboratory equipment and gives learners the possibility to perform even expensive
experiments within the virtual world as many times as they want. Such an approach presents itself as a possible
solution until mobile VR headsets can provide a similar degree of immersion and computing capabilities.
When using VR, an introduction of some kind is generally required to familiarize the user with the HMDs and
controls. We must consider the training time for users (Velev & Zlateva, 2017). This can be optimized by using
guided tutorials to explain how to use the controls (Abdelaziz, Alaa El Din, & Senousy, 2014), not to mention
that with tutorials teachers do not need to guide each student personally and hence more time is saved. In
addition, the setup time factor is significant if the VR equipment is to be used during traditional lessons. Hence,
we recommend using the virtual laboratories within project days or coordinate with other classes, so that the
equipment is in use throughout the whole day. In consequence the VR laboratory offers the possibility to change
experiment settings or reset the simulation with a few commands. In addition, there is no need to prepare
experiments in the real world, this is especially useful when having very time-consuming experiments.
To create a successful VR learning experience the knowledge gaining process should be optimized. Crucial for
the success is that users should not be overwhelmed when first entering the virtual world (Callaghan, Eguíluz,
McLaughlin, & McShane, 2015). Furthermore, it is important that the students know what the goal of the
simulation is and that they do not feel lost in the simulation. During the tests, we noticed that already having
117
background knowledge is a significant advantage for performing simulations. As Mayer’s cognitive theory of
multimedia states, learners receive information via two separate channels (visual and auditory) with limited
capacity. Learners must select and organize the relevant information and integrate it based upon prior knowledge
(Mayer, 2002). We found that a predefined quest list with clear task instructions in combination with worksheets
can help students to stay focused and motivated. Moreover, with the use of VR we encourage the motivation of
learners in a playful way while keeping them safe during the performance of dangerous experiments (Callaghan,
Eguíluz, McLaughlin, & McShane, 2015). Through the use of VR we can provide users with the possibility for
visualizing invisible phenomena, which proved to make the learning content more readily comprehensible for
students (Slavova & Mu, 2018). The greatest advantage, however, is that learners can explore and interact with
the simulations at their own pace and this motivates them, encourages their creativity and lets them utilize their
theoretical knowledge.
6.2. Limitations
Due to the classroom setup in schools, the number of participants and the time constraints, an A/B split user
study was not feasible. We therefore decided to set up the study as a workshop where participants tested the VR
environment in a classroom situation. The different room layouts, light conditions and the shape of play areas
had a marginal influence on the tracking accuracy, which caused slightly different experiences. While some
pupils had experience in programming, VR, and video games, others had not been much involved in computer
games, VR, and programming during their curriculum. The learning effects were determined by self-evaluation
and do not indicate long-term effects. Since not all participants (15 pupils and 3 student teachers) completed the
questionnaire, some data sets could not be included in the evaluation.
7. Conclusion
In conclusion, VR can offer an exciting and engaging ways to learn and teach. But there are still challenges that
need to be overcome to make it feasible for schools. We wish to offer some recommendations on how to create
an engaging VR learning experience, from which both students and teachers will benefit. Our core findings were
that an immersive experience motivates students and encourages them to learn more. Moreover, time and costs
are still crucial factors whether the VR laboratory will be included into school routines or not. Additionally, VR
offers many possibilities and improvements for the knowledge gaining process, such as enabling the
performance of dangerous experiments in a safe way or visualizing invisible processes.
Although participants rated the interaction with the laboratory as good, there is still potential for improvement,
especially in the design and usability. Both pupils and student teachers requested a more accessible method for
easier classroom integration. Future research should consider the potential of mobile VR devices such as
smartphones or standalone devices and explore how we can integrate them into classrooms. For future work, it
would be useful to port the current framework to mobile devices and explore how we can integrate these in
classrooms. An important step would be to discuss the current results with experienced teachers and to obtain
their feedback on the identified challenges and recommendations. Future studies could then investigate the
different use cases to develop a pedagogical model for schools. In future the effect of not wearing earphones and
of receiving tips, breaks and immersion could be investigated, or also if it could be a useful setup to have one
student performing an experiment while one or more peers give hints on how to complete the tasks involved.
Acknowledgement
At this point, we would like to thank all participants of this study and all those people who are and were involved
in the development process. This work was partly funded by the Ministry of Education, Science and Research of
Austria.
References
Abdelaziz, M. A., Alaa El Din, M., & Senousy, M. B. (2014). Challenges and issues in building virtual reality-based e-
learning system. International Journal of e-Education, e-Business, e-Management and e-Learning, 4(4), 320-328. doi:
10.7763/ijeeee.2014.V4.347
118
Abulrub, A. H. G., Attridge, A. N., & Williams, M. A. (2011). Virtual reality in engineering education: The future of creative
learning. In 2011 IEEE global engineering education conference (EDUCON) (pp. 751-757).
doi:10.1109/EDUCON.2011.5773223
Asıksoy, G., & Islek, D. (2017). The Impact of the Virtual Laboratory on Students’ Attitude in a General Physics Laboratory.
International Journal of Online Engineering, 13(4), 20-28. doi: 10.3991/ijoe.v13i04.6811
Bogusevschi, D., Muntean, C., & Muntean, G. M. (2020). Teaching and learning physics using 3D virtual learning
environment: A case study of combined virtual reality and virtual laboratory in secondary school. Journal of Computers in
Mathematics and Science Teaching, 39(1), 5-18.
Brooke, J. (1996). SUS: A quick and dirty usability scale. In Usability evaluation in industry (pp. 189-194). London, UK:
Taylor & Francis Ltd.
Callaghan, M. J., Eguíluz, A. G., McLaughlin, G., & McShane, N. (2015). Opportunities and challenges in virtual reality for
remote and virtual laboratories. In Proceedings of 2015 12th International Conference on Remote Engineering and Virtual
Instrumentation (pp. 235-237). doi:10.1109/REV.2015.7087298
Corter, J. E., Nickerson, J. V., Esche, S. K., Chassapis, C., Im, S., & Ma, J. (2007). Constructing reality: A Study of remote,
hands-on, and simulated laboratories. ACM Transactions on Computer-Human Interaction (TOCHI), 14(2), 7-es.
doi:10.1145/1275511.1275513
De Jong, T., & Van Joolingen, W. R. (1998). Scientific discovery learning with computer simulations of conceptual domains.
Review of educational research, 68(2), 179-201. doi: 10.3102/00346543068002179
De Jong, T., Linn, M., & Zacharia, Z. (2013). Physical and virtual laboratories in science and engineering education. Science,
American Association for the Advancement of Science, 340(6130), 305-308. doi: 10.1126/science.1230579
Dempsey, P. (2016). The teardown: HTC Vive VR headset. Engineering & Technology, 11(7-8), 80-81. doi:
10.1049/et.2016.0731
Dori, Y. J., Hult, E., Breslow, L., & Belcher, J. W. (2007). How much have they retained? Making unseen concepts seein in a
freshman electromagnetism course at MIT. Journal of Science Education and Technology, 16(4), 299-323. doi:
10.1007/s10956-007-9051-9
Freedman, R. A. (1996). Challenges in teaching and learning introductory physics. In from High-Temperature
Superconductivity to Microminiature Refrigeration (pp. 313-322). doi:10.1007/978-1-4613-0411-1_26
Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active
learning increases student performance in science, engineering, and mathematics. In Proceedings of the National Academy of
Sciences, 111(23), 8410-8415. doi:10.1073/pnas.1319030111
Hake, R. R. (1998). Interactive-engagement versus traditional methods: A Six-thousand-student survey of mechanics test data
for introductory physics courses. American journal of Physics, 66(1), 64-74. doi:10.1119/1.18809
Ip, H. H., & Li, C. (2015). Virtual reality-based learning environments: Recent developments and ongoing challenges. In
International Conference on Hybrid Learning and Continuing Education (pp. 3-14). doi:10.1007/978-3-319-20621-9_1
Kay, R., & Loverock, S. (2008). Assessing emotions related to learning new software: The computer emotion scale.
Computers in Human Behavior, 24(4), 1605-1623. doi:10.1016/j.chb.2007.06.002
Liou, W. K., & Chang, C. Y. (2018). Virtual reality classroom applied to science education. In Proceedings of the 23rd
International Scientific-Professional Conference on Information Technology (IT) (pp. 1-4). doi:10.1109/SPIT.2018.8350861
Mayer, R. E. (2002). Cognitive theory and the design of multimedia instruction: An Example of the two-way street between
cognition and instruction. New directions for teaching and learning, 2002(89), 55-71. doi:10.1002/tl.47
Moore, E. B., Chamberlain, J. M., Parson, R., & Perkins, K. K. (2014). PhET interactive simulations: Transformative tools
for teaching chemistry. Journal of Chemical Education, 91(8), 1191-1197. doi:10.1021/ed4005084
Olson, S., & Riordan, D. G. (2012). Engage to Excel: Producing one million additional college graduates with degrees in
science, technology, engineering, and mathematics. Report to the President. Executive Office of the President.
Pirker, J., Berger, S., Guetl, C., Belcher, J., & Bailey, P. H. (2012). Understanding physical concepts using an immersive
virtual learning environment. In Proceedings of the 2nd European Immersive Education Summit, Paris (pp. 183-191).
Pirker, J., Gütl, C., Belcher, J. W., & Bailey, P. H. (2013). Design and evaluation of a learner-centric immersive virtual
learning environment for physics education. In International Conference on Human Factors in Computing and Informatics
(pp. 551 - 561). doi:10.1007/978-3-642-39062-3_34
Pirker, J., Lesjak, I., & Guetl, C. (2017a). Maroon VR: A Room-scale physics laboratory experience. In 2017 IEEE 17th
International Conference on Advanced Learning Technologies (ICALT) (pp. 482-484). doi:10.1109/ICALT.2017.92
119
Pirker, J., Holly, M. S., Hipp, P., König, C., Jeitler, D., & Gütl, C. (2017b). Improving physics education through different
immersive and engaging laboratory setups. In Interactive mobile communication, technologies and learning (pp. 443-45).
doi:10.1007/978-3-319-75175-7_44
Pirker, J., Lesjak, I., Parger, M., & Gütl, C. (2018). An educational physics laboratory in mobile versus room scale virtual
reality-a comparative study. In Online Engineering & Internet of Things (pp. 1029-1043). doi:10.1007/978-3-319-64352-
6_95
Pirker, J., Holly, M., Almer, H., Gütl, C., & Belcher, J. W. (2019a). Virtual reality STEM education from a teacher’s
perspective. In Proceedings of 5th International Conference of the Immersive Learning Research Network. doi:10.3217/978-
3-85125-657-4-14
Pirker, J., Holly, M., Lesjak, I., Kopf, J., & Gütl, C. (2019b). MaroonVR—An Interactive and immersive virtual reality
physics laboratory. In Learning in a Digital World (pp. 213-238). doi:10.1007/978-981-13-8265-9_11
Pirker, J., Holly, M., & Gütl, C. (2020). Room scale virtual reality physics education: Use cases for the classroom. In
Proceedings of the 6th International Conference of the Immersive Learning Research Network (iLRN) (pp. 242-246).
doi:10.23919/iLRN47897.2020.9155167
Reeve, J., Jang, H., Carrell, D., Jeon, S., & Barch, J. (2004). Enhancing students’ engagement by increasing teachers’
autonomy support. Motivation and emotion, 28(2), 147-169. doi:10.1023/B:MOEM.0000032312.95499.6f
Safikhani, S., Holly, M., & Pirker, J. (2020). Work-in-Progress—Conceptual Framework for User Interface in Virtual
Reality. In Proceedings of 6th International Conference of the Immersive Learning Research Network, iLRN 2020 (pp. 332-
335). doi:10.23919/iLRN47897.2020.9155207
Sanders, M. (2008). Stem, stem education, stemmania. The Technology Teacher (pp. 20-26).
Slavova, Y., & Mu, M. (2018). A Comparative study of the learning outcomes and experience of VR in education. In
Proceedings of the Conference on Virtual Reality and 3D User Interfaces (VR) (pp. 685-686). IEEE.
doi:10.1109/VR.2018.8446486
Stark, L. W. (1995). How virtual reality works: Illusions of vision in “real” and virtual environments. In Human Vision,
Visual Processing, and Digital Display VI (Vol. 2411, pp. 277-287). International Society for Optics and Photonics.
doi:10.1117/12.207546
Velev, D., & Zlateva, P. (2017). Virtual reality challenges in education and training. International Journal of Learning and
Teaching, 3(1), 33-37. doi:10.18178/ijlt.3.1.33-37
Wieman, C., & Perkins, K. (2005). Transforming physics education. Physics Today, 58(11), 36. doi:
https://doi.org/10.1063/1.2155756
Zeidler, D. L. (2016). STEM education: A Deficit framework for the twenty first century? A Sociocultural socioscientific
response. Cultural Studies of Science Education, 11(1), 11-26. doi:10.1007/s11422-014-9578-z
