The use of virtual reality for language investigation and learning

Full text

OPINION ARTICLE
published: 06 November 2014
doi: 10.3389/fpsyg.2014.01280
The use of virtual reality for language investigation and
learning
Claudia Repetto*
Department of Psychology, Catholic University of the Sacred Heart, Milan, Italy
*Correspondence: claudia.repetto@unicatt.it
Edited by:
Gretchen M. Reevy, California State University, East Bay, USA
Reviewed by:
Mel Slater, ICREA-University of Barcelona, Spain
Stanley N. Bursten, Santa Barbara City College, USA
Keywords: embodied language, virtual agents, virtual reality, language learning, second language acquisition
Virtual Reality (VR) is a technological
tool traditionally used in psychology for
clinical purposes or self-empowerment:
a huge number of papers have docu-
mented the validity of treatments pro-
videdorsupportedbyVR.Mostofthese
have illustrated protocols for the treatment
of anxiety disorders: from simple pho-
bias (Krijn et al., 2007), to panic disorders
(Botella et al., 2007), post-traumatic stress
disorder (Gerardi et al., 2008), and gen-
eralized anxiety disorder (Repetto et al.,
2009, 2013b; Repetto and Riva, 2011);
recently, VR has also been effectively
employed for stress management in a non-
clinical population (Gaggioli et al., 2014).
However, in the last decades, growing
attention has been dedicated to this tool
in the field of neuroscience (Bohil et al.,
2011) for the study of spatial navigation,
multisensory integration, social neuro-
science, pain remediation, or neuroreha-
bilitation. Macedonia et al. (2014) took
a step in this direction, using one of the
VR basic elements for the investigation of
language learning. In particular, they built
Intelligent Virtual Avatars (IVAs) with
human appearance and assigned them the
task to teach a second language to differ-
ent trainee populations (adults and chil-
dren). The authors evaluated the results of
their studies and discussed two levels of
outcome: memory performance, and IVAs’
acceptability.
IVAs proved to be as effective as
their human counterparts at teaching
capabilities; learners memorized new
words better when they performed an
iconic gesture associated to the word
than when they only watched the teacher
performing the same gesture, whether the
teacher was virtual or human. This finding
can easily be interpreted within the frame-
work of Embodied Cognition Theory
(Barsalou, 2008): according to this theory,
the cognitive system is no longer consid-
ered as a processor of symbols and abstract
operations, but is deemed to be grounded
in multimodal representations molded
from human experience. In this view, cog-
nitive processes, including language, are
“embodied” in nature, since the perceptual
and motor systems influence the way we
construct concepts, make inferences and
use language. Several proofs of this claim
have been provided in the neuroscience lit-
erature: it has been found that the primary
motor cortex is involved in language com-
prehension (Repetto et al., 2013a), and
that the same neural structures needed
to process sensory information are also
active when processing words that embed
that sensory information, such as color
(Martin et al., 1995).
Considering these recent findings, the
useofVRinthestudyoflanguagepro-
cesses becomes even more reasonable. In
fact, VR can be considered an “embodied
technology” for its effects on body percep-
tions (Riva, 2002): it is possible to use VR
toinducecontrolledchangestotheexperi-
ence of the body; furthermore, the virtual
environment can be enriched to the extent
that it can become a plausible copy of the
real world.
The first feature seems very important
in the study of foreign language learning:
if the association of a gesture to a word can
enhance verbal memory (Macedonia et al.,
2011), then VR offers a privileged medium
in which to implement the training. In
fact, it gives users the opportunity to see
themselves moving in the environment
while being comfortably seated in a chair.
Thanks to different input devices, partici-
pants could virtually perform any action,
even those typically not performable in an
experimental or learning setting (e.g., kick
a ball). Thus, future research could com-
bine the use of IVAs with the possibility
for the learners to see themselves perform-
ing the gesture illustrated by the IVA: for
instance, if kicking a ball is the action to
imitate, the user (by manipulating a joy-
pad) could see in the VR his own leg and
foot raising up and hitting a ball. Recently,
a virtual environment (Riva et al., 2009)
was employed for the study of language
learning and comprehension; preliminary
results pointed out that, in second lan-
guage learning tasks, a virtual motion (a
motion performed in the virtual world
with a body part that is actually steel)
associated to action words can enhance
verbal memory if the environment is per-
ceived as true-to-life. Logically, in native
language tasks, the virtual action can pro-
mote language comprehension (papers in
preparation).
Similarly, the possibility to create very
detailed environments can support the
study of language: if representations in
the cognitive system are multimodal, then
to investigate their properties one should
recreate the multimodal experience that
can trigger the process. Thus, in the
future, navigation in high-quality virtual
worlds, associated with the execution of
gestures or actions, could possibly create
the optimum experience to improve the
www.frontiersin.org November 2014 | Volume 5 | Article 1280 |1

Repetto Virtual reality and language
processing of words, helping trainees to
learn (in case of new language acquisi-
tion) or re-learn/consolidate information
(just think about the rehabilitation of lan-
guage abilities in aphasic patients). New
research is needed to investigate the effi-
cacy of these trainings, and the findings by
Macedonia (Macedonia et al., 2014)testify
that this line of research deserves further
efforts.
As far as the IVAs’ acceptability is con-
cerned, Macedonia’s data seem to sup-
port the idea that, in general, both adults
and younger trainees accepted the avatar
Billie as a language teacher. Addressing
the issue of acceptability brings us to the
concept of “uncanny valley” (Mori et al.,
2012). According to Mori, the acceptabil-
ity of a virtual character is not a linear
function of its human likeness. In other
words, the more a virtual agent resembles
a human being, the more it is accept-
able, until it reaches a certain degree of
likeness: at this point, it is perceived as
“uncanny,” losing credibility and appeal.
Acceptability rises again when the level of
likeness exceeds this critical point, mak-
ing the character almost identical to the
human form. It should be noticed that the
uncanny valley effect represents a poten-
tial risk for the effectiveness of training:
if the trainee recognizes the virtual agent
as uncanny, he/she may be less willing
to adhere to the agent’s instructions, or
to the training itself. Apparently, this was
not the case in Macedonia’s research. In
fact, recent new findings have called the
uncanny valley effect into question. For
example, a recent study by Piwek (Piwek
et al., 2014) highlighted the fact that the
motion quality of a full-body animated
avatar can influence the acceptability of the
IVA: the characters classified in the deepest
location of the uncanny valley in their
static form, were judged later on as more
acceptable when motion was added. This
result seems to indicate that motion per
se may help to bypass the uncanny valley.
Nevertheless, the evaluation of the agent
acceptability appears mandatory: when
building an experiment that includes IVAs,
especially for psychological investigations,
researchers routinely should run pre-tests
that take into account the threat of the
uncanny valley effect.
In conclusion, Macedonia and col-
leagues opened new promising paths for
the study of language learning, by com-
bining neuroscientific knowledge and new
frontiers of technology that can help
address new questions, and by building
powerful tools for the improvement of lan-
guage abilities; the future direction is to
refine the paradigms taking advantage of
all the capabilities that VR offers.
REFERENCES
Barsalou, L. W. (2008). Grounded cogni-
tion. Annu. Rev. Psychol. 59, 617–645. doi:
10.1146/annurev.psych.59.103006.093639
Bohil, C. J., Alicea, B., and Biocca, F. A. (2011).
Virtual reality in neuroscience research and ther-
apy. Nat. Rev. Neurosci. 12, 752–762. doi: 10.1038/
nrn3122
Botella, C., García-Palacios, A., Villa, H., Baños, R.
M., Quero, S., Alcañiz, M., et al. (2007). Virtual
reality exposure in the treatment of panic dis-
order and agoraphobia: a controlled study. Clin.
Psychol. Psychother. 14, 164–175. doi: 10.1002/
cpp.524
Gaggioli, A., Pallavicini, F., Morganti, L., Serino,
S., Scaratti, C., Briguglio, M., et al. (2014).
Experiential virtual scenarios with real-time mon-
itoring (interreality) for the management of
psychological stress: a block randomized con-
trolled trial. J. Med. Internet Res. 16:e167. doi:
10.2196/jmir.3235
Gerardi, M., Rothbaum, B. O., Ressler, K., Heekin, M.,
and Rizzo, A. (2008). Virtual reality exposure ther-
apy using a virtual Iraq: case report. J. Trauma.
Stress 21, 209–213. doi: 10.1002/jts.20331
Krijn, M., Emmelkamp, P. M., Olafsson, R. P.,
Bouwman, M., Van Gerwen, L. J., Spinhoven,
P., et al. (2007). Fear of flying treatment meth-
ods: virtual reality exposure vs. cognitive behav-
ioral therapy. Aviat. Space Environ. Med. 78,
121–128.
Macedonia, M., Groher, I., and Roithmayr, F. (2014).
Intelligent virtual agents as language trainers facil-
itate multilingualism. Front. Psychol. 5:295. doi:
10.3389/fpsyg.2014.00295
Macedonia, M., Muller, K., and Friederici, A. D.
(2011). The impact of iconic gestures on for-
eign language word learning and its neural sub-
strate. Hum. Brain Mapp. 32, 982–998. doi:
10.1002/hbm.21084
Martin, A., Haxby, J. V., Lalonde, F. M., Wiggs, C.
L., and Ungerleider, L. G. (1995). Discrete corti-
cal regions associated with knowledge of color and
knowledge of action. Science 270, 102–105. doi:
10.1126/science.270.5233.102
Mori, M., Macdorman, K. F., and Kageki, N.
(2012). The uncanny valley [from the field].
IEEE Rob. Autom. Mag. 19, 98–100. doi:
10.1109/MRA.2012.2192811
Piwek, L., McKay, L. S., and Pollick, F. E. (2014).
Empirical evaluation of the uncanny val-
ley hypothesis fails to confirm the predicted
effect of motion. Cognition 130, 271–277. doi:
10.1016/j.cognition.2013.11.001
Repetto, C., Colombo, B., Cipresso, P., and Riva,
G. (2013a). The effects of rTMS over the pri-
mary motor cortex: the link between action
and language. Neuro psychol ogi a 51, 8–13. doi:
10.1016/j.neuropsychologia.2012.11.001
Repetto, C., Gaggioli, A., Pallavicini, F., Cipresso, P.,
Raspelli, S., and Riva, G. (2013b). Virtual reality
and mobile phones in the treatment of gener-
alized anxiety disorders: a phase-2 clinical trial.
Pers. Ubiquit . Comput. 17, 253–260. doi: 10.1007/
s00779-011-0467-0
Repetto, C., Gorini, A., Algeri, D., Vigna, C., Gaggioli,
A., and Riva, G. (2009). The use of biofeed-
back in clinical virtual reality: the intrepid project.
Stud. Health Technol. Inform. 144, 128–132. doi:
10.3233/978-1-60750-017-9-128
Repetto, C., and Riva, G. (2011). From virtual
reality to interreality in the treatment of anx-
iety disorders. Neuropsychiatry 1, 31–43. doi:
10.2217/npy.11.5
Riva, G. (2002). Virtual reality for health care: the sta-
tusofresearch.Cyber psychol. Beh av. 5, 219–225.
doi: 10.1089/109493102760147213
Riva, G., Carelli, L., Gaggioli, A., Gorini, A., Vigna, C.,
Algeri, D., et al. (2009). NeuroVR 1.5 in practice:
actual clinical applications of the open source VR
system. Stud. Health Technol. Inform. 144, 57–60.
doi: 10.3233/978-1-60750-017-9-57
Conflict of Interest Statement: The author declares
that the research was conducted in the absence of any
commercial or financial relationships that could be
construed as a potential conflict of interest.
Received: 13 August 2014; accepted: 22 October 2014;
published online: 06 November 2014.
Citation: Repetto C (2014) The use of virtual reality
for language investigation and learning. Front. Psychol.
5:1280. doi: 10.3389/fpsyg.2014.01280
This article was submitted to Educational Psychology, a
section of the journal Frontiers in Psychology.
Copyright © 2014 Repetto. This is an open-access arti-
cle distributed under the terms of the Creative Commons
Attribution License (CC BY). The use, distribution or
reproduction in other forums is permitted, provided the
original author(s) or licensor are credited and that the
original publication in this journal is cited, in accor-
dance with accepted academic practice. No use, distribu-
tion or reproduction is permitted which does not comply
with these terms.
Frontiers in Psychology | Educational Psychology November 2014 | Volume 5 | Article 1280 |2