Content uploaded by Cosimo Tuena
Author content
All content in this area was uploaded by Cosimo Tuena on Dec 19, 2017
Content may be subject to copyright.
98 Annual Review of Cybertherapy and Telemedicine 2017
How virtual embodiment affects episodic
memory functioning: a proof-of-concept
study
Cosimo TUENAa,c,
*
, Silvia SERINO a,b, Alexandre GASTON-BELLEGARDEc,d, Eric
ORRIOLSc,d, Dominique MAKOWSKIc,d Giuseppe RIVAa,b, Pascale PIOLINOc,d,e
a Department of Psychology, Università Cattolica del Sacro Cuore, Largo Gemelli, 1,
20100, Milan, Italy
b Applied Technology for Neuro-Psychology Lab, IRCCS Istituto Auxologico Italiano,
Via Magnasco, 2 20149 Milan, Italy
c Memory and Cognition Laboratory, Paris Descartes University, Sorbonne Paris Cité,
Institute of Psychology, Boulogne-Billancourt, France
d INSERM UMR S894, Center for Psychiatry and Neurosciences, Paris, France
eInstitut Universitaire de France (IUF), France
Abstract. Recent theories in the field of embodied cognition have pointed out the
role of the body for episodic memory, i.e. the memory for personally experienced
events. Although virtual embodiment has been used traditionally to investigate the
different components of bodily self, it provides great advantages to manipulate the
whole embodied experience. In the current study, we manipulated three different
levels of virtual embodiment (“full embodiment”, “medium embodiment”, and “low
embodiment”). All participants were asked to navigate in three different virtual
cities and memorize all the events that they encountered within each environment.
We evaluated the effect of different level of embodiment on the main feature of the
recall and recognition (i.e., what events have occurred) and sense of presence. Data
emerge with interesting consequences on embodied cognition hypothesis.
Accordingly, findings are discussed giving an innovative view of virtual reality as
an embodied tool able to influence cognitive processes such as episodic memory.
Keywords. Embodiment, Episodic Memory, Embodied Cognition, Virtual Reality
1. Introduction
Episodic memory is a neurocognitive system which allows humans to remember
personally experienced events (what) in their spatiotemporal context (where and when)
along with their perceptual and affective details [1, 2]. One crucial element of episodic
memory is binding, the process that connects these features [3]. Episodic retrieval is
accompanied by autonoetic consciousness (i.e., the ability of mentally travelling in
subjective time in past and future events) assessed by RKG paradigm (Remember, Know
and Guess paradigm) [4, 5].
Embodied cognition researchers have been recently drawn the attention to the body
as being crucial for cognition [5]. As the mind shapes the body, the body shapes our
*
Corresponding Author: cosimotuena@gmail.com
C. Tuena et al. / How virtual embodiment affects episodic memory functioning: a proof-of-concept study 99
cognitive processes. In particular, embodiment can be defined as the effect of our body
on cognition (e.g., memory, language, attention, action) [6]. This effect occurs thanks to
a sensorimotor simulation that can be defined as the process that allows the re-enactment
of previous perceptual configuration. In particular, sensorimotor details lead to better
memory performances due to a better encoding elaboration and useful traces during
retrieval [8]. This effect was also shown when interacting within virtual environments
by means of input devices [9, 11].
Virtual reality (VR) provides powerful multisensory experiences while interacting
in simulated environments and can be conceptualized as an embodied technology [10].
Virtual embodiment can be successfully induced by visuomotor synchronicity that
enables to alter self-consciousness [11]. Indeed, Kilteni et al. [14] showed that immersive
VR is an effective tool in manipulating embodiment (i.e., self-location, body ownership
and sense of agency). Moreover, embodiment in a virtual avatar provides a rich
interactive experience in simulated environments; this interaction could be exploited to
enhance cognition, such as episodic memory [13]. Finally, VR offers standardized trials
for studying episodic memory in ecological context similar to real life conditions [8].
The main aim of this study was to study whether different virtual embodiment levels
(full, medium and low) have an impact on episodic memory recall and recognition. We
expect that full embodiment condition, confirmed by high embodiment scores at the
questionnaire, would have high level of presence compared to medium and low condition
and that full embodiment leads to better recall and recognition performances compared
to medium and low condition.
2. Method
2.1 Participants and Design
14 young participants (7 males and 7 females; mean age = 22, SD = 2.08) with no history
of psychiatric or neurologic disorders were recruited. The experiment was carried out as
a within-group counterbalanced study.
2.2 Material
In the present study three virtual cities were created at the Memory and Cognition
laboratory using Unity 3D and in-house software (Editomen and Simulmem). The three
cities were built with a single main street with two turns; each city was composed of
three virtual blocks of the same size. Along the path buildings and other city elements
(e.g., cars, traffic lights, signals) were placed. Each city was composed by five relevant
landmarks (e.g., church, green skyscraper, temple, gas station, parking, playground) and
five events (e.g., woman asking the time, break-dancers, couple kissing, dog barking,
guitarist playing a song). For each city, landmarks and events were unique and placed
along the sidewalks in order to have the same amount of events and landmarks per side
across the path (i.e., four on the left sidewalk, four on the right sidewalk and two at the
end of the road in the middle). Landmarks and events were placed along the path: three
of them where placed at the beginning, four in the middle and three at the end. Microsoft
Kinect was used to detect knees and arms movements.
Participants watched the navigation with Oculus Rift DK2 from a first-person visual
perspective. We connected Kinect V2 for Windows with Oculus Rift DK2 on Windows
100 C. Tuena et al. / How virtual embodiment affects episodic memory functioning: a proof-of-concept study
8.1 64 Bits, Area 51 Alienware (i7-3.30Ghz; 16Go RAM; GC: Geforce Titan X),
which was used to launch the virtual environments and the recognition task.
2.3 Procedure
Each participant read and signed the consent form. Then, they were informed that they
would have to navigate in three different cities and they were informed that after
navigation their memory would be tested. Participants were randomly allocated to the
full, medium and low condition and each city was also randomly paired to one of
embodiment conditions. For the full condition participants saw their avatar in a mirror in
a virtual room. In order to induce virtual embodiment, they were asked to stand upright
in front of the Kinect and raise slowly each limb at a time five times. Then they were
placed in a training virtual city and they were asked to march in place to walk and to
touch their forehead with their right or left arm to turn. After they familiarized with the
interaction, the virtual task began. In the medium condition participants saw the
navigation executed by the experimenter by means of keyboard. In the training phase,
they were instructed by the experimenter on how to follow a passive navigation in the
virtual environment (i.e., “You will see the navigation of someone in the city. Touch
your right forehead when you “see” the path turning on the right”; touch your left
forehead when you “see” the path turning on the left”, “Move your foot on the spot when
the path continues”). Finally, in the low condition participants were asked to stand
upright and watch still the navigation executed by the experimenter with arrow keys. In
the last conditions the Kinect was turned off since no embodiment (i.e., no avatar) was
provided; participants watched the navigation by means of the head-mounted display. In
the medium condition a synchronous walk of a passive navigation was performed, while
in the low condition the participant was asked to stay still and simply watch the passive
navigation. In each condition, participants were told to memorize all the events that they
will encounter within the environment and pay attention to perceptive details such as
colors and spatial and temporal contexts (e.g., on the right or left, temporal order).
Participants were also asked to pay attention and memorize the buildings, shops, and the
other elements present in the city.
After each navigation, participants underwent a distractive task (ten minutes).
Neuropsychological tests were administered and short presence questionnaire (feeling of
presence) was filled in. The latter was specifically designed for this experiment but based
on the one administered by Lessiter et al. [14]. Embodiment questionnaire used by
Piryankova et al. [17] were administered only for the full embodiment condition, since
in the other conditions no avatar was provided. The questionnaire investigated location,
ownership and agency. We used a short version adapted for the current study. Then,
episodic recall was tested with a standard procedure developed by Memory and
Cognition Lab [e.g., 10, 18]. Participants were assessed on what (i.e., events and
landmarks) on a free recall and recognition task. For the free recall assessment, the
experimenter asked the participants to recall the main elements encountered along the
navigation (five minutes max.). To each correct response one point was assigned in a
grid by the experimenter. Free recall was calculated by summing the points. Therefore
the maximum score was ten for each recall session; additional items, which were not
considered relevant (e.g., cars, bins), were in any case recorded in the grid. This task was
administered after each condition. At the end of the session a computer-based recognition
task was carried out. We used the Neuropsydia module for Python [19] to build the task.
The computer displayed an element (e.g., an event or a landmark) and the participant had
Short et al. / Cyberharassment and Cyberbullying; Individual and Institutional Perspectives 122
to answer (“Yes” or “No”) with the mouse whether he/she saw or did not seethe image
in one of the cities. 30 images were presented, 15 were actually in the cities, whereas 15
were distractors; nine of the images were events, six were landmarks for both actual
items and distractors. If their answer was “Yes” their recognitions were assessed with
the RKG paradigm [20] and both source memory for the city in which they thought had
encountered the item and for the embodiment condition. Therefore, they had to indicate
if they saw the item in the first, second or third city and if they were in the full, medium
or low condition.
3. Results
In order to analyze embodiment questionnaire scores for full embodiment and presence
questionnaire a paired sample t-test was performed. We performed one-way within-
groups ANOVA for each total free recall and recognition scores for total scores and event
and landmark sub-scores. Paired t-test was conducted for post-hoc analysis. In the
analysis, one subject was excluded due to virtual sickness in the full embodiment
condition. Significance level was fixed at α = .0167 for Bonferroni correction and at α
= .05 for t-test.
3.1 Embodiment Questionnaire
The paired sample t-test revealed a significant difference in the scores for ownership (M
= 3.59, SD = 0.74) and location (M = 4.54, SD = 1.13), [t(12) = -3.59, p = .004]. A
significant difference was found also for ownership scores and agency scores (M = 4.77,
SD = 1.42), [t(12) = -3.03, p = .001]. Finally, any difference was found for location and
agency scores [t(12) = -3.03, p = .577].
3.2 Presence Questionnaire
Repeated measures ANOVA showed a significance [F(2,24) = 6.592, p = .005, ηp2
= .355] in the presence questionnaire scores. Post hoc comparisons reveled statistically
significant results (p = .013) between full embodiment (M = 3.22, SD = 0.43) and low
embodiment (M = 2.66, SD = 0.58) and between medium condition (M = 3.18, SD =
0.34) and low condition (p = .010). Finally, no difference was found in the scores
between full and medium conditions (p = .801).
3.3 Free recall task
Repeated measures ANOVA did not highlight significant differences for free recalls on
scores across the three conditions. Free recalls regardless the type of item (i.e., event or
landmark) did not show statically significant results [F(2,24) = .044, p = .957, ηp2 = .004].
Any significance was found for free recall for events [F(2,24) = .209, p = .813, ηp2 = .017]
and landmarks [F(2,24) = .947, p = .402, ηp2 = .073]. Last, any significance was found
for non-relevant items [F(2,24) = .888, p = .424, ηp2 = .064].
3.4 Recognition task
102 C. Tuena et al. / How virtual embodiment affects episodic memory functioning: a proof-of-concept study
Repeated measures ANOVA did not show statistically significant differences for correct
item recognition across the three conditions [F(2,24) = .293, p = .749, ηp2 = .024]. Again
any significance was found for item (what) recognition depending on the type of item,
respectively [F(2,24) = 1, p = .383, ηp2 = .077] for events and [F(2,24) = 1, p = .383, ηp2
= .077] for landmarks. Any significance was found for Remember responses [F(2,24)
= .794, p = .466, ηp2 = .062], Know responses [F(2,24) = .409, p = .669, ηp2 = .033] and
Guess responses across the three conditions regardless the type of item. Finally, source
scores (respectively city and city paired with condition) regardless the type of item, did
not evidence any statistically significant results [F(2,24) = 1.531, p = .237, ηp2 = .113].
However, interesting trends are reported in Table 1 for what recognition and source
memory scores.
Table 1. Mean (M) and standard deviation (SD) for item and source recognition evidence a positive effect of
the high embodiment condition on recognition memory.
Full embodiment
Medium embodiment
Low embodiment
Item recognition
Source city
Source city and
condition
M = 3,36, SD = 1.04
M = 3,08, SD = 1.19
M = 3,15, SD = .80
M = 2,69, SD = .95
M = 2,15, SD = 1.28
M = 3,08, SD = 1.26
M = 3,00, SD = 1.14
M = 2,31, SD = 1.11
M = 2,23, SD = 1.16
Discussion/Conclusion
In the present study we aim at investigating the effect of virtual embodiment on episodic
memory functioning. Results partially overlap with our hypothesis. First, embodiment
questionnaire revealed that location and agency were greater than ownership. When
comparing our median scores with median scores for each subscale obtained by
Piryankova et al. [17] , we observed that only our agency score was higher, however
overall scores were similar to the ones reported by Piryankova and co-authors.
Interestingly, the feeling of presence in the full embodiment was greater than low
condition, confirming how self-location and sense of presence are complementary
concepts in constructing spatial representations for the body and the environment [12].
However, medium embodiment also showed higher feeling of presence compared to the
low condition. These results highlight how motor control also affects sense of presence
without an avatar, conferring a critical role of action-intention comparison. The link
between intentions, actions and presence was reported by Triberti and Riva [21] and,
indeed, in the medium embodiment condition even though the participants did not have
the decisional level on the navigation their pantomimed actions were successfully paired
with navigational changes. The sense of presence in immersive virtual context might be
more related to self-location and agency rather than ownership. These findings confirm
the importance of embodiment for the sense of presence, however the sensorimotor
involvement, even if simulated (i.e., medium embodiment), contributes to provide
presence within virtual environments. Moreover these results evidence that the
methodology used to induce presence was correctly designed.
Concerning the episodic scores no effect of condition was found. However, trends
in Table 1 indicate a positive effect of full embodiment on episodic memory. Indeed,
source memory reflects recollection and is linked to autonoetic consciousness [20].
Finally, some studies showed that active navigation enhance episodic recall endorsing
the embodied cognition theories [e.g., 10, 11, 22]. However, the previous experiments
C. Tuena et al. / How virtual embodiment affects episodic memory functioning: a proof-of-concept study 103
used non-immersive environment and to our knowledge, this is the first study that
aims at investigating the effect of embodiment with active interaction on episodic
memory.
The main limit of the present experiment is the sample size; it is probable that the
number of participants mainly affected the results. Future directions for the current study
are to enlarge the sample and extend the analysis to the other episodic features
investigated by means of recall and recognition tasks (e.g., where, when, binding).
References
[1] E. Tulving, Episodic memory and common sense: how far apart?, Philosophical Transcription of the Royal
Society London B Biological Sciences 356(1413) (2001), 1505–1515.A.N. Author, Article title, Journal
Title 66 (1993), 856–890.
[2] S. C. Prebble, D. R. Addis, and L. J. Tippett, Autobiographical memory and sense of self, Psychological
Bulletin 139(4) (2013), 815–840.
[3] G. Plancher, V. Gyselinck, S. Nicolas, and P. Piolino, Age effect on components of episodic memory and
feature binding: A virtual reality study, Neuropsychology 24(3) (2010), 379–390.
[4] M. A. Wheeler, D. T. Stuss, and E. Tulving, Toward a theory of episodic memory: the frontal lobes and
autonoetic consciousness, Psychological Bulletin 121(3) (1997), 331–354.
[5] P. Piolino, B. Desgranges, and F. Eustache, Episodic autobiographical memories over the course of time:
Cognitive, neuropsychological and neuroimaging findings, Neuropsychologia 47(11) (2009), 2314–
2329.
[6] M. Wilson, Six views of embodied cognition, Psychonomic Bulletin & Review 9(4) (2002), 625–636.
[7] K. Dijkstra & L. Post, Mechanisms of embodiment, Frontiers in Psychology 6:1525 (2015).
[8] H. D. Zimmer & L. Cohen, R, Memory for action: A distinct form of episodic memory? Oxford University
Press, 2001.
[9] B. M. Brooks, E. A. Attree, F. D. Rose, B. R. Clifford, and A. G. Leadbetter, The Specificity of Memory
Enhancement During Interaction with a Virtual Environment, Memory 7(1) (1999), 65–78.
[10] N. Jebara, E. Orriols, M. Zaoui, A. Berthoz, and P. Piolino, Effects of enactment in episodic memory: A
pilot virtual reality study with young and elderly adults, Frontiers in Aging Neuroscience 6:338 (2014).
[11] G. Plancher, J. Barra, E. Orriols, and P. Piolino, The influence of action on episodic memory: A virtual
reality study, Quarterly Journal of Experimental Psychology 66(5) (2013), 895–909.
[12] G. Riva, From virtual to real body: Virtual reality as embodied technology, Journal of Cyber Therapy
and Rehabilitation 1(1) (2008), 7–22.
[13] B. Lenggenhager, T. Tadi, T. Metzinger, and O. Blanke, Video Ergo Sum: Manipulating Bodily Self-
Consciousness, Science 317(5841) (2007), 1096–1099.
[14] K. Kilteni, R. Groten, and M. Slater, The Sense of Embodiment in Virtual Reality, Presence:
Teleoperators and Virtual Environments, 21(4) (2012), 373–387.
[15] C. Repetto, S. Serino, M. Macedonia, and G. Riva, Virtual Reality as an Embodied Tool to Enhance
Episodic Memory in Elderly, Frontiers in Psychology 7:1839 (2016).
[16] J. Lessiter, J. Freeman, E. Keogh, and J. Davidoff, A Cross-Media Presence Questionnaire: The ITC-
Sense of Presence Inventory, Presence: Teleoperators and Virtual Environments 10(3) (2001), 282–297.
[17] I. V. Piryankova et al., Owning an overweight or underweight body: Distinguishing the physical,
experienced and virtual body, PLoS One 9(8) (2014).
[18] G. Plancher, A. Tirard, V. Gyselinck, S. Nicolas, and P. Piolino, Using virtual reality to characterize
episodic memory profiles in amnestic mild cognitive impairment and Alzheimer’s disease: Influence of
active and passive encoding, Neuropsychologia 50(5) (2012), 592–602.
[19] D. Makowski & L. Dutriaux, Neuropsydia.py: A Python Module for Creating Experiments, Tasks and
Questionnaires, Journal of Open Source Software (in review).
[20] J. M. Gardiner, Episodic memory and autonoetic consciousness: a first-person approach, Philosophical
Transcription of the Royal Society of London B Biological Sciences 356(1413) (2001), 1351–1361.
[21] S. Triberti & G. Riva, Being present in action: A theoretical model about the ‘interlocking’ between
intentions and environmental affordances, Frontiers in Psychology 6:2052 (2016).
[22] H. Sauzéon et al., The use of Virtual Reality for episodic memory assessment: Effects of active
navigation, Experimental Psychology 59(2) (2012), 99–108.
