Human Factors in Virtual Environments

Abstract

virtual environments human factors, user interfaces, computer graphics, head-mounted displays This is the final report detailing the research conducted during a three-year project funded by the HP Laboratories External Research Program (1997-2000) in collaboration with the Department of Computer Science, University of Bristol, UK. The project's initial goals were centred on the investigation of human factors issues related to interacting with distributed computer graphics worlds in real-time over a network. These objectives were extended and focused on fidelity metrics for computer graphics simulations displayed on Head Mounted Displays. Theories from cognitive psychology were employed to devise these metrics which were, subsequently, validated by formally designed experiments involving human judgements of spatial memory awareness states.

Full text

Human Factors in Virtual Environments
Katerina Mania1, Rycharde Hawkes
Internet Systems and Storage Laboratory
HP Laboratories Bristol
HPL-2003-216
October 21st , 2003*
E-mail: k.mania@sussex.ac.uk, rycharde.hawkes@hp.com
virtual
environments
human factors,
user interfaces,
computer
graphics,
head-mounted
displays
This is the final report detailing the research conducted during a
three-year project funded by the HP Laboratories External Research
Program (1997-2000) in collaboration with the Department of
Computer Science, University of Bristol, UK. The project's initial
goals were centred on the investigation of human factors issues
related to interacting with distributed computer graphics worlds in
real-time over a network. These objectives were extended and
focused on fidelity metrics for computer graphics simulations
displayed on Head Mounted Displays. Theories from cognitive
psychology were employed to devise these metrics which were,
subsequently, validated by formally designed experiments involving
human judgements of spatial memory awareness states.
* Internal Accession Date Only Approved for External Publication
1 Dept of Informatics, University of Sussex, Falmer, Brighton, BN1 9QT UK
 Copyright Hewlett-Packard Company 2003

1
HP Laboratories External Research Project Report (1997-2000):
Human Factors in Virtual Environments
Katerina Mania
Department of Informatics
University of Sussex
Falmer, Brighton BN1 9QT UK
k.mania@sussex.ac.uk
Rycharde Hawkes
HP Laboratories, Bristol
Stoke Gifford, Filton Road
BS34 8QZ, Bristol, UK
rycharde.hawkes@hp.com
INTRODUCTION
This is the final report detailing the research conducted during a three-year project funded by the
HP Laboratories External Research Program (1997-2000) in collaboration with the Department of
Computer Science, University of Bristol, UK.
The project’s generic goals were initially defined as the following:
• The construction of distributed computer graphics electronic spaces introduces critical
problems of scale. Some issues to be considered are network-, processing- and perceptual
scalability and the unavoidable latencies caused by the geographical separation of users
and nodes (Demuynck, K, 1996). Development of a shared environment has to take into
account all these limitations. Data management strategies will need to be introduced to
reduce these communication latencies.
• How should socially inhabited electronic spaces be structured and what tools are needed to
construct them? Furthermore, how should people be supported in exploring and navigating
such spaces and how might the structure afford possibilities for social interaction? One of
the most difficult issues is how to provide a sense of personal presence and awareness,
both direct and peripherally, with other people within an electronic space and how to
achieve this through user embodiment and other representation techniques. Most current
systems have major difficulties conveying presence of other users, awareness of what
other users are doing and providing mechanisms to represent a user as an embodiment within
a single application.
• Techniques for integrating electronic spaces with physical spaces are going to be
investigated. These environments are going to be addressed as inhabited social
environments capable of supporting participation in many different activities.
The proposed research was defined in terms as “inhabited”, “social interaction”, “awareness”,
“spatial metaphors”, “reactions”. It had a view of virtual reality and related technologies as
providing inhabited social spaces and by perceiving users as citizens and social beings both at

2
work and play.
This report is going to be divided in three separate stages. STAGE 1 represents the initial
investigations based on the generic goals stated above. STAGE 2 will present the revised
directions and first experimental results and STAGE 3 will describe the final, formally designed
experimental studies which formed the core of this project. STAGE 2 and 3 resulted in Katerina
Mania’s Ph.D. thesis1 submitted in June 2001. Her Ph.D. degree was awarded in October 2001.
1 Mania, K. (2001). Fidelity Metrics for Virtual Environment Simulations based on Spatial Memory Awareness States.
Ph.D thesis, University of Bristol, UK, Dept. of Computer Science

3
STAGE 12
SUMMARY
The general goal of Collaborative Virtual Environments (CVEs) is to provide a space within
which people may interact. CVEs are increasingly being used to support collaborative work
between geographically separated participants. User embodiment is concerned with the provision
of users with a representation of their choice so as to make others (and themselves) aware of
their presence in a virtual space. The taxonomy investigated in STAGE 1 detailed many of the
existing networked virtual environments and examined the fundamental interaction interfaces
which these systems provide. By initially discussing the features of communication which should
be supported regardless of the medium available, the following investigation revealed an
incomplete support for non-verbal communication cues over the range of the environments
examined.
1. Background
The first application of networked computer graphics appeared in 1972 on ARPANET, the
computer network developed by the Advanced Research Projects Agency. This network was
mainly intended for co-operative work and for sharing information. Today, multi-user virtual
environments are used for a variety of purposes, including shared scientific visualization,
training, co-operative work, battlefield simulation and entertainment games. Several platforms
exist for building multi-user virtual worlds, some of them free and easily accessible through the
Internet. Obviously, the performance of these systems is different from high-end applications
which are specialised, expensive and mostly running on dedicated networks. Although this gap is
shrinking, the future of networked environments which are able to accommodate a large number
of users and provide complex interfaces and rich user embodiments depends on aligning a
number of technical (networks, computer graphics capabilities, etc.) and social issues (telephone
companies, government regulators, etc.).
2. Non-Verbal Communication
New media such as distributed virtual environments, force researchers to analyse what is
fundamental about communication. Conversation relies on all channels of communication
through which information is exchanged by individuals during face-to-face interactions. Language
is closely linked with and supported by, non-verbal communication which adds to the meaning of
utterances, provides feedback, controls synchronisation and also plays a central role in human social
behaviour (Argyle, 1976).
Facial expressions: The face is one of the most important areas for non-verbal signaling. In
general, facial expressions are indicators of personality and emotions, serving also as interaction
signals. Facial expressions provide feedback and information about the listener's level of
understanding while revealing interest, puzzlement or disbelief. In addition, affective expressions
2 This work has been published in detail here:
Mania, K. & Chalmers, A. (1998). A Classification for User Embodiment in Collaborative Virtual Environments. Proc. of
the 4th International Conference on Virtual Systems and Multimedia (VSMM 98), 177-182. IOS Press - Ohmsha, Ltd.

4
allow listeners to infer the speaker’s current emotional state and communicate their audience's
emotional reaction to what is being said.
Gaze: Gaze is a general indicator of attention and can be directed at other conversational
participants in face-to-face interaction as well as at features of the physical environment. Gaze is
closely coordinated with verbal communication. It is used to obtain feedback on the other's
responses while talking and extra information about what is being said while listening. In
addition, shifts of gaze are used to regulate the synchronisation of speech. Gaze is also used as a
signal in starting encounters, in greetings, as a reinforcer and to indicate that a point is
understood.
Gestures: The hands and to a lesser extent the head and feet can produce a wide range of gestures.
Gestures are closely coordinated with speech and support multiple communication functions.
They are used to co-ordinate conversational content, achieve reference and assist in turn taking.
Conventional gestures are usually intended to communicate and are normally given and received
with full-awareness.
Posture: This is the information supplied by the orientation of a conversational participant's body.
Posture is an important means of conveying interpersonal attitudes and is associated with emotional
states. Posture accompanies speech in a way similar to that of gesture and provides feedback to the
speaker about how the message is being received. Body position and orientation can also be used to
include or exclude people from the conversation.
Self-Representation: Self-representation can be regarded as a special kind of non-verbal
communication. In general, the main purpose of manipulating appearance is to send messages
about one-self. Thus, people send messages about their social status, their occupation, their
personality or their mood. Appearance is also used to signal attitudes towards other people – for
example, aggression, rebelliousness and formality.
Bodily Contact: Physical touch seems to have a primitive significance of heightened intimacy
and it produces increased emotional arousal. Some forms of bodily contact are used as
interaction signals like greetings and farewells or as attention signals. However, the precise
meaning of a particular form of touch depends on the culture.
3. Taxonomy
Several distributed platforms were investigated.
DIVE: The Distributed Interactive Virtual Environment(DIVE) is an internet-based multiuser
virtual reality system developed by the Distributed Systems Laboratory of the Swedish Institute
of Computer Science (Carlsson & Hagsand, 1993). DIVE supports the development of shared
multi-user virtual environments, user interfaces and applications. Embodiments in DIVE have the
capability of head-movements, thus directing gaze while navigating.
MASSIVE: MASSIVE (Model, Architecture and System for Spatial Interaction in Virtual
Environments), a laboratory prototype from the University of Nottingham, UK, is a virtual
reality conferencing system which scales to large numbers of participants. Its users interact in the
same virtual world through a variety of different equipment, media and user interfaces(2D, 3D,
text, audio) (Benford et al., 1997). Head movement capabilities are available as well as a selection
of simple pre-programmed gestures such as sleeping (which is also used to indicate the user's
presence) and blushing.

5
VLNet: VLNet(Virtual Life Network) is a networked virtual environment developed in the
MIRALab of the University of Geneva and the Computer Graphics Lab of the Swiss Federal
Institute of Technology (Guye-Vuilleme et al., 1998). The system uses 3D human figures for avatar
representations. Virtual humans are directly controlled where face and joint representation is
modified using sensors attached to the user's body; user guided where the user defines tasks for
the embodiment to perform; and, autonomous that are self-governing and incorporate internal states
of actions.
Additional platforms investigated were dVs, OnLive!, Community Place, Quake, Worlds Chat,
SPLINE (Wilcox, 1998).
4. Discussion
The premise of this investigation was that communication is accomplished as a combination of
speech/language and non-verbal communication features. In addition, face-to-face interaction is
accompanied by involuntary expressions making communication live and more naturalistic. How
existing multi-user platforms incorporate non-verbal communication and the respective
interfaces concerned was examined. Most of the systems provided a limited set of gestures, facial
expressions or actions which are activated by mouse clicks on relevant buttons.
At the end of this investigation, it was clear that researchers cannot simulate the perceptual
complexity of real-world interactions and spaces in a straightforward way. Which aspects of
these interactions are essential to convey what needs to be communicated? STAGE 2 of this
research focused on simulating a real-world space and utilized an informal experimental design
comparing user task performance in the real-world space and computer graphics simulation
counterpart. The experimental methodology was based on a theory derived from cognitive
psychology research.

6
STAGE 23
SUMMARY
STAGE 2 outlines the experimental methodology employed and the relevant results derived from
the first, informally designed, preliminary study which compared spatial perception and memory
recall in a Virtual Environment (VE) displayed on a desktop monitor as well as on a Head Mounted
Display (HMD) with its real situation counterpart. The goal of this study was to identify the mental
processes participants followed after completing a memory task in addition to the amount of their
accurate recollections. The general scope was to identify variations of cognitive strategies (awareness
states) related to their processes of retrieval (visual or not visual) when task performance across
conditions does not differ. The actual task consisted of two parts: non-visual information recall for
participants experiencing a seminar-like situation and spatial recall of the environment where this
experience was taking place. For the non-visual part of the task, an audio-only condition was also
included in the experimental design. The computer graphics rendering of the real scene was non-
photorealistic, e.g. flat-shaded rendering. This preliminary study was designed to acquire a basic set of
data for the simplest rendering as well as HMD display (monocular, non-head tracked). This set of
elements will be built up in STAGE 2 to include photorealistic rendering, stereo graphics imagery and
more complex virtual interfaces such as head tracking. Participants were required, here, to complete a
memory task and provide self-reports of their level of perceived presence and simulator sickness, the
latter for the HMD condition.
1. Experimental Methodology
In STAGE 2, a methodology for simulation fidelity evaluation of VEs centred on a validated
theory of memory recall awareness states (the remember/know paradigm) is presented. Please
see STAGE 3 for a detailed description of this methodology from memory research. The actual
task consisted of two parts: non-visual information recall for participants experiencing a seminar-
like situation and spatial recall of the environment where this experience was taking place. The
study investigated how exposure to a computer-generated replica of the environment, displayed
on a typical desktop display and a HMD would compare to exposure to the same environment
and memory recall task in the real world from a cognitive rather than a task point of view.
Subjective measures such as memory awareness states selection and perceived presence
assessments were incorporated together with objective measures of memory recall, in a
comparative study of a VE against the real world. The remember/know paradigm focuses on the
actual awareness states that participants employ in order to complete a memory task rather than
on the actual scores of accurate memory recall. The resultant accurate seminar and spatial
memory recall scores and awareness states as well as participants’ sense of presence were
compared with those obtained from an analogous experiment in the actual real world space. The
extent to which judgements of memory recall, memory awareness states and presence in the
physical and VE are similar provides a measure for the fidelity of the simulation in question.
3 This work has been published in detail here:
• Mania, K. & Chalmers, A. (2001). The Effects of Levels of Immersion on Presence and Memory in Virtual
Environments: A Reality Centred Approach. Cyberpsychology & Behavior, 4(2), 247-264
• Mania, K., Chalmers, A., Troscianko, T., Hawkes, R. (2000). Presence and Task Performance: A Reality Centred
Approach. Technical Sketch, Proc. of ACM SIGGRAPH 2000, USA, ISBN 1-58113209-3, 245-245

7
Four groups of 18 participants were recruited to participate in this study from the student
population of the University of Bristol and Hewlett Packard Laboratories in Bristol, UK. 80% of
the subjects from each group were male. All used computers frequently in their daily activities.
Participants were randomly assigned to each group. A between-subject design was utilised
balancing groups for age and gender. Participants in all conditions were informed that they could
withdraw from participation at any time during the experiments and they were naïve as to the
purpose of the experiment. They were also asked if they had any knowledge relevant to the
historic topic of the seminar and if they did, they were excluded. Participants had either normal
or corrected-to-normal vision. According to the group they were assigned to, participants
completed the same memory task, in one of the following conditions:
1. In reality, attending to a 15-minute seminar in a seminar room in the University of
Bristol; referred to as the real-world condition.
2. Using a computer graphics simulation of the real world space with the real-world audio
on a desktop monitor; referred to as the desktop condition.
3. Using the same application on a monocular, non-head tracked HMD with the real world
audio and with a mouse for navigation; referred to as the HMD condition.
4. Listening to the audio recorded during the real-world condition and completing the non-
visual part of the task; referred to as the audio-only condition.
This study was based on the assumption that a 3D desktop display is less immersive than a
HMD. Audio used in one condition as the only experimental sensory stimulus is perceived, in this
study, as the least technologically ‘immersive’ condition.
Prediction
Presence and task performance were predicted to be significantly higher in the real-world
condition relative to the desktop, the HMD and the audio-only conditions, thus offering a high
benchmark. The main scope of this study was to show that by incorporating cognitive measures
together with task performance measures, variations of participants’ mental processes for memory
recall would be revealed across conditions.
The Real Situation
The first group of 18 participants attended a seminar presentation that took place in a specific
seminar room in the University of Bristol (Figure 1).
The seminar’s duration was 15 minutes. The historic content was chosen as none of the
participants had any prior knowledge on this matter. The lecturer utilised 12 slides on an
overhead projector. The seminar was digitally video recorded using a digital video camera on a
tripod. Subsequently, the audio was extracted (16-bit stereo, 44kHz) in order to be incorporated in the
computer graphics application for the desktop and HMD conditions.

8
Figure 1 The real seminar room and the computer graphics environment.
Table 1 Technical characteristic comparison between the desktop monitor and the HMD.
FoV (Field-of-View) Resolution Input Device
Desktop Monitor 38 degrees approx. hor. 1152*864 Mouse
HMD 30 degrees hor. XGA(1024*764) Mouse

9
The Graphical Simulation
The seminar room was modelled using the 3D Studio MAX modelling package and converted to
VRML (Figure 1). The geometry in the real room was measured using a regular tape measure
with accuracy of the order of one centimetre. The audio extracted during the real world seminar
was incorporated in the computer graphics application. The application included a slide-show
synchronised with the audio at the exact timings that the lecturer manipulated the slides in the
real seminar. A static billboard with a texture displaying the lecturer (who was always facing the
camera) was included in the application. The model was rendered flat-shaded and the application
had an average update rate of 45 frames per second for both the desktop and HMD condition.
The input device for navigation was a normal mouse in both the desktop and HMD conditions.
The second group of 18 participants used the desktop application which included the audio
recorded from the real seminar for the specified duration of the lecture (15 minutes) and their
navigation tendencies were informally monitored (desktop condition). The application was
displayed on a 21-inch typical desktop monitor. The Field-of-View (FoV) was calculated in
relation to the distance of the participant from the display. A third group of 18 participants used
the same application displayed on the HMD (HMD condition). A fourth set of 18 participants
just listened to the audio recorded during the real seminar and completed the part of the memory
task related to the seminar information. Obviously, the spatial perception task was not completed
since there was no visual stimulus for this group (audio-only condition).
The HMD employed was a HP Laboratories working prototype and, thus, was not a commercial
product available in the market. It was described as an ‘eye-glass’ display which features two
micro-displays and appropriate optics, one for each eye. Both eyes were presented with the same
image allowing for monocular imagery. Eyeglass displays allow for periphery vision and tend to
be smaller and much lighter than fully-fledged HMDs as shown in figures 4.3, 4.4. The
resolution of the desktop monitor employed in the desktop condition was kept at 1152*864;
respectively, the resolution of the HMD was 1024*764. This small difference of FoV and resolution
between the desktop monitor (38 degrees horizontal) and HMD (30 degrees horizontal) was
considered minimal since this study was preliminary (Table 1).
The real world was perceived as a control condition so the FoV of the participants in the real-
world condition was not restricted in this study. Participants in the desktop and HMD condition were
able to explore the room from a steady viewpoint, approximately placed in the centre of the room. They
had the ability to rotate on a full circle, horizontally, as well as on a half circle vertically, approximately
emulating the movement of the head, using a common mouse (Figure 3). The experimental room was
not darkened and participants in all conditions utilising computer graphics imagery were aware of their
surroundings.

10
Figure 2 The Hewlett Packard Laboratories HMD prototype.
Figure 3 Experimental set-up for the HMD condition (HP Laboratories, Bristol).
2. Materials
The four groups of participants were asked to fill in the same set of questionnaires after
exposure. This included the memory task and memory awareness states questionnaire and the
presence questionnaire (Slater et al., 1998) with the addition of the SSQ questionnaire (Kennedy et al.,

11
1993) for the HMD condition. These materials can be found in Appendix A1.
Memory recall task
The questionnaire relevant to the memory task was designed to test the participants’ accurate
memory recall of the information communicated in the seminar and their spatial awareness of the
environment and was based on the Conway et al., 1997 methodology. Overall, there were
twenty-two questions. Sixteen questions were related to the actual factual information
communicated in the seminar. The same set was incorporated in the real-world, desktop, HMD
and audio-only conditions. Six questions were relevant to the environment where the seminar
took place; these were incorporated in the real-world, desktop and HMD conditions. The correct
answers for nine of the questions related to the seminar were included in the slideshow and were
also mentioned by the lecturer. The remaining seven were only mentioned verbally and not included
in the actual slide show.
Each memory recall question had four possible answers and it included a confidence measure
with five possible states: No confidence, Low confidence, Moderate confidence, Confident, Certain.
Most importantly, it also included an awareness state measure with four possibilities: Remember,
Know, Familiar and Guess. Participants were required to select the correct answer for each
question according to their recollection, select a confidence level and also report on their strategy
of retrieval as expressed by one out of the four awareness states. Prior to filling out the core of
the questionnaire, participants were given instructions that were designed to explain what each of
the memory awareness states depicted as follows (Conway et al., 1997):
• You remembered a specific episode or image from the seminar. In this case you might have
images and feelings in mind relating to the recalled information. Perhaps you virtually ‘hear’
again or ‘see’ again the lecturer presenting some item of information or remember visually
the specific slide that information was included into. Answers such as these are called
REMEMBER answers.
• You might just ‘know’ the correct answer and the alternative you have selected just ‘stood
out’ from the choices available. In this case you would not recall a specific episode and
instead you would simply know the answer. Answers with this basis are called KNOW
answers.
• It may be, however, that you did not remember a specific instance, nor do you know the
answer. Nevertheless, the alternative you have selected may seem or feel more familiar than
any of the other alternatives. Answers made on this basis are called FAMILIAR answers.
• You may not have remembered, known, or felt that the choice you selected have been familiar.
In which case you may have made a guess, possibly an informed guess, e.g. some of the
choices look unlikely for other reasons so you have selected the one that looks least unlikely.
This is called a GUESS answer.
Presence
The second questionnaire was designed to measure participants’ level of perceived presence on a
Likert 7-point scale. The questions used in the Slater et al., 1998 study were adopted and the
questionnaire was administered in all conditions including the real world one. This particular set of
questions is exploring various aspects of the concept of presence itself and is not relevant to the

12
technology or interface used for the application. For example, issues investigated were the
dominance of the virtual world over the real one, the sense of visiting a ‘place’ versus viewing a scene
or listening to a sound and the level that the memory of the experiment resembled everyday
memories. Only questions of this nature could be applied to all four conditions without any per-
condition tailoring. The questionnaire included additional questions regarding gender, ratings of
background sounds, profession, level of computer-related expertise and level of losing track of time.
SSQ questionnaire
The widely used Simulator Sickness questionnaire (SSQ) was administered following
participants’ exposure to the VE for the HMD condition only. 16 symptoms were employed
indicated in the Kennedy et al., 1993 study. The questionnaire design is based on three
components: Nausea, Oculomotor problems and Disorientation. Participants report the degree to
which they experience each of the above symptoms as one of ‘none’, ‘slight’, ‘moderate’ and ‘severe’.
These are scored respectively as 0,1,2,3.
3. Summary of results
The incorporation of cognition-related measures, in this case, the report of the relevant memory
awareness state for each item of the memory recall task offered a valuable input towards a more
informative analysis. There was no statistical difference for the spatial memory task across
conditions, but prior probabilities relevant to memory awareness states showed that the
probability for an accurate response to fall under the ‘remember’ awareness state was higher for
the HMD condition compared to the real-world condition. Since ‘remember’ responses are linked
with visual mental imagery as a mechanism of retrieval, it could be argued that mental images
and subsequent memory responses associated with the HMD condition are more ‘vivid’ or
‘realistic’ and that could have an effect on spatial perception retained in time. It is therefore
suggested that usability studies involving only task performance measures while considering a
possible design or technology such as the Hewlett Packard HMD prototype, are not sufficient to
form conclusions regarding the effectiveness of the design or hardware in question. This is the
major premise of this work.
In general, presence did not follow the same trend as task performance in all cases. For example,
presence was significantly higher for the real-world condition compared to the desktop condition
but that was not reflected on memory recall. There was no statistical difference between the
scores for the spatial recall task between the real-world and the desktop condition. The presence
questionnaire also revealed no significant difference between the technological conditions. This
could mean that either these conditions do not have a varied effect on presence or that the measuring
device, in this case, the presence questionnaire could not pick up that difference. Inherently, this
could be an issue about the notion itself. There is an amount of ambiguity in terms of a scientific
representation of the notion that might reflect onto any possible measuring instrument.
Although the preliminary study gave confidence in the memory semantics methodology based on
the remember/know paradigm, the rendering used was basic and the spatial memory elements of
the task were limited. The preliminary study demonstrated the potential of the memory semantics
methodology as a simulation fidelity measure for VE applications in relation to the real world.
This measure focuses on the awareness states participants employ in order to complete a memory
task rather than on the actual scores of accurate completion commonly employed. The purpose of

13
the preliminary study was to adjust the memory semantics methodology for VE immersive
technology experimentation and reveal problems before a full study is made, therefore, the
experimental design was not strict. STAGE 3 incorporated a simpler spatial memory task, a
photorealistic stereo computer graphics simulation and head tracking. STAGE 3 forms the core
of this research.

14
STAGE 34
SUMMARY
This stage describes a methodology based on human judgments of memory awareness states for
assessing the simulation fidelity of a photorealistic Virtual Environment (VE) in relation to its
real scene counterpart. In order to demonstrate the distinction between task performance based
approaches and additional human evaluation of cognitive awareness states, a photorealistic VE
was created. Resulting scenes displayed on a Head Mounted Display (HMD) with or without
head tracking and desktop monitor were then compared to the real world task situation they
represented investigating spatial memory after exposure. Participants described how they
completed their spatial recollections by selecting one of four choices of awareness states after
retrieval in an initial test and a retention test a week after exposure to the environment. These
reflected the level of visual mental imagery involved during retrieval, the familiarity of the
recollection and also included guesses, even if informed. Experimental results revealed
variations in the distribution of participants’ awareness states across conditions while, in certain
cases, task performance failed to reveal any. Experimental conditions which incorporated head
tracking were not associated with confident visually-induced recollections. Generally, simulation
of task performance does not necessarily lead to simulation of the awareness states involved
when completing a memory task. The general premise of this research focuses on ‘how’ tasks are
achieved, rather than only on ‘what’ is achieved. The extent to which judgements of human
memory recall, memory awareness states and presence in the physical and VE are similar provides
a fidelity metric of the simulation in question.
1. Introduction
The mapping from the real world environment to the computer graphics environment is mediated
by environmental or visual fidelity (Waller et al., 1998). The term visual fidelity refers to the
degree to which visual features in the Virtual Environment (VE) conform to visual features in the
real environment. Interface or interaction fidelity refers to the degree to which the simulator
technology (visual and motor) is perceived by a trainee to duplicate the operational equipment
and the actual task situation. It is argued that training, for instance, in a VE with maximum
fidelity would result in transfer equivalent to real-world training since the two environments
would be indistinguishable (Waller et al., 1998). Robust metrics are essential in order to assess
the fidelity of VE implementations comprising of computer graphics imagery, display
technologies and 3D interaction metaphors across a range of application fields. Apart from
optimisation of technological characteristics such as resolution, Field-of-View (FoV), latency,
etc., one common belief is that efficient task performance measures should serve as fidelity
metrics for any application that mainly targets transfer of training in the real world (Bailey &
4 This work has been published in detail here:
• Mania, K., Troscianko, T., Hawkes, R., Chalmers, A. (2003). Fidelity Metrics for Virtual Environment
Simulations based on Human Judgments of Spatial Memory Awareness States. Presence, Teleoperators and
Virtual Environments Journal, 12(3), June 2003, MIT Press
• Mania, K., Chalmers, A., Troscianko, T., Hawkes, R. (2001). Simulation Fidelity Metrics for Virtual
Environments Based On Memory Semantics. Technical Sketch, Proc. of ACM SIGGRAPH 2001, USA, ISBN
1-58113-403-7, 258-258

15
Witmer, 1994, Waller et al., 1998, Lathrop & Kaiser, 2002). A commonly employed strategy,
therefore, for assessing the simulation fidelity of a VE is to compare task performance in a VE to
task performance in the real world scene represented in the VE. Another common approach is to
employ a cross-application construct, such as the sense of ‘presence’ to assess the effectiveness of a
VE or aspects of a VE according to its success in enhancing presence. There is a widespread belief that
presence should somehow improve task performance, although this has yet to be verified or indeed
reasons offered as to why this should be the case (Stanney et al., 1998).
The research at this stage argues that because of the wide-range of VE applications and
differences in participants across their background, ability and method of processing
information, an understanding of how tasks are undertaken within a VE complementing what is
achieved, is significant. This rationale is applied here to spatial memory recall. The utility of VEs,
regardless of the applications they are proposed for is predicated upon the accuracy of the spatial
representation formed in the VE. The framework to be presented has been drawn from traditional
memory research adjusted to form an experimental procedure in order to compare real scenes
and their computer graphics simulated counterparts. Here, participants could describe how they
achieved their spatial recollections after exposure to an environment by selecting one of four
awareness states (‘remember’, ‘know’, ‘familiar’ or ‘guess’) (Tulving, 1985, 1993, Conway et
al., 1997, Gardiner, 2000). These judgments reflect the level of visual mental imagery involved
at retrieval and the familiarity of the recollection including guesses, even if informed. In order to
demonstrate the varied distribution of cognitive activity even when task performance remains the
same, a photorealistic VE was created displayed on a Head Mounted Display (HMD) -
incorporating either mono or stereo rendering with or without head tracking - and desktop
display. Resulting scenes were then compared to the realworld task situation they represented
employing memory recall of elements of the space as well as report of awareness states on an initial
test and a retention test a week after the initial exposure. Central to this work is identifying whether
experimental conditions such as the real-world one and those incorporating head tracking (thus
including proprioceptive information) are associated with stronger visually-induced recollections
(‘remember’ awareness state) compared to conditions associated with a typical mouse interaction
interface. This work also aims to explore whether a cognitive shift between initial test and retest
is going to signify a performance shift. This study extends a preliminary study by Mania &
Chalmers, 2001.
2. Memory awareness states methodology
Memory, in the sense of ‘information’ for subsequent analysis, plays an important role in
perceptual systems such as the visual, auditory, haptic and kinesthetic. Memory is not a unitary
system (Baddeley, 1997). In the process of acquiring a new knowledge domain, visual or non-
visual, information retained is open to a number of different states. Some elements of a learning
experience or of a visual space may be ‘remembered’ linked to a specific recollection event and
mental image or could just pop-out, thus, could be just ‘known’. According to Tulving, 1985
recollective experiences are the hallmark of the episodic memory system. Knowing refers to those
in which there is no awareness of reliving any particular events or experiences, a mental theasurus
(semantic memory). Tulving, 1985 introduced a distinction between ‘remember’ and ‘know’
responses and provided the first demonstration that these responses can be made in a memory test,
item by item out of a set of memory recall questions, to report awareness states as well. He
reported illustrative experiments in which participants were instructed to report their states of

16
awareness at the time they recalled or recognised words they had previously encountered in a study
list. If they remembered what they experienced at the time they encountered the word, they made a
‘remember’ response. If they were aware they had encountered the word in the study list but did not
remember anything they experienced at that time, they expressed a ‘know’ response. The results
indicated that participants could quite easily distinguish between experiences of remembering and
knowing.
There is some preliminary evidence that the distinction between ‘remembering’ and ‘knowing’
reflects a difference in brain activity at the time of encoding (Smith, 1992). It is assumed that
recognition memory can be based largely on knowing, with little or no remembering. All that is
necessary for encoding into the semantic system is some initial awareness of events. In contrast,
encoding into episodic memory must depend on greater conscious elaboration of the events. Gregg
& Gardiner, 1994 showed that estimates of the strength of the memory trace are greater when
derived from remember plus know responses than when derived from only remember responses.
Knowing, thus, reflects an additional source of memory, not merely a difference in response
criteria. Although, ‘remember’ and ‘know’ awareness states have been controversially linked to
episodic and semantic memory types with ‘know’ responses more theoretically problematic,
recent research emphasised that ‘they can be used without commitment to any theory, but simply
to provide information on how various phenomena, including memory disorders, are characterised
experientially’ (Gardiner, 2000). In a relevant study, overall recognition performance in two groups
of participants was very similar, however, the reported states of awareness differed markedly.
One cannot make assumptions on what participants experience mentally from only their
performance, therefore, there is no alternative to the use of subjective reports. Thus, additional
information of awareness states provides an invaluable input into ‘how’ participants complete
recollections. Subsequent research to Tulving, 1985, summarised in Gardiner, 2000
demonstrated that some variables affect one or the other of the two states of awareness, that some
variables have opposing effects on them and that some variables have parallel effects on them. This
finding indicates that the two states of awareness are functionally independent.
Conway et al., 1997 argued that ‘familiarity’ can be defined as the feeling that something has been
encountered or experienced recently, although nothing about this recent occurrence can be
remembered. ‘Know’ responses, on the other hand, represent highly familiar memory items that
may come to mind without recollecting any particular encounter or any feeling of a recent
encounter and cannot be placed. Conway et al., 1997 showed that these finer grained judgements
could be dissociated from each other, just as different source memory judgements can. A
confidence scale cannot communicate awareness states. It is also suggested that when a new
knowledge domain is to be acquired, memory is represented initially in an episodic way. As time
goes by, the underlying representations may change such that they do not represent recollective
experiences and are simply ‘known’ leading to a semantic representation and schematised conceptual
knowledge. There is little evidence that feelings of familiarity reflect the semantic memory system
that supports highly familiar long-term knowledge. Gardiner, 2000, concludes: ‘… psychology of
memory should take on board subjective reports of conscious states and not just rely on more
conventional measures of performance. This evidence has established that the essential subjectivity of
remembering and knowing does not make reports of these states of awareness intractable to science’.
3. Experimental methodology
Five groups of 21 participants were recruited to participate in this study, from the University of

17
Bristol, UK undergraduate and M.Sc. student population and they received course credits for their
participation. 80% of the participants from each group were male. All used computers a great deal in
their daily activities. A between-subject design was utilised balancing groups for age and gender.
Participants in all conditions were informed that they could withdraw from participation at any time
during the experiments and they were naive as to the purpose of the experiment. Participants had
either normal or corrected-to-normal vision (self–report). According to the group they were
assigned to, participants completed the same memory task in one of the following conditions:
1. In reality, wearing custom made goggles to restrict their FoV, allowing for monocular
vision; referred to as the real-world condition.
2. Using a photorealistic computer graphics simulation on a monocular head-tracked HMD;
referred to as the HMD mono head tracked condition.
3. Using the same application on a stereo head-tracked HMD; referred to as the HMD
stereo head tracked condition.
4. Using the same application on a monocular HMD with a mouse interface; referred to as
the HMD mono mouse condition.
5. Using the same application displayed on a typical desktop monitor with a mouse interface,
wearing the same restrictive goggles as in the real-world condition; referred to as the desktop
condition.
A week after their experience, all participants were retested on the same memory task.
The Real Environment
The real environment consisted of a four by four meters room (Figure 4). Each wall of this room
had a different landmark; one wall consisted of a door and shelves, one wall of a door and a
greenboard, the third wall of a whiteboard and the fourth of smaller shelves on both its ends. The
existing window in the room was firmly covered with black lining to keep natural light out. The
light fixtures in the room were replaced with a standard incandescent bulb (assumed diffuse, light
emission in all directions). Several tables were placed close to the walls and 21 primitive objects of
approximately the same size (seven boxes, seven spheres and seven pyramids) were scattered
around the room, on the tables and shelves. All the objects were painted one shade of blue using the
same diffuse paint. A swivel chair was placed in the middle of the room.
The Computer Graphics Simulation
There was tight control over the visual appearance of the experimental space across realworld
and simulated conditions. The geometry in the real room was measured using a regular tape
measure with accuracy of the order of one centimetre. A photometry instrument (Minolta Spot
Chroma meter CS-100) was employed to measure the chromaticity CIE(x,y) and luminance (Y)
values of the light and materials in the real room. The Minolta chroma meter is a compact,
tristimulus colorimeter for non-contact measurements of light sources or reflective surfaces.
Luminance relates to the quality of a colour that most resembles the human’s notion of
brightness. Bright colours are generally of a high luminance and dark colours are generally of a
low luminance. The illuminant (light source) was measured by placing a white sheet of paper in
a specific position. Most of the materials (walls, objects, shelves, floor, plugframes) were measured at
the same position. To ensure accuracy, five measurements were recorded for each material, the

18
highest and lowest luminance magnitudes were discarded and an average was calculated of the
remaining three triplets. However, as this is a room in daily use some variations exist in all of the
surfaces due to texture, ageing and dirt.
The CIE (1931) colour space is based on colour matching functions derived by human
experimentation and it incorporates the trichromacy of the HVS. The usefulness of the CIE(x,y)
representation is that it allows colour specification in one language, however, equal geometric
steps of CIE(x,y) space do not correspond to equal perceptual steps. Before specifying display
colours, it is necessary to compute the tristimulus matrix of the display in question. In order to
compute the RGB tristimulus matrix, the chromaticity coordinates of the three display phosphors in
CIE(x,y) space are required. In addition, the chromaticity co-ordinates of the white that the three
phosphors of the display produce when turned on at their maximum are also required (Travis,
1991). Generally, the RGB system is a means for describing colours on a display monitor. It does
not take into account the energy that is produced in the physical world in terms of the distribution
over wavelength and also how the Human Visual System (HVS) responds to this distribution.
For the final measurements, the illuminant had to be taken into account. Measuring a diffuse
surface under a given light source results in Yxy values which include the contribution of the light
source itself. Incandescent bulbs are quite orange and fluorescent light is quite green, however, the
HVS perceives light in relative values and not as absolute measurements such as the ones out of the
chromameter. For example, if 1000 is the luminance in the real world, 100 the luminance of a real-
world material but 100 the luminance in the computer graphics simulation, then the luminance
for the simulated material needs to be 10 for the same ratio to be preserved.
The colour constancy attribute of the HVS, generally, is responsible for humans perceiving a
white sheet of paper as white under a wide range of illumination. If a participant is immersed into
a synthetic space on a display, theoretically, this should be true as well, however, the small size
of the displays prevents colour constancy from occurring. In relevant calculations for simulating
real-world illumination in a synthetic world, therefore, colour constancy needs to be enforced in the
rendering process since the HVS does not function as in the real world due to the nature of the
displays. The colour of the illuminant in RGB values was set as (1,1,1) for the radiosity
rendering, e.g. white.
In order to render the scene, the materials’ diffuse colour needs to be specified not the colour
observed under a particular light source. The final colour for each measured material in the scene
is estimated by dividing its RGB value by the RGB value of the observed white in the scene,
which is the colour of the light source in the scene. Using the relevant geometry and surfaces and
illuminant measurements converted to RGB triplets as input, the rendered model was created
using a radiosity rendering system (Figure 4). The final radiosity solution consisted of a finely
meshed model which could be interactively manipulated. This was the basis for the application
displayed on the desktop monitor and on the HMD. The desktop monitor and the HMD were
gamma corrected using the Minolta Spot Chromameter CS-100 in order to acquire relevant
luminance readings. When accurate colour specification is required as is often the case in scientific
applications, the non-linear relationship between display luminance and voltage is a significant
source of error and needs to be corrected to linearity.

19
Figure 4 The real-world and the computer graphics simulation.
Figure 5 The real-world and HMD mono/stereo condition (head-tracked).
3. Materials
The five groups of participants were asked to fill in the same set of questionnaires. This set
included the SSQ questionnaire (Kennedy et al., 1993) before and after the task, the memory task
and memory awareness states questionnaire and the presence questionnaire (Slater et al., 1998).

20
All participants across the five conditions completed the same memory task a week after the
initial experiment reporting on memory recall, confidence and awareness states.
Memory recall task
The memory recall questionnaire was designed to test the participants’ memory recall of the
positions and geometric shape of the 21 objects in the room. A diagram for each wall in the room
included numbered positions of objects in various locations. The diagrams were administered
together with the task questionnaire which consisted of 21 multiple choice questions representing
the 21 objects in the scene. Every question included three possible answers (box, sphere or
pyramid) and a confidence scale with five possible states: No confidence, Low confidence,
Moderate confidence, Confident, Certain. Every question also included an awareness states
report for every recollection, based on the memory awareness methodology offering four
choices: Remember, Know, Familiar or Guess. The participants were required to report on the
shape of the object in each numbered position on the diagram, starting with the positions they
were more confident they remembered. The design, thus, of the task questionnaire did not force
participants to start from a specified position in the room offering the capability to report,
initially, their most confident recollections. A pilot study was conducted in order to determine the
number of objects and, therefore, the number of questions of recall in relation to the exposure time
so as to avoid possible floor or ceiling effects (the task being too easy or too hard). Prior to filling
out the core of the task questionnaire, participants were given instructions designed to explain what
the memory awareness states depicted as follows:
• REMEMBER means that you can visualise clearly the object in the room in your head, in
that particular location. You virtually ‘see’ again elements of the room in your mind.
• KNOW means that you just ‘know’ the correct answer and the alternative you have selected
just ‘stood out’ from the choices available. In this case you can’t visualise the specific image
or information in your mind.
• FAMILIAR means that you did not remember a specific instance, nor do you know the
answer. It may seem or feel more familiar than any of the other alternatives.
• GUESS means that you may not have remembered, known, or felt that the choice you selected
have been familiar. You may have made a guess, possibly an informed guess, e.g. you have
selected the one that looks least unlikely.
Other measures
The presence questionnaire developed by Slater et al. 1998 was designed to measure the level of
presence on a Likert 7-point scale and was administered after the initial memory recall task
across conditions. The widely used Simulator Sickness questionnaire (SSQ) was administered
before and following participants’ exposure across conditions (Kennedy et al., 1993).
4. Procedures
The real-world condition
The SSQ questionnaire was administered before exposure. Following this procedure participants

21
were asked to wear any glasses or contact lenses they normally use when they have to focus at 2
meters distance (self-report). Subsequently, their dominant eye was identified by a widely used
‘sighting’ test. A pre-determined viewing position was set by manipulating the height of the
swivel chair according to the individual. Appropriate goggles were worn which restricted
participants’ FoV to 30 degrees to match the desktop and HMD’s FoV allowing for monocular
vision through the dominant eye only (Figure 5). The FoV was restricted in the real-world
condition to match the FoV of the displays. Although this action resulted in a ‘window’ to the
real world through the goggles, it was considered necessary in order to keep the FoV constant across
conditions. Participants were instructed that they would be guided to a room where they would spend
three minutes observing by rotating the swivel chair they would sit on placed in the middle of the
room, however, they were not aware of the post-exposure task. Navigational patterns and idle
time were monitored and recorded during exposure through a digital compass attached on the
swivel chair (Mania & Randell, 2002). After the set exposure time of three minutes, participants
were guided to the test room where the questionnaire pack was administered together with the
appropriate instructions.
The display conditions
The computer graphics application was displayed on a Kaiser Pro-View 30, gamma corrected
HMD (Figure 5). The viewpoint was set in the middle of the room and navigation was restricted
to a 360 degrees circle around that viewpoint and 180 degrees vertically in order to simulate
participants’ movement on the swivel chair in the real room (3 degrees of freedom). The geometric
FoV was calculated to be the same as the visual angle, through the goggles, in the real room. For the
HMD monocular conditions (headtracked and non-head-tracked) the dominant eye was identified and
the appropriate screen of the HMD was covered allowing for vision only through the dominant eye. For
the HMD stereo head tracked condition each participant’s interpupilary distance (IPD) was measured
and the stereo application’s parallax was set accordingly for the individual. For the desktop condition
utilizing a gamma corrected typical 21-inch desktop monitor, each participant’s dominant eye
was identified and the appropriate goggles were subsequently worn as in the real-world
condition. The frame of the monitor was covered with black cardboard to achieve a foreground
occlusion effect resulting in a stronger sense of depth. Horizontal rotation was monitored across
all conditions (Mania & Randell, 2002). There was no other source of light besides the HMD or
desktop display during exposure. The frame rate was retained at 14 frames per second across all
conditions. Although this is not a particularly high frame rate, it was considered adequate. The display
resolution was 640*480 (HMD maximum resolution) across technological conditions and the FoV
was constant (30 degrees) across all conditions including the realworld condition with restrictive
goggles fitted. The computer graphics rendering was computed taking into account real world
photometric measurements resulting in a photorealistic rendering as described in the previous
section. Texture mapping was applied only on the doors and tables in the room.
5. Memory awareness states’ statistical analysis
Awareness state data were represented as prior and posterior probabilities. Koriat & Goldsmith,
1994 have drawn an important distinction between the amount or quantity remembered
compared to the accuracy or quality of what is remembered. In the quantity analysis memory
awareness states data are represented as a priori or prior probabilities. Although this notation
does not follow the Bayesian probability theory principles for ‘prior’ probabilities, it is going to

22
be adopted as such in this paper following the characterisations of Koriat & Goldsmith, 1994 as
well as Conway et al., 1997. Prior probabilities are obtained by calculating the proportions of correct
answers falling in each of the four memory awareness categories for each participant. In the
accuracy analysis, correct recall scores are represented as posteriori or posterior probabilities. In
order to calculate posterior probabilities, the proportion of correct answers from the total of
answers given in each memory awareness category is computed for each participant.
For participant n,
in
xis the number of correct answers for the i awareness state,
'
in
xis the number of correct answers for the i awareness state,
i = { remember, know, familiar, guess } = { 1, 2, 3, 4 } then,
in
Pis the prior probability for awareness state i related to participant n (Equation 3.1),
∑
=
=4
1i
in
in
in
x
x
P
'
in
P is the posterior probability for awareness state i related to participant n (Equation 3.2),
'
'
inin
in
in xx
x
P+
=
Generally, prior probabilities reflect the following: Given that the response of a participant is
correct, what is the probability that the participant has chosen a particular state on that question?
Posterior probabilities, on the other hand, pose the following question: Given that a response of a
participant was assigned to one of the four memory awareness response categories, what is the
probability that the response is correct? For the purpose of this study each memory recall question
included a 5-scale confidence scale and a choice between ‘remember’, ‘know’, ‘familiar’ as well as
‘guess’ awareness states. The goal of this strategy was to identify the distributions of awareness
states responses across conditions focusing on visually induced recollections. This could reveal
variations that wouldn’t be possible by just counting right and wrong answers.
6. Discussion and summary of results
This investigation focuses on the effect of different viewing conditions (direct perception of
objects in a real-world setting versus perception of the computer graphics representation of this
setting) on observers' attributions regarding object-location memory. Accuracy of performance
per se is an imperfect reflection of the cognitive activity that underlies performance in memory
tasks. Accurate memory can be supported by either a recollection of prior specific experience
(remembering) or reliance on a general sense of knowing with little or no recollection of the
source of this sense (knowing) including familiarity and guesses even if informed. Training in a
VE system capable of perfectly simulating the real world should result in the same training effect

23
as that in the real world. The participants who mentally visualised the room and the objects in the
room during retrieval had a higher proportion of correct responses under the ‘remember’
awareness state. The participants that employed mnemonics’ strategies based on words instead of
visually retaining elements of the space reported the ‘know’ awareness state which resulted in a
proportion of correct responses linked with the ‘know’ awareness state. If a weaker trend of non-
visually induced recollections is employed by participants towards stronger visually induced
recollections linked to the ‘remember’ awareness state, it could be assumed that their mental
representation of a space involved more ‘vivid’ recollections.
There was a significant main effect of condition upon the ‘remember’ awareness state. It was
anticipated that the amount of correct ‘remember’ responses would be higher in conditions
incorporating more ‘naturalistic’ interfaces such as head tracking. However, results revealed that
the proportion of correct responses linked with the ‘remember’ awareness state was significantly
higher for the HMD mono mouse condition compared to the HMD mono head tracked and HMD
stereo head tracked conditions (initial task). Crucially, these responses correlated positively with
confidence scores. Therefore, an interface of high simulation fidelity such as head tracking does
not always correspond to visually induced memory awareness states. A similar result was revealed in
a preliminary study by Mania & Chalmers, 2001. If specific applications require a high amount of
recollections based on visual mental imagery, a ‘natural’ interface such as head tracking may not be
appropriate. Therefore, desirable variations of awareness states for specific application purposes could
be identified. It could be true, for instance, that for flight simulation applications it is crucial for trainees
to achieve a high level of visually induced recollections related to instruments as opposed to feelings of
familiarity of even confident recollections which are not accompanied by visual imagery. If
‘reality’ is associated to the degree of similarity to the real world task situation then, in this case, the
HMD mono mouse condition is not very ‘real’. The awareness states distribution is affected by
the degree of ‘realism’ of the motor response. Word based mnemonics and, generally,
recollections that were not linked to visually induced recollections were identifiable by the high
proportion of correct ‘know’ responses. The utilisation of a viewing method such as the HMD together
with an ‘unreal’ motor response such as the mouse, appeared to have prevented participants
employing non-visually induced recollections and resulted in a larger distribution of correct
responses assigned to the ‘remember’ awareness state. By decreasing the degree of ‘reality’ of
the motor response, participants -paradoxicallyadopted visually induced recollections. Achieving
high fidelity could incorporate the need for similar awareness states between a real-world task
situation and its computer graphics simulation. Here, something less ‘real’, therefore, less
computationally expensive but more demanding because of its novelty may restore a more
‘naturalistic’ or desirable awareness state. Research could identify such issues by using
methodologies that allow investigations based on the cognitive activity expressed by awareness
states responses. Additionally, a significant shift of correct ‘remember’ responses in the initial
task to correct ‘guess’ responses in the retest was observed. This shift was observed across all
conditions and it did signify a lower amount of correct recollections between initial test and
retest.
The task employed in this study did not allow for free navigation around the experimental space. The
FoV was restricted in the real-world setting to match the FoV of the displays for methodological
reasons. Future work could include a task which would allow freedom of navigation and also a
testing strategy which would incorporate transfer of training in the real-world. Matching
participants’ performance in simulations to performance in a real-world situation does not guarantee
that the cognitive activity linked with performance will be similar across the simulated conditions.

24
Task performance scores could, therefore, be taken into account according to specific awareness
states. By employing methodologies, such as the memory awareness states methodology,
computer graphics and VE technology research could exploit human perceptual mechanisms
towards successful applications.

25
REFERENCES
Argyle M. (1976). Bodily Communication, Methuen & Co Ltd, London. Baddeley, A.
(1997). Human Memory, Theory and Practice. Psychology Press.
Bailey, J.H., Witmer, B.G. (1994). Learning and Transfer of Spatial Knowledge in a Virtual
Environment. Proc. of the Human Factors & Ergonomics Society 38th Annual Meeting, 1158-1162,
Santa Monica, CA: Human Factors & Ergonomics Society.
Benford S., Bowers J., Fahlen L.E., Greenlhalgh C., Snowdon D. (1997). Embodiments, avatars,
clones and agents for multi-user, multi-sensory virtual worlds, Multimedia Systems, Springer-Verlag.
Carlsson C. Hagsand, O. (1993).DIVE A multi-user virtual reality system”, IEEE Virtual Reality
Annual International Symposium (Cat. No.93CH3336-5).
Conway, M.A., Gardiner, J.M., Perfect, T.J., Anderson, S.J., Cohen, G.M. Changes in memory
Awareness during Learning: The Acquisition of Knowledge by Psychology Undergraduates.
Journal of Experimental Psychology, Vol. 126, No4, 393-413, 1997.
Coolican, H. (1999). Research Methods and Statistics in Psychology, 3rd edition. Hodder &
Stoughton.
Demuynck, K. Broeckhove, J. Arickx, F. (1996) “The impact of communication mechanisms on
the performance in a distributed virtual reality system”, Proc. Of the Internat ional Conference and
Exhibition HPCN EUROPE.
Gardiner, J.M. (2000). Remembering and Knowing. In the E. Tulving and F.I.M. Craik (Eds.)
Oxford Handbook on Memory. Oxford University Press.
Gregg, V.H., Gardiner, J.M. (1994). Recognition Memory and Awareness: A Large Effect of
Study-Test Modalities on ‘Know’ Responses Following a Highly Perceptual Orienting Task.
European Journal of Cognitive Psychology, 6(2), 131-147.
Guye-Vuilleme A., Capin T. K., Pandzic I.S., Thalmann N.M, Thalmann D. (1998). Nonverbal
Communication Interface for Collaborative Virtual Environments, Proc. of CVE'98, Manchester,
UK.
Kennedy, R.S., Lane, N.E., Berbaum, K.S., Lilienthal, M.G. (1993). Simulator Sickness
Questionnaire: An Enhanced method for Quantifying Simulator Sickness. The International
Journal of Aviation Psychology, 3(3), 203-220.
Koriat, A., Goldsmith, M. (1994). Memory in Naturalistic and Laboratory Contexts: Distinguishing
the accuracy oriented and quantity oriented approaches to memory assessment. Journal of
Experimental Psychology: General, 123, 297-315.

26
Lathrop, W.B. Kaiser, M.K. (2002). Perceived Orientation in Physical and Virtual Environments:
Changes in Perceived Orientation as a Function of Idiothetic Information Available. Presence:
Teleoperators and Virtual Environments,11(1), 19-32. MIT Press.
Mania, K. (2001). On Simulation Fidelity and Naturalism of Virtual Interfaces (2001). Proc. of the
IEEE VR 2001 workshop on ‘The Future of VR and AR Interfaces: Multimodal, Humanoid,
Adaptive and Intelligent’, GMD Publications.
Mania, K. (2001). Connections between Lighting Impressions and Presence in Real and Virtual
Environments: An Experimental Study. Proc. of Afrigraph 200. ACM Publications.
Mania, K. (2001). Fidelity Metrics for Virtual Environment Simulations based on Spatial Memory
Awareness States. Ph.D thesis, University of Bristol, UK, Dept. of Computer Science.
Mania, K. (2001). Strategies for the Assessment of Simulation Fidelity in VEs. In Billinghurst, M.,
Ellis, S., Mania, K., Steed, A. Usability Evaluation Techniques for Virtual Reality Technologies course
notes, IEEE Virtual Reality 2001, Japan, 5-27.
Mania, K. (2002). Spatial Perception, Memory and Performance in Synthetic Worlds. In Biocca,
F., Brooks, F.P. Jr., Ellis, S., Mania, K., Slater, M., Steed, A., Whitton, M. Understanding Virtual
Environments: Immersion, Presence, and Performance. ACM Siggraph 2002 course notes, San
Antonio, USA, full-day course.
Mania, K. (2002). Realism and Fidelity in Computer Graphics. In Billinghurst, M., Ellis, S.,
Mania, K., Steed, A. Usability Evaluation Techniques for Virtual Reality Technologies: Human
Factors Issues, course notes, IEEE Virtual Reality 2002, USA, 3068.
Mania, K. & Chalmers, A. (1998). A Classification for User Embodiment in Collaborative Virtual
Environments. Proc. of the 4th International Conference on Virtual Systems and Multimedia
(VSMM 98), 177-182. IOS Press - Ohmsha, Ltd.
Mania, K. & Chalmers, A., (1999). Between Real and Unreal: Investigating Presence and Task
Performance, A Pilot Study. Proc. of 2nd International Workshop on Presence, University of Essex,
Colchester, UK. ISBN 9-386-1571-X.
Mania, K. & Chalmers, A. (2000). A User-Oriented Methodology for Evaluating Virtual
Environment Applications. Proc. of 3rd International Workshop on Presence, Delft, Netherlands.
ISBN 9-386-1571-X.
Mania, K. & Chalmers, A. (2001). The Effects of Levels of Immersion on Presence and Memory
in Virtual Environments: A Reality Centred Approach. Cyberpsychology & Behavior, 4(2), 247-
264.
Mania, K., Chalmers, A., Troscianko, T., Hawkes, R. (2000). Presence and Task Performance: A
Reality Centred Approach. Technical Sketch, Proc. of ACM SIGGRAPH 2000, 245.

27
Mania, K., Chalmers, A., Troscianko, T., Hawkes, R. (2001). Simulation Fidelity Metrics for Virtual
Environments based on Memory Semantics. Technical Sketch, Proc. of ACM SIGGRAPH 2001,
258-258.
Mania, K., Chalmers, A. (2001). The Effects of Levels of Immersion on Presence and Memory in
Virtual Environments: A Reality Centred Approach. Cyberpsychology & Behavior, 4(2), 247-264.
Mania, K., Randell, C. (2002). Monitoring Navigational Strategies and Idle Time in Real and Virtual
Environments: An Experimental Study. Proc. of the 8th International Conference on Virtual
Systems and Multimedia 2003 (VSMM), Korea, Kiwisoft Company Ltd., 327-335.
Slater, M., Steed, A., McCarthy, J., Maringelli, F. (1998). The Influence of Body Movement on
Subjective Presence in Virtual Environments. Human Factors: Journal of the Human Factors
Society, 40(3), 469-477.
Smith, M.E. (1992). Neurophysiological Manifestations of Recollective Memory Experience
during Recognition Memory Judgements. Journal of Cognitive Neuroscience, 5, 1-13.
Stanney, K.M., Salvendy, G., Deisigner, J., DiZio, P., Ellis, S., Ellison, E., Fogleman, G., Gallimore, J.,
Hettinger, L., Kennedy, R., Lackner, J., Lawson, B., Maida, J., Mead A., Mon-Williams, M.,
Newman, D., Piantanida, T., Reeves, L., Riedel, O., Singer, M., Stoffregen, T., Wann, J., Welch,
R., Wilson, J., Witmer, B. (1998). Aftereffects and Sense of Presence in Virtual Environments:
Formulation of a research and development agenda. Report sponsored by the Life Sciences
Division at NASA Headquarters. International Journal of Human-Computer Interaction, 10(2),
135-187.
Travis, D. (1991). Effective Color Displays. Academic Press.
Tulving, E. (1985). Memory and Concioussness. Canadian Psychologist, 26, 1-12.
Tulving, E. (1993). Varieties of Concioussness and Levels of Awareness in Memory. In A.D.
Baddeley and L. Weiskrantz (Eds.), Attention: Selection, Awareness and Control. A tribute to
Donald Broadbent, 283-299. London: Oxford University Press.
Waller, D., Hunt, E., Knapp, D. (1998). The Transfer of Spatial Knowledge in Virtual
Environment Training. Presence: Teleoperators and Virtual Environments, 7(2), MIT Press.
Wilcox S.K. (1998). Guide to 3D Avatars, John Wiley and Sons Inc.