Content uploaded by Panagiotis Kourtesis
Author content
All content in this area was uploaded by Panagiotis Kourtesis on Oct 28, 2024
Content may be subject to copyright.
Citation: Kourtesis, P.; Linnell, J.;
Amir, R.; Argelaguet, F.; MacPherson,
S.E. Cybersickness in Virtual Reality
Questionnaire (CSQ-VR): A
Validation and Comparison against
SSQ and VRSQ. Virtual Worlds 2023,2,
16–35. https://doi.org/10.3390/
virtualworlds2010002
Academic Editors: Radu Comes,
Dorin-Mircea Popovici, Calin
Gheorghe Dan Neamtu and
Jing-Jing Fang
Received: 19 December 2022
Revised: 9 January 2023
Accepted: 17 January 2023
Published: 29 January 2023
Copyright: © 2023 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
Cybersickness in Virtual Reality Questionnaire (CSQ-VR): A
Validation and Comparison against SSQ and VRSQ
Panagiotis Kourtesis 1, 2, * , Josie Linnell 2, Rayaan Amir 2, Ferran Argelaguet 3,4 and Sarah E. MacPherson 2
1Department of Psychology, National and Kapodistrian University of Athens, 15772 Athens, Greece
2Department of Psychology, University of Edinburgh, Edinburgh EH8 9YL, UK
3Institut National de Recherche en Sciences et Technologies du Numérique (Inria), de l’Universitéde Rennes,
35042 Rennes, France
4Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA), de Centre National de la Recherche
Scientifique (CNRS), 35042 Rennes, France
*Correspondence: pkourtesis@psych.uoa.gr
Abstract:
Cybersickness is a drawback of virtual reality (VR), which also affects the cognitive and
motor skills of users. The Simulator Sickness Questionnaire (SSQ) and its variant, the Virtual Reality
Sickness Questionnaire (VRSQ), are two tools that measure cybersickness. However, both tools
suffer from important limitations which raise concerns about their suitability. Two versions of
the Cybersickness in VR Questionnaire (CSQ-VR), a paper-and-pencil and a 3D–VR version, were
developed. The validation of the CSQ-VR and a comparison against the SSQ and the VRSQ were
performed. Thirty-nine participants were exposed to three rides with linear and angular accelerations
in VR. Assessments of cognitive and psychomotor skills were performed at baseline and after each
ride. The validity of both versions of the CSQ-VR was confirmed. Notably, CSQ-VR demonstrated
substantially better internal consistency than both SSQ and VRSQ. Additionally, CSQ-VR scores had
significantly better psychometric properties in detecting a temporary decline in performance due to
cybersickness. Pupil size was a significant predictor of cybersickness intensity. In conclusion, the
CSQ-VR is a valid assessment of cybersickness with superior psychometric properties to SSQ and
VRSQ. The CSQ-VR enables the assessment of cybersickness during VR exposure, and it benefits
from examining pupil size, a biomarker of cybersickness.
Keywords:
cybersickness; virtual reality; SSQ; VRSQ; sensitivity; cognition; reaction time; motor
skills; eye tracking; pupil size
1. Introduction
Virtual reality (VR) is a promising form of technology that facilitates applications
in many areas, such as education [
1
], professional training [
2
], cognitive assessment [
3
],
mental health therapy [
4
], and entertainment [
5
]. Nevertheless, beyond the advantages
that VR brings to these fields, a limitation of VR is the presence of cybersickness that affects
a percentage of users [
6
]. Cybersickness symptomatology includes nausea, disorientation,
and oculomotor symptoms. Although there are similarities between cybersickness and
simulator sickness, cybersickness differs from simulator sickness in terms of the frequency
and severity of the types of symptoms [
7
]. Specifically, users experiencing cybersickness
report increased general discomfort due to nausea and disorientation-related symptoms [
7
].
Cybersickness also differs from motion sickness as cybersickness is triggered by visual
stimulation, while motion sickness is triggered by actual movement [8].
Although there is not a comprehensive theoretical framework for cybersickness, the
most frequent and predominant one is the sensory conflict theory [
6
,
8
,
9
]. This theoretical
framework suggests that cybersickness symptomatology stems from a sensorial conflict
between the vestibular (inner ear) and the visual system [
6
,
9
]. In simple terms, the percep-
tion of postural balance relies on a combination of visual, vestibular, and proprioceptive
Virtual Worlds 2023,2, 16–35. https://doi.org/10.3390/virtualworlds2010002 https://www.mdpi.com/journal/virtualworlds
Virtual Worlds 2023,217
input. Conflicting motion perception cues of the visual, proprioception, and vestibular
systems postulate to cause cybersickness. The technological reason for this conflict is
vection, an illusory sense of motion that occurs in VR. Vection is one of the main reasons for
experiencing cybersickness in VR [
10
,
11
]. Specifically, motions such as linear and angular
accelerations appear to induce cybersickness in the user.
1.1. Cybersickness, Cognition, and Motor Skills
Beyond the obvious decrease in user experience in VR, cybersickness may also nega-
tively affect the cognitive and/or motor performance of the user. Given that VR is used for
applications that require intact cognitive and motor abilities (e.g., educational, research,
clinical, and training applications), the presence of cybersickness has serious consequences
for the implementation of VR in these applications. Recent systematic reviews of the litera-
ture suggest that cybersickness may substantially, yet temporarily, decrease the cognitive
and/or motor performance of the user in immersive VR studies [
12
–
14
]. Dahlman et al. [
15
]
postulated that motion sickness significantly decreases users’ verbal working memory.
Comparably, in immersive VR, Varmaghani et al. [
16
] conducted a study (N = 47) in which
the participants formed two groups: a VR group (N = 25) and a control group (N = 22;
playing a board game). The results indicated that the VR group did not show an increase in
visuospatial processing ability, while the control group did. Thus, this outcome postulated
that cybersickness affects visuospatial processing and/or learning ability.
In another study, Mittelstaedt et al. [
17
] examined cybersickness and cognition (reac-
tion time, spatial processing, visuospatial working memory, and visual attention processing)
in pre- and post-sessions in VR. The findings showed that cybersickness modulated a slower
reaction speed and prevented an expected improvement in visual processing speed [
17
].
These results suggest that cybersickness has a negative effect on attentional processing
and reaction times, while spatial abilities and visuospatial memory remain intact. In the
same vein, the studies of Nalivaiko et al. [
18
] (N = 26) and Nesbitt et al. [
10
] (N = 24)
examined the effect of cybersickness on reaction times. In both studies, reaction speed
substantially slowed. Interestingly, slower reaction times were significantly correlated with
an increase in the intensity of cybersickness [
10
,
18
], indicating that cybersickness intensity
may be associated with temporary cognitive and/or motor decline. However, no study
has examined whether cybersickness intensity predicts cognitive or motor decline. Finally,
while the above studies support the notion that cybersickness may decrease cognitive
and/or motor skills, they all evaluated cybersickness after VR exposure. No study has
assessed cybersickness during exposure.
1.2. Cybersickness Questionnaires
The Simulator Sickness Questionnaire (SSQ) is a 4-point Likert scale that was designed
to assess simulator sickness in aviators [
19
]. The SSQ is the tool that has been used
most frequently to measure cybersickness due to exposure to VR [
13
]. However, simulator
sickness differs from cybersickness symptomatology. In the latter, disorientation and nausea
symptoms are more frequent and intense [
7
]. Thus, despite its use in VR studies, the SSQ is
not specific to cybersickness symptoms that a user may experience in VR. Indeed, a recent
study showed that the SSQ does not have adequate psychometric properties to evaluate
cybersickness in VR [
20
]. However, there is a variant of the SSQ, namely the VR sickness
questionnaire (VRSQ), that was recently developed [
21
] using items directly derived from
the SSQ. In the development and validation study of the VRSQ, researchers attempted
to isolate the items of the SSQ that are pertinent to cybersickness [
21
]. Nevertheless, this
development and validation study suffered from serious limitations. Firstly, the sample size
was small (i.e., 24 participants), and the stimuli diversity was limited. Notably, the factor
analyses accepted only items pertinent to oculomotor and disorientation symptoms, while
they rejected all items pertinent to nausea (i.e., 7 items) [
21
]. The latter is very problematic
because it is well-established that nausea is the second (after disorientation) most frequent
type of symptom of cybersickness [
7
,
22
–
24
]. Furthermore, both the SSQ and the VRSQ
Virtual Worlds 2023,218
examine symptoms after VR exposure (not during) and produce scores that cannot be easily
interpreted. Finally, when developing the SSQ and the VRSQ, the available guidelines for
designing and developing a Likert scale tool were not considered.
There is scientific consensus regarding the design of Likert scale questionnaires. The
literature suggests that a 7-point Likert scale is substantially better than a 5-point (or less)
one [
25
–
28
]. The 7-point design offers a greater variety of responses, which better captures
the diversity of the individuals’ views or experiences. Furthermore, combining numbers
(e.g., 6) with corresponding text (e.g., strongly disagree) facilitates a better understanding
of the differentiation between the available responses [
25
–
28
]. These suggestions have been
considered and adopted in the development of the VR Neuroscience Questionnaire (VRNQ)
and the Cybersickness in VR Questionnaire (CSQ-VR) [
29
]. The CSQ-VR is derived from the
VR Induced Symptoms and Effects (VRISE) section of the VR Neuroscience Questionnaire
(VRNQ), which has been found to have very good structural and construct validity [
29
].
Additionally, the VRISE section of the VRNQ has been validated against the SSQ and the
Fast Motion Sickness Scale [
30
]. The advantages of the VRISE over the SSQ pertain to its
short administration (only 5 items/questions) and its production of easily comprehensible
outcomes [
30
]. However, the scoring of the VRISE is inverse (i.e., higher scores indicate
milder symptom intensity). In addition, oculomotor symptoms were assessed by only one
question in the VRISE.
The CSQ-VR was designed in line with the aforementioned guidelines (i.e., using
a 7-point scale and combining text with numbers), while also addressing the previous
shortcomings (i.e., inverse scoring and one oculomotor question) of the VRISE section of
the VRNQ. The CSQ-VR assesses the whole range of cybersickness symptoms, including
nausea, disorientation, and oculomotor symptoms. There are two questions for each type of
symptom. Each question is presented on a 7-item Likert Scale and the responses are offered
in the form of combined text and numbers, ranging from “1-absent feeling” to “7-extreme
feeling”. The CSQ-VR produces a total score and three sub-scores: nausea, disorientation,
and oculomotor. Each sub-score corresponds to a type of symptom and is calculated by
adding the two corresponding responses. The total score is the sum of the three sub-scores.
The design of the CSQ-VR yielded the maintenance of the advantages associated with
the VRISE of the VRNQ (i.e., very short administration, easy and interpretable scoring,
comprehensible questions and responses, and an examination of all types of cybersickness
symptoms) and the improvement of the weaker aspects (i.e., the addition of one more ocu-
lomotor question and positive scoring, where larger numbers indicate stronger symptoms).
Finally, the CSQ-VR was not only developed in a paper-and-pencil form, but also in a 3D
form that can be used in any virtual environment to examine cybersickness while the user
is in VR. This VR version of the CSQ-VR also benefits from eye tracking to measure gaze
fixations and pupil size (i.e., pupillometry). Because pupil size is associated with negative
emotions [31], pupillometry may offer a physiological metric of cybersickness intensity.
1.3. Research Aims
This study aims to examine the validity of the paper-and-pencil version and the VR
version of the CSQ-VR in detecting and evaluating cybersickness symptoms. The validity
is examined against that of the SSQ and the VRSQ, which are considered valid tools to
measure cybersickness. Furthermore, as there is an association between cybersickness and
cognitive and motor performance, this study offers a comparison between the CSQ-VR
(both versions), the SSQ, and the VRSQ in detecting temporary cognitive and/or motor
decline due to cybersickness. The VR version of the CSQ-VR is expected to facilitate an
ongoing examination of cybersickness while the user is immersed. Finally, the utility of
pupillometry in predicting cybersickness intensity is also be explored.
Virtual Worlds 2023,219
2. Materials and Methods
2.1. Virtual Environment Development
The virtual environment was developed using the Unity3D game engine. The in-
teractions with the environment were developed using SteamVR SDK. Because gaming
experience may modulate task performance [
3
], the virtual hands/gloves of SteamVR SDK
were used to ensure an ergonomic and effortless interaction. Notably, none of the inter-
actions required button presses. Instead, interactions were facilitated by simply touching
the object (initial selection) and continuously touching the object (to confirm the selection).
In addition, SteamVR virtual hands/gloves do not represent any gender or race, so their
utilization prevents confounding effects from these variables [32].
To ensure an understanding and the seamless completion of the tasks, users received
instructions in video, audio, and written form. For each task’s instructions, audio clips
with neutral naturalistic voices were produced using Amazon Polly. The audio feedback
was spatialized using the SteamAudio plugin. SRapinal SDK was used for eye tracking
and facilitating pupillometry. Finally, randomization of the experimental blocks within and
between participants and the extraction of the data into a CSV file, as well as the facilitation
of the experimental design and control, were achieved using bmlTUX SDK [33].
Linear and Angular Accelerations in VR
Based on the relevant literature, linear and angular accelerations are efficient in
inducing significant cybersickness symptoms in users in a relatively short time (e.g.,
5–10 min
) [
10
,
12
,
13
,
18
,
24
,
29
,
34
]. Correspondingly, a ride of 5 min was developed. Be-
cause the ride had to be repeated three times (i.e., a total 15 min ride) for each participant,
a 5 min duration was preferred. The ride was designed as an animation of the platform
that the user was standing on (see Figure 1). The direction of motion was always forward
(except in the last stage; see reversed z-axis). The movements of the platform were similar to
those of a roller coaster. The ride included the following accelerations in this specific order:
(1) linear (z-axis); (2) angular (z- and y-axes); (3) angular (z-, x-, and y-axes); (4) angular
(roll axis); (5) extreme linear (z-axis); (6) angular (yaw axis); and (7) extreme linear (y-axis
followed by reversed z-axis). The environment had simple black-and-white surround-
ings (see Figure 1). This background was used to ensure that the symptoms were strictly
induced by vection and not due to other reasons, such as intense colors. Additionally,
having the squared/tiled design offered cues for the participants to perceive vection and
altitude changes.
Virtual Worlds 2023, 2, FOR PEER REVIEW 4
ongoing examination of cybersickness while the user is immersed. Finally, the utility of
pupillometry in predicting cybersickness intensity is also be explored.
2. Materials and Methods
2.1. Virtual Environment Development
The virtual environment was developed using the Unity3D game engine. The inter-
actions with the environment were developed using SteamVR SDK. Because gaming ex-
perience may modulate task performance [3], the virtual hands/gloves of SteamVR SDK
were used to ensure an ergonomic and effortless interaction. Notably, none of the interac-
tions required button presses. Instead, interactions were facilitated by simply touching
the object (initial selection) and continuously touching the object (to confirm the selection).
In addition, SteamVR virtual hands/gloves do not represent any gender or race, so their
utilization prevents confounding effects from these variables [32].
To ensure an understanding and the seamless completion of the tasks, users received
instructions in video, audio, and written form. For each task’s instructions, audio clips
with neutral naturalistic voices were produced using Amazon Polly. The audio feedback
was spatialized using the SteamAudio plugin. SRapinal SDK was used for eye tracking
and facilitating pupillometry. Finally, randomization of the experimental blocks within
and between participants and the extraction of the data into a CSV file, as well as the
facilitation of the experimental design and control, were achieved using bmlTUX SDK
[33].
2.1.1. Linear and Angular Accelerations in VR
Based on the relevant literature, linear and angular accelerations are efficient in in-
ducing significant cybersickness symptoms in users in a relatively short time (e.g., 5–10
min) [10,12,13,18,24,29,34]. Correspondingly, a ride of 5 min was developed. Because the
ride had to be repeated three times (i.e., a total 15 min ride) for each participant, a 5 min
duration was preferred. The ride was designed as an animation of the platform that the
user was standing on (see Figure 1). The direction of motion was always forward (except
in the last stage; see reversed z-axis). The movements of the platform were similar to those
of a roller coaster. The ride included the following accelerations in this specific order: 1)
linear (z-axis); 2) angular (z- and y-axes); 3) angular (z-, x-, and y-axes); 4) angular (roll
axis); 5) extreme linear (z-axis); 6) angular (yaw axis); and 7) extreme linear (y-axis fol-
lowed by reversed z-axis). The environment had simple black-and-white surroundings
(see Figure 1). This background was used to ensure that the symptoms were strictly in-
duced by vection and not due to other reasons, such as intense colors. Additionally, hav-
ing the squared/tiled design offered cues for the participants to perceive vection and alti-
tude changes.
Figure 1. Examples of Linear (Left) and Angular (Centre and Right) Accelerations during the
Ride.
Figure 1.
Examples of Linear (
Left
) and Angular (
Centre
and
Right
) Accelerations during the Ride.
2.2. Cognitive and Psychomotor Skills’ Assessment
The aims of this study required the examination of cybersickness, cognition, and motor
skills to be repeated while the user was immersed in VR. For these reasons, immersive
VR versions of well-established tests were developed. For the development of these VR
cognitive and psychomotor tasks, the specific design and development guidelines and
recommendations for cognitive assessments in immersive VR were followed [35].
Virtual Worlds 2023,220
2.2.1. Verbal Working Memory
A VR version of the Backward Digit Span Task (BDST; [
36
]) was developed and used.
The VR BDST requires participants to listen to a series of digits and remember and recall
them in the reverse order of their presentation. For example, when the digits were 2, 4,
and 3, then participants had to respond in the reverse order (i.e., 3, 4, and 2). Therefore,
the first step involved listening to the digits. After this step, a keypad appeared in front of
the participants. Using the keypad, users had to provide the digits in the reverse order. To
indicate a number, participants had to touch the white box button displaying the equivalent
number (see Figure 2). Continuous touch of the button for one second confirmed the
response. After confirmation, if the response was correct, the button turned green and
made a positive sound. In contrast, if the response was incorrect, then the button turned
orange and made a negative sound. When a mistake was made or all the digits were
provided correctly, the trial ended. In every second successful trial, the length of the digit
sequence increased. When the participant made two subsequent mistakes within the same
digit sequence length (e.g., 3 digits), or when they finished the last trial (i.e., second trial
with a sequence length of 7 digits), then the task ended. The total score of the VR BDST
was determined by adding together the total number of correct trials and the highest digit
sequence length that was performed in at least one trial. A video displaying the task and its
procedures can be found here: https://www.youtube.com/watch?v=1H8cqci-lFs (accessed
on 16 January 2023).
Virtual Worlds 2023, 2, FOR PEER REVIEW 6
To assess reaction times, a VR version of the Deary–Liewald Reaction Time (DLRT)
task [38] was developed and used. The DLRT encompasses two tasks. One task assesses
simple reaction time (SRT), and the other task examines choice reaction time (CRT). For
the SRT task, participants had to observe a white box and touch it as soon as the box
changes color to blue (see Figure 2). There are 20 trials/repetitions in the SRT task. In the
CRT task, there are four boxes, which are aligned horizontally (see Figure 2). Randomly,
one of the four boxes changes its color to blue. When the box turns blue, participants are
required to touch the box as fast as possible (see Figure 2). The CRT task includes 40 tri-
als/repetitions. For both the SRT and CRT, the participants were instructed to touch the
boxes as fast as possible using the most convenient hand. There was a practice session at
the start of both the SRT and the CRT to ensure that the instructions were understood by
the participants. A video displaying the task and its procedures can be found here:
https://www.youtube.com/watch?v=wXdrt0PjNsk (accessed on 16 January 2023).
As in the original version, the SRT produces a score that is the average reaction time
across the 20 trials. Similarly, the CRT produces a score that is the average reaction time
across the 40 trials, for the correct responses only. However, in addition, given that the VR
version of the CRT is enhanced by eye tracking, the time required to attend to the target
was also measured (attentional time, i.e., the time from the appearance of the target until
the gaze of the user falls on it). Additionally, eye tracking facilitated the calculation of the
time required to touch the target once it had been attended to (motor time). Finally, simi-
larly to the original version, the overall time between the target’s presentation and its se-
lection (reaction time) was also calculated. Thus, the VR version of the CRT produces three
scores:
(1) the reaction time (RT) to indicate overall psychomotor speed,
(2) the attentional time (AT) to indicate attentional processing speed,
(3) the motor time (MT) to indicate movement speed.
Figure 2. Digit Span Task (Upper Left), Corsi Block Task (Upper Right), and Deary–Liewald Reac-
tion Time Tasks (Bottom).
Figure 2.
Digit Span Task (
Upper Left
), Corsi Block Task (
Upper Right
), and Deary–Liewald Reaction
Time Tasks (Bottom).
2.2.2. Visuospatial Working Memory
Visuospatial working memory was assessed using the Backward Corsi Block Test
(BCBT) [
37
]. A VR version of the BCBT was developed. This task consists of 27 white boxes
where each one is placed in a different position based on the x-, y-, and z-axes. Nevertheless,
only 9 boxes out of the 27 possible boxes were shown to the participants at one time (see
Virtual Worlds 2023,221
Figure 2). The 9 boxes were presented at the beginning of each trial. Then, a number of
these boxes (depending on the current sequence length) were randomly presented (turning
blue and making a bell sound) in sequential order, with each box presented for one second.
After the presentation of the sequence, participants had to select the boxes in reverse order.
Participants had to touch a cube (the cube turned blue on touch) and keep touching it
for one second to select the cube. When a cube was selected, it either turned green and
made a positive sound (i.e., correct response), or it turned orange and made a negative
sound (i.e., an error). The trial ended when the participants either made a mistake or
they correctly selected all the targets in their reverse order. The sequence lengths were
initially two boxes, with two trials for each length. The number of boxes in the sequence
was increased by one box when at least one of the two trials of the same length/span
was correct. When the participant incorrectly recalled two sequences of the same length,
the task ended. Equally, when the second trial of the last length/span (i.e., 7 cubes) was
performed, the task ended. The sequence lengths increased by up to seven cubes. The total
score is the sum of the span (the longest correct sequence length) and the total number
of correct sequences. A video displaying the task and its procedures can be found here:
https://www.youtube.com/watch?v=MLilvkyMt-g (accessed on 16 January 2023).
2.2.3. Psychomotor Skills
To assess reaction times, a VR version of the Deary–Liewald Reaction Time (DLRT)
task [
38
] was developed and used. The DLRT encompasses two tasks. One task assesses
simple reaction time (SRT), and the other task examines choice reaction time (CRT). For
the SRT task, participants had to observe a white box and touch it as soon as the box
changes color to blue (see Figure 2). There are 20 trials/repetitions in the SRT task. In the
CRT task, there are four boxes, which are aligned horizontally (see Figure 2). Randomly,
one of the four boxes changes its color to blue. When the box turns blue, participants
are required to touch the box as fast as possible (see Figure 2). The CRT task includes
40 trials/repetitions. For both the SRT and CRT, the participants were instructed to touch
the boxes as fast as possible using the most convenient hand. There was a practice session
at the start of both the SRT and the CRT to ensure that the instructions were understood
by the participants. A video displaying the task and its procedures can be found here:
https://www.youtube.com/watch?v=wXdrt0PjNsk (accessed on 16 January 2023).
As in the original version, the SRT produces a score that is the average reaction time
across the 20 trials. Similarly, the CRT produces a score that is the average reaction time
across the 40 trials, for the correct responses only. However, in addition, given that the VR
version of the CRT is enhanced by eye tracking, the time required to attend to the target
was also measured (attentional time, i.e., the time from the appearance of the target until
the gaze of the user falls on it). Additionally, eye tracking facilitated the calculation of
the time required to touch the target once it had been attended to (motor time). Finally,
similarly to the original version, the overall time between the target’s presentation and its
selection (reaction time) was also calculated. Thus, the VR version of the CRT produces
three scores:
(1)
the reaction time (RT) to indicate overall psychomotor speed,
(2)
the attentional time (AT) to indicate attentional processing speed,
(3)
the motor time (MT) to indicate movement speed.
2.3. Cybersickness Questionnaires
The Motion Sickness Susceptibility Questionnaire (MSSQ) [
39
] was completed prior
to enrolment to reduce the likelihood of a participant experiencing severe symptoms of
cybersickness. The MSSQ is a 3-point Likert scale with 18 items/questions examining the
experience of motion sickness using diverse means of transport. Nine items refer to the
experience of motion sickness as a child, and the other nine items refer to such experiences
as an adult. The nine questions are hence repeated in both sections. The MSSQ produces
Virtual Worlds 2023,222
three scores: a child score; an adult score; and a total score, which is the addition of the
previous two scores.
The SSQ and the VRSQ were administered pre- and post-exposure to VR to assess the
intensity of cybersickness symptoms. Both the SSQ and VRSQ are 4-point Likert scales. The
SSQ was developed for individuals that are trained using simulators (e.g., aviators) [
19
].
The SSQ has 16 questions, which are grouped under three categories: nausea; disorientation;
and oculomotor. Four scores are produced, including one for each category and a total score.
The calculation of the scores is made by a formula offered by the developers of the SSQ [
19
].
The maximum score is 100 for each category, and 300 for the total score. On the other hand,
the VRSQ is derived from the SSQ, and it contains 9 items (i.e., approximately half of the
SSQ items), which are grouped under two categories: disorientation and oculomotor (i.e.,
the nausea items are excluded) [
21
]. The VRSQ produces three scores, including one for
each category and a total score, which is the sum of the two sub-scores divided by two. The
maximum score for each sub-score is 100. As discussed above (see Section 1.2), while both
the SSQ and VRSQ appear to be valid tools, they suffer from certain limitations:
•
The SSQ is not specific to cybersickness, and the frequency and intensity of symptoms
substantially differ between simulator sickness and cybersickness.
•
The VRSQ does not consider nausea symptoms, and nausea symptoms are the second
most frequent type of symptoms in cybersickness.
•
VRSQ validation was performed in a study with a small sample size and a limited
diversity of stimuli.
•
Both the SSQ and VRSQ, being 4-point Likert scales, were not designed in line with
the design guidelines for Likert scale questionnaires.
Cybersickness in VR Questionnaire
The CSQ-VR is an improved version of the VRISE section of the VRNQ. The VRISE
section has been found to have very good structural validity in a study where participants
were exposed to three diverse kinds of VR software and environments [
29
]. Addition-
ally, the VRISE section of the VRNQ was previously validated and compared against the
SSQ [
30
]. The VRISE of the VRNQ appeared superior to the SSQ due to its shorter adminis-
tration time (i.e., 5 items instead of 16 items) and the enhanced interpretability of the scores
(i.e., scores calculated by a simple addition, instead of a complex formula). However, the
VRISE section of the VRNQ had only one item for oculomotor symptoms and the score
was inversed (i.e., a higher score indicated a weaker intensity of that symptom). To address
these limitations, the CSQ-VR was developed based on the VRISE section of the VRNQ.
Comparably to the VRNQ, the CSQ-VR was designed and developed by following the
design guidelines for Likert scales, i.e., a 7-point Likert scale, and combining text with num-
bers in the responses (see [
25
–
28
]). The CSQ-VR is a 7-point Likert scale that includes six
questions for the assessment of the three types of symptoms of cybersickness, which form
the following respective sub-scores: nausea; vestibular; and oculomotor. Each category
includes two questions. The total score is the sum of the three scores, which a maximum
score of 42 (14 for each sub-score). The paper-and-pencil version of the CSQ-VR can be
found in the Supplementary Materials.
Moreover, a 3D version of the CSQ-VR has also been developed to assess cybersickness
while the user is immersed in VR. A user interface (UI) for the VR version of the CSQ-VR
was designed and developed. In the UI, the question appears in the upper area and the
response (in red letters) appears in the middle area. The users change their response by
touching the corresponding number or sliding along the slider (see Figure 3). Furthermore,
based on the established link between pupil size and affective/emotional state [
31
], eye
tracking was integrated to facilitate ophthalmometry and pupillometry. To measure fixation
duration, invisible eye-tracking targets were placed in front of the text, while their height
and width were always matched to the displayed text per line (see Figure 3). Moreover,
the measurement of pupil size was continuous while the user responded to the CSQ-VR
questions. Pupillometry yields measurements of average pupil size (right and left), which
Virtual Worlds 2023,223
can be used as a physiological metric of negative emotion. Finally, a video showing the
procedures of a questionnaire in VR can be found here: Link to the Video (accessed on 16
January 2023).
Virtual Worlds 2023, 2, FOR PEER REVIEW 8
VR was designed and developed. In the UI, the question appears in the upper area and
the response (in red letters) appears in the middle area. The users change their response
by touching the corresponding number or sliding along the slider (see Figure 3). Further-
more, based on the established link between pupil size and affective/emotional state [31],
eye tracking was integrated to facilitate ophthalmometry and pupillometry. To measure
fixation duration, invisible eye-tracking targets were placed in front of the text, while their
height and width were always matched to the displayed text per line (see Figure 3). More-
over, the measurement of pupil size was continuous while the user responded to the CSQ-
VR questions. Pupillometry yields measurements of average pupil size (right and left),
which can be used as a physiological metric of negative emotion. Finally, a video showing
the procedures of a questionnaire in VR can be found here: Link to the Video (accessed on
16 January 2023).
Figure 3. User Interface and Eye-Tracking (ET) Targets of the VR version of CSQ-VR. Note: Eye-
tracking targets were not visible to the user.
2.4. Participants and Procedures
Thirty-nine participants (22 females, 17 males) were recruited with a mean age of
25.28 years [SD = 3.25, Range = 22–36] and a mean education of 17.23 years [SD = 1.60,
Range = 13–20]. The recruitment was performed via opportunity sampling using the Uni-
versity of Edinburgh’s internal mailing lists, alongside advertisements on social media.
The study was approved by the School of Philosophy, Psychology, and Language Sciences
(PPLS) Ethics Committee of the University of Edinburgh. Informed and written consent
was obtained from all participants prior to their participation. Participants were compen-
sated with 20 GBP each for their time and effort.
The MSSQ was completed before enrolment to reduce the likelihood of severe symp-
toms following VR exposure. In line with the MSSQ author’s suggestions [39], the 75th
percentile was used as a parsimonious cut-off score for inclusion in the study. This al-
lowed us to exclude individuals who are susceptible to experiencing strong cybersickness
symptomatology (i.e., the upper 25th percentile of the population). The included partici-
pants were then invited to attend the experiment. Upon arrival, participants were in-
formed of the study’s aims and procedures, and the adverse effects that they may experi-
ence. The participants then provided informed consent in written form.
Figure 3.
User Interface and Eye-Tracking (ET) Targets of the VR version of CSQ-VR. Note: Eye-
tracking targets were not visible to the user.
2.4. Participants and Procedures
Thirty-nine participants (22 females, 17 males) were recruited with a mean age of
25.28 years [SD = 3.25, Range = 22–36] and a mean education of 17.23 years [SD = 1.60,
Range = 13–20
]. The recruitment was performed via opportunity sampling using the Uni-
versity of Edinburgh’s internal mailing lists, alongside advertisements on social media.
The study was approved by the School of Philosophy, Psychology, and Language Sciences
(PPLS) Ethics Committee of the University of Edinburgh. Informed and written consent was
obtained from all participants prior to their participation. Participants were compensated
with 20 GBP each for their time and effort.
The MSSQ was completed before enrolment to reduce the likelihood of severe symp-
toms following VR exposure. In line with the MSSQ author’s suggestions [
39
], the 75th
percentile was used as a parsimonious cut-off score for inclusion in the study. This allowed
us to exclude individuals who are susceptible to experiencing strong cybersickness symp-
tomatology (i.e., the upper 25th percentile of the population). The included participants
were then invited to attend the experiment. Upon arrival, participants were informed of
the study’s aims and procedures, and the adverse effects that they may experience. The
participants then provided informed consent in written form.
Firstly, an induction on how to wear the headset and use and hold the controllers
was offered to every participant. An HTC Vive Pro Eye was used, which embeds an
eye-tracker with a 120 Hz refresh rate and a tracking accuracy of 0.5–1.1
◦
. Secondly, the
participants provided the following demographic data by responding to a questionnaire:
age; sex; gender; education; dominant eye; VR experience; computing experience; and
gaming experience. The dominant eye was determined using the Miles test [
40
]. Note that
VR/computing/gaming experiences were calculated by adding the scores from two ques-
tions (6-item Likert scale) for each one. The first question was pertinent to the participant’s
ability (e.g., 5: highly skilled) to operate a VR/computer/game, and the second one was
pertinent to the frequency of operating them (e.g., 4: once a week).
Virtual Worlds 2023,224
Before VR exposure, participants responded to the CSQ-VR (paper version), SSQ, and
VRSQ. Participants were then immersed in VR. Note that for the assessments and rides,
participants were always in a standing position in the middle of the VR area (see the X
mark in Figure 1). The first part included the tutorials, during which a video tutorial for
each task was offered, alongside the corresponding verbal and written instructions. After
each tutorial, the participant performed the corresponding task. This part formed the
baseline assessment of each participant. The baseline assessment included the following:
the VR version of CSQ-VR (Cybersickness); the verbal working memory task (BDST); the
visuospatial working memory task (BCBT); and the reaction time task (DLRT; see Figure 2).
After the baseline assessment, the first ride started. After each ride, the participants
performed an assessment identical to the baseline (i.e., CSQ-VR, BDST, BCBT, DLRT). On
top of the baseline assessment, the participants were exposed to three rides and three
respective assessments. The whole procedure in VR lasted approximately 100 min for
each participant. After the VR session, participants responded to the CSQ-VR (paper
version), SSQ, and VRSQ. Then, refreshments rich in electrolytes were offered to the
participants. Moreover, the participants rested for 10–15 min before leaving the premises.
The participants were instructed to avoid driving and using heavy machinery for the rest
of the day.
2.5. Statistical Analyses
Descriptive statistical analyses were performed to provide an overview of the sample.
Reliability analyses were conducted to examine the internal consistency of the CSQ-VR. The
recommended thresholds for Cronbach’s
α
were used to interpret the internal consistency
(i.e., adequate = 0.6–0.7, good = 0.7–0.8, and very good = 0.8–0.95) [
41
]. Pearson’s correla-
tional analyses were performed to examine the validity of the CSQ-VR versions against the
SSQ and the VRS post-exposure (i.e., after the VR session). Because the SSQ is considered
the gold standard and it has a structure (i.e., three sub-scores: nausea; oculomotor; and
disorientation) similar to the CSQ-VR, the convergent validity (i.e., correlations) of the
CSQ-VR was assessed against the SSQ. Receiver operating characteristic (ROC) and area
under the curve (AUC) analyses were performed to appraise the psychometric properties
of the CSQ-VR, SSQ, and VRSQ in detecting temporary cognitive and motor decline due
to cybersickness. The thresholds of AUC > 0.7 and metric score > 1.5 were used in line
with the respective recommendations for determining the suitability of the tool [
42
,
43
].
The temporary decline was based on the performance on the assessment after each ride.
In agreement with the consensus of the American Academy of Clinical Neuropsychology
for determining a substantial decrease in performance, two standard deviations from the
mean were used [
44
]. Thus, when the performance (i.e., score) on the assessment after
the respective ride was 2 standard deviations from the mean of the baseline assessment,
the performance was defined as abnormal (i.e., temporary decline). Note that the two
standard deviations had to indicate a worse performance and thus be greater for reaction
and motor times (i.e., slower reaction or motor speed) and smaller for the verbal and
visuospatial working memory (i.e., poorer performance). Finally, the predictive ability of
pupil size was examined by performing a mixed model regression analysis. The analysis
was performed using Jamovi statistical software (descriptive statistics, reliability, ROC, and
AUC analyses) [
45
], as well as R(transforming the data, plots design, and correlation and
regressions analyses) [
46
]. As the variables violated the normality assumption, we used
the bestNormalize R package [
47
] to transform and centralize the data. The distribution
of the data was then normal. The transformed data were used for parametric analyses
(i.e., correlations and mixed regression analysis). Furthermore, the psych (correlational
analyses) [
48
], the ggplot2 (plots) [
49
], and the lme4 (regression analyses) [
50
] R packages
were used to perform the respective analyses.
Virtual Worlds 2023,225
3. Results
The descriptive statistics of the sample are displayed in Table 1. Concerning the in-
tensity of cybersickness symptoms, it can be observed that the participants predominantly
experienced moderate symptoms. There were no dropouts during the experiment. The de-
scriptive statistics for the VR version of the CSQ-VR, per experimental stage, are presented
in Table 2.
Table 1. Descriptive Statistics.
Mean (SD) Range Max. Score
Sex (22F/17M) - - -
Age 25.28 (3.22) 22–36 -
Years of Education 15.14 (5.18) 13–20 -
VR Experience 2.67 (0.92) 2–6 14
Computing Experience 10.36 (0.80) 9–12 14
Gaming Experience 5.54 (2.97) 2–12 14
MSSQ Child Score 4.69 (3.34) 0–13.50 27
MSSQ Adult Score 3.91 (3.20) 0–11.25 27
MSSQ Total Score 8.60 (5.23) 0–20.13 54
Pupil Size (mm) 5.37 (0.90) 3.70–8.32 -
CSQ-VR (VR) Total Score * 10.63 (4.97) 6–28 42
CSQ-VR (VR) Nausea Score * 3.18 (1.56) 2–9 14
CSQ-VR (VR) Vestibular Score * 3.66 (2.43) 2–13 14
CSQ-VR (VR) Oculomotor Score * 3.79 (1.70) 2–9 14
CSQ-VR Total Score 12.23 (4.96) 6–27 42
CSQ-VR Nausea Score 3.51 (1.68) 2–9 14
CSQ-VR Vestibular Score 3.97 (2.41) 2–10 14
CSQ- VR Oculomotor Score 4.74 (1.81) 2–10 14
SSQ-Total Score 67.24 (48.09) 0–223.66 300
SSQ-Nausea Score 24.22 (22.09) 0–95.40 100
SSQ-Disorientation Score 9.40 (9.98) 0–44.88 100
SSQ-Oculomotor Score 33.62 (21.84) 0–83.38 100
VRSQ-Total Score 19.17 (13.27) 0–59.17 100
VRSQ-Disorientation Score 11.62 (13.27) 0–60.00 100
VRSQ-Oculomotor Score 26.71 (15.63) 0–58.33 100
* (VR) = VR version; Pupil Size measured while responding to the VR version of CSQ-VR.
Table 2. Descriptive Statistics of the VR version of CSQ-VR per Experimental Stage.
Experimental
Stage CSQ-VR Scores * Mean (SD) Range Max. Score
Baseline
Total Score 7.59 (2.09) 6–16 42
Nausea Score 2.23 (0.54) 2–4 14
Vestibular Score 2.38 (0.85) 2–6 14
Oculomotor Score 2.79 (1.11) 2–6 14
Ride 1
Total Score 10.79 (4.35) 6–24 42
Nausea Score 3.41 (1.37) 2–8 14
Vestibular Score 3.97 (2.47) 2–12 14
Oculomotor Score 3.41 (1.41) 2–8 14
Ride 2
Total Score 11.87 (5.03) 6–23 42
Nausea Score 3.54 (1.57) 2–8 14
Vestibular Score 4.13 (2.56) 2–12 14
Oculomotor Score 4.21 (1.73) 2–9 14
Ride 3
Total Score 12.26 (6.19) 6–28 42
Nausea Score 3.54 (2.02) 2–9 14
Vestibular Score 4.15 (2.91) 2–13 14
Oculomotor Score 4.56 (2.00) 2–9 14
* Scores of the VR version of CSQ-VR during the exposure to VR.
Virtual Worlds 2023,226
3.1. Reliability and Validity
Interpretation of the outcomes was based on the recommendations offered by Ur-
sachi et al. [
41
]. Based on them, Cronbach’s
α
of 0.6–0.7 is an acceptable score, 0.7–0.8 is
a good score, and 0.8–0.95 is a very good score. The overall internal consistency of the
questionnaire (i.e., the total score’s reliability) was evaluated by considering each question-
naire’s sub-scores. The internal consistency of the sub-categories (i.e., the reliability of the
sub-score) was examined by considering the respective items/questions. The reliability
analyses revealed that all sub-scores of the CSQ-VR had good internal consistency (see
Table 3). Specifically, the total score and the vestibular sub-score showed very good internal
consistency, while the nausea and oculomotor sub-scores revealed good internal consis-
tency. However, Cronbach’s
α
of the oculomotor sub-score was at the margins between
good and adequate internal consistency.
Table 3. Reliability (Internal Consistency) of CSQ-VR, SSQ, and VRSQ.
Questionnaire Scores Cronbach’s α
CSQ-VR
Total Score 0.865
Nausea 0.792
Vestibular 0.934
Oculomotor 0.704
SSQ
Total Score 0.810
Nausea 0.676
Disorientation 0.809
Oculomotor 0.744
VRSQ
Total Score 0.806
Disorientation 0.718
Oculomotor 0.654
The internal consistency was based on the sub-scores of the total score, and the sub-scores were based on their
respective items. Based on [
41
], Cronbach’s
α
of 0.6–0.7 is acceptable, 0.7–0.8 is good, and 0.8–0.95 is very good.
Both the SSQ and VRSQ total scores showed good internal consistency; however,
both were substantially lower than the internal consistency of the CSQ-VR total score
(see Table 3). The nausea score of the SSQ showed an acceptable internal consistency,
which was significantly lower than the almost very good internal consistency of the nausea
score of the CSQ-VR. The disorientation score of the SSQ revealed marginally very good
internal consistency, while the disorientation score of the VRSQ indicated marginally good
internal consistency. Both disorientation scores (SSQ and VRSQ) were significantly lower
than the almost excellent internal consistency of the CSQ-VR. Finally, the oculomotor
score of the SSQ showed good internal consistency that was higher than the marginally
good internal consistency of the oculomotor score of the CSQ-VR. On the other hand, the
oculomotor score of the VRSQ revealed adequate internal consistency. The oculomotor
score of the VRSQ and the nausea score of the SSQ were the two scores that were below the
parsimonious threshold of 0.7. Overall, the CSQ-VR appeared to have superior internal
consistency compared to the SSQ and the VRSQ.
The scores of the CSQ-VR (both versions) were significantly correlated with the corre-
sponding scores of the SSQ. Overall, the analyses revealed moderate to strong correlations
between the scores. The paper-and-pencil version of the CSQ-VR was strongly associated
with the SSQ (see Figure 4). Their total scores especially, as well as their oculomotor scores,
revealed a very strong correlation between them. Although the correlations for the nausea
and vestibular scores were weaker than those observed above, they were still strong cor-
relations (see Figure 4). Similarly, the VR version of the CSQ-VR was strongly associated
with the SSQ (see Figure 5). In particular, their total scores indicated a strong correlation
between them. While the correlations for their sub-scores were weaker than those between
the total score, they were still moderate to strong correlations (see Figure 5). These results
postulate the convergent validity of both versions of the CSQ-VR. Additionally, given that
Virtual Worlds 2023,227
all sub-scores were substantially associated, the construct validity of the CSQ-VR is strongly
supported.
Virtual Worlds 2023, 2, FOR PEER REVIEW 12
Oculomotor 0.654
The internal consistency was based on the sub-scores of the total score, and the sub-scores were
based on their respective items. Based on [41], Cronbach’s α of 0.6–0.7 is acceptable, 0.7–0.8 is good,
and 0.8–0.95 is very good.
The scores of the CSQ-VR (both versions) were significantly correlated with the cor-
responding scores of the SSQ. Overall, the analyses revealed moderate to strong correla-
tions between the scores. The paper-and-pencil version of the CSQ-VR was strongly asso-
ciated with the SSQ (see Figure 4). Their total scores especially, as well as their oculomotor
scores, revealed a very strong correlation between them. Although the correlations for the
nausea and vestibular scores were weaker than those observed above, they were still
strong correlations (see Figure 4). Similarly, the VR version of the CSQ-VR was strongly
associated with the SSQ (see Figure 5). In particular, their total scores indicated a strong
correlation between them. While the correlations for their sub-scores were weaker than
those between the total score, they were still moderate to strong correlations (see Figure
5). These results postulate the convergent validity of both versions of the CSQ-VR. Addi-
tionally, given that all sub-scores were substantially associated, the construct validity of
the CSQ-VR is strongly supported.
Figure 4. Correlations between the scores of the CSQ-VR (paper-and-pencil version) and the SSQ.
Figure 5. Correlations between the scores of the CSQ-VR (VR version) and the SSQ.
Furthermore, the scores for both versions of the CSQ-VR were strongly associated
with the VRSQ scores (see Table 4). The total scores of the CSQ-VR versions showed the
strongest correlations with the total score of the VRSQ. The oculomotor scores of the CSQ-
VR and VRSQ equally revealed robust associations between them. Although the vestibu-
lar scores indicated weaker correlations compared to the other scores, the correlations
were moderate (see Table 4). These outcomes further support the convergent and con-
struct validity of both versions of the CSQ-VR.
Table 4. Correlations between the scores of CSQ-VR (both versions) and the VRSQ.
Figure 4. Correlations between the scores of the CSQ-VR (paper-and-pencil version) and the SSQ.
Virtual Worlds 2023, 2, FOR PEER REVIEW 12
Oculomotor 0.654
The internal consistency was based on the sub-scores of the total score, and the sub-scores were
based on their respective items. Based on [41], Cronbach’s α of 0.6–0.7 is acceptable, 0.7–0.8 is good,
and 0.8–0.95 is very good.
The scores of the CSQ-VR (both versions) were significantly correlated with the cor-
responding scores of the SSQ. Overall, the analyses revealed moderate to strong correla-
tions between the scores. The paper-and-pencil version of the CSQ-VR was strongly asso-
ciated with the SSQ (see Figure 4). Their total scores especially, as well as their oculomotor
scores, revealed a very strong correlation between them. Although the correlations for the
nausea and vestibular scores were weaker than those observed above, they were still
strong correlations (see Figure 4). Similarly, the VR version of the CSQ-VR was strongly
associated with the SSQ (see Figure 5). In particular, their total scores indicated a strong
correlation between them. While the correlations for their sub-scores were weaker than
those between the total score, they were still moderate to strong correlations (see Figure
5). These results postulate the convergent validity of both versions of the CSQ-VR. Addi-
tionally, given that all sub-scores were substantially associated, the construct validity of
the CSQ-VR is strongly supported.
Figure 4. Correlations between the scores of the CSQ-VR (paper-and-pencil version) and the SSQ.
Figure 5. Correlations between the scores of the CSQ-VR (VR version) and the SSQ.
Furthermore, the scores for both versions of the CSQ-VR were strongly associated
with the VRSQ scores (see Table 4). The total scores of the CSQ-VR versions showed the
strongest correlations with the total score of the VRSQ. The oculomotor scores of the CSQ-
VR and VRSQ equally revealed robust associations between them. Although the vestibu-
lar scores indicated weaker correlations compared to the other scores, the correlations
were moderate (see Table 4). These outcomes further support the convergent and con-
struct validity of both versions of the CSQ-VR.
Table 4. Correlations between the scores of CSQ-VR (both versions) and the VRSQ.
Figure 5. Correlations between the scores of the CSQ-VR (VR version) and the SSQ.
Furthermore, the scores for both versions of the CSQ-VR were strongly associated with
the VRSQ scores (see Table 4). The total scores of the CSQ-VR versions showed the strongest
correlations with the total score of the VRSQ. The oculomotor scores of the CSQ-VR and
VRSQ equally revealed robust associations between them. Although the vestibular scores
indicated weaker correlations compared to the other scores, the correlations were moderate
(see Table 4). These outcomes further support the convergent and construct validity of both
versions of the CSQ-VR.
Table 4. Correlations between the scores of CSQ-VR (both versions) and the VRSQ.
Correlation Pair Pearson’s r p-Value
CSQ-VR–Total Score VRSQ–Total Score 0.77 <0.001
CSQ-VR–Oculomotor VRSQ–Oculomotor 0.75 <0.001
CSQ-VR–Vestibular VRSQ–Disorientation 0.55 <0.001
CSQ-VR (VR)–Total Score VRSQ–Total Score 0.65 <0.001
CSQ-VR (VR)–Oculomotor VRSQ–Oculomotor 0.62 <0.001
CSQ-VR (VR)–Vestibular VRSQ–Disorientation 0.52 <0.001
(VR) = VR version.
3.2. Detection of Temporary Decline due to Cybersickness
As mentioned above, the temporary decline was defined by two standard deviations
from the mean of the baseline assessment. This definition is in line with the guidelines of
the American Academy of Clinical Neuropsychology for determining whether performance
is abnormal [
44
]. Eleven observations were detected which met the criterion for temporary
cognitive/motor decline. Six of these were pertinent to reaction speed (i.e., longer reaction
times) and five of them were applicable to motor speed (i.e., slower). Thus, all temporary
declines were found for the DLRT task. A trend was also observed where, when motor
Virtual Worlds 2023,228
speed was substantially slower (i.e., a decline), reaction speed also substantially declined.
Finally, these declines were all found in three participants. Thus, only three participants
experienced a temporary decline in their psychomotor skills. These results indicate that
susceptibility to experiencing a temporary decline due to cybersickness may be attributed
to individual differences.
The ROC–AUC analyses provide cut-off scores for each questionnaire where the
optimal sensitivity (i.e., the detection of true positives) and specificity (i.e., the exclusion of
true negatives) are achieved. Following the recommendations for ROC–AUC analyses and
psychometrics [
42
,
43
] to determine the suitability of a questionnaire to detect temporary
decline, two criteria were set as follows: (1) AUC > 70% and (2) metric score > 1.5, both
of which had to be met. The ROC–AUC analyses for declines in reaction time (i.e., slower
reaction times) showed that only the total scores for both versions of the CSQ-VR met the
criteria (see Table 5). Similarly, the ROC–AUC analyses for motor speed decline indicated
that only the total scores for both versions of the CSQ-VR met the criteria. Furthermore,
the two versions of the CSQ-VR showed the best sensitivity and specificity in detecting a
temporary decline in reaction time and motor speed, while the VRSQ and SSQ showed
significantly smaller psychometric properties (see Tables 5and 6and Figure 6). These results
postulate that the total scores for both versions of the CSQ-VR have superior psychometric
properties to the total scores of the SSQ and VRSQ. Additionally, only the CSQ-VR total
scores are suitable for detecting a temporary decline in reaction speed and/or motor speed.
Table 5.
Psychometric Properties of the CSQ-VR, SSQ, and VRSQ in detecting Reaction Speed Decline.
Cybersickness Score Cut-Off Sensitivity (%) Specificity (%) PPV (%) NPV (%) AUC (%) Metric Score
CSQ-VR–Total Score 10 100% 75% 15.15% 100% 87% 1.75
CSQ-VR (VR)–Total Score 9 100% 75% 15.15% 100% 86.5% 1.75
SSQ–Total Score 83.36 80% 68.75% 10.26% 98.72% 66.1% 1.49
VRSQ–Total Score 20 100% 53.57% 8.77% 100% 66.6% 1.54
(VR) = VR version. Based on [
42
] and [
43
], the following thresholds were set and had to be met: AUC > 70% and
metric score > 1.5.; PPV = positive predictive value (i.e., the ratio of true positives); NPV = negative predictive
value (i.e., the ratio of true negatives).
Table 6.
Psychometric Properties of the CSQ-VR, SSQ, and VRSQ in detecting Motor Speed Decline.
Cybersickness Score Cut-off Sensitivity (%) Specificity (%) PPV (%) NPV (%) AUC (%) Metric Score
CSQ-VR–Total Score 10 100% 75.68% 18.18% 100% 86.9% 1.76
CSQ-VR (VR)–Total Score 9 100% 75.68% 18.18% 100% 88% 1.76
SSQ–Total Score 83.36 83.33% 69.37% 12.82% 98.72% 68% 1.53
VRSQ–Total Score 20 100% 54.05% 10.53% 100% 67.53% 1.54
(VR) = VR version. Based on [
42
] and [
43
], the following thresholds were set and had to be met: AUC > 70% and
metric score > 1.5; PPV = positive predictive value (i.e., the ratio of true positives); NPV = negative predictive
value (i.e., the ratio of true negatives).
The psychometric properties of the sub-scores of each questionnaire were also ex-
amined. In detecting a temporary decline in reaction speed or motor speed, only the
vestibular/disorientation scores of the questionnaires met the criteria of suitable psycho-
metric properties (see Tables 7and 8). However, the nausea score of the VR version of
the CSQ-VR also met the criteria for detecting both. The best sensitivity and specificity
in detecting a temporary decline in either reaction or motor speed was observed for the
vestibular score of the paper-and-pencil version of the CSQ-VR, closely followed by the
same score for the VR version of the CSQ-VR (see Tables 7and 8and Figure 7). The sensi-
tivity and specificity of the disorientation scores of the SSQ and VRSQ were substantially
lower compared to the CSQ-VR. However, the sensitivity and specificity of the disorienta-
tion score of the SSQ were significantly higher than the ones for the disorientation score of
the VRSQ.
Virtual Worlds 2023,229
Virtual Worlds 2023, 2, FOR PEER REVIEW 14
CSQ-VR–Total Score 10 100% 75.68% 18.18% 100% 86.9% 1.76
CSQ-VR (VR)–Total Score 9 100% 75.68% 18.18% 100% 88% 1.76
SSQ–Total Score 83.36 83.33% 69.37% 12.82% 98.72% 68% 1.53
VRSQ–Total Score 20 100% 54.05% 10.53% 100% 67.53% 1.54
(VR) = VR version. Based on [42] and [43], the following thresholds were set and had to be met: AUC
> 70% and metric score > 1.5; PPV = positive predictive value (i.e., the ratio of true positives); NPV
= negative predictive value (i.e., the ratio of true negatives).
Figure 6. Sensitivity and Specificity of the CSQ-VR, SSQ, and VRSQ Total Scores in detecting Reac-
tion Time (Left) and Motor Speed (Right) Decline.
Note: (VR) = VR version.
The psychometric properties of the sub-scores of each questionnaire were also exam-
ined. In detecting a temporary decline in reaction speed or motor speed, only the vestib-
ular/disorientation scores of the questionnaires met the criteria of suitable psychometric
properties (see Tables 7 and 8). However, the nausea score of the VR version of the CSQ-
VR also met the criteria for detecting both. The best sensitivity and specificity in detecting
a temporary decline in either reaction or motor speed was observed for the vestibular
score of the paper-and-pencil version of the CSQ-VR, closely followed by the same score
for the VR version of the CSQ-VR (see Tables 7 and 8 and Figure 7). The sensitivity and
specificity of the disorientation scores of the SSQ and VRSQ were substantially lower com-
pared to the CSQ-VR. However, the sensitivity and specificity of the disorientation score
of the SSQ were significantly higher than the ones for the disorientation score of the VRSQ.
Table 7. Psychometric Properties of the CSQ-VR, SSQ, and VRSQ Vestibular/Disorientation Scores
in detecting Reaction Speed Decline.
Cybersickness Score Cut-Off Sensitivity (%) Specificity (%) PPV (%) NPV (%) AUC (%) Metric Score
CSQ-VR–Nausea 3 60% 67.86% 7.69% 97.44% 65.3% 1.28
CSQ-VR–Vestibular 5 100% 77.68% 16.67% 100% 92.6% 1.78
CSQ-VR–Oculomotor 7 40% 93.75% 22.22% 97.22% 65.8% 1.34
CSQ-VR (VR)–Nausea 3 100% 66.96% 11.09% 100% 83.6% 1.67
CSQ-VR (VR)–Vestibular 4 100% 70.54% 13.16% 100% 86.7% 1.71
CSQ-VR (VR)–Oculomotor 6 40% 90.18% 15.38% 97.12% 61.2% 1.30
SSQ–Nausea 47.7 40% 88.39% 13.33% 97.06% 60.04% 1.28
Figure 6.
Sensitivity and Specificity of the CSQ-VR, SSQ, and VRSQ Total Scores in detecting Reaction
Time (Left) and Motor Speed (Right) Decline. Note: (VR) = VR version.
Table 7.
Psychometric Properties of the CSQ-VR, SSQ, and VRSQ Vestibular/Disorientation Scores in
detecting Reaction Speed Decline.
Cybersickness Score Cut-Off Sensitivity (%) Specificity (%) PPV (%) NPV (%) AUC (%) Metric Score
CSQ-VR–Nausea 3 60% 67.86% 7.69% 97.44% 65.3% 1.28
CSQ-VR–Vestibular 5 100% 77.68% 16.67% 100% 92.6% 1.78
CSQ-VR–Oculomotor 7 40% 93.75% 22.22% 97.22% 65.8% 1.34
CSQ-VR(VR)–Nausea 3 100% 66.96% 11.09% 100% 83.6% 1.67
CSQ-VR(VR)–Vestibular 4 100% 70.54% 13.16% 100% 86.7% 1.71
CSQ-VR (VR)–Oculomotor 6 40% 90.18% 15.38% 97.12% 61.2% 1.30
SSQ–Nausea 47.7 40% 88.39% 13.33% 97.06% 60.04% 1.28
SSQ–Disorientation 11.22 100% 64.29% 11.11% 100% 70.1% 1.64
SSQ–Oculomotor 45.48 80% 58.04% 7.84% 98.48% 67.9% 1.38
VRSQ–Disorientation 20 80% 74.01% 12.12% 98.81% 73.06% 1.54
VRSQ–Oculomotor 33.33 100% 53.57% 8.77% 100% 63.4% 1.54
(VR) = VR version. Based on [
42
] and [
43
], the following thresholds were set and had to be met: AUC > 70% and
metric score > 1.5; PPV = positive predictive value (i.e., the ratio of true positives); NPV = negative predictive
value (i.e., the ratio of true negatives).
Table 8.
Psychometric Properties of the CSQ-VR, SSQ, and VRSQ Vestibular/Disorientation Scores in
detecting Motor Speed Decline.
Cybersickness Score Cut-Off Sensitivity (%) Specificity (%) PPV (%) NPV (%) AUC (%) Metric Score
CSQ-VR–Nausea 2 100% 32.43% 7.41% 100% 62.6% 1.32
CSQ-VR–Vestibular 5 100% 78.38% 20% 100% 94.4% 1.78
CSQ-VR–Oculomotor 7 33.33% 93.69% 22.22% 96.3% 61% 1.27
CSQ-VR(VR)–Nausea 3 100% 67.57% 14.29% 100% 85.1% 1.68
CSQ-VR(VR)–Vestibular 4 100% 71.17% 15.79% 100% 89.3% 1.71
CSQ-VR (VR)–Oculomotor 6 33.33% 90.09% 15.38% 96.15% 56.5% 1.23
SSQ–Nausea 47.7 50% 89.19% 20% 97.06% 65.08% 1.39
SSQ–Disorientation 11.22 100% 64.86% 13.33% 100% 70.3% 1.65
SSQ–Oculomotor 45.48 83.33% 58.56% 9.8% 98.48% 67.8% 1.42
VRSQ–Disorientation 20 82.3% 74.77% 15.15% 98.81% 75% 1.58
VRSQ–Oculomotor 33.33 100% 54.05% 10.53% 100% 63.5% 1.54
(VR) = VR version. Based on [
42
] and [
43
], the following thresholds were set and had to be met: AUC > 70% and
metric score > 1.5; PPV = positive predictive value (i.e., the ratio of true positives); NPV = negative predictive
value (i.e., the ratio of true negatives).
Virtual Worlds 2023,230
Virtual Worlds 2023, 2, FOR PEER REVIEW 15
SSQ–Disorientation 11.22 100% 64.29% 11.11% 100% 70.1% 1.64
SSQ–Oculomotor 45.48 80% 58.04% 7.84% 98.48% 67.9% 1.38
VRSQ–Disorientation 20 80% 74.01% 12.12% 98.81% 73.06% 1.54
VRSQ–Oculomotor 33.33 100% 53.57% 8.77% 100% 63.4% 1.54
(VR) = VR version. Based on [42] and [43], the following thresholds were set and had to be met: AUC
> 70% and metric score > 1.5; PPV = positive predictive value (i.e., the ratio of true positives); NPV
= negative predictive value (i.e., the ratio of true negatives).
Table 8. Psychometric Properties of the CSQ-VR, SSQ, and VRSQ Vestibular/Disorientation Scores
in detecting Motor Speed Decline.
Cybersickness Score Cut-Off Sensitivity (%) Specificity (%) PPV (%) NPV (%) AUC (%) Metric Score
CSQ-VR–Nausea 2 100% 32.43% 7.41% 100% 62.6% 1.32
CSQ-VR–Vestibular 5 100% 78.38% 20% 100% 94.4% 1.78
CSQ-VR–Oculomotor 7 33.33% 93.69% 22.22% 96.3% 61% 1.27
CSQ-VR (VR)–Nausea 3 100% 67.57% 14.29% 100% 85.1% 1.68
CSQ-VR (VR)–Vestibular 4 100% 71.17% 15.79% 100% 89.3% 1.71
CSQ-VR (VR)–Oculomotor 6 33.33% 90.09% 15.38% 96.15% 56.5% 1.23
SSQ–Nausea 47.7 50% 89.19% 20% 97.06% 65.08% 1.39
SSQ–Disorientation 11.22 100% 64.86% 13.33% 100% 70.3% 1.65
SSQ–Oculomotor 45.48 83.33% 58.56% 9.8% 98.48% 67.8% 1.42
VRSQ–Disorientation 20 82.3% 74.77% 15.15% 98.81% 75% 1.58
VRSQ–Oculomotor 33.33 100% 54.05% 10.53% 100% 63.5% 1.54
(VR) = VR version. Based on [42] and [43], the following thresholds were set and had to be met: AUC
> 70% and metric score > 1.5; PPV = positive predictive value (i.e., the ratio of true positives); NPV
= negative predictive value (i.e., the ratio of true negatives).
Figure 7. Sensitivity and Specificity of the CSQ-VR, SSQ, and VRSQ Vestibular/Disorientation
Scores in detecting Reaction Time (Left) and Motor Speed (Right) Decline.
Note: (VR) = VR ver-
sion.
3.3. Mixed Model Regression Analysis
A mixed model regression analysis was conducted to determine whether pupil size
can be a biomarker/predictor of cybersickness. The analysis indicated that the model with
pupil size as a predictor of the total score on the VR version of the CSQ-VR was significant.
Figure 7.
Sensitivity and Specificity of the CSQ-VR, SSQ, and VRSQ Vestibular/Disorientation Scores
in detecting Reaction Time (Left) and Motor Speed (Right) Decline. Note: (VR) = VR version.
3.3. Mixed Model Regression Analysis
A mixed model regression analysis was conducted to determine whether pupil size
can be a biomarker/predictor of cybersickness. The analysis indicated that the model
with pupil size as a predictor of the total score on the VR version of the CSQ-VR was
significant. Pupil size also revealed a relatively high beta (negative) coefficient, postulating
that cybersickness intensity substantially increases as pupil size decreases (see Figure 8).
Furthermore, the fixed effects of pupil size, alongside the random effects of the participants,
appear to explain 50% of the variance in the intensity of cybersickness. These outcomes
postulate that pupil size is a significant predictor of the intensity of cybersickness, and it
can therefore be considered as a biomarker of cybersickness.
Virtual Worlds 2023, 2, FOR PEER REVIEW 16
Pupil size also revealed a relatively high beta (negative) coefficient, postulating that cy-
bersickness intensity substantially increases as pupil size decreases (see Figure 8). Fur-
thermore, the fixed effects of pupil size, alongside the random effects of the participants,
appear to explain 50% of the variance in the intensity of cybersickness. These outcomes
postulate that pupil size is a significant predictor of the intensity of cybersickness, and it
can therefore be considered as a biomarker of cybersickness.
Figure 8. Mixed Regression Model of Pupil Size Predicting Cybersickness Intensity.
4. Discussion
The CSQ-VR is an adapted and enhanced version of the VRISE section and sub-score
of the VNRQ. Based on the recommendations by Ursachi et al. [41] for Cronbach’s α, the
CSQ-VR displayed good to very good internal consistency. This finding is aligned with
the high structural validity and internal consistency of the VRNQ and its VRISE sub-score,
which have previously been observed [29]. Additionally, the total scores and sub-scores
of both versions of the CSQ-VR showed robust correlations with their respective sub-
scores and total scores of the SSQ and VRSQ. This finding supports the findings of Somrak
et al. [30] in which the VRISE sub-score of the VRNQ was significantly correlated with the
SSQ total score. Nevertheless, the current study meticulously examined the reliability and
validity of the total scores and the sub-scores of both versions of the CSQ-VR. Beyond
their convergent validity, the associations between the sub-scores of the CSQ-VR (both
versions) (i.e., nausea, vestibular, and oculomotor) and the equivalent sub-scores of the
SSQ support the construct validity of both versions of the CSQ-VR in examining the whole
range of cybersickness symptomatology. Therefore, both the paper-and-pencil and the VR
versions of the CSQ-VR are highly reliable and are valid tools for measuring the presence
and intensity of cybersickness symptoms in VR.
4.1. Comparison of CSQ-VR, SSQ, and VRSQ
Several studies have reported that the SSQ does not have adequate psychometric
properties for measuring cybersickness in VR [20,51,52]. The findings of this current study
are aligned with the previous literature. The inadequacy and inappropriateness of the SSQ
for measuring cybersickness in VR have also been confirmed. Specifically, the overall and
sub-scores for the SSQ and the VRSQ displayed internal consistency which was substan-
tially inferior to the respective internal consistency of the CSQ-VR total score and sub-
scores. Moreover, the nausea item of the SSQ revealed internal consistency that was below
the parsimonious threshold of 0.7 for Cronbach’s α, which is required for a tool to be used
in research and professional settings [53]. Likewise, the oculomotor sub-score of the VRSQ
was well below this threshold. Given that the VRSQ has only two sub-scores (i.e.,
Figure 8. Mixed Regression Model of Pupil Size Predicting Cybersickness Intensity.
4. Discussion
The CSQ-VR is an adapted and enhanced version of the VRISE section and sub-score
of the VNRQ. Based on the recommendations by Ursachi et al. [
41
] for Cronbach’s
α
, the
CSQ-VR displayed good to very good internal consistency. This finding is aligned with
Virtual Worlds 2023,231
the high structural validity and internal consistency of the VRNQ and its VRISE sub-score,
which have previously been observed [
29
]. Additionally, the total scores and sub-scores
of both versions of the CSQ-VR showed robust correlations with their respective sub-
scores and total scores of the SSQ and VRSQ. This finding supports the findings of Somrak
et al. [
30
] in which the VRISE sub-score of the VRNQ was significantly correlated with
the SSQ total score. Nevertheless, the current study meticulously examined the reliability
and validity of the total scores and the sub-scores of both versions of the CSQ-VR. Beyond
their convergent validity, the associations between the sub-scores of the CSQ-VR (both
versions) (i.e., nausea, vestibular, and oculomotor) and the equivalent sub-scores of the
SSQ support the construct validity of both versions of the CSQ-VR in examining the whole
range of cybersickness symptomatology. Therefore, both the paper-and-pencil and the VR
versions of the CSQ-VR are highly reliable and are valid tools for measuring the presence
and intensity of cybersickness symptoms in VR.
4.1. Comparison of CSQ-VR, SSQ, and VRSQ
Several studies have reported that the SSQ does not have adequate psychometric
properties for measuring cybersickness in VR [
20
,
51
,
52
]. The findings of this current
study are aligned with the previous literature. The inadequacy and inappropriateness of
the SSQ for measuring cybersickness in VR have also been confirmed. Specifically, the
overall and sub-scores for the SSQ and the VRSQ displayed internal consistency which was
substantially inferior to the respective internal consistency of the CSQ-VR total score and
sub-scores. Moreover, the nausea item of the SSQ revealed internal consistency that was
below the parsimonious threshold of 0.7 for Cronbach’s
α
, which is required for a tool to be
used in research and professional settings [
53
]. Likewise, the oculomotor sub-score of the
VRSQ was well below this threshold. Given that the VRSQ has only two sub-scores (i.e.,
disorientation and oculomotor), half of the test was found to be unreliable. This finding
agrees with the serious limitations reported in VRSQ development and validation, which
was conducted using smartphone VR (i.e., Samsung Gear VR) and not PC or standalone
VR, a very simplistic task (i.e., target selection) and stimuli (i.e., small and large buttons),
which were not efficient in inducing adequate levels of cybersickness in a relatively small
sample [
21
]. As a result, all the items pertinent to nausea, which is the second most frequent
symptom of cybersickness in VR [
7
,
22
–
24
,
51
], were dropped. Thus, it comes as no surprise
that both the SSQ and VRSQ displayed problematic consistencies in certain sub-scores and
overall inferior reliability for the total and sub-scores of the CSQ-VR.
Furthermore, the SSQ has received criticism for its highly complex structure and
scoring [
30
,
51
]. The CSQ-VR has previously been strongly preferred over the SSQ because
of its easily calculated and interpretable scores [
30
]. The VRSQ, derived from the SSQ,
has predominantly maintained the SSQ structure and scoring system, although the VRSQ
scoring system requires somewhat simpler calculations. Nevertheless, as was also seen
in this study, both the SSQ and VRSQ suffer in terms of structure. Additionally, given
that the design of the questions and available responses use a Likert scale that is essential
for collecting reliable and informative data [
25
–
28
], the CSQ-VR has an advantage over
the SSQ and VRSQ. Both the SSQ and VRSQ use a 4-point Likert scale, while the relevant
literature suggests that a 7-point Likert scale, especially when combining a number with
textual information (e.g., “6–Very Intense Feeling”) like in the CSQ-VR, are substantially
more efficient in providing useful and representative self-reports [
25
–
28
]. The design of
the general instructions (i.e., “Please, from 1 to 7, circle the response that better corresponds to
the presence and intensity of the symptom.”) and questions (e.g., “Nausea A: Do you experience
nausea (e.g., stomach pain, acid reflux, or tension to vomit)?”) are also more explicit in the
CSQ-VR than the equivalent design in the SSQ and VRSQ (i.e., general instruction: “Circle
how much each symptom below is affecting you now.”; question: “Nausea”). Finally, the SSQ has
16 questions measuring the whole range of symptoms. The VRSQ has nine questions, but
it measures only the vestibular and oculomotor-related symptoms, while the CSQ-VR is
shorter, measuring the whole range of cybersickness with only six questions. Therefore, the
Virtual Worlds 2023,232
CSQ-VR is a shorter questionnaire with an overall superior design to the SSQ and VRSQ,
which was also reflected in the psychometric properties examined in this study.
The previous literature has shown that cybersickness, particularly when symptoms
are strong, may affect the cognitive and/or motor skills of the user [
12
–
14
], especially their
reaction speed [
10
,
17
,
18
]. It is thus assumed that a questionnaire designed to measure cy-
bersickness would also be effective in detecting relevant declines in performance. The total
score for both versions of the CSQ-VR showed high sensitivity and specificity in detecting
these temporary declines in performance due to cybersickness, while the psychometric
properties of the total scores of the SSQ and VRSQ were substantially lower and inadequate.
Furthermore, two sub-scores (nausea and vestibular) of the CSQ-VR were also highly
sensitive and specific in the detection of temporary declines, while the equivalent scores of
the SSQ and VRSQ (which does not include a nausea score) were either significantly inferior
or inadequate. Thus, the CSQ-VR is the only questionnaire that is effective in detecting
these temporary declines in performance modulated by cybersickness. Given that VR is im-
plemented in education [
1
], professional training [
2
], neuropsychological assessments [
54
],
and therapy [
4
], where cognitive and motor skills should be reliable, it is essential that a tool
should be able to provide information that these skills may have been compromised by cy-
bersickness symptomatology. Finally, beyond these applications, VR is gradually becoming
established as a research tool in scientific fields, such as human–computer interactions [
55
]
and psychological sciences [
3
], where cybersickness may compromise the reliability of the
scientific findings [
12
]. Thus, the detection of a participant whose performance has been
compromised by cybersickness enables the exclusion of this participant or observation
from the analyses and assures the data’s reliability.
Nevertheless, as was also observed in this study, a participant’s performance may not
be affected throughout the experiment. In previous studies, changes in the intensity of
cybersickness during exposure can occur in terms of an increase due to aggravation [
22
]
or a decrease due to cultivated tolerance [
56
]. Therefore, the continuous or repetitive
assessment of cybersickness is required while the participant/user is immersed. Instead
of excluding all of a participant’s observations, the CSQ-VR allows a researcher to drop
only those particular observations where a participant’s performance was affected by
cybersickness. Considering that the VR version of the CSQ-VR has shown comparable
(and sometimes superior) psychometric properties to its paper-and-pencil version, it can
detect specific compromised observations/performance and suggest its exclusion from
analyses. Nevertheless, beyond self-reports, there are other neuro and biomarkers that
have been used to detect and measure cybersickness [
8
]. Specifically, researchers have
efficiently implemented electroencephalography [
57
,
58
] and eye tracking [
59
,
60
] to detect
and appraise the occurrence and intensity of cybersickness. The VR version of the CSQ-
VR also benefits from eye-tracking metrics. In this study, pupil size was found to be a
significant predictor of cybersickness. A decrease in pupil size indicated higher intensity
cybersickness and vice versa, a pattern that has been previously observed between pupil
size and negative emotions [
31
]. Previously, pupil size has been included in a deep fusion
model for predicting cybersickness [
60
]; however, its relationship, predictive ability and
contribution to this model were not evaluated, preventing a conclusion of whether pupil
size is a biomarker of cybersickness. This study provides evidence postulating that pupil
size is indeed a biomarker of cybersickness, as well as its intensity. The VR version of the
CSQ-VR thus has an additional advantage of incorporating pupillometry.
4.2. Limitations and Future Studies
The current study also has limitations that should be considered. The sample consisted
of young adults, which prevented the examination of cybersickness in a more age-diverse
population. Future studies should attempt to examine cybersickness in a sample with
a greater age spectrum to enable the study of age differences in tolerance and/or sus-
ceptibility towards cybersickness. Additionally, this study implemented a parsimonious
inclusion criterion based on the MSSQ scores (i.e., excluding individuals who scored higher
Virtual Worlds 2023,233
than the 75th percentile and could experience substantially more frequent and stronger
cybersickness symptomatology). Given that the intensity and prevalence of cybersickness
substantially differ across individuals, future studies should explore the effects of cyber-
sickness on cognitive and motor skills in a sample that may experience stronger symptoms.
Finally, the assessment included only working memory and psychomotor tests. Future
studies should strive to examine more complex cognitive functions (e.g., episodic memory
or decision making) and motor skills (e.g., tasks that require fine motor skills and accuracy).
5. Conclusions
The CSQ-VR is a short, valid and reliable tool of cybersickness, which has superior
psychometric properties to the SSQ and VRSQ. Additionally, the paper-and-pencil and
the VR versions of the CSQ-VR were highly sensitive and specific in detecting temporary
performance declines that were modulated by cybersickness. The VR version of the CSQ-
VR provides further advantages by facilitating an assessment of cybersickness in the virtual
environment while the participant/user is immersed. Finally, the VR version of the CSQ-VR
benefits from pupillometry (i.e., measurement of pupil diameter), which was found to
predict the presence and intensity of cybersickness. Pupillometry may thus be applied in
VR as a biomarker of positive (e.g., amusement) and negative (e.g., frustration) emotions.
Supplementary Materials:
The Cybersickness in Virtual Reality Questionnaire (CSQ-VR) can be
downloaded at: http://dx.doi.org/10.13140/RG.2.2.36571.03362.
Author Contributions:
Conceptualization, P.K., F.A. and S.E.M.; methodology, P.K., F.A. and S.E.M.;
software, P.K.; validation, P.K., F.A. and S.E.M.; formal analysis, P.K.; investigation, J.L. and R.A.;
resources, P.K.; data curation, J.L. and R.A.; writing—original draft preparation, P.K., J.L. and R.A.;
writing—review and editing, P.K., F.A. and S.E.M.; visualization, P.K.; supervision, P.K. and S.M;
project administration, P.K. and S.E.M.; funding acquisition, J.L. and R.A. All authors have read and
agreed to the published version of the manuscript.
Funding: This research received no external funding.
Institutional Review Board Statement:
The study was conducted in accordance with the Declaration
of Helsinki, and approved by the Philosophy, Psychology and Language Sciences Research Ethics
Committee of the University of Edinburgh (269-2122/4; 14 June 2022).
Informed Consent Statement:
Informed consent was obtained from all subjects involved in
the study.
Data Availability Statement:
The data presented in this study are available on request from the
corresponding author. The data are not publicly available due to the ethical approval requirements.
Acknowledgments:
This work was funded by the School of Philosophy, Psychology and Language
Sciences of the University of Edinburgh. The authors would like to thank the uCreate Studio of the
University of Edinburgh for providing them with the VR equipment and tech support.
Conflicts of Interest: The authors declare no conflict of interest.
References
1.
Radianti, J.; Majchrzak, T.A.; Fromm, J.; Wohlgenannt, I. A systematic review of immersive virtual reality applications for higher
education: Design elements, lessons learned, and research agenda. Comput. Educ. 2019,147, 103778. [CrossRef]
2.
Xie, B.; Liu, H.; Alghofaili, R.; Zhang, Y.; Jiang, Y.; Lobo, F.D.; Li, C.; Li, W.; Huang, H.; Akdere, M.; et al. A Review on Virtual
Reality Skill Training Applications. Front. Virtual Real. 2021,2, 645153. [CrossRef]
3.
Kourtesis, P.; MacPherson, S.E. How immersive virtual reality methods may meet the criteria of the National Academy of
Neuropsychology and American Academy of Clinical Neuropsychology: A software review of the Virtual Reality Everyday
Assessment Lab (VR-EAL). Comput. Hum. Behav. Rep. 2021,4, 100151. [CrossRef]
4.
Emmelkamp, P.M.; Meyerbröker, K. Virtual Reality Therapy in Mental Health. Annu. Rev. Clin. Psychol.
2021
,17, 495–519.
[CrossRef] [PubMed]
5. Cruz-Neira, C.; Fernández, M.; Portalés, C. Virtual Reality and Games. Multimodal Technol. Interact. 2018,2, 8. [CrossRef]
6. Rebenitsch, L.; Owen, C. Estimating cybersickness from virtual reality applications. Virtual Real. 2020,25, 165–174. [CrossRef]
7.
Stanney, K.M.; Kennedy, R.S.; Drexler, J.M. Cybersickness is Not Simulator Sickness. In Proceedings of the Human Factors and
Ergonomics Society Annual Meeting; SAGE Publications: New York, NY, USA, 1997; Volume 41, pp. 1138–1142.
Virtual Worlds 2023,234
8.
Davis, S.; Nesbitt, K.; Nalivaiko, E. A Systematic Review of Cybersickness. In Proceedings of the 2014 Conference on Interactive
Entertainment, Newcastle, Australia, 2 December 2014; Blackmore, K., Nesbitt, K., Smith, S.P., Eds.; ACM: New York, NY, USA,
2014. [CrossRef]
9. LaViola, J.J. A discussion of cybersickness in virtual environments. ACM SIGCHI Bull. 2000,32, 47–56. [CrossRef]
10.
Nesbitt, K.; Davis, S.; Blackmore, K.; Nalivaiko, E. Correlating reaction time and nausea measures with traditional measures of
cybersickness. Displays 2017,48, 1–8. [CrossRef]
11.
Kim, J.; Palmisano, S.; Luu, W.; Iwasaki, S. Effects of Linear Visual-Vestibular Conflict on Presence, Perceived Scene Stability and
Cybersickness in the Oculus Go and Oculus Quest. Front. Virtual Real. 2021,2, 582156. [CrossRef]
12.
Kourtesis, P.; Collina, S.; Doumas, L.A.A.; MacPherson, S.E. Technological Competence Is a Pre-condition for Effective Implemen-
tation of Virtual Reality Head Mounted Displays in Human Neuroscience: A Technological Review and Meta-Analysis. Front.
Hum. Neurosci. 2019,13, 342. [CrossRef]
13.
Saredakis, D.; Szpak, A.; Birckhead, B.; Keage, H.A.D.; Rizzo, A.; Loetscher, T. Factors Associated with Virtual Reality Sickness in
Head-Mounted Displays: A Systematic Review and Meta-Analysis. Front. Hum. Neurosci. 2020,14, 96. [CrossRef] [PubMed]
14.
Conner, N.O.; Freeman, H.R.; Jones, J.A.; Luczak, T.; Carruth, D.; Knight, A.C.; Chander, H. Virtual Reality Induced Symptoms
and Effects: Concerns, Causes, Assessment & Mitigation. Virtual Worlds 2022,1, 130–146. [CrossRef]
15.
Dahlman, J.; Sjörs, A.; Lindström, J.; Ledin, T.; Falkmer, T. Performance and Autonomic Responses during Motion Sickness. Hum.
Factors: J. Hum. Factors Ergon. Soc. 2009,51, 56–66. [CrossRef]
16.
Varmaghani, S.; Abbasi, Z.; Weech, S.; Rasti, J. Spatial and attentional after effects of virtual reality and relations to cybersickness.
Virtual Real. 2021,26, 659–668. [CrossRef]
17.
Mittelstaedt, J.M.; Wacker, J.; Stelling, D. VR aftereffect and the relation of cybersickness and cognitive performance. Virtual Real.
2018,23, 143–154. [CrossRef]
18.
Nalivaiko, E.; Davis, S.L.; Blackmore, K.; Vakulin, A.; Nesbitt, K. Cybersickness provoked by head-mounted display affects
cutaneous vascular tone, heart rate and reaction time. Physiol. Behav. 2015,151, 583–590. [CrossRef]
19.
Kennedy, R.S.; Lane, N.E.; Berbaum, K.; Lilienthal, M.G. Simulator Sickness Questionnaire: An Enhanced Method for Quantifying
Simulator Sickness. Int. J. Aviat. Psychol. 1993,3, 203–220. [CrossRef]
20.
Sevinc, V.; Berkman, M.I. Psychometric evaluation of Simulator Sickness Questionnaire and its variants as a measure of
cybersickness in consumer virtual environments. Appl. Ergon. 2019,82, 102958. [CrossRef] [PubMed]
21.
Kim, H.K.; Park, J.; Choi, Y.; Choe, M. Virtual reality sickness questionnaire (VRSQ): Motion sickness measurement index in a
virtual reality environment. Appl. Ergon. 2018,69, 66–73. [CrossRef]
22.
Sharples, S.; Cobb, S.; Moody, A.; Wilson, J.R. Virtual reality induced symptoms and effects (VRISE): Comparison of head
mounted display (HMD), desktop and projection display systems. Displays 2008,29, 58–69. [CrossRef]
23.
Bohil, C.; Alicea, B.; Biocca, F.A. Virtual reality in neuroscience research and therapy. Nat. Rev. Neurosci.
2011
,12, 752–762.
[CrossRef]
24.
Palmisano, S.; Mursic, R.; Kim, J. Vection and cybersickness generated by head-and-display motion in the Oculus Rift. Displays
2017,46, 1–8. [CrossRef]
25.
Dawes, J. Do Data Characteristics Change According to the Number of Scale Points Used? An Experiment Using 5-Point, 7-Point
and 10-Point Scales. Int. J. Mark. Res. 2008,50, 61–104. [CrossRef]
26.
Joshi, A.; Kale, S.; Chandel, S.; Pal, D.K. Likert Scale: Explored and Explained. Br. J. Appl. Sci. Technol.
2015
,7, 396–403. [CrossRef]
27.
Taherdoost, H. What is the best response scale for survey and questionnaire design; review of different lengths of rating
scale/attitude scale/Likert scale. Int. J. Acad. Res. Manag. 2019,8, 1–10.
28.
Wakita, T.; Ueshima, N.; Noguchi, H. Psychological Distance between Categories in the Likert Scale. Educ. Psychol. Meas.
2012
,72,
533–546. [CrossRef]
29.
Kourtesis, P.; Collina, S.; Doumas, L.A.A.; MacPherson, S.E. Validation of the Virtual Reality Neuroscience Questionnaire:
Maximum Duration of Immersive Virtual Reality Sessions without the Presence of Pertinent Adverse Symptomatology. Front.
Hum. Neurosci. 2019,13, 417. [CrossRef]
30.
Somrak, A.; Pogaˇcnik, M.; Guna, J. Suitability and Comparison of Questionnaires Assessing Virtual Reality-Induced Symptoms
and Effects and User Experience in Virtual Environments. Sensors 2021,21, 1185. [CrossRef]
31.
Partala, T.; Surakka, V. Pupil size variation as an indication of affective processing. Int. J. Hum.-Comput. Stud.
2003
,59, 185–198.
[CrossRef]
32.
Schwind, V.; Knierim, P.; Tasci, C.; Franczak, P.; Haas, N.; Henze, N. “These Are Not My Hands!”: Effect of Gender on the
Perception of Avatar Hands in Virtual Reality. In Proceedings of the 2017 CHI Conference on Human Factors in Computing
Systems, New York, NY, USA, 6–11 May 2017; pp. 1577–1582. [CrossRef]
33.
Bebko, A.O.; Troje, N.F. bmlTUX: Design and Control of Experiments in Virtual Reality and Beyond. I-Perception
2020
,11.
[CrossRef]
34.
Bonato, F.; Bubka, A.; Palmisano, S. Combined Pitch and Roll and Cybersickness in a Virtual Environment. Aviat. Space Environ.
Med. 2009,80, 941–945. [CrossRef] [PubMed]
35.
Kourtesis, P.; Korre, D.; Collina, S.; Doumas, L.A.A.; MacPherson, S.E. Guidelines for the Development of Immersive Virtual
Reality Software for Cognitive Neuroscience and Neuropsychology: The Development of Virtual Reality Everyday Assessment
Lab (VR-EAL), a Neuropsychological Test Battery in Immersive Virtual Reality. Front. Comput. Sci. 2020,1. [CrossRef]
Virtual Worlds 2023,235
36. Wechsler, D. Wechsler Bellevue Adult Intelligence Scale; Williams & Wilkins: Baltimore, MD, USA, 1939.
37. Corsi, P.M. Human memory and the medial temporal region of the brain. Diss. Abstr. Int. 1972,34, 891.
38.
Deary, I.J.; Liewald, D.; Nissan, J. A free, easy-to-use, computer-based simple and four-choice reaction time programme: The
Deary-Liewald reaction time task. Behav. Res. Methods 2010,43, 258–268. [CrossRef] [PubMed]
39.
Golding, J.F. Predicting individual differences in motion sickness susceptibility by questionnaire. Pers. Individ. Differ.
2006
,41,
237–248. [CrossRef]
40.
Roth, H.L.; Lora, A.N.; Heilman, K.M. Effects of monocular viewing and eye dominance on spatial attention. Brain
2002
,125,
2023–2035. [CrossRef]
41.
Ursachi, G.; Horodnic, I.A.; Zait, A. How Reliable Are Measurement Scales? External Factors with Indirect Influence on Reliability
Estimators. Procedia Econ. Financ. 2015,20, 679–686. [CrossRef]
42. Power, M.; Fell, G.; Wright, M. Principles for high-quality, high-value testing. Evid.-Base. Med. 2013,18, 5–10. [CrossRef]
43.
Streiner, D.L.; Cairney, J. What’s under the ROC? An Introduction to Receiver Operating Characteristics Curves. Can. J. Psychiatry
2007,52, 121–128. [CrossRef]
44.
Guilmette, T.J.; Sweet, J.J.; Hebben, N.; Koltai, D.; Mahone, E.M.; Spiegler, B.J.; Stucky, K.; Westerveld, M.; Participants, C.
American Academy of Clinical Neuropsychology consensus conference statement on uniform labeling of performance test scores.
Clin. Neuropsychol. 2020,34, 437–453. [CrossRef]
45. The Jamovi Project [Computer Software]. Available online: http://www.jamovi.org (accessed on 1 January 2020).
46.
R Core Team. A Language and Environment for Statistical Computing; R Foundation for Statistical Computing: Vienna, Austria, 2022.
Available online: https://www.R-project.org/ (accessed on 12 December 2022).
47.
Peterson, R.A.; Cavanaugh, J.E. Ordered quantile normalization: A semiparametric transformation built for the cross-validation
era. J. Appl. Stat. 2019,47, 2312–2327. [CrossRef] [PubMed]
48.
Revelle, W. psych: Procedures for Psychological, Psychometric, and Personality Research; Northwestern University: Evanston, IL, USA,
2018. Available online: https://CRAN.R-project.org/package=psych (accessed on 12 December 2022).
49.
Wickham, H. ggplot2: Elegant Graphics for Data Analysis; Springer-Verlag: New York, NY, USA, 2016. Available online: https:
//ggplot2.tidyverse.org (accessed on 12 December 2022)ISBN 978-3-319-24277-4.
50.
Bates, D.; Mächler, M.; Bolker, B.; Walker, S. Fitting Linear Mixed-Effects Models Using lme4. J. Stat. Softw.
2015
,67, 48. [CrossRef]
51.
Bouchard, S.; Berthiaume, M.; Robillard, G.; Forget, H.; Daudelin-Peltier, C.; Renaud, P.; Blais, C.; Fiset, D. Arguing in Favor of
Revising the Simulator Sickness Questionnaire Factor Structure When Assessing Side Effects Induced by Immersions in Virtual
Reality. Front. Psychiatry 2021,12, 739742. [CrossRef] [PubMed]
52.
Iii, W.B.S. Psychometric Evaluation of the Simulator Sickness Questionnaire as a Measure of Cybersickness. Ph.D. Thesis, Iowa
State University, Ames, IA, USA, 2017. [CrossRef]
53. Nunnally, J.C. Psychometric Theory—25 Years Ago and Now. Educ. Res. 1975,4, 7–21. [CrossRef]
54.
Kourtesis, P.; Collina, S.; Doumas, L.A.; MacPherson, S.E. Validation of the Virtual Reality Everyday Assessment Lab (VR-EAL):
An Immersive Virtual Reality Neuropsychological Battery with Enhanced Ecological Validity. J. Int. Neuropsychol. Soc.
2020
,27,
181–196. [CrossRef]
55.
Karray, F.; Alemzadeh, M.; Abou Saleh, J.; Arab, M.N. Human-Computer Interaction: Overview on State of the Art. Int. J. Smart
Sens. Intell. Syst. 2008,1, 137–159. [CrossRef]
56.
Stanney, K.; Lawson, B.D.; Rokers, B.; Dennison, M.; Fidopiastis, C.; Stoffregen, T.; Weech, S.; Fulvio, J.M. Identifying Causes
of and Solutions for Cybersickness in Immersive Technology: Reformulation of a Research and Development Agenda. Int. J.
Hum.–Comput. Interact. 2020,36, 1783–1803. [CrossRef]
57.
Jeong, D.; Yoo, S.; Yun, J. Cybersickness Analysis with EEG Using Deep Learning Algorithms. In Proceedings of the 2019 IEEE
Conference on Virtual Reality and 3D User Interfaces (VR), Osaka, Japan, 23–27 March 2019; pp. 827–835. [CrossRef]
58. Krokos, E.; Varshney, A. Quantifying VR cybersickness using EEG. Virtual Real. 2021,26, 77–89. [CrossRef]
59.
Lopes, P.; Tian, N.; Boulic, R. Eye Thought You Were Sick! Exploring Eye Behaviors for Cybersickness Detection in VR. In
Proceedings of the 13th ACM SIGGRAPH Conference on Motion, Interaction and Games, New York, NY, USA, 16–18 October
2020. [CrossRef]
60.
Islam, R.; Desai, K.; Quarles, J. Cybersickness Prediction from Integrated HMD’s Sensors: A Multimodal Deep Fusion Approach
using Eye-tracking and Head-tracking Data. In Proceedings of the 2021 IEEE International Symposium on Mixed and Augmented
Reality (ISMAR), Bari, Italy, 4–8 October 2021; pp. 31–40. [CrossRef]
Disclaimer/Publisher’s Note:
The statements, opinions and data contained in all publications are solely those of the individual
author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to
people or property resulting from any ideas, methods, instructions or products referred to in the content.
