ArticlePDF Available

Figures

Content may be subject to copyright.
(This is a sample cover image for this issue. The actual cover is not yet available at this time.)
This article appeared in a journal published by Elsevier. The attached
copy is furnished to the author for internal non-commercial research
and education use, including for instruction at the authors institution
and sharing with colleagues.
Other uses, including reproduction and distribution, or selling or
licensing copies, or posting to personal, institutional or third party
websites are prohibited.
In most cases authors are permitted to post their version of the
article (e.g. in Word or Tex form) to their personal website or
institutional repository. Authors requiring further information
regarding Elsevier’s archiving and manuscript policies are
encouraged to visit:
http://www.elsevier.com/copyright
Author's personal copy
Journal
of
Neuroscience
Methods
210 (2012) 125–
131
Contents
lists
available
at
SciVerse
ScienceDirect
Journal
of
Neuroscience
Methods
jou
rnal
h
om
epa
ge:
www.elsevier.com/locate/jneumeth
Computational
Neuroscience
Development
and
initial
assessment
of
a
new
paradigm
for
assessing
cognitive
and
motor
inhibition:
The
bimodal
virtual-reality
Stroop
Mylène
Henry,
Christian
C.
Joyal,
Pierre
Nolin
Laboratoire
de
Recherche
Interdisciplinaire
en
Réalité
Virtuelle
(LARI-RV),
Department
of
Psychology,
Université
du
Québec
à
Trois-Rivières,
Quebec,
Canada
h
i
g
h
l
i
g
h
t
s
We
developed
a
new
measure
of
inhibition
based
on
virtual
reality
(VR).
Our
bimodal
environment
allows
assessment
of
motor
and
cognitive
inhibition.
The
VR
environment
is
a
simple,
short,
and
multi-component
measure
of
inhibition.
a
r
t
i
c
l
e
i
n
f
o
Article
history:
Received
18
April
2012
Received
in
revised
form
20
July
2012
Accepted
31
July
2012
Keywords:
Virtual
reality
Inhibition
Impulsivity
Neuropsychology
Apartment
Stroop
Go
no-go
Adults
a
b
s
t
r
a
c
t
Assessing
and
predicting
inhibition
in
adults
is
a
common
assignment
for
clinicians.
However,
there
is
no
single
measure
of
inhibition
that
is
complete,
sensitive
and
enjoyable.
The
main
goal
of
this
study
was
to
develop
a
virtual
reality
neuropsychological
task
(the
bimodal
VR-Stroop)
capable
of
measuring
both
cognitive
(control
of
internal
and
external
interference)
and
motor
inhibition
(a
go
no-go
paradigm
with
reaction
time
variation,
commission
errors
and
omissions).
Preliminary
data
obtained
with
71
healthy
adult
participants
confirmed
that
the
VR-Stroop
is
capable
of
eliciting
the
Stroop
effect
with
bimodal
stimuli.
Initial
validation
data
also
suggested
that
measures
of
the
VR-Stroop
significantly
correlate
with
measures
of
the
Elevator
counting
with
distracters,
the
Continuous
Performance
Task
(CPT-II),
and
the
Stop-it
task.
Finally,
regression
analyses
indicated
that
commission
errors
and
variability
of
reaction
times
at
the
VR-Stroop
were
significantly
predicted
by
scores
of
the
Elevator
task
and
the
CPT-II.
These
preliminary
results
suggest
that
the
VR-Stroop
is
an
interesting
measure
of
cognitive
and
motor
inhibition
for
adults,
although
confirmatory
investigations
are
warranted.
© 2012 Elsevier B.V. All rights reserved.
1.
Introduction
Lack
of
inhibition
(or
impulsivity)
is
among
the
most
common
manifestations
of
mental
disorder
diagnoses
(APA,
2000;
Moeller
et
al.,
2001).
It
is
also
one
of
the
most
common
behaviors
assessed
by
clinicians
(e.g.
Lezak
et
al.,
2004).
Yet,
available
measures
of
inhibi-
tion/impulsivity
are
often
considered
unsatisfactory
or
incomplete
as
they
are
associated
with
low
sensitivity
and
poor
predictive
value,
especially
among
clinical
populations
(e.g.
with
psychiatric
and/or
neurological
impairments;
Mathias
et
al.,
2008;
Moeller
et
al.,
2001;
Reynolds
et
al.,
2006).
The
main
goal
of
this
study
was
to
develop
a
single,
yet
more
complete,
assessment
of
inhi-
bition/impulsivity
using
virtual
reality.
Corresponding
author
at:
Laboratoire
de
Recherche
Interdisciplinaire
en
Réalité
Virtuelle,
Département
de
psychologie,
Université
du
Québec
à
Trois-Rivières,
C.P.
500,
Trois-Rivières,
Québec,
Canada
G9A
5H7.
Tel.:
+1
819
376
5011x3559;
fax:
+1
819
376
5195.
E-mail
address:
christian.joyal@uqtr.ca
(C.C.
Joyal).
A
first
factor
explaining
the
difficulties
of
measuring
inhibi-
tion/impulsivity
is
the
traditional
use
of
questionnaires
and
verbal
self-reports
(e.g.
the
Barratt
Impulsivity
Scale;
Patton
et
al.,
1995;
the
I7subscale,
Eysenck
et
al.,
1985;
the
UPPS
impulsive
behavioral
scale,
Whiteside
et
al.,
2005).
Results
from
these
measures
depend
heavily
on
the
collaboration
and
comprehension
of
the
examinee,
which
is
not
always
attainable
in
certain
clinical
settings
(e.g.
foren-
sic
and
general
psychiatry).
Also,
questionnaires
tend
to
reflect
long
term
traits
of
impulsivity
(as
opposed
to
acute
states),
and
their
results
often
fail
to
correlate
with
those
of
direct
(behavioral)
mea-
sures
of
acute
impulsivity
(e.g.
Gerbing
et
al.,
1987;
Reynolds
et
al.,
2006;
Horn
et
al.,
2003).
Thus,
computerized
assessments
are
better
suited
to
evaluate
acute
states
of
inhibition/impulsivity,
especially
among
clinical
populations
(e.g.
Mathias
et
al.,
2008;
Moeller
et
al.,
2001).
A
second
factor
explaining
the
difficulties
of
developing
a
satis-
fying
measure
of
inhibition/impulsivity
is
the
relative
complexity
of
the
constructs.
Different
clinical
and
research
backgrounds,
rang-
ing
from
experimental
psychology
(e.g.
Logan
and
Cowan,
1984;
Patton
et
al.,
1995),
to
adult
psychiatry
(e.g.
Moeller
et
al.,
2001),
and
developmental
psychology
(e.g.
Nigg,
2000;
Barkley,
1997)
offered
0165-0270/$
see
front
matter ©
2012 Elsevier B.V. All rights reserved.
http://dx.doi.org/10.1016/j.jneumeth.2012.07.025
Author's personal copy
126 M.
Henry
et
al.
/
Journal
of
Neuroscience
Methods
210 (2012) 125–
131
different
theoretical
accounts
of
inhibition
and
its
corollary,
impul-
sivity.
Thus,
several
subtypes
of
both
inhibition
(e.g.
behavioral
vs.
cognitive,
intentional
vs.
non
intentional,
interference
control
vs.
dyscontrol,
Nigg,
2000;
behavioral
vs.
interference
control
vs.
cognitive,
Kipp,
2005),
and
impulsivity
have
been
proposed
dur-
ing
the
past
half
century
(e.g.
motor,
attentional,
and
unplanning;
Barratt,
1965;
Patton
et
al.,
1995;
lack
of
inhibitory
control,
low
decision
time,
sensation
seeking
and
low
persistence;
Buss
and
Plomin,
1975;
urgency,
lack
of
premeditation,
lack
of
perseverance
and
sensation
seeking;
Whiteside
and
Lynam,
2001).
Overall,
how-
ever,
direct
measures
(i.e.
behavioral)
of
inhibition
and
impulsivity
are
known
to
either
assess
cognitive
inhibition
or
motor
control
(e.g.
White
et
al.,
1994).
It
would
be
best
to
develop
a
task
capable
of
measuring
both
cognitive
inhibition
and
motor
control.
Cognitive
inhibition
is
sometime
viewed
as
the
capacity
to
inhibit
access
of
irrelevant
material
in
working
memory
(a
rather
higher-order
capacity
directly
associated
with
executive
func-
tioning;
Kipp,
2005),
or
decision-making
capacities
(more
closely
associated
with
risk
taking
and/or
thrill
seeking;
e.g.
Bechara
and
van
der
Linden,
2005).
Most
commonly,
cognitive
inhibition
is
considered
as
the
capacity
to
control
interference
from
exter-
nal
(environmental)
or
internal
(e.g.
intrusive
thoughts)
stimuli
(e.g.
Kipp,
2005).
In
that
sense,
interference
control
is
a
process
that
helps
maintaining
attention
focussed
on
a
task
in
spite
of
distracters.
Thus,
the
best
would
be
to
measure
both
type
of
inter-
ference
with
the
same
task.
With
virtual
reality
(VR),
it
is
relatively
simple
to
assess
external
interference
with
introduction
of
sur-
rounding
distracters
within
the
environment.
External
distracters
(auditory
and/or
visual
elements)
render
the
task
environment
more
sensitive
and
more
ecologically
valid
than
traditional
settings
(Adams
et
al.,
2009;
Nolin
et
al.,
2009).
Moreover,
distracters
pro-
voke
head
movements,
allowing
a
better
detection
of
subtle
deficits
among
clinical
populations
(Nolin
et
al.,
2009,
2012).
Therefore,
a
main
advantage
of
virtual
neuropsychological
tasks
is
to
evaluate
skills
and
abilities
in
an
environment
that
appears
more
sensitive
and
similar
to
the
real
world
(e.g.
Matheis
et
al.,
2007).
As
for
con-
trol
of
internal
interference,
it
is
best
measured
with
the
Stroop
task,
the
most
widely
used
assessment
of
cognitive
inhibition
and
interference
control
for
adults
(e.g.
MacLeod
and
MacDonald,
2000;
Strauss
et
al.,
2006).
That
task
is
based
on
the
classic
Stroop
effect
(Stroop,
1935;
MacLeod,
1991),
related
with
the
normal
habit
of
automatically
reading
a
written
word.
When
the
name
of
a
color
and
the
ink
color
of
the
name
differ
(e.g.
the
word
BLUE
is
written
in
red)
and
a
person
must
name
only
the
ink
color,
either
his/her
response
time
or
the
number
of
errors
(or
both)
increase
compared
to
trials
where
the
name
and
its
ink
matched.
Because
inhibition
assessments
are
much
more
numerous
for
children
than
adults
(e.g.
Simpson
and
Riggs,
2005;
Korkman
et
al.,
1998;
Manly
et
al.,
1999;
see
Lezak
et
al.,
2004
for
a
compendium),
and
the
Stroop
effect
is
stronger
in
adults
than
children
(being
based
on
reading
automatic-
ity;
e.g.
MacLeod,
1991),
a
virtual
environment
adapted
for
adults
with
measures
based
on
the
Stroop
effect
was
chosen.
Pairing
envi-
ronmental
distracters
with
the
Stroop
measure,
it
becomes
possible
to
assess
interference
control
for
both
external
and
internal
stimuli.
Thus,
a
single
instrument
could
measure
both
motor
impulsiv-
ity
and
cognitive
impulsivity.
Another
virtual
reality
Stroop
task
with
distracters
was
recently
developed
by
Parsons
et
al.
(2011),
although
the
Stroop
stimuli
are
unimodal
(visual
only),
and
the
environments
are
different
(Iraqi/Afghani
war
zones
for
army
vet-
erans).
Using
unimodal
stimulus
presentation
implies
that
at
least
three
different
response
keys
are
needed
(three
different
colors).
In
this
study,
a
VR-Stroop
assessment
with
bimodal
stimuli
was
developed
to
integrate
a
measure
of
motor
control,
which
implies
a
single
response
key
(go/no-go
reaction
times).
Moreover,
a
regular
environment
has
to
be
used
to
improve
environmental
ecological
validity
for
the
general
population.
Motor
control
is
generally
viewed
as
the
capacity
to
physically
and
voluntarily
withhold
a
prepotent
or
ongoing
motor
response
(e.g.
Evenden,
1999;
Dougherty
et
al.,
2009).
Motor
control
is
best
evaluated
with
computerized
measures,
which
are
generally
based
on
go
no-go
paradigms
(e.g.
CPT-II,
Conners
et
al.,
2003;
the
TOVA,
Leark
et
al.,
2007;
The
Stop-it
task;
Verbruggen
et
al.,
2008).
The
most
commonly
used
variable
for
motor
control
assessment
is
the
commission
error,
a
reactive
act
performed
with
low
reaction
time
and
reflection,
associated
with
low
impulse
control,
high
risk
taking
tendencies,
poor
decision
making,
low
gratification
delay
capacities
and
weak
resistance
to
temptation
(e.g.
Kipp,
2005;
White
et
al.,
1994).
Motor
inhibition
and
commission
errors
are
closely
depen-
dent
upon
integrity
of
brain
circuits
involving
the
lower
parts
of
the
frontal
lobes
(e.g.
Bechara
and
van
der
Linden,
2005;
Horn
et
al.,
2003).
Another
interesting
approach
with
go
no-go
paradigms
is
to
consider
intra-individual
(and
intra-test)
variability
of
reaction
times,
which
is
associated
with
certain
types
of
neurological
condi-
tions
such
as
Attentional
Deficit
and
Hyperactivity
Disorder
(ADHD
and
the
so-called
sluggish
cognitive
tempo;
e.g.
Carlson
and
Mann,
2002).
Go
no-go
paradigms
might
also
be
used
to
indirectly
assess
the
capacity
of
inhibition
processes
with
stop-signal
tasks
(Verbruggen
et
al.,
2008).
The
Stop-it
task
in
particular
allows
to
determine
the
time
required
between
a
visual
go
signal
and
a
auditory
no-go
sig-
nal
for
an
individual
to
withhold
a
response,
which
corresponds
to
the
Stop-Signal
Reaction
Time
(SSRT),
an
index
of
inhibition
capac-
ities
(Logan
and
Cowan,
1984;
Logan
et
al.,
1997;
Verbruggen
et
al.,
2008).
The
Stop-it
is
generally
considered
as
the
best
measure
of
motor
inhibition
(e.g.
Nolan
et
al.,
2011).
It
is
also
worth
noting
that
good
performances
at
the
Stop-it
activate
brain
regions
that
are
not
identical
(they
only
partially
overlap)
with
those
associated
with
good
performances
at
the
CPT-II
(Swick
et
al.,
2011).
Thus,
these
tasks
are
not
measuring
exactly
the
same
construct
and
might
be
used
concurrently.
The
main
problem
with
go
no-go
paradigms
is
their
notori-
ous
tediousness
and
monotony
for
the
examinee.
First,
they
were
generally
developed
to
assess
vigilance
(the
capacity
to
maintain
attention
focussed
for
a
relatively
long
period
of
time),
rendering
the
assessment
long
(up
to
20
min)
and
boring.
Second,
the
ecologi-
cal
validity
and
interest
for
their
environments
(typically
Xs
and
Os
appearing
on
a
black
screen
of
a
computer
in
a
quiet
experimental
room)
are
particularly
low.
The
use
of
virtual
reality
should
vastly
improve
these
aspects
of
the
assessment.
Finally,
there
are
important
differences
between
the
concepts
and
measurements
of
interference
control
(as
measured
with
the
Stroop),
and
motor
impulsivity
(as
measured
with
go
no-go
paradigms;
e.g.
Perugini
et
al.,
2000;
van
Mourik
et
al.,
2005).
Given
that
associations
between
the
Stroop
effect
and
other
types
of
inhi-
bition
capacities
might
be
weak
(e.g.
Heflin
et
al.,
2011),
it
would
be
interesting
to
develop
a
task
capable
of
measuring
more
than
one
subtype
of
inhibition/impulsivity.
The
first
objective
of
this
study
was
to
confirm
that
a
Stroop
effect
might
be
elicited
with
a
VR
task
based
on
bimodal
stim-
uli.
The
main
goal
was
to
develop
a
single
impulsivity
measure
assessing
control
of
internal
interference
(Stroop
effect),
control
of
external
interference
(environmental
distracters),
and
motor
inhi-
bition
(simple
reaction
times
based
on
a
go-no
go
paradigm).
A
third
goal
of
this
study
was
to
conduct
a
first-step
convergent
validation
of
the
VR
task
with
a
small
group
of
participants
and
traditional
impulsivity
measures.
2.
Methods
This
study
was
conducted
in
two
phases:
(1)
a
pilot
part
dur-
ing
which
optimal
experimental
conditions
were
determined
(e.g.
Author's personal copy
M.
Henry
et
al.
/
Journal
of
Neuroscience
Methods
210 (2012) 125–
131 127
comfort
of
the
experimental
room,
difficulty
levels
of
the
task,
clar-
ity
of
instructions;
best
inter-trial
intervals
-ISI)
and
program
bugs
were
fixed
and
(2)
a
preliminary
validation
phase
during
which
additional
participants
were
assessed
with
the
VR-Stroop
and
tra-
ditional
impulsivity/inhibition
tasks.
2.1.
Participants
A
total
of
71
volunteers
participated
in
the
study.
The
pilot
phase
was
conducted
with
33
adults
recruited
among
summer
stu-
dents,
research
assistants,
and
their
friends
(mean
age:
26.1
±
9.2,
10
males,
23
females).
The
validation
phase
was
conducted
with
40
additional
adult
volunteers
(minus
two
participants
with
invalid
data),
recruited
among
the
general
population
through
newspaper
advertisements,
relatives
or
acquaintances
of
research
assistants,
co-workers
of
relatives,
undergraduate
students
and
university
non-academic
staff
(mean
age:
33.8
±
15.2,
range
19–58
years
old;
14
males,
24
females;
15
full-time
workers,
15
full-time
students,
8
others).
The
research
was
completed
in
accordance
with
the
Helsinki
Declaration.
2.2.
Measures
All
participants
were
assessed
with
the
following
impulsiv-
ity/inhibition
measures:
(1)
the
conventional
Stroop
task
(D-KEFS
version;
Delis
et
al.,
2001
measure
of
control
for
internal
inter-
ference),
(2)
the
Elevator
Counting
task
with
distraction
(TEA;
Robertson
et
al.,
1994
measure
of
control
for
external
interfer-
ence);
(3)
the
Continuous
Performance
Task
second
edition
(CPT-II;
Conners
et
al.,
2003
measure
of
reaction
times,
commissions,
omissions
and
variability);
(4)
the
Stop-it
task
(Verbruggen
et
al.,
2008
measure
of
inhibition)
and
(5)
the
VR-Stroop
task
(Henry
et
al.,
2011).
Assessment
order
was
counterbalanced
across
partic-
ipants.
Assessments
were
given
by
university
research
assistants
who
received
specific
training
and
supervision
to
do
so.
The
Stroop
task,
D-KEFS
version
(Color
Word
Interference
Task,
Delis
et
al.,
2001)
is
very
similar
to
the
original
task
(Stroop,
1935).
It
is
considered
as
a
measure
of
cognitive
(or
internal
interference)
control
assessing
the
capacity
of
a
person
to
suppress
a
habitual
response
in
favor
of
a
less
familiar
one
while
maintaining
a
goal
in
mind
(Strauss
et
al.,
2006).
The
task
includes
4
conditions
but
only
the
data
from
the
first
and
third
conditions
were
analyzed
for
the
present
study
in
order
to
match
both
conditions
of
the
vir-
tual
task.
On
the
first
condition,
participants
must
name
the
color
of
color
blocks
(red,
blue
or
green)
presented
in
pseudo-random
order
as
fast
as
they
can.
That
condition
assesses
color-blindness
selec-
tive
attention
and
speed
processing.
The
third
condition
requires
to
name
the
color
of
the
ink
in
which
color
words
(same
colors
as
in
condition
1)
are
printed
(for
example,
RED
printed
in
blue).
That
condition
assesses
cognitive
interference,
the
“Stroop
effect”.
The
Elevator
Counting
with
distracters
is
a
subtest
of
the
Test
of
Everyday
Attention
battery
(TEA,
Robertson
et
al.,
1994;
Pearson
Assessments,
1998).
It
is
a
task
of
selective
attention
and
work-
ing
memory
with
external
interference
in
which
participants
have
to
use
an
imaginary
Elevator
with
an
inoperative
floor
indicator
(McAnespie,
2001).
In
order
to
know
on
which
floor
they
are
on,
auditory
stimuli
are
presented
to
the
examinee:
low-pitched
tones
must
be
counted
as
the
Elevator
going
up
one
floor,
while
inter-
spersed
high-pitched
tones
(distracters)
must
be
ignored.
The
task
has
10
trials
and
one
point
is
given
for
each
correctly
counted
string.
The
Continuous
Performance
Task-II
(Conners
et
al.,
2003;
Multi-Health
System,
2000).
The
CPT-II
is
a
14
min
computer-administered
task
used
to
assess
motor
impulsivity
and
vigilance
(Lezak
et
al.,
2004).
Participants
must
press
a
button
as
quickly
as
possible
in
response
to
a
target
stimulus
(a
letter
appearing
in
the
center
of
the
screen)
except
when
the
letter
X
is
presented.
A
total
of
360
stimuli
are
presented,
36
of
which
are
nontarget
(“X”)
refraining
the
participant
from
responding.
Each
letter
is
presented
for
a
total
of
250
ms
and
interstimulus
interval
rates
vary
between
1,
2
or
4
s.
Motor
impulsivity
is
measured
with
commission
errors
and
high
reaction
times
while
inattention
is
measured
with
omission
errors
and
slow
response
style.
Vigilance
is
related
to
the
stability
of
responses
across
blocks
of
trials.
Mean
reaction
times
for
good
responses
and
total
responses
are
also
computed,
as
well
as
their
variations
across
blocks
of
trials.
The
Stop-it
task
(Logan
et
al.,
1997;
Verbruggen
et
al.,
2008).
This
measure
is
considered
as
one
of
the
best
computerized
measure
of
motor
inhibition
(e.g.
Nolan
et
al.,
2011).
The
stop-it
is
an
indirect
measure
of
inhibition
processing
integrity
based
on
an
algorithm
that
gradually
adapts
to
the
mean
simple
reaction
time
of
each
par-
ticipant.
The
goal
is
to
induce
errors
in
approximately
50%
of
the
trials,
no
matter
what
the
mean
reaction
time
of
a
particular
par-
ticipant
is
(Verbruggen
et
al.,
2008).
The
primary
task
consists
of
simple
reaction
time
and
shape
discrimination
where
a
left
key
has
to
be
pressed
in
response
to
the
appearance
of
a
square,
and
a
right
key
when
the
shape
is
a
circle.
These
are
the
no-signal
trials.
On
25%
of
trials,
an
auditory
signal
(a
beep
sound)
succeeds
more
or
less
rapidly
the
visual
stimulus,
in
which
cases
participants
have
to
withhold
their
motor
response
(stop-signal
trials).
A
lack
of
inhi-
bition
will
manifest
either
as
a
too
quick
“go”
response
or
a
too
slow
“stop”
response
(Verbruggen
et
al.,
2008).
The
program
varies
the
time
elapsed
between
the
go
stimuli
and
the
stop
stimuli
as
a
function
of
the
response
speed
of
each
participant.
These
stop-
signal
delay
variations
greatly
improve
sensitivity
of
the
task
(Logan
and
Cowan,
1984;
Logan
et
al.,
1997)
stop-signal
delays
increase
by
50
ms
when
the
participant
correctly
inhibits
a
response,
ren-
dering
the
next
trial
harder
to
inhibit
or
decrease
by
50
ms
after
commission
errors,
rendering
the
subsequent
trial
easier,
until
50%
of
correct
responses
is
reached
(Verbruggen
et
al.,
2008;
Logan
et
al.,
1997).
The
Stop-Signal
Reaction
Time
(SSRT)
and
the
Stop-Signal
Delay
(SSD)
were
considered
here.
The
VR-Stroop
(ClinicaVR:
Apartment
Stroop)
task
was
devel-
oped
in
collaboration
with
Digital
MediaWorks
(www.dmw.ca)
as
an
attempt
to
obtain
a
more
complete
inhibition
task,
and
to
improve
sensitivity
of
impulsivity
assessments
(Henry
et
al.,
2011).
The
environment
is
the
interior
of
a
virtual
apartment
(see
Fig.
1).
Participants
are
seated
in
the
living
room,
in
front
of
a
flat-screen
TV
set,
a
kitchen
and
a
window.
A
head-mounted
display
(HMD)
was
used
(eMagin
Z800
visor)
to
recreate
a
3D-like
effect
and
partici-
pants
were
allowed
to
look
360around
themselves
and
explore
the
environment
by
turning
their
head.
The
task
is
based
on
the
Stroop
effect
(and
measures
internal
cognitive
interference),
with
go
no-go
components
(reaction
time,
commission
errors
and
omis-
sion
errors,
reaction
time
variability)
and
external
interference
(audio–visual
environmental
stimuli).
It
consists
of
two
conditions.
In
the
first
condition,
a
series
of
color
rectangles
appear
on
the
television
screen
(blue,
red
or
green,
pseudo-randomly)
while
names
of
colors
(blue,
red
or
green)
are
verbally
stated
through
the
computer
speakers
(male
or
female
voice,
the
female
voice
was
chosen
for
all
participants),
at
the
same
time
with
the
same
pace
(bimodal
presentations).
Participants
must
click
on
the
left
button
of
a
mouse
with
their
preferred
hand
as
quickly
as
possible
when
the
color
named
(audio
stimulus)
matches
the
color
shown
(visual
stimulus).
They
must
withhold
their
response
in
mismatched
trials.
A
total
of
144
stimuli
are
presented,
including
72
targets
(go
responses).
During
the
task,
distracters
appear
in
different
areas
of
the
environment
(center
(C),
left
(L)
or
right
(R)).
Some
distracters
are
audio–visual
(School
Bus
passing
on
the
street
(R);
SUV
(R);
iPhone
on
the
table
(C);
Toy
Robot
on
the
floor
(C)),
others
are
auditory
(Crumple
Paper
(L);
Drop
Pencil
Author's personal copy
128 M.
Henry
et
al.
/
Journal
of
Neuroscience
Methods
210 (2012) 125–
131
Fig.
1.
Capture
of
the
VR-Stroop
environment
(during
condition
2).
(For
interpretation
of
the
references
to
color
in
the
text,
the
reader
is
referred
to
the
web
version
of
the
article.)
(L);
Doorbell
(L)
Kat
Clock
(L)
Vacuum
Cleaner
(R)
Jack
Hammer
(R)
Sneeze
(L)
Jet
Noise
(C))
and
some
are
visual
(Paper
plane
(L
R),
Woman
walking
in
the
kitchen
(C)).
That
condition
was
designed
to
assess
reaction
times
(simple
and
complex),
selective
attention
(matching
the
auditory
and
visual
stimuli),
and
external
interfer-
ence
control
(environmental
distracters).
The
duration
of
condition
1
is
4.8
min.
Inter-Stimulus
Intervals
(ISI)
of
2000
ms
and
1000
ms
were
used
in
the
pilot
phase
to
determine
the
most
efficient.
In
the
second
condition,
color
words
are
presented
on
the
screen,
written
with
matched
ink
color
(BLUE
written
in
blue,
congruent
trial)
or
different
ink
color
(e.g.
BLUE
written
in
red,
incongruent
trial
see
Fig.
1).
The
colors
are
stated
by
the
same
voice
as
in
condition
1
and
participants
must
click
on
the
mouse
when
the
color
heard
is
the
same
as
the
ink
color
(target
stimuli,
congruent
or
incongruent)
but
not
the
color
word.
A
total
of
144
stimuli
are
pre-
sented,
including
72
targets
(go
responses),
divided
in
36
congruent
and
36
incongruent
stimuli.
During
the
task,
the
same
distracters
as
in
condition
1
appear
in
the
environment.
That
condition
was
designed
to
assess
cognitive
interference
(Stroop
effect)
in
addition
to
the
measures
of
condition
1
(go
no-go
variables
and
external
interference).
The
duration
of
condition
2
is
also
4.8
min,
for
a
total
task
duration
of
9.6
min.
Measures
include:
(1)
the
mean
total
reaction
time;
(2)
the
mean
reaction
time
for
correct
responses;
(3)
variation
(standard
deviations)
of
reaction
times;
(4)
variation
(standard
deviations)
of
reaction
times
for
correct
responses
(5)
the
number
of
correct
responses;
(6)
the
number
of
commission
errors
and
(7)
the
number
of
omission
errors.
Questionnaires.
After
completion
of
the
neuropsychological
tasks,
participants
(n
=
71)
filled
two
questionnaires
describing
their
VR
experience:
(1)
the
realistic
subscale
of
the
Presence
Questionnaire
(Witmer
and
Singer,
1994;
adapted
version
of
UQO
Cyberpsychology
Laboratory;
Robillard
et
al.,
2002)
evaluated
the
realism
of
the
VR
task
with
7
questions
arranged
on
a
Likert
scales
(from
1
to
7)
and
(2)
the
Simulator
Sickness
Questionnaire
(Kennedy
et
al.,
1993;
adapted
version
of
UQO
Cyberpsychology
Laboratory;
Bouchard
et
al.,
2007)
assessed
the
occurrence,
nature
and
severity
of
sickness
symptoms
induced
by
VR
environments
with
16
items
to
be
rated
on
a
scale
from
0
to
3.
2.3.
Statistical
analyses
Given
the
preliminary
and
exploratory
nature
of
this
study
(only
one
group
of
participants
was
involved
and
only
38
participants
were
included),
statistical
analyses
will
focus
more
on
avoiding
type
II
errors
(masking
genuine
effects
with
restrictive
analyses)
than
type
I
errors
(reporting
spurious
or
fortuitous
statistical
sig-
nificance).
The
primary
results
(verification
of
a
Stroop
effect
with
the
VR-Stroop)
will
be
analyzed
with
a
series
of
paired
t-tests
with
the
˛
level
set
at
0.001
because
the
number
of
comparisons
(10)
will
exceed
the
number
of
participants
divided
by
10
(approximately
4
comparisons
are
allowed
with
a
minimum
of
10
participants
per
comparison).
The
second
wave
of
results
(preliminary
validation
the
VR-
Stroop)
will
be
analyzed
with
correlations
between
fundamental
variables
of
the
different
inhibition
measures
(see
Parsons
and
Rizzo,
2008
for
a
similar
approach
for
initial
validation
of
a
VR
task).
As
an
attempt
to
keep
the
number
of
correlations
at
the
low-
est
possible
number,
the
following
variables
were
chosen
on
the
basis
of
their
theoretical
importance:
the
mean
time
completion
of
the
D-KEFS
Color-Word
Interference
Task;
the
mean
number
of
omissions
and
commissions,
the
mean
reaction
time
and
the
rate
(%)
of
non
ADHD
diagnoses
of
the
CPT-II;
the
Stop-Signal
Reac-
tion
Time
(SSRT)
and
Stop-Signal
delay
(SSD)
of
the
Stop-it;
and
the
total
score
of
the
TEA
Elevator
counting
with
distractions
task.
From
the
VR-Stroop,
the
mean
number
of
correct
responses,
com-
missions,
omissions,
double
clicks,
and
the
mean
reaction
time
total
and
for
correct
responses,
and
variation
of
reaction
time
(total
and
for
correct
responses)
were
considered.
The
third
and
final
stage
of
analyses
will
assess
the
capacity
of
traditional
measures
to
predict
the
level
of
virtual
measures
with
multiple
regressions.
Because
complete
data
for
only
38
participants
were
available,
only
4
variables
(approximately
10
participants
per
variable)
will
be
selected
from
those
significantly
correlating
as
predictors
of
VR-Stroop
performances.
The
predic-
tive
values
of
these
4
variables
will
be
assessed
for
condition
1
and
condition
2
of
the
VR-Stroop
and
only
the
most
significant
model
for
each
condition
will
be
retained.
Author's personal copy
M.
Henry
et
al.
/
Journal
of
Neuroscience
Methods
210 (2012) 125–
131 129
2.4.
Ethical
considerations
The
present
study
was
approved
by
the
ethical
committee
of
the
university
and
each
volunteers
signed
a
consent
form
after
expla-
nation
of
the
purposes
and
procedures
were
given.
Participants
in
the
validation
study
(part
two)
received
$15
in
compensation.
3.
Results
3.1.
Pilot
results
Pilot
results
showed
that
an
ISI
of
2000
ms
was
associated
with
a
ceiling
effect.
The
task
was
too
easy
for
control
participants,
with
high
ratios
of
correct/incorrect
responses,
low
standard
devi-
ations,
and
no
differences
between
conditions
(mean
numbers
of
correct
responses
in
condition
1:
71.6
±
0.7
vs.
69.9
±
6.6
in
con-
dition
2;
p
>
0.1;
commission
errors
during
condition
1:
2.21
±
3.2
vs.
2.24
±
2.7
during
condition
2;
p
>
0.1).
Thus,
the
ISI
was
set
at
1000
ms.
3.2.
Primary
results
With
an
ISI
of
1000
ms,
error
rates
were
higher
(15%
or
more),
and
differences
between
conditions
were
statistically
significant
for
the
mean
numbers
of
correct
responses
(67.1
±
4.9
vs.
62.3
±
7.8;
t
(1,
37)
=
4.79,
p
<
0.0001,
respectively),
the
mean
reaction
times
for
correct
responses
(0.5833
±
0.0488
s
vs.
0.6407
±
0.0659
s;
t
(1,
37)
=
6.38,
p
<
0.0001,
respectively),
the
mean
variations
of
reac-
tion
time
for
correct
responses
(0.127
±
0.04
s
vs.
0.156
±
0.04
s,
t
(1,
37)
=
4.28,
p
<
0.0001,
respectively),
the
mean
numbers
of
omissions
(3.5
±
3.9
vs.
8.3
±
6.8,
t
(1,
37)
=
5.30,
p
<
0.0001,
respectively),
and
the
mean
variations
for
total
reaction
time
(0.127
±
0.06
s
vs.
0.191
±
0.08
s,
t
(1,
37)
=
4.14,
p
<
0.0001,
respec-
tively).
The
mean
numbers
of
commissions
also
differed
between
condition
1
(6.9
±
4.5)
and
condition
2
(7.5
±
5.3),
although
stan-
dard
deviations
were
high
and
the
magnitude
of
difference
was
low
and
not
significant
(t
(1,
37)
=
1.09,
p
>
0.2).
During
condi-
tion
2,
the
mean
reaction
times
(0.599
±
0.054
s
vs.
0.691
±
0.099
s,
t
(1,
37)
=
7.22,
p
<
0.0001)
and
the
mean
variations
of
reaction
(0.126
±
0.037
s
vs.
0.164
±
0.049
s,
t
(1,
37)
=
5.06,
p
<
0.0001)
significantly
differed
between
congruent
and
incongruent
trials
(respectively),
suggesting
an
internal
interference
effect.
3.2.1.
Validation
correlations
Significant
correlations
between
variables
of
the
tasks
appear
in
Table
1
(condition
1)
and
Table
2
(condition
2).
As
illustrated
in
the
Tables,
scores
in
condition
1
(color
blocks)
of
the
VR-Stroop
were
mainly
associated
with
variables
of
the
Elevator
counting
with
dis-
tractions
and
the
mean
reaction
time
of
the
CPT-II
(Table
1),
while
scores
in
condition
2
(color
words)
were
associated
with
variables
of
the
Elevator
counting
with
distractions,
the
conventional
Stroop
task
and
the
Stop-it
task
(Table
2).
Certain
variables
of
condition
2
were
also
associated
(negatively)
with
variables
of
the
CPT-II,
including
the
rate
of
non-ADHD
diagnoses
(Table
2).
3.2.2.
Regression
models
Based
on
the
correlation
matrices,
the
following
4
variables
were
retained
as
possible
predictors
of
VR
results:
(1)
the
Elevator
counting
with
distractions
mean
score;
(2)
the
CPT-II
mean
reac-
tion
time;
(3)
the
SSRT
from
the
Stop-it
and
(4)
the
mean
time
completion
of
the
D-KEFS
Color-Word
Interference
Task.
In
con-
dition
1
(color
blocks),
these
4
variables
explained
a
significant
portion
of
the
variance
of
the
reaction
times
for
correct
responses
in
the
VR-Stroop
(R2=
27.6%;
p
=
0.03),
although
the
solution
was
better
for
the
commission
errors
(R2=
37.9%;
p
=
0.003).
The
Ele-
vator
task
score
was
significantly
involved
as
a
single
variable
(negatively;
ˇ
=
0.443;
p
=
0.011),
while
a
trend
for
a
positive
asso-
ciation
was
observed
with
the
D-KEFS
Stroop
time
completion
(ˇ
=
0.302;
p
=
0.07).
In
condition
2,
the
same
4
variables
explained
a
highly
significant
proportion
(approximately
half)
of
the
variance
for
the
RT
variation
on
correct
responses
(R2=
51.4%;
p
=
0.0001).
The
Elevator
task
mean
score
was
also
a
significant
inverse
contrib-
utor
(ˇ
=
0.414;
p
=
0.007),
with
the
SSRT
approaching
significance
(ˇ
=
0.253;
p
=
0.06).
3.2.3.
Cybersickness
symptoms
and
sense
of
realism
Overall,
participants
reported
very
few
post-VR
symptoms
related
with
the
task
(mean:
6.8
±
6.5,
range
0–31,
below
60th
per-
centile
according
to
the
norms;
Kennedy
et
al.,
1993)
with
eye
strain
being
the
most
frequent
complaint
(63%
of
the
sample).
The
sense
of
presence
was
good
(mean:
32.9
±
7.7
range
12–49),
slightly
above
the
norm
mean
(29.45
±
12.0).
4.
Discussion
The
main
goal
of
this
study
was
to
develop
a
single
VR
measure
of
inhibition
capable
of
assessing
three
different
abilities:
selective
attention,
control
of
cognitive
interference,
and
motor
inhibition.
Although
this
preliminary
study
represents
only
the
initial
valida-
tion
phase
of
the
measure,
interesting
results
emerged.
First,
results
confirmed
that
the
Stroop
effect
might
be
efficiently
elicited
in
a
VR
environment.
This
conclusion
is
similar
to
that
of
Parsons
et
al.
(2011),
who
used
the
Stroop
effect
in
a
war
environment
to
show
the
usefulness
of
VR
to
assess
cognitive
and
external
interference.
The
present
study
further
suggests
that
bimodal
presentation
of
the
stimuli
also
elicits
the
Stroop
effect.
That
type
of
stimulus
pre-
sentation
(visual
and
auditory)
lets
the
participant
to
continuously
use
the
same
unique
response
key,
which
in
turn
allows
the
exam-
iner
to
assess
motor
impulsivity
(simple
reaction
times,
omissions,
and
commissions).
Thus,
the
bimodal
VR-Stroop
seems
capable
of
measuring
internal
interference
control
and
motor
inhibition
simultaneously.
Initial
validation
of
the
task
also
suggests
that
bimodal
VR-
Stroop
scores
are
associated
significantly
with
important
measures
of
impulsivity.
More
specifically,
such
fundamental
variables
as
the
Elevator
counting
mean
score,
the
CPT-II
mean
reaction
time,
the
SSRT,
and
D-KEFS
interference
mean
time
were
all
signifi-
cant
predictors
of
VR-Stroop
scores.
These
results
suggest
that
the
VR-Stroop
would
be
sensitive
to
one
or
more
of
these
types
of
impulsivity.
Control
of
external
interference
and
selective
attention
might
also
be
assessed
via
head
movements
with
the
Head
Mounted
Dis-
play
during
the
task.
Although
these
variables
were
not
analyzed
for
the
present
study,
head
movements
provoked
by
environmental
distracters
in
a
virtual
classroom
were
found
to
be
the
best
detec-
tor
of
attentional
deficits
among
children
with
mild
brain
injuries
(Nolin
et
al.,
2009,
2012).
Thus,
the
VR-Stroop
might
be
useful
for
clinical
purposes.
On
one
hand,
the
task
has
the
potential
to
detect
a
wider
array
of
inhibition
deficits
than
traditional
neuropsychological
measures
and,
on
the
other
hand,
it
has
the
potential
to
discriminate
between
subgroups
of
clinical
populations.
These
possibilities
should
be
tested
with
different
clinical
and
nonclinical
groups
of
participants.
For
instance,
the
significant
and
negative
association
between
the
number
of
correct
responses
at
the
VR-Stroop
and
the
rate
of
CPT
ADHD
diagnoses
is
intriguing.
Variability
in
reaction
times
at
the
VR-Stroop
was
also
associated
with
the
CPT
diagnosis
of
ADHD.
Reaction
time
variability
is
linked
with
specific
subtypes
of
ADHD
(Carlson
and
Mann,
2002),
and
the
capacity
of
the
VR-Stroop
to
discriminate
between
controls
and
subtypes
of
ADHD
persons
should
be
tested
in
future
investigations.
Author's personal copy
130 M.
Henry
et
al.
/
Journal
of
Neuroscience
Methods
210 (2012) 125–
131
Table
1
Correlations
between
traditional
measures
of
impulsivity
and
the
condition
1
of
the
virtual
reality
Stroop
(bold
=
p
<
0.05
uncorrected).
VR
Stroop
StroopaCPT-ObCPT-CcCPT
RTd%ADHDeSSDfSSRTgElevatorh
Correct
responses
r
=
0.085
p
=
0.611
r
=
0.036
p
=
0.828
r
=
0.158
p
=
0.343
r
=
0.381
p
=
0.018
r
=
0.099
p
=
0.555
r
=
0.172
p
=
0.301
r
=
0.202
p
=
0.224
r
=
0.385
p
=
0.017
RT
on
correct
responses
r
=
0.202
p
=
0.223
r
=
0.186
p
=
0.263
r
=
0.175
p
=
0.293
r
=
0.420
p
=
0.009
r
=
0.319
p
=
0.051
r
=
0.169
p
=
0.311
r
=
0.337
p
=
0.039
r
=
0.195
p
=
0.241
RT
var
on
correct
responses
r
=
0.131
p
=
0.434
r
=
0.007
p
=
0.969
r
=
0.137
p
=
0.411
r
=
0.379
p
=
0.019
r
=
0.053
p
=
0.750
r
=
0.295
p
=
0.072
r
=
0.320
p
=
0.050
r
=
0.394
p
=
0.014
Commissions
r
=
0.459
p
=
0.004
r
=
0.076
p
=
0.648
r
=
0.010
p
=
0.953
r
=
0.072
p
=
0.669
r
=
0.098
p
=
0.560
r
=
0.069
p
=
0.680
r
=
0.023
p
=
0.889
r
=
0.546
p
=
0.0001
Omissions r
=
0.086
p
=
0.609
r
=
0.003
p
=
0.986
r
=
0.078
p
=
0.643
r
=
0.305
p
=
0.063
r
=
0.084
p
=
0.617
r
=
0.178
p
=
0.286
r
=
0.202
p
=
0.224
r
=
0.361
p
=
0.026
Double
clicks
r
=
0.184
p
=
0.270
r
=
0.105
p
=
0.530
r
=
0.103
p
=
0.540
r
=
0.374
p
=
0.021
r
=
0.043
p
=
0.800
r
=
0.087
p
=
0.603
r
=
0.279
p
=
0.089
r
=
0.352
p
=
0.030
RT
VAR
Total
r
=
0.127
p
=
0.448
r
=
0.027
p
=
0.872
r
=
0.255
p
=
0.123
r
=
0.327
p
=
0.045
r
=
0.051
p
=
0.762
r
=
0.213
p
=
0.199
r
=
0.197
p
=
0.236
r
=
0.308
p
=
0.060
RT
Total r
=
0.105
p
=
0.531
r
=
0.151
p
=
0.367
r
=
0.213
p
=
0.200
r
=
0.128
p
=
0.444
r
=
0.049
p
=
0.772
r
=
0.267
p
=
0.105
r
=
0.051
p
=
0.760
r
=
0.261
p
=
0.113
aD-KEFS
Color-Word
Interference
Task;
mean
time
completion.
bCPT-II;
number
of
omissions.
cCPT-II;
number
of
commissions.
dCPT-II;
mean
reaction
time.
eCPT-II;
rate
of
non
ADHD
diagnoses.
fStop-it;
Stop-Signal
delay.
gStop-it;
Stop-Signal
Reaction
Time.
hTEA
Elevator
task;
mean
number
of
correct
responses.
Initial
validation
condition
2.
Table
2
Correlations
between
traditional
measures
of
impulsivity
and
the
condition
2
of
the
virtual
reality
Stroop
(bold
=
p
<
0.05
uncorrected).
VR
Stroop
StroopaCPT-ObCPT-CcCPT
RTd%ADHDeSSDfSSRTgElevatorh
Correct
responses r
=
0.455
p
=
0.004
r
=
0.101
p
=
0.545
r
=
0.037
p
=
0.828
r
=
0.222
p
=
0.180
r
=
0.103
p
=
0.539
r
=
0.053
p
=
0.751
r
=
0.228
p
=
0.169
r
=
0.454
p
=
0.004
RT
on
correct
responses
r
=
0.377
p
=
0.020
r
=
0.050
p
=
0.765
r
=
0.066
p
=
0.692
r
=
0.455
p
=
0.004
r
=
0.453
p
=
0.004
r
=
0.014
p
=
0.932
r
=
0.497
p
=
0.001
r
=
0.228
p
=
0.168
RT
VAR
on
correct
responses
r
=
0.498
p
=
0.001
r
=
0.013
p
=
0.937
r
=
0.109
p
=
0.513
r
=
0.246
p
=
0.136
r
=
0.345
p
=
0.034
r
=
0.283
p
=
0.085
r
=
0.455
p
=
0.004
r
=
0.617
p
=
0.0001
Commissions
r
=
0.475
p
=
0.003
r
=
0.006
p
=
0.973
r
=
0.046
p
=
0.786
r
=
0.165
p
=
0.324
r
=
0.082
p
=
0.626
r
=
0.168
p
=
0.313
r
=
0.180
p
=
0.279
r
=
0.575
p
=
0.0001
Omissions
r
=
0.474
p
=
0.003
r
=
0.112
p
=
0.503
r
=
0.005
p
=
0.975
r
=
0.184
p
=
0.269
r
=
0.087
p
=
0.602
r
=
0.097
p
=
0.562
r
=
0.263
p
=
0.111
r
=
0.488
p
=
0.002
V-R
Double
clicks r
=
0.180
p
=
0.279
r
=
0.053
p
=
0.750
r
=
0.206
p
=
0.215
r
=
0.107
p
=
0.521
r
=
0.183
p
=
0.271
r
=
0.242
p
=
0.143
r
=
0.120
p
=
0.475
r
=
0.238
p
=
0.150
RT
VAR
Total
r
=
0.542
p
=
0.0001
r
=
0.000
p
=
0.998
r
=
0.136
p
=
0.417
r
=
0.226
p
=
0.172
r
=
0.320
p
=
0.050
r
=
0.070
p
=
0.676
r
=
0.438
p
=
0.006
r
=
0.383
p
=
0.018
RT
Total
r
=
0.256
p
=
0.120
r
=
0.046
p
=
0.782
r
=
0.221
p
=
0.183
r
=
0.242
p
=
0.144
r
=
0.286
p
=
0.082
r
=
0.274
p
=
0.097
r
=
0.311
p
=
0.057
r
=
0.192
p
=
0.249
aD-KEFS
Color-Word
Interference
Task;
mean
time
completion.
bCPT-II;
number
of
omissions.
cCPT-II;
number
of
commissions.
dCPT-II;
mean
reaction
time.
eCPT-II;
rate
of
non
ADHD
diagnoses.
fStop-it;
Stop-Signal
delay.
gStop-it;
Stop-Signal
Reaction
Time.
hTEA
Elevator
task;
mean
number
of
correct
responses.
Overall,
the
bimodal
VR-Stroop
seems
to
represent
a
single,
short
(10
min),
enjoyable,
portable
(with
a
laptop
computer),
and
multi-component
assessment
of
inhibition.
Further
investigations
with
measures
of
other
type
of
cognitive
functions
(reasoning,
deducing,
planning,
etc.)
will
help
demonstrating
the
discriminant
validity
of
the
task.
Concomitant
assessments
with
psychophysi-
ological
measures
(electrodermal
conductance,
eye-tracking,
etc.)
will
also
help
confirming
the
validity
of
the
construct.
Currently,
preliminary
data
suggest
that
the
bimodal
VR-Stroop
could
assess
different
aspects
of
inhibition,
including
selective
attention,
con-
trol
of
internal
interference,
and
motor
inhibition
either,
either
separately
or
as
a
global
index
of
inhibition.
Conflict
of
interest
statement
None.
Acknowledgments
The
authors
wish
to
thank
Roman
Mitura,
co-founder
of
Digi-
tal
Media
Work
(Kenata,
ON,
Canada)
for
his
collaboration
for
the
development
of
the
task.
Results
of
this
study
were
presented
in
part
at
the
39th
and
40th
meetings
of
the
International
Neuropsy-
chological
Society
(INS),
Boston
(MA,
USA,
2011)
and
Montreal
(QC,
Canada,
2012)
respectively.
Author's personal copy
M.
Henry
et
al.
/
Journal
of
Neuroscience
Methods
210 (2012) 125–
131 131
References
Adams
R,
Finn
P,
Moes
E,
Flannery
K,
Rizzo
AA.
Distractibility
in
atten-
tion/deficit/hyperactivity
disorder
(ADHD):
the
virtual
reality
classroom.
Child
Neuropsychol
2009;15:120–35.
American
Psychiatric
Association
– APA,
editor.
Diagnostic
and
statistical
manual
of
mental
disorders.
4th
ed.
Washington,
DC:
American
Psychiatric
Association;
2000.
Barkley
RA.
Behavioral
inhibition,
sustained
attention
and
executive
functions:
con-
structing
a
unifying
theory
of
ADHD.
Psychol
Bull
1997;121:65–94.
Barratt
ES.
Factor
analysis
of
some
psychometric
measures
of
impulsiveness
and
anxiety.
Psychol
Rep
1965;16:547–54.
Bechara
A,
van
der
Linden
M.
Decision-making
and
impulse
control
after
frontal
lobe
injuries.
Curr
Opin
Neurol
2005;18:734–9.
Bouchard
S,
Robillard
G,
Renaud
P.
Revising
the
factor
structure
of
the
simulator
sickness
questionnaire.
Annu
Rev
Cyberther
Telemed
2007;5.
Buss
AH,
Plomin
R.
Impulsivity.
In:
A
temperament
theory
of
personality
develop-
ment.
Oxford:
Wiley-Interscience;
1975.
p.
122–147.
Carlson
CL,
Mann
M.
Sluggish
cognitive
tempo
predicts
a
different
pattern
of
impair-
ment
in
the
attention
deficit
hyperactivity
disorder,
predominately
inattentive
type.
J
Clin
Child
Adolesc
Psychol
2002;31:123–9.
Conners
CK,
Epstein
JL,
Angold
A,
Klaric
J.
Continuous
Performance
Test
in
a
norma-
tive
epidemiological
sample.
J
Abnorm
Child
Psychol
2003;31:555–62.
Delis
DC,
Kaplan
E,
Kramer
JH.
Delis–Kaplan
executive
function
system.
San
Antonio,
TX: