ArticlePDF Available
Assessing
engagement
while
viewing
video
vignettes;
validation
of
the
Video
Engagement
Scale
(VES)
Leonie
N.C.
Visser
a,1,
*,
Marij
A.
Hillen
a,1
,
Mathilde
G.E.
Verdam
a,b
,
Nadine
Bol
c
,
Hanneke
C.J.M.
de
Haes
a
,
Ellen
M.A.
Smets
a
a
Department
of
Medical
PsychologyAcademic
Medical
Center,
University
of
Amsterdam,
Amsterdam,
The
Netherlands
b
Research
Institute
of
Child
Development
and
Education,
University
of
Amsterdam,
Amsterdam,
The
Netherlands
c
Amsterdam
School
of
Communication
Research/ASCoR,
University
of
Amsterdam,
Amsterdam,
The
Netherlands
A
R
T
I
C
L
E
I
N
F
O
Article
history:
Received
11
June
2015
Received
in
revised
form
13
August
2015
Accepted
22
August
2015
Keywords:
Patient-provider
communication
Video
vignettes
Analogue
patients
Engagement
Validation
A
B
S
T
R
A
C
T
Objectives:
In
health
communication
research
using
video
vignettes,
it
is
important
to
assess
viewers
engagement.
Engagement
scores
can
indicate
ecological
validity
of
the
design,
and
help
distinguish
between
different
engagement
types.
Therefore,
we
aimed
to
develop
and
validate
a
scale
assessing
viewers
engagement
with
video
vignettes.
Methods:
Based
on
an
existing
question
set,
the
15-item,
ve-dimensional
Video
Engagement
Scale
(VES)
was
developed.
The
VES
was
validated
in
two
video-vignettes
studies
to
investigate
patient-physician
communication.
In
addition
to
engagement,
we
assessed
its
presumed
correlates,
e.g.,
perceived
realism
of
the
video
and
identication
with
the
patient.
Results:
Internal
consistency
and
testretest
reliability
were
adequate
in
both
studies
(N
=
181
and
N
=
228).
Positive
correlations
between
the
VES
and
perceived
realism
of
the
video,
credibility
of
and
identication
with
the
patient
suggested
good
content
validity.
Conrmatory
factor
analysis
suggested
a
four-dimensional
model
t,
largely
resembling
our
hypothesized
model.
Conclusions:
The
VES
reliably
and
validly
measures
viewers
engagement
in
health
communication
research
using
video
vignettes.
It
can
be
employed
to
assess
ecological
validity
of
this
design.
Further
testing
of
the
scale
is
needed
to
more
solidly
establish
its
dimensionality.
Practice
Implications:
We
recommend
that
researchers
use
the
VES,
to
ensure
ecological
validity
of
future
video-vignettes
studies.
ã
2015
Elsevier
Ireland
Ltd.
All
rights
reserved.
1.
Introduction
Researchers
in
health
communication
increasingly
use
approx-
imations
of
reality
as
a
substitute
for
real-life
situations,
in
the
form
of
written
scenarios,
virtual
reality
or
role-playing
(see,
for
example,
[13]).
Video-taped
scenarios
are
a
specic
type
of
approximations
displaying
either
recordings
of
actual
situations
[4]
or
scripted
and
role-played
simulations
[5].
Scripted
videos
are
increasingly
used
in
studies
using
experimental
video-vignettes
designs
to
study
patient-provider
communication
[6].
Such
designs
allow
systematic
testing
of
the
effects
of
specic
communication,
by
creating
multiple
variations
of
the
video.
Moreover,
video
vignettes
allow
researchers
to
assess
viewers
experience
of
the
events
in
the
video
without
exposing
them
to
the
actual
situation
thus
providing
an
ethical
alternative
to
investi-
gating
effects
of
patient-provider
communication
in
medical
situations
[6].
Participants
in
video-vignettes
studies
focused
on
medical
communication
are
referred
to
as
analogue
patients
(APs).
APs
can
either
be
healthy
individuals
or
(former)
patients.
They
are
instructed
to
imagine
themselves
to
be
in
the
video-patients
situation
[7].
During
or
after
viewing
the
vignette,
APs'
perception
of
(aspects
of)
communication,
self-reported
emotional
distress,
information
recall,
physiological
arousal,
or
other
outcomes
can
be
assessed.
Thus,
APs
are
used
as
proxies
for
clinical
patients
in
real
medical
consultations.
Video-vignettes
designs
using
APs
were
found
to
be
valid
for
investigating
medical
communication
from
a
patients
perspective
[4,7].
To
ensure
ecological
validity
and
thereby
external
validity
of
the
video-vignettes
design,
it
is
important
that
APs
are
engaged
with
the
video
vignette
and
video
*
Corresponding
author
at:
Academic
Medical
Center,
University
of
Amsterdam,
Department
of
Medical
Psychology.
P.O.
Box
22700,
110 0
DE
Amsterdam,
The
Netherlands.
Fax:
+31
20
566
9104.
E-mail
address:
N.C.Visser@amc.uva.nl
(L.N.C.
Visser).
1
These
authors
have
contributed
equally
to
this
work.
http://dx.doi.org/10.1016/j.pec.2015.08.029
0738-3991/ã
2015
Elsevier
Ireland
Ltd.
All
rights
reserved.
Patient
Education
and
Counseling
xxx
(2015)
xxxxxx
G
Model
PEC
5151
No.
of
Pages
9
Please
cite
this
article
in
press
as:
L.N.C.
Visser,
et
al.,
Assessing
engagement
while
viewing
video
vignettes;
validation
of
the
Video
Engagement
Scale
(VES),
Patient
Educ
Couns
(2015),
http://dx.doi.org/10.1016/j.pec.2015.08.029
Contents
lists
available
at
ScienceDirect
Patient
Education
and
Counseling
journal
homepage:
www.else
vie
r.com/locate
/pateducou
patient.
To
assess
such
engagement,
we
need
more
understanding
of
this
construct.
The
construct
of
engagement
is
used
in
the
eld
of
persuasive
communication
to
assess
to
what
extent
a
reader
becomes
immersed
in
a
narrative.
Narrative
engagement
is
dened
as
someones
experience
of
a
narrative
[8]
and
is
known
to
inuence
the
persuasive
effect
(e.g.,
on
a
readers
beliefs
and
attitudes)
of
a
narrative
[9].
Narrative
engagement
is
a
multi-dimensional
construct
[8,10].
Although
the
precise
dimensional
structure
varies
across
approaches,
it
embodies
concepts
such
as
transportation
(the
extent
to
which
one
becomes
absorbed
into
the
story
and
mentally
enters
the
narrative
world
[11]),
identication
(the
extent
to
which
one
imagines
him-
or
herself
to
be
one
of
the
characters
in
the
text
[12]),
and
telepresence
(the
extent
to
which
someone
has
a
sense
of
being
present
in
a
virtual
environment
[13]).
For
studies
using
a
video-vignette
design
with
APs,
this
multi-dimensionality
of
engagement
is
particularly
relevant.
For
example,
whereas
some
studies
may
strive
to
have
the
APs
imagine
how
they
would
feel
if
they
would
be
in
the
situation
of
the
video
patient
(projection),
others
will
aim
to
get
APs
into
the
same
emotional
state
as
the
video-patient
(empathy).
APs
ability
to
project
oneself
and
their
ability
to
empathize
might
vary
per
AP
and
differentially
mediate
the
effect
on
outcomes
such
as
APs
level
and
type
of
emotional
distress
evoked
by
the
video
vignette.
Two
groups
of
researchers
have
previously
formulated
sets
of
questions
to
measure
narrative
engagement.
They
based
their
questions
mainly
on
the
concepts
mentioned
above,
such
as
transportation,
resulting
in
partial
overlap
with
regard
to
item
content
and
dimensions.
Yet,
both
sets
of
questions
are
unsuitable
for
measuring
APs
engagement
with
video
vignettes.
First,
Buselle
and
Bilandzic
[8]
developed
and
validated
a
four-dimensional
scale
to
measure
narrative
engagement.
Although
this
scale
is
aimed
at
measuring
lm
and
television
program
viewers
narrative
expe-
riences,
it
is
not
apt
for
measuring
APs
engagement
with
the
video
vignette
and
video
patient,
because
of
its
mere
focus
on
narrative
processing
and
understanding.
This
focus
is
less
relevant
for
APs
viewing
video
vignettes,
because
the
medical
encounters
depicted
in
the
vignettes
are
usually
short
with
an
easily
recognizable
thread
and
predened
roles
of
the
characters
(i.e.,
a
patient
and
a
health
care
provider).
Second,
De
Graaf
and
colleagues
[9,10]
constructed
a
set
of
questions
around
a
hypothesized
seven-
dimension
structure.
This
set
of
questions
has
more
relevance
with
regard
to
measuring
APs
engagement,
because
it
has
a
broader
scope,
encompassing
dimensions
such
as
empathy
with
the
main
character
and
adopting
the
main
characters
identity.
However,
it
aims
at
assessing
readers
engagement
with
written
narratives
and
therefore
it
needs
adaptation,
whereas
not
all
dimensions
and
questions
important
for
written
narratives
are
relevant
when
viewing
videos
[14].
For
example,
the
Imagery
dimension,
i.e.,
visually
imagining
the
events
described
(as
if
one
were
viewing
a
video),
is
irrelevant
for
videos,
as
the
events
are
already
visualized
in
the
video.
Based
on
the
above,
it
seems
important
to
develop
a
multi-
dimensional
scale
to
assess
APs
engagement
with
video
vignettes
to
allow
examining
the
ecological
validity
of
a
video-vignettes
design,
and
to
enable
distinguishing
between
different
dimensions
of
video-vignette
engagement.
Therefore,
we
aimed
to
rst
adapt
the
existing
set
of
questions
from
De
Graaf
et
al.
[9,10]
such
that
it
can
be
used
for
analogue
patients
viewing
video
vignettes.
The
resulting
questionnaire
should
include
all
relevant
dimensions
of
engagement,
yet
not
include
an
abundance
of
items.
We
sought
to
develop
a
valid
and
reliable
questionnaire,
usable
in
future
video-
vignettes
studies
using
APs.
Our
second
aim
was
to
examine
the
reliability
and
content
validity
of
the
questionnaire
when
used
in
video-vignettes
designs.
Finally,
we
tested
the
hypothesized
dimensional
structure
of
the
resulting
questionnaire,
thereby
also
examining
construct
validity.
Table
1
Overview
of
descriptive
properties
of
all
items
of
the
VES,
including
reliability
results
of
items
and
subscales.
Item
Content
Dimension*
Study
1
Study
2
M
(SD)
Skew
Item-
scale
R
Test
retest
R
Subscale
reliability
(
a
)
M
(SD)
Skew
Item-
scale
R
Subscale
reliability
(
a
)
1
During
viewing
I
was
fully
concentrated
on
the
video
ATT
6.08
(1.01)
1.46
.50
.49
.66
6.42
(0.82)
1.45
.35
.54
3
When
I
was
viewing
the
video,
my
thoughts
were
only
with
the
video
ATT
5.32
(1.43)
0.85
.39
.62
5.85
(1.28)
1.31
.26
12
During
viewing,
I
was
hardly
aware
of
the
space
around
me
ATT
4,47
(1.68)
0.44
.52
.75
3.86
(1.92)
0.02
.75
2
During
viewing
it
was
as
if
I
was
present
at
the
events
depicted
in
the
video
GNW
4.57
(1.46)
0.33
.75
.80
.82
4.95
(1.75)
0.75
.65
.83
4
After
the
video
was
nished,
I
had
the
feeling
I
came
back
into
the
real
world
GNW
4.42
(1.50)
0.26
.50
.78
3.85
(1.96)
0.00
.70
8
When
I
was
viewing
the
video,
I
was
in
the
world
of
the
video
in
my
thoughts
GNW
4.68
(1.51)
0.61
.78
.74
4.06
(1.85)
0.17
.78
5
When
I
had
been
viewing
for
a
while,
it
seemed
as
if
I
had
become
the
[video
character]**
in
my
thoughts
IDE
3.85
(1.46)
0.05
.78
.71
.92
3.87
(1.90)
0.03
.79
.91
13
I
had
the
feeling
I
went
through
what
the
[video
character]**
went
through
IDE
3.57
(1.62)
0.22
.73
.69
3.80
(1.98)
0.07
.76
14
In
my
imagination,
it
was
as
if
I
were
the
[video
character]**
IDE
3.62
(1.59)
0.13
.78
.71
3.41
(1.93)
0.34
.79
6
I
empathized
with
the
[video
character]**
EMP
5.63
(1.19)
1.16
.73
.67
.85
5.29
(1.46)
0.96
.66
.82
9
During
viewing,
I
felt
[particular
emotion]
***
when
the
[video
character]**
felt
[particular
emotion]***
EMP
4.58
(1.58)
0.44
.73
.67
3.60
(1.87)
0.25
.76
11
I
felt
for
the
[video
character]**
EMP
5.49
(1.24)
0.95
.75
.76
5.02
(1.49)
0.76
.65
7
The
video
affected
me
EMO
5.24
(1.35)
0.79
.75
.78
.83
4.69
(1.70)
0.51
.76
.82
10
I
found
the
video
moving
EMO
5.01
(1.41)
0.69
.65
.76
3.77
(1.79)
0.11
.78
15
Because
of
the
video,
feelings
arose
in
me
EMO
4.93
(1.39)
0.69
.55
.55
4.48
(1,96)
0.40
.60
Notes.
*ATT:
attention,
GNW:
going
into
a
narrative
world,
IDE:
identication,
EMP:
empathy,
EMO:
emotions.
**To
be
specied
depending
on
the
video
content.
In
the
present
validation
studies,
the
video-character
was
specied
as
patient.
***To
be
specied
depending
on
the
video
content.
Study
1
included
the
emotion
sadness;
Study
2
tense.
2
L.N.C.
Visser
et
al.
/
Patient
Education
and
Counseling
xxx
(2015)
xxxxxx
G
Model
PEC
5151
No.
of
Pages
9
Please
cite
this
article
in
press
as:
L.N.C.
Visser,
et
al.,
Assessing
engagement
while
viewing
video
vignettes;
validation
of
the
Video
Engagement
Scale
(VES),
Patient
Educ
Couns
(2015),
http://dx.doi.org/10.1016/j.pec.2015.08.029
2.
Methods
2.1.
Questionnaire
construction
The
development
of
the
Video
Engagement
Scale
(VES)
was
based
on
the
existing
set
of
39
questions
by
De
Graaf
et
al.,
constructed
to
assess
engagement
when
reading
written
narra-
tives
[9,10].
The
question
set
was
constructed
around
a
hypothe-
sized
seven-factor
structure,
encompassing:
emotions,
emotional
reactions
evoked
by
the
story;
attentional
focus,
attention
focused
on
the
story;
imagery,
imagery
of
the
story;
going
into
a
narrative
world,
the
sensation
of
going
into
a
narrative
world;
empathy,
empathizing
with
the
main
character;
identity,
adopting
the
identity
of
the
main
character;
and
adopting
the
perspective
of
the
character,
experiencing
the
story
from
the
position
of
the
main
character.
Items
were
either
adopted
from
existing
scales
or
based
on
previous
denitions
of
(forms
of)
engagement
[13,1517].
The
pool
of
questions
was
preliminarily
validated
by
De
Graaf
et
al.
in
three
experiments
[9,10].
Exploratory
factor
analyses
showed
evidence
for
a
ve-factor
and
four-factor
structure
respectively,
in
the
different
studies.
Based
on
De
Graaf
et
al.s
validation
results
[9,10],
two
authors
(LNCV
and
MAH)
critically
reviewed
the
question
set,
to
make
it
suitable
for
video
material
and
concise.
Changes
were
made
with
respect
to
the
dimensional
structure,
wording
of
items,
and
number
of
questions.
As
regards
to
the
dimensional
structure,
the
Imagery
dimension
was
removed
as
it
was
deemed
irrelevant
for
videos.
Adopting
the
perspective
of
the
character
was
removed
as
a
separate
dimension,
as
in
De
Graafs
research,
it
strongly
overlapped
both
conceptually
and
statistically
with
Adopting
the
identity
of
the
character.
Changes
to
wording
included
replacing
the
words
story
and
reading
with
video
and
viewing,
respectively.
Finally,
to
shorten
the
scale,
only
the
items
with
the
highest
factor
loadings
and
the
least
statistical
overlap
in
De
Graafs
studies
were
selected.
These
changes
resulted
in
the
15-item
Video
Engagement
Scale
(VES).
Responses
are
made
on
a
7-point
Likert
scale
(1
=
complete-
ly
disagree
to
7
=
completely
agree).
The
scale
encompasses
ve
proposed
dimensions:
(1)
emotions
(EMO),
emotional
reactions
evoked
by
the
video;
(2)
empathy
(EMP),
empathy
with
the
video-
character;
(3)
identity
(IDE),
adopting
the
video-characters
identity;
(4)
attention
(ATT),
attentional
focus
on
the
video,
with
reduced
access
to
the
real
world;
and
(5)
going
into
a
narrative
world
(GNW),
the
sensation
of
going
and
being
in
the
narrative
world.
Three
items
were
included
for
each
dimension.
Items
were
placed
in
random
order.
All
items
and
the
dimensions
to
which
they
belong
are
displayed
in
Table
1.
To
pilot
test
the
VES,
six
students
participated
as
APs.
After
viewing
a
video
vignette
of
a
physician-patient
consultation,
they
completed
the
questionnaire
while
thinking
out
loud
[18].
Pilot
results
conrmed
comprehensibility
of
the
questions
and
concep-
tual
distinction
between
the
different
items.
The
wording
of
one
item
was
slightly
changed
to
facilitate
understanding
(i.e.,
When
I
watched
the
video,
I
was
in
the
world
of
the
video
in
my
imagination
was
changed
into
When
I
was
watching
the
video,
(
.
.
.
)).
2.2.
Reliability
and
content
validity
The
VES
was
psychometrically
validated
in
the
context
of
two
video-vignettes
studies
focused
on
physician-patient
communica-
tion
using
APs.
The
two
studies
used
different
sample
populations
and
different
video
vignettes,
which
increases
generalizability
of
the
results.
2.2.1.
Study
1
Study
1
was
originally
designed
to
test
two
methodological
issues
relevant
to
research
using
experimental
video
vignettes
in
communication
research,
i.e.,
choice
of
camera
perspective
and
type
of
introduction
to
the
video.
In
total,
six
variations
were
created
of
a
video
depicting
a
bad-news
consultation
between
an
oncological
surgeon
and
an
esophageal
cancer
patient.
The
VES
was
completed
after
video
viewing.
2.2.1.1.
Sample
&
procedure.
University
students
(age
1840
years),
recruited
through
a
psychology
and
a
communication
science
program,
participated
as
APs.
They
were
randomly
assigned
to
view
one
of
the
six
variants
of
the
video
vignette
in
a
laboratory
room.
In
either
variant
of
the
introduction,
they
were
instructed
to
imagine
themselves
being
in
the
video-patients
situation.
Directly
after
viewing,
their
engagement
with
the
video
was
assessed
using
the
VES.
To
assess
test-retest
reliability,
a
subset
of
participants
was
invited
to
complete
the
VES
a
second
time,
2448
h
after
viewing.
2.2.1.2.
Measures.
Demographics
assessed
were
participants
age,
gender,
education
level
and
ethnicity.
Engagement
was
assessed
using
the
15-item
VES,
described
above.
We
assessed
how
well
patients
could
identify
with
the
video-patient
using
ve
items
from
Cohens
identication
scale
[12].
Perceived
realism
of
the
video
was
assessed
using
three
items,
asking
participants
how
realistic,
credible,
and
how
likely
to
happen
in
real
life
they
thought
the
events
in
the
video
were.
Perceived
realism
of
the
consultation
was
measured
with
two
questions
asking
how
real,
and
how
similar
to
a
real
bad-news
consultation
the
video-vignette
consultation
was.
Finally,
credibility
of
the
video-patient
was
assessed
by
two
items
asking
about
the
believability
of
the
video-
patients
behavior
and
appearance.
All
items
assessing
perceived
realism,
credibility
and
identication
had
a
7-point
Likert
scale
(1
=
completely
disagree
to
7
=
completely
agree).
We
measured
APs
changes
in
state
anxiety
by
calculating
the
difference
in
scores
on
the
STAI-State
short
form,
assessed
before
and
after
viewing
the
video.
The
STAI-State
short
form
contains
six
items
(4-point
Likert
scale;
1
=
not
at
all,
to
4
=
a
lot)[19].
Changes
in
sadness,
feeling
tense
and
fearfulness,
measured
with
three
single
items
using
visual
analogue
scale
(0100),
were
assessed
by
calculating
the
difference
between
scores
before
and
after
viewing
the
video.
2.2.1.3.
Analyses.
Unless
otherwise
reported,
all
analyses
were
conducted
using
SPSS
Version
20
[20].
For
our
analyses,
all
data
were
collapsed,
i.e.,
no
distinction
was
made
with
regard
to
the
video
variant
viewed
by
APs.
Internal
consistency
was
analyzed
for
the
overall
scale
and
for
its
subscales,
using
Cronbachs
Alpha,
which
was
expected
to
be
acceptable
(a
>
.70)
[21].
Inter-item
and
item-scale
correlations
were
calculated.
Inter-item
correlations
between
.20
and
.70
[22]
and
item-scale
correlations
higher
than
.20
were
considered
acceptable
[23].
Test-retest
reliability
was
assessed
by
correlating
VES
test
and
retest
scores.
We
expected
high
correlations
(r
>
.70)
[24].
To
assess
content
validity,
we
correlated
identication
with
the
video-patient,
perceived
realism
of
the
vignette
and
the
consultation,
and
credibility
of
the
patient
with
VES
scores.
Identication,
as
a
central
concept
within
the
multi-dimensional
construct
of
engagement,
should
correlate
positively
with
VES
scores.
In
previous
research,
a
positive
correlation
was
found
between
transportation,
also
a
central
concept
within
engagement,
and
perceived
realism
of
a
narrative,
its
setting
and
its
characters
[25]
.
Therefore,
positive
correlations
were
also
expected
between
VES
scores
and
perceived
realism
and
credibility.
Moreover,
because
of
the
emotional
content
of
the
bad
news
video
vignette,
we
expected
stronger
engagement
to
lead
to
more
emotional
distress.
Therefore
we
tested
whether
higher
VES
L.N.C.
Visser
et
al.
/
Patient
Education
and
Counseling
xxx
(2015)
xxxxxx
3
G
Model
PEC
5151
No.
of
Pages
9
Please
cite
this
article
in
press
as:
L.N.C.
Visser,
et
al.,
Assessing
engagement
while
viewing
video
vignettes;
validation
of
the
Video
Engagement
Scale
(VES),
Patient
Educ
Couns
(2015),
http://dx.doi.org/10.1016/j.pec.2015.08.029
scores
were
correlated
with
an
increase
in
state
anxiety,
sadness,
feeling
tense
and
fearfulness
as
a
result
of
viewing
the
video.
2.2.2.
Study
2
Study
2
was
originally
designed
to
test
the
effect
of
oncologists
non-verbal
communication
on
patients
trust.
A
basic
script
was
created
of
a
consultation
between
a
medical
oncologist
and
a
female
breast
cancer
patient.
Next,
variations
in
the
oncologists
amount
of
eye
contact,
bodily
posture
and
smiling
were
created.
This
resulted
in
8
versions
of
the
video.
The
VES
was
completed
after
viewing
the
video.
2.2.2.1.
Sample
&
procedures.
Female
breast
cancer
patients
and
healthy
women
served
as
APs.
Patients
were
recruited
through
radiotherapy
out-patient
clinics
of
an
academic
and
a
regional
hospital,
and
through
advertisements
distributed
among
breast
cancer
patient
organizations.
Patients
could
self-apply
by
reply
card,
email
or
telephone.
Participation
was
from
home,
via
the
computer.
APs
rst
completed
a
questionnaire
assessing
their
background
characteristics.
Next,
they
viewed
a
randomly
selected
video
version
after
being
instructed
to
imagine
themselves
in
the
perspective
of
the
video-patient.
Afterwards,
their
engagement
in
the
video
was
assessed.
2.2.2.2.
Measures.
Demographic
and
medical
characteristics
comprised
age,
gender,
education
level
and
medical
background.
Engagement
was
measured
using
the
newly
created
VES.
Perceived
realism
was
tested
using
the
same
3
items
as
in
Study
1.
2.2.2.3.
Analyses.
As
in
Study
1,
data
for
all
different
video
versions
were
collapsed
for
the
current
analyses.
Internal
consistency
was
analyzed
for
the
overall
scale
and
for
its
subscales
(Cronbachs
Alpha).
Inter-item
and
item-scale
correlations
were
calculated.
Inter-item
correlations
between
.20
and
.70
and
item-scale
correlations
higher
than
.20
were
considered
acceptable.
To
assess
content
validity,
engagement
(VES
score)
was
correlated
with
perceived
realism
of
the
video.
2.3.
Dimensionality/construct
validity
Structural
Equation
Modeling
(SEM)
was
used
to
investigate
the
underlying
dimensional
structure
of
the
questionnaire,
using
Lavaan
software
[26].
First,
we
investigated
whether
the
data
supported
the
hypothesized
underlying
ve-dimensional
structure
of
the
questionnaire.
To
achieve
identication
of
all
model
parameters,
scales
and
origins
of
the
common
factors
were
established
by
xing
the
factor
variances
at
one.
Goodness-of-t
was
evaluated
with
the
x
2
test
of
exact
t
(CHISQ;
signicant
x
2
indicates
a
signicant
difference
between
data
and
model).
Root
mean
square
error
of
approximation
(RMSEA;
[27,28])
was
used
as
an
approximate
t
index
(values
>.10
indicate
poor
t,
<.08
reasonable
t
and
<.05
close
t)
[29].
Modications
to
the
hypothesized
underlying
structure
were
based
on
inspection
of
correlation
residuals
(>.10)
and
signicant
modication
indices
[30].
Modications
were
incorporated
if
they
lead
to
improvement
for
both
data
sets
(Study
1
and
Study
2),
to
nd
a
common
dimensional
structure
and
thereby
improve
generalizability
of
results.
Each
modication
was
consistently
guided
by
substantive
consideration
to
retain
a
theoretical
sensible
model.
Chi-square
difference
tests
(CHISQD)
were
used
to
evaluate
differences
between
hierarchically
related
models
(signicant
CHISQD
indicates
a
signicantly
better
t
of
the
more
parsimoni-
ous
model).
The
nal
model
provides
information
about
the
number
of
factors
required
to
explain
the
relationships
between
items,
the
strength
of
the
relationships
between
items
and
the
underlying
factors,
the
strength
of
the
relationship
between
underlying
factors,
and
the
reliability
of
items.
3.
Results
3.1.
Reliability
and
content
validity
3.1.1.
Study
1
3.1.1.1.
Sample
and
descriptives.
Data
of
181
participants
were
collected
(Table
2).
There
were
no
missing
data,
because
we
used
a
Table
2
Demographic,
health
and
relationship
characteristics
of
the
sample
in
Study
1
and
Study
2.
Study
1
(n
=
181)
Study
2
(n
=
228)
University
students
Breast
cancer
patients
(n
=
159)
Healthy
women
(n
=
69)
Median
(range)
SD
Median
(range)
SD
Median
(range)
SD
Age
23
(1840)
4
56
(3173)
11
51
(3173)
11
No.
months
since
diagnosis
42
47
N
%
N
%
N
%
Gender
Male
55
30
0
0
Female
126
70
159
100
Educational
level
None/primary
school
0
0
2
1
0
0
Secondary/lower
level
vocat.
school
53
29
89
56
32
46
College/university
128
71
68
43
37
54
Ethnicity
Dutch
170
94
144
91
63
91
Other
11
6
15
9
6
9
Treatment
status
(n
=
159)
In
active
treatment
62
39
Regular
check
ups
94
59
No
treatment/check
ups
3
2
4
L.N.C.
Visser
et
al.
/
Patient
Education
and
Counseling
xxx
(2015)
xxxxxx
G
Model
PEC
5151
No.
of
Pages
9
Please
cite
this
article
in
press
as:
L.N.C.
Visser,
et
al.,
Assessing
engagement
while
viewing
video
vignettes;
validation
of
the
Video
Engagement
Scale
(VES),
Patient
Educ
Couns
(2015),
http://dx.doi.org/10.1016/j.pec.2015.08.029
forced
response
function.
Of
the
58
APs
invited
to
ll
in
the
retest,
44
completed
the
questionnaire.
Because
for
two
participants
retest
data
could
not
be
linked
to
the
original
data,
retest
data
of
42
of
the
58
invited
participants
were
available
(response
rate
=
72%).
Mean
VES
score
was
4.77
(SD
=
1.02;
range
1.6 7
6.87;
skewness
0.25,
see
Table
1).
Mean
VES
score
at
retest
(n
=
42)
was
4.69
(SD
=
1.0 0;
skewness
0.05).
Mean
score
for
identication
with
the
patient
was
2.84
(SD
=
0.89).
Mean
score
for
realism
of
the
consultation
was
5.03
(SD
=
1.16)
and
for
realism
of
the
video
was
5.50
(SD
=
1.08).
Credibility
of
the
patient
was
on
average
4.98
(SD
=
1.41).
Mean
state
anxiety
increased
from
1.67
before
the
video
(SD
=
0.47)
to
2.09
after
the
video
(SD
=
0.57),
averaging
an
increase
of
0.42
(SD
=
0.59).
Mean
increases
for
other
emotions
were:
19.08
for
Sadness
(SD
=
24.57),
9.79
for
Feeling
tense
(SD
=
23.87),
and
14.71
for
Fearfulness
(SD
=
22.10).
3.1.1.2.
Reliability.
Internal
consistency
for
the
VES
was
high
overall
(a
=
.93),
as
well
as
for
four
out
of
ve
subscales:
emotions
(a
=
.83),
empathy
(a
=
.85),
identity
(a
=
.92),
and
going
into
a
narrative
world
(a
=
.82).
For
attention,
consistency
was
moderate
(a
=
.66),
due
to
item
12
(During
viewing,
I
was
hardly
aware
of
the
space
around
me).
If
deleted,
internal
consistency
would
increase
to
an
acceptable
level
(a
=
.75).
Item-scale
correlations
ranged
between
.39
and
.78
(see
Table
1).
Inter-item
correlations
showed
an
acceptable
pattern,
mostly
ranging
between
.25
and
.80,
with
a
few
exceptions
as
low
as
.12.
Test-retest
reliability
was
high
for
mean
VES
scores
(r
s
=
.91),
and
ranged
between
r
s
=
.77
and
.89
for
the
ve
subscales
(Table
1).
3.1.1.3.
Content
validity.
As
hypothesized,
we
found
high
correlations
between
mean
VES
scores
and
mean
scores
for:
perceived
realism
of
the
video
(r
s
(181)
=
.50,
p
<
.001),
perceived
realism
of
the
consultation
(r
s
(181)
=
.44,
p
<
001),
and
credibility
of
the
patient
(r
s
(181)
=
.36,
p
<
.001).
The
correlation
with
identication
with
the
patient
scores
(r
s
(181)
=
.29,
p
<
.001)
was
substantial.
Moreover,
higher
VES
scores
were
associated
with
increased
state
anxiety
as
a
result
of
viewing
the
video
(r
s
(181)
=
.27,
p
<
.001),
as
well
as
with
increased
sadness
(r
s
(181)
=
.15,
p
<
.05)
and
fearfulness
(r
s
(181)
=
.20,
p
<
.01).
VES
scores
were
not
correlated
with
an
increase
in
feeling
tense
(r
s
(181)
=
.06,
p
<
.46).
3.1.2.
Study
2
3.1.2.1.
Sample
and
descriptives.
Of
the
234
participants,
six
(3%)
had
technical
problems
with
viewing
the
video
and
were
therefore
excluded
from
the
analyses.
The
denitive
sample
included
228
women
(159
(70%)
breast
cancer
patients
and
69
(30%)
healthy
women).
Because
we
used
a
forced
response
function,
there
were
no
missing
values.
Sample
characteristics
are
displayed
in
Table
2.
Mean
VES
score
was
4.46
(SD
=
1.26;
range
1.7 3 7. 0 0 ;
skewness
0.05).
For
individual
VES
items,
means
varied
between
3.41
and
6.42
(see
Table
1).
Mean
perceived
realism
of
the
video
was
5.49
(SD
=
1.38).
3.1.2.2.
Reliability.
Internal
consistency
of
the
VES
was
high
for
the
scale
overall
(a
=
.94).
For
the
subscales,
internal
consistency
was
good:
emotions
(a
=
.82),
empathy
(a
=
.82),
identity
(a
=
.91),
going
into
a
narrative
world
(a
=
.83),
except
for
the
attention
subscale
(a
=
.54).
As
in
Study
1,
deletion
of
item
12
would
increase
consistency
(a
=
.68).
Item-scale
correlations
ranged
between
.26
and
.79
(Table
1).
Inter-item
correlations
mostly
ranged
between
.20
and
.70,
with
a
few
exceptions
as
low
as
.06
or
as
high
as
.89.
3.1.2.3.
Content
validity.
Mean
VES
score
correlated
signicantly
with
mean
scores
for
perceived
realism
of
the
video
(r
s
(228)
=
.29,
p
<
.001).
3.2.
Dimensionality/construct
validity
The
hypothesized
model
(see
Fig.
1)
showed
poor
model
t
on
both
CHISQ
and
RMSEA
in
both
studies
(Model
1;
Table
3).
To
improve
model
t,
a
cross
loading
was
added
of
item
5
on
GNW
(Study
1:
CHISQD
(1)
=
24.64,
p
<
.001;
Study
2:
CHISQD
(1)
=
63.83,
p
<
.001),
indicating
that
the
item
When
I
had
been
viewing
for
a
while,
it
seemed
as
if
I
had
become
the
[video
character]
in
my
thoughts
measured
not
only
identication,
but
also
going
into
a
narrative
world.
Second,
a
residual
covariance
was
added
between
items
1
and
3
(Study
1:
CHISQD
(1)
=
21.23,
p
<
.001;
Study
2:
CHISQD
(1)
=
70.07,
p
<
.001).
This
indicates
that
the
items
During
viewing
I
was
fully
concentrated
on
the
video
(item
1)
and
When
I
was
viewing
the
video,
my
thoughts
were
only
with
the
video(item
3)
have
something
more
in
common
than
what
is
being
measured
by
the
underlying
factor
attention.
Finally,
AT T
Item
1
Item
4
Item
8
Item
13
Item
14
Res
.
Item
1
Res
.
Item
3
Res
.
Item
12
Res
.
Item
2
Res
.
Item
4
Res
.
Item
13
Res
.
Item
14
Res
.
Item
8
Item
6
Item
9
Item
11
Res
.
Item
6
Res
.
Item
11
Res
.
Item
9
Item
3
Item
12
Item
2
Item
5
Res
.
Item
5
GNW IDE EMP EMO
Item
7
Item
10
Item
15
Res
.
Item
7
Res
.
Item
15
Res
.
Item
10
Fig.
1.
Hypothesized
ve-dimensional
structure.
Notes:
Circles
represent
unobserved,
latent
variables
(common
and
residual
factors)
and
the
squares
represent
the
observed
variables
(item
scores).
The
ve
latent
variables
at
the
top
are
the
common
factors
attention
(ATT)
measured
by
items
1,
3
and
12,
going
into
a
narrative
world
(GNW)
measured
by
items
2,
4
and
8,
identity
(IDE),
measured
by
items
5,
13
and
14,
empathy
(EMP)
measured
by
items
6,
9
and
11,
and
emotions
(EMO)
measured
by
items
7,
10
and
15.
Other
latent
variables
are
the
residual
factors
that
represent
all
that
is
specic
to
item
1,
item
2,
item
3,
etc.,
plus
random
error
variation.
L.N.C.
Visser
et
al.
/
Patient
Education
and
Counseling
xxx
(2015)
xxxxxx
5
G
Model
PEC
5151
No.
of
Pages
9
Please
cite
this
article
in
press
as:
L.N.C.
Visser,
et
al.,
Assessing
engagement
while
viewing
video
vignettes;
validation
of
the
Video
Engagement
Scale
(VES),
Patient
Educ
Couns
(2015),
http://dx.doi.org/10.1016/j.pec.2015.08.029
a
residual
covariance
was
added
between
items
6
and
11
(Study
1:
CHISQD
(1)
=
13.67,
p
<
.001;
Study
2:
CHISQD
(1)
=
45.09,
p
<
.001),
indicating
that
the
items
I
empathized
with
the
[video
character]
(item
6)
and
I
felt
for
the
[video
character]
(item
11)
have
something
more
in
common
than
what
is
being
measured
by
the
underlying
factor
empathy.
Although
the
overall
model
t
of
this
model
for
both
studies
was
still
not
completely
satisfactory,
it
no
longer
indicated
poor
model
t
according
to
the
RMSEA
values
(Model
4;
Table
3).
Because
no
further
substantive
modications
could
be
identied
to
improve
model
t
for
both
studies,
this
model
was
retained.
Subsequently,
inspection
of
parameter
estimates
indicated
that
the
common
factors
EMO
and
EMP
could
not
be
distinguished
(i.e.,
correlations
were
not
signicantly
different
from
1;
Study
1:
CHISQD
(1)
=
0.51,
p
=
.48;
Study
2:
CHISQD
(1)
=
0.57,
p
=
.45).
Therefore,
the
nal
model
consists
of
a
four-dimensional
structure,
including
one
cross
loading
and
two
residual
covariances
(see
Fig.
2).
For
both
studies,
the
overall
t
of
the
nal
model
(Model
5,
Table
3)
is
still
signicant
according
to
the
CHISQ,
but
no
longer
shows
poor
t
according
to
the
RMSEA
(Table
4).
4.
Discussion
and
conclusion
4.1.
Discussion
We
developed
and
validated
the
15-item,
multi-dimensional
Video
Engagement
Scale
(VES)
to
assess
APs
engagement
with
video
vignettes.
The
VES
enables
researchers
to
ensure
the
ecological
validity
of
a
video-vignettes
design,
and
to
distinguish
between
different
dimensions
of
video-vignette
engagement.
Thus
far,
a
specic
scale
to
assess
APs
engagement
with
video
vignettes
did
not
exist.
As
a
result,
researchers
had
limited
insight
in
APs
engagement,
because
it
was
either
not
assessed
at
all,
or
using
single,
unvalidated
items.
Results
from
two
studies
indicate
that
the
VES
assesses
engagement
reliably,
as
indicated
by
high
internal
consistency
and
testretest
reliability.
Although
reliability
of
most
subscales
was
high,
we
found
lower
internal
consistency
for
the
attention
dimension.
This
was
due
to
Item
12
in
both
studies.
Item
12
reads:
During
viewing,
I
was
hardly
aware
of
the
space
around
me.
Possibly,
it
focuses
more
on
the
space
surrounding
the
participant,
whereas
the
other
items
in
this
dimension
focus
on
the
video.
For
participants,
to
devote
their
full
attention
to
the
video
might
not
necessarily
imply
that
they
forget
their
surroundings.
In
other
settings,
e.g.,
with
more
prominent
and
large
screens,
this
may
be
the
case.
However,
in
laboratory
settings
such
as
in
the
present
studies,
this
presumably
should
not
be
expected.
Satisfactory
content
validity
was
indicated
by
substantial
correlation
between
the
VES
and
identication
with
the
video-
patient,
and
high
correlations
between
the
VES
and
perceived
realism
of
the
video
and
consultation,
and
credibility
of
the
video-
patient.
Moreover,
as
expected,
high
correlations
were
found
between
the
VES
and
anxiety,
sadness
and
fearfulness
provoked
by
viewing
a
distressing
patient-provider
interaction.
Dimensionality
testing
suggested
a
four-dimensional
model
t,
largely
resembling
our
hypothesized
ve-dimensional
model.
The
hypothesized
dimensions
emotion
and
empathy
were
combined
to
one
dimension.
Empathizing
with
the
video-patient
seems
to
go
jointly
with
experiencing
emotions,
most
likely
those
modeled
by
the
patient
in
the
video
[31].
Scores
on
this
combined
scale
were
higher
than
on
the
identication
dimension.
Possibly,
APs
more
easilyempathizewiththe video-patient
andexperiencethe modeled
Table
3
Goodness
of
overall
model
t
of
structural
equation
models
to
assess
construct
validity.
Study
1
Study
2
Model
Description
CHISQ
df
RMSEA
[90%
CI]
CHISQ
df
RMSEA
[90%
CI]
Model
1
Hypothesized
model
(ve
dimensional
structure)
232.34
80
.103
[.087;
.118]
424.02
80
.137
[.125;
.150]
Model
2
Additional
cross
loading
Item
5
207.70
79
.095
[.079;
.111]
360.19
79
.125
[.112;
.138]
Model
3
Additional
residual
covariance
Item
1Item
3
186.47
78
.088
[.072;
.104 ]
290.12
78
.109
[.096;
.123]
Model
4
Additional
residual
covariance
Item
6Item
11
172.80
77
.083
[.066;
.099]
245.03
77
.098
[.084;
.112]
Model
5
Merged
factors
empathy
and
emotions
188.80
81
.086
[.070;
.102]
253.93
81
.097
[.083;
.110]
AT T
Item
1
Item
4
Item
8
Item
13
Item
14
Res
.
Item
1
Res
.
Item
3
Res
.
Item
12
Res
.
Item
2
Res
.
Item
4
Res
.
Item
13
Res
.
Item
14
Res
.
Item
8
Item
6
Item
9
Item
11
Res
.
Item
6
Res
.
Item
11
Res
.
Item
9
Item
3
Item
12
Item
2
Item
5
Res
.
Item
5
GNW IDE
EMP
EMO
Item
7
Item
10
Item
15
Res
.
Item
7
Res
.
Item
15
Res
.
Item
10
Fig.
2.
Four-dimensional
structure
of
the
nal
model.
Notes:
Circles
represent
unobserved,
latent
variables
(common
and
residual
factors)
and
the
squares
represent
the
observed
variables
(item
scores).
The
four
latent
variables
at
the
top
are
the
common
factors
attention
(ATT)
measured
by
items
1,
3
and
12,
going
into
a
narrative
world
(GNW)
measured
by
items
2,
4,
8
and
5,
identity
(IDE),
measured
by
items
5,
13
and
14,
and
empathy
and
emotions
(EMPEMO)
measured
by
items
6,
9,
11,
7,
10
and
15.
Other
latent
variables
are
the
residual
factors
that
represent
all
that
is
specic
to
item
1,
item
2,
item
3,
etc.,
plus
random
error
variation.
Dotted
lines
represent
the
modications
that
were
made
to
the
measurement
model,
and
include
two
residual
covariances
and
one
cross
loading.
6
L.N.C.
Visser
et
al.
/
Patient
Education
and
Counseling
xxx
(2015)
xxxxxx
G
Model
PEC
5151
No.
of
Pages
9
Please
cite
this
article
in
press
as:
L.N.C.
Visser,
et
al.,
Assessing
engagement
while
viewing
video
vignettes;
validation
of
the
Video
Engagement
Scale
(VES),
Patient
Educ
Couns
(2015),
http://dx.doi.org/10.1016/j.pec.2015.08.029
emotions,
than
they
project
themselves
in
the
role
of
the
video-
patient
and
experience
what
they
themselves
would
feel
in
that
situation.
This
is
an
important
realization
for
the
video-vignettes
methodology,
because
it
suggests
that
APs
emotions
when
viewing
the
video
may
be
largely
restricted
to
the
ones
displayed
by
the
video-patient.
More
research
is
needed
to
investigate
these
issues.
Future
research
should
further
test
the
four-dimensional
model
of
engagement
resulting
from
our
analyses.
For
now,
researchers
can
tentatively
use
scores
on
separate
dimensions
to
distinguish
between
different
types
of
engagement.
Further
conrmation
of
the
scales
dimensional
structure
will
ensure
an
even
richer
understanding
of
APs
experiences
of
video
vignettes.
Furthermore,
future
research
could
be
aimed
at
shortening
the
scale
to
further
improve
on
it.
At
present,
there
is
insufcient
evidence
to
eliminate
any
items
from
the
scale.
Reliability
scores
and
structural
equation
modeling
results
from
the
current
study
could
be
examined
in
future
research
to
select
the
items
with
the
best
t.
Next,
the
resulting
shortened
scale
would
need
to
be
empirically
validated
before
use.
Until
then,
use
of
the
15-item
scale
is
most
appropriate.
4.2.
Conclusion
In
conclusion,
we
developed
and
validated
a
15-item
scale
to
assess
video-vignette
viewers
engagement.
Results
from
two
studies
show
that
the
VES
reliably
and
validly
measures
APs
engagement
in
health
communication
research
using
video
vignettes.
First,
the
VES
can
be
employed
to
assess
the
validity
of
this
kind
of
research
and
it
therefore
contributes
to
the
rigor
of
future
video-vignette
studies.
Second,
the
VES
enables
distinguish-
ing
between
different
types
of
engagement,
but
this
should
happen
cautiously
for
now.
Further
testing
of
the
scale
is
needed
to
more
solidly
establish
its
dimensionality.
4.3.
Practice
implications
Results
from
two
studies
suggest
that
the
VES