Preliminary Evidence of Equivalence of Alternate Forms of the ImPACT

Abstract

The ImPACT (Immediate Postconcussion Assessment and Cognitive Testing) is a computerized neurocognitive test used to assist in the management of sport concussion management. A number of studies have documented the reliability and sensitivity of the ImPACT, but no studies have examined the equivalence of the ImPACT's alternate forms. The objective of our study was to determine the equivalence of the ImPACT's five alternate forms. Participants were administered alternate forms of ImPACT based on clinically relevant time frame derived from an extensive sports concussion database. Participants completed a baseline assessment followed by various combinations of the remaining alternate forms at 45 and 50 days. Inferential Confidence Intervals were calculated for each composite score for all alternate forms. We found non-equivalence between ImPACT form 1 and forms 2, 3, and 4 on the Verbal Memory and between forms 2 and 4. ImPACT forms 1 and 3 were not equivalent on the Visual Memory Composite. Finally, ImPACT forms 3 and 4 were not equivalent on the Visual Motor Speed and Reaction Time Composites. Alternate form equivalence is necessary to minimize measurement error and optimize clinical decision making. Clinicians using the ImPACT should consider non-equivalence of some forms on certain Composites when interpreting ImPACT following sports concussion.

Full text

This article was downloaded by: [Jacob Resch]
On: 09 October 2013, At: 06:34
Publisher: Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered
office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
The Clinical Neuropsychologist
Publication details, including instructions for authors and
subscription information:
http://www.tandfonline.com/loi/ntcn20
Preliminary Evidence of Equivalence of
Alternate Forms of the ImPACT
Jacob E. Resch
a
, Stephen Macciocchi
b
& Michael S. Ferrara
c
a
Brain Injury Laboratory, Department of Kinesiology, University of
Texas Arlington, TX, USA.
b
Shepherd Center, Atlanta, GA, USA.
c
The University of New Hampshire, Durham, NH, USA.
Published online: 08 Oct 2013.
To cite this article: Jacob E. Resch, Stephen Macciocchi & Michael S. Ferrara , The Clinical
Neuropsychologist (2013): Preliminary Evidence of Equivalence of Alternate Forms of the ImPACT,
The Clinical Neuropsychologist, DOI: 10.1080/13854046.2013.845247
To link to this article: http://dx.doi.org/10.1080/13854046.2013.845247
PLEASE SCROLL DOWN FOR ARTICLE
Taylor & Francis makes every effort to ensure the accuracy of all the information (the
“Content”) contained in the publications on our platform. However, Taylor & Francis,
our agents, and our licensors make no representations or warranties whatsoever as to
the accuracy, completeness, or suitability for any purpose of the Content. Any opinions
and views expressed in this publication are the opinions and views of the authors,
and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content
should not be relied upon and should be independently verified with primary sources
of information. Taylor and Francis shall not be liable for any losses, actions, claims,
proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or
howsoever caused arising directly or indirectly in connection with, in relation to or arising
out of the use of the Content.
This article may be used for research, teaching, and private study purposes. Any
substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,
systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &
Conditions of access and use can be found at http://www.tandfonline.com/page/terms-
and-conditions

Preliminary Evidence of Equivalence of Alternate Forms of
the ImPACT
Jacob E. Resch
1
, Stephen Macciocchi
2
, and Michael S. Ferrara
3
1
Brain Injury Laboratory, Department of Kinesiology, University of Texas Arlington, TX, USA
2
Shepherd Center, Atlanta, GA, USA
3
The University of New Hampshire, Durham, NH, USA
The ImPACT (Immediate Postconcussion Assessment and Cognitive Testing) is a computerized
neurocognitive test used to assist in the management of sport concussion management. A
number of studies have documented the reliability and sensitivity of the ImPACT, but no
studies have examined the equivalence of the ImPACT’s alternate forms. The objective of our
study was to determine the equivalence of the ImPACT’s five alternate forms. Participants were
administered alternate forms of ImPACT based on clinically relevant time frame derived from
an extensive sports concussion database. Participants completed a baseline assessment followed
by various combinations of the remaining alternate forms at 45 and 50 days. Inferential
Confidence Intervals were calculated for each composite score for all alternate forms. We
found non-equivalence between ImPACT form 1 and forms 2, 3, and 4 on the Verbal Memory
and between forms 2 and 4. ImPACT forms 1 and 3 were not equivalent on the Visual
Memory Composite. Finally, ImPACT forms 3 and 4 were not equivalent on the Visual Motor
Speed and Reaction Time Composites. Alternate form equivalence is necessary to minimize
measurement error and optimize clinical decision making. Clinicians using the ImPACT should
consider non-equivalence of some forms on certain Composites when interpreting ImPACT
following sports concussion.
Keywords: Equivalence; Alternate forms; Concussion; Computerized neuropsychological tests; Psychometric;
Cognitive.
INTRODUCTION
During the past 10 years, research and consensus panels have recommended a
multi-faceted clinical approach for managing sports concussions (Aubry et al., 2002;
Broglio, Macciocchi, & Ferrara, 2007; Guskiewicz et al., 2004; Harmon et al., 2013;
McCrory et al., 2005, 2009). Neurocognitive testing, balance assessment, and
self-reported symptom scales are typically utilized to assist healthcare professionals in
making return-to-play (RTP) decisions for athletes following the diagnosis of sport
concussion (McCrory et al., 2009). Despite controversy, computerized neurocognitive
testing (CNT) has been commonly accepted as a replacement for traditional neurocog-
nitive testing at all levels of sport. Rationale for this shift in clinical practice includes
the ability to assess large numbers of athletes, standardized delivery, and the availabil-
ity of alternate forms (Covassin, Elbin, Stiller-Ostrowski, & Kontos, 2009; Guskiewicz
et al., 2004; Meehan, d’Hem ecourt, Collins, Taylor, & Comstock, 2012). The ImPACT
Address correspondence to: Jacob E. Resch, Ph.D., ATC, Box 19259, 113 Maverick Activities Center,
Arlington, TX 76019-0259, USA. Email: resch@uta.edu
(Received 28 October 2012; accepted 11 September 2013)
The Clinical Neuropsychologist, 2013
http://dx.doi.org/10.1080/13854046.2013.845247
Ó 2013 Taylor & Francis
Downloaded by [Jacob Resch] at 06:34 09 October 2013

is a popular CNT and is used in many sports settings (Broglio, Ferrara, Macciocchi,
Baumgartner, & Elliott, 2007; Meehan et al., 2012; Randolph, McCrea, & Barr, 2005).
Psychometric studies conducted over the past 10 years examining the reliability
of the Immediate Postconcussion Assessment and Cognitive Testing (ImPACT) have
found variable test–retest reliability coefficients (Broglio, Ferrara et al., 2007; Resch
et al., 2013; Schatz, 2009). Broglio, Ferrara et al. (2007) and Resch et al. (2013)
reported sub-optimal ImPACT test–retest reliability coefficients ranging from .15 to .76
over a 50-day period. In contrast, intraclass correlation coefficients (ICC) ranged from
.42 to .85 in two studies employing 1- and 2-year test–retest intervals (Elbin, Schatz,
& Covassin, 2011; Schatz, 2009).
One possible explanation for variability in ImPACT test–retest reliability may be
the different test–retest time intervals used in existing research. Another plausible expla-
nation for the observed differences in test –retest reliability coefficients may be variability
in ImPACT forms used in test–retest reliability research (Broglio, Ferrara et al., 2007;
Elbin, 2012; Elbin et al., 2011; Resch et al. 2013; Schatz, 2009, 2012). The ImPACT
has five alternate or parallel forms designed to minimize practice effects associated with
repeat testing while simultaneously maintaining test sensitivity. While alternate forms
are a strength of any instrument, tests using multiple forms must show some evidence of
form equivalence (Crocker & Algina, 2008). For example, the Hopkins Verbal Learning
Test (HVLT) and Hopkins Verbal Learning Test-Revised have six alternate forms. Each
version of the HLVT has been demonstrated to possess equivalent alternate forms using
various statistical methods over varying periods of time (Brandt, 1991; O’Neil-Pirozzi,
Goldstein, Strangman, & Glenn, 2012). Although the ImPACT’s alternate forms are
presumed to be equivalent, no evidence exists to support or refute this claim.
The objective of the current study was to inves tigate form equivalence of the five
alternate forms of ImPACT using a clinically relevant post-concussive testing
paradigm. Evidence of alternate form equivalence or non-equivalence may help
identify and explain the reported variability in test–retest reliability. Our findings also
have the potential to contribute to the ImPACT’s development by examining form
equivalence and to determine which forms may be most appropriate in repeated
examinations of concussed athletes.
METHOD
Participants
The current study is based on a separate, but related, investigation which
addressed the test–retest reliability of forms 1, 2, and 3 of the ImPACT using clinically
relevant time points. (Resch et al., 2013), in which 152 collegiate-aged participants
were recruited from the general student body of a large metropolitan university to take
part. Participants were recruited via face-to-face classroom recruitment, word-of-mouth,
and flyers. Participants were excluded if they had a history of concussion diagnosed
by a physician or certified athletic trainer within 6 months prior to or during the study;
if English was not their primary language; if they were diagnosed with a learning
disability or attention deficit disorder; and if they consumed alcohol within 24 hours
of their testing session. Furthermore, participants were removed from analysis if their
performance on the first administration of ImPACT was considered invalid based on
2 JACOB E. RESCH ET AL.
Downloaded by [Jacob Resch] at 06:34 09 October 2013

the manufacturer’s criteria, or if he or she fell below acceptable performance (85%) on
Green’s Word Memory Test (WMT), a measure of effort (Green, 2003; M. Lovell,
Collins, & Maroon, 2009).
In order to assess the equivalence of alternate forms participants were randomly
assigned into one of six groups via systematic sampling. Each group was tested at
three time points (day 1, day 45, and day 50) with three form s of ImPACT. All groups
were administered form 1 at day 1. Groups randomly and systematically differed on
which form of ImPACT they received at day 45 and day 50. The ordering of forms
for each of the six groups was as follo ws: (Group 1) (forms 1,2,3), Group 2 (forms
1,2,4), Group 3 (forms 1,2,5), Group 4 (forms 1,3,4), Group 5 (forms 1,3,5), and
Group 6 (forms 1,4,5). While utilization of forms 1, 2, and 3 is presumably the most
frequently used combination of alternate forms, comparisons were designed to directly
compare each of the ImPACT’s alternate forms.
Tests
Green’s Word Memory Test (Green’s Publishing Inc, Edmonton, Alberta) was
used as a measure of participant effort. The Word Memory Test (WMT) provides
scores for immediate and delayed recall and consistency of responses, which is the
percent agreement between delayed recall sections of the test. The remaining compos-
ite scores include multiple choice, paired associates, and free recall. A score P 85%
for each composite score is needed to be considered a valid effort (Green, 2003). The
WMT has an extensive research background and is routinely used in clinical practice
for detecting suboptimal effort (Green, 2005).
The ImPACT (version 6.7.723, ImPACT applications, Pittsburgh, PA) assesses
attention, memory, reaction time, and information processing speed (M. Lovell,
2007b). The ImPACT is commonly used at the seconda ry, post-secondary, collegiate,
and professional levels of sport (M. Collins, Lovell, Iverson, Ide, & Maroon, 2006;
M. W. Collins et al., 1999; M. R. Lovell & Collins, 1998; M. R. Lovell et al., 2003;
Pellman, Lovell, Viano, & Casson, 2006). The ImPACT consists of eight tests
including immediate and delayed word recall, immediate and delayed design recall, a
symbol match test, a three letter recall, the Xs and Os test, and the color match test.
Combinations of two or more of the aforementioned tests are used to calculate five
composite scores including Visual and Verbal Memory, Reaction Time, Visual Motor
Speed, and Impulse Control.
Valid performance on the ImPACT is determined by several indicators which are
automated during the baseline (form 1) assessment. The validity indicators include
Impulse Control which is the total number of incorrect responses for the Xs and Os
interference task and the number of commissions during the Color Match Test
(M. Lovell, 2007a, 2007b). Impulse Control scores > 20 on a baseline test indicate an
invalid effort. Additional validity criteria include scores > 30 for the number of Xs
and Os incorrect, < 69% correct for word memory, < 50% correct for design
memory, and correctly recalling < 8 letters for the Three Letters test (M. Lovell,
2007a). In addition to reviewing each participant’s performance based on the
ImPACT’s automated criteria, the same criteria from each participant were reviewed
manually to ensure a valid baseline test. The invalidity criteria for ImPACT were cou-
pled with Green’s WMT and exclusion criteria (<85% for the immediate and delayed
ALTERNATE FORMS OF ImPACT 3
Downloaded by [Jacob Resch] at 06:34 09 October 2013

recall, consistency of responses, multiple choice, and paired associate composite
scores) found on the health questionnaire to determine the validity of each participant’s
data and subsequent inclusion for data analysis at each time point. Green’s WMT was
used during the second and third time points to ensure a valid effort.
Testing sessions
Time point 1. Recruited participants reported to the Athletic Training Research
Laboratory. Each participant was provided a separate work-station/office in order to
avoid any environmental distractions and no more than four participa nts were tested at
each session. All participants reviewed and signed an Institutional Research Board con-
sent form prior to beginning the study. Consented participants then completed a healt h
questionnaire which was reviewed to determine inclusion/exclusion from the study. Par-
ticipants then completed the initial portion of Green’s WMT followed by the completion
of the initial demographic, concussion history, and symptom portions of the ImPACT
test battery. Participants then completed the form 1 (baseline) of ImPACT. Following the
completion of ImPACT form 1, participants were asked to remain, seated quietly, until
the remainder of the 30-minute delay imposed by Green’s WMT expired. To complete
the baseline session, participa nts completed the remainder of Green’s WMT.
Time points 2 and 3. Participants returned to the Athletic Training Research
Laboratory on time point 2 (46.9 ± 3.4 days) and time point 3 (54 ± 2.5 days) at the
same time of day as time point 1. Participants completed the health questionnaire at
each session in order to determine continued eligibility for study inclusion. After
determining eligibility, participants completed the initial portion of Green’s WMT.
Dependent on random group assignment, participants completed the assigned alternate
form of ImPACT, followed by the remainder of Green’s WMT.
Statistical procedures
For this study, form equivalence was determined by using inferential confidence
intervals (ICIs). ICIs utilize a reduction factor to ensure statistically different values
also possess non-overlapping confidence intervals (Tryon, 2001). The ICIs for this
study employ the revised version of Tryon’s formula which takes into account varying
sample sizes leading to different critical t values utilized to determine statistical
equivalence (Tryon & Lewis, 2008). The formulas utilized for this study are as follows
for two hypothetical groups:
Group 1:
Lower ICI:

Y
1
 Et
v1
a=2
S

Y1
Upper ICI:

Y
1
þ Et
v1
a=2
S

Y1
Group 2:
Lower ICI:

Y
2
 Et
v2
a=2
S

Y2
Upper ICI:

Y
2
þ Et
v2
a=2
S

Y2
where

Y is equal to the mean test score, E represents the reduction factor, t
v1
α/2
repre-
sents the t value for the respective degrees of freedom (n – 1), and S

Y1
the standard
error of the mean. E and S

Y1
are calculated as follows:
4 JACOB E. RESCH ET AL.
Downloaded by [Jacob Resch] at 06:34 09 October 2013

E ¼
t
v12
a=2
S

Y2

Y1
t
v1
a=2
S

Y1
þ t
v2
a=2
S

Y2
S

Y 1
¼
S
ffiffiffi
n
p
where t
v12
α/2
is the t value determined by summing the sample size for group 1 and
group 2 with degrees of freedom defined by (n – 1), and S

Y2

Y1
which represents esti-
mated standard error of difference between sample means for two independent samples
which is =
ffiffi
ð
p
S
2

Y1
þ S
2
Y2
Þ (Tryon & Lewis, 2008).
When determining equivalence, the difference between the lower limit of the ICI
for the group with the higher test mean and the upper limit of the ICI for the group
possessing the lower test mean must be less than or equal to zero. If the difference is
greater than zero this means the ICIs do not overlap, suggesting non-equivalence.
For analysis of equivalence, alternate forms of ImPACT were compared only if
the tests preceding the form were the same. For example, for the comparison of forms
1, 2, and 3 all six groups (n = 108) were administered form 1 at time point 1. Only
groups 1, 2, and 3 (n = 54) were administered form 2 at day 45 and only group 1
(n = 18) was administered form 3 at day 50. Thus, form 1 (n = 1 08) was compared to
form 2 (n = 54). Form 2 (n = 54) was compared to form 3 (n = 18). All form
comparisons were performed employing this grouping methodology. To further analyze
equivalence between forms and demonstrate the advantages of using ICIs, Pearson
correlation coefficients and independent t tests were employed. As previously
discussed, these statistical tests may suggest equivalence (non-significant differences or
high correlation values) of alternate forms while ICIs may suggest differently. Our
results include all three statistics to further illustrate this point.
Repeated measures analysis of variance (ANOVA) was utilized to assess test
differences across time for the WMT. Greenhouse-Geisser corrections were imple-
mented when sphericity violations occurred. A Bonferroni adjustment was made for
multiple pair-wise comparisons used during post-hoc analysis. All data analyses were
performed utilizing SPSS version 18.0 (SPSS, Chicago, IL) and statistical equivalence
was set at α < .05 (Hinton-Bayre & Geffen, 2005).
RESULTS
Of the original sample, 44 participants did not meet inclusion criteria or did not
complete all testing sessions. The final sample consisted of 108 participants composed
of 33 male (31%) and 75 female (69%) healthy, college-aged students who met inclu-
sion criteria (age: 20.6 ± 1.5 years; height: 171.6 ± 9.7 cm; weight: 66.9 ± 11.9 kg;
and self-reported SAT scores were 1124.5 ± 117.66. Descriptive statistics for each of
the six groups can be found in Table 1. No statistical differences existed between
groups in terms of height, weight, and/or SAT scores. All participants scored above
85% on the WMT subscores suggesting a valid effort was provided at all time points.
Sample size, ImPACT means, and standard deviations for each group at each
time point are presented in Table 2. The results of our ICI analyses are depicted in
ALTERNATE FORMS OF ImPACT 5
Downloaded by [Jacob Resch] at 06:34 09 October 2013

Figure 1. Table 3 depicts which neurocognitive indices were equivalent (+) and
non-equivalent (–) to one another. Of a 74 comparisons, approximately 19% were
observed to be non-equivalent. Form 1 accounted for the majority (36%) of
non-equivalent comparisons. Specifically, Form 1 Verbal Memory, Visual Memory, and
Visual Motor Speed were found to be non-equivalent to the same neurocognitive index
on one or more alternate forms.
Weak correlation coefficients (r = – .304 to .263) were observed for the ImPACT
composite scores deemed non-equivalent by the ICI methods. A summary of
correlation coefficients between each alternate form and their composite scores may be
found in Table 4.
The results of our independent t-tests revealed significant differences between
form 1 and 2, t
(160)
= – 2.445, p = .016, and between forms 1 and 3, t
(142)
= – 3.421,
p = .001, on the Verbal Memory Composite. The remaining analyses revealed in no
additional significant differences. The conflicting results between the indepe ndent t-test
and the comparison’s ICIs are due to the more stringent nature of the ICI method.
Tryon (2001) suggests the ICI method is approximately 41% more stringent than the
standard t-test. Results for these analyses are described in Table 5.
DISCUSSION
The purpose of the current study was to investigate the equivalence of the
ImPACT’s alternate forms. Several of the ImPACT’s alternate form composite scores
(Verbal and Visual Memory, Visual Motor Speed, and Reaction Time Composite
scores) were observed to be non-equivalent in a healthy, college-aged sample who
provided good effort as evidenced by validity criteria set by the ImPACT’s manufac-
turer and the WMT (Green 2005; M. Lovell, 2007b; M. R. Lovell & Colli ns, 1998).
The observed non-equivalence of some composites on some alternate forms may
partially explain the variability of reliability coefficients reported in earlier research
(Broglio, Ferrara et al., 2007; Elbin et al., 2011; Resch et al., 2013; Schatz, 2009)
Table 1. Means and standard deviations (SD) for study participant demographics
Group n
Age
Years of Education SAT(years)
1 18 20.2 13.4 1183.1
(1.54) (1.14) (125.84)
2 18 20.9 13.9 1242.5
(1.68) (1.63) (140.40)
3 18 20.5 13.8 1209.9
(1.50) (1.22) (69.76)
4 18 21.00 14.1 1223.1
(1.28) (1.16) (130.47)
5 18 21.1 14.1 1207.9
(1.30) (24.40) (117.75)
6 18 20.3 13.6 1262.1
(1.74) (1.5) (117)
Total 108 20.6 13.8 1224.5
(1.52) (1.30) (117.66)
6 JACOB E. RESCH ET AL.
Downloaded by [Jacob Resch] at 06:34 09 October 2013

Table 2. Means and standard deviations (SD) for ImPACT composite scores at each time point for
comparisons exhibiting non-equivalence between forms
Form n Baseline Time Point 2 Time Point 3
Form 1 108
Verbal Memory 90.7 (7.3)
Visual Memory 81.0 (11.2)
Visual Motor Speed 42.09(6.26)
Reaction Time (ms) .54(.06)
Form 2 54
Verbal Memory 93.7 (7.2)
Visual Memory 81.7 (10.0)
Visual Motor Speed 42.34(5.22)
Reaction Time (ms) .53(.06)
Form 3 36
Verbal Memory 95.2(5.0)
Visual Memory 84.2(9.8)
Visual Motor Speed 44.11(5.64)
Reaction Time (ms) .54(.06)
Form 4 18
(Form 2 vs. Form 4)
Verbal Memory 90.5 (10.7)
Visual Memory 81.4 (8.9)
Visual Motor Speed 43.05(6.59)
Reaction Time (ms) .51(.04)
Form 4 18
(Form 3 vs. Form 4)
Verbal Memory 95.7 (4.3)
Visual Memory 84.5(8.2)
Visual Motor Speed 43.82(5.62)
Reaction Time (ms) .58(.06)
Form 5 18
(Form 2 vs. Form 5)
Verbal Memory 94.9 (5.7)
Visual Memory 81.8 (9.4)
Visual Motor Speed 42.99 (7.06)
Reaction Time (ms) .54 (.08)
Form 5
(Form 3 vs. Form 5) 18
Verbal Memory 94.2 (5.4)
Visual Memory 85.9 (11.1)
Visual Motor Speed 46.78 (5.41)
Reaction Time (ms) .52 (.06)
Form 5
(Form 4 vs. Form 5) 18
Verbal Memory 92.9 (8.4) 94.3 (6.7)
Visual Memory 78.6 (12.5) 80.5 (15.8)
Visual Motor Speed 42.78 (6.55) 43.22 (7.35)
Reaction Time (ms) .55 (.05) .57 (.08)
ALTERNATE FORMS OF ImPACT 7
Downloaded by [Jacob Resch] at 06:34 09 October 2013

Additionally, our results provi de information for clinicians who must determine if
changes between baseline and post-injury assessments are the result of the sport
concussion or psychometric characteristics of the ImPACT’s alternate forms.
Regardless of study design, the weakest reliability coefficients have been
routinely observed for the ImPACT’s Verbal (.37 to .62) and Visual (.26 to .70)
Memory Composite scores (Broglio, Ferrara et al., 2007; Elbin et al., 2011; Resch
Figure 1.
8 JACOB E. RESCH ET AL.
Downloaded by [Jacob Resch] at 06:34 09 October 2013

et al., 2013; Schatz, 2009). Using ICIs we observed non-equivalence between form 1
and forms 2, 3, and 4 for Verbal Memory Composite scores. We also observed
non-equivalence for the Visual Memory Composite score between forms 1 and 3.
The methodology employed by Broglio and Resch incorporated Forms 1, 2, and 3
which were delivered over a 50-day period. When reviewing a correlation matrix
specifically for these alternate forms (Table 6), correlations coefficients ranged from
–.07 to –.22 for Composite Verbal Memory, –.15 to .42 for Composite Visual Memory,
Table 3. ImPACT alternate form composite scores which were observed to be equivalent (+) and
non-equivalent (–) to one another
Composite Score Form 1 Form 2 Form 3 Form 4 Form 5
Form 1 Verbal Memory + –––NA
Visual Memory + + – +NA
VMS + + – +NA
Reaction Time (ms) + + + + NA
Form 2 Verbal Memory – ++– +
Visual Memory + ++++
VMS +++++
Reaction Time (ms) + ++++
Form 3 Verbal Memory – ++++
Visual Memory – ++++
VMS – ++++
Reaction Time (ms) + + + – +
Form 4 Verbal Memory ––+++
Visual Memory + ++++
VMS +++++
Reaction Time (ms) + + – ++
Form 5 Verbal Memory NA ++++
Visual Memory NA ++++
VMS NA++++
Reaction Time (ms) NA ++++
Table 4. Correlation coefficient ranges for ImPACT composite scores for each form (Forms used for
comparison)
Form Verbal Memory Visual Memory Visual Motor Speed Reaction Time
Form 1 .03 – .40 .11 – .45 .01 – .74 .00 – .43
Form 2 .03 – .31 .04 –.52 .03 – .74 .05 – .44
Form 3 .03 – .66 .13 – .49 .02 – .76 .06 – .71
Form 4 (1) .03 – .55 .04 – .73 .03 – .74 .18 – .74
Form 4 (2) .08 – .33 .02 – .52 .04 – .41 .03 – .44
Form 4 (3) .03 – .66 .02 – .49 .08 – .76 .02 – .71
Form 5 (2) .06 – .33 .05 – .51 .03 – .48 .00 – .44
Form 5 (3) .03 – .31 .05 – .46 .01 – .41 .01 – .32
Form 5 (4) .03 – .55 .02 – .73 .02 – .74 .03 – .74
ALTERNATE FORMS OF ImPACT 9
Downloaded by [Jacob Resch] at 06:34 09 October 2013

–.08 to .75 for Visual Motor Speed, and .08 to .43 for Reaction Time. These weak to
moderate correlation coef ficients may further explain the variable reliability results
reported by Broglio and Resch (Broglio, Ferrara et al., 2007; Resch et al., 2013).
Non-equivalence of the ImPACT’s Verbal and Visual Memory Composites may
be due to word and designs lists that vary in difficulty/complexity. For example, one
component of the ImPACT’s Verbal Memory Composite score is immedi ate and
delayed word memory. Athletes completing the ImPACT are presented with a series of
12 words twice during the first testing module. They are then asked to recall if a word
was or was not on the original list of 12 words. At the completion of the ImPACT
(approximately 15 minutes later), the individuals are asked to complete the same task
without a nother presentation of the original word set. The ImPACT administers differ-
ent words sets between a lternate forms, so non-equivalence for the Verbal Memory
Composite score may be due to varying difficulty of word sets on different forms.
Word composition has been shown to impact an individual’s ability to recall a
series of words over a period of time. Baddeley and Buchanan demonstrated that
participants are able to recall significantly fewer words when monosyllabic words (can,
van, dog) were delivered compared to polysyllabic (water, dinner, supper) words,
spoken duration increased (i.e., morphine versus décor), and the number of stimuli
was increased from two to eight administered over short periods of time (Baddeley,
Thompson, & Buchanan, 1975). A review of the ImPACT’s five alternate word lists
suggests varying word complexity based on the numbe r of syllables and duration. The
ImPACT form 1 word list consists of 10 monosyllabic words and 2 polysyllable
words. The form 2 word list consi sts of seven monosyllabic words and five two-sylla-
ble words. The words lists for forms 3, 4, and 5 consist of 11 monosyllabic words and
1 two-syllable word. This discrepancy in the number of mono- and two-syllable words
between alternate forms coupled with word list length of 12 words may partially
explain form non-equivalence. Additionally, the discrepancies in word composition
may also contribute to the weak to moderate reliability coefficients reported by Broglio
and Resch between the ImPACT alternate forms 1, 2 and 3 for Verbal Memory
Table 5. Results of alternate form comparison analyses
Comparison t (p value) Pearson’s r eRG
Form 1 vs. Form 2 –2.445
⁄
.263 .010
(Verbal Memory) (.016)
Form 1 vs. Form 3 –3.421
⁄
–.218 .030
(Verbal Memory) (.001)
Form 1 vs. Form 4 –1.188 .028 .002
(Verbal Memory) (.237)
Form 1 vs. Form 3 –1.543 –.147 .002
(Visual Memory) (.125)
Form 1 vs. Form 3 –1.719 –.117 .333
(Visual Motor Speed) (.757)
Form 3 vs. Form 4 –1.742 .712
⁄
.002
(Reaction Time) (.087)
Form 2 vs. Form 4 1.441 –.087 .005
(Verbal memory) (.154)
Equivalence Range is denoted by (eRG). In order for two forms to be deemed non-equivalent, eRG must
satisfy x > 0.
⁄
represents significance at p ≤ .05.
10 JACOB E. RESCH ET AL.
Downloaded by [Jacob Resch] at 06:34 09 October 2013

Table 6. Correlation matrix to demonstrate discriminant validity
Verbal Memory Visual Memory Visual Motor Speed Reaction Time
Form 1 Form 2 Form 3 Form 1 Form 2 Form 3 Form 1 Form 2 Form 3 Form 1 Form 2 Form 3
Verbal Memory Form 1 1 .26 –.22 .22† .27† .01 .07 .21 .25 –.05 .05 .05
(n = 108)
Form 2 1 –.07 .22 .37‡–.19 .18 .15 –.03 –.32† .14 –.09
(n = 54)
Form 3 1 .14 –.09 .33†–.08 .04 .20 .05 .05 .17
(n = 36)
Visual Memory Form 1 1 .42‡–.15 .03 .09 –.10 –.29‡–.07 .07
(n = 108)
Form 2 1 –.36‡ .28‡ .21 –.01 –.28†–.09 .00
(n = 54)
Form 3 1 .18 .15 .18 .11 –.03 –.07
(n = 36)
VMS Form 1 1 .75‡–.12 .33‡–.35† .07
(n = 108)
Form 2 1 –.08 –.21 –.40‡–.02
(n = 54)
Form 3 1 –.32 –.07 –.56‡
(n = 36)
Reaction Time Form 1 1 .43‡ .25
(n = 108)
Form 2 1 .08
(n = 54)
Form 3 1
(n = 36)
In order to possess discriminant validity, composite scores should not be significantly correlated to each other. †p ≤ .01, ‡p ≤ .001.
ALTERNATE FORMS OF ImPACT 11
Downloaded by [Jacob Resch] at 06:34 09 October 2013

Composite (Broglio, Ferrara et al., 2007; Resch et al., 2013). Using alternate word lists
composed of words possessing the same number of syllables may result in alternate
form equivalence and increase test–retest reliability.
Word frequency may also contribute to the non-equivalence associated with the
verbal memory composite score. Connine and colleagues reported the presentation of
high-frequency words in a recogniti on task resulted in significantly decreased participant
reaction time (ms) and an increase in the number of words correctly identified (Connine,
Mullenix, Shernoff, & Yelen, 1990). Additionally when participa nts were presented with
low-frequency words that they were familiar with, they responded similarly to the high-
frequency words (Connine et al., 1990). Future research should address the influence of
word frequency on computerized neurocognitive tests such as the ImPACT.
As with the ImPACT’s immediate and delayed word memory test, the visual
memory test involves presentation of with 12 stimuli (designs) consi sting of free form
lines. Following the second showing of the same 12 designs, athletes completing the
ImPACT are asked to recall if a series of the designs share the same orientation as
those previously shown. At the conclusion of the ImPACT (approximately 20 minutes
later), athletes are once again asked to recall if the orientation of a series of presented
designs matched the orientation of the original design list. As with word memory, the
ImPACT provides an alternate design list for each alternate form.
The non-equivalence between forms 1 and 3 of the ImPACT for the Visual
Memory Composite score may be due to several factors. First, research addressing
visual short-term memory (VSTM) has shown the ability to correctly recall complex
polygons significantly decreases after the presentation of four items (Luck & Vogel,
1997; Luria, Sessa, Gotler, Jolicoeur, & Dell’Acqua, 2010) An increased set of designs
(> 4 items) leads to an increased number of decisions to be made (i.e., the correct
orientation of each design) which results in decreased accuracy and an increased
number of errors during delayed design recall (Luck & Vogel, 1997). Variable design
complexity may also influence form equivalence. Design features such as shape, color,
and/or orientation have been shown to significantly reduce an individua l’s VSTM
capacity. Polygons, like those used in the ImPACT consist of several features.
Specifically, the ImPACT’s designs are defined by a conjunction of shape, size, and
orientation. The ImPACT’s alternate designs lists also vary by color but this feature is
consistent across shapes in a given design list. VSTM capacity decreases consequent
to design complexity (i.e. more features) (Luria et al., 2010).
Two ways to potent ially correct for form non-equivalence include decreas ing the
number of designs presented (< 4) while maintaining design complexity. The second
way to achieve form equivalence is to decrease the number of design features while
maintaining the same number of stimuli (i.e., 12). For instance, a series of 12 po lygons
consisting of the same number of straight lines (i.e., 4) and that are all the same color
would potentially allow healthy participants to increase VSTM resolution and increase
recall accuracy while still challenging those with limited VSTM capacity (i.e.,
concussed athletes).
Clinical implications
A recent study reported 93% of certified athletic trainers use computerized
neurocognitive testing use the ImPACT (Meehan et al., 2012). With this in mind, it is
12 JACOB E. RESCH ET AL.
Downloaded by [Jacob Resch] at 06:34 09 October 2013

important for healt h professionals, specifically athletic trainers, who use the ImPACT
to understand which composite scores are equivalent across alternate forms. When
using two alternate forms of the ImPACT, Table 3 may be used to assist clinicians in
determining which composite scores were observed to be equivalent or non-equivalent.
For example, when conducting the baseline assessment, the ImPACT automatically
delivers form 1. The delivery of form 1 also provides identification of impaired effort
and the automated calculation of reliable change indices upon repeated administration.
In the event of a concussion, typically forms 2, 3, 4, and/or 5 are administered
by clinicians. If a clinician chooses to deliver form 2 following a concussion, he/she
must be aware that the Verbal Memory Composite score may be non-equivalent when
compared to form 1. In this scenario the clinician should rely more on the remaining
composite scores such as Visual Memory, Visual Motor Speed, and Reaction Time
Composite scores which were observed to be equivalent. Likewise if a clinician
chooses to administer form 3 to compare to form 1, caution is warranted when
interpreting Verbal Memory, Visual Memory, and Visual Motor Speed composite
scores due to non-equivalence. Alternatively, since form 1 possessed the majority of
non-equivalent scores when compared to the remaining ImPACT alternate forms, clini-
cians may choose to deliver forms 2, 3, 4, or 5 as a baseline assessment. The major
disadvantages to this approach are clinicians would have to manually review each
baseline assessment to ensure test validity and calculate reliable change indexes
following a concussion. These calculations would be necess ary since ImPACT’s
validity criteria and reliable change index are calculated using form 1.
In terms of study limitations, unequal sample sizes were used to determine form
equivalence. Due to our study design, a varying number of participants had to com-
plete each alternate form of ImPACT. Additionally, the elaps ed time between test
administrations may have influenced our results. Though not typical of a study
addressing alternate form equivalence, our time points are consistent with the clinical
use of the ImPACT (Broglio, Ferrara et al., 2007). Typically when determining equiva-
lence of alternate forms participants complete both forms with minimal time between
administrations (Crocker & Algina, 2008). Alternate form s, as used by the ImPACT,
are delivered over relatively long periods of time. The current study employed clini-
cally relevant time points representative of the time between baseline, post-concussion,
and asymptomatic computerized neuropsychological testing at the collegiate level
(Broglio, Ferrara et al., 2007). Lastly, our study design did not allow for the calcula-
tion of practice effects. Though it appears that group scores improved across time for
each ImPACT composite score, the varying sample sizes make it difficult to determine
if improved performance is the results of repeated exposure.
CONCLUSION
The ImPACT has been commonly accepted as a replacement for traditional
neurocognitive testing at all levels of sport. Studies addres sing psychometric properties
of the ImPACT have shown high levels of sensitivity (Schatz, Pardini, Lovell, Collins,
& Podell, 2006) but variable levels of reliability with repeat testing (Broglio, Maccioc-
chi et al., 2007; Erlanger et al., 2003; Randolph et al., 2005; Schatz, 2009). Our results
suggest that the variable reliability reported in previous literature may partially be the
ALTERNATE FORMS OF ImPACT 13
Downloaded by [Jacob Resch] at 06:34 09 October 2013

result of non-equivalent composite scores across alternate forms. Ultimately, a
clinician’s primary concern when using computerized neuroco gnitive measures such as
the ImPACT is that any fluctuation in an athlete’s performance is due to their injury
rather than error (such as non-equivalent alternate forms) associated with the test. Our
results suggest that performance fluctuations following the baseline assessment may be
due to psychometric characteristics of the ImPACT. Many of the composite scores of
ImPACT’s alternate forms are equivalent, but several are not. Our primary concern is
the ImPACT form 1 compared to the remaining alternate forms. In order to enhance
form equivalence, test developers may want to address the psychometric properties of
form 1. Test manufacturers can also address form equiva lence in future iterations of
their software, which wi ll inherently improve the test–retest reliability coefficients and
ultimately make a more sensitive measure of sport concussion.
ACKNOWLEDGMENTS
The authors would like to thank Dr. Deborah Bandalos and Dr. Stephen Olejnik
for their statistical expertise and review throughout the development of this
manuscript. The authors would also like to thank Mikhail Bondarew, Jess Sandlin,
Teddy Sanders, and Anastasia Bobilev for their time and efforts in assisting in data
collection for this project.
REFERENCES
Aubry, M., Cantu, R., Dvorak, J., Graf-Baumann, T., Johnston, K. M., Kelly, J., Schamasch, P.
(2002). Summary and agreement statement of the 1st International Symposium on Concus-
sion in Sport, Vienna 2001. Clinical Journal of Sports Medicine, 12(1), 6–11.
Baddeley, A. D., Thomson, N., & Buchanan, M. (1975). Word length and structure of short-term
memory. Journal of Verbal Learning and Verbal Behavior, 14, 575–589.
Brandt, J. (1991). The Hopkins Verbal Learning Test: Development of a new memory test with
six equivalent forms. The Clinicial Neuropsychologist, 5(2), 125–142.
Broglio, S. P., Ferrara, M. S., Macciocchi, S. N., Baumgartner, T. A., & Elliott, R. (2007). Test–
retest reliability of computerized concussion assessment programs. Journal of Athletic
Training, 42(4), 509–514.
Broglio, S. P., Macciocchi, S. N., & Ferrara, M. S. (2007). Sensitivity of the concussion assess-
ment battery. Neurosurgery, 60(6), 1050–1057; discussion 1057–1058.
Collins, M., Lovell, M. R., Iverson, G. L., Ide, T., & Maroon, J. (2006). Examining concussion
rates and return to play in high school football players wearing newer helmet technology:
A three-year prospective cohort study. Neurosurgery, 58(2), 275–286; discussion 275–286.
Collins, M. W., Grindel, S. H., Lovell, M. R., Dede, D. E., Moser, D. J., Phalin, B. R., & McK-
eag, D. B. (1999). Relationship between concussion and neuropsychological performance in
college football players. JAMA, 282(10), 964–970.
Connine, C. M., Mullennix, J., Shernoff, E., & Yelen, J. (1990). Word familiarity and frequency
in visual and auditory word recognition. Journal of Experimental Psychology, Learning,
Memory, & Cognition, 16(6), 1084–1096.
Covassin, T., Elbin, R. J. III, Stiller-Ostrowski, J. L., & Kontos, A. P. (2009). Immediate post-
concussion assessment and cognitive testing (ImPACT) practices of sports medicine profes-
sionals. Journal of Athletic Training, 44(6), 639–644.
14 JACOB E. RESCH ET AL.
Downloaded by [Jacob Resch] at 06:34 09 October 2013

Crocker, L., & Algina, J. (2008). Introduction to classical and modern test theory. Mason, OH:
Cengage Learning.
Elbin, R. (2012, October 3). [Personal communication.].
Elbin, R. J., Schatz, P., & Covassin, T. (2011). One-year test–retest reliability of the online ver-
sion of ImPACT in high school athletes. The American Journal of Sports Medicine, 39(11),
2319–2324.
Erlanger, D., Feldman, D., Kutner, K., Kaushik, T., Kroger, H., Festa, J., … Broshek, D. (2003).
Development and validation of a web-based neuropsychological test protocol for sports-
related return-to-play decision making. Archives of Clinical Neuropsychology, 18, 293–316.
Green, P. (2003). Word Memory Test for Windows: User’s manual and program. Edmonton,
Canada: Green’s Publishing.
Green, P. (2005). Green’s Word Memory Test for Windows: User’s manual and program. Edmon-
ton, Canada: Green’s Publishing Inc.
Guskiewicz, K. M., Bruce, S. L., Cantu, R. C., Ferrara, M. S., Kelly, J. P., McCrea, M., Valo-
vich McLeod, T. C. (2004). National Athletic Trainers’ Association position statement:
Management of sport-related concussion. Journal of Athletic Training, 39(3), 280–297.
Harmon, K. G., Drezner, J. A., Gammons, M., Guskiewicz, K. M., Halstead, M., Herring, S. A.,
& Roberts, W. O. (2013). American Medical Society for Sports Medicine position state-
ment: Concussion in sport. British Journal of Sports Medicine, 47(1), 15–26.
Hinton-Bayre, A., & Geffen, G. (2005). Comparability, reliability, and practice effects on
alternate forms of the Digit Symbol Substitution and Symbol Digit Modalities tests.
Psychological Assessment, 17(2), 237–241.
Lovell, M. (2007a). ImPACT 2007 (6.0) clinical interpretation manual. Pittsburgh, PA: ImPACT
Applications Inc.
Lovell, M. (2007b). ImPACT 2007 (6.0) software user’s manual. Pittsburgh, PA: ImPACT Appli-
cations Inc.
Lovell, M., Collins, M. W., & Maroon, J. C. (2009). ImPACTtest.com Retrieved 12/01/2009,
2009, from http://www.impacttest.com/.
Lovell, M. R., & Collins, M. W. (1998). Neuropsychological assessment of the college football
player. Journal of Head Trauma Rehabilitation, 13(2), 9–26.
Lovell, M. R., Collins, M. W., Iverson, G. L., Field, M., Maroon, J. C., Cantu, R., … Fu, F. H.
(2003). Recovery from mild concussion in high school athletes. Journal of Neurosurgery,
98, 296–301.
Luck, S. J., & Vogel, E. K. (1997). The capacity of visual working memory for features and
conjunctions. Nature, 390(6657), 279–281.
Luria, R., Sessa, P., Gotler, A., Jolicoeur, P., & Dell
’Acqua, R. (2010). Visual short-term mem-
ory capacity for simple and complex objects. Journal of Cognitive Neuroscience, 22(3),
496–512.
McCrory, P., Johnston, K., Meeuwisse, W., Aubry, M., Cantu, R., Dvorak, J., … Schamasch, P.
(2005). Summary and agreement statement of the 2nd International Conference on Concus-
sion in Sport, Prague 2004. British Journal of Sports Medicine, 39(4), 196–204.
McCrory, P., Meeuwisse, W., Johnston, K., Dvorak, J., Aubry, M., Molloy, M., & Cantu, R.
(2009). Consensus statement on concussion in sport: The 3rd International Conference on
Concussion in Sport, held in Zurich, November 2008. British Journal of Sports Medicine,
43, i76–90.
Meehan, W. P.III, d’Hemecourt, P., Collins, C. L., Taylor, A. M., & Comstock, R. D. (2012).
Computerized neurocognitive testing for the management of sport-related concussions. Pedi-
atrics, 129(1), 38–44. doi: 10.1542/peds.2011-1972
O’Neil-Pirozzi, T. M., Goldstein, R., Strangman, G. E., & Glenn, M. B. (2012). Test–re-test reli-
ability of the Hopkins Verbal Learning Test-Revised in individuals with traumatic brain
injury. Brain injury, 26(12), 1425–1430.
ALTERNATE FORMS OF ImPACT 15
Downloaded by [Jacob Resch] at 06:34 09 October 2013

Pellman, E. J., Lovell, M. R., Viano, D. C., & Casson, I. R. (2006). Concussion in professional
football: Recovery of NFL and high school athletes assessed by computerized neuropsycho-
logical testing – Part 12. Neurosurgery, 58(2), 263–274; discussion 263–274.
Randolph, C., McCrea, M., & Barr, W. B. (2005). Is neuropsychological testing useful in the
management of sport-related concussion? Journal of Athletic Training, 40(3), 139–152.
Resch, J., Driscoll, A., McCaffrey, N., Brown, C. N., Macchiocchi, S., Baumgartner, T. A., …
Ferrara, M. S. (2013). ImPACT test retest reliability: Reliably unreliable? Journal of
Athletic Training, 48(4), 506–511.
Schatz, P. (2009). Long-term test–retest reliability of baseline cognitive assessments using
ImPACT. American Journal of Sports Medicine, 38(1), 47–53.
Schatz, P. (2012, October 3). [Personal communication.].
Schatz, P., Pardini, J. E., Lovell, M. R., Collins, M. W., & Podell, K. (2006). Sensitivity and
specificity of the ImPACT Test Battery for concussion in athletes. Archives of Clinical Neu-
ropsychology, 21(1), 91–99.
Tryon, W. (2001). Evaluating statistical difference, equivalence, and indeterminancy using infer-
ential confidence intervals: An integrated alternative method of conducting null hypothesis
statistical tests. Psychological Methods, 6(4), 371–386.
Tryon, W., & Lewis, C. (2008). An inferential confidence interval method of establishing statisti-
cal equivalence that corrects Tryon’s (2001) reduction factor. Psychological Methods, 13(3),
272–277.
16 JACOB E. RESCH ET AL.
Downloaded by [Jacob Resch] at 06:34 09 October 2013