Content uploaded by Thomas Beckman
Author content
All content in this area was uploaded by Thomas Beckman on Feb 26, 2019
Content may be subject to copyright.
Education Issues
Measuring Motivational Characteristics of
Courses: Applying Keller’s Instructional
Materials Motivation Survey to a Web-Based
Course
David A. Cook, MD, MHPE, Thomas J. Beckman, MD, Kris G. Thomas, MD,
and Warren G. Thompson, MD
Abstract
Purpose
The Instructional Materials Motivation
Survey (IMMS) purports to assess the
motivational characteristics of
instructional materials or courses using
the Attention, Relevance, Confidence,
and Satisfaction (ARCS) model of
motivation. The IMMS has received little
use or study in medical education. The
authors sought to evaluate the validity of
IMMS scores and compare scores
between standard and adaptive Web-
based learning modules.
Method
During the 2005–2006 academic year,
124 internal medicine residents at the
Mayo School of Graduate Medical
Education (Rochester, Minnesota) were
asked to complete the IMMS for two
Web-based learning modules.
Participants were randomly assigned to
use one module that adapted to their
prior knowledge of the topic, and one
module using a nonadaptive design.
IMMS internal structure was evaluated
using Cronbach alpha and
interdimension score correlations.
Relations to other variables were
explored through correlation with global
module satisfaction and regression with
knowledge scores.
Results
Of the 124 eligible participants, 79
(64%) completed the IMMS at least
once. Cronbach alpha was ⱖ0.75 for
scores from all IMMS dimensions.
Interdimension score correlations ranged
0.40 to 0.80, whereas correlations
between IMMS scores and global
satisfaction ratings ranged 0.40 to 0.63
(P⬍.001). Knowledge scores were
associated with Attention and Relevance
subscores (P⫽.033 and .01,
respectively) but not with other IMMS
dimensions (Pⱖ.07). IMMS scores were
similar between module designs (on a
five-point scale, differences ranged from
0.0 to 0.15, Pⱖ.33).
Conclusions
These limited data generally support the
validity of IMMS scores. Adaptive and
standard Web-based instructional
designs were similarly motivating.
Cautious use and further study of the
IMMS are warranted.
Acad Med. 2009; 84:1505–1509.
Motivation is widely acknowledged as
central to meaningful learning.
1–5
Although motivation is a characteristic of
the individual learner, for a teacher it is
useful to consider motivation from the
standpoint of the instructional design—
namely, the impact of course materials
and format on an individual’s motivation
to participate and learn, and specific
strategies to enhance the motivational
features of a course to better facilitate
engagement and learning. As new
technologies such as Web-based and just-
in-time learning bring personalized
instruction closer to reality, measurement
of and subsequent adaptation to
motivational needs may become
increasingly relevant.
6
Yet the assessment
of motivation has received little attention
in medical education.
Building on expectancy-value theories of
motivation, Keller
5,7
defined a four-
dimension model for motivation with
practical application to instructional
design—Attention, Relevance,
Confidence, and Satisfaction (ARCS):
• Attention, or interest, must be obtained
and maintained.
• Relevance to learners’ goals and needs
must be made clear.
• Learners must feel Confident in their
ability to succeed in learning
(expectancy for success).
• Learners should feel Satisfaction about
their accomplishments in the learning
opportunity.
Keller also developed an instrument, the
Instructional Materials Motivation
Survey (IMMS), to assess motivational
features of a course in each of the ARCS
dimensions (Keller JM. Development of
two measures of learner motivation:
Florida State University, 2006;
unpublished).
The information from such an
assessment could be used to improve a
course design generally, or to adapt a
course to an individual’s motivational
needs.
Only two studies
8,9
have used the IMMS
in medical education. Both used a Korean
translation of the IMMS to compare a
Web-based course with a series of face-
to-face lectures, and both found no
significant differences in overall
motivation scores between groups.
Dimension-specific subscores were not
Dr. Cook is associate professor of medicine and
director, Office of Education Research, College of
Medicine, Mayo Clinic, Rochester, Minnesota.
Dr. Beckman is associate professor of medicine,
College of Medicine, Mayo Clinic, Rochester,
Minnesota.
Dr. Thomas is assistant professor of medicine,
College of Medicine, Mayo Clinic, Rochester,
Minnesota.
Dr. Thompson is associate professor of medicine,
College of Medicine, Mayo Clinic, Rochester,
Minnesota.
Correspondence should be addressed to Dr. Cook,
Division of General Internal Medicine, Mayo Clinic
College of Medicine, Baldwin 4-A, 200 First Street
SW, Rochester, MN 55905; telephone: (507) 538-
0614; fax: (507) 284-5370; e-mail:
(cook.david33@mayo.edu).
Academic Medicine, Vol. 84, No. 11 / November 2009 1505
reported. Further research using the
IMMS seems warranted.
Evidence to support the validity of IMMS
scores is lacking. The validity of an
instrument’s scores can be supported by
evidence from five sources: content,
internal structure, relations to other
variables, response process, and
consequences.
10–12
Keller has provided a
detailed description of the development
of IMMS questions (content evidence),
and the internal consistency reliability
(internal structure evidence) is reportedly
excellent: ⱖ0.88 (Keller JM.
Development of two measures of learner
motivation: Florida State University,
2006; unpublished). A comprehensive
search of Medline, PsychInfo, and ERIC
revealed only one study evaluating the
validity of IMMS scores. In that study
13
of college students, factor analysis
suggested that 16 items could be
eliminated and that 11 of the remaining
20 loaded on the Attention dimension;
reliability information was not reported.
The studies using the IMMS Korean
translation reported internal consistency
reliabilities of 0.83
8
and 0.87
9
for the
entire instrument, but dimension-
specific information was not provided.
Additional evidence regarding the
validity of IMMS scores is needed.
We carried out the present study to
evaluate the validity of IMMS scores and,
as part of that evaluation, to measure
motivational differences between two
Web-based designs for teaching internal
medicine residents about ambulatory
medicine. We collected validity evidence
of internal structure (internal consistency
reliability and interdimension score
correlations) and relations to other
variables (discrimination of instructional
designs, and associations between IMMS
scores and knowledge scores, time spent
learning, and global course ratings). We
anticipated that higher motivation scores
would be associated with higher
knowledge scores (improved learning),
longer time on task (more motivated to
invest time), and higher course ratings.
Our study extends the findings of a
randomized trial comparing adaptive
and nonadaptive Web-based learning
designs.
14
In the adaptive design
(described in detail below), learners were
allowed to skip a section of the Web-
based course if they correctly answered a
multiple-choice question preceding that
section. We hypothesized that the
adaptive design would be more
motivating because the faster pace and
enhanced learner control would
command more attention and engender
greater satisfaction, and information
would seem more relevant when
presented in response to incorrectly
answered questions.
Method
Participants
The eligible participants for the present
study, conducted in the 2005–2006
academic year, were the 124 internal
medicine residents at the Mayo School
of Graduate Medical Education in
Rochester, Minnesota, who were enrolled
in a previously reported
14
randomized
trial. The Mayo institutional review
board deemed the study exempt, and all
participants consented to participate.
Interventions
The instructional interventions were two
Web-based learning modules, one on
asthma and the other on depression, each
developed using two instructional
designs.
14
In the standard design, the
module began with a patient scenario and
a multiple-choice question. After
responding to the question, the learner
was given the correct answer, a brief
rationale, and then detailed information
(including tables and figures, as
appropriate) on the topic. After reading
the detailed information, the learner
proceeded to the next question or patient
scenario (there was a total of five or
six scenarios, each with two to five
questions). The second design, an
adaptive design, was identical to the
standard design if the learner responded
incorrectly to a given question. However,
if the learner responded correctly, then
only the correct answer and rationale
were presented; the detailed information
associated with that question could be
skipped or reviewed (accessed via a Web
link) at the learner’s discretion.
Information was identical between the
two module designs. Participants were
randomly assigned to complete the
asthma module using one design
(standard or adaptive) and to complete
the depression module using the other
design (a randomized, crossover study
design).
Instruments
The IMMS consists of 36 statements with
response options ranging from 1 (not
true) to 5 (very true). After correction for
negatively phrased items (i.e., reverse
scoring), higher scores indicate greater
motivation. The 12 Attention items focus
on how well a course’s content, writing
style, and organization capture and
maintain attention or help avoid
boredom. Nine Relevance items assess
how well the information links to the
learner’s prior knowledge and experience,
perceived needs, and potential future
applications. Nine Confidence items
address the material’s apparent difficulty
and how the course presentation provides
assurance that learning would be
successful. Six Satisfaction items assess
enjoyment during the course and
perceived accomplishment afterward.
The IMMS was administered
immediately after each module.
Knowledge was assessed following IMMS
administration using a multiple-choice
test (Cronbach alpha 0.68). A global
measure of satisfaction with the module
(i.e., global module satisfaction) was then
assessed using the question, “On a scale
from 1 (poor) to 6 (excellent), what is
your overall rating of this module?” Time
spent completing modules was calculated
from computer logs.
14
We also wished to
determine the association between
motivation and satisfaction with the
course as a whole (i.e., overall course
rating). We measured this outcome
approximately one month after the last
module, with a question, “On a scale
from 1 to 6, with 1 being awful and 6
being excellent, what is your overall
rating of this Web-based course?”
Data analysis
Each dimension’s subscores were
standardized, as suggested by Keller,
by dividing by the number of questions
to yield a score for each dimension
ranging from 1 to 5. Internal consistency
was calculated using Cronbach alpha
(each module considered separately).
Discrimination between IMMS
dimensions was determined by
calculating the correlation (Pearson r)
between IMMS subscores within each
module. Discrimination was also
evaluated by determining the degree to
which IMMS scores distinguished the
two module designs. This was done by
comparing average IMMS scores between
Education Issues
Academic Medicine, Vol. 84, No. 11 / November 20091506
the module designs using general linear
models, and by comparing the responses
of individuals between designs using the
intraclass correlation coefficient (ICC).
Convergence was evaluated by calculating
correlation (Pearson r) between IMMS
scores and global module satisfaction
scores. Finally, associations between
IMMS scores and knowledge scores
(percentage of responses correct), time
spent, and overall course ratings were
determined using mixed linear models.
Statistical analysis used SAS 9.1 (SAS
Institute, Cary, NC). All available data
were used in each analysis. All analyses
used a two-sided alpha of 0.05.
Results
Between March and August 2006, 79 of the
124 residents (a response rate of 64%)
completed the IMMS at least once. Twenty-
six of the 79 (33%) were women, and 29
(37%), 29 (37%), and 21 (26%) were in
their first, second, and third postgraduate
years, respectively. Of the 79 residents, 66
completed the IMMS after using standard
modules, 53 completed it after using
adaptive modules, and 40 completed it after
both modules.
Validity evidence
Internal consistency (Cronbach alpha)
was ⱖ0.75 for all IMMS dimensions,
suggesting adequate reliability (Table 1).
Cronbach alpha for all 36 items together
was 0.93 to 0.95. Deleting individual
items had little effect on alpha for
individual dimension or total scores
except for one item in the Relevance
dimension, “This lesson was not relevant
to my needs because I already knew most
of it.” Deleting this item improved the
Relevance alpha from 0.78 to 0.82 for
Standard and from 0.80 to 0.84 for
Adaptive. This item was one of 10
“reverse-scored” items (i.e., agreement
with this item indicated low motivation).
Alphas remained stable or dropped after
individually deleting the other reverse-
scored items, suggesting that reverse
scoring did not lead to poor item
performance.
Correlation coefficients between scores
from different IMMS dimensions (Table
2) are lower than the internal consistency
reliability (Cronbach alpha), indicating
some degree of discrimination between
motivation dimensions. However,
interdimension correlations are still fairly
high (rranging 0.4 to 0.8), particularly
for the Attention dimension correlated
with others, suggesting at least modest
overlap between dimensions.
Correlations between IMMS scores and
the global module satisfaction scores
(assessed at the end of each module)
ranged from 0.40 to 0.63 (Table 2).
We found positive associations between
IMMS total scores and dimension-
specific subscores and knowledge
scores, time spent, and overall course
ratings (assessed after a several-week
delay); see Table 3. These associations
were statistically significant for
regression of Attention and Relevance
scores with knowledge (P⫽.033 and
P⫽.01, respectively), and all IMMS scores
with overall course ratings (P⬍.001).
Comparison between designs
There were no significant differences in
mean IMMS scores between module
Table 1
Reliability and Differences Between Instructional Designs for Scores (From 79
Internal Medicine Residents) on the Instructional Materials Motivation Survey
(IMMS) and Global Ratings, Mayo School of Graduate Medical Education,
2005–2006
*
Standard (n ⴝ66) Adaptive (n ⴝ53)
Differences between
designs (n ⴝ40)
Scale Mean (SD) Alpha Mean (SD) Alpha
Mean difference
(95% CI) ICC
†
IMMS Attention 3.1 (0.6) .82 3.2 (0.6) .87 0.1 (⫺0.2, 0.3)
‡
0.41
.........................................................................................................................................................................................................
IMMS Relevance 3.2 (0.6) .78 3.2 (0.7) .80 0.05 (⫺0.2, 0.3)
‡
0.58
.........................................................................................................................................................................................................
IMMS Confidence 3.6 (0.6) .75 3.6 (0.6) .83 0.0 (⫺0.2, 0.3)
‡
0.56
.........................................................................................................................................................................................................
IMMS Satisfaction 2.5 (0.8) .89 2.6 (0.9) .90 0.15 (⫺0.2, 0.5)
‡
0.59
.........................................................................................................................................................................................................
IMMS total
(entire scale)
3.1 (0.5) .93 3.2 (0.6) .95 0.1 (⫺0.1, 0.3)
‡
0.51
.........................................................................................................................................................................................................
Global module
satisfaction
§
4.3 (0.9) — 4.2 (1.1) — ⫺0.15 (⫺0.5, 0.2)
‡
0.52
* The scale for all IMMS dimensions and the total score ranges from 1 to 5. Participating residents were asked to
complete the IMMS after two modules, and 79 completed the IMMS at least once: 66 following the standard
module, 53 following the adaptive module, and 40 after both modules. SD indicates standard deviation; CI,
confidence interval.
†
ICC indicates intraclass correlation coefficient comparing ratings for Standard versus Adaptive dimension.
‡
All P⬎.33 comparing instructional designs.
§
Global module satisfaction was assessed using a separate question, with responses ranging from 1 (poor) to 6
(excellent).
Table 2
Correlations of Scores (From 79 Internal Medicine Residents) Among
Instructional Materials Motivation Survey Dimensions and With a Global Rating,
Mayo School of Graduate Medical Education, 2005–2006
*
Scale Attention Relevance Confidence Satisfaction
Standard (n ⫽66)
.........................................................................................................................................................................................................
IMMS Relevance 0.73
.........................................................................................................................................................................................................
IMMS Confidence 0.70 0.68
.........................................................................................................................................................................................................
IMMS Satisfaction 0.69 0.64 0.40
.........................................................................................................................................................................................................
Global module satisfaction
†
0.50 0.47 0.54 0.53
Adaptive (n ⫽53)
.........................................................................................................................................................................................................
IMMS Relevance 0.80
.........................................................................................................................................................................................................
IMMS Confidence 0.69 0.62
.........................................................................................................................................................................................................
IMMS Satisfaction 0.73 0.75 0.60
.........................................................................................................................................................................................................
Global module satisfaction
†
0.50 0.63 0.45 0.40
* Values in this matrix represent correlations within individual modules. P⬍.001 for each coefficient. Participating
residents were asked to complete the IMMS after two modules, and 79 completed the IMMS at least once: 66
following the standard module, 53 following the adaptive module.
†
Global module satisfaction was assessed using a separate question, with responses ranging from 1 (poor) to 6
(excellent).
Education Issues
Academic Medicine, Vol. 84, No. 11 / November 2009 1507
designs (Table 1). The mean difference
ranged 0.0 to 0.15 points, indicating that
residents had not consistently perceived
one module design to be more
motivating than the other. However,
ICCs ranging 0.41 to 0.59 showed only
modest consistency from one design to
the next, suggesting that individual
residents provided somewhat different
IMMS scores for the two module designs
(i.e., evidence of discrimination, or
sensitivity to change).
Discussion
In this study, we sought evidence to
support the validity of IMMS scores for
making inferences about the motivational
aspects of instructional events.
Supportive validity evidence included
good to very good internal consistency
reliability, variability in individual
participants’ ratings for different
module designs, lower interdimension
correlation than internal consistency,
significant positive associations with
overall course ratings, and consistently
positive (albeit rarely statistically
significant) associations with
knowledge scores and time spent.
Conversely, IMMS scores were not
sensitive to variation in instructional
design overall, and at least one item in
the Relevance dimension seemed
inconsistent with other scale items.
Correlations with global module
satisfaction ratings were generally lower
than correlations with other IMMS
dimension scores. However, global
module satisfaction reflects not only
motivation but also other aspects of
instructional design, and it may make sense
to see higher correlations among the IMMS
dimensions (all of which focus on
motivation) than between the IMMS
dimensions and this global measure.
The failure to detect a difference in mean
IMMS scores between module designs
could reflect insensitivity of IMMS
scores, perceptual differences in what
constitutes motivational materials, or
similarly motivating module designs. We
did not intend the designs to illustrate
unusually large motivational differences
but, rather, to reflect the variation found
between instructional alternatives used in
real life. Modest ICC values could
indicate that individual residents
perceived and rated the designs
differently (i.e., individuals discriminated
between designs even though averages
did not). However, if, in fact, the modules
were similarly motivating, then the ICCs
would indicate poor test-retest
reproducibility. To address this concern,
future studies might compare modules on
different topics using identical instructional
designs, or module designs that vary more
than those in this study. Of course, if
between-design differences in research
settings exceed those commonly found in
actual educational practice, then differences
in scores will have limited application.
Research might also explore the use of
individual, rather than average, IMMS
scores in certain applications (such as
motivationally adaptive courses).
Internal consistency data in this study are
similar to those reported by Keller (Keller
JM. Development of two measures of
learner motivation: Florida State
University, 2006; unpublished) and those
found in a Korean translation of the
IMMS.
8,9
Further evidence of internal
structure might come from research on
the temporal stability or factor structure
of IMMS scores. Relations to other
variables evidence is available from
studies demonstrating that IMMS scores
are sensitive to instructional design
changes intended to highlight the ARCS
model.
15,16
Additional evidence might
come from studies correlating IMMS
scores with scores from other
instruments assessing motivation using,
for example, the multitrait–multimethod
matrix.
17
Content evidence has been
provided by the IMMS’s author in a
detailed description of item development
(Keller JM. Development of two
measures of learner motivation: Florida
State University, 2006; unpublished).
Evidence that assessing and acting on
IMMS scores can affect learning
outcomes would constitute evidence of
consequences, but no such studies were
found. Likewise, no response process
evidence was found.
In accord with our predictions,
participants who perceived the course as
more relevant and attention-getting had
higher knowledge scores. By contrast, the
associations between motivation and
time were not statistically significant.
Although these findings are preliminary,
they set the stage for further study of
motivation in medical education. IMMS
scores will be most useful when they
inform evidence-based course changes—
for example, through formative feedback
to instructors (after the course) or
through dynamic adaptations in response
to motivational needs of individual
learners (during the course). These, in
turn, will require research investigating
what instructional changes actually
enhance motivation, what motivational
enhancements improve learning
outcomes, and how educators can apply
these enhancements effectively and
efficiently. For example, studies could
investigate how to enhance the relevance
of material and whether that promotes
better learning. Alternatively, educators
might consider adapting Web-based
learning to motivational needs. A study
in high school students found that a
motivationally adaptive computer-
assisted learning design was more
effective than designs with fixed
motivational features.
15
To our
Table 3
Associations Between Instructional Materials Motivation Survey (IMMS) Scores
(From 79 Internal Medicine Residents) and Other Study Outcomes, Mayo School
of Graduate Medical Education, 2005–2006
*
Outcome
IMMS dimension
Attention:
b (SE) P
Relevance:
b (SE) P
Confidence:
b (SE) P
Satisfaction:
b (SE) P
IMMS total:
b (SE) P
Knowledge
score
†
4.6 (2.1) .033 5.0 (1.9) .010 3.7 (2.2) .088 1.4 (1.3) .27 3.4 (2.4) .16
.........................................................................................................................................................................................................
Time
†
3.4 (5.0) .50 6.0 (4.5) .19 4.0 (5.2) .45 6.1 (3.4) .074 8.1 (5.8) .16
.........................................................................................................................................................................................................
Overall
course
rating
†
1.2 (0.2) ⬍.001 0.8 (0.2) ⬍.001 1.2 (0.2) ⬍.001 0.5 (0.1) ⬍.001 1.3 (0.2) ⬍.001
* Numbers in table represent slope (b) of the regression line indicating the average difference in outcome for a
one-unit change in motivation ratings. For example, on average, a one-point increase in Attention ratings would
correspond to a 4.6-point improvement in knowledge scores. Motivation ratings are standardized to a five-point
scale in which higher ratings indicate higher motivation. The IMMS total score is an unweighted average of the
four individual dimension scores.
†
Units are as follows: knowledge score (percent correct); time (minutes); course rating (1 ⫽awful, 6 ⫽excellent).
Education Issues
Academic Medicine, Vol. 84, No. 11 / November 20091508
knowledge, no such studies have been
done in medical education.
This study has limitations. First, the
validity evidence presented is restricted to
internal consistency and relations to
other variables. Future studies might
explore associations with scores from
other instruments measuring motivation.
Second, the sample size precluded factor
analysis. Third, we conducted multiple
independent hypothesis tests. A
conservative threshold for statistical
significance using Bonferroni adjustment
would be alpha ⫽0.003. Finally, score
validity is context-specific, and thus these
results may not apply to other learners or
to non-Web-based learning
environments.
In conclusion, this study presents the first
validity evidence regarding IMMS scores
in medical education. Though limited,
these data are generally supportive and
suggest that cautious use and further
study of this instrument’s scores are
warranted.
Acknowledgments
Funding for this study was from the Mayo Clinic
College of Medicine Education Innovation
program.
References
1Maslow AH. A theory of human motivation.
Psychol Rev. 1943;50:370–396.
2Bandura A. Self-efficacy. In: Ramachaudran
VS, ed. Encyclopedia of Human Behavior.
Vol 4. New York, NY: Academic Press; 1994:
71–81.
3Gagne´ RM. The Conditions of Learning and
Theory of Instruction. 4th ed. New York, NY:
Holt, Rinehart and Winston; 1985.
4Biggs JB. Good learning: What is it? How can
it be fostered? In: Biggs JB, ed. Teaching for
Learning: The View From Cognitive
Psychology. Hawthorn, Australia: The
Australian Council for Educational Research;
1991.
5Keller JM. Motivation and instructional
design: A theoretical perspective. J Instr Dev.
1979;2(4):26–34.
6Astleitner H, Keller JM. A model for
motivationally adaptive computer-assisted
instruction. J Res Comput Educ. 1995;27:
270–280.
7Keller JM. Development and use of the ARCS
model of instructional design. J Instr Dev.
1987;10(3):2–10.
8Jang KS, Park OJ, Hong MS, et al. A study on
development of Web-based learning program
with multimedia ECG monitoring and its
application. J Korean Soc Med Inform. 2003;
9:101–110.
9Jang KS, Hwang SY, Park SJ, Kim YM, Kim
MJ. Effects of a Web-based teaching method
on undergraduate nursing students’ learning
of electrocardiography. J Nurs Educ. 2005;44:
35–39.
10 Messick S. Validity. In: Linn RL, ed.
Educational Measurement. 3rd ed. New
York, NY: American Council on Education/
Macmillan; 1989.
11 American Educational Research Association;
American Psychological Association;
National Council on Measurement in
Education. Standards for Educational and
Psychological Testing. Washington, DC:
American Educational Research Association;
1999.
12 Cook DA, Beckman TJ. Current concepts in
validity and reliability for psychometric
instruments: Theory and application. Am J
Med. 2006;119:166.e7–e16.
13 Huang W, Huang W, Diefes-Dux H, Imbrie
PK. A preliminary validation of attention,
relevance, confidence and satisfaction
model-based instructional material
motivational survey in a computer-based
tutorial setting. Br J Educ Technol. 2006;37:
243–259.
14 Cook DA, Beckman TJ, Thomas KG,
Thompson WG. Adapting Web-based
instruction to residents’ knowledge
improves learning efficiency: A randomized
controlled trial. J Gen Intern Med. 2008;23:
985–990.
15 Song SH, Keller JM. Effectiveness of
motivationally adaptive computer-assisted
instruction on the dynamic aspects of
motivation. Educ Technol Res Dev. 2001;
49(2):5–22.
16 Gabrielle DM. The Effects of Technology-
Mediated Instructional Strategies on
Motivation, Performance, and Self-Directed
Learning [dissertation]. Tallahassee, Florida:
Florida State University; 2003. Available at:
(http://etd.lib.fsu.edu/theses/available/etd-
11142003-171019). Accessed July 20, 2009.
17 Campbell DT, Fiske DW. Convergent and
discriminant validation by the multitrait–
multimethod matrix. Psychol Bull. 1959;56:
81–105.
Education Issues
Academic Medicine, Vol. 84, No. 11 / November 2009 1509
