Educational Research: Time to Reach the Bar, Not Lower It

Departmrnt of Emergency Medicine, Harbor-UCLA Medical Center, Torrance, CA, USA.
Academic Emergency Medicine (Impact Factor: 2.01). 04/2005; 12(3):247-8. DOI: 10.1197/j.aem.2004.12.007
Source: PubMed


Available from: Roger J Lewis, Nov 03, 2014
Educational Research: Time to Reach the Bar, Not
Lower It
I believe that emergency physicians are among the best
clinical educators in academic medicine and that
a tradition of outstanding clinical education is one of
the defining characteristics of our field. Being a great
educator, of course, requires knowing not only what to
teach but how to teach. When accepted educa tional
methods are systematically studied, however, we
sometimes find that they are either less effective than
we believed or that alternative methods are more
effective. For example, it is now well documented
that the traditional didactic lecture on a clinical topic
is ineffective for changing postresidency physicians’
clinical beha vior.
Defining the most effective meth-
ods for teaching resident physicians is increasingly
important as we attempt to fit a larger and larger
quantity of medical, scientific, social, and administra-
tive content into residency training. Moreover, the
definition of six areas of competency to be achieved
by residency graduates in all fields by graduation does
little to define practical and effective methods for
teaching this material or measuring success.
Our tradition of clinical and educational excellence
notwithstanding, the field of emergency medicine is
still striving to define and distinguish itself in the realm
of research. While tremendous gains have been made
in this area, there is still much to be done if we are to
gain an equal footing with the traditional core disci-
plines within academic medicine. Regardless of the
quality of our clinical activities, the currency within
academic medical centers that most reliably garners
respect is research productivity measured by extramu-
ral funding. This is unlikely to change in the foresee-
able future.
These two factors, our tradition of excellence in
clinical education and our need, as a field, to increase
the quality and impact of our research activities, result
in a key opportunity for our field. Emergency medicine
is poised to be the academic specialty at the cutting
edge of research into educational methodology—to
determine the best methods for training physicians to
be effective health care providers in the 21st century.
While we may be poised, are we ready to take the lead?
Performing high-quality educational research is
hard. The basic rules of educational research are no
different than for clinical research—one must have
a clearly defined and important hypothesis to be tested
that is answerable using reliable and valid measures on
a suitable population to which one has access. Care
must be taken to ensure that the study design mini-
mizes bias and, when bias is unavoidable, one must
include provisions to estim ate its direction and mag-
nitude. Following these rules while conducting clinical
research is difficult; while performing educational
research it is often much harder. Moreover, establish-
ing a career based on achievement in educational
research requires not one or two studies but a pattern
of investigation that demonstrates a growth in the
sophistication of the hypotheses being tested and of the
investigator. Establishing such a pattern requires
long-term commitment, both by the investigator and
on the part of their department and institution.
Consider, for example , the problem of defining the
study population in educational research. There are
really two populations: the students or residents being
taught and the faculty or other personnel doing the
teaching. In order to obtain results that are reliable and
externally valid, it is important that both of these
populations be large enough and representative of
residents and educators at many different institutions.
It is virtually impossible to obtain large and diverse
enough samples at a single institution. An emergency
medicine residency training program may have 20–40
residents and 10–30 faculty. These numbers are really
too small to form the basis for a clinical trial. Thus, high-
quality educational research must virtually always be
It is often difficult to define the outcome of interest in
an educational study. Many of the skills we are trying
to teach are complex and multidimensional and in-
volve the ability of the resident to apply knowledge to
a varied and diverse set of patient care problems. Once
the skill or knowledge to be taught is defined, there still
remains the task of developing measures of the out-
come and demonstrating the reliability and validity of
those measures. Ironically, much of the educational
research currently published, across many fields, is
based on outcome measures that would never be
deemed reliable or valid enough to be used in an
educational testing setting. Consider, in contrast, the
extraordinary effort that is devoted to ensuring the
reliability and validity of American Board of Emer-
gency Medicine examinations.
Given the obvious importance of obtaining reliable
answers to fundamental questions regarding best
methods of teaching emergency medicine to our
residency trainees, how can we overcome the chal-
lenges associated with performing high-quality edu-
cational research? First, we must acknowledge that
research examining educati onal methodologies is an
important, fundamental, and necessary part of the
March 2005, Vol. 12, No. 3
d 247
Page 1
career of emergency medicine educators. Any com-
mitment to educational research must be backed up,
however, with protected time abov e and beyond that
required to simply administer educational and resi-
dency programs to allow the medical educator the
time to acquire research skills, set up multi-institu-
tional collaborations, complete the research, and
publish the findings. Second, we must raise our
expectations regarding the domain expertise of our
medical educators. While I believe emergency medi-
cine sets, in general, very high standards regarding
clinical knowledge and general teaching skills, we
must also expect our educators to have an under-
standing of modern educational principles and edu-
cational research methodology, including principles of
psychometric measurement (questionnaire and test
development, measuring internal validity, and so on).
Third, we must ensure that those devoting the time
and effort necessary to produce high-quality educa-
tional research receive the professional status and
recognition they deserve. Specifically, educational
research must be viewed as equally valuable as bas ic
scientific and clinical research in the eyes of academic
promotions committees, chairs, and deans. It would
be shortsighted to acknowledge the contributions of
those who generate the basic scientific and clinical
results that form the basis of emergency medical care
while devaluing the contributions of those who
ensure these advances are incorporated into the
practice arma mentarium of the next generation of
physicians. Lastly, and perhaps most controversially,
we should gradually reduce and eventually eliminate
the publication in peer-reviewed journ als of educa-
tional research studies that fail to meet the quality
standards required of clinical research studies. Exam-
ples of such publications include small, single-center
studies in which the number of educators is so small
that the results are not generalizable to any other
setting. (Consider the generalizability of a study in
which all instruction is provided by two or three
people.) Simil arly, educational concept papers pub-
lished in the peer-review literature should be required
to include data that demonstrate the effectiveness of
the educational techniques being discussed. In the
basic sciences and clinical sciences, opinion and
concept pieces are usually written by those with long
investigative careers who include summaries of results
from multiple primary studies to support their anal-
ysis or insights. Currently, many educational concept
papers are not required to meet this standard for
supporting evidence. While I understand this last
recommendation will be offensive to some, I believe
respect for educational research will only be ensured
by raising the bar to a level equivalent to that for other
types of research (hence the title of this commentary).
There are great opportunities in educational re-
search. There are opportunities to improve the effec-
tiveness of resident and medical student education,
both within emergency medicine and across all clinical
disciplines; opportunities to improve the quality of
clinical practice of the next generation of emergency
physicians; and opportunities to set the standard
for investigations into a fu ndamental part of what
we do as academic emergency physicians: teach. It is
time to reach the bar.—
RogerJ.Lewis,MD,PhD(roger@, Department of Emergency Medicine,
Harbor-UCLA Medical Center, Torrance, CA
1. Davis DA, Thomson MA, Oxman AD, Haynes RB. Changing
physician performance. A systematic review of the effect of
continuing medical education strategies. JAMA. 1995; 274:
2. Davis D, O’Brien MAT, Freemantle N, Wolf FM, Mazmanian P,
Taylor-Vaisey A. Impact of formal continuing education. Do
conferences, workshops, rounds, and other traditional
continuing education activities change physician behavior
or health care outcomes? JAMA. 1999; 282:867–74.
3. Jouriles NJ, Emerman CL, Cydulka RK. Direct observation for
assessing emergency medicine core competencies: interpersonal
skills. Acad Emerg Med. 2002; 9:1338–41.
4. Legome E, Pancu D, Nadel E, Manko J. A novel approach to
resident evaluation and the core competencies [letter]. Acad
Emerg Med. 2003; 10:97.
5. American Educational Research Association. Standards for
Educational and Psychological Testing, 1999. Washington, DC:
American Educational Research Association, Feb 2000.
6. Bianchi LC, Gallagher EJ, Korte RC, Ham HP. Interexaminer
agreement on the American Board of Emergency Medicine oral
certification examination. Ann Emerg Med. 2003; 41:859–64.
248 Lewis
Page 2
  • Source
    Preview · Article · Jan 2009 · Academic Emergency Medicine
  • Source
    [Show abstract] [Hide abstract] ABSTRACT: Education research in emergency medicine has made some advances, but still suffers from poorly designed studies and isolated projects that are small and cannot be generalized to other institutions. This commentary argues for the need of an emergency medicine education research group (EMERG). EMERG would facilitate and coordinate better quality educational research projects at multiple institutions. This in turn would promote faculty development in education research and potentially result in improved educational outcomes and patient care. ACADEMIC EMERGENCY MEDICINE 2010; 17:S11–S12 © 2010 by the Society for Academic Emergency Medicine
    Full-text · Article · Oct 2010 · Academic Emergency Medicine
  • Source
    [Show abstract] [Hide abstract] ABSTRACT: In 2010 the Council of Emergency Medicine Residency Directors (CORD) established an Academy for Scholarship in Education in Emergency Medicine to define, promote, recognize, and reward excellence in education, education research, and education leadership in emergency medicine. In this article we describe the mission and aims of the Academy. Academies for medical educators are widespread in medical schools today and have produced many benefits both for faculty and for educational programs. Little effort, however, has been devoted to such a model in graduate medical education specialty societies. While CORD and other emergency medicine organizations have developed numerous initiatives to advance excellence in education, we believe that this effort will be accelerated if housed in the form of an Academy that emphasizes scholarship in teaching and other education activities. The CORD Academy for Scholarship in Education in Emergency Medicine is a new model for promoting excellence in education in graduate medical education specialty societies.
    Preview · Article · Oct 2010 · Academic Emergency Medicine
Show more