A 2-year study of Gram stain competency assessment in 40 clinical laboratories.

Abstract

We used a computer-based competency assessment tool for Gram stain interpretation to assess the performance of 278 laboratory staff from 40 laboratories on 40 multiple-choice questions. We report test reliability, mean scores, median, item difficulty, discrimination, and analysis of the highest- and lowest-scoring questions. The questions were reliable (KR-20 coefficient, 0.80). Overall mean score was 88% (range, 63%-98%). When categorized by cell type, the means were host cells, 93%; other cells (eg, yeast), 92%; gram-positive, 90%; and gram-negative, 88%. When categorized by type of interpretation, the means were other (eg, underdecolorization), 92%; identify by structure (eg, bacterial morphologic features), 91%; and identify by name (eg, genus and species), 87%. Of the 6 highest-scoring questions (mean scores, > or = 99%) 5 were identify by structure and 1 was identify by name. Of the 6 lowest-scoring questions (mean scores, < 75%) 5 were gram-negative and 1 was host cells. By type of interpretation, 2 were identify by structure and 4 were identify by name. Computer-based Gram stain competency assessment examinations are reliable. Our analysis helps laboratories identify areas for continuing education in Gram stain interpretation and will direct future revisions of the tests.

Full text

Page 1

28
28
Am J Clin Pathol 2006;125:28-33
DOI: 10.1309/40WD3015CH1RYH58
© American Society for Clinical Pathology
Microbiology and Infectious Disease / GRAM STAIN COMPETENCY ASSESSMENT
A 2-Year Study of Gram Stain Competency Assessment
in 40 Clinical Laboratories
Nancy Goodyear, PhD,1Sara Kim, PhD,2,3Mary Reeves,1and Michael L. Astion, MD, PhD1
Key Words: Gram stain; Competency assessment; Computer-assisted instruction
DOI: 10.1309/40WD3015CH1RYH58
A b s t r a c t
We used a computer-based competency assessment
tool for Gram stain interpretation to assess the
performance of 278 laboratory staff from 40
laboratories on 40 multiple-choice questions. We report
test reliability, mean scores, median, item difficulty,
discrimination, and analysis of the highest- and lowest-
scoring questions. The questions were reliable (KR-20
coefficient, 0.80). Overall mean score was 88% (range,
63%-98%). When categorized by cell type, the means
were host cells, 93%; other cells (eg, yeast), 92%;
gram-positive, 90%; and gram-negative, 88%. When
categorized by type of interpretation, the means were
other (eg, underdecolorization), 92%; identify by
structure (eg, bacterial morphologic features), 91%;
and identify by name (eg, genus and species), 87%. Of
the 6 highest-scoring questions (mean scores, ≥99%) 5
were identify by structure and 1 was identify by name.
Of the 6 lowest-scoring questions (mean scores, <75%)
5 were gram-negative and 1 was host cells. By type of
interpretation, 2 were identify by structure and 4 were
identify by name. Computer-based Gram stain
competency assessment examinations are reliable. Our
analysis helps laboratories identify areas for continuing
education in Gram stain interpretation and will direct
future revisions of the tests.
Competency assessment (CA) is an integral component
of laboratory operations. CA results in improved laboratory
performance through identification of targets for improvement
of processes or training. CA also allows laboratories to pro-
vide documentation of laboratory quality for staff, manage-
ment, surveyors, and customers. In addition, CA is mandated
by the Clinical Laboratory Improvement Amendments
(CLIA) regulations and included in the College of American
Pathologists and Joint Commission on Accreditation of
Healthcare Organizations requirements.1-3CLIA regulations
specify that laboratories must assess competency by different
methods, including direct observation, review of test and qual-
ity control results, instrument records, and written testing.1
Gram stain CA is of particular importance because
Gram stains often are performed in urgent care settings
where patient care and treatment decisions are made based
on the results. Gram stain misinterpretation has been report-
ed widely.4-8Yuan et al9recently demonstrated that Gram
stain misinterpretation can have significant clinical impact.
Gram stains other than urethral or endocervical are classi-
fied as high complexity under CLIA.10The Department of
Laboratory Medicine, University of Washington, Seattle,
has been developing laboratory tutorials and computer-
based CA tools for the last 10 years.11-14Computer-based
CA is an objective method that functions effectively as a
supplement to other methods of CA. This work is a logical
extension of our previous work. In this study, we detail the
performance of 278 laboratory staff from 40 facilities on the
Web-based Gram Stain CA examination. We identify and
discuss the CA questions that were most and least problem-
atic for the laboratory staff studied. Overall, the study
demonstrates successful wide-scale implementation of

Page 2

Am J Clin Pathol 2006;125:28-33 29
DOI: 10.1309/40WD3015CH1RYH58
29 29
© American Society for Clinical Pathology
Microbiology and Infectious Disease / ORIGINAL ARTICLE
computer-based CA and points out key areas that US labo-
ratory staff found to be problematic.
Materials and Methods
Description of the CA System
The University of Washington Department of Laboratory
Medicine and Medical Training Solutions (http://www.med-
training.org, Seattle, WA) have collaborated to produce an
online CA system comprising more than 30 different CA top-
ics to date. The minimum system requirements are Microsoft
Internet Explorer, version 6.0 (Microsoft, Redmond, WA);
Windows Media Player 6.4 (Microsoft); Microsoft Windows
98 (Microsoft); a Pentium-class processor (Intel, Santa Clara,
CA); and a 16-bit color display with resolution of 800 × 600.
A sound card and speakers are recommended but not required,
and a printer is necessary if paper copies of examination
results are desired.15Administrative features of the system
include the ability to assign examinations to individual
employees and to send automated e-mail reminders, including
direct links to the examinations, to employees. Examinations
are scored automatically, and individual or laboratory results
are available immediately for review or printing.
Each examination consists of 10 multiple-choice ques-
tions, written and reviewed by content experts and educators
in the Department of Laboratory Medicine. Each question is
accompanied by an image, which is integral to answering the
question. ❚Image 1❚ shows an example of a question from the
Gram Stain examination. After answering the question, the
examination taker is shown the correct answer, with an
explanation of the question. There are 2 examination periods
for each year (January 1–June 30 and July 1–December 31).
New examination questions are written for each testing period.
Subjects
Subjects included laboratory staff from facilities in the
United States who completed 4 consecutive Gram stain exam-
inations administered in 2003 and 2004. We applied 2 criteria
for including test scores in the analysis: (1) Only users who
took all 4 tests were included. (2) Data from clinical laborato-
ries were included in the study. We obtained approval from the
University of Washington Institutional Review Board to con-
duct the study.
Test Items
We combined 10 items from each of the 4 Gram stain
tests and based our analysis on a total of 40 items. We ana-
lyzed the items using 2 different categorizations. The first set
of categories were based on cell type and included the follow-
ing: gram-positive, 12 questions; gram-negative, 15 questions;
host cells, including WBCs, RBCs, squamous epithelial cells,
and columnar epithelial cells, 11 questions; and other cells,
including yeast and no organisms seen, 10 questions. Some
questions fit more than 1 category, eg, a question may have
asked about WBCs and gram-positive cocci in an image.
The second categorization was based on the type of inter-
pretation required and included the following: identify by
structure, in which the test taker was asked to identify cell
types, Gram reaction, or bacterial morphologic features, 22
questions; identify by name, in which the test taker was asked
to correlate the image to organism identification by genus or
genus and species, 15 questions; and other, which included
❚Image 1❚ Screen capture from the Gram Stain Competency examination showing a typical explanation screen for an image-
based multiple-choice question.

Page 3

30
30
Am J Clin Pathol 2006;125:28-33
DOI: 10.1309/40WD3015CH1RYH58
© American Society for Clinical Pathology
Goodyear et al / GRAM STAIN COMPETENCY ASSESSMENT
questions about overdecolorization or underdecolorization, 3
questions. The identify by structure category requires only 1
cognitive step, that of identifying the structures in the image.
Questions in the identify by name category require an addi-
tional step, that of matching the structure(s) identified in the
image with organism names. This requires consideration of
clinical significance, knowledge of the Gram reaction and
morphologic features for many organisms, and distinguishing
subtle differences in morphologic features. Each question was
classified into only 1 of the 3 categories.
Data Analysis
Scores recorded on a server were transferred to Microsoft
Excel (Microsoft) and SPSS version 10.0.5 (SPSS, Chicago,
IL) for analysis. We conducted the following data analyses:
KR-20 Reliability
We calculated the reliability of the test to determine its
internal consistency based on dichotomous test scores (1 for
correct; 0 for incorrect).
Descriptive Statistics
We calculated the mean and median scores for the over-
all test and for the cell and interpretation types covered in
the test.
Item Difficulty and Discrimination
To assess the quality of individual items and the test as a
whole, we performed item difficulty and item discrimination
analyses. For each test item, we calculated the percentage of
users who answered the items correctly to determine the level
of difficulty of the items. We used the data to identify easy
items (defined as >95% of the users answering correctly) or
difficult (<75% of the users answering correctly). Then we
derived an item discrimination based on correlations between
a user’s total score on the overall test and that user’s score on
an item. The discrimination index is used to determine
whether an item appropriately differentiates the ability of
high- and low-performing users. We applied the following
standard to judge the discriminatory quality of the items:
good, more than 0.30; fair, 0.10 to 0.30; and poor, less than
0.10.16An ideal question would include a correct answer with
a strong positive correlation and distracter items with negative
correlations. In other words, users who attained a high score
on the overall test would tend to choose the correct answer
and, therefore, not select distracter items.
Subset Analysis
We present 6 questions that were answered correctly by
99% or more of the users, and we highlight in detail 6 items
that were answered correctly by fewer than 75% of the users
to illustrate why these items were challenging to them.
Results and Discussion
Scores of 278 users from 40 laboratories were analyzed.
The number of users per laboratory ranged between 1 and 25.
The average overall score on the combined 40-question exam-
ination was 88% (range, 63%-98%).
Reliability
We obtained a KR-20 coefficient of 0.80 (out of 1.00),
which suggests that test items are dependable for assessing a
user’s mastery of Gram stain content.
Descriptive Statistics
We calculated the mean and median scores and ranges
based on users’scores on 40 test items. ❚Table 1❚ and ❚Table 2❚
present the percentage of correct answers selected by 278
users, based on categorization by cell type and type of inter-
pretation, respectively. Overall, users correctly identified
approximately 90% of the items. When questions are catego-
rized by cell type (Table 1), users attained the highest scores
in host cells (93%) followed by other cells (92%), gram-posi-
tive cells (90%), and gram-negative cells (88%). When cate-
gorized by type of interpretation (Table 2), users attained the
highest scores in the other category (92%), followed by iden-
tify by structure (91%), and identify by name (87%).
Item Difficulty and Discrimination
Of 40 items, 6 were answered correctly by 99% or more
of users, suggesting that these items were easy. Examination
of the item discrimination index for these items revealed the
level of discrimination was poor (<0.10). When categorized
by cell type, these questions were spread across all 4 cate-
gories: host cells, 3; gram-negative, 3; other cells, 2; and
gram-positive, 1 (note that 3 questions were classified into
more than 1 category). When categorized by type of interpre-
tation, only 1 question was in the identify by name category, 5
were in the identify by structure category, and none were in
the other category.
There were 6 items that were answered correctly by
fewer than 75% of the users, suggesting that these items were
difficult. All of these items showed fair to good discrimina-
tion levels (>0.10). Five items were in the gram-negative cat-
egory (range, 67%-74%), and 1 was in the host cells catego-
ry. When classified by type of interpretation, 2 of the difficult
items were classified as identify by structure and 4 as identi-
fy by name.
We found item difficulty and discrimination to be useful
tools for assessing the quality of the examinations. Based on
these study results, items with poor discrimination (≤0.10)
will be reviewed when new questions are written. Although
all questions are retired at the end of each examination peri-
od, some new questions will be similar in style and content to

Page 4

Am J Clin Pathol 2006;125:28-33 31
DOI: 10.1309/40WD3015CH1RYH58
3131
© American Society for Clinical Pathology
Microbiology and Infectious Disease / ORIGINAL ARTICLE
previous questions. Items with poor discrimination will be
avoided or revised to improve discrimination.
Subset Analysis
Of the 6 questions with mean scores of 99% or more, the
only one that fell in the identify by name category asked the
test taker to identify Neisseria gonorrhoeae from a classic
image with many WBCs, one of which is packed with intra-
cellular gram-negative diplococci (Image 1). Four questions
involved recognizing WBCs, epithelial cells, budding yeast,
gram-negative rods, and gram-positive cocci. One question
involved differentiating background material from true bacte-
ria (“no organisms seen”).
The 6 questions with mean scores of 75% or less are
shown in ❚Table 3❚, with the percentage of users who selected
each possible answer and the item discrimination indices for
each item. The discrimination indices for all correct answers
ranged between +0.267 and +0.521 and for all distracter
items, between –0.084 and –0.395. Therefore, these 6 ques-
tions are judged to be of good quality even if they were diffi-
cult for the users. All 6 questions, accompanied by images, are
available at www.medtraining.org/Gramstudy.
Questions 3, 10, and 22 involved variations in the mor-
phologic features of gram-negative rods or diplococci. For
question 3, 18% of users selected Clostridium species instead
of the correct answer, Klebsiella species. Although at first
glance this seems an unlikely error, Clostridium species are
well known to overdecolorize easily, and the specimen source
was a biliary abscess, which is consistent with an anaerobic
infection. Question 10 involved a different sort of distinction.
The correct answer was Moraxella catarrhalis, a gram-nega-
tive diplococcus, but 13% of users answered Haemophilus
influenzae, a tiny gram-negative coccobacillus. An additional
10% answered “no infection present,” which may reflect the
infrequency of M catarrhalis as a pathogen in their institution.
Only 1 WBC was present in the image, which may have led
them to believe the organism was most likely an upper respi-
ratory tract contaminant such as a saprophytic Neisseria
species. The subtle morphologic distinction in question 22
involved Klebsiella pneumoniae appearing as short, wide
rods, several of which exhibit bipolar staining. The bipolar
staining rods were thought to be Neisseria species by 11% of
users and Pseudomonas aeruginosa, usually longer and thin-
ner, by 15% of users.
The difficulty in correctly answering questions 6 and 9
seems to stem from confusion about the appearance of
Fusobacterium nucleatum and the term fusiform. This is an
uncommon isolate, although it is quite distinctive in appear-
ance. In question 6, 30% of users answered F nucleatum
instead of the correct answer, Serratia marcescens. The rods
in this image are much shorter and wider than true
Fusobacterium species. It is possible that users were misled
by background material. In question 9, 16% of users described
the appearance of the gram-negative rods as long and narrow,
and 13% described them as curved. Neither answer is com-
pletely false; however, fusiform also incorporates the needle-
like appearance of the rods.
Question 24 asked users to distinguish columnar epithe-
lial cells from squamous epithelial cells. One other question in
the sample set required the same differentiation (question not
❚Table 1❚
Ability of 278 Users to Identify Cell Types in a 40-Question Examination*
Cell TypeMean (%) Median (%) Minimum Score†(%)Maximum Score‡(%)
Host cells (n = 11)
Gram-positive (n = 12)
Gram-negative (n = 15)
Other (n = 10)
93
90
88
92
91
92
93
90
64
33
40
30
100
100
100
100
*The 40 questions sort into 48 items when categorized by cell type because some questions fit more than 1 category.
†The lowest individual score for the 278 users.
‡The highest individual score for the 278 users.
❚Table 2❚
Ability of 278 Users From 40 Laboratories to Perform Different Types of Gram Stain Interpretations*
Type of Interpretation Mean (%) Median (%)Minimum Score†(%)Maximum Score‡(%)
Identify by name (n = 15)
Identify by structure (n = 22)
Other (n = 3)
87
91
92
93
91
100
33
59
0
100
100
100
*Each of the 40 questions corresponded to 1 type of interpretation.
†The lowest individual score for the 278 users.
‡The highest individual score for the 278 users.

Page 5

32
32
Am J Clin Pathol 2006;125:28-33
DOI: 10.1309/40WD3015CH1RYH58
© American Society for Clinical Pathology
Goodyear et al / GRAM STAIN COMPETENCY ASSESSMENT
shown); the mean score for this question was 91%. In this
question, the columnar epithelial cells were more classic in
appearance than in question 24, very oblong, and included vis-
ible villi. In question 24, the cells are more rounded; however,
they can be distinguished from squamous epithelial cells by
their size relative to several WBCs in the field and in the large
size of their nuclei relative to the cytoplasmic volume. It also
is possible that many subscribing laboratories do not distin-
guish between columnar and squamous epithelial cells in
reporting routine Gram stains; therefore, their employees lack
experience in this area.
Conclusion
We successfully implemented a computer-based CA tool
in 40 laboratories. In this study, we looked at the performance
of 278 laboratory staff who took Gram stain competency
examinations in 4 examination periods during 2 years. Our
data show that the items on the Gram stain CA examination
are reliable overall. Questions with the poorest responses
involved distinguishing columnar and squamous epithelial
cells, identifying fusiform organisms, and distinguishing a
variety of gram-negative morphologic features.
Our program allows laboratories to determine their own
benchmarks or pass rates for each examination to accommo-
date differences in practice in different laboratories; however,
laboratory managers may consider interventions in the areas
identified as difficult. The ability to show only 1 microscopic
field is a limitation of this test system. Our ongoing analysis
of questions results in high-quality CA tools. Our future plans
include examining other CA test topics (eg, safety), perform-
ing a more extensive analysis involving a larger pool of test
questions and subjects, and continuing quality improvement
of the CA tests based on these results.
From the Departments of 1Laboratory Medicine, 2Medical
Education and Biomedical Informatics, and 3Family Medicine,
University of Washington, Seattle.
The University of Washington receives licensing revenue, in
relation to the competency assessment Web site, from Medical
Training Solutions.
❚Table 3❚
Questions Correctly Answered by Fewer Than 75% of Users
Image*DescriptionQuestion and Multiple Choice Options†
Users Selecting Option (%)Item Discrimination
WBCs; few GNRs3. This directly stained biliary abscess fluid specimen suggests
an infection caused by which of the following?
a. Clostridium spp
b. Haemophilus spp
c. Neisseria spp
d. Klebsiella spp
6. This direct Gram stain is from an infected surgical wound.
Which of the following is the most likely identification?
a. Coagulase-negative staphylococci
b. Fusobacterium nucleatum
c. Serratia marcescens
d. Legionella spp
9. This is a direct Gram stain of a peritonsillar abscess. Which
of the following best describes the organism seen?
a. Gram-negative rods, fusiform
b. Gram-negative rods, curved
c. Gram-negative rods, long, narrow
d. Gram-negative rods, coccobacilli
10. This direct Gram stain of sputum suggests an infection caused by:
a. Haemophilus influenzae
b. Klebsiella pneumoniae
c. Moraxella catarrhalis
d. No infection present
22. This direct Gram stain from a urine specimen suggests infection with:
a. Haemophilus influenzae
b. Klebsiella pneumoniae
c. Neisseria spp
d. Pseudomonas aeruginosa
24. Evaluate this direct Gram stain from a bronchial wash specimen.
a. WBCs, columnar epithelial cells
b. WBCs, epithelial cells, GPC
c. WBCs, gram-positive cocci, GNRs
d. WBCs, squamous epithelial cells
18
9
1
72
–0.395
–0.238
–0.210
0.521
WBCs; few GNRs; moderate
pink background
2 –0.236
–0.247
0.368
–0.230
30
67
1
Long, thin GNRs, some
curved; 2 WBCs
70
13
16
1
0.514
–0.358
–0.306
–0.160
1 WBC; many GNDC; streaky
background13
3
74
10
–0.380
–0.284
0.454
–0.084
7 WBCs; GNRs, short, some
bipolar staining2 –0.266
0.474
–0.331
–0.206
72
11
15
WBCs; columnar epithelial
cells; no organisms70
3
1
26
0.267
–0.314
–0.298
–0.141
GNDC, gram-negative diplococci; GNR, gram-negative rod; GPC, gram-positive cocci; spp, species; WBC, white blood cell.
*See www.medtraining.org/Gramstudy for questions and images as they appear in the Gram stain competency test.
†Correct answer in bold.