Effectiveness of Clinical Decision
Support in Controlling
C. Craig Blackmore, MD, MPH, Robert S. Mecklenburg, MD,
Gary S. Kaplan, MD
Background: Decision support systems for advanced imaging are being implemented with increased fre-
quency and are mandated under some new governmental health care initiatives. However, evidence of effec-
tiveness in reducing inappropriate imaging utilization is limited.
Methods: A retrospective cohort study was performed of the staged implementation of evidence-based
clinical decision support built into ordering systems for selected high-volume imaging procedures: lumbar
MRI, brain MRI, and sinus CT. Brain CT was included as a control. Imaging utilization rates (number of
patients imaged as a proportion of patients with selected clinical conditions) and overall imaging utilization
before and after the interventions were determined from billing data from a regional health plan and from the
institutional radiology information system.
Results: The use of imaging clinical decision support was associated with substantial decreases in the
utilization rate of lumbar MRI for low back pain (risk ratio, 0.77; 95% confidence interval, 0.87-0.67; P ?
.0001), head MRI for headache (risk ratio, 0.76; 95% confidence interval, 0.91-0.64; P ? .001), and sinus CT
volumes (all diagnoses) for lumbar MRI, head MRI, and sinus CT, with no observed effect for the head CT
Conclusion: Targeted use of imaging clinical decision support is associated with large decreases in the
inappropriate utilization of advanced imaging tests.
Key Words: Imaging utilization, appropriateness, computer decision support
J Am Coll Radiol 2011;8:19-25. Copyright © 2011 American College of Radiology
Health care expenses in the United States continue to
spiral upward, now representing more than 17% of the
gross domestic product . Imaging is one of the most
important contributors to health care costs, encompass-
ing more than 14% of Medicare Part B expenditures
[2-4]. Although identified as the most significant ad-
has become a target for cost containment. A major driver
for increasing imaging cost is the inappropriate utiliza-
tion of advanced imaging, including CT and MRI [4,6-
8]. Accordingly, health care providers are under increas-
ing pressure to limit imaging to evidence-based
Payers have initiated several approaches to control im-
aging utilization, including external authorization meth-
ods and clinical decision support systems . Clinical
decision support systems are point-of-order decision
aids, usually through computer order entry systems, that
tests, including information on test appropriateness for
specific indications. Such systems may be purely educa-
tional, or they may be restrictive in not allowing imaging
test ordering to proceed when accepted indications are
absent. Although data on the efficacy of imaging clinical
decision support systems are limited , adoption is
increasing and has spread to include state-level initiatives
in Washington  and Minnesota . Imaging clini-
cal decision support systems can range from simple aids
Center for Health Care Solutions, Virginia Mason Medical Center, Seattle,
Corresponding author and reprints: C. Craig Blackmore, MD, MPH, Vir-
ginia Mason Medical Center, Department of Radiology, Mailstop C5-XR,
1100 Ninth Avenue, PO Box 900, Seattle, WA 98111; e-mail:
© 2011 American College of Radiology
for small numbers of studies and indications to broad
systems encompassing the thousands of possible pairs of
indications and imaging procedures. To date, there are
no published studies demonstrating decreased imaging
sion support, though a decrease in the rate of growth of
utilization of imaging has been reported. We hypothe-
sized that imaging clinical decision support could de-
crease imaging utilization when targeted to select imag-
ing studies and indications that included high volumes
and high cost [13,14].
The objective of this investigation was to identify
changes in imaging utilization associated with the initia-
tion of an imaging management program based on clin-
ical decision support for selected CT and MRI studies at
a single integrated health care delivery system.
The overall study design was a retrospective cohort eval-
uation of the effect of the staged implementation of a
clinical decision support system on imaging utilization,
with historical and concurrent controls. The study was
granted a waiver from the institutional review board.
The study setting was Virginia Mason Medical Cen-
ter, an integrated multidisciplinary health care net-
work in the Pacific Northwest with approximately 450
physicians, 800,000 outpatient visits, 17,000 hospital
visits, and 260,000 radiology examinations annually.
The institution includes a central urban campus as
well as multiple suburban satellite imaging and outpa-
tient care centers.
Lumbar MRI, head MRI, and sinus CT were identified
as frequently performed, high-cost procedures with high
variability in utilization [2,14,15] and with at least some
medical evidence to guide appropriate utilization .
Accordingly, these procedures were targeted for the ini-
tial implementation of the decision support system,
rather than a more global approach. The intervention
was based on a set of locally derived evidence-based de-
cision rules for when imaging is appropriate. These deci-
sion rules were developed by Virginia Mason providers
from the involved specialties after review of national and
international evidence-based guidelines and primary lit-
erature and were vetted extensively in the institution
before implementation. The system was not designed to
be comprehensive but rather to focus on areas where
there was potential for improvement, which we defined
to enable guideline development.
eral assumptions: (1) that physician education alone is
insufficient to change practice, (2) that patient and pro-
vider expectations mandate that an alternative be offered
occur at the point of care, to avoid disrupting care.
The imaging intervention was a mandatory series of
that confirmed adherence to the institutional evidence-
based imaging indications (Figure 1). Providers ordering
studies were required to check appropriate boxes corre-
sponding to approved imaging indications. Failure to
document compliance with approved indications would
vention was systemwide but was limited to outpatient
aging clinical decision support intervention was accom-
panied by an institutional educational effort including
e-mails, small conferences, and personal communica-
tion. Additional periodic audits were performed with
but had not documented appropriate indications in the
medical record. The evidence-based imaging protocols
for MRI for low back pain and head MRI for headache
were implemented in 2005. The protocol for sinus CT
for suspected sinus disease was implemented in 2007.
Because of patient and provider expectations, alterna-
tives to imaging that might be beneficial to patients were
also offered, with information provided in the order en-
try system. For lumbar back pain, physical therapy was
offered, with availability of same-day or next-day con-
sultation with a (nonoperative) spine specialist. For
headache and sinus disease, prompt neurologist or
allergist consultation was available. The subspecialist
consultants were authorized to override the clinical
decision support system when they considered imag-
ing clinically indicated.
To determine the effectiveness of the intervention in
decreasing inappropriate imaging utilization, we used
International Classification of Disease, 9th ed., Clinical
Modification (ICD-9-CM) and Common Procedural
Terminology®(CPT®) codes to interrogate the data
records of a large regional health insurance carrier to
determine the rates of relevant imaging for patients with
specified diagnoses cared for in our system. Data were
available for January 1, 2003, through December 31,
2009. For each of the clinical conditions (low back pain,
of ICD-9-CM codes. For the patients with the clinical
scenarios defined by the codes, we used CPT codes to
back pain, the included ICD-9-CM codes were 344.6,
20 Journal of the American College of Radiology/Vol. 8 No. 1 January 2011
720, 721.3, 721.42, 721.5 to 721.9, 722.10, 722.32,
722.52, 722.73, 722.83, 722.93, 724.02, 724.2 to
724.9, 846, and 847.2 to 847.4. For lumbar MR, the
included CPT codes were 72148, 72149, and 72158,
encompassing all lumbar MR examinations. For head-
ache, the included ICD-9-CM codes were 307.81, 339,
346, and 784.0. The associated CT and MR codes were
70450, 70460, 70470, 70541, 70551, 70552, and
70553, encompassing all head MR and CT examina-
478.1. The sinus CT CPT code was 70486, which in-
cluded all CT sinus studies.
Total volumes of imaging were also determined from
the radiology information system (IDX Imagecast 10;
GE Healthcare, Fairfield, Connecticut) on the basis of
the CPT codes detailed above. These volumes are irre-
spective of payer.
Primary analysis was a comparison of the rate of imaging
in the years preceding the intervention with the rate of
imaging in the years after the intervention, for the single
commercial payer. For imaging rate, the numerator was
the total number of patients with a given clinical condi-
was used in the primary analysis to control for temporal
clinical condition. We assessed for significant change in
trends, using the likelihood ratio test to compare linear
regression models of rate as a function of year vs rate as a
function of year and intervention. Estimates of the abso-
lute magnitude in decrease in imaging rate after the in-
tervention were made by comparing the imaging rate in
rate in the years after the intervention, using ?2analysis.
For the magnitude analysis, the actual year of interven-
tion was excluded. Similar analysis was also performed
there was no substitution of head CT for head MRI after
the intervention. Because there was no intervention for
head CT, for the analysis, the intervention year for head
CT was considered to be 2005, the year of the head MR
Secondary analysis included the determination of
changes in trends and overall volumes of the specific
imaging studies associated with the intervention,
Fig 1. Sample imaging
clinical decision support
tool for low back pain and
Blackmore, Mecklenburg, Kaplan/Clinical Decision Support 21
throughout the network (for all purchasers and for all
diagnoses). Overall volumes were not adjusted for clini-
cal condition but provide an estimate of overall effect of
the intervention on health care utilization and cost. We
assessed for significant change in overall volume of imag-
ing studies after the intervention, adjusted for temporal
trends, using the likelihood ratio test to compare linear
regression models of volume as a function of year vs
volume before and after the intervention using linear
with a value of ?1.0 indicating decreased imaging after
the intervention. In addition, results are reported as a
percentage change (reduction) in imaging. Statistical
LP, College Station, Texas).
We found clinically and statistically significant de-
creases in utilization rates for the targeted procedures
after the intervention. Table 1 details the raw counts
of imaging procedures, as well as the counts of patients
with the corresponding diagnoses and the rate of im-
aging among affected individuals before and after the
intervention. The rates of imaging after the interven-
tion were 23.4% lower for low back pain lumbar MRI
.001), 23.2% lower for headache head MRI (RR, 0.76;
95% CI, 0.91-0.64; P ? .001), and 26.8% lower for
sinusitis sinus CT (RR, 0.73; 95% CI, 0.82-0.65; P ?
.001). The peak rate occurred in the year before the
decrease in imaging rate was significant in the multiple
regression analysis after adjustment for temporal trend
for lumbar MRI (P ? .001), head MRI (P ? .05), and
sinus CT (P ? .003), with a nonsignificant result for the
head CT control group (P ? .88).
After the intervention-associated decline, the rate of
MRI of the lumbar spine increased at approximately 3%
per year (RR, 1.003; 95% CI, 1.002-1.004; P ? .007),
MRI (RR, 1.000; 95% CI, 0.99-1.01, P ? .99). Postint-
Fig 2. Imaging rates vs time for patients with disease-
specific billing codes from a single regional payer. Ar-
rows indicate the year before the intervention.
Table 1. Imaging volume, patient volumes, and imaging rate
Lumbar MRI for low
Head MRI for
Sinus CT for sinusitis 285
Head CT for
Low back pain
Lumbar MRI rate?
Brain MRI rate?
Sinus CT rate?
Head CT rate?
149 165224171186 186191
Note: Data on patients from a single regional commercial payer. Numbers in boldface italics represent the year of intervention (no
intervention in the head CT control group).
?Rate is defined as the number of patients with a given procedure divided by the total number of patients with a specific clinical
22 Journal of the American College of Radiology/Vol. 8 No. 1 January 2011
ervention trend analysis for head MRI, head CT, and
lumbar MRI was limited by the small sample size
(4 years). Sinus CT could not be explored for trend after
For the head CT control group, we identified no
significant change in the rate of imaging (RR, 0.97;
95% CI, 1.21-0.78, P ? .37) after the head MRI
intervention (no head CT intervention was per-
formed). There was also no trend in head CT rate in
the years after the intervention (RR, 1.0; 95% CI,
0.99-1.01, P ?.96).
Secondary analysis revealed that the decision support
intervention was also associated with decreases in the
overall volumes of lumbar MRI, head MRI, and sinus
CT studies, regardless of diagnosis. For head MRI, the
the regression model (P ? .0001) after adjustment for
temporal volume trends and continued to decrease after
the intervention by 162 studies per year (95% CI, 88-
236; P ? .01). For lumbar MRI, adjusted volumes after
before the intervention, with no significant change in
subsequent years (estimated subsequent decrease, 34;
95% CI, decrease 279 to increase 210, P ? .60). For
sinus CT, there was a significant decrease in adjusted
volumes after the intervention (P ? .010), with insuffi-
cient data to assess for a further decrease. For the head
CT control group, there was no significant change in
overall volume associated with the time of the head MRI
intervention (P ? .52).
Clinical decision support is potentially an ideal method
for improving the evidence-based use of imaging. Clini-
cal decision support tools have the desired properties of
being educational, transparent, efficient, practical, and
ical decision support is limited. Prior investigation has
focused on the use of a global system encompassing vir-
tually all CT and MRI studies and indications and has
demonstrated only a relative attenuation in the rate of
increase in imaging utilization. However, in the prior
report, actual imaging utilization of both CT and MRI
continued to grow .
In this report, we detail a significant and sustained
decrease in the utilization of targeted advanced imag-
ing studies through the use of clinical decision support
based on a simple set of locally derived evidence-based
imaging guidelines. Our approach has several impor-
tant innovations from other reports of imaging clinical
decision support systems [9,10] that may have con-
tributed to success. We targeted areas of high and
potentially inappropriate utilization, concentrating
effort where there is potential for benefit rather than
globally applying computer decision support to all
higher imaging, as others have advocated [9,10]. Also,
we incorporated denial of imaging for inappropriate
indications, preventing orders that did not meet evi-
dence-based indications from proceeding in the com-
puter order entry system. Finally, we offered the pro-
vision of alternate resources, in the form of prompt
specialist consultation or therapy, where indicated.
The study setting likely had a substantial effect on the
success of the program. The intervention was performed
at Virginia Mason Medical Center, a multispecialty in-
tegrated health care network, with all providers being
salaried employees of the institution. Thus, financial in-
centives and risks were shared by the entire institution
and providers. Although the providers received no direct
financial incentive or avoidance of precertification, there
was pressure on the institution from local commercial
of imaging. The clinical decision support intervention,
coupled with rapid access to appropriate clinical care,
increased the quality and efficiency of providing care at
the institution, potentially providing overall benefit de-
spite decrease in radiology volumes. This overall institu-
tional benefit allowed radiology to participate in practice
improvements that may have resulted in decreased radi-
extent that financial incentives in the health care system
are based on volumes and reward inefficiency through
cial disadvantage as a result of providing better quality,
more evidence based care.
A second advantage to being a multispecialty net-
work is that most referrals for imaging were from
within the system, enhancing the ability to influence
physician ordering behavior. The elimination of un-
necessary imaging was defined by the institution as a
component of quality, motivating providers to sup-
port the mandatory clinical decision support program.
Also, the concept of evidence-based medicine had
wide penetration throughout our institution, with a
concordant high acceptance of evidence-based imag-
ing protocols. In addition, the institutional culture,
with a pervasive focus on efficiency and Lean health
care management methodology , provided a
framework to enable relatively rapid change.
There have been important challenges in the imple-
mentation of the imaging clinical decision support sys-
tem. Although built using evidence-based medicine
methodology, our protocols were often limited by the
availability of quality data and nationally accepted evi-
dence-based guidelines. Accordingly, global evidence
was applied locally through the work of institutional
Blackmore, Mecklenburg, Kaplan/Clinical Decision Support 23
expertise only where evidence was lacking . How-
ever, because our protocol development process was lo-
cal, critical buy-in from stakeholders was achieved in the
development stage, enhancing implementation through-
out the network.
We acknowledge the limitations of this analysis.
The study was performed retrospectively with data
from only 7 years because earlier data are not available
within our data systems. Temporal events indepen-
dent of our intervention may affect the rates of imag-
ing, and although we did adjust for year in the regres-
sion analyses, residual confounding may exist. The use
of head CT as an internal control provided some reas-
surance that there was not a generalized trend toward
a decrease in imaging utilization over the study time
frame, as we observed no significant change in head
CT rate and volume during the study period. In addi-
tion, the fact that the CT sinusitis intervention oc-
curred 2 years after the lumbar and brain MR inter-
ventions, but with similar results, lends strength to the
argument that the decrease in imaging is a function of
the intervention. Finally, national trends in the time
frame of this study have reported continued substantial
increases in imaging volumes, in sharp contrast to our
decreases [19,20]. We also acknowledge that other fac-
tors in addition to the clinical decision support likely
contributed to the success of our program, including the
Hawthorne effect, peer pressure, and the fact that the
results of our periodic audits would potentially be avail-
able to the referring physician’s employer.
Also, the analysis was based on administrative data
without patient identifiers. Therefore, we were not able
to directly evaluate the appropriateness of imaging for
each subject. It is possible that inappropriate utilization
continues. We also lack the ability to confirm that the
decrease in utilization is appropriate. However, given
that the computer order entry intervention is based on the
best available evidence, we have confidence that appropri-
patients in whom imaging was not performed at our insti-
tution sought care elsewhere. This would provide an argu-
ment for more global adoption of evidence-based imaging
protocols but not lessen the significance of our results in
improving care at our institution.
With clinical decision support or other barriers to image
ordering, there is always the potential that providers will
“game” the system, developing ways to continue to order
of imaging to determine the outcome when a request was
initially denied by the system. However, we report our re-
sults in terms of imaging rate and total volume of imaging
metrics, imaging rate and total imaging volume represent
actual utilization outcomes that cannot be “gamed” by al-
tering indications or other techniques.
Our data were acquired in the real world of quality
improvement, so we lack the ability to randomize or to
perform a multicenter controlled study. Furthermore,
the limited number of institutions with a focus on Lean
process and quality may restrict the generalizability of
our results. However, we do provide evidence of the
potential value of targeted imaging clinical decision sup-
port and provide an example of a successful approach.
Finally, as of this report, we have implemented imaging
clinical decision support only for a limited number of
imaging studies and indications. However, a large pro-
the potential for improvement, occurs in a relatively lim-
ited number of high-use, high-cost procedures [14,15].
In conclusion, we demonstrate that the implementa-
tion of imaging clinical decision support for selected
high-utilization imaging procedures can have a substan-
tial effect on imaging rate and volume in an integrated
multidisciplinary health care network. The use of such
systems can aid the elimination of unnecessary imaging,
increasing both patient safety and quality and decreasing
health care costs.
We gratefully acknowledge the collaboration of Premera
1. Truffer CJ, Keehan S, Smith S, et al. Health spending projections through
2. Levinson DR. Office of Inspector General report: growth in advanced
imaging paid under the Medicare physician fee schedule. Washington,
DC: Office of Inspector General; 2007.
3. Medicare Payment Advisory Commission. Report to Congress: variation
and innovation in medicine. Washington, DC: Medicare Payment Advi-
sory Commission; 2003.
4. Bernardy M, Ullrich CG, Rawson JV, et al. Strategies for managing
imaging utilization. J Am Coll Radiol 2009;6:844-50.
5. Fuchs VR, Sox HC. Physicians’ views of the relative importance of thirty
medical innovations. Health Aff (Millwood) 2001;20:30-42.
6. Hillman BJ. Foreword. In: Medina LS, Blackmore CC, eds. Evidence
based imaging: optimizing imaging for patient care. New York: Springer;
7. Reinhardt UE, Hussey PS, Anderson GF. US healthcare spending in an
international context. Health Aff (Millwood) 2004;23:10-25.
8. Lehnert BE, Bree RL. Analysis of appropriateness of outpatient CT and
MRI referred from primary care clinics at an academic medical center:
how critical is the need for improved decision support. J Am Coll Radiol
9. Rosenthal DI, Weilburg JB, Schultz T, et al. Radiology order entry with
decision support: initial clinical experience. J Am Coll Radiol 2006;3:
24 Journal of the American College of Radiology/Vol. 8 No. 1 January 2011
Effect of computerized order entry with integrated decision support on
the growth of outpatient procedure volumes: seven-year time series anal-
ysis. Radiology 2009;251:147-55.
11. An act relating to diagnostic imaging service; adding a new chapter to
Title 70 RCW; and declaring an emergency. In: Revised Code of Wash-
ington. 61st ed. 2009.
htdi_executive_summary_35959/. Accessed March 23, 2010.
13. Institute of Medicine. 2009 Institute of Medicine workshop series: low-
ering costs and improving outcomes. Washington, DC: National Acade-
mies Press. In press.
14. Mecklenburg RS, Kaplan GS. The marketplace collaborative project.
15. Washington State agency utilization—advanced imaging priority. Avail-
able at: http://www.hta.hca.wa.gov/documents/StateAgencyUtilization-
AIPriority.pdf. Accessed March 9, 2010.
for patient care. New York: Springer; 2006.
17. Bush RW. Reducing waste in US healthcare system. JAMA 2007;297:
local decisions. Health Aff (Millwood) 2005;24:151-62.
19. Smith-Bindman R, Miglioretti DL, Larson EB. Rising use of diagnostic
imaging in a large integrated health system. Health Aff (Millwood) 2008;
20. Parker L, Levin DC, Frangos A, Rao VM. Geographic variation in the
utilization of noninvasive diagnostic imaging: national Medicare data,
1998-2007. AJR Am J Roentgenol 2010;194:1034-9.
Learning, for more information.
Blackmore, Mecklenburg, Kaplan/Clinical Decision Support 25