other factors such as the possibility that declines in referrals
reﬂect a drop in the available pool of eligible or interested
candidates rather than alert fatigue. However, the population of
potentially eligible participants (ie, patients with a recent
stroke) remained relatively constant during the study, making
this less likely. Nevertheless, it is probable that the reasons for
the declines were multi-factorial, reﬂecting the combined inﬂu-
ence of alert fatigue and other factors. Additional studies,
including qualitative studies to assess physician-user percep-
tions, are ongoing and should help clarify other reasons for the
Comparison of physician response patterns over time and
apparent alert fatigue with those when similar CDS approaches
are employed for clinical use would be useful. Unfortunately,
data on such changes over time in CDS response rates appear to
be lacking in the published literature. As noted above, plentiful
circumstantial evidence of this aspect of alert fatigue in many
studies reveals less than ideal average rates of response to CDS
with some studies commenting on the
common behavior of overriding alerts,
and still others
addressing changes that can increase average response rates by
improving the usability or appropriateness of alerts.
although this form of alert fatigue over time undoubtedly exists,
there has been surprisingly little empirical evidence of it, or data
to characterize the nature of the phenomenon. Our study
appears to be among the ﬁrst to empirically demonstrate this
aspect of alert fatigue by tracking changes in clinician response
to alerts over time. Therefore, we believe it has implications
beyond recruitment using CTAs, and that such an approach to
measuring responses over time can help advance understanding
of alert fatigue in general. We also believe that the methodology
employed here could be used to evaluate and reﬁne the design
and application of decision support alerts in the future.
Although the randomized study design and multi-user, multi-
environment setting strengthen these ﬁndings and advance our
understanding of CTA usage, this study has some limitations.
These ﬁndings were derived from a single study of CTAs
employed in a single trial of patients with recent stroke.
Whether these ﬁndings would differ if the CTA were applied to
another type of trial or in different settings remains to be
determined. Also, while the CTA approach has been demon-
strated to be effective using multiple EHR platforms,
study employed a single EHR and these ﬁndings might differ
with the use of another EHR. Furthermore, this alert was
employed in a setting where other alerts were rarely triggered.
Another factor possibly impacting response rates over time is
the threshold setting (ie, sensitivity vs speciﬁcity) for a given
alert. Whether the ﬁndings of this study would differ if there
were multiple or more frequent alerts is not known but is
possible given that multiple simultaneous alerts are a commonly
cited factor leading to alert fatigue as noted above, and should be
Physician response rates to CTAs started and remained relatively
high even after a period of use, although they gradually but
signiﬁcantly declined over time. While overall response rates
were lower among generalists than subspecialists, the rates of
decline in CTA responses and referrals varied signiﬁcantly only
between university-based versus community-based physicians,
and not between generalists versus subspecialists. These data
also suggest that alert fatigue over time is likely a factor that
must be taken into account when CTAs are employed.
While it is currently unclear how the nature and degree of
alert fatigue for CTAs compares to that of other types of CDS
alerts, this study has implications for the implementation and
management of such alerts. The methodology used here also
appears to have implications for studies into the relative impact
of alert fatigue across a range of decision support alert inter-
ventions. Overall, these ﬁndings offer much-needed empirical
data about the performance characteristics of CTAs, data that
should help inform the tailoring and application of CTAs in real-
world environments in order to overcome the major research
challenge of improving and accelerating participant recruitment.
Acknowledgments Preliminary ﬁndings from this study were presented in abstract
form at the 2011 AMIA Joint Summits on Translational Science. Special thanks go
to our collaborators on the associated intervention study: Drs Mark Eckman, Philip
Payne, Nancy Elder, Sian Cotton, and Emily Patterson, and Ms. Ruth Wise.
Contributors PJE contributed to the conception, design, and acquisition and
interpretation of data, and drafted and revised the manuscript. AL contributed to the
design, interpretation of data, and critical revisions to the manuscript. Both authors
approved the ﬁnal version to be published.
Funding This project was supported by a grant from the National Library of Medicine
of the National Institutes of Health, R01-LM009533.
Competing interests None.
Ethics approval Ethics approval was granted by the University of Cincinnati
Institutional Review Board.
Provenance and peer review Not commissioned; externally peer reviewed.
1. Nathan DG, Wilson JD. Clinical research and the NIH: a report card. N Engl J Med
2. Campbell EG, Weissman JS, Moy E, et al. Status of clinical research in academic
health centers: views from the research leadership. JAMA 2001;286:800e6.
3. Mansour EG. Barriers to clinical trials. Part III: knowledge and attitudes of health
care providers. Cancer 1994;74(9 Suppl):2672e5.
4. Siminoff LA, Zhang A, Colabianchi N, et al. Factors that predict the referral of breast
cancer patients onto clinical trials by their surgeons and medical oncologists. J Clin
5. Somkin CP, Altschuler A, Ackerson L, et al. Organizational barriers to physician
participation in cancer clinical trials. Am J Manag Care 2005;11:413e21.
6. Winn RJ. Obstacles to the accrual of patients to clinical trials in the community
setting. Semin Oncol 1994;21(4 Suppl 7):112e17.
7. Embi PJ, Jain A, Clark J, et al. Effect of a clinical trial alert system on physician
participation in trial recruitment. Arch Intern Med 2005;165:2272e7.
8. Rollman BL, Fischer GS, Zhu F, et al. Comparison of electronic physician prompts
versus waitroom case-ﬁnding on clinical trial enrollment. J Gen Intern Med
9. Grundmeier RW, Swietlik M, Bell LM. Research subject enrollment by primary care
pediatricians using an el ectronic health record. AMIA Annu Symp Proc 2007:289e93.
10. Embi PJ, Jain A, Harris CM. Physicians’ perceptions of an electronic health record-
based clinical trial alert approach to subject recruitment: a survey. BMC Med Inform
Decis Mak 2008;8:13.
11. Ash JS, Sittig DF, Campbell EM, et al. Some unintended consequences of clinical
decision support systems. AMIA Annu Symp Proc 2007:26e30.
12. van der Sijs H, Aarts J, Vulto A, et al. Overriding of drug safety alerts in
computerized physician order entry. J Am Med Inform Assoc 2006;13:138e47.
13. Shah NR, Seger AC, Seger DL, et al. Improving acceptance of computerized
prescribing alerts in ambulatory care. J Am Med Inform Assoc 2006;13:5e11.
14. Horsky J, Zhang J, Patel VL. To err is not entirely human: complex technology and
user cognition. J Biomed Inform 2005;38:264e6.
15. Cash JJ. Alert fatigue. Am J Health Syst Pharm 2009;66:2098e101.
16. Shah A. Alert Fatigue. 2011. http://clinfowiki.org/wiki/index.php/Alert_fatigue
(accessed 15 Jan 2012).
17. Embi PJ, Eckman MH, Payne PR, et al. EHR-based clinical trial alert effects on
recruitment to a neurology trial across settings: interim analysis of a randomized
controlled Study. AMIA Summits Transl Sci Proc; March 2010. San Francisco, CA,
18. Embi PJ, Lieberman MI, Ricciardi TN. Early Development of a Clinical Trial Alert
System in an EHR Used in Small Practices: Toward Generalizability. Phoenix, AZ:
AMIA Spring Congress, 2006.
19. Embi PJ, Jain A, Clark J, et al. Development of an electronic health record-based
Clinical Trial Alert system to enhance recruitment at the point of care. AMIA Annu
Symp Proc 2005:231e5.
20. Weingart SN, Toth M, Sands DZ, et al. Physicians’ decisions to override
computerized drug alerts in primary care. Arch Intern Med 2003;163
PAGE fraction trail=4
e148 J Am Med Inform Assoc 2012;19:e145ee148. doi:10.1136/amiajnl-2011-000743
Research and applications