Quality and Outcomes
The Hospital Compare Mortality Model
and the Volume–Outcome
Jeffrey H. Silber, Paul R. Rosenbaum, Tanguy J. Brachet,
Richard N. Ross, Laura J. Bressler, Orit Even-Shoshan,
Scott A. Lorch, and Kevin G. Volpp
Objective. We ask whether Medicare’s Hospital Compare random effects model
correctlyassessesacutemyocardialinfarction (AMI)hospitalmortality rates whenthere
is a volume–outcome relationship.
Data Sources/Study Setting. Medicare claims on 208,157 AMI patients admitted in
3,629 acute care hospitals throughout the United States.
average adjusted mortality based on the Hospital Compare random effects model. We
thenfitrandomeffectsmodelswith the same patientvariables asinMedicare’s Hospital
Compare mortality model but also included terms for hospital Medicare AMI volume
and another model that additionally included other hospital characteristics.
Principal Findings. Hospital Compare’s average adjusted mortality significantly un-
derestimates average observed death rates in small volume hospitals. Placing hospital
volume in the Hospital Compare model significantly improved predictions.
Conclusions. The Hospital Compare random effects model underestimates the typ-
ically poorer performance of low-volume hospitals. Placing hospital volume in the
indicated when using a random effects model to predict outcomes. Care must be taken
to insure the proper method of reporting such models, especially if hospital charac-
teristics are included in the random effects model.
Key Words. Hospital Compare, mortality, acute myocardial infarction, random
Medicare’s web-based ‘‘Hospital Compare’’ is intended to provide the public
patients with certain medical conditions’’ (U.S. Department of Health & Hu-
man Services 2007a). For acute myocardial infarction (AMI) mortality in
rHealth Research and Educational Trust
Health Services Research
2007, of 4,477 U.S. hospitals (many of which presumably have no experience
with AMI), the Medicare Hospital Compare model asserted that 4,453 (99.5
‘‘better,’’and 7 were‘‘worse’’than theU.S.national rate.In2008,the Hospital
Compare model suggests that of 4,311 hospitals, none were worse than
average, and nine were better than average.
These evaluations are surprising. Some hospitals treat only a few AMIs
every year, and others treat a few each week. One of the more consistent
Chassin 2002; Gandjour, Bannenberg, and Lauterbach 2003; Shahian and
Normand 2003) is that, after adjusting for patient risk factors, there is often a
higher risk of death when a patient is treated at a low-volume hospital. Indeed,
this pattern is unmistakable in Medicare data, the data used to construct
Hospital Compare. However, Hospital Compare reports no such pattern.
THE SMALL NUMBERS PROBLEM
Over 20 years ago, in a paper published in this journal, Chassin et al. (1989)
described the small numbers problem when attempting to rank hospitals by
adjusted mortalityrates.Toaccountfordeath rateinstabilitywithlowvolume,
the Chassin study chose to rank hospitals by the statistical significance asso-
ciated with their observed and expected death rates. This, in retrospect, was
Address correspondence to Jeffrey H. Silber, M.D., Ph.D., Center for Outcomes Research, 3535
Philadelphia, Philadelphia, PA. Jeffrey H. Silber, M.D., Ph.D., and Tanguy J. Brachet, Ph.D., are
with the Department of Anesthesiology & Critical Care, University of Pennsylvania School of
Medicine, Philadelphia, PA. Jeffrey H. Silber, M.D., Ph.D., is with the Department of Pediatrics,
University of Pennsylvania School of Medicine, Philadelphia, PA. Jeffrey H. Silber, M.D., Ph.D.,
and Kevin G. Volpp, M.D., Ph.D., are with the Department of Health Care Management, The
Wharton School, University of Pennsylvania, Philadelphia, PA. Jeffrey H. Silber, M.D., Ph.D.,
Paul R. Rosenbaum, Ph.D., Tanguy J. Brachet, Ph.D., Orit Even-Shoshan, M.S., Scott A. Lorch,
M.D.,M.S.C.E., and KevinG.Volpp, M.D.,Ph.D., are with the Leonard DavisInstitute ofHealth
Economics, University of Pennsylvania, Philadelphia, PA. Paul R. Rosenbaum, Ph.D., is with the
Department of Statistics, The Wharton School, University of Pennsylvania, Philadelphia, PA.
Scott A. Lorch, M.D., M.S.C.E., is with the Department of Pediatrics, Division of Neonatology,
The Children’s Hospital of Philadelphia, Philadelphia, PA. Kevin G. Volpp, M.D., Ph.D., is with
the Center for Health Equity Research and Promotion, Veteran’s Administration Hospital, Phil-
adelphia, PA. Kevin G. Volpp, M.D., Ph.D., is with the Department of Medicine, University of
Pennsylvania School of Medicine, Philadelphia, PA.
Hospital Compare and the Volume–Outcome Relationship 1149
not a good decision. Large hospitals could have extreme ranks because their
significance and would be forced to be ranked near the middle.
Twenty years later, a new solution for the small numbers problem in
AMI has been introduced by Medicare’s Hospital Compare. The Hospital
Compare model for AMI is based on a random effects model published
by Krumholz et al. (2006b) as well as a technical report funded by a contract
from Medicare (Krumholz et al. 2006a). Consistent with the technical report,
in a section titled Adjusting for Small Hospitals or a Small Number of Cases
web page says:
The [Medicare] hierarchical regression model also adjusts mortality rates results
for. . .hospitalswith fewheartattack . ..casesina givenyear. ...This reducesthe
chance that such hospitals’ performance will fluctuate wildly from year to year or
that they will be wrongly classified as either a worse or better performer .. . .In
essence, the predicted mortality rate for a hospital with a small number of cases is
moved toward the overall U.S. National mortality rate for all hospitals. The es-
timates of mortality for hospitals with few patients will rely considerably on the
pooled data for all hospitals, making it less likely that small hospitals will fall into
either of the outlier categories. This pooling affords a ‘‘borrowing of statistical
strength’’ that provides more confidence in the results.
After ‘‘moving’’ (‘‘shrinking’’) many hospitals’ AMI mortality rates toward the
national rate, the 2008 Hospital Compare concludes that 4,302/4,311 or 99.8
percent of hospitals are ‘‘no different than U.S. national rate’’ and zero hos-
pitals are ‘‘worse than U.S. national rate.’’ This study will attempt to explain
why Hospital Compare came to these conclusions.
The ‘‘hierarchical’’ or ‘‘random effects’’ model used to construct
Hospital Compare utilizes the fact that low volume at a hospital implies that
its empirical mortality rate is imprecisely estimated. However, it assumes that
there is no relationship between volume and mortality (Panageas et al. 2007),
not a reasonable assumption, given that the literature suggests AMI mortality
rates tend to be higher when volume is lower (Luft et al. 1987; Farley and
Ozminkowski 1992; Thiemann et al. 1999; Tu, Austin, and Chan 2001; Halm
et al. 2002). Hospital Compare could have developed a random effects model
that allowed the empirical data to speak to the issue of whether a systematic
volume–outcome relationship is present, but the Hospital Compare model
used an overriding assumption that true hospital mortality rates are random
hospital is judged by Hospital Compare to be unstable and in need of
1150 HSR: Health Services Research 45:5, Part I (October 2010)
08/15/2007’’ [accessed on October 10, 2007]. Available at http://www.cms.
Chassin, M. R., R. E. Park, K. N. Lohr, J. Keesey, and R. H. Brook. 1989. ‘‘Differences
among Hospitals in Medicare Patient Mortality.’’ Health Services Research 24:
Cox, D. R., and M. Y. Wong. 2010. ‘‘A Note on the Sensitivity to Assumptions of a
Generalized Linear Mixed Model.’’ Biometrika 97: 209–14.
for Predicting Surgical Mortality in the Hospital.’’ Health Affairs 28: 1189–98.
Farley, D. E., andR. J. Ozminkowski. 1992. ‘‘Volume–OutcomeRelationships and In-
Hospital Mortality: The Effect of Changes in Volume over Time.’’ Medical Care
Francis, T. 2007. ‘‘How to Size Up Your Hospital: Improved Public Databases Let
People Compare Practices and Outcomes; The Importance of Looking Past the
Numbers.’’ The Wall Street Journal, July 10, New York, pp. D.1.
Freedman, D. A. 2006. ‘‘On the So-Called ‘Huber Sandwich Estimator’ and ‘Robust
Standard Errors.’’’ American Statistician 60: 299–302.
Gandjour, A., A. Bannenberg, and K. W. Lauterbach. 2003. ‘‘Threshold Volumes
Associated with Higher Survival in Health Care: A Systematic Review.’’ Medical
Care 41: 1129–41.
Gelman, A. 2006. ‘‘Multilevel (Hierarchical) Modeling: What It Can and Cannot Do.’’
Technometrics 48: 432–5.
York: Chapman & Hall.
York State’s Approach.’’ New England Journal of Medicine 332: 1229–32.
Halm, E. A., C. Lee, and M. R. Chassin. 2002. ‘‘Is Volume Related to Outcome in
Health Care? A Systematic Review and Methodologic Critique of the Litera-
ture.’’ Annals of Internal Medicine 137: 511–20.
Iezzoni, L. I. 2003. Risk Adjustment for Measuring Health Care Outcomes. Chicago, IL:
Health Administration Press.
M. M. Ward. 2006a. Risk-Adjustment Models for Ami and Hf 30-Day Mortality, Sub-
contract #8908-03-02. Baltimore, MD: Centers for Medicare and Medicaid Services.
and S.-L. T. Normand. 2006b. ‘‘An Administrative Claims Model Suitable for
Profiling Hospital Performance Based on 30-Day Mortality Rates among
Patients with an Acute Myocardial Infarction.’’ Circulation 113: 1683–92.
Li, Y., X. Cai, L. G. Glance, W. D. Spector, and D. B. Mukamel. 2009. ‘‘National
Release of the Nursing Home Quality Report Cards: Implications of Statistical
Methodology for Risk Adjustment.’’ Health Services Research 44: 79–102.
Luft, H. S., S. S. Hunt, and S. C. Maerki. 1987. ‘‘The Volume–Outcome Relationship:
Practice-Makes-Perfect or Selective-Referral Patterns?’’ Health Services Research
Morris, C. N. 1983. ‘‘Parametric Empirical Bayes Inference: Theory and Applica-
tions.’’ Journal of the American Statistical Association 78: 47–55.
Hospital Compare and the Volume–Outcome Relationship 1165
for Public Reporting of Health Provider Quality: Making It Meaningful to
Patients.’’ American Journal of Public Health 100: 264–9.
Normand, S.-L. T., M. E. Glickman, and C. A. Gatsonis. 1997. ‘‘Statistical Methods for
Profiling Providers of Medical Care: Issues and Applications.’’ Journal of the
American Statistical Association 92: 803–14.
Panageas, K. S., D. Schrag, A. Russell Localio, E. S. Venkatraman, and C. B. Begg.
2007. ‘‘Properties of Analysis Methods That Account for Clustering in Volume–
Outcome Studies When the Primary Predictor Is Cluster Size.’’ Statistics in
Medicine 26: 2017–35.
Ruppert, D., M. P. Wand, and R. J. Carroll. 2003. Semiparametric Regression. New York:
Cambridge University Press.
SAS Institute Inc. 2004. ‘‘SAS/STAT 9.1 User’s Guide. Chapter 42. The Logistic Proce-
dure’’ [accessed on February 29, 2008]. Available at http://support.sas.com/docu
SAS Institute Inc. 2008. ‘‘Production Glimmix Procedure’’ [accessed on January 28,
2008]. Available at http://support.sas.com/rnd/app/da/glimmix.html
Shahian, D. M., and S.-L. T. Normand. 2003. ‘‘The Volume–Outcome Relationship:
From Luft to Leapfrog.’’ Annals of Thoracic Surgery 75: 1048–58.
Silber, J. H., P. S. Romano, A. K. Rosen, Y. Wang, R. N. Ross, O. Even-Shoshan, and
K. Volpp. 2007. ‘‘Failure-to-Rescue: Comparing Definitions to Measure Quality
of Care.’’ Medical Care 45: 918–25.
Silber, J. H., P. R. Rosenbaum, P. S. Romano, A. K. Rosen, Y. Wang, Y. Teng, M. J.
Halenar, O. Even-Shoshan, and K. G. Volpp. 2009. ‘‘Hospital Teaching
Intensity, Patient Race, and Surgical Outcomes.’’ Archives of Surgery 144:
The Leapfrog Group. ‘‘White Paper Available to Explain Leapfrog’s New Ebhr
Survival Predictor’’ [accessed on April 19, 2010]. Available at http://www.
Thiemann, D. R., J. Coresh, W. J. Oetgen, and N. R. Powe. 1999. ‘‘The Association
between Hospital Volume and Survival after Acute Myocardial Infarction in
Elderly Patients.’’ New England Journal of Medicine 340: 1640–8.
Patients Treated by Admitting Physician and Mortality after Acute Myocardial
Infarction.’’ Journal of the American Medical Association 285: 3116–22.
U.S. Department of Health & Human Services. 2007a. ‘‘Hospital Compare——A Qual-
ity Tool for Adults, Including People with Medicare (December 19)’’ [accessed
on January 28, 2008]. Available at http://www.hospitalcompare.hhs.gov/
U.S. Departmentof Health& Human Services. 2007b. ‘‘Information for Professionals.
Outcome Measures: Adjusting for Small Hospitals or a Small Number of Cases
1166HSR: Health Services Research 45:5, Part I (October 2010)
Werner, R. M., and D. A. Asch. 2005. ‘‘The Unintended Consequences of Publicly
Reporting Quality Information.’’ Journal of the American Medical Association
Additional supporting information may be found in the online version of this
Appendix SA1: Author Matrix.
Appendix SA2: Electronic Appendix: Additional Models.
Please note: Wiley-Blackwell is not responsible for the content or func-
tionality of any supporting materials supplied by the authors. Any queries
(other than missing material) should be directed to the corresponding author
for the article.
Hospital Compare and the Volume–Outcome Relationship1167