Comparison of 19 pre-operative risk stratification
models in open-heart surgery
Johan Nilsson1*, Lars Algotsson2, Peter Ho ¨glund3, Carsten Lu ¨hrs1, and Johan Brandt1
1Department of Cardiothoracic Surgery, Heart and Lung Centre, Lund University Hospital, SE 221 85 Lund, Sweden;
2Department of Cardiothoracic Anesthesiology, Heart and Lung Centre, Lund University Hospital, Lund, Sweden; and
3Competence Centre for Clinical Research, Lund University Hospital, Lund, Sweden
Received 23 August 2005; revised 2 November 2005; accepted 16 December 2005; online publish-ahead-of-print 18 January 2006
See page 768 for the editorial comment on this article (doi:10.1093/eurheartj/ehi792)
Aims To compare 19 risk score algorithms with regard to their validity to predict 30-day and 1-year
mortality after cardiac surgery.
Methods and results Risk factors for patients undergoing heart surgery between 1996 and 2001 at a
single centre were prospectively collected. Receiver operating characteristics (ROC) curves were
used to describe the performance and accuracy. Survival at 1 year and cause of death were obtained
in all cases. The study included 6222 cardiac surgical procedures. Actual mortality was 2.9% at 30
days and 6.1% at 1 year. Discriminatory power for 30-day and 1-year mortality in cardiac surgery was
highest for logistic (0.84 and 0.77) and additive (0.84 and 0.77) European System for Cardiac Operative
Risk Evaluation (EuroSCORE) algorithms, followed by Cleveland Clinic (0.82 and 0.76) and Magovern
(0.82 and 0.76) scoring systems. None of the other 15 risk algorithms had a significantly better discrimi-
natory power than these four. In coronary artery bypass grafting (CABG)-only surgery, EuroSCORE fol-
lowed by New York State (NYS) and Cleveland Clinic risk score showed the highest discriminatory
power for 30-day and 1-year mortality.
Conclusion EuroSCORE, Cleveland Clinic, and Magovern risk algorithms showed superior performance
and accuracy in open-heart surgery, and EuroSCORE, NYS, and Cleveland Clinic in CABG-only surgery.
Although the models were originally designed to predict early mortality, the 1-year mortality prediction
was also reasonably accurate.
Despite technological advancements, open-heart operations
still carry a risk of mortality and morbidity. To aid in the
selection of patients for cardiac surgery, several risk-
scoring systems have been developed during the last
decades. These aim to estimate the risk of peri-operative
death, based on the occurrence of different risk factors.
Operative mortality is also increasingly used as an indicator
of the quality of cardiac surgery.1
To make an accurate comparison between different
institutions or surgeons, mortality data must be adjusted
to the risk profiles of the patients.2,3Differences between
the available risk algorithms regarding score design and the
patient population on which the score development was
based could influence their accuracy and performance.
Ideally, a risk model should be useful for outcome prediction
at different surgical centres, both at the institutional level
and for individual patients.4Operative mortality is the
outcome variable most commonly used as a quality indi-
cator, but long-term mortality may be more relevant from
a patient perspective.
A few comparative studies of different risk algorithms
exist.4–8However, the relative performance of the risk-
scoring systems currently used remains unclear. The
purpose of this study was to compare 19 open-source risk
score algorithms with regard to their validity to predict
30-day and 1-year mortality after cardiac surgery in a
large single-institution patient population.
Study design and patients
The study was approved by the Ethics Committee of the Medical
Faculty at Lund University. Risk factors for all adult patients under-
going heart surgery at the University Hospital of Lund between
January 1996 and February 2001 were prospectively collected
Cardiothoracic Surgery. The patient record form contained a total
of 248 variables (80 pre-, 106 intra-, and 62 post-operative) based
on the Society of Thoracic Surgeons (STS)9patient record form.
The data was stored in a local adult cardiac surgery database.
Data collection and risk-score calculation
From the total of 248 variables, those corresponding to the risk
factors in the different risk models were selected. Thus, a subset
& The European Society of Cardiology 2006. All rights reserved. For Permissions, please e-mail: firstname.lastname@example.org
*Corresponding author. Fax: þ46 46 15 86 35.
E-mail address: email@example.com
European Heart Journal (2006) 27, 867–874
by guest on May 14, 2011
1. Dubois RW, Rogers WH, Moxley JH III, Draper D, Brook RH. Hospital
inpatient mortality. Is it a predictor of quality? N Engl J Med 1987;
2. Dubois RW, Brook RH, Rogers WH. Adjusted hospital death rates: a poten-
3. Hannan EL, Kilburn H Jr, Racz M, Shields E, Chassin MR. Improving the out-
comes of coronary artery bypass surgery in New York State. JAMA
4. Asimakopoulos G, Al-Ruzzeh S, Ambler G, Omar RZ, Punjabi P, Amrani M,
Taylor KM. An evaluation of existing risk stratification models as a tool for
comparison of surgical performances for coronary artery bypass grafting
between institutions. Eur J Cardiothorac Surg 2003;23:935–942.
5. Bridgewater B, Neve H, Moat N, Hooper T, Jones M. Predicting operative
risk for coronary artery surgery in the United Kingdom: a comparison of
various risk prediction algorithms. Heart 1998;79:350–355.
6. Geissler HJ, Holzl P, Marohl S, Kuhn-Regnier F, Mehlhorn U, Sudkamp M,
de Vivie ER. Risk stratification in heart surgery: comparison of six score
systems. Eur J Cardiothorac Surg 2000;17:400–406.
7. Nilsson J, Algotsson L, Hoglund P, Luhrs C, Brandt J. Early mortality in
coronary bypass surgery: the EuroSCORE versus The Society of Thoracic
Surgeons risk algorithm. Ann Thorac Surg 2004;77:1235–1239; discussion
8. Pinna-Pintor P, Bobbio M, Colangelo S, Veglia F, Giammaria M, Cuni D,
Maisano F, Alfieri O. Inaccuracy of four coronary surgery risk-adjusted
models to predict mortality in individual patients. Eur J Cardiothorac
9. Edwards FH, Clark RE, Schwartz M. Coronary artery bypass grafting: the
Society of Thoracic Surgeons National Database experience. Ann Thorac
10. Schemper M, Smith TL. Efficient evaluation of treatment effects in the
presence of missing covariate values. Stat Med 1990;9:777–784.
11. Eagle KA, Guyton RA, Davidoff R, Ewy GA, Fonger J, Gardner TJ, Gott JP,
Herrmann HC, Marlow RA, Nugent W, O’Connor GT, Orszulak TA,
Rieselbach RE, Winters WL, Yusuf S, Gibbons RJ, Alpert JS, Garson A Jr,
Gregoratos G, Russell RO, Ryan TJ, Smith SC Jr. ACC/AHA guidelines for
coronary artery bypass graft surgery: executive summary and recommen-
dations. A report of the American College of Cardiology/American Heart
Association Task Force on Practice Guidelines (Committee to revise the
1991 guidelines for coronary artery bypass graft surgery). Circulation
12. Pepe MS. The receiver operating characteristic curve. The Statistical
Evaluation of Medical Tests for Classification and Prediction. New York:
Oxford University Press; 2003. p66–94.
13. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver
operating characteristic (ROC) curve. Radiology 1982;143:29–36.
14. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under
two or more correlated receiver operating characteristic curves: a non-
parametric approach. Biometrics 1988;44:837–845.
15. LustedLB. Logical analysis
16. Nashef SA, Roques F, Michel P, Cortina J, Faichney A, Gams E, Harjula A,
Jones MT. Coronary surgery in Europe: comparison of the national subsets
of the European system for cardiac operative risk evaluation database.
Eur J Cardiothorac Surg 2000;17:396–399.
17. Nashef SA, Roques F, Hammill BG, Peterson ED, Michel P, Grover FL,
Wyse RK, Ferguson TB. Validation of European System for Cardiac
Operative Risk Evaluation (EuroSCORE) in North American cardiac
surgery. Eur J Cardiothorac Surg 2002;22:101–105.
18. Nilsson J, Algotsson L, Hoglund P, Luhrs C, Brandt J. EuroSCORE predicts
intensive care unit stay and costs of open heart surgery. Ann Thorac Surg
2004;78:1528–1534; discussion 1534–1535.
19. Murphy-Filkins R, Teres D, Lemeshow S, Hosmer DW. Effect of
changing patient mix on the performance of an intensive care unit
AmJ Public Health
severity-of-illness model: how to distinguish a general from a specialty
intensive care unit. Crit Care Med 1996;24:1968–1973.
20. Schemper M, Heinze G. Probability imputation revisited for prognostic
factor studies. Stat Med 1997;16:73–80.
21. Wyse RK, Taylor KM. Using the STS and multinational cardiac surgical
databases to establish risk-adjusted benchmarks for clinical outcomes.
Heart Surg Forum 2002;5:258–264.
22. Lippmann RP, Shahian DM. Coronary artery bypass risk prediction using
neural networks. Ann Thorac Surg 1997;63:1635–1643.
23. Huijskes RVHP, Rosseel PMJ, Tijssen JGP. Outcome prediction in coronary
artery bypass grafting and valve surgery in the Netherlands: development
of the Amphiascore and its comparison with the Euroscore. Eur J
Cardiothorac Surg 2003;24:741–749.
24. Kurki TS, Kataja M. Preoperative prediction of postoperative morbidity in
coronary artery bypass grafting. Ann Thorac Surg 1996;61:1740–1745.
25. Higgins TL, Estafanous FG, Loop FD, Beck GJ, Blum JM, Paranandi L.
Stratification of morbidity and mortality outcome by preoperative risk
factors in coronary artery bypass patients. A clinical severity score.
26. Nashef SA, Roques F, Michel P, Gauducheau E, Lemeshow S, Salamon R.
European system for cardiac operative risk evaluation (EuroSCORE).
Eur J Cardiothorac Surg 1999;16:9–13.
27. Roques F, Michel P, Goldstone AR, Nashef SA. The logistic EuroSCORE.
Eur Heart J 2003;24:882.
28. Roques F, Gabrielle F, Michel P, De Vincentiis C, David M, Baudet E.
Quality of care in adult heart surgery: proposal for a self-assessment
approach based on a French multicenter study. Eur J Cardiothorac Surg
1995;9:433–439; discussion 439–440.
29. Magovern JA, Sakert T, Magovern GJ, Benckart DH, Burkholder JA,
Liebler GA, Magovern GJ, Sr. A model that predicts morbidity and
mortality after coronary artery bypass graft surgery. J Am Coll Cardiol
30. Coronary Artery Bypass Surgery in New York State 1996–1998. [pdf] 2001
[cited 8 April 2005]. www.health.state.ny.us/nysdoh/consumer/heart/
31. Tu JV, Jaglal SB, Naylor CD. Multicenter validation of a risk index for mor-
tality, intensive care unit stay, and overall hospital length of stay after
cardiac surgery. Steering Committee of the Provincial Adult Cardiac
Care Network of Ontario. Circulation 1995;91:677–684.
32. Parsonnet V, Dean D, Bernstein AD. A method of uniform stratification of
risk for evaluating the results of surgery in acquired adult heart disease.
33. Gabrielle F, Roques F, Michel P, Bernard A, de Vicentis C, Roques X,
Brenot R, Baudet E, David M. Is the Parsonnet’s score a good predictive
score of mortality in adult cardiac surgery: assessment by a French
multicentre study. Eur J Cardiothorac Surg 1997;11:406–414.
34. Pons JM, Granados A, Espinas JA, Borras JM, Martin I, Moreno V. Assessing
open heart surgery mortality in Catalonia (Spain) through a predictive
risk model. Eur J Cardiothorac Surg 1997;11:415–423.
35. Ivanov J, Tu JV, Naylor CD. Ready-made, recalibrated, or remodeled?
Issues in the use of risk indexes for assessing mortality after coronary
artery bypass graft surgery. Circulation 1999;99:2098–2104.
36. Ivanov J, Borger MA, David TE, Cohen G, Walton N, Naylor CD. Predictive
accuracy study: comparing a statistical model to clinicians’ estimates of
37. Tremblay NA, Hardy JF, Perrault J, Carrier M. A simple classification of
38. Tuman KJ, McCarthy RJ, March RJ, Najafi H, Ivankovich AD. Morbidity and
duration of ICU stay after cardiac surgery. A model for preoperative risk
assessment. Chest 1992;102:36–44.
39. Grover FL, Johnson RR, Marshall G, Hammermeister KE. Factors predic-
tive of operative mortality among coronary artery bypass subsets. Ann
Thorac Surg 1993;56:1296–1306; discussion 1306–1307.
874 J. Nilsson et al.
by guest on May 14, 2011