Page 1

American Journal of Transplantation 2010; 10: 2279–2286

Wiley Periodicals Inc.

C ?2010 The Authors

Journal compilationC ?2010 The American Society of

Transplantation and the American Society of Transplant Surgeons

doi: 10.1111/j.1600-6143.2010.03179.x

A Risk Prediction Model for Delayed Graft Function

in the Current Era of Deceased Donor Renal

Transplantation

W. D. Irisha,*, J. N. Ilsleyb, M. A. Schnitzlerc,

S. Fengdand D. C. Brennane

aBiostatistics and Health Outcomes Research, CTI Clinical

Trial and Consulting Services, Cincinnati, OH

bMedical Affairs, Genzyme Corporation, Cambridge, MA

cCenter for Outcomes Research, Saint Louis University,

St. Louis, MO

dDepartment of Surgery, University of California,

San Francisco, San Francisco, CA

eDepartment of Medicine, Washington University,

St. Louis, MO

*Corresponding author: William D. Irish,

birish@ctifacts.com

Delayed graft function (DGF) impacts short- and long-

term outcomes. We present a model for predicting

DGF after renal transplantation. A multivariable logis-

tic regression analysis of 24337 deceased donor re-

nal transplant recipients (2003–2006) was performed.

We developed a nomogram, depicting relative contri-

bution of risk factors, and a novel web-based calcula-

tor (www.transplantcalculator.com/DGF) as an easily

accessible tool for predicting DGF. Risk factors in the

modern era were compared with their relative impact

in an earlier era (1995–1998). Although the impact of

many risk factors remained similar over time, weight

of immunological factors attenuated, while impact of

donor renal function increased by 2-fold. This may re-

flect advances in immunosuppression and increased

utilization of kidneys from expanded criteria donors

(ECDs) in the modern era. The most significant factors

associated with DGF were cold ischemia time, donor

creatinine, body mass index, donation after cardiac

death and donor age. In addition to predicting DGF, the

model predicted graft failure. A 25–50% probability of

DGF was associated with a 50% increased risk of graft

failure relative to a DGF risk <25%, whereas a >50%

DGF risk was associated with a 2-fold increased risk of

graft failure. This tool is useful for predicting DGF and

long-term outcomes at the time of transplant.

Key words: Deceased donor kidneys, delayed graft

function (DGF), risk assessment modeling, risk factors

Abbreviations:

dence Interval;

tinuous Variable; DCD, Donation After Cardiac Death;

DGF, Delayed Graft Function; ECD, Expanded Criteria

BMI, Body Mass Index;

CIT, Cold Ischemia Time;

CI, Confi-

CV, Con-

Donor; GAM, Generalized Additive Model; HLA, Hu-

man Leukocyte Antigen; HR, Hazard Ratio; IV, Indica-

tor Variable; OPTN, Organ Procurement and Transplan-

tation Network; OR, Odds Ratio; PRA, Panel Reactive

Antibody; ROC,ReceiverOperatingCharacteristic; SD,

Standard Deviation; UNOS, United Network for Organ

Sharing; WIT, Warm Ischemia Time.

Received 02 December 2009, revised 23 April 2010 and

accepted for publication 24 April 2010

Introduction

Delayed graft function (DGF), commonly defined as need

for dialysis within the first week posttransplantation, oc-

curs in approximately 25% of deceased donor transplants

(1,2); although the incidence varies depending on the risk

profile of the donor and recipient (2). For example, DGF is

more common in Black recipients of deceased donor kid-

neys (2,3), while donors over the age of 60 years confer

a risk of DGF nearly three times that of donors less than

40 years of age (2).

DGF is associated with increased risk of acute rejection

and poorer long-term graft survival (1,3–5) and is asso-

ciated with significant economic costs due to prolonged

hospitalization and costly patient management (6–8). In

2003, Irish and colleagues developed a multivariable lo-

gistic regression model to quantify the risk of DGF using

donor and recipients characteristics known at the time of

transplantation (2). The original model included deceased

donor renal transplant recipients transplanted over a 4-year

period (1995–1998). Using the combination of donor and

recipients risk factors identified in this model, a nomogram

was developed as a tool for clinicians to identify patients

at greater risk of developing DGF .

Criticisms of the original model were that the study pop-

ulation included multiple-organ transplants, preemptive

transplants and organs, which had undergone machine-

perfusion. Moreover, the study did not include risk factors

such as warm ischemia time (WIT) (9), duration of pretrans-

plant dialysis (10), donor weight (11) and recipient body

mass index (BMI) (12,13). Additionally, the practical usabil-

ity of a paper-based nomogram to manually calculate DGF

risk is limited (14).

2279

Page 2

Irish et al.

Since the development of the original model, there have

been significant advances in the field of transplantation

with regard to both immunosuppression and organ allo-

cation strategies (15). In light of these changes and the

criticisms cited above, we set out to refine the previously

published model using a more recent refined data set and

considering additional risk factors in the analysis. A com-

parison of the old model with the model developed in the

current era provided a basis for understanding the changes

in the relative impact of risk factors over time. We also

aimed to improve the utility of the model by developing a

web-based DGF Risk calculator with potential applicability

as a tool for defining individuals or populations at greatest

risk for DGF as well as predicting long-term outcomes.

Methods

Study design

Risk factors for DGF were analyzed in data obtained from United Network

for Organ Sharing/Organ Procurement and Transplantation (UNOS/OPTN)

on adult (≥16 years) recipients of a solitary, nonpreemptive, nonmachine-

perfused, deceased donor kidney between January 1, 2003 and December

31, 2006. To minimize the inclusion of grafts experiencing technical failures

or primary nonfunction, grafts lost within 24 h posttransplant were excluded

from the model. DGF was defined as the need for dialysis within the first 7

days posttransplant, as reported to UNOS. All risk factors identified by Irish

et al. (2) were included in the updated analysis as were four additional risk

factors (WIT, duration of dialysis, recipient BMI and donor weight).

Statistical analysis

Risk factors for DGF and outcomes were studied using a multivariable

binary logistic regression modeling technique; a statistical method that is

typically used when modeling a dichotomous variable (16). Measures of

model adequacy were obtained, including lack of fit (linearity assumptions)

and predictive accuracy (calibration and discrimination).

To test whether quantitative covariates were linearly related to the log

odds of response, we used generalized additive models (GAMs) (17). The

log odds scale used in logistic regression (16) was defined as the natural

logarithm of the odds of DGF (log (p/q) where p is the probability of DGF and

q = 1 − p). Graphical techniques were used to investigate the functional

form of quantitative covariates (i.e. nonlinear trends, discrete effects) (17).

In an effort to minimize the number of patients deleted from the logistic

regression analysis, patientswerenotexcludedonthebasisof missing data

elements. Missing data of continuous variables (CVs) were accounted for in

the model by creating two indicator variables (IV1 and IV2): If CV = missing

then IV1 = 1 or 0 otherwise; and If CV = missing then IV2 = 0 otherwise

IV2 = CV. For example, if patient is missing CIT then IV = 1 and IV2 = 0. If

patient is not missing CIT then IV1 = 0 and IV2 = CIT. For categorical data,

missing or unknown were grouped into the ‘no’ or ‘absence’ category.

Predictive accuracy of the model was assessed in an external validation

data set (patients transplanted in 2007) using the concordance c index (area

under the receiver operating characteristic [ROC] curve), which estimates

the probability of concordance between predicted and observed responses

(0.5 indicating no predictive discrimination; 1.0 indicating perfect separation

of patients with different outcomes). Further validation was performed by

evaluating the model in ECD and donation after cardiac death (DCD) sub-

groups of the 2007 validation data set. We used decision-curve analysis as

an approach to quantify the clinical usefulness of the DGF prediction tool

(18). This was done by calculating the net benefit of the model for different

threshold probabilities for DGF with different weights for false-positive and

false-negative classifications.

A nomogram was developed using ‘R’—a software environment for statis-

tical computing and graphics (www.r-project.org). A web-based calculator

was also developed using the parameter estimates for the logistic regres-

sion model to serve as a tool for predicting the likelihood of DGF for an

individual or patient population.

Statistical analyses were performed by the primary author (Dr. Irish) using

SAS for windows software (SAS Institute, Cary, NC), except for the general-

ized additive regression analysis, which utilized the GAM function of S-Plus

for windows software (Tibco Software Inc., Palo Alto, CA). Kaplan–Meier

method (19) was used to estimate graft survival among populations strat-

ified by predicted probability of DGF . Cox hazards model (20) was used to

calculate the hazards ratio (HR) of graft failure associated with DGF risk

strata. The relationship between the odds of DGF and risk of graft failure

was evaluated using ordinary least-squares linear regression.

Continuous data are presented as mean ± standard deviation (SD) and

categorical data are reported as counts and percentages.

Results

Patient population

Data on 24653 adult, nonpreemptive, non-machine-

perfused, deceased renal transplant recipients were avail-

able for analysis. Kidneys transplanted with other or-

gans (kidney–pancreas, kidney–liver, kidney–heart were ex-

cluded from the analysis); double adult kidney recipients

and recipients of pediatric en bloc kidney were included in

the analysis. Graft losses within the first 24 h (n = 315)

and patients where DGF was unknown (n = 1) were ex-

cluded. A multivariable logistic regression model was fit to

the data for the remaining patients. Recipient and donor

characteristics are presented in Table 1. Compared to the

previous era, mean donor age increased (34.1 ± 16.9 vs.

36.7 ± 16.7) and there were more donors over the age of

60 (6.8% vs. 7.2%). There was also an increase in donation

after cardiac death (1.1% vs. 3.1%) and in donor hyperten-

sion (16.0% vs. 22.4%). In this analysis, 13.8% of donor

kidneys met the current UNOS definition for ECD kidney.

Predictive model

Incidence of DGF between 2003 and 2006 was 25.7%.

Several variables (HLA mismatches, donor age, peak panel

reactiveantibody[PRA],terminalserumcreatinineandcold

ischemia time [CIT]) had a similar functional relationship

with the log odds of DGF as previously described (2). For

donor age, the log odds of DGF increases linearly after

approximately 16 years of age, but remains relatively con-

stant for donor ages less than 16. WIT was linearly related

to the log odds of DGF , while donor weight and dialysis du-

ration had a nonlinear relationship and were incorporated

into the model using a second-order quadratic term.

Results of the model are presented in Table 2. The five

most significant factors associated with DGF , were CIT

2280

American Journal of Transplantation 2010; 10: 2279–2286

Page 3

DGF Risk Calculator

Table

transplantation

1: Recipientand donor characteristicsat timeof

Recipient demographics

Age (in years)

Race (%)

White

Black

Other

Sex (%)

Male

Female

Recipient clinical characteristics

Previous transplants (%)

Primary indication for renal transplantation (%)

Diabetes

Hypertension

Other

Missing

Peak PRA (in percent) (n)

Pretransplant transfusions (%)

Yes

No

Unknown

Total HLA mismatches @

HLA-A,-B,-DR loci (n)

Percent 6-antigen mismatch

Body mass index (kg/m2) (n)

Duration of dialysis prior to

transplantation (days) (n)

Donor demographics

Age (in years) (n)

Age ≥60 years

Donor clinical characteristics

Weight (kg)

Cold ischemia time (in hours) (n)

Total warm ischemia time

(in minutes) (n)

Donation after cardiac death (%)

History of hypertension (%)

Yes

No

Unknown

Terminal serum creatinine

(mg/dL) (n)

Expanded criteria donor1(%)

Primary cause of death (%)

Head trauma

Cerebrovascular accident/stroke

Anoxia

Other

N = 24,337

49.8 ± 13.7

11 314 (46.5)

7336 (30.1)

5687 (23.4)

14 822 (60.9)

9515 (39.1)

3001 (12.3)

5809 (24.1)

5145 (21.3)

13 178 (54.6)

205

17.8 ± 31.0 (23,489)

6119 (25.1)

14 945 (61.4)

3273 (13.4)

3.7 ± 1.9 (24,323)

13.4

27.0 ± 5.2 (23,526)

1409.2 ± 1019.1

(23,086)

36.6 ± 16.7 (24,333)

7.2%

76.0 ± 23.5

17.8 ± 7.8 (21,693)

39.3 ± 17.6 (14,577)

754 (3.1)

5447 (22.4)

18 721 (76.9)

169 (0.7)

1.0 ± 0.5 (24,226)

3353 (13.8)

10 987 (45.1)

9304 (38.2)

3406 (14.0)

640 (2.7)

1Defined as all donors older than 60 years of age or donors older

than 50 years with any two of the following conditions: (1) history

of hypertension; (2) terminal serum creatinine >1.50 mg/dL; and

(3) cerebrovascular cause of brain death.

(odds ratio [OR] per 1-h increase = 1.041; 95% CI = 1.036–

1.045),terminalserumcreatinine(ORper1mg/dLincrease

= 1.693; 95% CI = 1.579–1.815), recipient BMI (OR per

1 kg/m2increase = 1.043; 95% CI = 1.037–1.049), DCD

kidney (OR yes vs. no = 3.063; 95% CI = 2.614–3.589) and

donor age ≥16 years (OR per 1 year increase = 1.017; 95%

CI = 1.014–1.019). For factors measured on a continuous

scale, the OR represents odds of DGF per unit change in

the scale of the variable. For example, the OR for CIT is

1.041, which suggests the odds of DGF increases by 4%

for every 1-h increase in CIT. A 10-h increase would mean

that the odds of DGF increase by approximately 50%.

DGF Risk nomogram

A functional nomogram was developed graphically depict-

ing the impact of risk factors on the likelihood of DGF

(Figure 1A). Specific points relative to the impact on DGF

were assigned to the nine categorical variables. A point

scale was used to assign points to the 11 continuous vari-

ables, including two quadratic terms for donor weight and

duration of dialysis prior to transplantation. The nomogram

can be used to calculate the risk of DGF manually for an in-

dividual patient by adding the points associated with each

risk factor and using a straightedge to correlate the result-

ing point score with the corresponding predicted risk of

developing DGF . When calculating the points associated

with donor weight, both the donor weight point scale (Part

A) and the donor weight-squared point scale (Part B) need

to be taken into account. For example, if a donor weighed

80 kg then the total of points contributed to DGF risk would

be Part A + Part B = 20 + 11 = 31 points. A total point

score of approximately 230, for example, correlates with

the current average rate of DGF of 25% while a point score

of 265 correlates a 50% chance of DGF .

Web-based DGF risk calculator

Themathematical

dictive model was used to program a web-based

DGFrisk calculator,which

www.transplantcalculator.com/DGF.

transplant-relatedinformationspecifictoindividualpatients

or patient groups can be entered into this interactive tool

to obtain a predicted probability of developing DGF .

formula derivedfrom the pre-

can be

Demographic

accessed via

and

Model validation

The logistic regression model was validated using a sep-

arate data set of patients transplanted in 2007 meeting

the inclusion criteria of this study (n = 5234). Using the

final model an ROC curve was developed (Figure 1B). The

ROC curve lies above the line of unity signifying the predic-

tive accuracy of the model. The c index (or area under the

ROC curve) was 0.704, which indicates a good degree of

discrimination versus 0.665 when applying the 2003 DGF

model (2).

Further validation was performed by evaluating how well

themodeldiscriminatedamongsubgroupsofpatientscon-

sidered at risk for DGF (i.e. ECD and DCD recipients). Re-

sults presented in Table 3 suggest good agreement be-

tween the observed prevalence of DGF and the model-

based predicted probability of DGF within each risk cat-

egory. The predictive probability of DGF using the 2003

model (2) by ECD versus SCD was 0.580 and 0.433, re-

American Journal of Transplantation 2010; 10: 2279–2286

2281

Page 4

Irish et al.

Table 2: Results of the multivariable logistic regression analysis

95% Confidence

interval

Estimate Standard errorOdds ratioLowerUpper

Recipient

Race/ethnicity: black vs. non-black

Gender: male vs. female

Previous transplant: yes vs. no

Diabetes vs. other

Peak panel reactive antibody (per% increase)

Pretransplant transfusions: yes vs. no/unknown

HLA mismatches (per unit increase)

Body mass index (per unit increase)

Duration of dialysis (per day increase)

Duration of dialysis—squared

Donor

Donor age (per year increase beyond 16 years)

Cold ischemia time (per hour increase)

Warm ischemia time (per minute increase)

Donation after cardiac death: yes vs. no

History of hypertension: yes vs. no/unknown

Terminal serum creatinine (per mg/dL increase)

Donor cause of death: anoxia vs. other

Donor cause of death: cerebrovascular/stroke vs. other

Weight (per kg increase)

Weight-–squared

0.235

0.375

0.134

0.277

0.003

0.225

0.045

0.042

0.00051

−6.26E-8

0.034

0.039

0.053

0.037

0.00058

0.036

0.009

0.003

0.000042

8.4E-9

1.266

1.454

1.144

1.319

1.003

1.252

1.046

1.043

1.001

1.000

1.183

1.361

1.031

1.228

1.002

1.167

1.028

1.037

1.000

1.000

1.354

1.555

1.269

1.417

1.004

1.343

1.064

1.049

1.001

1.000

0.017

0.040

0.007

1.119

0.265

0.526

0.204

0.191

−0.011

0.00009

0.001

0.002

0.001

0.081

0.040

0.036

0.049

0.040

0.003

0.000015

1.017

1.041

1.007

3.063

1.303

1.693

1.226

1.210

0.989

1.000

1.014

1.036

1.005

2.614

1.205

1.579

1.114

1.119

0.984

1.000

1.019

1.045

1.010

3.589

1.410

1.815

1.349

1.310

0.995

1.000

Likelihood ratio chi-square = 2512.291; p < 0.0001 on 25 degrees of freedom.

spectively; which is much higher than the observed preva-

lence of DGF reported in Table 3. Agreement between

predicted probabilities of DGF and observed prevalence

of DGF is provided in Figure 2A. The plot shows good

agreement between the predicted probabilities and the

observed prevalence of DGF .

Net benefit of the prediction model for different threshold

probabilities is provided in Figure 2B. If we assume that

theharmofunnecessarytreatmentofDGF(afalse-positive

decision)isrelativelylimited,thecut-offshouldbebetween

10% and 30%. In contrast, if treatment is quite harmful,

we should use a higher cut-off (>40%) before a treatment

decision is made.

Predicted probability of DGF and graft survival

AKaplan–Meier plotof graftsurvivalforpatientpopulations

stratified by predicted probability of DGF is presented in

Figure 3. Patients with a probability of DGF between 25%

and 50% are associated with a 50% increase in the risk of

graft failure relative to patients with a probability of DGF

less than 25%, whereas patients with a predicted proba-

bility of DGF greater than 50% are associated with a 2-fold

increase in the risk graft failure. The relationship between

the odds of DGF and risk of graft failure can be mathemat-

ically defined as HR = OR0.387, whereas a 2-fold increase

in the predicted odds of DGF is associated with a 30%

increase in the risk of graft failure.

Discussion

Since development of the original DGF prediction model

by Irish and colleagues (2), UNOS has instituted a new

allocation policy, which resulted in increased utilization of

ECD kidneys (21,22). In the last decade, the proportion of

deceased donors over age 50 increased from 21% to 30%

(23,24). Nonetheless, waiting times have doubled and the

waiting list has increased by 260%, while the number of

deceased donor renal transplants has increased by only

16% (22–24).

Immunosuppressiveprotocolshavealsochangedsincede-

velopment of the original model (25), as has knowledge of

how immunosuppression strategies can be used to miti-

gate DGF risk and improve early outcomes despite the use

of poorer quality organs.

Given the changing landscape, we updated the previous

model to reflect the current era of renal transplantation.

The study population was refined to exclude subsets of

patients likely to complicate evaluation of factors influ-

encing DGF (i.e. multiorgan, preemptive and machine-

perfused transplants) (2,26). To enhance predictive accu-

racy of the model, we included additional risk factors in

the updated analysis. Since the usability of a paper-based

nomogram to manually calculate DGF risk is limited (14),

the utility, accessibility and functionality of the updated

model were enhanced by making this tool available via

2282

American Journal of Transplantation 2010; 10: 2279–2286

Page 5

DGF Risk Calculator

Continuous Variables

Peak PRA (%)

Duration Dialysis Part A

Duration Dialysis Part B

Recipient BMI (kg/m2)

HLA Mismatch

Cold Ischemia Time (hours)

Warm Ischemia Time (minutes)

Donor Terminal Creatinine (mg/dL)

Donor Age – 16 years

Donor Weight (kg) : Part A

Donor Weight (kg) : Part B

8000

0

1020

30 40 50

6070

80

90

100

0 40 80

0

1000 20003000

40005000

60007000

8000

7000

6000

50004000

3000

1000

05 10

15

2025

30

3540 45

0 2 4 6

05

10

1520

25

30

35

40 45

0

30

6090

0 0.51 1.52 2.53 3.54

0 10

25

40 55

200160 120 80400

0 60 80

100 120

140

160

180

200

Points

Total Points

Risk of DGF

6

Donor Cause of Death-Cerebrovascular

6

Donor Cause of Death-Anoxia

6

Donor History of Hypertension

27

Donation after Cardiac Death

6

Recipient Pre-transplant Transfusion

8

Recipient Diabetes

5

Previous Transplant

9

Male Recipient

6

Black Recipient

Points Categorical Variables

Sensitivity

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1-Specificity

0.00.10.2 0.3 0.40.5 0.60.70.8 0.91.0

c-index = 0.704

Line of Unity

150200250 300 350

0.100.30 0.500.700.90

A

B

0.78 0.600.40 0.220.15

Figure 1: Delayed graft function (DGF) risk nomogram. (A) A functional nomogram graphically depicting the impact of risk factors on

predicting the likelihood of DGF . The nomogram can be used to calculate a patient’s risk of DGF by adding the points associated with

each risk factor and using a straightedge to correlate the resulting point score with the corresponding predicted risk of developing DGF .

(B) The receiver operating characteristic (ROC) curve representing the predictive accuracy of the DGF prediction model. The concordance

c index (or area under the ROC curve) was 0.704.

the world-wide-web. Demographic and transplant-related

information specific to individual patients or patient pop-

ulations can be entered into the novel interactive tool,

www.transplantcalculator.com/DGF, to obtain a predicted

probability of developing DGF , which also can be used to

predict long-term graft survival immediately prior to trans-

plantation.

The relative impact of most risk factors remained consis-

tent with the original model, with the exception of two

immunological factors (peak PRA and history of a previous

transplant), which diminished by approximately 50%, and

an important donor factor (terminal serum creatinine), the

impact of which increased by approximately 150%. The at-

tenuated effect of these immunological factors may be due

in part to improvements in the sensitivity of HLA antibody

testing, optimization of immunosuppression regimens and

greater use of antibody induction (26). However, this is off-

set by a greater contribution of the relative impact of donor

renal function, possibly as a result of the increased use of

poorer donors. Thus, the DGF rate has remained relatively

constant over time.

In this analysis, WIT and recipient BMI were found to be

linearly related to DGF risk, while duration of pretransplant

dialysis and donor weight were found to have a nonlinear

relationship. These findings are unique in that no study has

previously examined the functional relationship of these

emerging risk factors. Interestingly, the effect of pretrans-

plant dialysis was more pronounced up to 2000 days,

whereas the incremental change in the risk of DGF is 0.160

per year compared to 0.086 per year beyond 2000 days.

The reasons underlying this relationship are not clear but

may be related to diminished renal function with long-term

dialysis potentially altering the environment into which the

new kidney is transplanted (10).

American Journal of Transplantation 2010; 10: 2279–2286

2283

Page 6

Irish et al.

Table 3: Observed prevalence (and 95% confidence interval) ver-

sus predicted probability of delayed graft function (DGF) by risk

group

Observed

prevalence

of DGF

0.482 (0.419, 0.545)

Predicted

probability

of DGF

0.459

Risk group

Donation after

cardiac death

(n = 254)

Donation after brain

death (n = 4910)

Expanded criteria

donors1(n = 698)

Standard criteria

donors (n = 4466)

1Defined as all donors older than 60 years or donors older than

50 years with any two of the following conditions: (1) history of

hypertension; (2) Terminal serum creatinine >1.50 mg/dL; and (3)

cerebrovascular cause of brain death.

0.255 (0.243, 0.267)0.261

0.341 (0.306, 0.378) 0.377

0.255 (0.242, 0.268)0.254

Figure 2: (A) Predicted probability of delayed graft function

(DGF) by observed prevalence of DGF. Validation set was di-

vided into five equal groups based on the rank order of the pre-

dicted probabilities of DGF . Within each group, the mean predicted

probability of DGF was calculated (x-axis) and plotted against the

observedprevalenceofDGF(y-axis)forthatgroup.Theplotshows

good agreement between the predicted probabilities and the ob-

served prevalence of DGF . (B) Decision curve for predicted proba-

bilities of DGF based on the validation set. The performance of a

prediction model can be summarized as a net benefit: NB = (TP

− wFP)/N where TP is the number of true-positive decisions, FP

the number of false-positive decisions, N is the total number of

patients and w is a weight equal to the odds of threshold prob-

ability or the ratio of harm to benefit. For a threshold probability

of 0.25, net benefit of the model is 0.089. Using the model is the

equivalent of a strategy that identified the equivalent of 8.9 cases

of DGF per 100 patients with no unnecessary treatment.

Kaplan-Meier Graft Survival by Predicted Probability of DGF Strata

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Graft Survival Probability

Years Post-Transplant

10% Probability of DFG

10-25% Probability of DFG

25-50% Probability of DFG

Figure 3: Kaplan–Meier graft survival by predicted probability

of delayed graft function (DGF) strata. A Kaplan–Meier plot of

graft survival by strata defined on the predicted probability of DGF .

Patients with a predicted probability of DGF between 25% and

50% are associated with a 50% increase in the risk of graft failure

versus patients with a predicted probability of DGF less than 25%

(Hazard ratio [HR] = 1.523; 95% CI = 1.429–1.623). Patients with

a predicted probability of DGF greater than 50% are associated

with a 2-fold increase in the risk graft failure relative to patients

with a predicted probability of DGF less than 25% (HR = 2.165;

95% CI = 1.960–2.393). The relationship between the odds of

DGF and risk of graft failure can be mathematically defined as

HR = OR0.387, whereas a 2-fold increase in the predicted odds of

DGF is associated with a 30% increase in the risk of graft failure.

A strength of this study is that the model was validated

using a separate analysis data set that included patients

transplanted in 2007. The c index, a standard measure

of discrimination was 0.704, suggesting the model has

good discrimination. The ability of the model to discrim-

inate between subgroups of patients at high versus low

risk for DGF (ECD vs. standard criteria donors and DCD vs.

brain dead donors) (27–30) further suggests good discrim-

inatory power. However, the model should not be used

as a basis for clinical decisions but can be used as tool

to complement the decision making process. We have

provided a decision curve that depicts the net increase

in the proportion of appropriately treated patients over a

range of threshold probabilities of DGF . These threshold

probabilitiesrepresentpossibletriggersforinfluencingclin-

ical decision making. Decision curve analysis allows one

to vary the threshold probability over an appropriate range.

This is important because, often, either: (1) there are in-

sufficient data on which to calculate a rational threshold

or (2) patients/clinicians may disagree about the appropri-

ate threshold, due to different preferences for alternative

health states.

Another important finding of our analysis is that we have

used this model to show that increasing categories of DGF

risk are associated with increased risk of graft failure, such

that a 2-fold increase in DGF risk is associated with a 30%

increase in the risk of graft failure, irrespective of whether

DGF occurs or not. For example, a predicted probability of

2284

American Journal of Transplantation 2010; 10: 2279–2286

Page 7

DGF Risk Calculator

DGF between 25% and 50% was found to be associated

with a 50% increase in the risk of graft failure relative to

a predicted probability of DGF less than 25%, whereas a

predicted probability of DGF greater than 50% was found

to be associated with a 2-fold increase in the risk graft

failure.WhiletheprecisecontributionofDGFtograftlossis

debatable, the detrimental clinical effect of DGF after renal

transplantation is well documented (1,3) and many studies,

includingours,haveshownDGFtobeapredisposingfactor

for decreased graft survival (31). This is significant as the

predictive value for graft survival can be determined at the

time of transplant and does not require analysis of serum

creatinine or other factors at 6 months or 1 year or later

after transplantation.

These results indicate the potential utility of the model,

not only for predicting a patient’s risk of developing DGF

but for predicting risk of DGF in different cohorts of pa-

tients and its impact. For example, the model could be

used to compare the observed incidence of DGF versus

predicted at a given center or under a specific immuno-

suppression protocol. Moreover, the model could be used

to evaluate the impact of specific risk factors or different

donor pairs on the risk of DGF and 5-year survival. Thus,

one potential use of the model would be to determine or-

gan allocation. The model could also be used as a research

tool for identifying patients at higher risk for DGF for a clin-

ical trial either as an inclusion/exclusion criterion or as a

stratification factor for randomization. Although there are

no currently approved therapies for the prevention of DGF ,

the ability to predict a patient’s likelihood of developing

DGF may be useful in guiding clinical practice. A greater

risk of DGF may enhance the need to minimize additional

insults to the kidney by means of minimizing acute rejec-

tion, nephrotoxicity and ischemia-reperfusion injury with

strategies such as early introduction of polyclonal induc-

tion, delayed initiation of calcineurin inhibitors and poten-

tially the use of now-investigational compounds that aim

to prevent ischemia-reperfusion injury.

There are limitations to our study, most notably the na-

ture of the definition of DGF . Typically, DGF is defined by

requirement for dialysis within the first week posttrans-

plant. However, postoperative requirement of dialysis is

not standardized and the decision to dialyze is subjective

(27).Thishasledtoconsiderablevariationincenter-specific

incidences of DGF . While we aimed to minimize techni-

cal failures and cases of primary nonfunction by excluding

grafts lost within 24 h of transplant (mean predicted risk

of DGF in the 315 patients excluded was 32.4%), it re-

mains impossible to uniformly identify diagnoses of true

DGF-based solely on objective criteria. With this degree of

uncertainty, it is unlikely that any statistical model can pre-

dict with high degree of accuracy the probability of DGF

(27,32).

It is also important to recognize that there must be a bal-

ance between identifying a ‘clean’ sample population on

which to base a model, and excluding too many patients

on the basis of missing data elements, as these patients

may still confer additional predictive value to the model.

In our model, we chose not to exclude patients that were

missing certain data elements and rather we accounted

for missing variables in the model in order to retain a

more robust sample size. The use of machine-perfusion

differs between transplant centers; hence, we excluded it

from the updated model. Its use, however, is becoming

increasingly common, especially in ECD and DCD kidneys.

A recent multicenter clinical trial has shown that machine-

perfusion is associated with lower rates of DGF versus

cold storage (26). We found that use of our model to pre-

dict the likelihood of DGF in a cohort of patients whose

donor organs were machine-perfused overestimated the

risk of DGF by approximately 10%. This finding suggests

the need for either a separate model for DGF in machine-

perfusedtransplantsor modificationofthismodeltoadjust

for the use of machine-perfusion.

Over the past decade there have been significant advances

in the field of transplantation, particularly with regard to im-

munosuppression and the increased utilization of ECD and

DCD kidneys (15). In light of these changes, we set out

to develop a predictive model for DGF in the modern era

of deceased donor renal transplantation. We showed that

an increased risk of DGF based on our model is also as-

sociated with inferior long-term allograft survival. We also

improved upon the utility of the model by developing a

web-based DGF risk calculator with potential applicability

as a tool for defining individuals or study populations at

greatest risk for DGF , or designing clinical trials whose ob-

jective is to evaluate the impact of immunosuppressive

strategies or novel agents on the development of DGF and

long-term graft survival.

Acknowledgments

Funding Source: This work was supported by Genzyme Corporation.

Disclosure

The authors of this manuscript have no conflicts of inter-

est to disclose as described by the American Journal of

Transplantation.

References

1. Shoskes DA, Cecka JM. Deleterious effects of delayed graft func-

tion in cadaveric renal transplant recipients independent of acute

rejection. Transplant 1998; 66: 1697–1701.

2. Irish WD, McCollum DA, Tesi RJ et al. Nomogram for predicting

the likelihood of delayed graft function in adult cadaveric renal

transplant recipients. J Am Soc Nephrol 2003; 14: 2967–2974.

3. Ojo AO, Wolfe RA, Held PJ, Port FK, Schmouder RL. Delayed graft

function: Risk factors and implications for renal allograft survival.

Transplant 1997; 63: 968–974.

American Journal of Transplantation 2010; 10: 2279–2286

2285

Page 8

Irish et al.

4. Tropmann C, Gillingham KJ, Bennedetti E et al. Delayed graft func-

tion, acute rejection, and outcome after cadaveric renal transplan-

tation: The multivariate analysis. Transplant 1995; 59: 962–968.

5. Yarlagadda SG, Coca SG, Formica RN, Poggio ED, Parikh CR. Asso-

ciationbetweendelayedgraftfunctionandallograftandpatientsur-

vival: A systematic review and meta-analysis. Nephrol Dial Trans-

plant 2009; 24: 1039–1047.

6. Rosenthal JT, Danovitch GM, Wilkinson A, Ettenger RB. The high

cost of delayed graft function in cadaveric renal transplantation.

Transplant 1991; 51: 1115–1118.

7. Almond PS, Troppmann C, Escobar F, Frey DJ, Mata AJ. Eco-

nomic impact of delayed graft function. Transplant Proc 1991; 23:

1304.

8. Freedland SJ, Shoske DA. Economic impact of delayed graft func-

tion and suboptimal donors. Transplant Rev 1999; 13: 23–30.

9. Harper SJ, Hosgood SA, Waller HL et al. The effect of warm is-

chemic time on renal function and injury in the isolated hemoper-

fused kidney. Transplantation 2008; 85: 88–92.

10. Keith DS, Cantarovich M, Paraskevas S, Tchervenkov JK. Duration

of dialysis pretransplantation is an important risk factor for de-

layed recovery of renal function following deceased donor kidney

transplantation. Transpl Int 2008; 21: 126–132.

11. Figueiredo A, Moreira P, Parada B et al. Risk factors for delayed

renal graft function and their impact on renal transplantation out-

come. Transplant Proc 2007; 39: 2473–2475.

12. DraftsHH,AnjumMR,WynnJJ,MulloyLL,BowleyJN,Humphries

AL. The impact of pre-transplant obesity on renal transplant out-

comes. Clin Transplant 1997; 11: 493–496.

13. Meier-Kriesche HU, Kaplan B. Waiting time on dialysis as the

strongest modifiable risk factor for renal transplant outcomes: A

paired donor kidney analysis. Transplantation 2002; 73: 70–74.

14. Grimes DA. The nomogram epidemic: Resurgence of a medical

relic. Ann Intern Med 2008; 149: 273–275.

15. UNOS Policy 3.5.1. Expanded criteria donor definition and point

system. Richmond, VA: United Network for Organ Sharing, 2002:

1–26.

16. Hosmer DW, Lemeshow SL. Applied logistic regression. New

York: Wiley, 1989.

17. Hastie T, Tibshirani R. Generalized additive models. London: Chap-

man and Hall, 1990.

18. Vickers AJ, Elkin EB. Decision curve analysis: A novel method for

evaluating prediction models. Med Decis Making 2006; 26: 656–

574.

19. Kaplan EL, Meier P. Nonparametric estimation from incomplete

observations. J Am Stat Assoc 1958; 53: 457–481.

20. Cox DR. Regression models and life time tables. J R Stat Soc B

1972; 34: 187–202.

21. Stratta RJ, Rohr MS, Sundberg AK et al. Increased kidney trans-

plantation utilizing expanded criteria deceased organ donors with

results comparable to standard criteria donor transplant. Ann Surg

2004; 239: 688–697.

22. Metzger RA, Delmonico FL, Feng S et al. Expanded criteria donors

for kidney transplantation. Am J Transplant 2003; 3: 114–125.

23. Port FK. Organ donation and transplantation trends in the United

States, 2001. Am J Transplant 2003; 3: 7–12.

24. NathanHM,ConradSL,HeldPJetal.OrgandonationintheUnited

States. Am J Transplant 2003; 3: 29–40.

25. Meier-Kriesche H-U, Li S, Gruessner RWG et al. Immunosuppres-

sion: Evolution in practice and trends, 1994–2004. Am J Transplant

2006; 6: 1111–1131.

26. Moers C, Smits JM, Maathuis M-HJ et al. Machine perfusion or

cold storage in deceased-donor kidney transplantation. N Engl J

Med 2009; 360: 7–19.

27. Yarlagadda SG, Coca SG, Garg AX et al. Marked variation in the

definition and diagnosis of delayed graft function: A systematic

review. Nephrol Dial Transplant 2008; 23: 2995–3003.

28. OjoAO,HansonJA,Meier-KriescheHetal.Survivalinrecipientsof

marginal cadaveric donor kidneys compared with other recipients

and wait-listed transplant candidates. J Am Soc Nephrol 2001; 12:

589–597.

29. Port FK, Bragg JL, Metzger RA et al. Donor characteristics asso-

ciated with reduced graft survival: An approach to expanding the

pool of kidney donors. Transplantation 2002; 74: 1281–1286.

30. Weber M, Dindo D, Demartines N, Amb¨ uhl PM, Clavien P-A. Kid-

ney transplantation from donors without a heartbeat. N Engl J

Med 2002; 347: 248–255.

31. Hassanain M, Tchervenkov J, Cantarovich M et al. Delayed graft

function has an equally bad impact on deceased donor renal graft

survival in both standard criteria donors and expanded criteria

donors. Transplant Proc 2009; 41: 133–134.

32. Daly P, Power RE, Healy DA. Delayed graft function: A dilemma in

renal transplantation. BJU International 2005; 96: 498–501.

2286

American Journal of Transplantation 2010; 10: 2279–2286