Impact of Outcomes Monitoring on Innovation
and Risk in Liver Transplantation
John Paul Roberts
Division of Transplantation, University of California San Francisco, San Francisco, CA
1. The reporting of liver transplant center outcomes
is required by the final rule of the Department of
Health and Human Services. The reported patient and
graft survival outcomes are risk-adjusted for specific
donor and recipient factors, and the observed survival
is compared to the expected survival. Both the Cen-
ters for Medicare and Medicaid Services and the Organ
Procurement and Transplantation Network flag pro-
grams for corrective action when the observed sur-
vival is significantly less than the expected survival.
Both agencies can take action up to the closure of a
center. In the last 5 years, the Organ Procurement and
Transplantation Network has not taken an adverse
action that required the closure of a liver transplant
center because of outcomes.
2. Center survey data suggest that centers may try to
select donors and recipients to minimize poor out-
comes. This strategy may not be effective if centers
stop accepting donors or recipients according to fac-
tors that are included in the risk adjustment model.
For example, limiting recipients to those less than
65 years old may improve the observed outcomes,
but the expected outcomes will also improve because
a recipient 65 years or older is included in the mod-
el’s risk adjustment.
3. For factors such as cardiovascular risk that are not
included in the model, it may be reasonable to exclude
patients in an attempt to improve the observed out-
comes without affecting the expected outcomes. Other
examples of these types of factors are smoking, nutri-
tional status, and donor liver biopsy findings.
4. Currently, there is no exemption for patients
undergoing experimental protocols. Down-staging for
hepatocellular carcinoma, transplantation for human
immunodeficiency virus–positive recipients, and the
use of left lobe grafts with inflow modification are
relatively recent areas of innovation in liver trans-
plantation. Because innovation is frequently associ-
ated with a learning curve and, therefore, poor
outcomes, the inclusion of patients in innovative
protocols potentially could lead to centers being
subjected to an adverse action by the Organ Procure-
ment and Transplantation Network or the Centers
for Medicare and Medicaid Services. Active consider-
ation is being given to the exclusion of patients in
innovative protocols from center-specific outcomes.
Liver Transpl 18:S59-S63, 2012. V
C 2012 AASLD.
Received July 9, 2012; accepted August 7, 2012.
Outcomes monitoring has existed for liver transplan-
tation for many years. Initially, center outcomes were
used to designate centers of excellence by insurance
companies. The Organ Procurement and Transplanta-
tion Networkbegan using
adjusted patient and graft survival models produced
by the Scientific Registry of Transplant Recipients to
flag centers whose observed outcomes were less than
expected. Centers submit data on recipients and liv-
ing donors, and organ procurement organizations sub-
mit data on donors. Multivariate Cox models are cre-
ated to compare expected patient and graft outcomes
based on the collection of all submitted data (national
data) to a center’s observed outcomes based on indi-
vidual patient characteristics.1The Scientific Registry
of Transplant Recipients publicly reports these center-
specific outcomes every 6 months, and they are used
by both payers and patients to decide the relative
Abbreviation: CUSUM, cumulative sum.
Potential conflict of interest: Nothing to report.
Address reprint requests to John Paul Roberts, M.D., Division of Transplantation, University of California San Francisco, 505 Parnassus Avenue,
Box 0780, San Francisco, CA 94143-0780. Telephone: 415-353-1590; FAX: 415-353-8709; E-mail: firstname.lastname@example.org
View this article online at wileyonlinelibrary.com.
LIVER TRANSPLANTATION.DOI 10.1002/lt. Published on behalf of the American Association for the Study of Liver Diseases
LIVER TRANSPLANTATION 18:S59-S63, 2012
Liver Transplantation, Vol 18, No 11, Suppl 2 (November), 2012: pp S59-S63
performance of transplant centers. The Membership
and Professional Standards Committee of the Organ
Procurement and Transplantation Network has a peer
review process to help flagged centers to improve their
outcomes. Figure 1 provides an example of the public
report format available from the Scientific Registry of
Transplant Recipients Web site.2
The Centers for Medicare and Medicaid Services
began to use Scientific Registry of Transplant Recipi-
ents data to flag centers for corrective action in 2007.
This has led to both voluntary and involuntary clo-
sures of centers, although these primarily have been
kidney transplant programs. There is evidence that
centers have changed their behavior because of con-
cerns about the potential effects of being flagged. Fig-
ure 2 demonstrates this effect in a survey of trans-
Although the processes of both the Centers for Med-
icare and Medicaid Services and the Organ Procure-
ment and Transplantation Network have required cen-
ter site surveys and corrective action plans, many
centers have had observed outcomes significantly
lower than the expected outcomes for many years.
Figure 3 presents an examination of the reports
posted between January 2005 and July 2010, which
included 10 reporting periods. As can be seen, many
centers had multiple periods in which graft survival
was significantly less than expected, and 1 center had
outcomes worse than expected in all periods. These
data suggest that the process does not rapidly change
a center’s results.
One reason for this lack of change is that the
reporting periods are 2.5 years in length. This means
that only 20% of the patients in a report turn over in
any reporting period, so it is difficult for the Organ
Procurement and Transplantation Network, the Cen-
ters for Medicare and Medicaid Services, and the cen-
ter to decide whether or not corrective action plans
have been effective. This slow report turnover also
creates uncertainties for centers that may wish to
take action on outcomes before they are flagged.
One possible solution is to use an outcomes signal-
ing measure: the cumulative sum (CUSUM). The
CUSUM tool meets many of the requirements for an
ideal monitoring tool in that it can be timely, is easily
interpretable, and can be risk-adjusted in the same
manner in which center outcomes are reported. The
CUSUM is a graphical representation of the changes
in risk-adjusted outcomes over time. Changes in the
plot line suggest improving or declining outcomes.
specific outcomes every 6 months. These reports are used by both payers and patients to decide the relative performance of transplant
centers. Reprinted with permission from the Scientific Registry of Transplant Recipients.2Copyright 2012, Scientific Registry of
Example of a public report available from the Scientific Registry of Transplant Recipients Web site, which provides center-
S60 ROBERTS LIVER TRANSPLANTATION, November 2012
Figures 4 and 5 present CUSUM plots for 2 kidney
transplant programs: one with declining performance
and another with improving performance.4If a center
updates its data on a frequent basis, it can act early
on deviations of observed outcomes from expected
outcomes and try to improve reported outcomes.
Improvements in a center’s outcomes may be effected
by changes in specific care processes at the center. It is
more difficult to improve a center’s observed outcomes
in comparison with expected outcomes from the Scien-
tific Registry of Transplant Recipients risk-adjusted
model through changes in donor or recipient charac-
teristics.5The following is a partial list of the donor and
recipient characteristics included in the model in the
order of their importance.
Recipient factors: retransplantation, life support,
malignant neoplasms other than hepatocellular
carcinoma, functional status, portal vein thrombo-
sis, recipient age ? 65 years, hepatitis C virus, re-
cipient age of 60 to 64 years, abdominal surgery,
creatinine, and albumin.
Donor factors: donation after cardiac death, split
liver, donor age > 70 years, ischemic time ? 12
hours, ischemic time of 9 to 11 hours, race, and
cause of death.
A center could try to improve its observed outcomes
by making changes in the recipients through the elim-
ination of transplantation for patients older than 65
years. This could result in an improvement in the cen-
ter’s observed outcomes, but because this is con-
tained in the risk adjustment model for recipients, the
expected outcomes would also improve, and a change
in thedifference between
observed survival might not occur. If the center
actually had better outcomes for patients older than
65 years in comparison with other centers in the
nation, the difference could actually increase. Simi-
larly, if the transplant center used deceased cardiac
donors who were significantly older or had longer
warm ischemia times in comparison with the national
average, the center’s observed results could be worse
Because not all clinically important factors are
included in the risk adjustment models, a center may
be able to improve its observed outcomes by minimiz-
ing transplantation for patients with these clinically
significant factors, such as cardiovascular disease.
Other potential areas for risk mediation are nutri-
tional status, smoking, and steatosis on donor liver
are excluded from proportions).
Reprinted with permission from
Copyright 2010, North American
Transplant Coordinators Organi-
center’s observed graft survival was significantly less than expected.
The data have been taken from Scientific Registry of Transplant
Recipients center-specific reports2(January 2005 to July 2010).
Number of reporting periods in which an individual
LIVER TRANSPLANTATION, Vol. 18, No. 11, 2012ROBERTSS61
Figure 6 presents changes in the cardiovascular
risk profile for patients receiving liver transplants
since the Centers for Medicare and Medicaid Services
began site surveys in 2007.6
The potential effects of the flagging of worse-than-
expected outcomes on innovations in transplantation
are of great concern.7
Liver transplantation has
always been at the cutting edge of medical therapy,
and the field cannot move forward without some risk
taking by transplant centers with patients, donors,
and therapies. Recent examples of innovations in liver
transplantation include the down-staging of hepato-
cellular cancers to the Milan criteria, transplantation
for human immunodeficiency virus–positive patients,
and the use of left lobe grafts with inflow modulation.
Trials of new therapies may turn out badly and result
in decreased patient and/or graft survival. If the trials
involve a significant number of a center’s patients, the
center’s outcomes may be flagged.
How do we allow for innovation within the program-
specific report framework? One potential method is to
allow the designation of a therapy as an innovation
and exclude these patients from the risk adjustment
models. Reaching consensus with the Organ Procure-
ment and Transplantation Network and the Centers
for Medicare and Medicaid Services is going to require
a significant amount of work. Important decisions
need to be made about the process for the inclusion
of protocols, patients, consent processes, and so
forth. It is hoped that progress can be made to allow
innovations to move the field of liver transplantation
1. Schaubel DE, Dykstra DM, Murray S, Ashby VB, McCul-
lough KP, Dickinson DM, et al. Analytical approaches for
transplant research, 2004. Am J Transplant 2005;5(pt 2):
2. Scientific Registry of Transplant Recipients. US hospitals
with liver transplant centers. http://www.srtr.org/csr/
Accessed August 2012.
evidence of improvement consistent with a learning curve effect.
Transplantation.4Copyright 2006, John Wiley and Sons, Inc.
CUSUM chart of a renal transplant center with
American Journal of Transplantation.4Copyright 2006, John
Wiley and Sons, Inc.
CUSUM chart of a renal transplant center with
Medicare and Medicaid Services began site surveys in 2007.
Reprinted with permission from Hepatology.6Copyright 2011,
John Wiley and Sons, Inc.
S62ROBERTSLIVER TRANSPLANTATION, November 2012
3. Schold JD, Arrington CJ, Levine G. Significant alterations
in reported clinical practice associated with increased
oversight of organ transplant center performance. Prog
4. Axelrod DA, Guidinger MK, Metzger RA, Wiesner RH,
Webb RL, Merion RM.
adjusted technique (CUSUM). Am J Transplant 2006;6:
5. Dickinson DM, Arrington CJ, Fant G, Levine GN, Schau-
bel DE, Pruett TL, et al. SRTR program-specific reports on
outcomes: a guide for the new reader. Am J Transplant
6. Wang E, Lyuksemburg V, Abecassis MM, Skaro AI.
Donor and recipient risk aversion in liver transplanta-
tion [oral presentation]. Hepatology 2011;54(suppl 1):
7. Abecassis MM, Burke R, Klintmalm GB, Matas AJ, Merion
RM, Millman D, et al.; for American Society of Transplant
transplant center outcomes requirements—a threat to
innovation. Am J Transplant 2009;9:1279-1286.
LIVER TRANSPLANTATION, Vol. 18, No. 11, 2012ROBERTSS63