Bias and asymmetric loss in expert forecasts: a study of physician prognostic behavior with respect to patient survival.
ABSTRACT We study the behavioral processes undergirding physician forecasts, evaluating accuracy and systematic biases in estimates of patient survival and characterizing physicians' loss functions when it comes to prediction. Similar to other forecasting experts, physicians face different costs depending on whether their best forecasts prove to be an overestimate or an underestimate of the true probabilities of an event. We provide the first empirical characterization of physicians loss functions. We find that even the physicians subjective belief distributions over outcomes are not well calibrated, with the loss characterized by asymmetry in favor of overpredicting patients' survival. We show that the physicians' bias is further increased by (1) reduction of the belief distributions to point forecasts, (2) communication of the forecast to the patient, and (3) physicians own past experience and reputation.

Article: A random world is a fair world.
Proceedings of the National Academy of Sciences 01/2013; · 9.81 Impact Factor  SourceAvailable from: James Wellehan[Show abstract] [Hide abstract]
ABSTRACT: Reptile medicine is a broad field with a relatively incomplete data set to work from in the relevant literature. Despite the popularity of pet reptiles, few veterinary schools provide significant education in reptile medicine. The lack of professional veterinary medical education presents challenges to the clinician seeking to provide the best care for their reptile patients. R eptiles are increasingly popular pets. In Britain, the number of pet reptiles has been reported to exceed the number of pet dogs. 1 The field of reptile medicine presents unique challenges to the clinician due, in large part, to the diversity of species that commonly present as pets. There are significant differences in reptile physiology from the common domestic mammals, and owners frequently lack basic knowledge of appropriate care for these animals. There is a paucity of wellsupported information on the topic of reptile medicine and an abundance of misinformation. A recent survey found that only 4 North American Veterinary colleges provided more than 6 hours of lecture on reptile medicine, despite 4% of households in the United States owning reptiles. 2 Given these limitations, it is essential for the reptile clinician to be able to identify and evaluate information on a topic and to make the best decisions possible based on incomplete data.Journal of Exotic Pet Medicine 07/2009; 18(3):194201. · 0.40 Impact Factor 
Article: Computers, conversation, utilization, and commoditization: the 2008 Herbert Abrams Lecture.
American Journal of Roentgenology 06/2009; 192(5):137581. · 2.74 Impact Factor
Page 1
Journal of Health Economics 27 (2008) 1095–1108
Contents lists available at ScienceDirect
Journal of Health Economics
journal homepage: www.elsevier.com/locate/econbase
Bias and asymmetric loss in expert forecasts: A study of physician
prognostic behavior with respect to patient survival
Marcus Alexandera,∗, Nicholas A. Christakisb,c
aHarvard University, Institute for Quantitative Social Science and Department of Government, CGIS 1737 Cambridge Street, Cambridge 02138, United States
bDepartment of Health Care Policy, Harvard Medical School, 180 Longwood Avenue, Boston, MA 02115, United States
cDepartment of Sociology, FAS, 33 Kirkland Street, Cambridge, MA 02138, United States
a r t i c l e i n f o
Article history:
Received 23 November 2006
Received in revised form 1 February 2008
Accepted 10 February 2008
JEL classification:
I10, I12, I19, D01, D80, C53
Keywords:
Loss function
Forecasting
Behavioral economics
Survival
Prognosis
a b s t r a c t
We study the behavioral processes undergirding physician forecasts, evaluating accuracy
and systematic biases in estimates of patient survival and characterizing physicians’ loss
functions when it comes to prediction. Similar to other forecasting experts, physicians face
different costs depending on whether their best forecasts prove to be an overestimate or an
underestimate of the true probabilities of an event. We provide the first empirical charac
terization of physicians’ loss functions. We find that even the physicians’ subjective belief
distributionsoveroutcomesarenotwellcalibrated,withthelosscharacterizedbyasymme
tryinfavorofoverpredictingpatients’survival.Weshowthatthephysicians’biasisfurther
increased by (1) reduction of the belief distributions to point forecasts, (2) communication
of the forecast to the patient, and (3) physicians’ own past experience and reputation.
© 2008 Elsevier B.V. All rights reserved.
In this paper, we investigate the accuracy of physicians’ forecasts of survival. We ask whether a physician’s prognosis
exhibits systematic biases, and we explore the sources of such biases. Our investigation uncovers a systematic tendency of
physicians to overpredict their patients’ survival at three stages: first, with respect to the survival distributions that doctors
construct, second in their summarization of this distribution through the selection of a point estimate, and third in their
choice about how to further modify this estimate during communication.
The strategic role of communication between physicians and patients has been studied by Caplin and Leahy (2004),
illustrating how the standard model of preferences breaks down once agents draw psychological utility from their beliefs.
Extending this model, Koszegi (2006) also used physician–patient communication to investigate how provision of informa
tion by experts becomes distorted in the presence of anticipatory feelings. These important theoretical contributions lay
the groundwork for empirically examining the systematic tendencies of physicians to distort their prognosis when both
formulating it and communicating it to their patients.
More specifically, findings from the literature on emotional agency lead us to expect that a closer relationship between
a physician and a patient should be associated with more upwardly biased loss. In this model, the physicians’ utility func
tion includes their patients’ emotional status, therefore providing an incentive for physicians to formulate an upwardly
biased prognosis. This theoretical framework also sheds light on why we would expect a doctor to be even more upwardly
biased when communicating than when formulating an expectation. It is clearly more emotionally stressful to share bad
news than merely to think about it. Additionally, communication provides for a strategic environment consistent with
∗Corresponding author. Institute for Quantitative Social Science and Department of Government, Harvard University, CGIS, 1737 Cambridge Street,
Cambridge, MA 02138, United States. Tel. +1 617 909 4618; fax: +1 617 432 5891.
Email address: malexand@fas.harvard.edu (M. Alexander).
01676296/$ – see front matter © 2008 Elsevier B.V. All rights reserved.
doi:10.1016/j.jhealeco.2008.02.011
Page 2
1096
M. Alexander, N.A. Christakis / Journal of Health Economics 27 (2008) 1095–1108
Koszegi’s (2006) model, whereby a physician has an opportunity as an agent to affect the emotional state of the patient as a
principal.
Furthermore, as Koszegi (2003) indicates, physician–patient communication may be accompanied by deeper psycholog
ical biases such as Samuelson’s (1963) fallacy of large numbers—simply defined as accepting a large number of unfavorable
gambles even when the agent is unwilling to take any one individual gamble on its own. In fact, some of the classic biases
in behavioral economics have been first characterized by studying physician behavior. These are most notably associated
with the process by which physicians formulate diagnosis, and they most famously include the role of hindsight in distorting
probability estimates (Arkes et al., 1981; Slovic and Fischhoff, 1977), base rate neglect (Casscells et al., 1978; Kahneman
and Tversky, 1973), and the conjunction fallacy (Tversky and Kahneman, 1983). In all of these situations, agents produce
inaccurate probability estimates given the uncertainty they face over the true state of the world.
Thekeyquestionthatariseshereishowphysiciansunderstandandprocesstheinformationabouttheirpatients’likelihood
of survival, and how they use their own subjective belief distributions to formulate a point forecast. In other words, even
before the strategic component of physician–patient communication enters the picture, we can ask whether systematic
biases characterize the process by which physicians arrive at their own best point forecast of patients’ survival.
In analyzing the asymmetry of physicians’ prognosis, it is useful to draw on the broader economic literature on expert
forecasts. In one of the first studies of asymmetric loss in economics, Varian (1974) documented an important fact that
experts in a market face different costs depending whether their best prediction is an overestimate or an underestimate of
the market price. In his study of the market for single family homes in a 1965 California town, Varian noticed that assessors
faced a significantly higher cost if they happened to overestimate the value of a house. While in the case of an underestimate,
the assessor’s office faced the cost in the amount of the underestimate, conversely, in the case of the overestimate by an
identical amount, the assessor’s office faced a possibility of a lengthy and costly appeal process. Since this classic study, loss
functions have become an important aspect of the study of expert forecasts.
The two key empirical puzzles surrounding the question of expert forecasts became to determine whether forecasters’
loss functions were symmetric, and if not, how optimal forecasts can be made given loss asymmetry, as addressed most
recently by Elliott et al. (2005). For example, government experts making budget forecasts may be influenced by political
incentives, as the costs of wrongly projecting a surplus may lead to public disapproval, while wrongly projecting a deficit
may lead to an impression of exceptional government performance. Artis and Marcellino (2001), as well as Campbell and
Ghysels (1995), document that budget deficit forecasts have asymmetric loss. Furthermore, expert opinion varies greatly and
systematically. For example, research by Lamont (1995) indicates that factors such as forecasters’ experience and reputation
are reliable determinants of experts’ willingness to deviate from consensus forecasts of GDP, unemployment, and prices.
In financial and macroeconomic forecasting, Granger and Newbold (1986) have concluded that economic theory does not
suggest that experts even should have a symmetric loss function. An improved understanding of behavioral biases arising in
agents’ decisions, such as those associated with loss functions, can contribute to answering puzzles about risky behavior in
the labor market and education decisions (e.g., Abowd and Card, 1989; Card and Hyslop, 1997; Card and Lemieux, 2001a,b)
and in health economics (e.g., Koszegi, 2003, 2006).
Because in most economic situations, such as Varian’s (1974) realestate market, agents formulate and report point
predictions as their forecasts, the agents’ true subjective belief distributions are lost and cannot be recovered from their
forecasts. Hence the problem of characterizing the loss function is compounded by the fact that we do not know anything
about the behavioral process by which agents reduce their belief distributions into singlepoint predictions, a process which
itself reflects the extent of asymmetry in their unobserved loss function. Furthermore, because of strategic considerations,
the prediction that agents communicate may be different from both the point prediction and the prediction implied by
the agents’ full subjective belief distributions. Unfortunately, due to data limitations, no study has been able to examine all
of these aspects of forecasting simultaneously. To date, the study of loss function asymmetry has been largely limited to
studying point forecasts (e.g., the Livingstone survey), while the study of forecasters’ fuller subjective belief distributions has
been confined to surveys of national output by experts (e.g., Survey of Professional Forecasters), as illustrated by the work
that originated with Victor Zarnowitz’s (1985) study of rational expectations.
Our study addresses the extent of intrinsic bias in forecast predictions and asks how the forecast bias and the symmetry
of the loss function change as agents move from a full subjective distribution to a point prediction and then to commu
nicating their formulated forecast. We focus on the first part of the processes because psychological research by Tversky
and Kahneman (1973, 1974) has documented that individuals exhibit different types of biases when using probability dis
tributions to infer a possibility of an outcome. Analogous to the biases that arise from the use of inference heuristics such
as representativeness or availability, agents may also exhibit biases when narrowing their subjective belief distributions
to singlepoint predictions. In particular, because the standard symmetric loss function requires minimization of the mean
squared error, individuals may show systematic bias due to failure to compute a correct mean or because t have asymmetric
loss. Much like econometric estimators that are biased when certain assumptions fail, the behavioral mechanism leading
to a point forecast from subjective beliefs may be biased due to computational limitations or a misinterpretation of the
optimization problem by agents. We also focus on the latter part of the process – the role of communication – because, with
the exception of independent, disinterested expert forecasters, the communication of an agent’s forecast is likely to play a
strategic role in a market. Therefore, any bias that led to formulation of the forecast may be further compounded by the
agent’s strategic biases in communicating the prediction. To study all of this, we need a record of forecasts that documents
both the process of reduction from subjective beliefs to a point forecast and the process of communication of that forecast.
Page 3
M. Alexander, N.A. Christakis / Journal of Health Economics 27 (2008) 1095–1108
1097
To investigate the problem, we use a unique prospective study of Chicago metropolitan area physicians referring termi
nally ill patients to hospice care. To our knowledge, this is the first survey that collected a combination of forecasts allowing
us to study all the aspects of the above question. First, the study asked physicians to make interval forecasts of the patients’
survival probability, approximating a subjective belief distribution. Second, the physicians’ best point prediction of survival
was recorded. Third, physicians were asked what prediction they would communicate to their patients. Fourth, a series of
questions regarding the physicians’ confidence, optimism, and experience was recorded, as well as the patients’ characteris
tics. Finally, the prospective nature of the study allowed us to compare the different forecasts to the patients’ actual survival
time. Together, these features of the data give us a unique opportunity to study loss functions that characterize physician
decisionmaking.
We quantify the extent of asymmetry in how much physicians value overpredicting versus underpredicting their
patients’ survival. We also demonstrate that the physicians’ bias increases when they communicate their prognosis to their
patients. The physicians’ own loss function becomes more asymmetric, favoring overprediction of survival, when they move
from formulating a point prediction to communicating a prognosis to their patients. We also show that the asymmetry in the
physicians’ loss function moves in the other direction when a fuller subjective belief distribution is elicited from the physi
cians. In contrast to the point forecast, the physicians’ bias decreases when they forecast a subjective probability distribution
over their patients’ odds of survival.
We also asked which physician and patient characteristics serve as determinants of the level of asymmetry in the physi
cians’ loss function. Our findings indicate that the patients’ gender, race, and type of disease, as well as the physicians’
experience, are important determinants. Together, these results point to the fact that physicians may rationally prefer to
overestimate survival of their patients. Given the economic and clinical nature of the doctor–patient relationship, overesti
mating the odds of a patient’s survival can be expected to serve as a commitment device to a prescribed choice of therapy
and contributes to the physicians’ sense of confidence. In the setting of hospice care in particular, evidence of upward bias
suggests that emotional agency described above comes to the forefront, playing an important role in physicians’ behavior
above and beyond the commitment mechanism observed elsewhere. However, the evidence that patients’ race and gender
play a role in the degree of loss asymmetry indicates that physicians’ forecasts are also subject to biases beyond a rational
calibration of the loss function.
Because forecasting of patients’ survival is an important part of the medical profession (Christakis, 1999), our character
ization of physicians’ loss functions serves two purposes: (1) in general, it carries implications for understanding behavior
of experts whose performance depends on forecast accuracy, and (2) more particularly, it has downstream implications for
understanding the supply of health care and for health care expenditures.
The paper is organized as follows. Section 1 introduces the data. Section 2 presents the method we use for estimation of
loss functions. Section 3 presents the results. Section 4 discusses the conclusions.
1. Data
1.1. Patient and physician data
To study prognostic accuracy and bias among physicians, we use data from a 1996 prospective cohort study, conducted in
the Chicago metropolitan area. The study approached all hospices in Chicago that admitted more than 200 patients per year.
Five of the six such hospices participated in the study, producing a cohort of all patients admitted during 130 consecutive
days in 1996 (Christakis and Lamont, 2000).
Forallpatientsinthestudy,thephysicianwhoreferredthepatienttohospicecarewasempaneled(noneoftheparticipat
ing physicians were the hospice medical directors). In some cases, the referring physician was the primary care doctor and in
others the physician was a specialist (such as the treating oncologist). We collected a prognosis from only one doctor for each
patient. We collected individual physician data (e.g., their sex, specialty, year of graduation from medical school, board cer
tification, etc.) and three variables that characterizes the relationship between the doctor and the patient, namely, duration
of contact (when they first met), frequency of contact, and recency of contact (when the doctor last examined the patient).
All physicians were surveyed at the same point in time, typically within 48h of the time they referred the patient to hospice.
The descriptive statistics are summarized in Table 1. We studied a total of 504 patients referred by 365 physicians. All
patients were followed until their deaths. At the time of hospice referral, all patients were terminally ill. The most frequent
diagnoses were lung cancer (18%), AIDS (12%), colorectal cancer (7%), breast cancer (6%), chronic heart failure (5%), and
stroke (5%).
The main variables of interest measure the physicians’ forecast of their patients’ survival. Physicians were surveyed to
record three different types of prognosis: (1) the point prediction is an answer to a question about the physicians’ best
estimate of how long this patient has to live; (2) the communicated prediction is an answer to a question about what
prognosis the doctor would communicate to the patient if the patient or the family insisted on receiving an estimate of
survival; (3) the subjective distribution prediction is the physicians’ stated percent estimate that the patient would still be
alive 7, 30, 90, 180 and 360 days after referral. Because we recorded the time of death, we can measure actual survival directly
and estimate the accuracy and biases physicians exhibit when they formulate their prognosis.
The explanatory variables analyzed below include: patients’ basic demographics (age, gender, race), income (based on
the patients’ ZIP codes), the duration of the disease that led to their final prognosis, and the Eastern Cooperative Oncology
Page 4
1098
M. Alexander, N.A. Christakis / Journal of Health Economics 27 (2008) 1095–1108
Table 1
Descriptive statistics
Variable Mean (S.D.)Variable Proportions (%)
Patients
Age (year)
Household income ($)
Disease duration (days)
ECOG physical activity
68.6 (17.4)
33,186 (11,178)
83.5 (135.8)
2.80 (1.01)
Sex (female)
Race (not white)
Cancer patients
55.4
32.4
64.5
Physicians
Hospice referrals (last quarter) 12.3 (16.9)Sex (female)
Speciality (family or GP)
Board certification
Selfdescribed optimist
Graduated from medical school ranked top 10th percentile
19.8
54.8
80.3
73.3
17.5
Physician–patient relationship
Time since first meeting (days)
Number of contacts in the past 3 months
159 (308)
11.1 (13.9)
Prognosis and survival
Point prediction of patient survival (days)
Communicated prediction of survival (days)
Actual survival (days)
106.6 (123.2)
116.1 (111.0)
62.2 (104.5)
Group (ECOG) score (measuring patients’ performance status: 0 for normal activity and 4 for completely bedbound). The
physicians’ data includes their gender, a dummy for whether a physician has a specialty, a prestige indicator of whether the
physicians’ medical school was ranked in the top 10th percentile of all medical schools, the number of hospice referrals in
the past quarter, and whether the physician considers himself or herself an optimist (based on Seligman, 1991). The time
since first meeting is the number of days elapsed since the physician first met the patient, and the frequency of contact is
measured as the number of days the physician has seen or spoken with the patient in the last 3 months.
1.2. The subjective belief distribution
Another key feature of the dataset is that it allows us to use the distribution of subjective beliefs to study the bias in
how prognoses are formulated and how this bias changes as physicians move in their decisionmaking from a full belief
distribution to a point prediction and then to a communicated prognosis. To study the subjective belief distribution, we
focus on the mean. To calculate the mean of the subjective distribution, we used the physicians’ interval predictions of the
probability that a patient would survive for 7, 30, 60, 180, and 360 days. We assumed that the subjective probability of a
patient’s survival at day 0 is 100% (i.e. the patient was alive on day 0).
We used a nonparametric approach to obtain a mean of the subjective distribution, given that we had point estimates
of the survival probability. The probabilities of survival for every day between 0 and 360 days were interpolated using linear
regression, and a lowess regression was then used to smooth our observations, giving us a nonparametric survival function
for each patient (as formulated subjectively by the physician). Finally, using this nonparametric survival function, the mean
was computed by minimizing the distance between the probability of a patient’s survival and the 50% value (using the
minimum squared error). This gave us a mean survival probability from the physician’s subjective distribution of beliefs over
his or her patient’s survival.
In Fig. 1, we analyze the relationship of the mean survival resulting from a full subjective belief distribution with three
othervalues:(1)actualsurvivalofthepatient,(2)pointprediction,and(3)communicatedsurvival.Wepresentascatterplot,
followed by a fractional polynomial regression line with confidence intervals. The advantage of the fractional polynomial
regression is that it does not rely on a linear assumption of the relationship between our mean of the belief distribution and
the other three variables.1We also plot a 45◦line to evaluate the extent of symmetry or asymmetry in the relationship.
The results in this figure give us the first indications of the significance and the direction of bias in physicians’ prognoses.
The physicians’ subjective probability distributions are poorly calibrated, as the mean of this distribution overestimates the
patients’ actual survival. As physicians move from their subjective distributions to point predictions, this bias increases. We
seethisbecausetheextentofasymmetryisgreaterwhenthepointpredictioniscomparedwithactualsurvivalthanwhenthe
belief distribution is compared to actual survival. The same happens when physicians move to the communicated prognosis.
Hence, this initial evidence suggests that physicians overestimate their patients’ survival, and that this bias may be further
increased as physicians move from a subjective belief distribution to a point prediction and then to a communicated survival.
1Whilemanyothernonlinearornonparametricmodelscouldbeused,theadvantageofthisapproachisthatitiseasilyimplementedandthatpolynomials
with a sufficient number of higherorder terms offer a good enough approximation of most wellbehaved, continuous functions.
Page 5
M. Alexander, N.A. Christakis / Journal of Health Economics 27 (2008) 1095–1108
1099
Fig. 1. Physicians’ prognosis of patient survival. Note: Upperleft to lowerright: (1) Up to the left: The distribution of physicians’ predicted survival, based
on a mean of their subjective belief probability distribution. (2) Up to the right: The relationship between the mean of the belief distribution and the point
prediction of survival. (3) Down to the left: The relationship between the mean of the belief distribution and the physicians’ communicated prognosis. (4)
Down to the right: The relationship between the mean of the belief distribution and actual survival. Red straight line represents symmetry. The blue curved
line is the estimated relationship with 95% CI shaded. (For interpretation of the references to colour in this figure legend, the reader is referred to the web
version of the article.)
2. Estimation of the loss functions
Our question in this section is to characterize the shape of asymmetry in the loss function of an average physician in our
sample. We use a flexible loss function approach which is then applied to two common forms of asymmetric loss functions
in forecasting, the lin–lin and the quad–quad function (Elliott et al., 2003). The general loss function is given by
L(p,˛) = [˛ + (1 − 2˛) × 1(yi− ˆ yi< 0)] × yi− ˆ yip,
where p∈N, the set of all positive integers, ˛∈(0,1), and yi− ˆ yiis the forecast error.
(1)
2.1. An estimation method for the average physician loss function
To estimate the average asymmetry parameter for our physician sample, we use an estimator developed by Elliott et al.
(2003):
(1/N)?N+?−1
(1/N)
i=?
For the lin–lin function, p=1, and the estimator becomes simply:
?N+?−1
i=?
and for ?=1, ˆ ˛ =?N
?N+?−1
i=?
?N+?−1
i=?
ˆ ˛ =
i=?
yi− ˆ yip−1× (1/N)?N+?−1
i=?
1(yi− ˆ yi< 0)yi− ˆ yip−1
yi− ˆ yip−1?2
??N+?−1
.
(2)
ˆ ˛l=
i=?
yi− ˆ yi0×?N+?−1
i=?
1(yi− ˆ yi< 0)yi− ˆ yi0
yi− ˆ yi0
?N+?−1
=
?N+?−1
i=?
?N+?−1
1(yi− ˆ yi< 0)
yi− ˆ yi0
i=?
(3)
i=11(yi− ˆ yi< 0)/N
For the quad–quad function, p=2, and the estimator becomes:
yi− ˆ yi ×?N+?−1
ˆ ˛q=
i=?i=?
1(yi− ˆ yi< 0)yi− ˆ yi
?2
??N+?−1
yi− ˆ yi
,
(4)
ˆ ˛q=
i=?
1(yi− ˆ yi< 0)yi− ˆ yi
?N+?−1
yi− ˆ yi
.
(5)