Page 1

Journal of Health Economics 27 (2008) 1095–1108

Contents lists available at ScienceDirect

Journal of Health Economics

journal homepage: www.elsevier.com/locate/econbase

Bias and asymmetric loss in expert forecasts: A study of physician

prognostic behavior with respect to patient survival

Marcus Alexandera,∗, Nicholas A. Christakisb,c

aHarvard University, Institute for Quantitative Social Science and Department of Government, CGIS 1737 Cambridge Street, Cambridge 02138, United States

bDepartment of Health Care Policy, Harvard Medical School, 180 Longwood Avenue, Boston, MA 02115, United States

cDepartment of Sociology, FAS, 33 Kirkland Street, Cambridge, MA 02138, United States

a r t i c l e i n f o

Article history:

Received 23 November 2006

Received in revised form 1 February 2008

Accepted 10 February 2008

JEL classification:

I10, I12, I19, D01, D80, C53

Keywords:

Loss function

Forecasting

Behavioral economics

Survival

Prognosis

a b s t r a c t

We study the behavioral processes undergirding physician forecasts, evaluating accuracy

and systematic biases in estimates of patient survival and characterizing physicians’ loss

functions when it comes to prediction. Similar to other forecasting experts, physicians face

different costs depending on whether their best forecasts prove to be an overestimate or an

underestimate of the true probabilities of an event. We provide the first empirical charac-

terization of physicians’ loss functions. We find that even the physicians’ subjective belief

distributionsoveroutcomesarenotwellcalibrated,withthelosscharacterizedbyasymme-

tryinfavorofover-predictingpatients’survival.Weshowthatthephysicians’biasisfurther

increased by (1) reduction of the belief distributions to point forecasts, (2) communication

of the forecast to the patient, and (3) physicians’ own past experience and reputation.

© 2008 Elsevier B.V. All rights reserved.

In this paper, we investigate the accuracy of physicians’ forecasts of survival. We ask whether a physician’s prognosis

exhibits systematic biases, and we explore the sources of such biases. Our investigation uncovers a systematic tendency of

physicians to overpredict their patients’ survival at three stages: first, with respect to the survival distributions that doctors

construct, second in their summarization of this distribution through the selection of a point estimate, and third in their

choice about how to further modify this estimate during communication.

The strategic role of communication between physicians and patients has been studied by Caplin and Leahy (2004),

illustrating how the standard model of preferences breaks down once agents draw psychological utility from their beliefs.

Extending this model, Koszegi (2006) also used physician–patient communication to investigate how provision of informa-

tion by experts becomes distorted in the presence of anticipatory feelings. These important theoretical contributions lay

the groundwork for empirically examining the systematic tendencies of physicians to distort their prognosis when both

formulating it and communicating it to their patients.

More specifically, findings from the literature on emotional agency lead us to expect that a closer relationship between

a physician and a patient should be associated with more upwardly biased loss. In this model, the physicians’ utility func-

tion includes their patients’ emotional status, therefore providing an incentive for physicians to formulate an upwardly

biased prognosis. This theoretical framework also sheds light on why we would expect a doctor to be even more upwardly

biased when communicating than when formulating an expectation. It is clearly more emotionally stressful to share bad

news than merely to think about it. Additionally, communication provides for a strategic environment consistent with

∗Corresponding author. Institute for Quantitative Social Science and Department of Government, Harvard University, CGIS, 1737 Cambridge Street,

Cambridge, MA 02138, United States. Tel. +1 617 909 4618; fax: +1 617 432 5891.

E-mail address: malexand@fas.harvard.edu (M. Alexander).

0167-6296/$ – see front matter © 2008 Elsevier B.V. All rights reserved.

doi:10.1016/j.jhealeco.2008.02.011

Page 2

1096

M. Alexander, N.A. Christakis / Journal of Health Economics 27 (2008) 1095–1108

Koszegi’s (2006) model, whereby a physician has an opportunity as an agent to affect the emotional state of the patient as a

principal.

Furthermore, as Koszegi (2003) indicates, physician–patient communication may be accompanied by deeper psycholog-

ical biases such as Samuelson’s (1963) fallacy of large numbers—simply defined as accepting a large number of unfavorable

gambles even when the agent is unwilling to take any one individual gamble on its own. In fact, some of the classic biases

in behavioral economics have been first characterized by studying physician behavior. These are most notably associated

with the process by which physicians formulate diagnosis, and they most famously include the role of hindsight in distorting

probability estimates (Arkes et al., 1981; Slovic and Fischhoff, 1977), base rate neglect (Casscells et al., 1978; Kahneman

and Tversky, 1973), and the conjunction fallacy (Tversky and Kahneman, 1983). In all of these situations, agents produce

inaccurate probability estimates given the uncertainty they face over the true state of the world.

Thekeyquestionthatariseshereishowphysiciansunderstandandprocesstheinformationabouttheirpatients’likelihood

of survival, and how they use their own subjective belief distributions to formulate a point forecast. In other words, even

before the strategic component of physician–patient communication enters the picture, we can ask whether systematic

biases characterize the process by which physicians arrive at their own best point forecast of patients’ survival.

In analyzing the asymmetry of physicians’ prognosis, it is useful to draw on the broader economic literature on expert

forecasts. In one of the first studies of asymmetric loss in economics, Varian (1974) documented an important fact that

experts in a market face different costs depending whether their best prediction is an overestimate or an underestimate of

the market price. In his study of the market for single family homes in a 1965 California town, Varian noticed that assessors

faced a significantly higher cost if they happened to overestimate the value of a house. While in the case of an underestimate,

the assessor’s office faced the cost in the amount of the underestimate, conversely, in the case of the overestimate by an

identical amount, the assessor’s office faced a possibility of a lengthy and costly appeal process. Since this classic study, loss

functions have become an important aspect of the study of expert forecasts.

The two key empirical puzzles surrounding the question of expert forecasts became to determine whether forecasters’

loss functions were symmetric, and if not, how optimal forecasts can be made given loss asymmetry, as addressed most

recently by Elliott et al. (2005). For example, government experts making budget forecasts may be influenced by political

incentives, as the costs of wrongly projecting a surplus may lead to public disapproval, while wrongly projecting a deficit

may lead to an impression of exceptional government performance. Artis and Marcellino (2001), as well as Campbell and

Ghysels (1995), document that budget deficit forecasts have asymmetric loss. Furthermore, expert opinion varies greatly and

systematically. For example, research by Lamont (1995) indicates that factors such as forecasters’ experience and reputation

are reliable determinants of experts’ willingness to deviate from consensus forecasts of GDP, unemployment, and prices.

In financial and macroeconomic forecasting, Granger and Newbold (1986) have concluded that economic theory does not

suggest that experts even should have a symmetric loss function. An improved understanding of behavioral biases arising in

agents’ decisions, such as those associated with loss functions, can contribute to answering puzzles about risky behavior in

the labor market and education decisions (e.g., Abowd and Card, 1989; Card and Hyslop, 1997; Card and Lemieux, 2001a,b)

and in health economics (e.g., Koszegi, 2003, 2006).

Because in most economic situations, such as Varian’s (1974) real-estate market, agents formulate and report point

predictions as their forecasts, the agents’ true subjective belief distributions are lost and cannot be recovered from their

forecasts. Hence the problem of characterizing the loss function is compounded by the fact that we do not know anything

about the behavioral process by which agents reduce their belief distributions into single-point predictions, a process which

itself reflects the extent of asymmetry in their unobserved loss function. Furthermore, because of strategic considerations,

the prediction that agents communicate may be different from both the point prediction and the prediction implied by

the agents’ full subjective belief distributions. Unfortunately, due to data limitations, no study has been able to examine all

of these aspects of forecasting simultaneously. To date, the study of loss function asymmetry has been largely limited to

studying point forecasts (e.g., the Livingstone survey), while the study of forecasters’ fuller subjective belief distributions has

been confined to surveys of national output by experts (e.g., Survey of Professional Forecasters), as illustrated by the work

that originated with Victor Zarnowitz’s (1985) study of rational expectations.

Our study addresses the extent of intrinsic bias in forecast predictions and asks how the forecast bias and the symmetry

of the loss function change as agents move from a full subjective distribution to a point prediction and then to commu-

nicating their formulated forecast. We focus on the first part of the processes because psychological research by Tversky

and Kahneman (1973, 1974) has documented that individuals exhibit different types of biases when using probability dis-

tributions to infer a possibility of an outcome. Analogous to the biases that arise from the use of inference heuristics such

as representativeness or availability, agents may also exhibit biases when narrowing their subjective belief distributions

to single-point predictions. In particular, because the standard symmetric loss function requires minimization of the mean

squared error, individuals may show systematic bias due to failure to compute a correct mean or because t have asymmetric

loss. Much like econometric estimators that are biased when certain assumptions fail, the behavioral mechanism leading

to a point forecast from subjective beliefs may be biased due to computational limitations or a misinterpretation of the

optimization problem by agents. We also focus on the latter part of the process – the role of communication – because, with

the exception of independent, disinterested expert forecasters, the communication of an agent’s forecast is likely to play a

strategic role in a market. Therefore, any bias that led to formulation of the forecast may be further compounded by the

agent’s strategic biases in communicating the prediction. To study all of this, we need a record of forecasts that documents

both the process of reduction from subjective beliefs to a point forecast and the process of communication of that forecast.

Page 3

M. Alexander, N.A. Christakis / Journal of Health Economics 27 (2008) 1095–1108

1097

To investigate the problem, we use a unique prospective study of Chicago metropolitan area physicians referring termi-

nally ill patients to hospice care. To our knowledge, this is the first survey that collected a combination of forecasts allowing

us to study all the aspects of the above question. First, the study asked physicians to make interval forecasts of the patients’

survival probability, approximating a subjective belief distribution. Second, the physicians’ best point prediction of survival

was recorded. Third, physicians were asked what prediction they would communicate to their patients. Fourth, a series of

questions regarding the physicians’ confidence, optimism, and experience was recorded, as well as the patients’ characteris-

tics. Finally, the prospective nature of the study allowed us to compare the different forecasts to the patients’ actual survival

time. Together, these features of the data give us a unique opportunity to study loss functions that characterize physician

decision-making.

We quantify the extent of asymmetry in how much physicians value over-predicting versus under-predicting their

patients’ survival. We also demonstrate that the physicians’ bias increases when they communicate their prognosis to their

patients. The physicians’ own loss function becomes more asymmetric, favoring over-prediction of survival, when they move

from formulating a point prediction to communicating a prognosis to their patients. We also show that the asymmetry in the

physicians’ loss function moves in the other direction when a fuller subjective belief distribution is elicited from the physi-

cians. In contrast to the point forecast, the physicians’ bias decreases when they forecast a subjective probability distribution

over their patients’ odds of survival.

We also asked which physician and patient characteristics serve as determinants of the level of asymmetry in the physi-

cians’ loss function. Our findings indicate that the patients’ gender, race, and type of disease, as well as the physicians’

experience, are important determinants. Together, these results point to the fact that physicians may rationally prefer to

overestimate survival of their patients. Given the economic and clinical nature of the doctor–patient relationship, overesti-

mating the odds of a patient’s survival can be expected to serve as a commitment device to a prescribed choice of therapy

and contributes to the physicians’ sense of confidence. In the setting of hospice care in particular, evidence of upward bias

suggests that emotional agency described above comes to the forefront, playing an important role in physicians’ behavior

above and beyond the commitment mechanism observed elsewhere. However, the evidence that patients’ race and gender

play a role in the degree of loss asymmetry indicates that physicians’ forecasts are also subject to biases beyond a rational

calibration of the loss function.

Because forecasting of patients’ survival is an important part of the medical profession (Christakis, 1999), our character-

ization of physicians’ loss functions serves two purposes: (1) in general, it carries implications for understanding behavior

of experts whose performance depends on forecast accuracy, and (2) more particularly, it has downstream implications for

understanding the supply of health care and for health care expenditures.

The paper is organized as follows. Section 1 introduces the data. Section 2 presents the method we use for estimation of

loss functions. Section 3 presents the results. Section 4 discusses the conclusions.

1. Data

1.1. Patient and physician data

To study prognostic accuracy and bias among physicians, we use data from a 1996 prospective cohort study, conducted in

the Chicago metropolitan area. The study approached all hospices in Chicago that admitted more than 200 patients per year.

Five of the six such hospices participated in the study, producing a cohort of all patients admitted during 130 consecutive

days in 1996 (Christakis and Lamont, 2000).

Forallpatientsinthestudy,thephysicianwhoreferredthepatienttohospicecarewasempaneled(noneoftheparticipat-

ing physicians were the hospice medical directors). In some cases, the referring physician was the primary care doctor and in

others the physician was a specialist (such as the treating oncologist). We collected a prognosis from only one doctor for each

patient. We collected individual physician data (e.g., their sex, specialty, year of graduation from medical school, board cer-

tification, etc.) and three variables that characterizes the relationship between the doctor and the patient, namely, duration

of contact (when they first met), frequency of contact, and recency of contact (when the doctor last examined the patient).

All physicians were surveyed at the same point in time, typically within 48h of the time they referred the patient to hospice.

The descriptive statistics are summarized in Table 1. We studied a total of 504 patients referred by 365 physicians. All

patients were followed until their deaths. At the time of hospice referral, all patients were terminally ill. The most frequent

diagnoses were lung cancer (18%), AIDS (12%), colorectal cancer (7%), breast cancer (6%), chronic heart failure (5%), and

stroke (5%).

The main variables of interest measure the physicians’ forecast of their patients’ survival. Physicians were surveyed to

record three different types of prognosis: (1) the point prediction is an answer to a question about the physicians’ best

estimate of how long this patient has to live; (2) the communicated prediction is an answer to a question about what

prognosis the doctor would communicate to the patient if the patient or the family insisted on receiving an estimate of

survival; (3) the subjective distribution prediction is the physicians’ stated percent estimate that the patient would still be

alive 7, 30, 90, 180 and 360 days after referral. Because we recorded the time of death, we can measure actual survival directly

and estimate the accuracy and biases physicians exhibit when they formulate their prognosis.

The explanatory variables analyzed below include: patients’ basic demographics (age, gender, race), income (based on

the patients’ ZIP codes), the duration of the disease that led to their final prognosis, and the Eastern Cooperative Oncology

Page 4

1098

M. Alexander, N.A. Christakis / Journal of Health Economics 27 (2008) 1095–1108

Table 1

Descriptive statistics

Variable Mean (S.D.)Variable Proportions (%)

Patients

Age (year)

Household income ($)

Disease duration (days)

ECOG physical activity

68.6 (17.4)

33,186 (11,178)

83.5 (135.8)

2.80 (1.01)

Sex (female)

Race (not white)

Cancer patients

55.4

32.4

64.5

Physicians

Hospice referrals (last quarter)12.3 (16.9) Sex (female)

Speciality (family or GP)

Board certification

Self-described optimist

Graduated from medical school ranked top 10th percentile

19.8

54.8

80.3

73.3

17.5

Physician–patient relationship

Time since first meeting (days)

Number of contacts in the past 3 months

159 (308)

11.1 (13.9)

Prognosis and survival

Point prediction of patient survival (days)

Communicated prediction of survival (days)

Actual survival (days)

106.6 (123.2)

116.1 (111.0)

62.2 (104.5)

Group (ECOG) score (measuring patients’ performance status: 0 for normal activity and 4 for completely bed-bound). The

physicians’ data includes their gender, a dummy for whether a physician has a specialty, a prestige indicator of whether the

physicians’ medical school was ranked in the top 10th percentile of all medical schools, the number of hospice referrals in

the past quarter, and whether the physician considers himself or herself an optimist (based on Seligman, 1991). The time

since first meeting is the number of days elapsed since the physician first met the patient, and the frequency of contact is

measured as the number of days the physician has seen or spoken with the patient in the last 3 months.

1.2. The subjective belief distribution

Another key feature of the dataset is that it allows us to use the distribution of subjective beliefs to study the bias in

how prognoses are formulated and how this bias changes as physicians move in their decision-making from a full belief

distribution to a point prediction and then to a communicated prognosis. To study the subjective belief distribution, we

focus on the mean. To calculate the mean of the subjective distribution, we used the physicians’ interval predictions of the

probability that a patient would survive for 7, 30, 60, 180, and 360 days. We assumed that the subjective probability of a

patient’s survival at day 0 is 100% (i.e. the patient was alive on day 0).

We used a non-parametric approach to obtain a mean of the subjective distribution, given that we had point estimates

of the survival probability. The probabilities of survival for every day between 0 and 360 days were interpolated using linear

regression, and a lowess regression was then used to smooth our observations, giving us a non-parametric survival function

for each patient (as formulated subjectively by the physician). Finally, using this non-parametric survival function, the mean

was computed by minimizing the distance between the probability of a patient’s survival and the 50% value (using the

minimum squared error). This gave us a mean survival probability from the physician’s subjective distribution of beliefs over

his or her patient’s survival.

In Fig. 1, we analyze the relationship of the mean survival resulting from a full subjective belief distribution with three

othervalues:(1)actualsurvivalofthepatient,(2)pointprediction,and(3)communicatedsurvival.Wepresentascatterplot,

followed by a fractional polynomial regression line with confidence intervals. The advantage of the fractional polynomial

regression is that it does not rely on a linear assumption of the relationship between our mean of the belief distribution and

the other three variables.1We also plot a 45◦line to evaluate the extent of symmetry or asymmetry in the relationship.

The results in this figure give us the first indications of the significance and the direction of bias in physicians’ prognoses.

The physicians’ subjective probability distributions are poorly calibrated, as the mean of this distribution over-estimates the

patients’ actual survival. As physicians move from their subjective distributions to point predictions, this bias increases. We

seethisbecausetheextentofasymmetryisgreaterwhenthepointpredictioniscomparedwithactualsurvivalthanwhenthe

belief distribution is compared to actual survival. The same happens when physicians move to the communicated prognosis.

Hence, this initial evidence suggests that physicians over-estimate their patients’ survival, and that this bias may be further

increased as physicians move from a subjective belief distribution to a point prediction and then to a communicated survival.

1Whilemanyothernonlinearornon-parametricmodelscouldbeused,theadvantageofthisapproachisthatitiseasilyimplementedandthatpolynomials

with a sufficient number of higher-order terms offer a good enough approximation of most well-behaved, continuous functions.

Page 5

M. Alexander, N.A. Christakis / Journal of Health Economics 27 (2008) 1095–1108

1099

Fig. 1. Physicians’ prognosis of patient survival. Note: Upper-left to lower-right: (1) Up to the left: The distribution of physicians’ predicted survival, based

on a mean of their subjective belief probability distribution. (2) Up to the right: The relationship between the mean of the belief distribution and the point

prediction of survival. (3) Down to the left: The relationship between the mean of the belief distribution and the physicians’ communicated prognosis. (4)

Down to the right: The relationship between the mean of the belief distribution and actual survival. Red straight line represents symmetry. The blue curved

line is the estimated relationship with 95% CI shaded. (For interpretation of the references to colour in this figure legend, the reader is referred to the web

version of the article.)

2. Estimation of the loss functions

Our question in this section is to characterize the shape of asymmetry in the loss function of an average physician in our

sample. We use a flexible loss function approach which is then applied to two common forms of asymmetric loss functions

in forecasting, the lin–lin and the quad–quad function (Elliott et al., 2003). The general loss function is given by

L(p,˛) = [˛ + (1 − 2˛) × 1(yi− ˆ yi< 0)] × |yi− ˆ yi|p,

where p∈N, the set of all positive integers, ˛∈(0,1), and yi− ˆ yiis the forecast error.

(1)

2.1. An estimation method for the average physician loss function

To estimate the average asymmetry parameter for our physician sample, we use an estimator developed by Elliott et al.

(2003):

(1/N)?N+?−1

(1/N)

i=?

For the lin–lin function, p=1, and the estimator becomes simply:

?N+?−1

i=?

and for ?=1, ˆ ˛ =?N

?N+?−1

i=?

?N+?−1

i=?

ˆ ˛ =

i=?

|yi− ˆ yi|p−1× (1/N)?N+?−1

i=?

1(yi− ˆ yi< 0)|yi− ˆ yi|p−1

|yi− ˆ yi|p−1?2

??N+?−1

.

(2)

ˆ ˛l=

i=?

|yi− ˆ yi|0×?N+?−1

i=?

1(yi− ˆ yi< 0)|yi− ˆ yi|0

|yi− ˆ yi|0

?N+?−1

=

?N+?−1

i=?

?N+?−1

1(yi− ˆ yi< 0)

|yi− ˆ yi|0

i=?

(3)

i=11(yi− ˆ yi< 0)/N

For the quad–quad function, p=2, and the estimator becomes:

|yi− ˆ yi| ×?N+?−1

ˆ ˛q=

i=?i=?

1(yi− ˆ yi< 0)|yi− ˆ yi|

?2

??N+?−1

|yi− ˆ yi|

,

(4)

ˆ ˛q=

i=?

1(yi− ˆ yi< 0)|yi− ˆ yi|

?N+?−1

|yi− ˆ yi|

.

(5)