Bias and asymmetric loss in expert forecasts: A study of physician prognostic behavior with respect to patient survival

Institute for Quantitative Social Science, Harvard University, CGIS 1737 Cambridge Street, Cambridge, MA 02138, United States.
Journal of Health Economics (Impact Factor: 2.25). 08/2008; 27(4):1095-108. DOI: 10.1016/j.jhealeco.2008.02.011
Source: PubMed

ABSTRACT We study the behavioral processes undergirding physician forecasts, evaluating accuracy and systematic biases in estimates of patient survival and characterizing physicians' loss functions when it comes to prediction. Similar to other forecasting experts, physicians face different costs depending on whether their best forecasts prove to be an overestimate or an underestimate of the true probabilities of an event. We provide the first empirical characterization of physicians loss functions. We find that even the physicians subjective belief distributions over outcomes are not well calibrated, with the loss characterized by asymmetry in favor of over-predicting patients' survival. We show that the physicians' bias is further increased by (1) reduction of the belief distributions to point forecasts, (2) communication of the forecast to the patient, and (3) physicians own past experience and reputation.

  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Regression learning methods in real world applications often require cost minimization instead of the reduction of various metrics of prediction errors. Currently in the literature, there is a lack of white box solutions that can deal with forecasting problems where under-prediction and over-prediction errors have different consequences. To fill this gap, we introduced the Cost-sensitive Global Model Tree (CGMT), which applies a fitness function that minimizes an average misprediction cost. Proposed specialized genetic operators improve searching for optimal tree structure and cost-sensitive linear regression models in the leaves. Experimental validation is performed on loan charge-off data. It is known to be a difficult forecasting problem for banks due to the asymmetric cost structure. Obtained results show that specialized evolutionary algorithm applied to model tree induction finds significantly more accurate predictions than tested competitors. Decisions generated by the CGMT are simple, easy to interpret, and can be applied directly.
    Decision Support Systems 04/2015; 74. DOI:10.1016/j.dss.2015.03.009 · 2.04 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: The field of clinical decision making is polarized by two predominate views. One holds that treatment recommendations should conform with guidelines; the other emphasizes clinical expertise in reaching case-specific judgments. Previous work developed a test for a proposed alternative, that clinical judgment should systematically incorporate both general knowledge and patient-specific information. The test was derived from image theory's two phase-account of decision making and its ``simple counting rule'', which describes how possible courses of action are pre-screened for compatibility with standards and values. The current paper applies this rule to clinical forecasting, where practitioners indicate how likely a specific patient will respond favorably to a recommended treatment. Psychiatric trainees evaluated eight case vignettes that exhibited from 0 to 3 incompatible attributes. They made two forecasts, one based on a guideline recommendation, the other based on their own alternative. Both forecasts were predicted by equally- and unequally-weighted counting rules. Unequal weighting provided a better fit and exhibited a clearer rejection threshold, or point at which forecasts are not diminished by additional incompatibilities. The hypothesis that missing information is treated as an incompatibility was not confirmed. There was evidence that the rejection threshold was influenced by clinician preference. Results suggests that guidelines may have a de-biasing influence on clinical judgment. Subject to limitations pertaining to the subject sample and population, clinical paradigm, guideline, and study procedure, the data support the use of a compatibility test to describe how clinicians make patient-specific forecasts.
  • American Journal of Roentgenology 06/2009; 192(5):1375-81. DOI:10.2214/AJR.08.2063 · 2.74 Impact Factor


Available from