The impossibility of experimental elicitation of subjective probabilities

Theory and Decision (Impact Factor: 0.48). 01/1995; 38(3):313-320. DOI: 10.1007/BF01362238

ABSTRACT We show that if decision makers may have stakes in certain events then the experimental elicitation of their subjective probabilities of these events is impossible.

  • [Show abstract] [Hide abstract]
    ABSTRACT: Understanding risky choice requires knowledge of beliefs and preferences. A variety of methods have been proposed to elicit beliefs. Theory shows that some methods should produce biased estimates except under restrictive assumptions, while others should produce unbiased estimates with fewer restrictive assumptions. The efficacy of alternative methods, however, has not been systematically documented empirically. We use an experiment to test whether induced beliefs can be recovered using a scoring rule and prediction-based elicitation method. Pooling subject responses, we are unable to recover the induced beliefs with either method. The bias associated with the prediction method tends to be larger than the bias associated with the scoring rule method. Using individual responses, we find the performance of each method is subject- specific. For some subjects, the scoring rule works, while for others, the prediction method works. Overall, one of the two methods successfully recovered the induced beliefs for 70 percent of subjects.
  • [Show abstract] [Hide abstract]
    ABSTRACT: We introduce a simple method to construct an incentive compat- ible scoring rule to elicit an agent's subjective belief about a random variable. The method does not depend on the exact form of the agent's utility function. Independent of her risk-preference, it is optimal for her to report the value that minimizes the expected value of a loss func- tion specified in the incentive scheme. Under this incentive scheme, the agent receives a fixed prize when her prediction error, defined by the relevant loss function, is smaller than a randomly generated value and earns nothing otherwise. Adjusting the loss function according to our goal for eliciting the agent's beliefs, the scoring rule can be used in a rich assortment of situations. We run experiments using our scoring rule and the quadratic scoring rule and find that our scor- ing rule performs as well as the quadratic scoring rule. In fact, our scoring rule's performance seems to improve as subjects become more familiar with it.
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Proper scoring rules (PSR) are among the most popular incentivized belief elicitation mechanisms. A well known result is that risk averters facing PSR misreport their beliefs by stating more uniform probabilities. We show that this result does not generalize when i) the PSR payments are increased, ii) the agent has a …nancial stake in the event she is predicting, and iii) the agent can hedge her prediction by taking an additional action. Instead, combining theory and experiment, we …nd that agents distort their reported probabilities in complex, yet mostly predictable manners. We argue that our results have implications for the elicitation of beliefs in most environments of interest to economists, both in academia and in practice.