David R. Mandel

• PhD, Psychology, UBC 1996
• Senior Researcher at Defence Research and Development Canada

Framing effects have long been viewed as compelling evidence of irrationality in human decision making, yet that view rests on the questionable assumption that numeric quantifiers used to convey the expected values of choice options are uniformly interpreted as exact values. Two experiments show that when the exactness of such quantifiers is made e...

The accuracy of 1,514 strategic intelligence forecasts abstracted from intelligence reports was assessed. The results show that both discrimination and calibration of forecasts was very good. Discrimination was better for senior (versus junior) analysts and for easier (versus harder) forecasts. Miscalibration was mainly due to underconfidence such...

Intelligence analysis plays a vital role in policy decision-making. Key functions of intelligence analysis include accurately forecasting significant events, appropriately characterizing the uncertainties inherent in such forecasts, and effectively communicating those probabilistic forecasts to stakeholders. We review decision research on probabili...

Uncertainty is both inherent in nature and endemic to national security decision-making. Intelligence communities throughout the Western world, however, rely on vague language to communicate uncertainty—both the probability of critical events and the confidence that analysts have in their assessments—to decision-makers. In this article, we review t...

National security decision-making is informed by intelligence assessments, which in turn depend on sound information evaluation. Recognizing this fact, intelligence organizations have promulgated methods for evaluating source and evidential characteristics. We critically examine these methods and identify several limitations that undermine the fide...

People are often overconfident in their probabilistic judgments of future events or the state of their own knowledge. Some training methods have proven effective at reducing bias, but these usually involve intensive training sessions with experienced facilitators. This is not conducive to a scalable and domain‐general training program for improving...

Experts are expected to make well‐calibrated judgments within their field, yet a voluminous literature demonstrates miscalibration in human judgment. Calibration training aimed at improving subsequent calibration performance offers a potential solution. We tested the effect of commercial calibration training on a group of 70 intelligence analysts b...

In this retrospective honoring the exemplary psychologist Daniel Kahneman (1934–2024), the authors present a curated selection of quotes from the academic community reflecting on his ideas. These submissions, gathered from a wide range of scholars, highlight Kahneman’s contributions to fields spanning attention, judgment, decision-making, and well-...

Expert judgment often involves estimating magnitudes, such as the frequency of deaths due to a pandemic. Three experiments ( N s = 902, 431, and 755, respectively) were conducted to examine the effect of outcome framing (e.g., half of a threatened group expected to survive vs. die), probability level (low vs. high), and probability format (verbal,...

People are generally overconfident in their probabilistic judgements of future events or the state of their own knowledge. Some training methods have proven effective at reducing bias, but these often involve intensive training sessions with experienced facilitators. This is not conducive to a scalable and domain-general training program for improv...

It is a physical fact that humans have access to records of the past but not of the future and that this creates an epistemic asymmetry in time. We explore the epistemic asymmetry, hypothesizing that retrospection about past psychological states is less biased than prospection about future states. Leveraging data from the Pew Research Center’s Glob...

The quality of information that informs decisions in expert domains such as law enforcement and national security often requires assessment based on meta-informational cues such as source reliability and information credibility. Across two experiments with intelligence analysts (n = 74) and non-experts (n = 175), participants rated the accuracy, in...

People commonly exhibit a bias blind spot, judging themselves to be less susceptible to bias than the “average other.” However, less is known about how people attribute bias to familiar others who evoke strong affect. The present work (N = 1,980) examines the degree to which participants’ attributions of bias are sensitive to their affective impres...

Generalizations strengthen in traditional sciences, but in psychology (and social and behavioral sciences, more generally) they decay. This is usually viewed as a problem requiring solution. It could be viewed instead as a law-like phenomenon. Generalization decay cannot be squelched because human behavior is metastable and all behavioral data coll...

We examine the opportunities and challenges of expert judgment in social sciences, scrutinizing the way social scientists make predictions. While social scientists show above chance accuracy in predicting lab-based phenomena, they often struggle to predict real-world societal changes. We argue that most causal models used in social sciences are ove...

This correction pertains to an error in the copyright information of the original publication. There were no revisions to the content of the chapter itself. The copyright holder of this chapter has been retrospectively corrected. The correct copyright holder name is: © His Majesty the King in Right of Canada as represented by Department of Nation...

Strict framing effects such as Huizenga et al. (2023) examine require that alternative frames are judged to be extensionally equivalent from the participant’s viewpoint. Ascertaining subjective equivalence requires measuring linguistic individual differences in frame interpretation, which is seldom performed. Studies that have examined the issue ca...

The primary aim of this chapter is to introduce a novel approach to strengthen contemporary intelligence community practices for establishing intelligence collection priorities based on expected information value. We propose the integration of quantitative measures of information utility that have been discussed in the literature on information the...

Previous research shows that variation in coherence (i.e., degrees of respect for axioms of probability calculus), when used as a basis for performance-weighted aggregation, can improve the accuracy of probability judgments. However, many aspects of coherence-weighted aggregation remain a mystery, including both prescriptive issues (e.g., how best...

Experts often communicate probabilities verbally (e.g., unlikely) rather than numerically (e.g., 25% chance). Although criticism has focused on the vagueness of verbal probabilities, less attention has been given to the potential unintended, biasing effects of verbal probabilities in communicating probabilities to decision-makers. In four experimen...

The benefits of judgment aggregation are intuitive and well-documented. By combining the input of several judges, practitioners may enhance information sharing and signal strength while cancelling out biases and noise. The resulting judgment is more accurate than the average accuracy of the individual judgments—a phenomenon known as the wisdom of c...

Intelligence communities regularly produce important assessments that inform policymakers. The Analysis of the Competing Hypotheses technique (ACH) is one of the most widely-touted methods for improving the accuracy of those assessments. But does ACH work? This systematic review identified seven articles describing six experiments testing the ACH....

Expert judgment often involves estimating magnitudes, such as the frequency of deaths due to a pandemic. Two experiments (Ns = 902 and 431, respectively) were conducted to examine the effect of outcome framing (half of a threatened group expected to survive vs. die), probability level (low vs. high), and probability format (verbal, numeric range, o...

This is a pre-print of a chapter intended for publication in the book "JUDGMENT IN PREDICTIVE ANALYTICS" (Ed. Matthias Seifert, Ph.D.). The chapter represents a primer and review of various performance-weighted aggregation techniques, including: history-based weighting methods, disposition-based weighting methods, and coherence-based weighting meth...

How well can social scientists predict societal change, and what processes underlie their predictions? To answer these questions, we ran two forecasting tournaments testing the accuracy of predictions of societal change in domains commonly studied in the social sciences: ideological preferences, political polarization, life satisfaction, sentiment...

How well can social scientists predict societal change, and what processes underlie their predictions? To answer these questions, we ran two forecasting tournaments testing the accuracy of predictions of societal change in domains commonly studied in the social sciences: ideological preferences, political polarization, life satisfaction, sentiment...

As replications of individual studies are resource intensive, techniques for predicting the replicability are required. We introduce the repliCATS (Collaborative Assessments for Trustworthy Science) process, a new method for eliciting expert predictions about the replicability of research. This process is a structured expert elicitation approach ba...

Expectations about future events underlie practically every decision we make, including those in medical research. This paper reviews five studies undertaken to assess how well medical experts could predict the outcomes of clinical trials. It explains why expert trial forecasting was the focus of study and argues that forecasting skill affords insi...

Previous research shows that variation in coherence (i.e., degrees of respect for axioms of probability calculus), when used as a basis for performance-weighted aggregation, can improve the accuracy of probability judgments. However, many aspects of coherence-weighted aggregation remain a mystery, including both prescriptive issues (e.g., how best...

Strict framing effects such as Huizenga et al. examine require that alternative frames are judged to be extensionally equivalent from the participant’s viewpoint. Ascertaining subjective equivalence requires measuring linguistic individual differences in frame interpretation, which is seldom performed. Studies that have examined the issue call the...

Previous research has shown greater risk aversion when people make choices about lives than cash. We tested the hypothesis that compared to placebo, exogenous testosterone administration would lead to riskier choices about cash than lives, given testosterone’s association with financial risk-taking and reward sensitivity. A double-blind, placebo-co...

Meta‐information is information about information that can be used as cues to guide judgments and decisions. Three types of meta‐information that are routinely used in intelligence analysis are source reliability, information credibility, and classification level. The first two cues are intended to speak to information quality (in particular, the p...

Individuals often assess themselves as being less susceptible to common biases compared to others. This bias blind spot (BBS) is thought to represent a metacognitive error. In this research, we tested three explanations for the effect: The cognitive sophistication hypothesis posits that individuals who display the BBS more strongly are actually les...

Expert judges often express probabilities verbally (e.g., unlikely) rather than numerically (e.g., 25% chance). Researchers criticize the practice because verbal probabilities are vague, but the pragmatic consequences may be equally important. In four experiments (Ns = 201, 439, 435, 696), we show that probability format (i.e., verbal vs. numeric)...

For the last half-century, the US and Allied intelligence community has sought to minimize the ostensibly detrimental effects of cognitive biases on intelligence practice. The dominant approach to doing so has been to develop structured analytic techniques (SATs), teach them to analysts in brief training sessions, provide the means to use SATs on t...

Individuals often assess themselves as being less susceptible to common biases compared to others. This bias blind spot (BBS) is thought to represent a metacognitive error. In this research, we tested three explanations for the effect: The cognitive sophistication hypothesis posits that individuals who display the BBS more strongly are actually les...

How well can social scientists predict societal change, and what processes underlie their predictions? To answer these questions, we ran two forecasting tournaments testing accuracy of predictions of societal change in domains commonly studied in the social sciences: ideological preferences, political polarization, life satisfaction, sentiment on s...

Organizations in several domains including national security intelligence communicate judgments under uncertainty using verbal probabilities (e.g., likely) instead of numeric probabilities (e.g., 75% chance), despite research indicating that the former have variable meanings across individuals. In the intelligence domain, uncertainty is also commun...

With their spotlight on Laplace’s inductive rule of succession, Fiedler et al. (2022) revisit important questions about what constitutes rationality and how rationality can be well tested in experimental research. In commentary on their article, I propose that Popper’s evolutionary epistemology can help address both questions.

Meta-information is information about information that can be used as cues to guide judgments and decisions. Three types of meta-information that are routinely used in intelligence analysis are source reliability, information credibility and classification level. The first two cues are intended to speak to information quality (in particular, the pr...

As the US-China great power competition intensifies, public opinion polling may help gauge internal drivers of foreign policy decision-making. Using Pew Research Center data, the authors analyzed how American and Chinese respondents viewed their own and each other’s countries between 2008–2016. They further examined how American attitudes towards C...

Bermúdez’s case for rational framing effects, while original, is unconvincing and gives only parenthetical treatment to the problematic assumptions of extensional and semantic equivalence of alternative frames in framing experiments. If the assumptions are false, which they sometimes are, no valid inferences about “framing effects” follow and, then...

Life in an increasingly information-rich but highly uncertain world calls for an effective means of communicating uncertainty to a range of audiences. Senders prefer to convey uncertainty using verbal (e.g., likely) rather than numeric (e.g., 75% chance) probabilities, even in consequential domains, such as climate science. However, verbal probabil...

Bermudez’s case for rational framing effects, while original, is unconvincing and gives only parenthetical treatment to the problematic assumptions of extensional and semantic equivalence of alternative frames in framing experiments. If the assumptions are false, which they sometimes are, no valid inferences about “framing effects” follow and, then...

Objective To assess the accuracy of principal investigators’ (PIs) predictions about three events for their own clinical trials: positivity on trial primary outcomes, successful recruitment and timely trial completion. Study design and setting A short, electronic survey was used to elicit subjective probabilities within seven months of trial regis...

On November 23, 2021, Dr. David Mandel presented Communicating Uncertainty in Warning Intelligence at the 2021 CASIS West Coast Security Conference. The primary concepts of Dr. Mandel’s presentation centered on the utilization of verbal versus numeric probabilities, the variability in understandings of verbal probabilities, and the relationship bet...

With their spotlight on Laplace’s inductive rule of succession, Fiedler et al. (2022) revisit important questions about what constitutes rationality and how rationality can be well tested in experimental research. In commentary on their article, I propose that Popper’s evolutionary epistemology can help address both questions.

Organizations in several domains including national security intelligence communicate judgments under uncertainty using verbal probabilities (e.g., likely) instead of numeric probabilities (e.g., 75% chance), despite research indicating that the former have variable meanings across individuals. In the intelligence domain, uncertainty is also commun...

Following significant intelligence failures, the United States intelligence community adopted Intelligence Community Directive 203 (ICD203) to promote analytic rigor. We developed two psychometric scales to examine how strongly intelligence professionals (N=108) endorsed the analytic standards comprising ICD203 and how strongly they believed their...

An influential program of psychological research suggests that people’s judgements and decisions depend on the way in which information is presented, or ‘framed’. In a central choice paradigm, decision-makers seem to adopt different preferences, and different attitudes to risk, depending on whether the options specify the number of people who will...

With their spotlight on Laplace’s inductive rule of succession, Fiedler et al. (2022) revisit important questions about what constitutes rationality and how rationality can be well tested in experimental research. In commentary on their article, I propose that Popper’s evolutionary epistemology can help address both questions.

An influential program of psychological research suggests that people’s judgements and decisions depend on the way in which information is presented, or ‘framed’. In a central choice paradigm, decision-makers seem to adopt different preferences, and different attitudes to risk, depending on whether the options specify the number of people who will...

Structured protocols offer a transparent and systematic way to elicit and combine/aggregate, probabilistic predictions from multiple experts. These judgements can be aggregated behaviourally or mathematically to derive a final group prediction. Mathematical rules (e.g., weighted linear combinations of judgments) provide an objective approach to agg...

This article surveys the latest research on risky-choice framing effects, focusing on the implications for rational decision-making. An influential program of psychological research suggests that people’s judgements and decisions depend on the way in which information is presented, or ‘framed’. In a central choice paradigm, decision-makers seem to...

Forecasting plays a vital role in intelligence assessment and contributes to national se- curity decision-making by improving strategic foresight. Remarkably, most intelligence organizations do not proactively track their forecasting accuracy and, therefore, do not know how accurate their forecasts are or what types of biases intelligence analysts...

Lustick and Tetlock outline an intellectually ambitious approach to scoping the future. They are particularly interested in sectors of national security and foreign policy decision-making that require an-ticipatory strategic intelligence that is difficult to produce because there is insufficient data, even if relevant theories are available. They p...

This article surveys the latest research on risky-choice framing effects, focusing on the implications for rational decision-making. An influential program of psychological research suggests that people’s judgements and decisions depend on the way in which information is presented, or ‘framed’. In a central choice paradigm, decision-makers seem to...

As US-China great power competition intensifies, public opinion polling may help gauge internal drivers of foreign policy decision-making. Using Pew Research Center data, we analyzed how Americans and Chinese perceived their own and each other’s countries between 2008-2016. We also compared these samples’ perceptions of current economic and future...

Forecasting plays a vital role in intelligence assessment and contributes to national security decision-making by improving strategic foresight. Remarkably, most intelligence organizations do not proactively track their forecasting accuracy and, therefore, do not know how accurate their forecasts are or what types of biases intelligence analysts (o...

Research suggests political identity has strong influence over individuals’ attitudes and beliefs, which in turn can affect their behavior. Likewise, firsthand experience with an issue can also affect attitudes and beliefs. A large (N = 6,383) survey (Pew Research and Ipsos W64) of Americans was analyzed to investigate the effects of both political...

In a recent article, Wall, Crookes, Johnson and Weber (2020) claim that Query Theory has better explanatory success in accounting for recent data than the Explicated Valence Account of Tombu and Mandel (2015). In this commentary, I first argue that this claim is not supported by the full range of available evidence. I then draw attention to the per...

Across a wide range of domains, experts make probabilistic judgments under conditions of uncertainty to support decision-making. These judgments are often conveyed using linguistic expressions (e.g., x is likely). Seeking to foster shared understanding of these expressions between senders and receivers, the US intelligence community implemented a c...

Fergnami and Chermack are to be credited for calling out the lack of scientific discipline in futures and foresight studies. The crux of their proposal is that the area is essentially in a prescientific stage of development due to a mix of historical antecedents, economic and ideological motives, and ignorance about science, both of its principles...

Experts are often asked to represent their uncertainty as a subjective probability. Structured protocols offer a transparent and systematic way to elicit and combine probability judgements from multiple experts. As part of this process, experts are asked to individually estimate a probability (e.g., of a future event) which needs to be combined/agg...

Replication is a hallmark of scientific research. As replications of individual studies are resource intensive, techniques for predicting the replicability are required. We introduce a new technique to evaluating replicability, the repliCATS (Collaborative Assessments for Trustworthy Science) process, a structured expert elicitation approach based...

Lustick and Tetlock outline an intellectually ambitious approach to scoping the future. They are particularly interested in sectors of national security and foreign policy decision-making that require anticipatory strategic intelligence that is difficult to produce because there is insufficient data, even if relevant theories are available. They pr...

Organizations tasked with communicating expert judgments couched in uncertainty often use numerically bounded linguistic probability schemes to standardize the meaning of verbal probabilities. An experiment (N = 1,202) was conducted to ascertain whether agreement with such a scheme was better when probabilities were presented verbally, numerically...

Research suggests political identity has strong influence over individuals’ attitudes and beliefs, which in turn can affect their behavior. Likewise, firsthand experience with an issue can also affect attitudes and beliefs. A large (N = 6,383) survey (Pew Research and Ipsos W64) of Americans was analyzed to investigate the effects of both political...

Research suggests political identity has strong influence over individuals’ attitudes and beliefs, which in turn can affect their behavior. Likewise, firsthand experience with an issue can also affect attitudes and beliefs. A large (N = 6,383) survey (Pew Research and Ipsos W64) of Americans was analyzed to investigate the effects of both political...

This is a commentary on a target article by Fergnami and Chermack. It addresses the issue of how the study of futures and foresight can ensure a scientific future. I comment on two aspects of the authors' essay: first, their emphasis on theory development, which I believe is overemphasized (at the cost of an emphasis on method) and, second, their a...

Probability information is regularly communicated to experts who must fuse multiple estimates to support decision-making. Such information is often communicated verbally (e.g., “likely”) rather than with precise numeric (point) values (e.g., “.75”), yet people are not taught to perform arithmetic on verbal probabilities. We hypothesized that the ac...

This paper examines intelligence analysis as a form of reasoning and judgment under conditions of uncertainty. As such it is susceptible to cognitive and motivational biases. I examine how the intelligence community has attempted to mitigate such biases and comment on the fact that it has adopted a fundamentally prescientific approach. Recent psych...

Following significant intelligence failures, the United States intelligence community adopted Intelligence Community Directive 203 (ICD203) to promote analytic rigor. This study developed two reliable psychometric scales to examine how strongly intelligence professionals (N=108) endorsed the ICD203 facets and the extent to which they believed their...

Key Points: • Several lines of research evidence point to the ineffectiveness of numerically bounded linguistic probability schemes. • Methods for effective communication of probability information should aim to boost users' capability to process quantitative information. • Lewis et al.'s (2019) recommendation for a numerically bounded linguistic p...

When people use high and low verbal probabilities such as probable and improbable, they expect them to have discriminable meanings. However, the context in which these terms are applied can affect how they are interpreted and how well they are discriminated. In an experiment, we examined the effect of positive and negative context (i.e., event occu...

Although the Analysis of Competing Hypotheses method (ACH) is a structured analytic technique promoted in several intelligence communities for improving the quality of probabilistic hypothesis testing, it has received little empirical testing. Whereas previous evaluations have used numerical evidence assumed to be perfectly accurate, in the present...

Three experiments (N = 550) examined the effect of an interval construction elicitation method used in several expert elicitation studies on judgment accuracy. Participants made judgments about topics that were either searchable or unsearchable online using one of two order variations of the interval construction procedure. One group of participant...

Organizations tasked with communicating expert judgments couched in uncertainty often use numerically bounded linguistic probability schemes to fix the meaning of verbal probabilities. An experiment (N=1,202 after exclusions) was conducted to ascertain whether agreement with such a scheme was better when probabilities were presented verbally, numer...

A reply to Simmons and Nelson (2013) was posted at https://sites.google.com/site/themandelian/data-colada in December, 2013. As it has been cited in literature, and as the website on which it appeared will soon be inactive, the aim of this document is to ensure the reply has a permanent record going forward. This version has been edited for style,...

This is a commentary on a forthcoming essay by Paul Schoemaker entitled, "How Historical Analysis Can Enrich Scenario Planning."

Across a wide range of domains, experts make probabilistic judgments under conditions of uncertainty to support decision-making. These judgments are often conveyed using linguistic expressions (e.g., x is likely). Seeking to foster shared understanding of these expressions between senders and receivers, the US intelligence community implemented a c...

Background. Decisions about trial funding, ethical approval, or clinical practice guideline recommendations require expert judgments about the potential efficacy of new treatments. We tested whether individual and aggregated expert opinion of oncologists could predict reliably the efficacy of cancer treatments tested in randomized controlled trials...

Intelligence analysis is fundamentally an exercise in expert judgment made under conditions of uncertainty. These judgments are used to inform consequential decisions. Following the major intelligence failure that led to the 2003 war in Iraq, intelligence organizations implemented policies for communicating probability in their assessments. Virtual...

Uncertainty is both inherent in nature and endemic to national security decision-making. Intelligence communities throughout the Western world, however, rely on vague language to communicate uncertainty—both the probability of critical events and the confidence that analysts have in their assessments—to decision-makers. In this article, we review t...

This is a postprint of the 20-chapter final report of the NATO SAS Research Technical Group on Assessment and Communication of Uncertainty in Intelligence to Support Decision-Making.

The primary objective of the SAS-114 Research Task Group was to investigate methods of improving intelligence assessments and communicating the uncertainties surrounding such assessments clearly to decision-makers. For this undertaking, SAS-114 members collected and evaluated a wide range of uncertainty communication standards currently used in def...

This is a commentary on a forthcoming essay by Paul Schoemaker entitled, "How Historical Analysis Can Enrich Scenario Planning."

Although the Analysis of Competing Hypotheses method (ACH) is a structured analytic technique promoted in several intelligence communities for improving the quality of probabilistic hypothesis testing, it has received little empirical testing. Whereas previous evaluations have used numerical evidence assumed to be perfectly credible, in the present...

This chapter reviews intelligence community standards for communicating probability in intelligence assessments.

Three experiments (N = 854) examined the effect of a four-step elicitation method used in several expert elicitation studies on judgment accuracy. Participants made judgments about topics that were either searchable or unsearchable online using one of two order variations of the four-step procedure. One group of participants provided their best jud...

This chapter from the forthcoming NATO SAS-114 final report examines the standards used in the US and UK to communicate probability in intelligence analysis in light of the extant literature on the communication of uncertainty.

In a recent issue of Earth’s Future [vol. 7, pp. 1020-1026], S. C. Lewis et al. recommended a numerically bounded linguistic probability (NBLP) scheme for communicating probabilistic information in extreme event attribution studies. We provide a critique of NBLP schemes in general and of Lewis et al.’s in particular, noting two key points. First, e...

Intelligence analysis is fundamentally an exercise in expert judgment made under conditions of uncertainty. These judgments are used to inform consequential decisions. Following the major intelligence failure that led to the 2003 war in Iraq, intelligence organizations implemented policies for communicating probability in their assessments. Virtual...

As in other areas of expert judgment, intelligence analysis often requires judging the probability that hypotheses are true. Intelligence organizations promote the use of structured methods such as “Analysis of Competing Hypotheses” (ACH) to improve judgment accuracy and analytic rigor, but these methods have received little empirical testing. In t...