No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Behavioral Consequences of Probabilistic Precision: Experimental Evidence from National Security Professionals
Abstract
National security is one of many fields where experts make vague probability assessments when evaluating high-stakes decisions. This practice has always been controversial, and it is often justified on the grounds that making probability assessments too precise could bias analysts or decision makers. Yet these claims have rarely been submitted to rigorous testing. In this paper, we specify behavioral concerns about probabilistic precision into falsifiable hypotheses which we evaluate through survey experiments involving national security professionals. Contrary to conventional wisdom, we find that decision makers responding to quantitative probability assessments are less willing to support risky actions and more receptive to gathering additional information. Yet we also find that when respondents estimate probabilities themselves, quantification magnifies overconfidence, particularly among low-performing assessors. These results hone wide-ranging concerns about probabilistic precision into a specific and previously undocumented bias that training may be able to correct.
... Analysts typically express uncertainty through verbal probability phrases, rather than numbers. They argue that doing so avoids the "illusion of rigor" (Friedman et al., 2017;Friedman & Zeckhauser, 2012), wherein decision-makers perceive a greater degree of scientific precision associated with judgments than is warranted. This may be especially true when analysts lack base rate information. ...
... Participants read three scenarios adapted from the "optimistic" version of the scenarios from Friedman et al. (2017). Before reading the first scenario, participants were provided abridged instructions from NIC (2023) regarding the meaning of confidence levels; this can be found in Part 2 of the supporting information and the scenarios can be found in Part 3. Scenario 1 described an operation to be conducted by US Special Forces to retrieve American civilian hostages being held by ISIS. Figure 1 displays this scenario in its entirety. ...
... Scores on the test range from 0 to 4 and higher scores represent greater numeracy. Numeracy is often included as a covariate in studies of the interpretation of verbal probability phrases (e.g., Budescu et al., 2012;Friedman et al., 2017;Wintle et al., 2019) because it is a trait that affects the basic evaluation of numbers, choice strategies, risk perceptions, and relative use of numeric and non-numeric information (Peters et al., 2006;Reyna et al., 2009). ...
Intelligence agencies communicate uncertainty to decision‐makers through verbal probability phrases that correspond to numerical ranges (i.e., probability lexicons) and ordinal levels of confidence. However, decision‐makers may misinterpret the relationship between these concepts and form inappropriate interpretations of intelligence analysts' uncertainty. In two experiments, four ways of conveying second‐order probability to decision‐makers were compared: (a) probability and confidence phrases written in the text of a report, (b) the addition of a probability lexicon, (c) the addition of a probability lexicon that varied numerical ranges according to the level of confidence (i.e., revised lexicon), and (d) a probability phrase written in text followed by a numerical range that varied according to the level of confidence. The revised lexicon was expected to improve interpretations of second‐order probability. The 275 participants in Experiment 1 and 796 participants in Experiment 2 provided numerical estimates corresponding to analytic judgments provided in descriptions about three overseas military operations and also indicated their support for approving or delaying the operations. The results demonstrated that providing the numerical range in the text of the report or providing a probability lexicon, improved interpretations of probability phrases above the verbal phrase‐only condition, but not interpretations of confidence. Participants were unable to correctly interpret confidence with respect to the precision of their estimate intervals and their decisions about the operations. However, in Experiments 2 and 3 the effects on these variables of providing decision‐makers with information about the source of the analyst's uncertainty were examined. In Experiment 3 ( n = 510), providing this information improved correspondence between confidence level and approval of the operation. Recommendations are provided regarding additional methods of improving decision‐makers' interpretation of second‐order probability conveyed in intelligence reporting.
... Conversely, those who are held accountable may feel trapped by numeric probabilities and would therefore prefer vague verbal probabilities when they are seeking to avoid accountability. Friedman, Lerner, and Zeckhauser (2017), surveying national security professionals, found decision-makers were less willing to support risky actions when those risks were conveyed quantitatively and more willing to seek additional information to improve their understanding of the situation. ...
... However, because numeric probabilities are evaluable and precise, policymakers may find themselves cut-off from employing such self-justifying behavior. Indeed, Friedman et al. (2017), surveying national security professionals, found decision-makers were less willing to support risky actions when those risks were conveyed quantitatively and more willing to seek additional information to improve their understanding of the situation. From this, we hypothesize that: ...
... Finally, what is politically rational might still be cognitively suboptimal in several ways. Friedman, Lerner, and Zeckhauser (2017) found that decision makers tended to be less willing to support risky actions after viewing numeric probabilities and were more willing to seek additional information. Numeric probabilities essentially exposed their gaps in knowledge. ...
Accountability is often presented as a panacea for behavioral ailments. This one-size-fits-all approach to a multi-dimensional construct ignores a key component of the effectiveness of accountability systems: situational context. Situational contexts such as highly stochastic environments (e.g., financial markets, world politics) and politically-charged domains (e.g., national security decision-making, domestic policy) form accountability boundary conditions, beyond which previous experimental effects may not generalize. In a series of studies, I explore the relatively under-explored frontiers of accountability effects, including those that apply to highly stochastic environments; politically-charged outcomes, where the tendency towards motivated reasoning dominates; and rapidly evolving states of information, where one’s ability to update one’s beliefs has serious implications for the quality of one’s judgments and decisions. In this series of studies, I find that accountability effects only appeared under certain conditions. In general, holding people accountable for their judgments did not improve performance on highly stochastic or politically-charged tasks—in fact, it sometimes made performance worse. However, certain types of accountability were able to boost performance in some contexts. These studies demonstrate the value of incorporating situational context into accountability experiments.
... Debates over uncertainty communication have long pervaded the intelligence discourse (e.g., Refs. [6], [7], [8]) and became acute following major intelligence failures; namely, the 9/11 attacks and Iraq WMD fiasco [9], [10]. To address the goal of mitigating subjectivity and the potential for miscommunication, some intelligence organizations have developed standardized lexicons for communicating estimative probability. ...
... Standards may omit numerical values due to the assumption that quantitative expressions lead consumers to be overconfident and overly risk seeking in their subsequent decision making. However, contrary to the first assumption, Friedman, Lerner, and Zeckhauser [9] find that national security officials presented numerical probability assessments were in fact less confident in supporting proposed actions and they were more amenable to additional information gathering. ...
This chapter reviews intelligence community standards for communicating probability in intelligence assessments.
... Despite the literature advocating numerical over verbal probabilities (e.g., [3,12,13,14]), many organisations and professionals that communicate uncertainty, from the Intelligence Community to doctors, prefer to use words to convey their probabilistic forecasts [6,13,15,16]. Words are often seen as a more appropriate vehicle for expressing vagueness and avoiding false precision [17], and to avoid giving an "illusion of rigor" [18]. In an effort to achieve more reliable interpretation, many such organisations have adopted standardised guidelines. ...
... The literature on verbal probabilities describes some opposition in professionals and organisations to reporting numbers alongside verbal expressions of uncertainty [6,13,15,16,18]. However, since the revision of intelligence standards outlined in the policy document ICD 203 [24], readers of intelligence reports have at least had access to numerical guidelines tables, just not directly in text. ...
People interpret verbal expressions of probabilities (e.g. ‘very likely’) in different ways, yet words are commonly preferred to numbers when communicating uncertainty. Simply providing numerical translations alongside reports or text containing verbal probabilities should encourage consistency, but these guidelines are often ignored. In an online experiment with 924 participants, we compared four different formats for presenting verbal probabilities with the numerical guidelines used in the US Intelligence Community Directive (ICD) 203 to see whether any could improve the correspondence between the intended meaning and participants’ interpretation (‘in-context’). This extends previous work in the domain of climate science. The four experimental conditions we tested were: 1. numerical guidelines bracketed in text, e.g. X is very unlikely (05–20%), 2. click to see the full guidelines table in a new window, 3. numerical guidelines appear in a mouse over tool tip, and 4. no guidelines provided (control). Results indicate that correspondence with the ICD 203 standard is substantially improved only when numerical guidelines are bracketed in text. For this condition, average correspondence was 66%, compared with 32% in the control. We also elicited ‘context-free’ numerical judgements from participants for each of the seven verbal probability expressions contained in ICD 203 (i.e., we asked participants what range of numbers they, personally, would assign to those expressions), and constructed ‘evidence-based lexicons’ based on two methods from similar research, ‘membership functions’ and ‘peak values’, that reflect our large sample’s intuitive translations of the terms. Better aligning the intended and assumed meaning of fuzzy words like ‘unlikely’ can reduce communication problems between the reporter and receiver of probabilistic information. In turn, this can improve decision making under uncertainty.
... Compared with verbal probabilities, point numeric values are also less likely to result in the selection of extreme outcomes in a distribution [66,67], leak information about the sender's reference point [38], or damage a sender's credibility after making erroneous predictions [18][19][20]. One study found that expressing uncertainty via precise numeric as opposed to verbal probabilities did not result in national security officials making riskier decisions, even for optimistic scenarios, and did not lead to overconfidence in decisions, but did increase willingness to gather additional information, desirable in this circumstance [80]. ...
Life in an increasingly information-rich but highly uncertain world calls for an effective means of communicating uncertainty to a range of audiences. Senders prefer to convey uncertainty using verbal (e.g., likely) rather than numeric (e.g., 75% chance) probabilities, even in consequential domains, such as climate science. However, verbal probabilities can convey something other than uncertainty, and senders may exploit this. For instance, senders can maintain credibility after making erroneous predictions. While verbal probabilities afford ease of expression, they can be easily misunderstood, and the potential for miscommunication is not effectively mitigated by assigning (imprecise) numeric probabilities to words. When making consequential decisions, recipients prefer (precise) numeric probabilities.
... On failures of intelligence, seeBar-Joseph and Kruglanski (2003),Jervis (2010), andShlaim (1976. 12 On this, seeZeckhauser (2012, 2018) andFriedman, Lerner, and Zeckhauser (2017).Downloaded from https://academic.oup.com/fpa/article/18/1/orab036/6458602 by guest on 11 December 2021 ...
A study of the daily briefings of US presidents by the intelligence community offers a useful test of whether governments can surmount intragovernmental influences in the acquisition and processing of information. A finding that the briefs somehow anticipate events would suggest that governments—their leaders and organizations—rise above political incentives and institutional practices to approach the rationality that realist and liberal scholars attribute to states. This study, thus, examines which countries appear in (the now declassified) daily intelligence briefs of the 1961–(January)1977 period, covering the Kennedy, Johnson, Nixon, and Ford years. It not only finds evidence that the selection of countries for the briefs favors countries referenced in prior briefs (per the foreign-policy literature) but also finds significant evidence that the appearance of countries, in the briefs, anticipates their increased activity in the period to follow (per a rational model).
... Hafner-Burton, Hughes, and Victor (2013), for example, find that elites Q9 behave more like rational decision-makers because they are less prone to loss aversion and are able to use heuristics more effectively when processing complex1000 information. Friedman, Lerner, and Zeckhauser (2017) find that national security managers have a stronger ability to understand probability estimates than does the general public. Future research could focus on the degree to which such differences impact elites' nuclear weapons preferences in peacetime and decision-making processes in crises. ...
That nuclear weapons have not been used in war since 1945 is one of the most intriguing research puzzles in the field of international relations. It has sparked a fruitful scholarly debate: Can the persistence of the nonuse of nuclear weapons be understood with reference to a normative “taboo” subject to a constructivist logic of appropriateness, or does it rather constitute a prudent tradition based on a logic of consequences as rationalist scholars would have it? Recently, a study by Daryl Press, Scott Sagan, and Benjamin Valentino provided further impetus for this debate and opened up a “second generation” of “taboo” research. Unlike the first generation, the second wave examined attitudes toward nuclear use among the general public rather than elite decision-makers and used large-N experimental surveys rather than in-depth interviews and archival research. In particular, these studies raised several methodological questions on how to capture the “atomic aversion”: Is it meaningful to examine public attitudes in order to grasp the validity of the nuclear “taboo” (as opposed to elite perspectives) and can we infer a weakening of the normative aversion toward nuclear use from public surveys? Bringing together the pioneers of the original debate as well as more recent contributors, this special forum seeks to take stock of the progress that has been made by discussing the theoretical, conceptual, and methodological underpinnings of the research on the nonuse of nuclear weapons. Specifically, the contributions critically reflect upon the second wave of nuclear taboo scholarship with the overall aim to build bridges between different theoretical approaches and to identify avenues for further research in this area. Ultimately, this forum seeks to present the relevance of re-envisioning nuclear taboo research to a broader audience.
... For instance, Timmermans (1994) found that medical professionals were more likely to agree on treatment decisions and were better at Bayesian reasoning when probabilistic information was presented numerically rather than verbally. In a study comparing 407 US national security officials' responses to verbal and numeric probability estimates, Friedman, Lerner, and Zeckhauser (2017) found that quantifying probabilities did not prompt officials to take riskier actions, even for optimistic scenarios. Nor did quantification result in overconfidence. ...
Intelligence analysis is fundamentally an exercise in expert judgment made under conditions of uncertainty. These judgments are used to inform consequential decisions. Following the major intelligence failure that led to the 2003 war in Iraq, intelligence organizations implemented policies for communicating probability in their assessments. Virtually all chose to convey probability using standardized linguistic lexicons in which an ordered set of select probability terms (e.g., highly likely) is associated with numeric ranges (e.g., 80–90%). We review the benefits and drawbacks of this approach, drawing on psychological research on probability communication and studies that have examined the effectiveness of standardized lexicons. We further discuss how numeric probabilities can overcome many of the shortcomings of linguistic probabilities. Numeric probabilities are not without drawbacks (e.g., they are more difficult to elicit and may be misunderstood by receivers with poor numeracy). However, these drawbacks can be ameliorated with training and practice, whereas the pitfalls of linguistic probabilities are endemic to the approach. We propose that, on balance, the benefits of using numeric probabilities outweigh their drawbacks. Given the enormous costs associated with intelligence failure, the intelligence community should reconsider its reliance on using linguistic probabilities to convey probability in intelligence assessments. Our discussion also has implications for probability communication in other domains such as climate science.
... For instance, Timmermans (1994) found that medical professionals were more likely to agree on treatment decisions and were better at Bayesian reasoning when probabilistic information was presented numerically rather than verbally. In a study comparing 407 US national security officials' responses to verbal and numeric probability estimates, Friedman, Lerner, and Zeckhauser (2017) found that quantifying probabilities did not prompt officials to take riskier actions, even for optimistic scenarios. Nor did quantification result in overconfidence. ...
Intelligence analysis is fundamentally an exercise in expert judgment made under conditions of uncertainty. These judgments are used to inform consequential decisions. Following the major intelligence failure that led to the 2003 war in Iraq, intelligence organizations implemented policies for communicating probability in their assessments. Virtually all chose to convey probability using standardized linguistic lexicons in which an ordered set of select probability terms (e.g., highly likely) is associated with numeric ranges (e.g., 80-90%). We review the benefits and drawbacks of this approach, drawing on psychological research on probability communication and studies that have examined the effectiveness of standardized lexicons. We further discuss how numeric probabilities can overcome many of the shortcomings of linguistic probabilities. Numeric probabilities are not without drawbacks (e.g., they are more difficult to elicit and may be misunderstood by receivers with poor numeracy). However, these drawbacks can be ameliorated with training and practice, whereas the pitfalls of linguistic probabilities are endemic to the approach. We propose that, on balance, the benefits of using numeric probabilities outweigh their drawbacks. Given the enormous costs associated with intelligence failure, the intelligence community should reconsider its reliance on using linguistic probabilities to convey probability in intelligence assessments. Our discussion also has implications for probability communication in other domains such as climate science.
... Moreover, intelligence communities should disabuse themselves of the false notion that precision will trigger unwarranted risk taking in decision makers. Recent evidence shows the opposite-decision makers are more cautious when they receive clearer information presented in numeric form than when they receive it via verbal probabilities [21]. Intelligence professionals might be concerned about such outcomes not only for virtuous reasons such as wanting to avoid false pretences for war, as in the 2003 Iraq weapons of mass destruction fiasco, but also for political cover from blame. ...
Command and policy decision-making alike depend on timely and accurate intelligence. As Sherman Kent had noted in the 1950s and 60s, intelligence assessments are seldom cold, hard facts but rather judgments made by experts under conditions of uncertainty. Since Kent, history has taught us that misjudging and miscommunicating uncertainties threaten prospects for operational and strategic successes. Nevertheless, NATO and its members' intelligence communities have persisted in using inadequate methods for assessing and communicating uncertainty, which rely on vague linguistic probabilities (e.g., "likely" or "unlikely"). Here, drawing on recent scientific evidence including from NATO SAS-114 and other research, I describe the principal reasons why the intelligence community should change course and consider using numeric probabilities for estimates that support important decisions. This is arguably more important now than ever since changes in the global security environment, which augment the importance of non-munitions targeting, call for characterization of deep uncertainties related to second-and higher-order effects.
... In at least one case, our studies were republished as part of NATO's "Security through Science" program . Our newer work, for example, studies assessing the accuracy of probability estimates made by defense and intelligence officers (Friedman, Lerner, & Zeckhauser, 2017), will hopefully provide useful guidance on when it is effective to express uncertainty using words rather than numbers. Advising Navy initiatives on the basis of decision-science findings, more generally, also falls into this category. ...
... Individuals are, for instance, risk-seeking when confronted with choices between losses and risk-averse when confronted with choices between gains (prospect theory, Kahneman & Tversky, 1979). While this insight has previously been applied to the field of foreign policy crisis management (McDermott, 1998), research about the risk calculations political and military leaders make when sending troops in harm's way is still at an early stage of development (e.g., Friedman, Lerner, & Zeckhauser, 2017;Haerem, Kuvaas, Bakken, & Karlsen, 2011;Kertzer, 2017;Macdonald & Schneider, 2017;Saunders, 2011;Trenta, 2016). ...
Risks are omnipresent in contemporary international security. Despite a long tradition in security studies going at least back to Von Clausewitz, we consider that the topic of risk remains under-examined. This forum seeks to advance the research agenda on risk in security studies by showcasing work of scholars using advanced concepts of risk, based on insights from sociology, biology, psychology, and safety studies, to better understand the role of risk in international security. As a way of introduction, this short article sets out the main debates.
... The US National Intelligence Council (2007, iv) thus explained its use of qualitative probability phrasings by writing that "assigning precise numerical ratings to such judgments would imply more rigor than we intend." For a recent theoretical and empirical examination of this "illusions of rigor" thesis, see Friedman, Lerner, and Zeckhauser (2017). ...
Scholars, practitioners, and pundits often leave their assessments of uncertainty vague when debating foreign policy, arguing that clearer probability estimates would provide arbitrary detail instead of useful insight. We provide the first systematic test of this claim using a data set containing 888,328 geopolitical forecasts. We find that coarsening numeric probability assessments in a manner consistent with common qualitative expressions—including expressions currently recommended for use by intelligence analysts—consistently sacrifices predictive accuracy. This finding does not depend on extreme probability estimates, short time horizons, particular scoring rules, or individual attributes that are difficult to cultivate. At a practical level, our analysis indicates that it would be possible to make foreign policy discourse more informative by supplementing natural language-based descriptions of uncertainty with quantitative probability estimates. More broadly, our findings advance long-standing debates over the nature and limits of subjective judgment when assessing social phenomena, showing how explicit probability assessments are empirically justifiable even in domains as complex as world politics.
... 5. The latter could explain why our study finds little effect of attributional certainty while that of Friedman et al. (2016) finds that individuals do care about changes in probability estimates. Our study varies two aspects of attribution, and it may be that the partisan modifier is diluting the effect of the probability estimates themselves, meaning that the two sets of results are not necessarily incompatible. ...
Accusations of Russian hacking in the 2016 US presidential election has raised the salience of cyber security among the American public. However, there are still a number of unanswered questions about the circumstances under which particular policy responses are warranted in response to a cyber-attack and the public’s attitudes about the conditions that justify this range of responses. This research investigates the attributes of a cyber-attack that affect public support for retaliation. It finds that cyber-attacks that produce American casualties dramatically increase support for retaliatory airstrikes compared to attacks with economic consequences. Assessments of attribution that have bipartisan support increase support to a lesser extent but for a broader range of retaliatory measures. The findings have important implications for ongoing debates about cyber security policy.
Globalization has led to an increasingly interconnected international community and transmitted risks to every corner of the world through the chain of globalization. Security risks arising from international conflicts seem inescapable. This paper proposes four hypotheses about the phenomenon of “risks and security disembedding” in post-Cold War international society and uses them to explain the United Kingdom's behavior in the Bosnian War and the Russo-Ukrainian War. In the Bosnian War, confident in its security and focused on maintaining European stability, the UK has therefore chosen to maintain a very limited intervention in Bosnia's affairs. In contrast, the instability of the EU and NATO's security mechanisms in the Russo-Ukrainian war heightened The UK's anxiety, and the volatile international situation led it to show a strong tendency towards security disembedding. Analysis suggests that security mechanisms are also the starting point of conflict and that the state will rely more on disembedding mechanisms to counteract global security risks.
The International Auditing and Assurance Standards Board (IAASB) issued for public comment Proposed International Standard on Auditing 570 (Revised) Going Concern. The Australian Auditing and Assurance Standards Board (AUASB) and New Zealand Auditing and Assurance Standards Board (NZAuASB) also called for comments. The Auditing and Assurance Standards Committee of AFAANZ prepared a submission, based on the findings reported in extant research, informing a number of the questions asked by the IAASB. This technical note presents the formal submission made to the IAASB.
Assessing resolve and interpreting costly signals are crucial tasks for leaders engaging in international diplomacy. However, leaders rarely make these decisions in isolation, relying on advisers to help assess adversary intentions. How do group dynamics change the way costly signals are interpreted? We field a large‐scale group experiment to examine how assessments of resolve vary across group settings. We find groups make significantly higher initial assessments of adversary resolve than individuals do, but also update their beliefs less after receiving new information. In small group contexts, first impressions may play a stronger role in shaping beliefs than any signals—costly or otherwise—that come afterwards. This has important implications for our understanding of signaling, providing further evidence that costly signals are less straightforward than often assumed.
Human analysts working for threat intelligence leverage tools powered by artificial intelligence to routinely assemble actionable intelligence. Yet, threat intelligence sources and methods often have significant uncertainties and biases. In addition, data sharing might be limited for operational, strategic, or legal reasons. Experts are aware of these limitations but lack formal means to represent and quantify these uncertainties in their daily work. In this chapter, we enunciate the technical, legal, and societal challenges for building explainable AI for threat intelligence. We also discuss ideas for overcoming these challenges.
Fazy przygotowania, analizy, integracji, wstępnej interpretacji danych i informacji wywiadowczych są stosunkowo szeroko opisane w literaturze. Jedynie dziedzina oceny pewności źródeł oraz wiarygodności danych i informacji wywiadowczych nie nadąża za rozwojem innych elementów w domenie działań informacyjnych. Ze względu na coraz intensywniejsze działania prowadzone przez potencjalnych przeciwników należy obecnie stosowane metody, techniki i narzędzia poddać krytycznej ocenie i zidentyfikować ich ograniczenia, a także podjąć próby opracowania i wdrożenia nowych procesów i procedur. Przede wszystkim trzeba zwiększać możliwości przygotowania i komunikowania coraz dokładniejszych ocen pewności źródeł oraz wiarygodności danych i informacji. W związku z tym należy: kwantyfikować dokładność informacji, przygotować nowe procedury i oprogramowanie, badać stopień nadmiarowości informacji, jej kompletność i poziom diagnostyczności. Pracownicy aparatów zdobywającego i analitycznego muszą mieć wiedzę na te-mat istniejących ograniczeń i poszukiwać sposobów rozwiązywania problemów. Takie poszukiwania nie powinny skupiać się na meto-dach uniwersalnych, lecz na pragmatycznym podejściu do każdego z elementów. Practical dimension of issues related to assessing the reliability of sources and the trustworthiness of data and information: The phases of preparation, analysis, integration, initial interpretation of data and intelligence are relatively widely described in the literature. Only the field of assessing the certainty of sources and the reliability of data and intelligence has not kept pace with the development of other elements in the domain of information operations. In view of the increasing intensity of activities carried out by potential adversaries, the methods, techniques and tools currently in use should be critically evaluated and their limitations identified, and attempts should be made to develop and implement new processes and procedures. Above all, the capacity to prepare and communicate increasingly accurate assessments of the certainty of sources and the reliability of data and information must be enhanced. Therefore, it is necessary to: quantify the accuracy of the information, prepare new procedures and software, study the degree of information redundancy, its completeness and level of diagnosticity. Acquisition and analytical apparatus staff must be aware of existing limitations and search for ways to solve problems. Such a search should not focus on one-size-fits-all methods, but on a pragmatic approach to each element.
The phases of preparation, analysis, integration, initial interpretation of data and intelligence are relatively widely described in the literature. Only the field of assessing the certainty of sources and the reliability of data and intelligence has not kept pace with the development of other elements in the domain of information operations. In view of the increasing intensity of activities carried out by potential adversaries, the methods, techniques and tools currently in use should be critically evaluated and their limitations identified, and attempts should be made to develop and implement new processes and procedures. Above all, the capacity to prepare and communicate increasingly accurate assessments of the certainty of sources and the reliability of data and information must be enhanced. Therefore, it is necessary to: quantify the accuracy of the information, prepare new procedures and software, study the degree of information redundancy, its completeness and level of diagnosticity. Acquisition and analytical apparatus staff must be aware of existing limitations and search for ways to solve problems. Such a search should not focus on one-size-fits-all methods, but on a pragmatic approach to each element.
We interviewed national security professionals to understand why the U.S. Intelligence Community has not systematically incorporated prediction markets or prediction polls into its intelligence reporting. This behavior is surprising since crowdsourcing platforms often generate more accurate predictions than traditional forms of intelligence analysis. Our interviews suggest that three principal barriers to adopting these platforms involved (i) bureaucratic politics, (ii) decision-makers lacking interest in probability estimates, and (iii) lack of knowledge about these platforms’ capabilities. Interviewees offered many actionable suggestions for addressing these challenges in future efforts to incorporate crowdsourcing platforms or other algorithmic tools into intelligence tradecraft.
One of the major developments in political science in the past decade has been the rise of experiments and surveys on political elites. Yet, the increase in the number of elite studies has outpaced our collective understanding of best practices and how we know a good elite experiment when we see one. In this article, we discuss some of the challenges in the study of political elites—from who counts as an elite to how to best utilize elite experiments in the context of broader research designs. We also offer recommendations on questions of access, recruitment, and representativeness, as well as designs that researchers can use to study “eliteness” without access to elites.
How effective are states at assessing and predicting the nuclear intentions of foreign countries? Drawing on close to 200 US assessments of foreign countries’ proliferation intentions between 1957 and 1966, this research note finds that close to 80 percent of testable US assessments were correct and that they shifted from highly inaccurate in the late 1950s to highly accurate in the 1960s. Based on quantitative and qualitative analysis, I conclude that learning from early failures led the intelligence community to achieve higher accuracy.
Political scientists often criticize psychological approaches to the study of politics on the grounds that many psychological theories were developed on convenience samples of college students or members of the mass public, whereas many of the most important decisions in politics are made by elites, who are presumed to differ systematically from ordinary citizens. This paper proposes an overarching framework for thinking about differences between elites and masses, presenting the results of a meta‐analysis of 162 paired treatments from paired experiments on political elites and mass publics, as well as an analysis of 12 waves of historical elite and mass public opinion data on foreign policy issues over a 43 year period. It finds political scientists both overstate the magnitude of elite‐public gaps in decision‐making, and misunderstand the determinants of elite‐public gaps in political attitudes, many of which are due to basic compositional differences rather than to elites' domain‐specific expertise.
Uncertainty is both inherent in nature and endemic to national security decision-making. Intelligence communities throughout the Western world, however, rely on vague language to communicate uncertainty—both the probability of critical events and the confidence that analysts have in their assessments—to decision-makers. In this article, we review the status-quo approach taken by the intelligence community and, drawing on abundant research findings, we describe fundamental limitations with the approach, including the inherent vagueness, context-dependence, and implicit meanings that attend the use of verbal uncertainty expressions. We then present an alternative approach based on the use of imprecise numeric estimates supplemented by clear written rationales, highlighting the affordances of this alternative. Finally, we describe institutional barriers to reform and address common objections raised by practitioners. While we focus our discussion on the domain of national security intelligence, the case for numeric probabilities is relevant to any organizational field where high-stakes decisions are made under conditions of uncertainty.
Uncertainty is both inherent in nature and endemic to national security decision-making. Intelligence communities throughout the Western world, however, rely on vague language to communicate uncertainty—both the probability of critical events and the confidence that analysts have in their assessments—to decision-makers. In this article, we review the status-quo approach taken by the intelligence community and, drawing on abundant research findings, we describe fundamental limitations with the approach, including the inherent vagueness, context-dependence, and implicit meanings that attend the use of verbal uncertainty expressions. We then present an alternative approach based on the use of imprecise numeric estimates supplemented by clear written rationales, highlighting the affordances of this alternative. Finally, we describe institutional barriers to reform and address common objections raised by practitioners. While we focus our discussion on the domain of national security intelligence, the case for numeric probabilities is relevant to any organizational field where high-stakes decisions are made under conditions of uncertainty.
National security decision-making is informed by intelligence assessments, which in turn depend on sound information evaluation. Recognizing this fact, intelligence organizations have promulgated methods for evaluating source and evidential characteristics. We critically examine these methods and identify several limitations that undermine the fidelity of information evaluation. We argue that these limitations are symptomatic of a deep-seated tendency in the intelligence community to mask rather than effectively guide subjectivity in intelligence assessment. Drawing on the guidance metaphor, we propose that rigid 'all-purpose' information evaluation methods be replaced by flexible 'context-sensitive' guidelines aimed at improving the soundness, precision, accuracy, and clarity of irreducibly subjective judgments. Specific guidelines, supported by empirical evidence, include use of numeric probability estimates to quantify the judged likelihood of information accuracy, promoting collector-analyst collaboration, and periodic revaluation of information as new information is acquired. We also offer a set of questions to guide information evaluation.
Political psychology in international relations (IR) has undergone a dramatic transformation in the past two decades, mirroring the broader changes occurring in IR itself. This review examines the current state of the field.Webegin by offering a data-driven snapshot analyzing four years of manuscript classifications at a major IR journal to characterize the questions that IR scholars engaged in psychological research are and are not investigating. We then emphasize six developments in particular, both present-day growth areas (an increased interest in emotions and hot cognition, the rise of more psychologically informed work on public opinion, a nascent research tradition we call the first image reversed, and the rise of neurobiological and evolutionary approaches) and calls for additional scholarship (better integration of the study of mass and elite political behavior and more psychological work in international political economy). Together, these developments constitute some of the directions in which we see the next generation of scholarship heading. Expected final online publication date for the Annual Review of Political Science Volume 21 is May 11, 2018. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.
This book is the first work to build a conceptual framework describing how the pursuit of military effectiveness can present military and political tradeoffs, such as undermining political support for the war, creating new security threats, and that seeking to improve effectiveness in one aspect can reduce effectiveness in other aspects. Here are new ideas about military effectiveness, covering topics such as military robotics, nuclear weapons, insurgency, war finance, public opinion, and others. The study applies these ideas to World War II, the Korean War, the Vietnam War, and the 1973 October War, as well as ongoing conflicts and public policy debates, such as the War on Terror, drone strikes, ISIS, Russian aggression against Ukraine, US-Chinese-Russian nuclear competitions, and the Philippines insurgency, among others. Both scholarly and policy-oriented readers will gather new insights into the political dimensions of military power, and the complexities of trying to grow military power. Develops new ideas about important elements of military effectiveness. Mathematical treatments are avoided. Addresses contemporary policy topics such as drone strikes, military robotics, counterinsurgency, and others.
The Behavioral Revolution and International Relations - Volume 71 Issue S1 - Emilie M. Hafner-Burton, Stephan Haggard, David A. Lake, David G. Victor
As Amazon’s Mechanical Turk (MTurk) has surged in popularity throughout political science, scholars have increasingly challenged the external validity of inferences made drawing upon MTurk samples. At workshops and conferences experimental and survey-based researchers hear questions about the demographic characteristics, political preferences, occupation, and geographic location of MTurk respondents. In this paper we answer these questions and present a number of novel results. By introducing a new benchmark comparison for MTurk surveys, the Cooperative Congressional Election Survey, we compare the joint distributions of age, gender, and race among MTurk respondents within the United States. In addition, we compare political, occupational, and geographical information about respondents from MTurk and CCES. Throughout the paper we show several ways that political scientists can use the strengths of MTurk to attract respondents with specific characteristics of interest to best answer their substantive research questions.
Intelligence analysis plays a vital role in policy decision-making. Key functions of intelligence analysis include accurately forecasting significant events, appropriately characterizing the uncertainties inherent in such forecasts, and effectively communicating those probabilistic forecasts to stakeholders. We review decision research on probabilistic forecasting and uncertainty communication, drawing attention to findings that could be used to reform intelligence processes and contribute to more effective intelligence oversight. We recommend that the intelligence community regularly and quantitatively monitor its forecasting accuracy to better understand how well it is achieving its functions. We also recommend that the intelligence community use decision science to improve these functions (namely, forecasting and communication of intelligence estimates made under conditions of uncertainty). In the case of forecasting, decision research offers suggestions for improvement that involve interventions on data (e.g., transforming forecasts to debias them) and behavior (e.g., via selection, training, and effective team structuring). In the case of uncertainty communication, the literature suggests that current intelligence procedures, which emphasize the use of verbal probabilities, are ineffective. The intelligence community should, therefore, leverage research that points to ways in which verbal probability use may be improved as well as exploring the use of numerical probabilities wherever feasible.
The accuracy of 1,514 strategic intelligence forecasts abstracted
from intelligence reports was assessed. The results show that
both discrimination and calibration of forecasts was very good.
Discrimination was better for senior (versus junior) analysts and
for easier (versus harder) forecasts. Miscalibration was mainly due
to underconfidence such that analysts assigned more uncertainty
than needed given their high level of discrimination. Underconfidence
was more pronounced for harder (versus easier) forecasts
and for forecasts deemed more (versus less) important for policy
decision-making. In spite of the observed underconfidence, there
was a paucity of forecasts in the least informative 0.4-0.6 probability
range. Recalibrating the forecasts substantially reduced
underconfidence. The findings offer cause for tempered optimism
about the accuracy of strategic intelligence forecasts and indicate
that intelligence producers aim to promote informativeness while
avoiding overstatement.
Five university-based research groups competed to recruit forecasters, elicit their predictions, and aggregate those predictions to assign the most accurate probabilities to events in a 2-year geopolitical forecasting tournament. Our group tested and found support for three psychological drivers of accuracy: training, teaming, and tracking. Probability training corrected cognitive biases, encouraged forecasters to use reference classes, and provided forecasters with heuristics, such as averaging when multiple estimates were available. Teaming allowed forecasters to share information and discuss the rationales behind their beliefs. Tracking placed the highest performers (top 2% from Year 1) in elite teams that worked together. Results showed that probability training, team collaboration, and tracking improved both calibration and resolution. Forecasting is often viewed as a statistical problem, but forecasts can be improved with behavioral interventions. Training, teaming, and tracking are psychological interventions that dramatically increased the accuracy of forecasts. Statistical algorithms (reported elsewhere) improved the accuracy of the aggregation. Putting both statistics and psychology to work produced the best forecasts 2 years in a row.
We introduce the Berlin Numeracy Test, a new psychometrically sound instrument that quickly assesses statistical numeracy and risk literacy. We present 21 studies (n=5336) showing robust psychometric discriminability across 15 countries (e.g., Germany, Pakistan, Japan, USA) and diverse samples (e.g., medical professionals, general populations, Mechanical Turk web panels). Analyses demonstrate desirable patterns of convergent validity (e.g., numeracy, general cognitive abilities), discriminant validity (e.g., personality, motivation), and criterion validity (e.g., numerical and non-numerical questions about risk). The Berlin Numeracy Test was found to be the strongest predictor of comprehension of everyday risks (e.g., evaluating claims about products and treatments; interpreting forecasts), doubling the predictive power of other numeracy instruments and accounting for unique variance beyond other cognitive tests (e.g., cognitive reflection, working memory, intelligence). The Berlin Numeracy Test typically takes about three minutes to complete and is available in multiple languages and formats, including a computer adaptive test that automatically scores and reports data to researchers (www.riskliteracy.org). The online forum also provides interactive content for public outreach and education, and offers a recommendation system for test format selection. Discussion centers on construct validity of numeracy for risk literacy, underlying cognitive mechanisms, and applications in adaptive decision support.
What is the role of intelligence agencies in strategy and policy? How do policymakers use (or misuse) intelligence estimates? When do intelligence–policy relations work best? How do intelligence-policy failures influence threat assessment, military strategy, and foreign policy? These questions are at the heart of recent national security controversies, including the 9/11 attacks and the war in Iraq. In both cases the relationship between intelligence and policy broke down—with disastrous consequences. This book explores the complex interaction between intelligence and policy and shines a spotlight on the problem of politicization. Major episodes in the history of American foreign policy have been closely tied to the manipulation of intelligence estimates. The book describes how the Johnson administration dealt with the intelligence community during the Vietnam War; how President Nixon and President Ford politicized estimates on the Soviet Union; and how pressure from the George W. Bush administration contributed to flawed intelligence on Iraq. It also compares the U.S. case with the British experience between 1998 and 2003, and demonstrates that high-profile government inquiries in both countries were fundamentally wrong about what happened before the war.
This article examines the ethics of crowdsourcing in social science research, with reference to my own experience using Amazon’s Mechanical Turk. As these types of research tools become more common in scholarly work, we must acknowledge that many participants are not one-time respondents or even hobbyists. Many people work long hours completing surveys and other tasks for very low wages, relying on those incomes to meet their basic needs. I present my own experience of interviewing Mechanical Turk participants about their sources of income, and I offer recommendations to individual researchers, social science departments, and journal editors regarding the more ethical use of crowdsourcing.
According to a growing tradition in International Relations, one way governments can credibly signal their intentions in foreign policy crises is by creating domestic audience costs: leaders can tie their hands by publicly threatening to use force since domestic publics punish leaders who say one thing and do another. We argue here that there are actually two logics of audience costs: audiences can punish leaders both for being inconsistent (the traditional audience cost), and for threatening to use force in the first place (a belligerence cost). We employ an experiment that disentangles these two rationales, and turn to a series of dispositional characteristics from political psychology to bring the audience into audience cost theory. Our results suggest that traditional audience cost experiments may overestimate how much people care about inconsistency, and that the logic of audience costs (and the implications for crisis bargaining) varies considerably with the leader's constituency.
We examine the trade-offs associated with using Amazon.com's Mechanical Turk (MTurk) interface for subject recruitment. We first describe MTurk and its promise as a vehicle for performing low-cost and easy-to-field experiments. We then assess the internal and external validity of experiments performed using MTurk, employing a framework that can be used to evaluate other subject pools. We first investigate the characteristics of samples drawn from the MTurk population. We show that respondents recruited in this manner are often more representative of the U.S. population than in-person convenience samples-the modal sample in published experimental political science-but less representative than subjects in Internet-based panels or national probability samples. Finally, we replicate important published experimental work using MTurk samples. © The Author 2012. Published by Oxford University Press on behalf of the Society for Political Methodology. All rights reserved.
Status has long been implicated as a critical value of states and leaders in international politics. However, decades of research on the link between status and conflict have yielded divergent findings, and little evidence of a causal relationship. I attempt to resolve this impasse by shifting the focus from status to relative status concerns in building a theory of status from the ground up, beginning with its behavioral microfoundations. I build on and extend previous work through an experimental study of status threats and the escalation of commitment, operationalized here as a new behavioral escalation task using real financial incentives and framed around a narrative of war and peace. I utilize a unique sample of high-profile political and military leaders from the Senior Executive Fellow (SEF) program at the Harvard Kennedy School, as well as a group of demographically matched control subjects, allowing me to evaluate the moderating effect of power on status concerns while also addressing typical concerns about external validity in IR experiments. I find strong evidence that the fear of losing status impedes decision making and increases the tendency to “throw good money after bad,” but that power aids decision making by buffering high-power subjects against the worst effects of status loss.
The Intergovernmental Panel on Climate Change (IPCC) uses verbal descriptions of uncertainty (for example, Unlikely) to convey imprecision in its forecasts and conclusions. Previous studies showed that the American public misinterprets these probabilistic statements. We report results from a multi-national study involving 25 samples in 24 countries and 17 languages. As predicted, laypeople interpret IPCC statements as conveying probabilities closer to 50% than intended by the IPCC authors. We show that an alternative presentation format supplementing the verbal terms with numerical ranges increases the correspondence between the publicâ (tm) s interpretations and the IPCC guidelines, and the terms are better differentiated. These qualitative patterns are remarkably stable across all samples and languages. In fact, interpretations of the terms in various languages are more similar under the new presentation format. These results suggest changing the way the IPCC communicates uncertainty.
A number of researchers have advocated the use of explicit numeric probabilities in the drafting and presentation of intelligence assessments. A trial of this methodology by a Canadian assessment unit has demonstrated that numeric probabilities can be used effectively in the preparation of intelligence reports. The practice leads analysts to pay greater attention to the estimative judgments they make and allows for greater transparency in understanding the degree of certainty that they attach to their conclusions. The use of numeric probabilities highlights the need for a consistent mapping standard for verbal probability terms. However, implementing the methodology requires overcoming a widely shared aversion among analysts and intelligence managers to thinking in numeric terms.
At conferences, at seminars, and on political science blogs, the potential utility of experimental methods for international relations (IR) research continues to be a hotly contested topic. Given the recent rise in creative applications of experimental methods, now is a useful moment to reflect more generally on the potential value of experiments to study international affairs, how these inherently micro-level methods can shed light on bigger-picture questions, what has been learned already, what goals are probably out of reach, and how various research agendas in IR might productively incorporate experiments.
In a series of reports and meetings in Spring 2011, intelligence analysts and officials debated the chances that Osama bin Laden was living in Abbottabad, Pakistan. Estimates ranged from a low of 30 or 40 per cent to a high of 95 per cent. President Obama stated that he found this discussion confusing, even misleading. Motivated by that experience, and by broader debates about intelligence analysis, this article examines the conceptual foundations of expressing and interpreting estimative probability. It explains why a range of probabilities can always be condensed into a single point estimate that is clearer (but logically no different) than standard intelligence reporting, and why assessments of confidence are most useful when they indicate the extent to which estimative probabilities might shift in response to newly gathered information.
One of the most striking findings in political science is the democratic peace: the absence of war between democracies. Some authors attempt to explain this phenomenon by highlighting the role of public opinion. They observe that democratic leaders are beholden to voters and argue that voters oppose war because of its human and financial costs. This logic predicts that democracies should behave peacefully in general, but history shows that democracies avoid war primarily in their relations with other democracies. In this article we investigate not whether democratic publics are averse to war in general, but whether they are especially reluctant to fight other democracies. We embedded experiments in public opinion polls in the United States and the United Kingdom and found that individuals are substantially less supportive of military strikes against democracies than against otherwise identical autocracies. Moreover, our experiments suggest that shared democracy pacifies the public primarily by changing perceptions of threat and morality, not by raising expectations of costs or failure. These findings shed light on a debate of enduring importance to scholars and policy makers.
How strong are normative prohibitions on state behavior? We examine this question by analyzing anti-nuclear norms, sometimes called the “nuclear taboo,” using an original survey experiment to evaluate American attitudes regarding nuclear use. We find that the public has only a weak aversion to using nuclear weapons and that this aversion has few characteristics of an “unthinkable” behavior or taboo. Instead, public attitudes about whether to use nuclear weapons are driven largely by consequentialist considerations of military utility. Americans’ willingness to use nuclear weapons increases dramatically when nuclear weapons provide advantages over conventional weapons in destroying critical targets. Americans who oppose the use of nuclear weapons seem to do so primarily for fear of setting a negative precedent that could lead to the use of nuclear weapons by other states against the United States or its allies in the future.
Alan Beyerchen teaches German history in the Department of History at Ohio State University, where he is also affiliated with the Division for Comparative Studies and the Mershon Center. This paper is the first part of a larger project on the implications of nonlinear science for the liberal arts.
For their comments and encouragement, I am indebted to Kaushik Bagchi, Christopher Bassford, Lisa Barber, Leonard Jossem, Edward Merta, Raymond Muessig, Williamson Murray, Barbara J. Reeves, Oliver Schmidt, David Staley, Raymond Stokes, Ruud van Dijk, Paul Watkins, Barry Watts, Bostwick Wyman, David Young, Keith Zahniser; my students in Humanities 792 during Spring Quarter 1991; participants in colloquia held at the Institute for Contemporary History in Athens, Ohio, in November 1990 and November 1991; and members of the Implications of Nonlinear Studies Working Group at Ohio State University, especially Randolph Roth.
1. Carl von Clausewitz, On War, ed. and trans. Michael Howard and Peter Paret (Princeton: Princeton University Press, 1976). I use this edition for all quotations from On War in English unless otherwise indicated. For the German, see Vom Kriege, 18th ed. (complete edition of original text), ed. Werner Hahlweg (Bonn: Dümmlers, 1973). For other works in English, see von Clausewitz, Historical and Political Writings, ed. and trans. Peter Paret and Daniel Moran (Princeton: Princeton University Press, 1992).
2. Hans Rothfels, "Clausewitz," in Edward Mead Earle, ed., Makers of Modern Strategy (New York: Atheneum, 1969), p. 93. Christopher Bassford offers one impression of the reception of Clausewitz's work in his study of the Anglo-American reception of Clausewitz, 1815-1945 (Oxford: Oxford University Press, in press).
3. Raymond Aron, Clausewitz: Philosopher of War, trans. Christine Booker and Norman Stone (London: Routledge and Kegan Paul, 1983), p. 6. Original Penser la guerre, Clausewitz, 2 vols. (Paris: Gallimard, 1976). The suggestion has recently been made that the text was actually much more finished than has hitherto been thought: Azar Gat, "Clausewitz's Final Notes," Militärgeschichtliche Mitteilungen, Vol. 45, No. 1 (1989), pp. 45-50.
4. Peter Paret, Clausewitz and the State: The Man, His Theories and His Times (Princeton: Princeton University Press, 1985), pp. 8-9 (originally published by Oxford University Press, 1976). Azar Gat's argument, that Clausewitz's work is best understood as part of the Romantic backlash against the Enlightenment, also belongs to this approach. See Gat, The Origins of Military Thought: From the Enlightenment to Clausewitz (Oxford: Oxford University Press, 1989).
5. Michael I. Handel, War, Strategy and Intelligence (London: Frank Cass, 1989), p. 60.
6. Alan Beyerchen, "Nonlinear Science and the Unfolding of a New Intellectual Vision," Papers in Comparative Studies, Vol. 6 (1988-89), pp. 26-29.
7. The principle of proportionality means that if f is a function or an operator, a is a constant, and u is the system input (either a variable or itself a function), then f(au) = af(u). A more precise way of stating the principle of additivity is that the effect of adding the system inputs together first and then operating on their sum is equivalent to operating on two inputs separately and then adding the outputs together, so that f(u1 + u2) = f(u1) + f(u2). If f does not meet both of these conditions, it is nonlinear. In effect, if a system can be described adequately by the mathematical operations of addition, subtraction, multiplication by a constant, integration with respect to time or differentiation with respect to time, it can appropriately be thought of as linear. If it is necessary to multiply or divide variables by each other, raise to powers, extract roots, or integrate or differentiate with respect to dependent variables (that is, variables other than time), then the system is nonlinear.
8. The meaning of a "synergistic" interaction is indicated by the contrast between a common linear operation and a common nonlinear one. A linear operation such as multiplying by a constant obeys the principle of additivity: let f(u) = au, then f(u1 + u2) = a(u1 + u2) = au1 + au2, which is just f(u1) + f(u2) again. A nonlinear operation such as squaring, however, is different: let...
There is a well-settled maxim that the standard of persuasion in criminal trials—proof beyond a reasonable doubt—is unquantifiable.
However, the usual reasons given for the unquantifiability of reasonable doubt are unsatisfactory; and a recent case, United States v. Copeland, serves as a reminder that strong considerations favour quantification of at least some standards of persuasion. This comment
attempts to bring greater clarity to the question of the advantages and disadvantages of some form of quantification of the
reasonable doubt standard.
An effective terrorism alert system in a federal government has one central task: to motivate actors to take costly protective measures. The United States' color-coded Homeland Security Advisory System (HSAS) failed in this mission. In federal systems, national leaders cannot compel protective actions by setting an alert level; they must convince constituent governments and private parties that the desired actions are worth the costs. Such beliefs can be generated either by sharing the information behind an alert or by developing enough confidence in the alert system that the government's word alone suffices. The HSAS did neither, largely because it was not designed to generate confidence. Rather, the system's creators assumed that the public would trust the national leadership and believe in the utility of the system's information. Over time, as the HSAS became increasingly perceived as politically manipulated, there was no built-in mechanism to recover confidence in the system. An alternative, trust-based terrorist alert system could solve this problem. Building on the notion of "procedural fairness" from the psychological and legal traditions, this system would retain the political advantages of the HSAS, facilitate greater compliance among the requisite actors, and ameliorate many of the strategic problems inherent in terror alert systems.
I. Are there uncertainties that are not risks? 643. — II. Uncertainties that are not risks, 647. — III. Why are some uncertainties
not risks? — 656.
There is a marked gap between the demands on forecasting and the results that numerical forecasting techniques usually can provide. It is suggested that this gap can be closed by the implementation of experts' qualitative predictions into numerical forecasting systems. A formal analysis of these predictions can then be integrated into quantitative forecasts.In the framework of possibility theory, a model is developed which accounts for the verbal judgments in situations where predictions are made or knowledge is updated in the light of new information. The model translates verbal expressions into elastic constraints on a numerical scale. This numerical interpretation of qualitative judgments can then be implemented into numerical forecasting procedures.The applicability of this model was tested experimentally. The results indicate that the numerical predictions from the model agree well with the actual judgments and the evaluation behavior of the subjects.The applicability of this model is demonstrated in a study where bank clerks had to predict exchange rates. The analysis of qualitative judgments according to this model provided significantly more information than numerical predictions.A general framework for an interactive forecasting systems is suggested for further developments.
A perplexing yet persistent empirical finding is that individuals assess probabilities in words and in numbers nearly equivalently, and theorists have called for future research to search for factors that cause differences. This study uses an accounting context in which individuals are commonly motivated to reach preferred (rather than accurate) conclusions. Within this context, I predict new differences between verbal and numerical probability assessments, as follows: first, individuals will justify an optimistic verbal assessment (e.g., somewhat possible) by retaining the option of re-defining it, in case of negative outcomes, as though the phrase really means something different, and, for that matter, means more things. This re-definition will maintain some connection to the original meaning of the phrase, but de-emphasized relative to the new meaning. Second, based on this behavior, I also predict individuals' verbal probability assessments to be (1) more biased and yet (2) perceived as more justifiable than their numerical assessments. I find supportive evidence in an experiment designed to test the hypotheses. This study contributes to motivated reasoning and probability assessment theories (1) with new evidence of how individuals can word-smith in multiple attributes of a phrase to justify reaching a preferred conclusion, and (2) with new, reliable differences between verbal and numerical probability assessments. This study has important theoretical and practical implications relevant to organizational contexts in which people assess the likelihoods of uncertainties in words or numbers, and with motivations to reach a preferred conclusion.
For 20 different studies, Table 1 tabulates numerical averages of opinions on quantitative meanings of 52 qualitative probabilistic expressions. Populations with differing occupations, mainly students, physicians, other medical workers, and science writers, contributed. In spite of the variety of populations, format of question, instructions, and context, the variation of the averages for most of the expressions was modest, suggesting that they might be useful for codification. One exception was possible, because it had distinctly different meanings for different people. We report new data from a survey of science writers. The effect of modifiers such as very or negation (not, un-, im-, in-) can be described approximately by a simple rule. The modified expression has probability meaning half as far from the appropriate boundary (0 or 100) as that of the original expression. This paper also reviews studies that show stability of meanings over 20 years, mild effects of translation into other languages, context, small order effects, and effects of scale for reporting on extreme values. The stem probability with modifiers gives a substantial range 6% to 91% and the stem chance might do as well if tried with very. The stems frequent, probable, likely, and often with modifiers produce roughly equivalent sets of means, but do not cover as wide a range as probability. Extreme values such as always and certain fall at 98% and 95%, respectively, and impossible and never at 1%. The next step will be to offer codifications and see how satisfactory people find them.
Planning Peace: US Noncombat Operations and Major Wars; citation_author=Rapport
- War
Field Manual 5-0: The Operations Process
- Us Army
Field Manual 101-5: Staff Organization and Operations
- Us Army
An Experiment in Decision Analysis in Israel in 1975
- Lanir