
J. Scott ArmstrongUniversity of Pennsylvania | UP · The Wharton School
J. Scott Armstrong
PnD. MIT Management
About
570
Publications
525,299
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
42,192
Citations
Introduction
Additional affiliations
July 2011 - August 2011
March 1997 - April 1997
June 1968 - present
Education
September 1965 - May 1968
Publications
Publications (570)
The scientific method delivers prosperity, yet scientific practice has become subject to corrupting influences from within and without the scientific community. This essential reference is intended to help remedy those threats. The authors identify eight essential criteria for the practice of science and provide checklists to help avoid costly fail...
This chapter addresses how stakeholders of scientific research can assess whether research is compliant with the scientific method and then promote useful scientific research to improve products, services, processes, methods, and decision-making. We address stakeholders in sections for universities, scientific journals, governments, regulators and...
Objectivity underlies all of the criteria for complying with the scientific method. Only an objective description of an objective study and its findings can be considered scientific. Yet objectivity does not come naturally, and so scientists must struggle to overcome a tendency to advocate for subjectively preferred hypotheses when they practice sc...
The invitation for those nominating candidates for the Nobel Prize in economics, the “Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel,” described the award of the prize as being “based solely on scientific merit.” No criteria for judging scientific merit were provided, but nominators were directed to “consider origin and gend...
Gerd Gigerenzer, Director of the Max Planck Institute for Human Development, commented on reading papers in the Journal of Experimental Psychology from the 1920s and 1930s. He observed that “This was professionally a most depressing experience, but not because these articles were methodologically mediocre. On the contrary, many of them make today’s...
We believe that the primary role of a scientist is to make discoveries that can help to improve peoples’ lives, whether directly such as through the discovery of a vaccine against a disease or indirectly such as through the invention of a procedure that can improve efficiency or lead to better decisions. In this chapter, we provide practical advice...
Researchers who have been applying the scientific method to important problems for over 20 centuries are responsible for saving lives and improving our quality of life. Their efforts have provided us with the comforts and the myriad of opportunities that we have to live fulfilling lives that could barely be imagined in earlier times.
In this chapter, we raise making a useful contribution as a motive for university researchers, discuss some of the recent history of government involvement in science, and examine the evidence on whether or not government funding and regulation of research is beneficial.
List of works referred to in Armstrong & Green (2022) The Scientific Method
The scientific method is largely responsible for improving life expectancies and the quality of life over the past 2000 years. Individual scientists, in their efforts to discover how things work and how to make them better have used the method on their own or in collaboration with others to make the world a better place.
The scientific method delivers prosperity, yet scientific practice has become subject to corrupting influences from within and without the scientific community. This essential reference is intended to help remedy those threats. The authors identify eight essential criteria for the practice of science and provide checklists to help avoid costly fail...
The scientific method delivers prosperity, yet scientific practice has become subject to corrupting influences from within and without the scientific community. This essential reference is intended to help remedy those threats. The authors identify eight essential criteria for the practice of science and provide checklists to help avoid costly fail...
The scientific method delivers prosperity, yet scientific practice has become subject to corrupting influences from within and without the scientific community. This essential reference is intended to help remedy those threats. The authors identify eight essential criteria for the practice of science and provide checklists to help avoid costly fail...
The scientific method delivers prosperity, yet scientific practice has become subject to corrupting influences from within and without the scientific community. This essential reference is intended to help remedy those threats. The authors identify eight essential criteria for the practice of science and provide checklists to help avoid costly fail...
The following books and articles were discussed in the second edition of Long-Range Forecasting, published in 1985. In the few cases where unpublished sources are cited, instructions are provided on how to obtain a copy of the book or article. An attempt was also made to choose the most readable source when multiple sources were available. The list...
We asked 52 university students to make predictions about conflict situations. We gave each student four conflicts from a set of eight. We asked half our participants to assume there were 100 situations similar to a target conflict. We then asked them “…in how many of these situations would…” each one of a list of possible outcomes occur? The full...
Rules for choosing a single-decision forecast from a set of up to five analogies that have been rated for similarity to a target conflict
From “Bad Environmental and Resource Scares”, Chapter 18 of Julian L Simon’s 1996 “ The Ultimate Resource 2”, Princeton University Press, pp 258-273, and “Healing the Planet” afternote on p 274. Reproduced with the kind permission of the Simon family.
Electronic (Excel) version of the Checklist for Writing Management Reports
Proposed remedies for harmful regulation
The Global Warming Challenge, aka The Climate Bet (theclimatebet.com), is a record of an experiment to test the relative accuracy of Al Gore's 2007 dangerous man made global warming "tipping point" alarming projection relative to the no-change (no-trend) forecasts of Green, Armstrong, and Soon (2009) proposed by Scott Armstrong in his challenge to...
Flyer for Armstrong & Green presentation at ICCC Ten Washington, D.C., Including the Golden Rule of Forecasting Checklist and abstracts of papers with evidence on "Long-range climate and public policy forecasting"
Ten checklists for Armstrong & Green's "The Scientific Method" book in the form of pdf forms. For more information on "The Scientific Method: A Guide to Finding Useful Knowledge" (Cambridge, 2022), see thescientificmethod.info.
"Do we face dangerous global warming?"
Scott Armstrong gave a talk at Lehigh University on June 7, 2019 about research with Kesten Green on scientific climate forecasting. The talk was invited by the graduating Class of 1959, of which he is a member, for their 60th Reunion.
Purpose: Commentary on M4-Competition and findings to assess the contribution of data models--such as from machine learning methods--to improving forecast accuracy.
Methods: (1) Use prior knowledge on the relative accuracy of forecasts from validated forecasting methods to assess the M4 findings. (2) Use prior knowledge on forecasting principles...
Purpose: Commentary on M4-Competition and findings to assess the contribution of data models—such as from machine learning methods—to improving forecast accuracy. Methods: (1) Use prior knowledge on the relative accuracy of forecasts from validated forecasting methods to assess the M4 findings. (2) Use prior knowledge on forecasting principles and...
In the mid-1900s, there were two streams of thought about forecasting methods. One stream-led by econometricians-was concerned with developing causal models by using prior knowledge and evidence from experiments. The other was led by statisticians, who were concerned with identifying idealized "data generating processes" and with developing models...
Problem
Do conservative econometric models that comply with the Golden Rule of Forecasting provide more accurate forecasts?
Methods
To test the effects of forecast accuracy, we applied three evidence-based guidelines to 19 published regression models used for forecasting 154 elections in Australia, Canada, Italy, Japan, Netherlands, Portugal, Spai...
Relative absolute error (RAE) of forecasts from damping compared to forecasts from the original regression models.
(DOCX)
Relative absolute error (RAE) of forecasts from equalizing compared to forecasts from the original regression models.
(DOCX)
Commentary of the findings of the M4-Competition presented by Scott Armstrong at the M4 Conference on Monday, December 10, 2018 in New York City.
Problem: Multiple regression analysis (MRA) is commonly used to develop forecasting models that inform policy and decision making, but the technique does not appear to have been validated for that purpose.
Methods: The predictive validity of published least squares MRA models is tested against naive benchmarks, alternative methods that are either p...
We examined two papers with long-range forecasts of global mean temperatures (IPCC 2007, and Green, Armstrong, & Soon 2009) that included forecasts of global mean temperatures, and one paper that used the IPCC projections to forecast a dramatic decline in the population of polar bears for compliance with the scientific method. Ratings for complianc...
Problem: The scientific method is unrivaled for generating useful knowledge, yet papers published in scientific journals frequently violate the scientific method.
Methods: A definition of the scientific method was developed from the writings of pioneers of the scientific method including Aristotle, Newton, and Franklin. The definition was used as...
Problem
How to help practitioners, academics, and decision makers use experimental research findings to substantially reduce forecast errors for all types of forecasting problems.
Methods
Findings from our review of forecasting experiments were used to identify methods and principles that lead to accurate forecasts. Cited authors were contacted to...
Decision making is improved by avoiding unaided expert judgment, and using structured judgmental procedures instead. The Red Team-Blue Team approach is one such structured approach. It recognizes that it is difficult for people-including scientists and public officials-to remain objective about the consequences of public policies and regulations. T...
Super Bowl advertisers ignore many persuasion principles, to their cost.
Problem: How to help practitioners, academics, and decision makers use experimental research findings to substantially reduce forecast errors for all types of forecasting problems.
Methods: Findings from our review of forecasting experiments were used to identify methods and principles that lead to accurate forecasts. Cited authors were contacted...
The PollyVote uses evidence-based techniques for forecasting the popular vote in presidential elections. The forecasts are derived by averaging existing forecasts generated by six different forecasting methods. In 2016, the PollyVote correctly predicted that Hillary Clinton would win the popular vote. The 1.9 percentage-point error across the last...
Earlier versions of this Tree appear in various publications and presentations. For more information on the methods and how to implement them, see "Forecasting Methods and Principles: Evidence-based Checklists."
A Recap of the 2016 Election Forecasts - Volume 50 Issue 2 - James E. Campbell, Helmut Norpoth, Alan I. Abramowitz, Michael S. Lewis-Beck, Charles Tien, James E. Campbell, Robert S. Erikson, Christopher Wlezien, Brad Lockerbie, Thomas M. Holbrook, Bruno Jerôme, Véronique Jerôme-Speziari, Andreas Graefe, J. Scott Armstrong, Randall J. Jones, Alfred...
See also Green (2018) "The harm of regulation: Evidence and solutions" a presentation at the 6th Annual Friedman Conference. DOI: 10.13140/RG.2.2.29921.02407
The PollyVote Forecast for the 2016 American Presidential Election - Volume 49 Issue 4 - Andreas Graefe, Randall J. Jones, J. Scott Armstrong, Alfred G. Cuzán
Checklist for multiple raters to rate a proposal for an advertisement against persuasion principles.
Purpose: To respond to issues posed in the four commentaries on Armstrong, Du, Green and Graefe (this issue) regarding the immediate usefulness of that paper’s test of advertisements’ compliance with persuasion principles, and regarding the need for further research.
Approach: Address commentators’ concerns using logic, prior research findings, an...
Purpose
– This paper aims to test whether a structured application of persuasion principles might help improve advertising decisions. Evidence-based principles are currently used to improve decisions in other complex situations, such as those faced in engineering and medicine.
Design/methodology/approach
– Scores were calculated from the ratings o...
Excel spreadsheet checklists for comparing written and oral reports to evidence-based persuasion principles supported by Armstrong's Persuasive Advertising book, or logic.
Purpose: To test whether a structured application of persuasion principles might help improve advertising decisions. Evidence-based principles are currently used to improve decisions in other complex situations, such as those faced in engineering and medicine. Approach: Scores were calculated from the ratings of 17 self-trained novices who rated 96...
This article proposes a unifying theory, or the Golden Rule, of forecasting. The Golden Rule of Forecasting is to be conservative. A conservative forecast is consistent with cumulative knowledge about the present and the past. To be conservative, forecasters must seek out and use all knowledge relevant to the problem, including knowledge of methods...
The Golden Rule of Forecasting is a general rule that applies to all forecasting problems. The Rule was developed using logic and was tested against evidence from previously published comparison studies. The evidence suggests that a single violation of the Golden Rule is likely to increase forecast error by 44%. Some commentators argue that the Rul...
The Golden Rule of Forecasting is a general rule that applies to all forecasting problems. The Rule was developed using logic and was tested against evidence from previously published comparison studies. The evidence suggests that a single violation of the Golden Rule is likely to increase forecast error by 44%. Some commentators argue that the Rul...
This article proposes a unifying theory, or Golden Rule, of forecasting. The Golden Rule of Forecasting is to be conservative. A conservative forecast is consistent with cumulative knowledge about the present and the past. To be conservative, forecasters must seek out and use all knowledge relevant to the problem, including knowledge of methods val...
A checklist of requirements for following The Golden Rule of Forecasting. From Armstrong, Green, and Graefe (2015).
The Golden Rule of Forecasting requires forecasters to be conservative by forecasting in a way that is consistent with cumulative knowledge about the situation and about forecasting. Violating the average guideline increases forecast errors by more than 40%. We examined whether the United Nations Intergovernmental Panel on Climate Change’s (IPCC’s)...
With the objective of improving the accuracy of election forecasts, we examined three evidence-based forecasting guidelines that are relevant to forecasting with causal models. The guidelines suggest: (1) modifying estimates of the strength of variable effects to account for uncertainty, (2) combining forecasts from diverse models, and (3) utilizin...
The United Nations Intergovernmental Panel on Climate Change’s (IPCC’s) forecast of dangerous manmade global warming is an important influence on public policy decisions in many countries. Rational policies require that policy makers obtain forecasts from evidence-based methods. A review of comparative studies found that sophisticatedly simple meth...
The Golden Rule of Forecasting Checklist was used to evaluate IPCC “business as usual” global warming scenario and the Green, Armstrong & Soon no-change model forecasts.
Consensus ratings by Armstrong and Green indicated that of 20 relevant Golden Rule Checklist guidelines:
• the IPCC scenarios followed none
• the no-change model followed 95%
This article introduces this JBR Special Issue on simple versus complex methods in forecasting. Simplicity in forecasting requires that (1) method, (2) representation of cumulative knowledge, (3) relationships in models, and (4) relationships among models, forecasts, and decisions are all sufficiently uncomplicated as to be easily understood by dec...
This article examines whether decomposing time series data into two parts - level and change - produces forecasts that are more accurate than those from forecasting the aggregate directly. Prior research found that, in general, decomposition reduced forecasting errors by 35%. An earlier study on decomposition into level and change found a forecast...
Spreadsheet checklist of persuasion principles for creating advertisements - The Persuasion Principles Audit software.
Supporting document: Persuasion Principles Audit Report Sample Format.
Supporting document: Persuasion Principles Checklist Instructions
This article introduces the Special Issue on simple versus complex methods in forecasting. Simplicity in forecasting requires that (1) method, (2) representation of cumulative knowledge, (3) relationships in models, and (4) relationships among models,forecasts, and decisions are all sufficiently uncomplicated as to be easily understood by decision-...
Purpose: To test whether a structured application of persuasion principles might help improve advertising decisions. Evidence-based principles are currently used to improve decisions in other complex situations, such as those faced in engineering and medicine.Approach: Scores were calculated from the ratings of 17 self-trained novices who rated 96...
Purpose: To respond to issues posed in the four commentaries on Armstrong, Du, Green and Graefe (this issue) regarding the immediate usefulness of that paper’s test of advertisements’ compliance with persuasion principles, and regarding the need for further research. Approach: Address commentators’ concerns using logic, prior research findings, and...
Checklist for developing and using a time line for planning
The interactive training module for the Advertising Principles Audit
Persuasive Advertising (Armstrong, 2010) focuses on how to develop persuasive ads, but not on how to deliver those ads to customers. Here are five methods that can be used to decide a media budget as well as how to allocate that spending across various media. I start with the weakest method and work towards the strongest. Ideally, one should use al...
The evaluation of advertising agencies and proposals rests heavily of judgments. Fortunately, much is known about how to improve judgments.
Earlier versions of this Tree appear in various publications and presentations. Note that the presence of a method in the Tree does not signal its predictive validity. For evidence on which methods are valid, see "Forecasting Methods and Principles: Evidence-based Checklists."
The Golden Rule of Forecasting counsels forecasters to be conservative when making forecasts. We tested the value of three of the four Golden Rule guidelines that apply to causal models: modify effect estimates to reflect uncerainty; use all important variables; and combine forecasts from diverse models. These guidelines were tested using out-of-sa...
We review the performance of the PollyVote, which combined forecasts from polls, prediction markets, experts’ judgment, political economy models, and index models to forecast the two-party popular vote in the 2012 U.S. Presidential Election. Throughout the election year the PollyVote provided highly accurate forecasts, outperforming each of its com...
We summarize the literature on the effectiveness of combining forecasts by assessing the conditions under which combining is most valuable. Using data on the six US presidential elections from 1992 to 2012, we report the reductions in error obtained by averaging forecasts within and across four election forecasting methods: poll projections, expert...
A letter to Mr Wanamaker about the findings of Armstrong, Du, Green, & Graefe (2015).
The Big-Issue Model predicts election outcomes based on voters’ perceptions of candidates’ ability to handle the most important issue. The model provided accurate forecasts of the 2012 U.S. presidential election. The results demonstrate the usefulness of the model in situations where one issue clearly dominates the campaign, such as the state of th...
The Golden Rule of Forecasting requires that forecasters be conservative by making proper use of cumulative knowledge and by not going beyond that knowledge. The procedures that have been used to forecast dangerous manmade global warming violate the Golden Rule. Following the scientific method, we investigated competing hypotheses on climate change...
Our review of the evidence found that the notion that higher pay leads to the selection of better executives is undermined by the prevalence of poor recruiting methods. Moreover, higher pay fails to promote better performance. Instead, it undermines the intrinsic motivation of executives, inhibits their learning, leads them to ignore other stakehol...
Questions
Question (1)
In our current paper, the "Golden Rule of Forecasting" we are searching for evidence on whether, over the past half-century, the practice of forecasting in any area of the social or management sciences has been shown to produce more accurate forecasts or less accurate forecasts or if there has been no measurable change. Please provide citations to relevant sources.