Assessing Regulations: Red Team versus Blue Team, and Related Methods

Abstract

Decision making is improved by avoiding unaided expert judgment, and using structured judgmental procedures instead. The Red Team-Blue Team approach is one such structured approach. It recognizes that it is difficult for people-including scientists and public officials-to remain objective about the consequences of public policies and regulations. The solution that the approach provides is akin to the adversarial system that we use in our courts.

Full text

Assessing Regulations: Red Team versus Blue Team, and
Related Methods
By Scott Armstrong and Kesten Green
January 15, 2018
Decision making is improved by avoiding unaided expert judgment, and using structured
judgmental procedures instead. The Red Team-Blue Team approach is one such structured
approach. It recognizes that it is difficult for people—including scientists and public officials—to
remain objective about the consequences of public policies and regulations. The solution that the
approach provides is akin to the adversarial system that we use in our courts.
One of the teams must be genuinely skeptical of the benefits of government action and of any
studies offered in support of it. While such skepticism should be the natural attitude of public
officials and scientists, it cannot be taken for granted.
If one team is genuinely advocating for government intervention and the other is genuinely
skeptical, they are unlikely to reach an agreement. For that reason, a panel of independent judges
will likely be needed to weigh the relative merits of the scientific evidence presented by the Red and
Blue teams.
Again similarly to our court system, there should be a presumption that citizens are innocent of the
need to be disciplined by regulation. In other words, the pro-regulation team would need to prove
beyond a reasonable doubt that regulation would lead to a substantial net benefit relative to no
regulation over the long term. To maintain transparency and facilitate any challenges of their
findings, the judges should be required to publish their reasoning and recommendation of regulation
or deregulation.
In order to guarantee the integrity of the process, regulatory agencies should commit to
implementing the judges’ recommendations, so long as they meet pre-specified scientific criteria.
That commitment is critical: consider the harm caused by inaugural EPA Administrator
Ruckelshaus’s decision to ignore Judge Sweeney’s conclusion—after hearing seven months of
testimony—that DDT was not dangerous to people, and had substantial net benefits (Zubrin 2013).
We describe our suggestions for Red Team-Blue Team procedures under four headings. The first
suggestion is critical.
1. Accept only scientific studies as evidence. Not all studies are equal. We recommend
considering as evidence only findings from studies that comply with the criteria for a
scientific study, as described in the checklist available at guidelinesforscience.com (also
attached as Appendix A to this memo). We created that operational checklist for users to
determine whether a study meets the eight criteria for science using definitions of science
provided by pioneering scientists. Details and evidence are provided in Armstrong and
Green (2018).
2. Conduct meta-analyses. To decide whether a regulation is justified, consider all available
relevant scientific evidence—including uncertainty, existing knowledge gaps, and scientific
forecasts based on knowledge of the situation. For example, to be able to recommend
regulations in response to possible climate changes, evidence-based forecasts are needed on
(1) whether a persistent and substantive long-term global temperature trend will occur, (2)
whether such a trend would be harmful, and (3) whether regulations could be devised that
would reduce any forecasted harm in a cost-effective way.

1. A general call should be made to submit relevant scientific papers. To reduce the
burden imposed by the submission of unscientific papers, we suggest levying a fee
for submission expenses. The fee should be refunded for papers that meet the criteria
for being a relevant and useful scientific paper.
2. At least three raters should independently rate each paper using the Guidelines for
Science checklist (Appendix A). The process takes less than one hour per paper. Our
studies show that raters have found it easy to rate the papers because the criteria are
expressed operationally. For example, “Did the study test alternative reasonable
hypotheses?”
3. The Red and Blue teams would independently prepare meta-analyses of the relevant
papers that met the criteria for useful science and identify important gaps in
knowledge of the situation and on the effects of regulation/no regulation alternatives.
3. Use Multiple Anonymous Authentic Dissent (MAAD). For regulation problems in which
more conclusive evidence is needed than can be obtained from competing meta-analyses,
use the Multiple Anonymous Authentic Dissent approach to design and conduct new studies
comparing the scientific evidence on competing hypotheses. In MAAD, researchers act as
dissenters at key points in the study, beginning with the design. As the project develops,
each researcher is asked to independently record all potential defects in the research. An
administrator collates and edits the anonymous submissions and circulates the compiled list
to the team. Each researcher on the team then assumes that each of the objections has merit,
describes ways to deal with them, and sends their anonymous suggestions to the
administrator. The administrator then summarizes the suggestions for the team to make
anonymous written revisions.
1. Fund researchers with diverse viewpoints and backgrounds—in particular, different
stakeholders relevant to the regulations—to work in teams, with each team using the
method of multiple reasonable hypotheses. Funding for their research should be
contingent upon complying with scientific criteria.
2. To ensure independent thinking, the individual researchers in each group should not
communicate face-to-face. Instead, they should work in virtual groups, a process that
has additional benefits of reducing travel and meeting costs and providing a written
record of the reasons for the group’s decisions. The MAAD process should be
conducted via an administrator to help ensure that suggestions for improvements are
kept anonymous.
3. The findings from each team would be submitted to a “Science Court, which would
operate like a law court. Each team would present its case in writing to the judges.
The names of the judges should be included with their decision, along with full
disclosure of their reasoning and access to the relevant scientific studies.MAAD is
likely to be efficient because the teams’ efforts and time go towards improving the
research programs rather than travel, meetings, or defending their viewpoint. Related
research on group processes indicates that MAAD should be effective at providing
evidence for judging the merits of regulation alternatives.
4. Ask researchers to use evidence-based checklists: To determine whether a regulation
would be beneficial, researchers should complete a checklist to assess whether the regulation
meets the logical precursors for success. In order to do so, they will also need to complete
checklists for scientific forecasting to identify whether or not forecasts of the effects of
regulation and deregulation are valid.
1. We have developed a checklist for evaluating regulations based on logic and findings
from studies on the effects of regulation by leading scientists. The “Conditions
Necessary for Successful Regulation” checklist is available at the
ironlawofregulation.com website.
2. We suggest the use of evidence-based forecasting methods to predict whether a
regulation will achieve the desired effects while avoiding unintended consequences.

The checklists are described in “Forecasting methods and principles: Evidence-based
checklists.” Armstrong and Green (2018).
References
Armstrong, J.S. & Green, K.C. (2018a). Guidelines for Science: Evidence and Checklists. Working
paper available from ResearchGate.
Armstrong, J.S. & Green, K.C. (2018b). Forecasting methods and principles: Evidence-based
checklists. Working paper available from ResearchGate.
Zubrin, R. (2013). Merchants of Despair. New York: Encounter Books.

Appendix A.
!
Checklist of Criteria for
Compliance with The Scientific Method a,b !
Paper title:
Reviewer:
Date:
Time spent (minutes):
Instructions for Raters
You should skim the paper while you complete the checklist as a
skeptical reviewer.
1. Rate each lettered item, (a-d), below, with a checkbox (ý) as
True if the research complies,
na (not applicable), or
F/? (False/Unclear) if the research does not comply, or if you
are unsure.
IMPORTANT: If you are not convinced that the paper
complied, rate the item F/?.
2. If you rate an item True, give reasons for your rating in your own
words after the ✍ symbol.
(Items marked ¬ are necessary for science, but are not
individually sufficient.)
3. Rate criteria 1-8 as True with a checkbox (ý), only if all
necessary lettered items (¬) for the criterion are rated True.
First assess whether the paper complies with the lettered
items under each criterion, below. Then assess whether it
complies with each of the eight criterion based on
compliance with the lettered items. Do not speculate.
True
na
F/?
1. Problem is important for decision making, policy, or method
development
✍
o
a.
Importance of the problem clear from the titleo, abstracto, result
tableso, or conclusionso (Check each that applies)
✍
o!¬!o!
b.
The findings add to cumulative scientific knowledge
✍
o!¬!o!
c.
The findings can be used to improve people’s lives without resorting to
duress or deceit
✍
o!¬!o!
d.
Uses of the findings are important and clear to you
✍
o!¬!o!

!
2. Prior knowledge comprehensively reviewed and summarized
✍
o
a.
The paper describes objective and comprehensive procedures used to
search for prior useful scientific knowledge
✍
o!¬!o!
b.
The paper describes how prior substantive findings were used to
develop hypotheses (e.g. direction and magnitude of effects of each
variable) and research procedures
✍
o!¬!o!
3. Disclosure is sufficiently comprehensive for understanding
and replication
✍
o!
a.
Methods are fully and clearly described so as to be understood by
researchers, students, and managers, or are well-known to readers,
including potential users
✍
o!¬!o!
b.
Data are easily accessible using information provided in the paper
✍
o!¬!o!
c.
Sources of funding are described, or absence of external funding noted
✍
o!¬!o!
4. Design was objective (unbiased by advocacy)
✍
o!
a.
Prior hypotheses are clearly described (e.g., regarding directions and
magnitudes of relationships, and effects of conditions)
✍
o!¬!o!
b.
All reasonable hypotheses (including credible naïve, no-meaningful-
difference, and current-practice hypotheses) included in design
✍
o!¬!o!
c.
Revisions to hypotheses are described, or absence of revisions noted
✍
o!¬!o!
5. Data are valid (true measures) and reliable (repeatable
measures)
✍
o!
a.
Data were shown to be relevant to the problem, or was obvious
✍
o!¬!o!
b.
All relevant data were used, including longest relevant series for time-
series problems
✍
o!o!o!
c.#
Reliability of data was assessed, or was obvious
✍
o!¬!o!
d.#
Other information needed for assessing the validity of the data is
provided, such as known shortcomings and potential biases
✍
o!o!o!

!
6. Methods were valid (proven fit for purpose) and simple
✍
o!
a.
Methods were explained clearly and shown valid—unless well known to
intended readers, users, and reviewers, and validity is obvious
✍
o!¬!o!
b.
Methods were sufficiently simple for potential users to understand
✍
o!o!o!
c.
Multiple validated methods were used
✍
o!o!o!
d.
Methods used cumulative scientific knowledge explicitly
✍
o!o!o!
7. Experimental evidence was used to compare alternative
hypotheses
✍
o!
a.
Experimental evidence was used to compare hypotheses under explicit
conditions
✍
o!¬!o!
b.
Predictive validity of hypotheses was tested using out-of-sample data
✍
o!¬!o!
8. Conclusions follow logically from the evidence presented
✍
o!
a.
Conclusions do not go beyond the evidence in the paper
✍
o!¬!o!
b.
Conclusions are not the product of confirmation bias
✍
o!¬!o!
c.
Conclusions do not reject a hypothesis by denying the antecedent
✍
o!¬!o!
d.
Conclusions do not support a hypothesis by affirming the consequent
✍
o!¬!o!
Describe the most important scientific finding using your own words
✍!
Sum$the$criteria$(1–8)$rated$True$for$compliance:$[$$]$of$8.$
aA#version#of#this#checklist#is#available#at#GuidelinesforScience.com.##
bResearchers#should#rate#their#paper#against#this#checklist#before#submitting.
Version 82-book