ENVIRONMENTAL REGULATION GOING RETRO:
LEARNING FORESIGHT FROM HINDSIGHT
JONATHAN B. WIENER* AND DANIEL L. RIBEIRO**
I. INTRODUCTION ...................................................................... 1
II. ENVIRONMENTAL POLICY FORESIGHT ................................. 10
A. CBA of Infrastructure Projects ...................................... 10
B. EIA ............................................................................ 13
C. RIA ............................................................................ 16
D. Risk Assessment ............................................................ 19
III. FROM FORESIGHT TO HINDSIGHT: THE RISE OF
RETROSPECTIVE REVIEW ..................................................... 21
A. Ex Post RIA in the U.S. ................................................. 21
B. Ex Post RIA Beyond the U.S. ........................................ 27
1. United Kingdom ...................................................... 28
2. European Union ...................................................... 30
3. Australia .................................................................. 32
C. Ex Post EIA, CBA, and RA ............................................ 33
IV. THE STATE OF PLAY OF RETROSPECTIVE REVIEW IN U.S.
ENVIRONMENTAL LAW ......................................................... 36
A. Prior Assessments of the Practice of Ex Post RIA
in the U.S. ...................................................................... 38
B. OMB/OIRA Reports to Congress .................................. 44
C. State Experience ............................................................ 50
D. EPA “Retrospective Reviews” under EO 13,563 ............ 51
V. GOING RETRO: ADVANCING REGULATORY HINDSIGHT ........ 60
VI. CONCLUSION ........................................................................ 71
Environmental law demands foresight. Much environmental
law seeks to prevent dangers that “may reasonably be anticipated,”1
invoking precaution against future risks before they occur.2 Even
* Perkins Professor of Law, Public Policy and Environmental Policy, Duke
University; University Fellow, Resources for the Future (RFF).
** SJD Candidate, Duke University School of Law. The authors thank Andrea Renda
for comments on an earlier draft, Joe Aldy and John Graham for discussions during their
visits to Duke, and the Bass Connections project on “Reviewing Retrospective Regulatory
Review” at Duke University, 2015-16 (including Ed Balleisen, Lori Bennear, Elizabeth Brake,
Josh Bruce, Mercy DeMenno, Andrea Renda, and our team of graduate and undergraduate
students), for extensive discussion and support.
1. E.g., Clean Air Act, 42 U.S.C. § 108 (2004).
2. E.g., Ethyl Corp. v. US, 541 F.2d 377, 377 (D.C. Cir. 1976). For a recent review of
precaution in environmental law, see Jonathan B. Wiener, Precaution and Climate Change,
2 JOURNAL OF LAND USE [Vol. 32:1
environmental laws that seek to remedy past damage and restore
ecological systems still depend on foreseeing the future effects of
such measures. Environmental Impact Assessment (EIA)—the
flagship of modern environmental law, now adopted around the
world—calls for foresight before taking action.3 Similarly,
Regulatory Impact Assessment (RIA)—required by every United
States (U.S.) President of the past four decades, and increasingly
adopted in other countries—has emphasized prospective ex ante
assessment of the future impacts of proposed new rules or rule
revisions.4 Each of these impact assessment (IA) tools incorporates,
to some degree, the analytic methods of risk assessment (RA) and
cost-benefit analysis (CBA).
Yet foresight is inevitably imperfect. Humans may be unusual
among species in trying to make decisions via foresight, by
envisioning hypothetical scenarios of future consequences (and how
they will feel about them),5 but humans also tend to be flawed
forecasters.6 Choosing among options is challenging, because
anticipating the consequences of alternative actions involves
foreseeing future outcomes with and without each option and
furthermore foreseeing future preferences about these outcomes.7
Even when making decisions with the best intentions, humans are
susceptible to biases and heuristics. The future scenarios that the
human brain constructs tend to be made of collages of memories,
which helps explain why humans tend to overemphasize events that
they recall as more salient (the availability heuristic).8 Humans
may overstate the importance of their current state of affairs as a
reference point (status quo bias); they may find it difficult to
in THE OXFORD HANDBOOK OF INT’L CLIMATE CHANGE LAW (Cinnamon Carlarne et al., eds.,
Oxford Univ. Press, 2016).
3. The U.S. National Environmental Policy Act (NEPA), 42 U.S.C. § 4321 et seq.,
requires federal agencies to stop and think ahead about the reasonably foreseeable significant
environmental impacts of their major actions. On the international adoption of EIA, see NEIL
CRAIK, INTERNATIONAL LAW OF ENVIRONMENTAL IMPACT ASSESSMENT (Cambridge Univ.
Press, 2008). On EIA as policy foresight and its international diffusion, see Jonathan B.
Wiener & Daniel L. Ribeiro, Impact Assessment: Diffusion and Integration, in COMPARATIVE
LAW AND REGULATION: UNDERSTANDING THE GLOBAL REGULATORY PROCESS (Francesca
Bignami & David Zaring eds., 2016).
4. See Wiener & Ribeiro, supra note 3.
5. See DANIEL GILBERT, STUMBLING ON HAPPINESS 81-106 (Vintage Canada ed. 2009).
6. Id.; LEONARD MLODINOW, THE DRUNKARD’S WALK: HOW RANDOMNESS RULES OUR
LIVES (Vintage Books 2008); PHILIP E. TETLOCK & DAN GARDNER, SUPERFORECASTING: THE
ART AND SCIENCE OF PREDICTION (Crown 2015); DANIEL KAHNEMAN, THINKING, FAST AND
SLOW (1st ed. 2011); NASSIM NICHOLAS TALEB, FOOLED BY RANDOMNESS: THE HIDDEN ROLE
OF CHANCE IN LIFE AND IN THE MARKETS (Random House 2005).
7. GILBERT, supra note 5.
8. Daniel T. Gilbert & Timothy D. Wilson, Prospection: Experiencing the Future, 317
SCIENCE 1351 (2007); D. L. Schacter, D. R. Addis & R. L. Buckner, Episodic Simulation of
Future Events: Concepts, Data, and Applications, 1124 ANN. N. Y. ACAD. SCIENCE 39 (2008).
Fall. 2016] REGULATION GOING RETRO 3
appreciate randomness, expecting to see a cause behind every event;
they may attribute patterns when there is only noise; and they may
overstate the skills or errors of the decision maker.9
Benjamin Franklin understood both the need for foresight and
its inescapable imperfection when he advised his friend, the British
scientist Joseph Priestley, who was considering whether to accept a
job offer made by Lord Shelburne to work as the librarian and tutor
of Shelburne’s children.10 Franklin proposed a process of
envisioning and weighing “all the Reasons pro and con” for each
decision option, recognizing that:
tho’ the Weight of Reasons cannot be taken with the
Precision of Algebraic Quantities, yet, when each is thus
considered … and the whole lies before me, I think I can
judge better, and am less liable to make a rash Step; and in
fact I have found great Advantage from this kind of
Equation, in what may be called Moral or Prudential
Taking Franklin’s advice, Priestley considered his objectives and
collected information on Lord Shelburne and his offer.12 He sought
to foresee and weigh the possible consequences of his alternatives
and make his decision.13 Yet, as Franklin noted, even such foresight
is inevitably imprecise.
Foresight can be improved, notably through astute hindsight:
learning from the past.14 The key is to reassess past foresight in
light of experience and thereby increase the accuracy of our
foresight methods.15 Informing foresight from hindsight is an
9. MLODINOW, supra note 6, at 9 (mentioning examples of how adverse outcomes can
be misperceived as indicative of bad decisions or bad decision skills).
10. JOHN TOWILL RUTT, LIFE AND CORRESPONDENCE OF JOSEPH PRIESTLEY: VOLUME I
11. Benjamin Franklin, Letter to Joseph Priestley (September 19, 1772), in BENJAMIN
FRANKLIN: REPRESENTATIVE SELECTIONS, WITH INTRODUCTION, BIBLIOGRAPHY AND NOTES
348-49 (Frank Luther Mott & Chester E. Jorgenson, eds., New York: American Book
Company). The context and influence of Franklin’s letter on the development of CBA is
discussed in Jonathan B. Wiener, The Diffusion of Regulatory Oversight, in THE
GLOBALIZATION OF COST-BENEFIT ANALYSIS IN ENVIRONMENTAL POLICY (Richard L. Revesz
& Michael A. Livermore eds., 2013); RUTT, supra note 10, at 182-183. The discussion of
Priestley’s decision making process before and after Franklin’s advice is also mentioned in
CHIP HEATH & DAN HEATH, DECISIVE: HOW TO MAKE BETTER CHOICES IN LIFE AND WORK
12. See RUTT, supra note 10, at 178, 181, 183, 185, 188.
14. TETLOCK & GARDNER, supra note 6, at 13 (“Forecast, measure, revise. Repeat. It’s a
never ending process of incremental improvement that explains why weather forecasts are
good and slowly getting better. . . . [W]ithout revision, there can be no improvement”).
15. See generally id.
4 JOURNAL OF LAND USE [Vol. 32:1
essential inferential method of science. From hypothesis testing
through experimentation and observation, to the Bayes-Laplace
Theory of updating prior beliefs, the essence of scientific inquiry is
that additional information can enable us to test past assumptions
and predictions and improve our ability to foresee.16
In this sense, environmental law needs to learn17 to improve its
foresight via hindsight—it needs to couple prospection with
retrospection. The point of such retrospection is not to return to a
past state of the world; it is not a reactionary nostalgia, but rather
a reflective (at times bittersweet) process of learning.18 Measuring
past forecasts against policy performance can promote learning and
improvement in subsequent decisions. Such a forecast-revise-adapt
approach is a central feature of the new wave of developments in
artificial intelligence and deep learning.19 It can be part of our legal
institutions as well.
IA, developed in the U.S. and diffused throughout the world,
has become the institutional and legal mechanism for policy
foresight.20 As noted, EIA and RIA have both been adopted widely
as prospective ex ante procedures for policy foresight, seeking to
foster environmental quality and better regulation.21
The emphasis of both RIA and EIA over the past five decades
has been prospective: estimating the future consequences of a policy
decision.22 Researchers have observed that these ex ante forecasts
may, understandably, exhibit significant uncertainties and
inaccuracies.23 Several studies have found that only a plurality of ex
16. SHARON BERTSCH MCGRAYNE, THE THEORY THAT WOULD NOT DIE: HOW BAYES’
RULE CRACKED THE ENIGMA CODE, HUNTED DOWN RUSSIAN SUBMARINES, AND EMERGED
TRIUMPHANT FROM TWO CENTURIES OF CONTROVERSY (Reprint ed. 2012); MLODINOW, supra
17. For an early call to incorporate learning into environmental law, including through
experimentation and review, see Daniel A. Farber, Environmental Protection as a Learning
Experience, 27 Loy. L.A. L. Rev. 791 (1994).
18. SVETLANA BOYM, THE FUTURE OF NOSTALGIA (Basic Books 2008).
19. Nicola Lettieri, Computational Social Science, the Evolution of Policy Design and
Rule Making in Smart Societies, 8 FUTURE INTERNET 19 (2016); Rise of the Machines,
ECONOMIST (May 2015), http://www.economist.com/news/briefing/21650526-artificial-
20. Wiener & Ribeiro, supra note 3.
21. Wiener, supra note 11. See Wiener & Ribeiro, supra note 3.
22. See Jos Arts, Paula Caldwell & Angus Morrison-Saunders, Environmental Impact
Assessment Follow-up: Good Practice and Future Directions—Findings from a Workshop at
the IAIA 2000 Conference, 19 IMPACT ASSESS. PROJ. APPRAIS. 175, 175-85 (2001); JOSEPH
ALDY, LEARNING FROM EXPERIENCE: AN ASSESSMENT OF THE RETROSPECTIVE REVIEWS OF
AGENCY RULES AND THE EVIDENCE FOR IMPROVING THE DESIGN AND IMPLEMENTATION OF
REGULATORY POLICY 7 (2014); EUR. COMM'N, Smart Regulation in the European Union, COM
(2010) 543 final, at 3 (Oct. 8, 2010) [hereinafter EC, Smart Regulation].
23. See Adam Finkel, The Cost of Nothing Trumps the Value of Everything: The Failure
of Regulatory Economics to Keep Pace with Improvements in Quantitative Risk Analysis, 4
MICH. J. ENVTL. & ADMIN. L. 91 (2014).
Fall. 2016] REGULATION GOING RETRO 5
ante IAs turn out to be accurate (even defined loosely as +/- 25%),
with errors of both overestimation and underestimation of actual
impacts, for reasons including: industry overestimation of costs,
assumptions of static technology followed by actual innovation, and
mis-projection of compliance rates.24 In some cases, the ex ante IA
may appear inaccurate because the policy was changed after the ex
ante IA was prepared on a prior version of the policy.25 Yet there
have still been “only . . . a handful” of retrospective studies of
prospective accuracy,26 and they have examined only partial
samples which may not be representative of the broader universe of
policies and IAs.27
Governments have increasingly called for regular conduct of
retrospective review or ex post IA, chiefly to secure cost savings or
other gains from revising older regulations.28 Retrospective review
of existing regulations was the objective of section 5 of President Bill
Clinton’s Executive Order (EO) 12,866 (1993)29; section 6 of
President Barack Obama’s EO 13,563 (2011)30; President Obama’s
EO 13,579 (2011) calling on independent agencies to conduct similar
reviews31; President Obama’s EO 13,610 (2012) giving further
details on the review process32; and the Organization for Economic
Co-operation and Development’s (OECD) recommendation number
5 on regulatory policy and governance (2012).33 The Regulatory
24. See Winston Harrington, Richard D. Morgenstern & Peter Nelson, On the Accuracy
of Regulatory Cost Estimates, 19 J. POL’Y ANALYSIS & MGMT. 297 (2000); OFFICE OF MGMT. &
BUDGET, 2005 REPORT TO CONGRESS ON THE COSTS AND BENEFITS OF FEDERAL REGULATIONS
AND UNFUNDED MANDATES ON STATE, LOCAL, AND TRIBAL ENTITIES (2005) [hereinafter OMB
2005 REPORT]; Winston Harrington, Grading Estimates of the Benefits and Costs of Federal
Regulation: A Review of Reviews, RESOURCES FOR THE FUTURE (2006); Richard D.
Morgenstern, The RFF Regulatory Performance Initiative: What Have We Learned?,
RESOURCES FOR THE FUTURE (2015) [hereinafter Morgenstern, RFF].
25. Such changes could occur during the legislative/rulemaking process after the ex
ante IA is prepared, or during implementation after adoption of the policy. One of the main
criticisms of ex ante IA expressed by some officials from Directorates-General of the European
Commission is that the proposed policy action examined in the ex ante IA gets significantly
amended after the proposal leaves the Commission and traverses the European Parliament
and Council—without an update to the IA to assess the impacts of the final policy action.
Interview with Two Officials from Directorates-General of the European Commission (2015),
on file with authors.
26. Finkel, supra note 23, at 118.
27. See id.
28. See infra Section III.
29. Exec. Order No. 12,866, 58 Fed. Reg. 51,735 (Sept. 30, 1993). Previously, President
Jimmy Carter’s Exec. Order 12,044 (1978) addressed review of existing regulations in §§
2(d)(8) and 4; and President Ronald Reagan’s Exec. Order 12,291 (1981) addressed review of
existing regulations in § 3(i).
30. Exec. Order No. 13,563, 76 Fed. Reg. 3821 (Jan. 18, 2011).
31. Exec. Order No. 13,579, 76 Fed. Reg. 41,587 (July 11, 2015).
32. Exec. Order No. 13,610, 77 Fed. Reg. 28,469 (May 10. 2012).
33. ORG. FOR ECON. CO-OPERATION & DEV., RECOMMENDATION OF THE COUNCIL ON
REGULATORY POLICY AND GOVERNANCE (2012).
6 JOURNAL OF LAND USE [Vol. 32:1
Flexibility Act (RFA) requires agencies to review within 10 years of
issuance those regulations that have “a significant economic impact
upon a substantial number of small entities.”34 Some statutes
require reviews every few years.35 The Administrative Conference
of the United States (ACUS) endorsed the call for retrospective
review as early as 1995 (just after the Clinton EO),36 commissioned
an expert appraisal in 2014 by Joseph Aldy of retrospective review
efforts to date (soon after the Obama EO),37 and adopted a set of
recommendations in late 2014 for strengthening retrospective
review.38 Countries around the world have been adopting versions
of retrospective review (whether called ex post IA, follow up policy
evaluation, post-implementation review, retrospective review, or
Yet, these government measures to require retrospective review
have not yet fulfilled the goal that we emphasize here: using
retrospective review to learn to improve prospective review—using
hindsight to improve foresight. Calls for retrospective review have
yielded only partial and slow progress in practice. After his term at
the helm of Office of Information and Regulatory Affairs (OIRA),
where he was a key architect of the Obama administration’s
retrospective review orders and supervised their implementation,
Cass Sunstein wrote that “[i]t is an astonishing fact that until very
recently, there has been no sustained effort to gather, let alone act
on, that information [about what regulatory policies actually do]—
and that existing efforts remain highly preliminary and partial.”40
The Aldy report found that the Obama administration’s measures
generated retrospective reviews of several hundred specific rules,
and helped build a culture of retrospective review; however, the
track record remained “mixed” and very few of the administration’s
34. 5 U.S.C. § 602(a)(1) (2012).
35. E.g., 42 U.S.C. § 7409(d) (requiring reviews of national ambient air quality
standards [NAAQS] every five years).
36. Admin. Conference of the U.S., Recommendation 95-3, Review of Existing Agency
Regulations, 60 Fed. Reg. 43,108, 43,109 (Aug. 18, 1995).
37. ALDY, supra note 22.
38. Admin. Conference of the U.S., Recommendation 2014-5, Retrospective Review of
Agency Rules, adopted December 4, 2014, at 79 Fed. Reg. 75,114, 75,114-117 (Dec. 17, 2014).
ACUS Recommendation 5(c) notes that one factor in selecting rules for retrospective analysis
is “[u]ncertainty about the accuracy of initial estimates of regulatory costs and benefits.” Id.
at 75,116. Retrospective review was also advocated by the American Bar Association. SECTION
OF ADMIN. LAW AND REGULATORY PRACTICE, AM. BAR ASSOC., IMPROVING THE
ADMINISTRATIVE PROCESS: A REPORT TO THE PRESIDENT-ELECT 12-13 (2016),
39. ORG. FOR ECON. CO-OPERATION & DEV., REGULATORY POLICY OUTLOOK § 5 (2015)
[hereinafter OECD, POLICY OUTLOOK].
40. Cass R. Sunstein, The Regulatory Look-Back, 94 B.U. L. REV. 579, 588 (2014).
Fall. 2016] REGULATION GOING RETRO 7
newly issued rules were revisions based on a retrospective review or
required a future retrospective review.41 Cary Coglianese observed
that “retrospective review is today where prospective analysis was
in the 1970s: ad hoc and largely unmanaged.”42 OECD remarked:
“ex post evaluation by [U.S.] federal agencies remains patchy and
It is understandable that agencies told to conduct retrospective
reviews may see this task as low priority compared to issuing the
new policies demanded by Congress, the President, and the public;
an agency may hesitate to conduct reviews that might cast doubt on
its own past analyses, or subject its policies to revision or rescission.
Hence, there is a need for presidential exhortation (or another
institutional mechanism) to promote retrospective review. The
Obama Administration continued to seek and report additional
retrospective reviews by agencies each year.44
So far, government retrospective review has mainly been aimed
at assessing each regulatory policy individually, with a view to
revising that specific policy, often to reduce its cost burden.45 In this
article, we argue that the retrospective review effort should be
broader, assessing the comprehensive scope of important impacts
(not only costs, but also benefits and ancillary impacts, with a view
not just to reducing burdens, but also to increasing net benefits).46
41. ALDY, supra note 22, at 4-6. Similarly, Sofie Miller studied twenty-two rules
promulgated in 2014 and found that very few included plans for future retrospective review.
Sofie E. Miller, Learning from Experience: Retrospective Review of Regulations in 2014 (Geo.
Wash. U. Regulatory Studies Ctr., Working Paper, 2015), https://regulatorystudies.
42. Cary Coglianese, Moving Forward with Regulatory Lookback, 30 YALE J. ON REG.
57, 59 (2013).
43. OECD, POLICY OUTLOOK, supra note 39, at 123; see also Randall Lutter, Regulatory
Policy: What Role For Retrospective Analysis and Review?, 4 J. BENEFIT-COST ANALYSIS, 17-
38 (2013) (similar).
44. See, e.g., Howard Shelanski, Making All Levels of Government More Efficient and
Effective Through Retrospective Review, THE WHITE HOUSE (March 4, 2016),
and-effective-through-retrospective (reporting on “more than 50 new retrospective initiatives”
and stating that the administration’s “regulatory lookback effort to date [since 2011] has
achieved an estimated $28 billion in net 5-year savings”). Howard Shelanski was the
Administrator of OIRA during President Obama’s second term.
45. See infra Section II.
46. Exec. Order No. 13,563, 76 Fed. Reg. 3821 (Jan. 18, 2011), recognizes the possibility
that retrospective review will find that a rule is “insufficient” as well as that it is “outmoded,
ineffective . . . or excessively burdensome” (section 6), but the emphasis so far has been on
reducing costs; see Shelanski, supra note 44 (noting large cost savings, but also an example
of expanding federal policy on hearing aids). A useful analogy may be to outcomes studies in
medical care, the objective of which is not necessarily to reduce (or increase) medication, but
to improve patient health outcomes; similarly, retrospective review should be aimed
evenhandedly not at reducing (or increasing) regulation, but at improving societal outcomes.
See Jonathan B. Wiener, Managing the Iatrogenic Risks of Risk Management, 9 RISK: HEALTH
8 JOURNAL OF LAND USE [Vol. 32:1
Furthermore, we argue that retrospective review should emphasize
learning—by assessing larger and representative samples of
multiple ex post IAs compared to ex ante IAs, in order to improve
foresight through more accurate ex ante IA methodologies and to
learn about better policy designs.47 Under EO 13,563, “each agency
is directed to use the best available techniques to quantify
anticipated present and future benefits and costs as accurately as
possible”48—which should include using retrospective review to test
and improve the accuracy of prospective IA. Cary Coglianese
recommends “rigorous retrospective review [of multiple rules
sharing common estimation issues] to evaluate their benefits and
costs retrospectively [and] help validate or improve prospective
estimation techniques applicable to other rules.”49 Aldy likewise
notes the value of using retrospective review (ex post IA) to test and
improve the accuracy of methodologies for prospective ex ante IA.50
President Obama declared in his 2009 inaugural address that
“[t]he question we ask today is not whether our government is too
big or too small, but whether it works.”51 Regulations can protect
SAFETY & ENV’T 39, 78-79 (1998) (proposing national outcomes studies of regulation, akin to
outcomes studies in medicine).
47. OIRA appears to agree with this goal of using retrospective IA to enhance the
accuracy of prospective IA:
Prospective analysis may overestimate or underestimate both benefits and costs;
retrospective analysis can be important as a corrective mechanism. Executive
Orders 13563 and 13610 specifically call for such analysis, with the goal of
improving relevant regulations through modification, streamlining, expansion, or
repeal. The aim of retrospective analysis is to improve understanding of the
accuracy of prospective analysis and to provide a basis for potentially modifying
rules as a result of ex post evaluations. Rules should be written and designed to
facilitate retrospective analysis of their effects, including consideration of the data
that will be needed for future evaluation of the rules’ ex post costs and benefits.
OFFICE OF MGMT. & BUDGET, 2015 REPORT TO CONGRESS ON THE BENEFITS AND COSTS OF
FEDERAL REGULATIONS AND AGENCY COMPLIA NCE WITH THE UNFUNDED MANDATES REFORM
ACT 6 (2016) (with footnote 9 citing Michael Greenstone, Toward a Culture of Persistent
Regulatory Experimentation and Evaluation, in NEW PERSPECTIVES ON REGULATION (David
Moss & John Cisternino, eds., 2009)). However, in response to two commenters on the 2015
draft report who suggested that OMB should report the findings of retrospective reviews
alongside OMB’s reports of agencies’ prospective IAs for major rules over the past decade,
OMB replied that it hopes the agencies and outside researchers will do so. Id. at 109.
48. Exec. Order No. 13,563, 76 Fed. Reg. 3821 (Jan. 18, 2011).
49. Coglianese, supra note 42, at 65.
50. ALDY, supra note 22, at 22-26. See also Adam J. White, Retrospective Review, for
Tomorrow’s Sake, YALE J. ON REG.: NOTICE & COMMENT BLOG (Nov. 28, 2016),
(“retrospective review’s greatest virtue actually has nothing to do with repealing regulations.
Rather, retrospective review’s greatest value is forward-looking . . . to confront how accurate
or inaccurate the agencies’ own projections were in forecasting the rules’ impacts in the first
51. Barrack Obama, INAUGURAL ADDRESS BY BARACK OBAMA, Jan. 21, 2009,
visited Jan. 21, 2017).
Fall. 2016] REGULATION GOING RETRO 9
environmental quality and public health, but if poorly designed or if
conditions change, they can also induce new problems.52 If policy
makers try to foresee the expected consequences of proposed policy
actions, then efforts should be undertaken to validate these
forecasts and improve their accuracy over time. Without a
mechanism to learn what really works and how well (or poorly), it
will be unknown if government policies are achieving their intended
or optimal outcomes, and the government will not be able to improve
its foresight for subsequent policy decisions. Without ex post review,
ex ante IA can err in foreseeing impacts, and can more easily be
used to justify a policy choice already taken, rather than to learn
about actual impacts.53
Sections II and III of this article trace the evolution of IA, first
the rise and diffusion of ex ante analysis and then the more limited
emergence of ex post review, including EIA, RIA, and other related
tools intended to improve policy decisions and increase
accountability. Section IV offers an original contribution to the
literature with a new empirical analysis of agency reporting data on
the extent to which U.S. environmental regulation—in particular,
regulation by the Environmental Protection Agency (EPA)—is going
retro, in the sense of incorporating a learning mechanism by which
hindsight can improve foresight. We find low levels of
implementation of ex post EIA and RIA, and a focus on reducing the
cost burden of each policy taken individually, rather than
evaluation of a comprehensive scope of impacts or multi-policy
retrospective to test and learn to improve the accuracy of
prospective IA. Section V comments on the possible causes of and
remedies for the shortcomings we have observed, and makes
recommendations for future research and for institutional reforms
to improve the implementation of ex post IA, so better foresight can
evolve from better hindsight.
52. Wiener, supra note 11, at 124; JOHN D. GRAHAM & JONATHAN B. WIENER, RISK VS.
RISK: TRADEOFFS IN PROTECTING HEALTH AND THE ENVIRONMENT (Harvard Univ. Press,
1995). As regulatory impacts affect different people, further analysis is needed to assure a
fair distribution of welfare. See MATTHEW D. ADLER, WELL-BEING AND FAIR DISTRIBUTION:
BEYOND COST-BENEFIT ANALYSIS (Oxford Univ. Press, 2012).; Matthew D. Adler, Cost-Benefit
Analysis and Distributional Weights: An Overview (Duke Envtl. & Energy Econ., Working
Paper EE 13-05, 2013), http://scholarship.law.duke.edu/faculty_scholarship/3110.
53. See Claudio M. Radaelli, Rationality, Power, Management and Symbols: Four
Images of Regulatory Impact Assessment, 33 SCANDINAVIAN. POL. STUDIES 164-188, 171
(2010) (mentioning the de-coupling of the “talk” and practice of regulatory assessment
10 JOURNAL OF LAND USE [Vol. 32:1
II. ENVIRONMENTAL POLICY FORESIGHT
U.S. environmental law has evolved by progressively
incorporating analytical tools and methods of policy foresight. Four
such tools gained importance as not only methods for improving
regulation and other policy decisions, but also as measures to
increase accountability and better communicate decisions to the
public. The early application of CBA to government infrastructure
projects laid the methodological basis for the subsequent
deployment of prospective EIA and RIA. Similarly, formal methods
of prospective RA were developed to inform policy decisions.
A. CBA of Infrastructure Projects
The conceptual elements of CBA were evident in Benjamin
Franklin’s letter to Joseph Priestley in 1772, quoted above.54 There
is some evidence that these ideas then influenced pivotal thinkers
in late 18th century Paris, notably Jeremy Bentham and French
engineer-economists such as Jules Dupuit (professor at the Ecole
des Ponts et Chaussées), and the French military engineers later
brought these ideas back to America to train the U.S. Army Corps
of Engineers (Army Corps).55 CBA in U.S. policymaking first
appeared as a practice of the Army Corps in selecting projects.56
With the beginning of the professionalization of the civil service in
the 1880s, the Army Corps began to develop a systematized
planning process for designing and choosing priorities for
infrastructure projects based on economic analysis of anticipated
costs and benefits.57 The longstanding use of CBA by the Army
Corps’ archetype, the French Corps des Ponts et Chaussées, was a
direct influence.58 Since 1807, the French Corps had been
quantifying and monetizing the social costs and benefits of
infrastructure projects as a method of measuring their “public
utility” and ranking different projects competing for public funds.59
French influence—starting with engineers assisting the Americans
during the Revolutionary War—combined with the distinct political
54. Franklin, supra note 11.
55. See Wiener, supra note 11.
56. See THEODORE M. PORTER, TRUST IN NUMBERS ch. 7, at 148 (Princeton Univ. Press,
57. Id. at 151.
58. Id. at 148, 150.
59. Id. at 120.
Fall. 2016] REGULATION GOING RETRO 11
setting under which the U.S. Army Corps developed, stimulated the
gradual adoption and implementation of CBA.60
The U.S. Congress played a major role in formalizing and
routinizing CBA in the 20th century by making it a legal
requirement preceding funding decisions for public water projects—
first in the Rivers and Harbors Act of 1902 (RHA) and then in the
Flood Control Act of 1936 (FCA).61 The main policy motivations for
the two Acts were to improve governance and promote stronger
accountability in public spending; to enhance the perception of
fairness in the selection of water projects; and to control “pork
barrel” politics.62 The RHA and the FCA employed different
approaches to improving the rationality of water project funding
decisions. The RHA’s approach was mainly institutional, creating
an advisory body—a national-level advisory Board of Engineers for
Rivers and Harbors—but also procedural/methodological, i.e.,
creating the possibility of a CBA before the approval of a water
project.63 The FCA, on the other hand, built upon the RHA to
transform CBA into a binding normative standard for Congressional
approval of funds for every public water project.64 The standard
introduced by the FCA for Congressional approval was “if the
benefits to whomsoever they may accrue are in excess of the
estimated costs, and if the lives and social security of people are
otherwise adversely affected.”65 For the two types of studies,
forecasting the positive and negative, direct and indirect, effects of
public projects worked as a preceding step to the calculation of its
60. Wiener, supra note 11, at 134; PORTER, supra note 56; See infra note 67; JOE N.
BALLARD, THE HISTORY OF THE U.S. ARMY CORPS OF ENGINEERS 17 (1988) (on the
participation of French engineers in the Revolutionary War).
61. 33 U.S.C. §§ 541-579 (1902); 33 U.S.C. §§ 701-709 (1936).
62. BEATRICE HORT HOLMES, A HISTORY OF FEDERAL WATER RESOURCES PROGRAMS,
1800-1960, at 8 (1972); PORTER, supra note 56, at 149, 155.
63. According to the Rivers and Harbors Act of 1902 (RHA):
[I]n the consideration of such works and projects the board shall have in view the
amount and character of commerce existing or reasonably prospective which will be
benefited by the improvement, and the relation of the ultimate cost of such work,
both as to cost of construction and maintenance, to the public commercial interest
involved, and the public necessity for the work and propriety of its construction,
continuance, or maintenance at the expense of the United States.
33 U.S.C. § 541. The Act also stipulated that “all facts, information, and arguments which are
presented to the board for its consideration in connections with any matter referred to it by
the Chief of Engineers shall be reduced to and submitted in writing, and made a part of the
records.” Still, the board acted in an advisory board capacity, since only the projects referred
to it by the Chief of Engineers underwent a CBA analysis. Id.
64. See Flood Control Act of 1936, 33 U.S.C. §§ 701-709 (1936).
65. Id. at § 1(a), 33 U.S.C. § 701(a).
66. In the RHA, the idea of forecasting, with its inevitable uncertainty, is explicit in the
use of expression “reasonably prospective” to refer to the estimation of benefits. See id. at § 3,
33 U.S.C. § 541.
12 JOURNAL OF LAND USE [Vol. 32:1
The next period of significant methodological and institutional
developments of CBA, as a method for informing and promoting
accountability for policy decisions, occurred between the 1940s and
the late 1960s. To a remarkable extent, the standardization of CBA
methods was a product of American demand for transparency in
government decision-making.67 To resolve disputes over how to
conduct CBA analyses, the Bureau of the Budget—the predecessor
of the Office of Management and Budget (OMB)—used powers
vested by EO 9,384 of 1943 and, in 1952, issued Circular A-47 with
the first set of interagency guidelines for CBA of water projects.68
Circular A-47 consolidated years of evolution and methodological
debate about CBA, transforming CBA of water and related land
programs and projects into a process of sequential decisions, from
problem definition to the calculation of net benefits.69 CBA became
“an essential part of the process of formulating and selecting
projects.”70 The forecasting nature of CBA was once again evident,
now in the guidelines (“a concise but complete estimate of all the
benefits and all of the economic costs. Because any long-term
estimates are subject to wide margins of error, the results should be
expressed in ranges rather than in single figures”).71
67. A. R. Prest & R. Turvey, Cost-Benefit Analysis: A Survey, 75 ECON. J. 683-735, 684
(1965); PORTER, supra note 56, at 149, 162 (explaining how the most powerful advocates for
standardized methods of CBA were the opponents of the Army Corps, namely utilities,
railroads, the Soil and Conservation Service of the Department of Agriculture, and the
Bureau of Reclamation, in the Department of the Interior). Attitudes toward transparency
and access to information differed in France and the U.S., while decisions by the French
Conseil général about alternative programs—all backed by economic quantification—were
made in closed session and the Corps des Ponts protected itself by withholding information,
the U.S. Army Corps of Engineers was compelled to disclose its findings. Id. at 116, 144. In
Porter’s view, this transparency was one of the key factors explaining why CBA evolved in
the U.S., surpassing the French approach in methodological sophistication. Id.
68. BUREAU OF THE BUDGET, CIRCULAR A-47, REPORTS AND BUDGET ESTIMATES
RELATING TO FEDERAL PROGRAMS AND PROJECTS FOR CONSERVAT ION, DEVELOPMENT, OR USE
OF WATER AND RELATED LAND RESOURCES (1952). Circular A-47 established minimum criteria
that would be used by the Executive Office of the President when reviewing proposed water
project reports and budget estimates, with the goal of promoting “more uniform agency
policies and standards,” and to inform better priority setting among projects competing for
funds. Id. Circular A-47 was preceded by a series of studies by the Subcommittee on Benefits
and Costs, established in 1946 at the Inter-Agency Committee on Water Resources (IACWR),
with the goal of formulating uniform principles and procedures for CBA of water resources
projects. Executive Order 9,384 of 1943 required agencies to submit to the Bureau of the
Budget reports relating to or affecting Federal public works and improvement projects.
69. INTER-AGENCY COMM. ON WATER RESOURCES, PROPOSED PRACTICES FOR ECONOMIC
ANALYSIS OF RIVER BASIN PROJECTS 3, 11, 18, 22 (1958); John F. Timmons, Economic
Framework for Watershed Development, 36 J. FARM ECON. 1170, 1173 (1954).
70. INTER-AGENCY COMM. ON WATER RESOURCES, supra note 69, at 11.
71. BUREAU OF THE BUDGET, supra note 68, at 5; See also INTER-AGENCY COMM. ON
WATER RESOURCES, supra note 69, at 17. When discussing the treatment of risks in CBA, the
IACRW Report mentions:
Risks in the form of uncertainties for which no appropriate basis is available for
prediction include the probability of errors in estimating benefits and costs due to
Fall. 2016] REGULATION GOING RETRO 13
Influenced by the developments in welfare economics, the use of
CBA expanded from water projects to inform project decisions in
other areas, such as health, recreation, and land use.72 Soon,
planners overseas began to advocate the use of ex ante “evaluation
in planning” centered on CBA as the ideal approach to making
rational and transparent planning choices.73 Even with
methodological limitations, CBA was seen as an improvement
compared to open-ended concepts of the time (e.g., “best use of land
in the public interest,” “a pattern of land use that is reasonably
convenient, pleasing and cheap,” or “advantages and
CBA would soon become a key component of IA—in particular in
the U.S.75 During and after the 1970s, several U.S. environmental
statutes incorporated CBA for agency decisions or regulations,
independent of an IA.76
The creation of EIA in the U.S. was a landmark in the evolution
of normative frameworks of ex ante IA systems and policy foresight.
The National Environmental Policy Act (NEPA) (1969) was a
response to the countervailing environmental risks of government
actions by mission oriented-agencies, such as in the transportation
and energy sectors.77 EIA became by far the most operational and
such factors as fluctuations in levels of economic activity, technological changes and
innovations, and other unforeseeable developments adversely affecting the cost of
value of project services.
INTER-AGENCY COMM. ON WATER RESOURCES, supra note 69, at 23.
72. Nathaniel Lichfield, Cost-Benefit Analysis in Plan Evaluation, 35 TOWN PLAN. REV.
159, 163 (1964); ROBERT DORFMAN, MEASURING BENEFITS OF GOVERNMENT INVESTMENTS;
PAPERS PRESENTED AT A CONFERENCE OF EXPERTS HELD NOVEMBER 7-9, 1963, at 7-9 (1965);
PORTER, supra note 56, at 187 ("The new welfare economics presupposed that all pleasures
and pains in life were commensurable under a single, coherent, quantifiable utility function.
It seemed both intellectually serious and practically useful to try to work this out for such
difficult issues as recreation, health, and the saving or loss of life").
73. Lichfield, supra note 72.
74. Id. at 161, 168.
75. CBA is often a key part of RIA. See RICHARD L. REVESZ & MICHAEL A. LIVERMORE,
RETAKING RATIONALITY: HOW COST-BENEFIT ANALYSIS CA N BETTER PROTECT THE
ENVIRONMENT AND OUR HEALTH (Reprint ed. 2011). But RIA can be undertaken without full
CBA (for example, if RIA takes a goal as given and employs cost-effectiveness analysis).
76. For example, the Toxic Substances Control Act (TSCA) of 1976, § 6(c); the Safe
Drinking Water Act (SDWA) Amendments of 1996, authorizing EPA to determine whether
the benefits justify the costs before setting drinking water standards; or the Amended Gas
Pipeline Safety Standards of 1996, requiring CBA before setting safety standards. See Robert
W. Hahn, State and Federal Regulatory Reform: A Comparative Analysis, 29 J. LEG. STUD.
873, 889 (2000).
77. 42 U.S.C. § 4321; Jonathan B. Wiener, Managing the Iatrogenic Risks of Risk
Management, 9 RISK 39, 42-43 (1998). NEPA was a reaction against agencies neglecting
14 JOURNAL OF LAND USE [Vol. 32:1
significant of NEPA’s provisions, covering policy decisions of
different scales, including permits, projects, programs, plans,
regulations, and legislative proposals submitted by the Executive
Branch to Congress.78 The logic of EIA is to improve the
environmental outcomes of government decisions via analysis,
transparency, and public participation in the policy decision process
before implementation.79 EIA works as both a precautionary and
evidence-based tool with the potential of avoiding unintended
consequences and unnecessary environmental harms.80
Foresight is at the core of EIA. Agencies must undertake EIA to
foresee the environmental impacts of their actions, as the language
of foresight in the guidelines issued under NEPA by the Council on
Environmental Quality (CEQ) explicitly indicates.81 Regarding
uncertainty, CEQ guidelines require agencies to be clear about the
lack of sufficient information “when an agency is evaluating
reasonably foreseeable significant adverse effects on the human
environment in an environmental impact statement.”82 When
defining which cumulative impacts and indirect impacts EIA must
consider, the guidelines also mention “reasonably foreseeable”
impacts and future actions.83 In other countries, EIA regulations
environmental impacts. In the early NEPA case of Calvert Cliffs Coordinating Committee v.
AEC, 449 F.2d 1109 (DC Cir. 1971), Judge Skelly Wright saw in NEPA a requirement of CBA
to include the environmental costs of federal agency projects. See A. Dan Tarlock, The Story
of Calvert Cliffs: A Court Construes the National Environmental Policy Act to Create a
Powerful Cause of Action, in ENVIRONMENTAL LAW STORIES, 84 (Richard J. Lazarus ed., 2005);
Robert L. Rabin, Federal Regulation in Historical Perspective, 38 STAN. LAW REV. 1189, 1279
(1986). Agencies that did not have to prepare CBA analysis of their own projects and major
policy decisions under their own statutes, such as the Atomic Energy Commission and the
Department of Transportation, were most in need of this broader CBA incorporating
environmental impacts. Id. at 1299; JOANNA L. GRISINGER, THE UNWIELDY AMERICAN STATE:
ADMINISTRATIVE POLITICS SINCE THE NEW DEAL 12, 246 (2014); Tarlock, supra note. In the
first 8 years of implementation of NEPA, the Department of Transportation was the agency
most frequently involved in NEPA litigation, with 211 cases. COUNCIL ON ENVTL. QUALITY,
THE NINTH ANNUAL REPORT OF THE COUNCIL ON ENVIRONMENTAL QUALITY (1978). Later, the
U.S. Supreme Court held that NEPA did not impose a substantive CBA requirement, only a
procedural stop and think requirement. See Strycker’s Bay Neighborhood Council v. Karlen,
444 U.S. 223, 228 (1980).
78. National Environmental Policy Act of 1969 § 102(c)(i)-(ii), 42 U.S.C. § 4332 (2012).
79. See Craik, supra note 3.
80. The required EIS must be prepared and submitted early in the policy process. 40
C.F.R. § 1501.2. EISs should be based “upon the analysis and supporting data from the
natural and social sciences and the environmental design arts.” 40 C.F.R. § 1502.8.
81. Sections 102(c)(i) and (ii) of NEPA includes the core content of EIA, requiring its
report (i.e, the “detailed statement” prepared by the responsible agency official) to include
environmental impacts and adverse environmental effects from the action, without language
denoting uncertainty, such as “estimates” or “potential.” National Environmental Policy Act
of 1969 § 102(c)(i)-(ii), 42 U.S.C. § 4332 (2012).
82. 40 C.F.R. § 1502.22 (1978). The expression “reasonably foreseeable significant
adverse impacts” is repeated in other provisions of the rule.
83. 40 C.F.R. § 1508.7 (1978).
Fall. 2016] REGULATION GOING RETRO 15
and guidelines employ language denoting foresight at the core of
From its concise formulation in section 102 (C) of NEPA, EIA
evolved through the 1970s into a sophisticated and detailed set of
guidelines resulting from repeated interactions among Congress,
the President, courts, non-governmental actors, and the CEQ.85
From the U.S., the concept of EIA diffused throughout the globe and
reached over a hundred countries.86 Also, many states in the U.S.
adopted their versions of NEPA (or “little NEPAs”). In less than nine
years, over 10,000 environmental impact statements (EIS) had been
filed before federal agencies in the U.S., and many times this
number of environmental assessments.87 In most countries that
have adopted EIA, it only applies to project level decisions—perhaps
because it remained unclear, at least until the 1978 CEQ
regulations, which kinds of agency policy decisions would be
considered “major federal actions” to trigger an EIA under NEPA.88
Expanding the scope of EIA to cover programs and plans, the
European Union (EU) and its Member States passed legislation
creating the Strategic Environmental Assessment (SEA).89
84. In the U.K., for instance, EIAs should include “an estimate, by [the] type and
quantity, of expected residues and emissions (water, air and soil pollution, noise, vibration,
light, heat, radiation, etc[.]) resulting from the operation of the proposed development.” Town
and Country Planning Regulations 2011, No. 1824, Schedule 4, Part 1(c). In Canada, EIAs
should consider “environmental effects . . . that may occur in connection with the designated
project and any cumulative environmental effects that are likely to result from the designated
project . . . .” S.C. 2012, ch. 19, s. 52, § 19(1)(a). In Australia, a controlled action for which an
environmental assessment may be required should consider as relevant impacts those that
the action “(a) has or will have; or (b) is likely to have.” Environmental Protection and
Biodiversity Conservation Act 1999, Compilation No. 51 (2016), Div. 2, 821(1).
85. Herbert F. Stevens, The Council on Environmental Quality’s Guidelines and Their
Influence on the National Environment Policy Act, 23 CATH. U. L. REV. 547, 556 (1973). One
year after CEQ passed its 1978 regulation, the Supreme Court validated its legal force.
Andrus v. Sierra Club, 442 U.S. 347, 357-58 (1979); accord. Robertson v. Methow Valley
Citizens Council, 490 U.S. 332, 348-53 (1989); Marsh v. Oregon Natural Resources Council,
490 U.S. 360, 377 (1988); Warm Springs Dam Task Force v. Gribble, 417 U.S. 1301, 1309-10
86. Craik, supra note 3, at 23; Wiener & Ribeiro, supra note 3.
87. COUNCIL ON ENVTL. QUALITY, supra note 77, at 407. In the U.S. federal process,
environmental assessments are preliminary studies aimed at informing the agency decision
whether to conduct a full EIS. See 40 C.F.R. § 1508.9 (1978).
88. Jerry B. Edmonds, The National Environmental Policy Act Applied to Policy-Level
Decisionmaking, 3 ECOL. L.Q. 799, 799 (1973) (explaining how at first there were doubts on
whether the EIA should cover policy decisions at levels other than the project-level).
89. The creation of SEA as a supposedly distinct tool from EIA can be seen as a
rebranding effort, compared to the option of simply expanding the scope of EIA, as it has been
in the U.S. at least since 1978. “Policies,” a category that is usually used to include legislation,
regulation, and policy documents, has not been covered by SEA, except by the 2003 SEA
Protocol to the Espoo Convention, which focuses on transboundary effects. One possible
reason is the overlap with RIA. See infra Section II.D.
16 JOURNAL OF LAND USE [Vol. 32:1
RIA was also set up in the U.S. during the 1970s, partly to help
the Executive Branch oversee the flow of rules emanating from the
new environmental and other social legislation passed by Congress
during this period, and modeled in part on EIA.90 When Congress
enacted NEPA in 1969, commentators were discussing the
expansion of CBA from water projects, programs, and budget
planning to agency regulations.91 Responding to concerns over the
compliance costs of new environmental regulations, President
Nixon created the National Industrial Pollution Control Council and
transformed the Bureau of the Budget into its current form of the
OMB.92 The first formulation of what became the RIA in the U.S.
was issued by a memorandum from the OMB Director in May 1971,
creating the Quality of Life Review (QLR).93 Under the QLR
requirement, every agency had to submit proposed rules to OMB for
review and clearance before publishing a notice of proposed
rulemaking (NPRM).94 Agencies had to prepare a summary
description containing the principal objectives, alternatives
considered, costs and benefits of each alternative, and the reason for
selecting the preferred alternative.95 In practice, the QLR was
applied almost exclusively to environmental regulation from the
Since the 1970s, every American president of both major
political parties has maintained or expanded the ex ante RIA
framework. In 1978, President Carter issued EO 12,044 and created
90. NEPA was a source of inspiration for the development of the Quality of Life Review,
which responded to the perceived need for an “Economic Impact Statement.” Joe Conley II,
Environmentalism Contained: A History of Corporate Responses to the New
Environmentalism 162 (2006).
91. ALLAN SCHMID, EFFECTIVE PUBLIC POLICY AND THE GO VERNMENT BUDGET: A
UNIFORM TREATMENT OF PUBLIC EXPENDITURE AND PUBLIC RULES 579-91 (1969). According
to one participant, the tools of benefit-cost analysis and centralized review used at the Army
Corps of Engineers (discussed in Section II.A., supra) were then promoted by Allan Schmid
as a way to oversee regulation generally, and adopted by the Nixon Administration. See Jim
Tozzi, OIRA’s Formative Years: The Historical Record of Centralized Regulatory Review
Preceding OIRA’s Founding, 63 ADMIN. L. REV. (special ed.) 37, 41-43 (2011).
92. Christopher S. Yoo, Steven G. Calabresi & Anthony J. Colangelo, The Unitary
Executive in the Modern Era, 1945-2004, 90 IOWA L. REV. 601, 658 (2005); William H. Rodgers
Jr, National Industrial Pollution Control Council: Advise or Collude, 13 BC INDUS. COM. REV.
93. OFFICE OF MGMT. & BUDGET, OMB PAPERS: QUALITY OF LIFE REVIEW #1, AGENCY
REGULATIONS, STANDARDS, AND GUIDELINES PERTAININ G TO ENVIRONMENTAL QUALITY,
CONSUMER PROTECTION, AND OC CUPATIONAL AND PUBLIC HEALTH AND SAFETY 2 (1971); see
Tozzi, supra note 91, at 44-45.
Fall. 2016] REGULATION GOING RETRO 17
the “Regulatory Analysis” requirement, overseen by a “Regulatory
Analysis Review Group.”97 In 1980, Congress enacted and President
Carter signed the Paperwork Reduction Act (PRA), creating OIRA
within OMB.98 In 1981, President Reagan issued EO 12,291,
replacing Carter’s EO and giving the tool its current name, the
“Regulatory Impact Analysis,” as well as giving OMB/OIRA the
authority to oversee RIAs.99 In 1993, President Clinton issued EO
12,866, replacing and improving upon Reagan’s EO; subsequent
presidents have maintained EO 12,866 in effect.100 In 2011,
President Obama issued EO 13,563, supplementing without
rescinding EO 12,866,101 notably by requiring retrospective review
as discussed above.102 In 2003, OIRA issued Circular A-4, which
continues to serve as the main RIA guidelines in the U.S.103 These
EOs use language calling for foresight in RIA.104 Similarly, the
97. Exec. Order No. 12,044, 43 Fed. Reg. 12,661 (1978). Earlier, in 1974, President Ford
(while maintaining the QLR requirement) issued EO 11,821, creating the Inflation Impact
Statement (IIS) (renamed in 1976 “Economic Impact Statements”) as an additional
requirement to executive agency rulemaking. Exec. Order No. 11,821, 39 Fed. Reg. 41,501
(1974); Exec. Order No. 11,949, 41 Fed. Reg. 23,663 (1976).
98. Pub. L. No. 96-511, 94 Stat. 2812 (1980) (codified at 44 U.S.C. §§ 3501-3521).
99. Exec. Order No. 12,291, 3 C.F.R. § 127 (1981).
100. Exec. Order No. 12,866, 3 C.F.R. § 638 (1993).
101. Exec. Order No. 13,563, 76 Fed. Reg. 3821 (2011). The scope of impacts to be covered
has evolved across these EOs. EO 12,044 referred to the “economic consequences” of the
proposed rule, EO 12,044, § 3(b)(1) (not using the word “benefits”). EO 12,291 used the
language of costs and benefits, but without a specific mention of environmental and social
impacts (section 3(d)(1)). Under EOs 12,866 and 13,563, RIA must assess costs and benefits,
including impacts on the environment, public health and safety, and on discrimination or bias
(section 6(a)(3)(C)(i) of EO 12,866 and section 1(b)(3) of EO 13,563). Moreover, EO 12,291
called for benefits to “outweigh” costs, whereas EO 12,866 changed this language to call for
benefits to “justify” costs. See Exec. Order No. 12,044, 43 Fed. Reg. 12,661 (1978); Exec. Order
No. 12,291, 46 Fed. Reg. 13,193 (1981); Exec. Order No. 12,866, 58 Fed. Reg. 51,735 (1993).
102. See supra notes 29-32. The current RIA framework in the U.S. is comprised mainly
of EOs 12,866, 13,563, and 13,610. But in addition to these basic requirements applicable to
significant rulemaking actions of executive agencies (and EO 13,579 regarding independent
agencies), the overall picture of RIA in the U.S. is more complex and fragmented. In addition
to the RIA, agencies are subject to RIA-like requirements focusing on specific classes of
stakeholders or kinds of impacts, such as on takings of private property (EO 12,630 of 1988),
Indian tribal governments (EO 13,175 of 2000), children (EO 13,045 of 1997), health and
environmental impacts on minorities (EO 12,898 of 1994), and energy (EO 13,211 of 2001). In
1980, drawing inspiration from the same political and economic circumstances of the late-
1970s, Congress passed two statutes: the Regulatory Flexibility Act (RFA) and the Paperwork
Reduction Act (PRA) of 1980, both calling for ex ante analysis—the RFA, for impacts on small
businesses; and the PRA, for impacts resulting from information requirements. In addition to
the RFA and the PRA, Congress also passed the Unfunded Mandates Reform Act (UMRA) of
1995, requiring ex ante RIA of any proposed agency rule that may result in the expenditure
by a state, local, tribal government, or by the private sector, in the aggregate, of more than
$1 million in any one year. See Wiener & Ribeiro, supra note 3, at 175.
103. OFFICE OF MGMT. & BUDGET, CIRCULAR A-4, REGULATORY ANALYSIS 1 (2003)
[hereinafter OMB, A-4].
104. “[T]he expected benefits or accomplishments and the costs,” OFFICE OF MGMT. &
BUDGET, supra note 93, at 2 (emphasis added); “an estimate … of the new reporting burdens
or recordkeeping requirements,” Exec. Order No. 12,044, § 2(d)(6), 43 Fed. Reg. 12,661 (1978)
18 JOURNAL OF LAND USE [Vol. 32:1
foresight nature of RIA is evident in Circular A-4’s provisions
related to uncertainty.105 In addition to using the same language of
the EOs denoting forecasting (e.g., “anticipate and evaluate the
likely consequences of rules”), Circular A-4 has detailed sections on
the uncertainty elements involved in foreseeing the effects of
Similarly, but less rapidly than EIA, the concept of RIA has
diffused throughout other national and subnational jurisdictions. At
the state level in the U.S., ex ante RIA was adopted by many states,
under different names and with different scopes, also figuring in the
Model State Administrative Procedure Act.107 Internationally, ex
ante RIA became the cornerstone of the Better Regulation
movement in Europe, and was adopted by every OECD member
country.108 The United Kingdom (U.K.) and the EU represent two
(emphasis added); “an estimate of the number of small entities to which the proposed rule will
apply,” and “the projected reporting, recordkeeping and other compliance requirements,” 5
U.S.C. § 603(b)(3)-(4) (2012) (emphasis added); “estimate of the burden that shall result from
the collection of information” and “a description of the likely respondents and proposed
frequency of response to the collection of information,” 44 U.S.C. § 3507(a)(D)(ii)(V)-(IV)
(2012) (emphasis added); “[a] description of the potential benefits [and costs] of the rule . . .
and the identification of those likely to receive the benefits [and bear the costs,” Exec. Order
No. 12,291, § 3(d)(1), 3 C.F.R. § 127 (1981) (emphasis added); “assessment of the potential
costs and benefits of the regulatory action” and “[a]n assessment . . . of benefits [and costs]
anticipated from the regulatory action.” Exec. Order No. 12,866, § (6)(a)(3)(B)(ii), -(C)(i),
3 C.F.R. § 638 (1993) (emphasis added); and “to quantify anticipated present and future
benefits and costs.” Exec. Order No. 13,563, § (1)(c), 3 C.F.R. § 13,563 (2011) (emphasis
105. OMB, A-4, supra note 103.
106. Id. Some key examples are stipulating and measuring the baseline (“what the world
would be like if the proposed rule is not adopted”) and dealing with uncertainty (with
emphasis on identifying key uncertainties and conducting sensitivity analysis, as a way of
anticipating the effect of changing forecasting assumptions). Id. at 2. In one section, Circular
A-4 stipulates: “[y]our estimates cannot be more precise than their most uncertain
component. Thus, your analysis should report estimates in a way that reflects the degree of
uncertainty and not create a false sense of precision.” Id. at 40.
107. See Russell S. Sobel & John A. Dove, Analyzing the Effectiveness of State Regulatory
Review, 44 PUB. FIN. REV. 446 (2016); JASON A. SCHWARTZ, 52 EXPERIMENTS WITH
REGULATORY REVIEW 87 (2010) (with a detailed view of each state, finding that "45 states
require[d] some form of [ex ante] economic impact analysis, besides specialized reviews like
regulatory flexibility analysis."); see also Stuart Shapiro & Deborah Borie-Holtz, Regulatory
Reform in the States: Lessons from New Jersey, (March 24, 2011) (available at
http://papers.ssrn.com/abstract=1794172). In another 2011 study, Shapiro and Borie-Holtz
classified the states of New York, Virginia, Michigan, and Pennsylvania with a maximum
score of stringency of RIA requirements. STUART SHAPIRO & DEBORAH BORIE -HOLTZ,
INSTITUTE FOR POLICY INTEGRITY, DOES PROCESS MATTER: REGULATORY PROCEDURE AND
REGULATORY OUTPUT IN THE STATES 8 n.13 (2011), http://policyintegrity.org/publications/
detail/does-process-matter; NAT’L CONFERENCE O F COMM’RS ON UNIF. STATE LAWS, REVISED
MODEL STATE ADMINISTRATIVE PROCEDURE ACT § 305 (2010) (suggesting implementation
comparison of alternatives on the basis of CBA of a proposed rule, and a determination that
the benefits of the proposed rule justify its costs).
108. Jonathan B. Wiener, Better Regulation in Europe, 59 CURRENT LEGAL PROB. 447
(2006); Wiener, supra note 11, at 126-28. OECD has been a major supplier of information and
experiences on regulatory quality improvement, helping to spread RIA. Id. at 130. See also
Fall. 2016] REGULATION GOING RETRO 19
jurisdictions in which ex ante RIA has been making significant
advances, leading to innovative institutional transformations in
recent years.109 In the latest version of its guidelines, the U.K. RIA
system mentions the use of sensitivity analysis, while also
discussing how to report the uncertainty of parameters assessed in
RIA.110 In 2015, the European Commission also issued a new set of
guidelines for its IA system, with similar language and provisions
revealing the forecasting basis of IA.111
D. Risk Assessment
Alongside or within CBA, EIA, and RIA, ex ante risk assessment
(RA) has been a key analytical tool for foreseeing future potential
harms of pollutants and other stressors.112 EPA has employed
formal RA practices since its early days, having issued its first RA
document at least as early as 1975.113 The National Academy of
Sciences (NAS) encouraged federal agencies to conduct RAs and
FABRIZIO DE FRANCESCO, TRA NSNATIONAL POLICY INNOVATION: THE OECD AND THE
DIFFUSION OF REGULATORY IMPACT ANALYSIS (Dario Castiglione et al. eds., 2013)
(documenting the influence of OECD on the diffusion of RIA); OECD, POLICY OUTLOOK, supra
109. For recent developments in the U.K. and EU systems, respectively, see DEP'T FOR
BUS. INNOVATION & SKILLS, BETTER REGULATION FRAMEWORK MANUAL (2015), and EUR.
COMM'N, Better Regulation Guidelines, COM (2015) 215 final (April 19, 2015) [hereinafter EC,
Better Regulation Guidelines].
110. DEP'T FOR BUS. ENERGY & INDUS. STRATEGY, BETTER REGULATION FRAMEWORK
MANUAL § 2.2.5 (2015). The Manual is also explicit when discussing the estimate levels and
underlying uncertainty of compliance with the proposed regulation. Id. at § 2.3.50. It has a
specific topic for “Key assumptions, sensitivities and risks” in which the foresight nature of
RIA becomes evident: “[i]n order to reflect the inherent uncertainty of costs and benefits
estimates, you may need to provide a range for your costs and benefits estimates. Highlight
the factors determining the outcome within any range and how any risks will be mitigated.”
Id. The Better Regulation Framework Manual refers to the Green Book as the main source of
detailed methodological guidelines. Id. at § 1.5.5.
111. EC, Better Regulation Guidelines, supra note 109 (mentioning “assumptions,”
“uncertainty,” “estimates,” and “sensitivity” in many parts of the document). For example:
“When quantifying [all relevant impacts], spurious precision should be avoided and ranges
provided . . . . Whenever an assumption is particularly important or uncertain, sensitivity
analysis should be used to check whether changing it would lead to significantly different
results.” Id. at 27.
112. See Alon Rosenthal, George M. Gray & John D. Graham, Legislating Acceptable
Cancer Risk from Exposure to Toxic Chemicals, 19 Ecology L.Q. 269 (1992) (documenting
requirements for RA in many environmental laws); RICHARD L. REVESZ, ENV’L L. & POL’Y 51
(3d ed. 2015) (“Risk assessment is generally recognized as the first step in the regulatory
process—a regulatory agency must first analyze the magnitude of an environmental risk
before it can intelligently decide on whether and how much risk should be regulated—a
process known as risk management“).
113. U.S. ENVTL. PROT. AGENCY, EPA/100/B-04/001, AN EXAMINATION OF EPA, RISK
ASSESSMENT PRINCIPLES & PRACTICE 4 (2004) [hereinafter EPA, 2004 EXAMINATION]; see also
CARNEGIE COMM'N ON SCI., TECH., & GOV'T, RISK AND THE ENV'T 27 (1993).
20 JOURNAL OF LAND USE [Vol. 32:1
outlined guidelines for best practices.114 In some cases,
environmental statutes incorporate RA as a requirement for agency
decision or rulemaking—either independently or combined with
EIA, RIA, and CBA.115 The Clean Air Act (CAA), for instance,
requires EPA to make findings that a pollutant “may reasonably be
anticipated to endanger public health or welfare” for setting
national ambient air quality standards, and to conduct residual RAs
after setting emissions standards for major sources of hazardous air
pollutants;116 the Resource Conservation and Recovery Act (RCRA)
mandates that EPA make findings of endangerment to public health
or the environment to regulate hazardous waste sites;117 the Federal
Insecticide, Fungicide, and Rodenticide Act (FIFRA) stipulates a
risk/benefit analysis for the registration of pesticides.118 Many other
environmental statutes impose criteria or standards without
formally requiring (but in practice leading to) RA processes.119
Forecasting is at the center of RA, because RA attempts to
characterize the likelihood and severity of future adverse events
with the purpose of informing decisions marked by uncertainty.120
The NAS’s National Research Council has acknowledged that risk
assessors rely on assumptions and make use of “inferential bridges”
in order to conduct ex ante RA in the face of uncertainty.121 The
114. NAT'L RESEARCH COUNCIL, RIS K ASSESSMENT IN THE FEDERAL GOVERNMENT:
MANAGING THE PROCESS 57-58 (1983); NAT'L RESEARCH COUNCIL, SCIENCE AND DECISIONS:
ADVANCING RISK ASSESSMENT 26 (2009); see also Junius C. McElveen, Jr., Risk Assessment
in the Federal Government: Trying to Understand the Process, 5 TUL. ENVTL. L.J. 45, 53
(1991); EPA, 2004 EXAMINATION, supra note 113, at 3 (describing the use of risk assessment
as a routine activity by EPA for making multiple kinds of decisions).
115. RA of pollutants or other stressors should, in principle, provide the information for
the harm estimates in EIA and for the risk reduction benefits estimates in RIA. See Alan L.
Porter & Frederick A. Rossini, Integrated Impact Assessment, 6 INTERDISC. SCI. REV. 346
(1981); Fred Anderson et al., Regulatory Improvement Legislation: Risk Assessment, Cost-
Benefit Analysis, and Judicial Review, 11 DUKE ENVTL. L. & POL'Y F. 89, 93 (2000); Hossein
Mahmoudi et al., A framework for combining social impact assessment and risk assessment,
43 ENVTL. IMPACT ASSESS. REV. 1 (2013).
116. 42 U.S.C. §§ 7408(a)(1)(A), 7412(f)(1) (2012); see also U.S. ENVTL. PROT. AGENCY,
EPA-453/R- 99-001, RESIDUAL RISK: REPORT TO CONGRESS (1999).
117. McElveen, supra note 114, at 48 n.3.
118. See GOV'T ACCOUNTABILITY OFFICE, GAO/RCED-91-52, EPA’S USE OF BENEFIT
ASSESSMENTS IN REGULATING PESTICIDES 9 (1991).
119. Rosenthal et al., supra note 112; McElveen, supra note 114.
120. For a historical account of the evolution of risk analysis from probability theory, see
PETER L. BERNSTEIN, AGAINST THE GODS: THE REMARKABLE STORY OF RISK (1996). On de ep
uncertainty in RA, see Robert J. Lempert & Myles T. Collins, Managing the Risk of Uncertain
Threshold Responses: Comparison of Robust, Optimum, and Precautionary Approaches, 27
RISK ANALYSIS 1009 (2007). On RA and management of extreme catastrophic risks, see Nick
Bostrom, Existential Risk Prevention as Global Priority, 4 GLOB. POL'Y 15 (2013); Jonathan
B. Wiener, The Tragedy of the Uncommons: On the Politics of Apocalypse, 7 GLOB. POL'Y 67
121. NAT'L RESEARCH COUNCIL, RISK ASSESSMENT IN THE FEDERAL GOVERNMENT, supra
note 114, at 3, 28. By contrast, RIA fits into the NRC’s definition of risk management: “the
process of weighing policy alternatives and selecting the most appropriate regulatory action,
Fall. 2016] REGULATION GOING RETRO 21
analytical steps of RA (hazard identification, dose-response
assessment, exposure assessment, and risk characterization) are
necessarily inferential, resulting in estimates with ranges of
III. FROM FORESIGHT TO HINDSIGHT: THE RISE OF
The development of prospective analytical tools for policy
foresight—such as ex ante RA, CBA, EIA, and RIA—has enabled
important advances in protection of public health, environment and
security against uncertain future risks, but it has also prompted the
question whether these ex ante tools are generating accurate
foresight. There is growing interest in developing evidence-based
tools to enable retrospective, ex post, or look-back reviews of past
policies.123 The precautionary approach underlying ex ante IA tools
(“look before you leap”) also suggests the value of revisiting earlier
estimates in light of data on actual experience: prudent precaution
is provisional, to be revised as knowledge improves.124 Different
forms of retrospective, ex post, and periodic reviews have gained
ground in the literature and have gradually been adopted by
governments, supplementing ex ante RA, CBA, EIA, and RIA tools.
This Section describes these developments. A key finding of our
inquiry is that retrospective reviews have more often been aimed at
reducing the cost of individual rules, and less often at learning from
experience to improve the accuracy of ex ante foresight.
A. Ex Post RIA in the U.S.
Ex ante analysis of regulatory impacts of proposed rules, and ex
post evaluation of existing rules, developed as intertwined ideas
since the early years of RIA in the U.S.125 While addressing
Congress in 1974, President Ford asked Congress to “undertake a
integrating the results of risk assessment with engineering data and with social, economic,
and political concerns to reach a decision.” Id. at 3; see also Anderson et al., supra note 115,
at 91. See NAT'L RESEARCH COUNCIL, RISK ASSESSMENT IN THE FEDERAL GOVERNMENT, supra
note 114, at 7 (stressing the importance of communicating uncertainty and variability in the
results of RA).
122. See M. Granger Morgan, Risk Analysis and Management, 269 SCI. AM. 32, 34 (1993)
(explaining the different uncertainties inherent to risk analysis and, consequently, the need
to represent them with probability distributions).
123. See ALDY, supra note 22; Coglianese, supra note 42.
124. See Wiener, supra note 2.
125. Nixon’s QLR, however, focused only on the estimated impacts of new regulation,
given its predominant focus—the recent new wave of environmental regulation. OFFICE OF
MGMT. & BUDGET, supra note 93, at 1.
22 JOURNAL OF LAND USE [Vol. 32:1
long-overdue total reexamination of the independent regulatory
agencies” as part of a joint effort to “identify and eliminate existing
federal rules and regulations that increase costs to the consumer
without any good reason in today’s economic climate.”126 But soon
after, when he issued EO 11,821, Ford’s Inflation Impact Statement
(IIS) focused only on proposals for legislation or promulgation of new
regulations and rules by executive agencies.127 In 1978, President
Carter’s EO 12,044 not only expanded the ex ante RIA requirement
to address all economic impacts, but also innovated significantly by
introducing ex post RIA.128 Carter’s EO had a specific section on
“Review of Existing Regulations,” requiring agencies to “periodically
review their existing regulations to determine whether they are
achieving the policy goals of this Order.”129 In addition to this
central mandate, EO 12,044 also stipulated
procedural/methodological rules, as well as selection criteria,
communication and participation requirements of such regulatory
reviews.130 Methodologically, regulatory reviews should “follow the
same procedural steps outlined for the development of new
regulations,”131 i.e., ex ante regulatory analysis.132 The criteria
developed by each agency for selecting rules for review—based on
the general criteria stipulated by the EO—and the list of regulations
selected for review were to be published and included in the
semiannual agency agendas.133 EO 12,044 also required that new
regulations include “a plan for evaluating the regulation after its
126. Gerald Ford, “WHIP INFLATION NOW” SPEECH (OCTOBER 8, 1974), MILLER CENTER,
UNIVERSITY OF VIRGINIA (Oct. 8, 1974), http://millercenter.org/president/ford/speeches/
speech-3283 (last visited Jan. 21, 2017).
127. See OMB, A-4, supra note 103. Exec. Order No. 11,821, Preamble, 39 Fed. Reg.
41,501 (1974). The Council on Wage and Price Stability, created in 1974 by Congress, to which
EO 11,821 allowed OMB to delegate its oversight functions related to the IIS, employed broad
language to describe its role, which could potentially include reviewing the performance of
existing programs and activities. Council on Wage and Price Stability Act, Pub. L. No. 93-387,
§ 3(A)(7), 88 Stat. 750, 750 (1974).
128. See Exec. Order No. 12,044, 43 Fed. Reg. 12,661 (1978).
129. Id. at § 4. The goals of the EO are stipulated in section 1, according to which
“[R]egulations shall be as simple and clear as possible. They shall achieve legislative goals
effectively and efficiently. They shall not impose unnecessary burdens on the economy, on
individuals, on public or private organizations, or on State and local governments.” Id. at § 1.
130. Id. at § 4.
132. Including, in the case of significant regulations with major consequences, “a careful
examination of alternative approaches” and a “succinct statement of the problem; a
description of the major alternative ways of dealing with the problem that were considered
by the agency; an analysis of the economic consequences of each of these alternatives and a
detailed explanation of the reasons for choosing one alternative over the others.” Id. at §
133. Id. at § 2(a).
Fall. 2016] REGULATION GOING RETRO 23
issuance has been developed”134—a prospective provision for
After Carter’s EO, every other EO issued on RIA included a
provision regarding retrospective reviews of existing regulation,
although typically with a less comprehensive framework than in EO
12,044. For example, section 3 of EO 12,291, issued by President
Reagan in 1981, included a subsection requiring agencies to “initiate
reviews of currently effective rules in accordance with the purposes
of this Order, and perform Regulatory Impact Analyses of currently
effective major rules.”135 The provision requiring agencies to include
in ex ante RIA a plan for future review disappeared, as well as
mentions of selection criteria for review.136 On the other hand, OMB
was given express authority to designate currently effective rules
for review and establish schedules for reviews and analyses under
the EO.137 Then, in 1993 with EO 12,866, President Clinton included
section 5 on ex post evaluation of existing regulations, requiring
publication of regulations selected for review in each agency’s
annual plan and regulatory agenda, empowering the Vice President
to identify rules for review, and instructing agencies to conduct
reviews to make existing rules “more effective in achieving the
regulatory objectives, less burdensome, or in greater alignment with
the President’s priorities.”138
President Obama supplemented EO 12,866 with three
additional EOs, all with rules for retrospective review of existing
regulations.139 Like EO 12,866, EO 13,563 dedicates one section to
134. Id. at § 2(d)(8).
135. Exec. Order No. 12,291, § 3(i), 46 Fed. Reg. 13,193 (Feb. 17, 1981).
136. See id.
137. In 1985, President Reagan issued EO 12,498, once again addressing the need to
reduce the burdens of “existing and future regulations.” It created a requirement that
agencies should annually state their regulatory policies, goals, and objectives for the coming
years, including “information concerning all significant regulatory actions underway or
planned.” Exec. Order No. 12,498, § 1, 50 Fed. Reg. 1036 (Jan. 4, 1985). In 1992, President
Bush announced in his State of the Union Address a 90-day moratorium on new regulation,
and a review of federal regulations, which was then directed to agencies via a memorandum
on the same day. The memorandum defines the standards for review, mirroring much of the
process applicable to ex ante RIA under EO 12,291. Neil R. Eisner et al., Federal Agency
Reviews of Existing Regulations, 48 ADMIN. L. REV. (AM. BAR ASS'N) 139, 142 (1996). President
Clinton followed the same approach and mandated another one-time review effort of existing
regulations via memorandum issued to federal agencies in 1995. Hahn, supra note 76, at 887.
138. Exec. Order No. 12,866, § 5(a), 58 Fed. Reg. 51,735 (Sep. 30, 1993). The goal of such
review is defined in the same provision as “to determine whether any such regulations should
be modified or eliminated so as to make the agency’s regulatory program more effective in
achieving the regulatory objectives, less burdensome, or in greater alignment with the
President’s priorities and the principles set forth in this Executive Order.”
139. Exec. Order No. 13,563, 76 Fed. Reg. 3821 (Jan. 18, 2011); Exec. Order No. 13,579,
76 Fed. Reg. 41,585 (Jul. 11, 2011); Exec. Order No. 13,610, 77 Fed. Reg. 28,469 (May 10
24 JOURNAL OF LAND USE [Vol. 32:1
what it calls “Retrospective Analysis of Existing Rules.”140 By
reaffirming the provision in section 5 of EO 12,866, President
Obama signaled that at least some agencies had not complied with
it so far, requiring them again to submit to OIRA “a preliminary
plan . . . under which the agency will periodically review its existing
regulations . . . .”141 In the following year, President Obama issued
EO 13,610, on “Identifying and Reducing Regulatory Burdens.”142
This new EO added to the ex post RIA system a provision on public
participation, and created a complementary duty requiring agencies
to report semiannually to OIRA “on the status of their retrospective
review efforts,”143 describing “progress, anticipated
accomplishments, and proposed timelines for relevant actions . . .
.”144 EO 13,610 also stipulated in section 3 a set of factors that
agencies should consider when setting priorities and selecting rules
for review.145 OIRA has issued a series of memoranda pressing the
agencies to identify existing rules and conduct reviews, but OIRA
has not yet issued a full guideline document for ex post RIA akin to
Circular A-4 for ex ante RIA.146
In addition to presidents using their executive powers to
promote ex post RIA in the U.S., Congress has also created statutory
ex post evaluation requirements. For example, the Regulatory
Flexibility Act (RFA) provisions on periodic regulatory review
require that every regulation with a significant economic impact on
a substantial number of small entities must undergo a review
140. Exec. Order No. 13,563, 76 Fed. Reg. 3821 (Jan. 18, 2011).
141. Id. at § 6(b). The provision announces the same goal of the review “to determine
whether any such regulations should be modified, streamlined, expanded, or repealed so as
to make the agency’s regulatory program more effective or less burdensome in achieving the
regulatory objectives.” Id. The other operational provision in section 6—this one original—
directs agencies to release “[s]uch retrospective analyses, including supporting data, . . .
online whenever possible.” Id. at § 6(a).
142. Exec. Order No. 13,610, 77 Fed. Reg. 28,469 (May 10, 2012). EO 13,610 renames
“review of existing regulations” as “retrospective review.” The same overall purpose is
reaffirmed: “to conduct retrospective analyses of existing rules to examine whether they
remain justified and whether they should be modified or streamlined in light of changed
circumstances, including the rise of new technologies.” Id. at § 1.
143. Id. at §§ 3-4.
144. Id. at §§ 2, 4. EO 13,610 also requires that such semiannual reports be made
available to the public, as well as the “retrospective analyses of regulations, including
supporting data”—the latter, “wherever practicable.” Id.
145. The factors are: (a) reviews that will “produce significant quantifiable monetary
savings or significant quantifiable reductions in paperwork burdens while protecting public
health, welfare, safety, and our environment;” (b) reviews that will “reduce unjustified
regulatory burdens or simplify or harmonize regulatory requirements imposed on small
business;” (c) reforms that would make “significant progress in reducing those burdens while
protecting public health, welfare, safety, and our environment;” and (d) “consideration to the
cumulative effects of agency regulations, including cumulative burdens.” Id. at § 3.
146. Coglianese, supra note 42, at 61-62; see also Cary Coglianese, Empirical Analysis
and Administrative Law, 2002 UNIV. ILL. L. REV. 1111, 62 (2002).
Fall. 2016] REGULATION GOING RETRO 25
within ten years of being issued.147 The Paperwork Reduction Act
(PRA) allowed any interested party to request that OMB review an
existing information collection requirement, which could lead to a
“remedial” action by OMB and the agency.148 Also, the PRA called
for new regulations to have their information collection
requirements reviewed every three years after initial approval;
based on the review report, OMB can approve or disapprove the
extension.149 The Unfunded Mandates Act (UMRA) also has a
provision regarding review of existing regulations, although with a
provisional nature.150 Several specific laws also require periodic
reviews of past policies: examples include the five year reviews of
national ambient air quality standards in the CAA, and the six year
reviews of drinking water quality standards in the Safe Drinking
Water Act (SDWA).151
At least since 1996, Congress began to include in appropriations
legislation a requirement directing OMB to annually submit reports
containing “estimates of the total annual costs and benefits of
Federal Regulatory programs, including quantitative and non-
quantitative measures of regulatory costs and benefits.”152 Initially,
the requirement also stipulated that OMB should include in its
report “recommendations from the Director . . . to reform or
eliminate any Federal regulatory program or program element that
is inefficient, ineffective, or is not a sound use of the Nations’
147. 5 U.S.C. § 610(a) (“[t]he purpose of the review shall be to determine whether such
rules should be continued without change, or should be amended or rescinded, consistent with
the stated objectives of applicable statutes, to minimize any significant economic impact of
the rules upon a substantial number of such small entities.”).
148. 44 U.S.C. §§ 3507(g)-(h)(1) (2000).
150. Title III of the Unfunded Mandates Act of 1995 (UMRA) addresses “Review of
Federal Mandates,” granting the Advisory Commission on Intergovernmental Relations
powers to investigate and review the role and impact of existing Federal mandates. As a result
of such review—which appears in the Act to be a one-time analysis—the Commission may
make a recommendation for “suspending, on a temporary basis, Federal mandates which are
not vital to public health and safety and which compound the fiscal difficulties of State, local,
and tribal governments, including recommendations for triggering such suspension.” 2 U.S.C.
§ 1552(a)(3)(d) (1995).
151. 42 U.S.C. § 7409(d) (Clean Air Act provision for NAAQS to be reviewed every five
years); 42 U.S.C. § 300g-1(b)(9) (1996) (SDWA provision for six year reviews). The Frank R.
Lautenberg Chemical Safety for the 21st Century Act of 2016, amending the Toxic Substances
Control Act (TSCA), 15 U.S.C. § 2601 et seq., now calls for periodic reviews of policies every
five years. Pub. L. 114-182 (June 22, 2016), amending TSCA to insert section 26(l), to be
codified at 15 U.S.C. § 2625(l). The Telecommunications Act of 1996 calls for biennial reviews
by the Federal Communications Commission. Robert Hahn et al., Assessing the Quality of
Regulatory Impact Analyses, 23 HARV. J. OF LAW & PUB. POL'Y 889 (2000).
152. OFFICE OF INFO. & REGULATORY AFFAIRS, 1997 REPORT TO CONGRESS ON THE COSTS
AND BENEFITS OF FEDERAL REGULATIONS (1997) [hereinafter OIRA, 1997 REPORT].
26 JOURNAL OF LAND USE [Vol. 32:1
resources.”153 The provisions were annually renewed in
appropriations legislation until, in 2001, it became a permanent
feature of what is now known as the Regulatory Right-to-Know
Act.154 In 2012, Congress passed the Consolidated Appropriations
Act, which also requires OMB to include in its annual report to
Congress information on agency implementation of EO 13,563; in
particular, it requires OMB to identify “existing regulations that
have been reviewed and determined to be outmoded, ineffective, and
Following the same pattern of diffusion of ex ante RIA, U.S.
states have also adopted requirements for periodic ex post reviews
of existing regulations.156 The 1981 edition of the Model State
Administrative Procedure Act (MSAPA) suggests a provision
requiring periodic review of all agency regulations in no longer than
seven years.157 In 2000, Robert Hahn reported that nearly one-third
of the states had adopted comprehensive review requirements of all
existing regulations.158 The extensive analysis conducted in 2010 by
Jason Schwartz of the Institute for Policy Integrity found thirty
states in which agencies were either encouraged or required to
reevaluate their existing regulations periodically.159 The trigger for
153. Treasury, Postal Services and General Government Appropriations Act of 1997, §
645, Pub. L. No. 104-208, 110 Stat. 3009 (codified as amended in 28 U.S.C. § 2241 (2008)).
154. Treasury and General Government Appropriations Act of 1998, § 625, Pub. L. No.
105-277, 112 Stat. 2681 (1998); Treasury and General Government Appropriations Act of
2001, § 624, Pub. L. No. 106-554, 115 Stat. 514 (2001) (“[f]or the calendar year 2002 and each
year thereafter…”). Starting in 1999, the language used in the two provisions changed:
regarding the recommendations for reform, the new statute directed OMB to only include in
its report “recommendations for reform.” Id.; Omnibus Consolidated and Emergency
Supplemental Appropriations Act of 1999, Pub. L. No. 105-227, §§ 683(a)(1)-(3), 112 Stat. 2681
(1999) (requiring that in the accounting statement and associated report submitted by OIRA
there should be “recommendations for reform.”). Section 638(a)(1) became “an estimate of the
total annual costs and benefits (including quantifiable and non-quantifiable effects) of Federal
rules and paperwork, to the extent feasible: (A) in the aggregate; (B) by agency and agency
program; and (C) by major rule.” Id.
155. Consolidated Appropriations Act of 2012, Pub. L. 112-74, 125 Stat. 786; OFFICE OF
INFO. & REGULATORY AFFAIRS , 2012 REPORT TO CONGRESS ON THE BENEFITS AND COSTS OF
FEDERAL REGULATIONS AND UNFUNDED MANDATES ON STATE, LOCAL, AND TRIBAL ENTITIES
56 (2013) [hereinafter OIRA, 2012 REPORT].
156. See SCHWARTZ, supra note 107, at 46; Hahn, supra note 76. See infra Section IV.
157. Interestingly, the 2010 edition does not have the same provision. The 1981 version
was substituted for another rule creating the possibility of periodic review of agency
regulations by a legislative committee. NAT’L CONFERENCE OF COMM’RS ON UNIF. STATE
LAWS, supra note 107; SCHWARTZ, supra note 107, at 34, 37, 115.
158. Hahn, supra note 76, at 874, 876 (the study relied on interviews and survey data,
sometimes with only one response per state).
159. SCHWARTZ, supra note 107, at 86. Another study, published in 2016, found twenty-
five states that enacted requirements to review existing regulations, from 2006 through 2013.
See Stuart Shapiro, Debra Borie-Holtz & Ian Markey, Retrospective Review in Four States, 39
REG. 32 (2016) (narrating the recent history and reporting interview data on the adoption and
implementation of review of existing regulations in four states: Delaware, Nevada, Florida,
and Rhode Island).
Fall. 2016] REGULATION GOING RETRO 27
review in these systems is the passage of time from the initial date
of when an agency issues a regulation, with the selection of rules to
review including, in many states, all regulations.160
B. Ex Post RIA Beyond the U.S.
As with ex ante RIA, ex post RIA has also become a global
element of regulatory governance. In 2012 OECD published a new
set of recommendations from its Council on Regulatory Policy and
Governance.161 Along with recommending adoption of ex ante RIA,
it called for member countries to “[c]onduct systematic programme
reviews of the stock of significant regulation against clear defined
policy goals, including consideration of costs and benefits, to ensure
that regulations remain up to date, cost justified, cost effective and
consistent, and deliver the intended policy objectives.”162 It directed
countries to do this by “[m]aintain[ing] a regulatory management
system, including both ex ante assessment and ex post evaluation
as key parts of evidence-based decision making.”163
OECD reported that by 2011, twenty-four of its member
countries had mandatory periodic evaluation requirements of
existing regulations.164 Of the jurisdictions that have adopted ex
160. SCHWARTZ, supra note 107, at 115-123.
161. ORG. FOR ECON. CO-OPERATION & DEV., 2012 OECD RECOMMENDATION OF THE
COUNCIL ON REGULATORY POLICY GOVERNANCE 3 (2012) [hereinafter OECD, 2012
RECOMMENDATION] (building upon the 1997 OECD Report on Regulatory Reform, the 2005
Guiding Principles for Regulatory Quality and Performance, the 2005 APEC-OECD
Integrated Checklist for Regulatory Reform, and the 2009 Recommendation of the Council on
162. Id. at 4.
163. Id. at 6. OECD also said that “[t]he use of a permanent review mechanism should
be considered for inclusion in rules, such as through review clauses in primary laws and
sunsetting of subordinate legislation.” Id. at 12. The 2012 Recommendation builds on the
1995 Recommendation, in which no mention to ex post RIA existed. See ORG. FOR ECON. CO-
OPERATION & DEV., RECOMMENDATION OF THE COUNCIL ON IMPROVING THE QUALITY OF
GOVERNMENT REGULATION (1995). It also supplements the 2005 OECD Guiding Principles
for Regulatory Quality and Performance, which already suggested that member countries
“[a]ssess impacts and review regulations systematically to ensure that they meet their
intended objectives efficiently and effectively in a changing and complex economic and social
environment . . . .” ORG. FOR ECON. CO-OPERATION & DEV., OECD GUIDING PRINCIPLES FOR
REGULATORY QUALITY AND PERFORMANCE 14 (2005). Synthesizing 10 years of work on
regulatory reform, OECD published in 2011 a report identifying ex post evaluation as one of
the essential tools of regulatory policy alongside ex ante RIA. ORG. FOR ECON. CO-OPERATION
& DEV., REGULATORY POLICY AND GOVERNANCE: SUPPORTING ECONOMIC GROWTH AND
SERVING THE PUBLIC INTEREST (2011) [hereinafter OECD, SUPPORTING ECONOMIC GROWTH].
164. OECD, SUPPORTING ECONOMIC GROWTH, supra note 163, at 31. In 2015, OECD
reported that the number was 20 countries plus the European Commission see OECD, POLICY
OUTLOOK, supra note 39; see infra Section III, for observations about the methodology of the
28 JOURNAL OF LAND USE [Vol. 32:1
post RIA, three systems have a distinct level of sophistication and
detail: the U.K., the EU, and Australia.165
1. United Kingdom
Initially influenced by the U.S. RIA system, the U.K. IA system
eventually outpaced its American archetype in its ex post IA
framework.166 From the initial phase of development in the mid-
1980s, the U.K. IA system followed a dual approach, targeting both
the flow of new regulations and the stock of existing regulations.167
To address the first, the central government developed the
Compliance Cost Assessment (CCA).168 The CCA was later
transformed into the U.K. RIA with the launch of the Better
Regulation Initiative in 1998, and rebranded in 2007 as simply
“Impact Assessment.”169 On the side of existing regulations, reform
initiatives evolved from ad hoc to a sophisticated program of
evaluation and reform with two distinct components. One aimed at
reviewing the stock of existing regulations, often organized by
sectors of policy themes, with the purpose of reducing compliance
costs by repealing or improving rules (leading to the creation of an
ongoing program called Cutting Red Tape).170 The second
165. See infra Section III.
166. HOUSE OF COMMONS, LIFTING THE BURDEN, 1985, Cmnd. 9571, at 2-3 (UK)
[hereinafter HOUSE OF COMMONS, LIFTING THE BURDEN] (“[w]e have considered carefully the
work done in other countries, in particular in the U.S.A.”). Comparisons with the U.S.
regulatory policy appeared in the other reports of the time, such as in a 1986 White Paper:
The secret of the American experience undoubtedly lies in a more entrepreneurial
society. Yet that is not all. If we examine their economy we will see that individuals
are far less restricted if they wish to work for themselves, to start a new business,
or to employ people. They enjoy a freedom from regulations foreign to most
Europeans. Are they too free, or are we too regulated?
HOUSE OF COMMONS, BUILDING BUSINESSES . . . NOT BARRIERS, 1986, Cmnd. 9794, at 1 (UK)
[hereinafter HOUSE OF COMMONS, BUILDING BUSINESSES].
167. HOUSE OF COMMONS, LIFTING THE BURDEN, supra note 166, at 3.
168. Since the inception of CCA, it had some requirements aimed at enabling a future
review of a new proposed regulation to which the CCA applied. See HOUSE OF COMMONS,
BUILDING BUSINESSES, supra note 166, at 12 (stipulating as one of elements of the CCA the
clarification of “what steps can be taken to measure the effectiveness of the new regulation in
meeting its objectives?”). This was accompanied by a prescription regarding (partial)
monitoring, directing departments to “keep adequate records of the effects of regulations—
old as well as new—on business.” Id. at 72. The necessary integration between ex ante and
ex post IA was consolidated in further guidance documents. See UK HOUSE OF LORDS, THE
MANAGEMENT OF SECONDARY LEGISLATION, 2006, HL 149-I, at 13 (UK) (proposing a policy
cycle approach to integrated ex ante and ex post IA, and mentioning that officials should “use
[ex ante] Impact Assessment of the starting point for the [post-implementation] review”).
169. NAT'L AUDIT OFFICE, POST IMPLEMENTATION REVIEW OF STATUTORY INSTRUMENTS:
ANALYSIS OF THE EXTENT OF REVIEW BY GOVERNMENT DEPARTMENTS 7 (2009).
170. HOUSE OF COMMONS, LIFTING THE BURDEN, supra note 166, at 1; ROLF G. ALTER ET
AL., FROM RED TAPE TO SMART TAPE: ADMINISTRATIVE SIMPLIFICA TION IN OECD COUNTRIES
197 (2003); HOUSE OF COMMONS, BUILDING BUSINESSES, supra note 166, at 4; HOUSE OF
COMMONS, RELEASING ENTERPRISE, 1988, Cm. 512, AT 1 (UK). Starting in 1988, the
Fall. 2016] REGULATION GOING RETRO 29
component of reviewing rules included planned ex post
evaluations—known as post-implementation reviews (PIR).171 PIRs
would take place after a period of implementation of new proposed
regulations in order to measure their performance against goals and
decision criteria stipulated in ex ante IAs.172 After 2011, every
regulation imposing regulatory burdens on businesses or civil
society had to contain either a sunset or a review clause—both
triggering a PIR.173 The government has published detailed
guidelines covering the method PIRs must follow.174
There is also ex post evaluation of primary legislation in the
U.K.—called post-legislative scrutiny (PLS).175 In 2001, the
Regulatory Reform Act passed by Parliament required legislative
proposals to include a description of the “burdens which the existing
law . . . has the effect of imposing.”176 In 2004, the House of Lords
published a report acknowledging its co-responsibility in making
sure legislation was “fit for purpose.”177 For achieving this goal, it
proposed the adoption of PLS, which would be triggered after no
longer than six years of implementation, by a review clause included
in every piece of legislation.178 The same policy cycle approach that
government committed to adopting a more systematic review of the stock of existing
regulations, which would take place as a rolling annual program. Id. at 1; see HM GOV'T,
REDUCING REGULATION MADE SIMPLE 13 (2010) (UK) (mentioning the adopting of thematic
reviews); NAT'L AUDIT OFFICE, DELIVERING REGULATORY REFORM 29 (2011) (UK) (chronicling
the creation of the first online platform for ongoing review of existing rules based on public
input—initially called “Your Freedom”).
171. See ALTER ET AL., supra note 170.
172. Since at least 2003, the RIA guidelines mention the policy cycle approach and
underscore necessary links between ex ante and ex post RIA (such as the importance of
monitoring and the feedback of the resulting data into the “policy making process”).
REGULATORY IMPACT UNIT, BETTER POLICY MAKING: A GUIDE TO REGULATORY IMPACT
ASSESSMENT 29 (2003). These developments led OECD to consider the U.K. in the same year
a “primary example of the increasing international emphasis on regulatory quality.” ALTER
ET AL., supra note 170, at 196.
173. BETTER REGULATION TASK FORCE, REGULATION – LESS IS MORE 7 (2005). HM GOV'T,
supra note 170, at 11. In 2015, PIR gained statutory basis with the Small Business,
Enterprise, and Employment Act 2015, c. 26.
174. The guidelines are stipulated in the Magenta Book, which is applicable to
evaluation of other policy decisions and programs. HM TREASURY, THE MAGENTA BOOK:
GUIDANCE FOR EVALUATION 11 (2011).
175. U.K. DEP'T FOR BUS. INNOVATION & SKILLS, CLARIFYING THE RELATIONSHIP
BETWEEN POLICY EVALUATION, POST-LEGISLATIVE SCRUTINY AND POST-IMPLEMENTATIO N
176. Regulatory Reform Act 2001, ch. 6 § (2)(a). Periodic review of existing and future
reviews of new legislation were considered in the U.K. since the early 1990s. HOUSE OF LORDS,
PARLIAMENT & THE LEGISLATIVE PROCESS, Report, 2003-4, HL 173, at 8 (UK) (also
highlighting the integration between ex ante and ex post IA, with PLS being able to work as
“a means of assessing the utility of pre-legislative scrutiny). Id. at 43.
177. HOUSE OF LORDS, supra note 176, at 8 (also highlighting the integration between
ex ante and ex post IA, with PLS being able to work as “a means of assessing the utility of
pre-legislative scrutiny). Id. at 43.
178. Id. at 27, 44.
30 JOURNAL OF LAND USE [Vol. 32:1
influenced the design of PIR also inspired the framing of PLS by
Parliament, which saw PLS as a complementary tool to pre-
legislative scrutiny, and pre-legislative scrutiny as a facilitator of
PLS.179 Based on current guidelines, after three to five years after
enactment of an Act of Parliament, the department responsible for
implementation must submit a memorandum with the results of a
preliminary ex post assessment of its performance.180 Based on this
report, a committee from Parliament decides whether to conduct a
2. European Union
Since its early years, the EU RIA program also reflected a
concern for measuring the performance of existing regulations.182
Ex ante IA evolved in the European Commission from the Business
Impact Assessment adopted in 1986 under the U.K. Presidency and
modeled after the U.K. CCA.183 During the 1990s, the Commission
added new tools aimed at implementing ex post assessment of
existing regulation.184 When IA took its shape in the EU during the
early 2000s, it implicitly (and, later, explicitly) followed the policy
cycle model, with continual learning via integration of ex ante and
ex post IA.185 One decisive step in this direction coincided with the
2010 rebranding of the Better Regulation agenda to “Smart
Regulation.”186 Along with the explicit adoption of the policy cycle
approach, the Commission announced in the document an increased
179. LAW COMM'N, POST-LEGISLATIVE SCRUTINY, 2006, Cm. 6945, at 9 (UK).
180. U.K. CABINET OFFICE, GUIDE TO MAKING LEGISLATION 288 (2015).
181. Id. at 263.
182. See ANDREA RENDA, IMPACT ASSESSMENT IN THE EU: THE STATE OF THE ART AND
THE ART OF THE STATE 45 (2006).
183. Id. at 45-48. The influence of U.S. RIA was also a factor propelling the Better
Regulation movement in the EU. See Wiener, supra note 108, at 451.
184. Such as the SLIM project (Simplification of the Legislation on the Internal Market),
the creation of the BEST (Business Environment Simplification Task Force), in 1997; and the
creation of the Business Test Panel in 1998, with the aim of acting as a permanent body for
consultation of firms affected by EU regulations. ANDREA RENDA, LAW AND ECONOMICS IN
THE RIA WORLD: IMPROVING T HE USE OF ECONOMIC ANALYSIS IN PUBLIC POLICY AND
LEGISLATION 51 (2011).
185. EUR. COMM'N, Focus on Results: Strengthening Evaluation of Commission
Activities, at 7, SEC (2000) 1051 final (July 26, 2000) (conveying what would later be
consolidated and made mandatory as the “evaluate first principle” and mentioning that “[a]s
a rule, the preparation of proposals with budgetary and resource implications should include
information on: . . . lessons learned from any past intervention, . . . ; plan for monitoring and
evaluation during the course of the intervention”). In 2012, with the REFIT Program: “the
evaluation process could be designed alongside the policy itself with better monitoring and
reporting.” EUR. COMM'N, EU Regulatory Fitness, at 7, COM (2012) 746 final (Dec. 12, 2012)
[hereinafter EC, EU Regulatory Fitness].
186. EC, Smart Regulation, supra note 22.
Fall. 2016] REGULATION GOING RETRO 31
emphasis on ex post evaluation.187 The Commission, implementing
the new vision, followed the same strategy as adopted in the U.K.,
of two distinct programs of reviews of existing regulations.188 The
first focused on the flow of new rules by requiring a review or sunset
clause in every new proposed regulation, based on which ex post RIA
must take place after a period of implementation (planned ex post
evaluation); the second created a program of review of the stock of
Going beyond the U.K. model, the EU added two new features to
its RIA system. For the flow of new regulations (or regulatory
amendments), the EU added a requirement called the “evaluate
first principle,” which links the new rule to a prior ex post RIA of
the existing rule being revised.190 For the stock of existing rules, the
Commission created the Regulatory Fitness and Performance
Program (REFIT), which included two types of review: evaluation
and cutting red tape measures.191 Moreover, within the category of
evaluation, it created three different species: evaluation (of
individual rules), fitness checks (of a thematic body of rules), and
cumulative cost studies (usually focusing on a specific industry
Also in 2010, when the Commission formalized the adoption of
the REFIT Program, the European Parliament created its
187. Id. In this key white paper, the Commission said: “[s]mart regulation policy will
therefore attach greater importance than before to evaluating the functioning and
effectiveness of existing legislation.” Id. at 3.
188. See EC, Better Regulation Guidelines, supra note 109, at 30, 37; see also supra notes
176 and 177.
189. See id. (“[l]egislative proposals should also foresee when, how and on what basis
legislation will be evaluated in the future”); EUR. COMM'N, BETTER REGULATION TOOLBOX 260
(2015) [hereinafter EC, BETTER REGULATION TOOLBOX]. The requirement is less stringent
than in the U.K. system, where ex post RIA should take place in no longer than five years of
implementation. In contrast, Directorate-Generals (DG) have discretion in the EU to stipulate
when ex post RIA will take place. DEP'T FOR BUS. INNOVATION & SKILLS, supra note 109, at
31, 33. EC, BETTER REGULATION TOOLBOX 260. The decision about when to conduct an
evaluation must be made at the time of the proposal, i.e., early in the policy cycle: “[l]egislative
proposals should also foresee when, how and on what basis legislation will be evaluated in
the future.” By not specifying limits and guidance to this decision, the guidelines give
discretion to each DG to define when ex post RIA should take place. The guidelines also
mention the use of sunset clauses as a possibility (“may be used”). EC, Better Regulation
Guidelines, supra note 109, at 37.
190. Also differently from the U.K., the EU system describes not only the methodological
details of evaluations, but also the procedure that must be followed. See EC, Smart
Regulation, supra note 22, at 6; EC, Better Regulation Guidelines, supra note 109, at VI. There
is a specific guideline document covering the prescribed methods for evaluations. EUR.
COMM'N, EVALSED SOURCEBOOK: METHOD AND TECHNIQUES (2013).
191. EC, EU Regulatory Fitness, supra note 185. Within cutting red tape measures, the
REFIT Program includes two other sub-categories: studies and “legislative initiatives”—the
latter include “consolidation, simplification, recast, and codification.” EC, BETTER
REGULATION TOOLBOX, supra note 189, at 33.
192. EC, EU Regulatory Fitness, supra note 185.
32 JOURNAL OF LAND USE [Vol. 32:1
Directorate for Impact Assessment and European Added Value
(Directorate).193 The new Directorate mission was defined as
“enhanc[ing] Parliament’s capacity to undertake scrutiny and
oversight of the executive, particularly through ex ante and ex post
evaluation of EU legislation.”194 Now under the umbrella of the
European Parliamentary Research Service, one of the services
corresponding to units of the former Directorate is aimed at
“[e]valuating the results of existing European legislation.”195 The
Directorate issued succinct procedural guidelines for conducting
supplemental ex ante IA, but not for ex post evaluations.196
Beyond the U.S. and Europe, Australia gained the reputation of
a having a strong RIA system, considered by OECD as the member
country with “the most developed system [of ex post evaluation] in
both primary and subordinate legislation.”197 Ex ante RIA was
adopted in Australia in 1985.198 Three decades later, the Australian
RIA system had evolved to adopt a multi-track approach addressing
both the flow of new regulation and the stock of existing
regulations.199 One of the ten principles disciplining the work of
193. The creation of the Directorate for Impact Assessment and European Added Value
was the Parliament’s institutional solution to fulfilling its obligations under the 2003 Inter-
Institutional Agreement on Better Law-Making signed with the Commission and the Council.
See EUR. PARLIAMENT RESEARCH SERV., EUROPEAN PARLIAMENT WORK IN THE FIELDS OF EX-
ANTE IMPACT ASSESSMENT AND EUROPEAN ADDED VALUE: ACTIVITY REPORT FOR JUNE 2012 –
JUNE 2014, at 5 (2014).
194. EUR. PARLIAMENT RESEARCH SERV., IMPACT ASSESSMENT AND EUROPEAN ADDED
VALUE: DIRECTORATE C (2015), http://www.europarl.europa.eu/EPRS/Welcome_to_EPRS-
Dir_C-Mar2015.pdf [hereinafter EPRS, IMPACT ASSESSMENT AND EUROPEAN ADDED VALUE]
195. European Parliament Research Service, EUR. PARLIAMENT, http://www.europarl.
(last visited Jan. 21, 2017).
196. See EPRS, IMPACT ASSESSMENT AND EUROPEAN ADDED VALUE, supra note 194,
197. OECD, POLICY OUTLOOK, supra note 39, at 129.
198. AUSTL. PRODUCTIVITY COMM'N, IDENTIFYING AND EVALUATING REGULATION
REFORMS: PRODUCTIVITY COMMISSION RESEARCH REPORT XII (2011).
199. The principle is stated in a 2011 report by the Australian Productivity Commission:
[t]he regulatory system should ensure that new regulation and the existing ‘stock’
are appropriate, effective and efficient. This requires the robust vetting of proposed
regulation; ‘fine tuning’ of existing regulations and selecting key areas for reform.
… There is a range of approaches to reviewing existing regulation and identifying
necessary reforms. Some are more ‘routine’, making incremental improvements
through ongoing management of the stock; some involve reviews that are
programmed, and some are more ad-hoc. Designed for different purposes, the
techniques within these three categories can complement each other, through their
Id. at X; see Lorenzo Allio, Ex Post Evaluation of Regulation: An Overview of the Notion and
of International Practices, in REGULATORY POLICY IN PERSPECTI VE: A READER’S COMPANION
TO THE OECD REGULATORY POLICY OUTLOOK 2015 at 191 (2015).
Fall. 2016] REGULATION GOING RETRO 33
Australian policy makers stipulates that “[a]ll regulation must be
periodically reviewed to test its continuing relevance.”200 For new
regulations, a PIR must be completed within a period of no longer
than five years (in some cases, two years) of rule implementation.201
All regulatory changes with a substantial or widespread impact on
the Australian economy must undergo a PIR within five years of
implementation.202 The system also has links integrating ex ante
with ex post RIA, representing the same idea of a policy cycle: the
findings from a PIR that concludes that a regulatory change is
necessary are used to inform a decision about, and incorporated into
a new ex ante RIA of, a proposed regulatory revision.203 In addition,
an ex ante RIA should plan and make arrangements enabling a
future ex post evaluation.204 As to the stock of existing regulations,
the Australian system promotes “stock-takes” of regulation by
either relying on public input to select rules for review or choosing
a specific industry sector or theme to have all regulations
C. Ex Post EIA, CBA, and RA
EIA systems have seen some tentative requirements for ex post
evaluation, but in a less systematized form when compared to RIA.
In the U.S., for instance, NEPA directs CEQ to “review and appraise
the various programs and activities of the Federal Government in
the light of the policy set forth in title I of this Act.”206 Still, this duty
200. DEP'T OF THE PRIME MINISTER & CABINET, THE AUSTRALIAN GOVERNMENT GUIDE
TO REGULATION 2 (2014).
201. OECD, POLICY OUTLOOK, supra note 39, at 129.
202. OFFICE OF BEST PRACTICE REGULATION, POST-IMPLEMENTATION REVIEWS:
GUIDANCE NOTICE 2 (2016) (also mentioning that a PIR is required if a regulatory change
that is not minor nor “machinery in nature” had not been preceded by an ex ante RIA). If the
ex ante RIA prepared for a regulation is considered inadequate by the Office of Best Practice
Regulation (OBPR), a PIR must be completed within two years instead of five. Id. at 4.
203. Id. at 7.
204. OFFICE OF BEST PRACTICE REGULATION, USER GUIDE TO THE AUSTRALIAN
GOVERNMENT GUIDE TO REGULATION 7 (2016). One of the elements OBPR assesses when
overseeing ex ante RIA reports (RIS) is whether “it ha[s] a clear implementation and
evaluation plan.” Id. at 11. Implementation and evaluation also correspond to the topic of one
of the seven RIS questions agencies must address in ex ante RIAs, according to the guidelines.
See DEPT. OF THE PRIME MINISTER & CABINET, supra note 200, at 5.
205. AUSTL. PRODUCTIVITY COMM'N, supra note 198, at XXVIII (referring to this
approach as “Principle-based reviews strategies”).
206. 42 U.S.C. § 204(3). Also, NEPA requires the preparation and submission to
Congress of an annual Environmental Quality Report including “a review of the programs
and activities (including regulatory activities) of the Federal Government, the State and local
governments, and nongovernmental entities or individuals with particular reference to their
effect on the environment and on the conservation, development, and utilization of natural
resources; and . . . a program for remedying the deficiencies of existing programs and
activities.” Id. at §§ 201(4)-(5).
34 JOURNAL OF LAND USE [Vol. 32:1
has not been taken by CEQ as a mandate to conduct or require ex
post EIA.207 Neither NEPA nor the CEQ guidelines require agencies
to plan and conduct a future review of ex ante EIAs in light of new
information gathered from implementing the action that triggered
it.208 Nevertheless, the topic has been adopted by the EIA epistemic
community, generating significant literature on what is called
follow-up, post-implementation audit of EIAs, and adaptive
environmental assessment and management.209 Commentators
emphasize the role of ex post EIA audits in promoting learning, with
the potential for improving the accuracy of future predictions.210 The
main perception, however, is that lack of institutionalized follow-up
in EIA frameworks has been a systemic weakness, even though
some jurisdictions have incorporated follow-up requirements in
their EIA systems.211 One factor that may explain the relative lack
207. Under EO 13,563, CEQ implemented a series of NEPA pilots, but these were not
meant to conduct or promote ex post evaluation of each EIA. Rather, the pilots were meant to
(a) review CEQ’s own NEPA regulations, and (b) review the evaluation of EIAs by EPA under
Clean Air Act section 309. See COUNCIL ON ENVTL. QUALITY, COUNCIL ON ENVIRONMENTAL
QUALITY PLAN FOR RETROSPECTIVE REVIEW OF EXISTING REGULATIONS (2011). A partial move
in the direction of ex post EIA is the requirement issued by CEQ in 2010 and 2011,
emphasizing the need for better post-decision monitoring when an agency issues a Finding of
No Significant Impact (FONSI) and periodic reviews of categorical exclusions. Id. at 3.
208. See Farber, supra note 17.
209. On follow-up, see ANGUS MORRISON-SAUNDERS & JOS ARTS, ASSESSING IMPACT:
HANDBOOK OF EIA AND SEA FOLLOW-UP (2004) [hereinafter MORRISON-SAUNDERS & ARTS,
HANDBOOK]. On studies of post-implementation audits (not framed and treated under the
umbrella of follow-up measures), usually focusing on the accuracy of predictions contained in
ex ante EIAs, see Angus Morrison-Saunders & John Bailey, Exploring the EIA/Environmental
Management Relationship: Follow-up for Performance Evaluation (2000) (presented at IAIA
'00 Back to the Future conference, June 19-23, Hong Kong), http://researchrepository.
murdoch.edu.au/2443/; see also Ronald Bisset, Problems and Issues in the Implementation of
EIA Audits, 1 ENVTL. IMPACT ASSESSMENT REV. 379, 380 (1980) (identifying the ex post
measurement of accuracy against ex ante predictions as one of the approaches to assessing
the effectiveness of EIA); Ralf Buckley, Environmental audit: review and guidelines, 7 ENVTL.
PLAN. L.J. 127 (1990); Ralf Buckley, Auditing the Precision and Accuracy of Environmental
Impact Predictions in Australia, 18 ENVTL. MONIT. ASSESSMENT 1-23 (1991) (providing an
example of results from multi-projects audit, assessing the accuracy of predictions in light of
new information from monitoring). On adaptive environmental assessment and management,
see INT’L INST. FOR APPLIED SYSTEMS ANALYSIS, ADAPTIVE ENVIRONMENTAL ASSESSMENT AND
MANAGEMENT (Crawford S. Holling ed., 1978); Craig R. Allen et al., Adaptive Management for
a Turbulent Future, 92 J. ENVTL. MGMT. 1339, 1339-45 (2011); Bernard T. Bormann et al.,
Adaptive management, in ECOLOGICAL STEWARDSHIP: A COMMON REFERENCE FOR
ECOSYSTEM MANAGEMENT 505-34 (W.T. Sexton, A.J. Malk, R.C. Szaro, N.C. Johnson 1999
eds.); DEP’T OF THE INTERIOR, ADAPTIVE MANAG EMENT: THE US DEPARTMENT OF THE
INTERIOR TECHNICAL GUIDE (2009).
210. Ben Dipper, Monitoring and Post-auditing in Environmental Impact Assessment: A
Review, 41 J. ENVTL. PLAN. MGMT. 731, 733 (1998); see also Daniel A. Farber, Bringing
Environmental Assessment into the Digital Age, in TAKING STOCK OF ENVIRONMENTAL
ASSESSMENT (Jane Holder and Donald McGillivray ed., 2007) (advocating collection and
analysis of past EIAs).
211. See MORRISON-SAUNDERS & ARTS, HANDBOOK, supra note 209, at 66, 158, 238-239
(describing the requirements for monitoring and auditing of EIA in Canada, the Netherlands,
Western Australia, Hong Kong, and Finland, and for regional planning in the U.K.); Dipper,
Fall. 2016] REGULATION GOING RETRO 35
of adoption of ex post EIA is that the typical policy decision to which
EIA applies—usually involving a project, e.g., building a highway or
permitting the installation of a facility—means that making
changes after it has been constructed is often costly or moot.212
There tend to be few autonomous and systematic ex post
requirements of CBA for infrastructure projects; these analyses
tend to be reviewed, if at all, through ex post RIA and ex post EIA.
One prominent example of ex post CBA applied to an entire
regulatory program—a kind of programmatic ex post RIA—is the
requirement included in section 812 of the 1990 Amendments to the
Clean Air Act (codified at 42 U.S.C. § 312), requiring EPA to conduct
and report to Congress a comprehensive analysis of the impact of
major programs under the CAA on the public health, economy, and
environment.213 The law requires EPA to consider “costs, benefits
and other effects associated with compliance” with different
standards defined under the authority delegated by the CAA.214
Ex post reviews of RA sometimes occur through statutory
periodic reviews of the scientific basis for regulatory standards (as
for national ambient air quality standards and drinking water
quality standards, discussed above in Section III.A). One
requirement that comes close to an ex ante-ex post system for RA is
the post-market evaluation of drugs by the U.S. Food and Drug
Administration (FDA). In 2007, Congress amended the Federal
Food, Drug, and Cosmetic Act to “enhance the post-market
authorities of the [FDA].”215 The Act, as amended, provides for the
possibility of a post-market surveillance system based on
performance standards as “rigorous as the ones already developed
for premarket review,”216 and possibly leading to an obligation to
conduct and periodically report on post-approval studies or clinical
supra note 210, at 735 (reflecting on the consequences of lack of mandatory monitoring
requirements by stating that from the project developer’s point of view, it really does not
matter if predictions are accurate: the developer will suffer no consequences, and all that the
developer needs ex ante is educated guesswork). There is some degree of overlap in the
literature between follow-up and adaptive environmental management. Usually, follow-up
measures are referred to as including monitoring, audit, ex post evaluation, and management
activities. See MORRISON-SAUNDERS & ARTS, HANDBOOK, supra note 209, at 3; Jos Arts, Paula
Caldwell & Angus Morrison-Saunders, Environmental Impact Assessment Follow-up: Good
Practice and Future Directions—Findings from a Workshop at the IAIA 2000 Conference, 19
IMPACT ASSESSMENT PROJECT APPRAIS. 175-185 (2001).
212. For a list of EISs submitted to EPA with the description of the policy decision to
which they apply, see Environmental Impact Statement (EIS) Database, U.S. ENVTL. PROT.
AGENCY, https://cdxnodengn.epa.gov/cdx-enepa-public/action/eis/search (last visited Jan. 21,
213. Clean Air Act, Title VIII, § 812(a), 104 Stat. 2399 (1990) (current version at 42
U.S.C. § 312 (2004)).
215. Federal Food, Drug, and Cosmetic Act, 121 Stat. 823 (2007).
216. Id. at § 910(6).
36 JOURNAL OF LAND USE [Vol. 32:1
trials of a drug.217 As a result of the post-approval study, FDA might
require safety labeling changes, or other risk evaluation and
IV. THE STATE OF PLAY OF RETROSPECTIVE REVIEW
IN U.S. ENVIRONMENTAL LAW
Observing the formal adoption of retrospective review or ex post
RIA requirements is not the same as assessing their actual
implementation. There can be a gap between adoption and
implementation. Discussing the diffusion of ex ante RIA, Claudio
Radaelli called attention to the idea that RIA can sometimes travel
lightly and serve different justification logics.218 The result can be a
common RIA “bottle” but containing different “wines”—or “even no
wine at all.”219 This assessment has been confirmed by OECD and
other studies of diffusion of ex ante RIA.220 What is true about ex
ante RIA can also be said about ex post IA requirements. Different
institutional structures in different jurisdictions make it even
harder to assess and compare what has been truly implemented. As
with ex ante RIA, different systems of ex post RIA of existing
regulations might be designed to work with varied institutions and
toward different goals.221
In the case of RIA, it becomes further complicated to compare
systems and assess implementation due to the terminological
imprecision of the word “review”—often used in normative
requirements, oversight bodies, agency reports and academic
literature to address different types of regulatory initiatives. As
mentioned in Section III, in some systems, as in the U.K., EU, and
217. Id. at § 901(o)(3)(B). The goal of post-approval studies or clinical trials are: “(i) [t]o
assess a known serious risk related to the use of the drug involved; (ii) [t]o assess signals of
serious risk related to the use of the drug; (iii) [t]o identify an unexpected serious risk when
available data indicates the potential for a serious risk.”
218. Claudio M. Radaelli, Diffusion Without Convergence: How Political Context Shapes
the Adoption of Regulatory Impact Assessment, 12 J. EUR. PUB. POL'Y 924, 924 (2005).
220. The adoption-implementation gap issue has been mentioned by several studies.
OECD, POLICY OUTLOOK, supra note 39, at 103; Radaelli, supra note 218; Fabrizio De
Francesco, Claudio M. Radaelli & Vera E. Troeger, Implementing Regulatory Innovations in
Europe: the Case of Impact Assessment, 19 J. EUR. PUB. POL'Y 491 (2012); RENDA, supra note
182, at 81. In 2011, OECD itself had reported after surveying the implementation of RIA in
member countries: “[e]x ante impact assessment remain a weak area. Nearly all countries are
struggling to establish the process so that it is taken seriously by officials and politicians;”
but also that “[t]here is growing awareness that this is a key tool.” OECD, SUPPORTING
ECONOMIC GROWTH, supra note 163, at 112, 122.
221. Radaelli, supra note 218, at 929 (mentioning that RIA might perform different
functions in different countries, and could correspond to the “rebranding” of preexisting and
partial tools, with purposes that overlap with the most recent approach explicitly directed to
reviewing existing regulations—e.g., administrative burden reduction).
Fall. 2016] REGULATION GOING RETRO 37
Australia, there exist different programs within the broad category
of reviews of existing regulation. In others, different variations can
be conflated under just a single label.222 “Regulatory review” can
mean revision of an existing rule, i.e., a proposed policy change or
repeal, with little or no analysis of the past performance of the rule
being “reviewed.” In these cases, there might be little or no
hindsight and learning. On the other hand, “regulatory review” can
also mean a comprehensive ex post evaluation of an existing rule,
comparing expected to realized impacts—positive and adverse—
before any policy revision is considered.
Where the literature suggests an adoption-implementation gap
in ex post RIA, this gap might reflect divergent understandings of
what is being implemented under the heading of “ex post RIA” or
“evaluation” or “retrospective review.” As noted above, Coglianese
and OECD have criticized current practice as incomplete and
inadequate.223 After trying to make sense of what kinds of “review”
are actually occurring, the general impression from secondary
sources is one of low or spotty implementation of ex post RIA.224
Governments have only partially implemented retrospective review,
focusing on revisions to individual existing rules with the goal of
222. OECD’s recommendation on ex post RIA attempts to measure more than just one
tool or approach to assessing and improving existing regulations, as it calls for member
countries to “[c]onduct systematic programme reviews of the stock of significant regulation .
. . .” OECD, 2012 RECOMMENDATION, supra note 161, at 4 (Recommendation 5). Even under
Recommendation 1, it specifies that countries should “[m]aintain a regulatory management
system, including both ex ante impact assessment and ex post evaluation as key parts of
evidence-based decision making.” Id. at 6. Also, under Recommendation 3, it suggests
“[c]o-ordinating ex post evaluation for policy revision and for refinement of ex ante methods.”
Id. at 9. Finally, in Recommendation 5 itself, it uses “review” and “revision” to mean different
practices, suggesting that “[t]he methods of Regulatory Impact Analysis should be integrated
in programmes for the review and revision of existing regulations.” Id. at 12.
223. See Coglianese, supra note 42 (finding that retrospective review “is today where
prospective analysis was in the 1970s: ad hoc and largely unmanaged.”); OECD, POLICY
OUTLOOK, supra note 39, at 123 (“ex post evaluation by [U.S.] federal agencies remains patchy
224. Allio, supra note 199, at 196, 221, 240 (“post-implementation evaluations have not
yet been systematically implemented in most countries;” “ex post evaluation has remained
relatively side-lined;” “systematic ex post evaluation is less common and the number and
performance of such reviews are rarely measured systematically;” “very few OECD countries
have actually deployed the tool systematically”); ORG. FOR ECON. CO-OPERATION & DEV.,
ASSESSING PROGRESS IN THE IMPLEMENTATION OF THE 2012 RECOMMENDATION OF THE OECD
COUNCIL ON REGULATORY POLICY AND GOVERNANCE 9 (2013) (“few countries are actually
doing it systematically;” “some countries have undertaken pilot projects in ex-post
assessment, which have not yet been transformed into a systematic approach”); CHRISTIANE
ARDNT ET AL., 2015 INDICATORS OF REGULATORY POLICY AND GOVERNANCE: DESIGN,
METHODOLOGY AND KEY RESULTS 7 (2015) (“[c]ountries are less advanced in ex post
evaluation where only a few countries systematically evaluate the impact of their regulations
ex post”); OECD, POLICY OUTLOOK, supra note 39, at 112 (“[o]verall, however, very few OECD
countries have actually deployed the ex post evaluation systematically”).
38 JOURNAL OF LAND USE [Vol. 32:1
cutting administrative burden (red tape).225 Often it is difficult to
determine from survey responses what, if anything, has really been
A. Prior Assessments of the Practice
of Ex Post RIA in the U.S.
National audit offices have played a major role in investigating
and promoting implementation and compliance with RIA systems.
In the U.S., the Government Accountability Office (GAO) has since
the mid-1990s repeatedly assessed compliance of federal agencies
with different ex post RIA requirements, among other analytical
225. Allio, supra note 199, at 200. (“few countries assess whether underlying policy goals
of regulation have been achieved, whether any unintended consequences have occurred and
whether there is a more efficient solution to achieve the same objective . . . . A more frequent
practice in OECD countries is partial ex post assessments focusing exclusively on regulatory
burdens”); OECD, SUPPORTING ECONOMIC GROWTH, supra note 163, at 9 (“[a] more frequent
practice in OECD countries is partial ex-post assessment, focusing exclusively on regulatory
burdens”); OECD, POLICY OUTLOOK, supra note 39, at 113-14 (“[m]ost countries have had ex
post evaluations based on administrative burden reduction with an assessment of compliance
cost using the standard cost model;” “[t]he survey results confirm the findings by Allio (2015)
that countries focus on partial ex post assessment of regulatory burdens and rarely assess
whether underlying policy goals of regulation have been achieved”).
226. This is a problem of survey-based studies on both adoption and implementation of
ex post RIA. Sometimes well-intended studies contribute to the lack of clarity on what
exactly—and at what level—is being implemented. The OECD 2015 Regulatory Policy
Outlook study is based on survey data, including responses from government officials. It
proposed to measure, among other variables, systematic implementation of ex post evaluation
by OECD member countries. The research design is vulnerable, though, in the validity and
accuracy of its findings, because it is not clear if the answers truly measure different aspects
of ex post evaluation as defined by OECD itself to mean an analysis of how a regulation has
performed. The questionnaire on ex post evaluation uses at least seven different terms
referring to ex post evaluation (“ex post evaluations,” “ex post evaluations by RIA,” “major
review,” “regular reviews to examine complaints and other problems,” “internal review an
evaluation by the regulator,” “reviews of existing regulation,” and “ex post evaluations of
existing regulation”). The answers to questions using different terms are aggregated. This is
particularly worrisome, as the report itself admits that “[t]he experience of conducting ex post
evaluation varies considerably across countries and also domestically across different
Ministries or agencies within governments” and that “[t]his is in part due to the different
interpretations and understanding of what ex post evaluation means,” and “there is the
opportunity to develop a broader understanding of ex post evaluation among OECD
countries.” OECD, POLICY OUTLOOK, supra note 39, at 112, 113. Accuracy is also an issue in
the study, because responses came from government officials (the study claims that evidence
was gathered to verify the answers, but without specifying what evidence it gathered and
examined), and also due to the vague character of answers. For example, to the question “do
subordinate regulations include automatic evaluation requirements? (3C4b_S)” the responses
could be “for some subordinate regulations;” to the question “have ex-post evaluations of
existing subordinate regulations been undertaken in the last three years? (3C1-S)” the
response could be “yes, some” or “yes, frequently.” The same issue is seen in prior
questionnaires used to measure adoption of ex post RIA systems; see ORG. FOR ECON. CO-
OPERATION & DEV., OECD REGULATORY INDICATORS QUE STIONNAIRE 2008 22 (2008)
(“periodic ex post evaluation of existing regulation,” “review,” “modif[ication of] specific
Fall. 2016] REGULATION GOING RETRO 39
evidence-based tools used by federal agencies.227 In its analysis,
GAO also addressed methodological and institutional challenges
facing agency practice of ex post RIA explaining its findings.228
While trying to measure the level and quality of implementation of
ex post RIA requirements, GAO faced the issue of imprecise
meaning of “review,” leading to different practices by each agency.229
Evidence in the reports reveal a track record of either full lack of
compliance or the simpler type of regulatory “review,” i.e., cost-
cutting revision of existing rules without evidence of a formal ex post
analysis of their past performance—something also perceived by
other oversight bodies, such as OIRA.230 OIRA emphasized the
227. In its first reports of the kind, the focus of