Making Evidence-Based Organizational Decisions in an Uncertain World

Full text

Making
Evidence-Based
Organizational
Decisions
in
an
Uncertain
World
$
Denise
M.
Rousseau
Contents
Organizational
decisions
are
similar
to
but
different
from
other
decisions
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
2
Organizational
practices
can
repair
decision
biases
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
2
Implications
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
6
Decision
Processes
Matter:
Three
Evidence-Based
Decision
Processes.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
7
Routine
Decisions
Involving
Known
Knowns
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
8
Non-Routine
Decisions
with
Known
Unknowns.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
9
Novel
Decisions
Involving
Unknown
Unknowns
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
10
Caveats.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
11
Selected
bibliography.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
11
Make
everything
as
simple
as
possible,
but
not
simpler.
Albert
Einstein
This
article
deals
with
decision
making
practices
that
work.
Based
on
decades
of
decision
research,
it
is
a
primer
on
how
to
make
better
organizational
decisions.
Organizational
decision
makers
are
often
familiar
with
the
well-established
research
on
cognitive
biases
in
individual
judgment
(e.g.,
seeking
informa-
tion
to
confirm
existing
beliefs,
relying
on
easily
available
information),
expertly
described
in
Daniel
Kahneman’s
Think-
ing
Fast
and
Slow.
That
research
focuses
on
the
biases
of
non-
experts
making
simulated
decisions–— and
indicates
how
diffi-
cult
it
is
for
individuals
to
reduce
their
own
cognitive
biases.
Organizational
decisions
are
somewhat
different.
They
take
place
in
a
social
setting
where
participants
can
hold
different
information
and
perspectives,
and
under
the
right
circum-
stances,
can
act
in
ways
that
actually
help
de-bias
organiza-
tionaldecisions.The focusof this articleis on the organizational
decision
making
processes
that
can
reduce
errors
in
judgment,
improve
the
quality,
relevance
and
understanding
of
the
infor-
mation
considered,
and
better
suit
the
challenging
environ-
ments
in
which
organizational
decisions
are
made.
In
support
of
the
movement
toward
more
evidence-based
practice
(EBP)
in
management,
I
lay
out
the
what,
why
and
how
of
making
good
organizational
decisions–— using
pro-
cesses
scientific
research
shows
increase
the
odds
of
success.
The
need
for
a
better
informed
approach
to
decisions
in
organizations
is
straightforward:
The
best
available
manage-
ment
research
suggests
that
around
half
of
organizational
decisions
fail
to
achieve
their
goals.
This
failure
tends
to
be
tied
to
managers
who
rush
to
judgment,
impose
their
pre-
ferred
solutions,
fail
to
confront
the
politics
behind
the
decision,
ignore
uncertainty,
downplay
risks,
and
discourage
search
for
alternatives.
At
the
same
time,
contemporary
organizations
have
many
notable
successes.
NASA
put
a
man
on
the
moon.
Smallpox
has
been
eradicated.
Farmers
in
third
world
countries
use
cellphones
to
obtain
crop
prices.
Half
of
organizational
decisions
may
well
succeed,
and
science
shows
some
key
reasons
why.
This
article
aims
to
promote
better
organizational
deci-
sions
by
offering
evidence-based
insights
to
improve
decision
quality.
I
describe
how
the
research
evidence
can
inform
Organizational
Dynamics
49
(2020)
100756
$
This
paper
was
written
at
the
Rockefeller
Foundation’s
Bellagio
Center
and
supported
by
an
H
J
Heinz
II
professorship.
Jessica
Cooke,
Trish
Greenhalgh,
Mari
Kuraishi,
Maria
Tomprou
and
Glen
Whyte
provided
insights
in
its
writing.
Available
online
at
www.sciencedirect.com
ScienceDirect
jo
u
rn
al
h
om
ep
ag
e:
ww
w.els
evier.c
o
m/lo
c
ate/o
rg
d
yn
https://doi.org/10.1016/j.orgdyn.2020.100756
0090-2616/©
2018
The
Author(s).
Published
by
Elsevier
Inc.
This
is
an
open
access
article
under
the
CC
BY-NC-ND
license
(http://
creativecommons.org/licenses/by-nc-nd/4.0/).

organizational
decision
making
and
specify
the
building
blocks
of
successful
decisions:
Use
of
organizational
de-
biasing
practices
and
appropriate
decision
processes.
This
article
develops
three
key
ideas.
Organizational
decisions
are
similar
to
and
yet
different
from
individual
decisions
in
ways
that
introduce
new
biases
and
at
the
same
time
can
compensate
for
them.
Limitations
to
human
judg-
ment
and
biases
in
organizational
decisions
can
be
overcome
to
some
extent
by
using
a
bundle
of
six
evidence-based
practices
useful
for
de-biasing
decisions
made
with
others.
These
de-biasing
practices
underpin
three
decision
pro-
cesses
a
well-managed
organization
needs
in
its
repertoire
in
order
to
master
effective
decision
making.
ORGANIZATIONAL
DECISIONS
ARE
SIMILAR
TO
BUT
DIFFERENT
FROM
OTHER
DECISIONS
Human
beings
have
cognitive
limits
that
introduce
error
into
judgments
and
decisions.
Organizations
by
virtue
of
their
structuring
and
the
broader
environments
in
which
they
operate
tend
to
introduce
their
own
peculiar
decision
chal-
lenges.
At
the
same
time,
organizational
decisions
differ
from
personal
decisions
and
from
the
professional
decisions
many
practitioners
make
(Table
1).
These
differences
will
be
important
to
the
kinds
of
processes
that
help
organizations
improve
their
decision
practices.
Organizational
decision
makers
work
with
and
through
others.
Unlike
many
clinical
psychologists
and
accountants,
managers
don’t
practice
solo.
Not
only
responsible
for
their
own
decisions,
managers
often
are
also
accountable
for
the
decisions
others
make,
responsibilities
associated
with
gaps
in
both
information
and
understanding
that
create
uncertainty
and
concern
with
control.
All
judgments
and
decisions
are
affected
by
the
roles
people
play.
Ask
physicians
whether
they
prefer
to
focus
their
efforts
on
treatment
or
disease
prevention,
let’s
say
for
diabetes
or
asthma,
and
they
are
likely
to
express
different
preferences
when
asked
about
their
patients
or
their
grand-
children.
Organizations
place
people
in
multiple
roles
simul-
taneously.
The
teacher,
nurse
or
engineer
who
is
also
a
department
head
has
different
information
and
beliefs
than
the
new
teacher,
nurse
or
engineer
just
starting
out.
Orga-
nizational
decision
makers
tend
to
pay
attention
to
certain
information
because
it
fits
their
role
and
avoid
other
infor-
mation
if
it
seems
irrelevant
or
exposes
them
to
risk.
The
manager
who
says,
“Don’t
tell
me
about
the
problem
because
then
I’ll
have
to
deal
with
it,”
is
trying
to
manage
her
accountability.
Differences
in
roles,
interests,
and
dif-
ferences
in
levels
of
authority
or
power
make
organizations
political
environments
with
myriad
informal
and
formal
ways
to
constrain,
block,
or
enable
decisions.
People
with
differ-
ent
organizational
roles
possess
both
different
interests
and
information,
one
reason
for
creating
task
forces
made
up
of
people
from
varied
backgrounds.
Organizational
decision
makers
face
considerable
uncer-
tainty
and
complexity
in
making
decisions.
Managers
often
don’t
get
timely
feedback
on
the
outcomes
of
their
deci-
sions,
unlike
the
surgeon
who
makes
post-surgery
rounds
to
find
out
how
patients
are
doing.
Organizational
decisions
often
have
lots
of
stakeholders,
inside
and
outside
the
organization,
prompting
a
tendency
for
managers
to
avoid
paying
attention
to
those
on
whom
less
information
is
avail-
able
or
to
work
around
those
who
make
a
decision
more
complicated.
Organizational
decision
makers
can
face
highly
dynamic
situations.
Missing
information
and
difficulty
inter-
preting
data
are
common
in
dynamic
environments,
making
appropriate
courses
of
action
difficult
to
determine.
In
contrast,
an
accountant
is
likely
to
know
what
information
is
needed
to
close
the
books
and
where
to
find
it.
Last,
organizational
decision
makers
often
face
a
diverse
array
of
decisions
made
concurrently,
rather
than
similar
decisions
over
and
over.
Not
surprisingly
then,
there
tend
to
be
fewer
frameworks,
checklists
and
decision
supports
to
guide
orga-
nizational
decisions
than
in
professions
like
nursing
and
medicine
or
engineering
and
accountancy.
For
all
these
reasons,
organizational
decisions
differ
in
important
ways
from
personal
decisions
and
the
decisions
many
professionals
make.
ORGANIZATIONAL
PRACTICES
CAN
REPAIR
DECISION
BIASES
The
second
key
idea
is
that
using
appropriate
organizational
decision
practices
helps
overcome
both
well-established
indi-
vidual
biases
and
biases
peculiar
to
organizations.
Research
demonstrates
a
variety
of
cognitive
and
organizational
repairs
Table
1
Organizational
Decisions
Differ
from
Individual
Decisions
Qualities
Organizational
Decision
Makers
(managers,
teams,
work
groups)
Individual
Decision
Makers
(homelife,
private
professional
practice)
Nature
of
Work
Work
with
and
through
others
Personal
choices/solo
practitioners
Political
Influences
Strong
political
pressures
can
block
or
constrain
decisions
Some
but
fewer
political
pressures
Accountability
Accountable
for
decisions
by
self
and
others
Accountable
for
own
decisions
Decision
Types
Make
many
kinds
of
decisions,
often
concurrently
Make
fewer
kinds
of
decisions
Stakeholder
Diversity
Many
stakeholders
affect
and
affected
by
decisions
Fewer
stakeholders
affect
and
affected
by
decisions
Levels
of
Uncertainty
Greater
uncertainty
due
to
missing
information,
difficulty
interpreting
situations,
and
environmental
change
Uncertainty
can
vary,
often
low
to
moderate
Decision
Supports
Typically
few
decision
supports
and
protocols
Professionals
often
have
decision
supports
and
protocols
(e.g.,
checklists)
2
D.M.
Rousseau/Organizational
Dynamics
49
(2020)
100756

that
can
help
reduce
the
effects
of
biases.
One
repair
sug-
gested
by
a
study
from
the
University
of
Chicago
applies
to
the
multilingual–— hinking
in
a
foreign
language!
A
native
English
speaker
who
speaks
Korean
as
a
second
language
is
likely
to
be
more
deliberate
in
his
thinking
in
Korean
and
less
affected
by
emotion
than
he
is
when
thinking
in
his
native
tongue.
But
organizations
don’t
need
to
hire
linguists
to
de-bias
how
decision
makers
think.
Populating
labs,
teams
and
committees
with
people
from
different
backgrounds
can
help
do
that.
Working
in
diverse
groups
prompts
more
deliberate
thinking
by
exposing
people
to
information
that
challenges
existing
beliefs.
Although
it
is
very
difficult
for
individuals
to
reduce
their
own
biases
despite
careful
training
and
personal
effort,
making
decisions
in
a
social
setting
can
reduce
bias
and
improve
decision
quality.
The
basic
idea
is
this:
it’s
easier
to
recognize
biases
in
other
people
than
in
ourselves.
A
common
individual
bias
is
valuing
personal
experience
over
relevant
organizational
data
or
scientific
perspectives.
Peo-
ple
typically
use
less
objective
or
external
information
in
making
individual
decisions
than
they
think
they
do
and
base
many
judgments
on
unquestioned
personal
beliefs.
The
manager
who
favors
candidates
who
went
to
the
same
university
as
he
did
may
not
recognize
the
reason
for
his
preference.
However,
making
the
hiring
decision
in
a
group
using
de-biasing
practices
can
surface
and
overcome
that
manager’s
biased
assumptions.
There
are
six
organizational
biases
shown
to
lead
to
failed
decisions.
Each
comes
with
targetted
repairs
known
to
be
effective
(Table
2).
These
repairs
reflect
findings
from
the
work
of
many
scholars
including
Chip
Heath,
David
Larrick,
Paul
Nutt,
and
others.
These
repairs
are
synergistic
and
work
together
to
improve
organizational
decisions.
1
Solving
the
wrong
problem
!
Start
with
search
and
get
the
right
decision
frame
No
shortcuts
are
apparent
for
taking
time
at
the
outset
of
the
decision
process.
–Paul
C.
Nutt
Organizational
decisions
arise
from
problems,
opportu-
nities
or
crises,
and
pretty
much
in
that
order.
None
of
these
may
be
well
defined.
At
the
outset
of
a
decision,
it
is
critical
to
figure
out
the
“decision
frame,”
that
is,
the
decision
makers’
conception
of
the
problem
to
solve,
or
the
oppor-
tunity
or
crisis
to
address.
A
good
start
begins
with
asking
questions–— lots
of
questions.
Asking
questions
kicks
off
the
deliberate
search
for
both
information
and
understanding
(not
the
same
thing!),
gathering
intelligence
to
understand
the
need,
opportunity
or
crisis.
Active
search
informs
the
rest
of
the
decision
process,
making
it
easier
to
figure
out
how
best
to
make
the
decision
and
the
actions
to
take.
In
contrast,
shortcuts
at
the
start
often
translate
into
frustration
and
failure
later.
Stopping
the
search
process
too
soon
often
leads
to
solving
the
wrong
problem,
settling
on
a
decision
frame
before
really
understanding
the
situation.
Questioning
and
search
can
come
to
an
abrupt
halt
when
managers
impose
their
own
take
on
a
problem.
The
execu-
tive
who
begins
a
meeting
extolling
the
virtues
of
bold
action
can
skew
the
discussion
that
follows.
The
situation
is
no
different
than
the
police
arresting
their
first
suspect
without
looking
for
further
evidence
of
who
else
might
have
done
the
crime.
Imposing
an
idea
generally
leads
to
worse
decisions
than
first
taking
the
time
to
make
sense
of
the
underlying
problem.
Uncertainty
can
seem
intolerable
and
even
respon-
sible
managers
often
assume
it
away,
acting
as
if
the
expected
results
are
clear
and
guaranteed.
In
a
rush
to
make
things
concrete,
the
search
for
relevant
information
can
be
suppressed
and
uncertainty
glossed
over.
The
resultant
lack
of
information
and
insight
leads
to
solving
the
wrong
problem
or
pursuing
a
questionable
opportunity
with
limited
results.
Search
can
also
be
short-circuited
when
the
organization
members
involved
think
so
much
alike
that
they
fail
to
recognize
important
aspects
of
the
problem.
As
we
will
discuss
below,
attention
to
the
many
stakeholders
a
decision
might
have
can
help
broaden
problem
conceptualization
and
avoid
the
decision
frame
becoming
concrete
too
soon.
Deliberate
search
can
help
maintain
an
open
mind,
per-
mitting
a
discovery
process
to
unfold.
This
critical
initial
phase
shapes
everything
that
happens
later.
Search
can
clarify
the
nature
of
the
decision,
what
needs
to
be
improved,
what
results
are
wanted,
the
support
needed,
and
the
stakeholders
affected.
At
the
start,
its
often
better
to
be
unsure
about
the
problem
(or
comfortable
with
uncer-
tainty)
rather
than
superconfident.
The
key
idea
here
is
that
uncertainty
is
inevitable
in
organizational
decisions
and
the
Table
2
Organizational
Repairs
for
Better
Decisions
Organizational
Biases
Repairs
1.
Solving
the
wrong
problem
(Idea-led
not
problem-driven)
Taking
time
at
the
start
to
ask
diagnostic
questions
Engage
in
active
search
processes
2.
Ignoring
politics
affecting
the
process
(Sponsor
biases,
pet
projects)
Addressing
the
politics
of
the
decision
Legitimate
a
de-biased
decision
focus
3.
Considering
just
one
option
(Pet
project,
gut
feeling)
Entertaining
multiple
options
4.
Focusing
on
a
single
outcome
(Narrow
view
of
success)
Using
several
criteria
for
decision
success
and
effectiveness
5.
Letting
narrow
interests
dominate
(Stakeholders
ignored)
Broadening
the
kinds
of
stakeholders
considered
and
involved
6.
Relying
on
easily
available
information
(Stories
and
“hippos”)
Expanding
sources
of
information
to
include
scientific
evidence,
organizational
data,
expert
judgment
and
stakeholder
concerns
D.M.
Rousseau/Organizational
Dynamics
49
(2020)
100756
3

start
of
the
decision
is
the
place
to
begin
managing
it.
Uncertainty
is
inherent
in
organizational
decisions
and
best
confronted
early.
Managers
typically
prefer
to
focus
on
concrete
evidence
and
distance
themselves
from
informa-
tion
that
raises
doubt.
Instead,
it’s
better
to
manage
the
uncertainty
or
even
try
to
reduce
it–— by
gathering
facts,
reflecting
or
waiting
for
more
information–— rather
than
assuming
it
away.
There
is
no
shortcut
to
a
thoughtful
start
to
the
decision
process,
regardless
of
urgency
or
the
resources
poured
into
a
problem.
Spending
more
time
assessing
the
problem
means
less
time
required
to
solve
it.
In
tighter-time
frames,
rehear-
sals
during
slow
periods
can
help
build
decision
making
capacity
(example,
flight
crews
anticipating
bad
weather
ahead).
Or
decision
makers
can
buy
time,
as
in
the
case
of
a
cockpit
crew
who
goes
into
a
holding
pattern
until
they
have
figured
out
how
to
solve
a
problem.
If
time
really
cannot
be
bought,
the
decision
may
require
simpler
rules
that
take
critical
action
and
avoid
harms.
And
the
question
to
be
asked
later
is
“why
did
we
wait
so
long
to
prepare
for
this
deci-
sion?”
2
Pretending
It’s
Not
Political
!
Address
the
politics
and
there’s
always
politics
Where
fear
is
present,
wisdom
cannot
be.
–Lacantius
Organizations
are
political
places.
When
subordinates
and
consultants
are
reluctant
to
challenge
how
their
bosses
or
clients
frame
a
decision,
politics
are
at
work.
Politics
refers
to
explicit
or
implicit
use
of
personal
power
and
resources
to
influence
others.
Organizational
politics
can
be
a
positive
practice
when
integrative
goals
are
attained,
and
a
negative
practice
when
personal
or
local
interests
take
precedence.
Politics
cannot
be
ignored
in
making
a
good
decision.
Politics
can
be
overt
or
subtle,
known
by
all
or
invisible.
Negative
politics
can
take
many
forms:
Pushing
a
leader’s
pet
project,
ignoring
sensitive
information,
involving
some
sta-
keholders
while
ignoring
others
are
some
examples.
Positive
politics
entail
being
aware
of
a
decision’s
social
implications
such
as
how
different
interest
groups
interpret
and
prioritize
issues.
Negative
politics
and
good
decision
making
don’t
go
together.
Good
decisions
are
made
in
psychologically
safe
settings
where
political
interests
can
be
addressed
directly.
Attention
to
politics
is
part
of
a
good
start
to
the
decision
process,
raising
issues
regarding
the
framing
of
the
decision.
The
initial
framing
may
reflect
the
decision
sponsor’s
infor-
mation
limits,
biases
and
interests.
A
more
broadly
useful
frame
can
result
from
paying
attention
to
the
array
of
interests
a
decision
reflects.
Negative
politics
can
sometimes
be
turned
into
positive
.
One
way
to
address
political
issues
is
to
broaden
attention
early
to
the
concerns
of
additional
stakeholders,
including
implementers,
users,
and
others
ultimately
affected
by
the
decision.
The
folks
in
the
room
where
the
decision
is
being
made,
literally
or
figuratively,
will
inevitably
bring
their
own
biases.
But
appeals
to
broader
interests
and
superordinate
goals,
or
arguments
regarding
push
back
from
stakeholders,
can
prompt
the
decision
to
be
framed
in
a
more
legitimate
way.
Taking
time
to
develop
and
legitimate
the
decision’s
frame
broadens
the
problem
conceptualization
and
ulti-
mately
helps
generate
more
options
and
better
choices.
Poli ti cs
reflects
the
moral
component
of
decision
making,
giving
rise
to
the
interests
attended
to
in
making
a
decision.
Negative
politics
can
disproportionately
impose
the
costs
of
paying
high
executive
compensation
upon
lower
level
employ-
ees
via
paycuts.
Or,
it
can
kick
the
can
down
the
road,
leaving
unresolved
problems
from
a
decision
to
somebody
else.
Positi ve
politics
can
broaden
a
decision’s
appeal
and
distribute
its
benefits
and
costs
in
a
principled
fashion,
as
in
the
case
where
costs
are
shared
and
start
at
the
top.
Politi cs
skews
the
infor-
mation
sought in search
and
in evaluating alternatives.
As
such
a
high
quality
decision
process
addresses
politics
upfront
and
subsequently
as
needed.
And,
if
politics
discourage
agreement
on
a
decision,
some
of
the
best
outcomes
can
result
from
using
field
tests
or
experiments
to
evaluate
the
effects
of
competing
approaches
or
resolve
disputes
regarding
likely
outcomes.
3
Considering
just
one
alternative
!
Generate
several
distinct
alternatives
The
preference
for
a
“quick
fix”
is
motivated
by
prag-
matics
and
by
fear.
–Paul
C.
Nutt
A
key
predictor
of
decision
success
is
whether
more
than
one
alternative
is
considered.
In
decisions
where
managers
impose
their
views
regarding
a
preferred
solution,
the
search
for
alternatives
is
cut
short.
“One
and
done”
is
not
enough.
Considering
only
a
single
alternative
ignores
relevant
infor-
mation.
Such
a
quick
fix
can
seem
timely
and
efficient,
but
outcomes
tend
to
be
poorer.
Where
only
one
alternative
is
considered,
discussion
of
this
(non)
choice
become
conflated
with
support
for
its
expected
results.
If
the
single
alternative
emerges
early,
like
a
preferred
IT
solution
or
adoption
of
a
leader’s
pet
idea,
it
can
remain
unchanged
as
the
search
process
shuts
down,
resulting
in
little
new
information
regard-
ing
the
problem
or
the
preferred
solution’s
costs
and
benefits.
Identifying
multiple
alternatives
leads
to
gathering
more
information
regarding
expected
differences
in
results.
Doing
so
also
can
expand
the
array
of
criteria
or
success
indicators
considered,
addressing
another
bias
discussed
below.
One
pitfall
with
multiple
options
is
the
tendency
to
fall
back
on
intuition
rather
than
analysis
in
trying
to
quickly
weigh
the
alternatives.
But
analysis,
by
which
I
mean
systematic
atten-
tion
to
each
alternative’s
pros
and
cons,
is
important
to
improving
the
odds
of
success.
An
added
benefit,
evaluating
multiple
alternatives
side
by
side
can
reduce
the
reliance
on
self-interest
on
the
part
of
decision
makers
and
expand
their
consideration
of
other
stakeholders.
4
Using
only
one
criterion
of
success!Use
multiple
criteria
“Not
everything
that
can
be
counted
counts,
and
not
everything
that
counts
can
be
counted.”
—
Albert
Einstein
Evidence
indicates
that
decision
success
is
more
likely
when
decision
makers
consider
multiple
criteria
or
outcomes.
4
D.M.
Rousseau/Organizational
Dynamics
49
(2020)
100756

Decision
makers
often
consider
only
one
criterion
of
success
or
decision
effectiveness.
Short-term
outcomes,
particularly
fi-
nancial
ones,
tend
to
grab
more
attention
than
longer-term
qualitative
outcomes.
But
how
many
things
should
this
deci-
sion
really
accomplish?
Having
several
criteria
in
mind
informs
search
processes
and
deepens
the
evaluation
of
alternatives.
Importantly,
the
concerns
of
more
than
one
interest
group
are
better
represented
when
multiple
criteria
are
used.
Consid-
eration
of
both
multiple
criteria
and
several
alternatives
increases
the
complexity
of
the
decision
but
also
its
likelihood
of
success.
Complexity
means
that
an
analysis
of
pros
and
cons
is
needed
in
order
to
make
the
decision–— not
just
gut
feel.
Consider
the
decision
whether
to
outsource
an
organization’s
training
activities,
build
in-house
training
capabilities,
or
hybrids
of
the
two.
When
simultaneously
considering
several
alternatives
such
as
these,
decision
makers
are
less
likely
to
fall
back
on
intuition
and
more
likely
to
deliberate
on
the
relative
advantages
and
disadvantages
of
each,
resulting
in
more
mindful,
less
biased
considerations
of
opportunities
and
costs.
5
Interests
of
a
few
dominate
while
other
stakeholders
are
ignored
!
Reflect
the
broad
array
of
stakeholders
“What
would
the
best
organization
in
the
world
do?”
–Paul
O’Neill
Aside
from
politiking,
the
interests
of
a
few
participants
can
predominate
when
their
biases
and
limited
information
form
the
basis
of
the
decision.
Actively
seeking
information
regarding
the
concerns
of
other
stakeholders
helps
improve
decision
outcomes
through
better
understanding
of
both
the
problem
and
issues
of
implementation.
Importantly,
the
subjective
opinions
of
users,
like
clients
and
customers,
are
associated
with
better
outcomes
than
relying
on
the
opinions
of
sponsors
and
decision
participants.
Decisions
also
can
have
long
reaches
and
some
stakeholders
ultimately
affected
by
a
decision
may
not
initially
come
to
mind.
Participants
also
may
not
share
the
same
views
of
a
deci-
sion’s
other
stakeholders.
Thus
it
may
help
to
map
the
potential
array
of
internal
(employees,
departments,
the
board)
and
external
stakeholders
(users,
regulators,
com-
munities).
To
prompt
decisions
reflecting
broad
ethical
stakeholder
concerns,
Paul
O’Neill,
former
CEO
of
Alcoa,
regularly
asked
the
question
“what
would
the
best
company
in
the
world
do?”
A
broad
stakeholder
frame
in
conceptualizing
the
pro-
blem
contributes
to
increased
quality
of
information
search
and
identification
of
important
criteria
for
evaluating
alter-
natives.
It
can
also
aid
implementation
by
identifying
con-
flicts,
and
helping
to
develop
more
integrative
solutions.
For
example,
the
hiring
of
a
new
university
HR
director
was
at
first
seen
as
an
issue
related
only
to
the
concerns
of
non-
academic
staff,
since
academic
department
heads
handled
faculty
recruitment
and
performance
management.
After
talking
with
deans,
faculty
and
other
administrators
at
the
department-level,
the
selection
team
realized
there
was
another
angle
to
hiring
the
kind
of
HR
director
needed,
one
that
affected
staff
and
faculty
collaborations
in
entre-
preneurial
activities.
Since
various
stakeholders
interpreted
the
position’s
needs
from
their
own
perspectives,
the
task
force
re-examined
its
initial
take
on
the
decision,
ultimately
reframing
it
as
a
search
for
an
HR
professional
with
broader
experience
supporting
teams.
Attending
to
a
decision’s
multiple
stakeholders
also
pro-
vides
the
opportunity
to
anticipate
how
the
decision’s
ben-
efits
and
costs
might
be
appropriately
distributed.
It
raises
important
questions
regarding
who
benefits,
who
is
harmed,
and
what
other
outcomes
might
be
possible.
Since
the
real
interests
of
stakeholders
often
differ
from
the
stated
objec-
tives
of
a
problem,
there
are
two
issues.
One
is
the
imple-
mentation
challenges
associated
with
ignoring
end
users.
This
classic
problem
is
found
in
philanthropy
where
the
donor’s
notion
of
how
to
solve
a
problem
may
conflict
with
the
community’s
values
or
priorities.
Stakeholders
can
hold
very
different
mental
models
of
both
the
problem
and
appropriate
action,
which
can
be
important
information
not
readily
available
at
the
outset
of
the
decision
process.
The
second
issue
is
that
uneven
allocation
of
benefits
and
harms
across
stakeholders
raises
moral
concerns.
Stake-
holder
interests
sometimes
may
need
to
be
balanced
across
several
decisions
when
they
cannot
be
addressed
in
a
single
decision.
6
Relying
on
easily
available
information
!
Use
several
types
of
information
and
appraise
quality
“What
would
I
really
like
to
know?”
–An
evidence-based
practitioner
Reliance
on
easily
available
information
means
being
swayed
by
the
salient
opinion
of
the
decision’s
sponsor,
a
manager’s
own
experiences
or
the
intuitions
of
the
folks
in
the
room,
without
attention
to
other
sources
of
relevant
information.
The
aim
of
EBP
is
systematic
use
of
the
best
available
information
from
multiple
sources
in
order
to
improve
decision
outcomes.
The
bias
of
easily
available
information
can
be
reduced
by
searching
for
different
kinds
of
relevant
evidence,
and
then
appraising
how
good
it
is
(its
reliability,
validity,
consistency
and
relevance)
in
helping
make
the
decision.
Tw o
key
ideas
here
are:
1)
using
relevant
high-quality
evidence
tends
to
lead
to
better
decisions
than
easily
available
information,
and
2)
quality
evidence
comes
in
several
different
forms.
Consider
the
case
of
a
small
non-profit
OwnHome
with
ten
employees
whose
mission
is
to
help
inner-city
residents
build
the
financial
stability
to
support
homeownership.
The
social
workers
employed
by
OwnHome
had
high
turnover,
com-
plaining
of
being
stretched
too
thinly.
The
executive
direc-
tor’s
first
attributed
the
turnover
problem
to
bad
hires
made
by
the
previous
director.
As
turnover
continued
among
their
replacements,
she
recognized
that
there
was
more
to
the
problem.
At
an
all-staff
retreat,
issues
surfaced
regarding
the
executive
director’s
failure
to
structure
strategy
imple-
mentation,
resulting
in
a
lack
of
critical
resources
and
supports
for
helping
clients.
At
the
retreat,
organization
members
developed
a
plan
to
get
a
better
understanding
regarding
effective
strategy
implementation
and
alliance
building.
The
plan
called
for
some
members
to
talk
with
faculty
at
a
local
university,
some
to
search
the
local
library
D.M.
Rousseau/Organizational
Dynamics
49
(2020)
100756
5

data
bases
for
research
articles
on
the
topics,
and
others
to
benchmark
other
local
nonprofits.
Three
weeks
later,
they
pooled
their
information.
They
noted
areas
of
agreement
across
the
evidence
they
had
found,
in
particular,
the
need
to
focus
on
key
activities
that
ensured
the
flow
of
important
resources.
As
a
result,
they
developed
a
set
of
success
criteria
and
possible
alternative
approaches.
Finally,
they
decided
on
a
strategy
of
building
alliances
with
local
orga-
nizations
in
order
to
help
clients
gain
and
keep
stable
employment,
manage
their
personal
finances,
and
access
credit.
Having
a
specific
framework
that
all
staff
understood
provided
a
roadmap
both
the
director
and
staff
could
follow
in
their
day
to
day
activities.
Periodic
meetings
monitored
progress
in
alliance
building
and
plans
were
updated
as
the
organization
learned
what
worked.
In
this
example,
multiple
sources
of
evidence
were
important
to
problem
recognition,
sensemaking,
developing
an
action
plan,
and
its
execution.
Reliance
on
easily
available
information
is
often
reflected
in
the
sway
that
tradition
(past
practices)
and
authority
(powerful
people)
have
on
the
decision
frame
and
alterna-
tives
considered.
At
first,
HomeOwn’s
executive
director
thought
the
problem
was
with
people,
not
recognizing
how
her
own
day-to-day
focus
on
fire
fighting
kept
her
from
overseeing
strategy
implementation.
In
many
organizations,
easily
available
information
takes
the
form
of
personal
experience,
stories,
and
what
we
in
EBP
have
come
to
call
“the
HIPPO,”
that
is,
the
highest-paid
person’s
opinion.
Generally
speaking
where
available
information
is
ambigu-
ous,
organizational
decision
makers
fall
back
on
their
own
subjective
judgment
as
did
the
executive
director
of
Home-
Own–— at
first.
Subjective
judgment
means
intuitive
infer-
ences,
hunches
and
guesses
where
the
readily
available
opinions
of
others,
sponsors,
participants,
experts
or
end
users
are
interpreted
without
making
an
effort
to
evaluate
how
trustworthy
or
unbiased
the
information
might
be.
Reliance
on
subjective
judgment,
guesses
and
intuition
has
been
found
to
coincide
with
a
lot
of
confidence
and
wishful
thinking,
less
effective
than
analysis
of
information
from
multiple
sources.
So
in
general,
lots
of
confidence
is
not
necessarily
a
good
thing
where
decisions
are
concerned.
The
antidote
to
over-reliance
on
readily
available
infor-
mation
is
an
active
search
process
such
as
HomeOwn
engaged
in,
gathering
pertinent
information
from
multiple
sources.
Sources
include
scientific
evidence
relevant
to
the
problem
to
be
solved
(e.g.,
on-line
search
for
research
on
strategy
implementation);
organizational
data
if
they
can
be
found
(e.g.,
in
company
records)
or
created
(e.g.,
social
workers
sharing
experiences
at
the
staff
retreat);
expert
judgment
(e.g.,
faculty
specializing
in
strategy
implementa-
tion);
and
of
course
stakeholder
concerns
as
people
who
affect
or
are
affected
by
the
decision
(in
a
sense
the
social
workers
provide
both
organizational
and
stakeholder
data).
A
key
question
to
ask
in
the
search
for
relevant
evidence
is
what
would
you
like
to
know?
It
is
easy
to
overlook
what
is
missing
because
our
minds
fill
in
the
missing
information
with
hunches
that
seem
like
facts.
Broadening
the
knowledge
base
applied
to
a
decision
permits
testing
taken-for
granted
assumptions.
It
also
can
raise
awareness
on
the
part
of
the
decision
maker
about
how
much
uncertainty
a
decision
may
really
involve.
In
the
case
of
HomeOwn,
the
decision
makers
learned
from
faculty,
literature
search,
and
local
contacts
that
strategy
implementation
in
their
small
organization
might
need
to
be
fleshed
out
little
by
little
in
order
to
learn
by
doing
what
works
and
what
doesn’t.
All
evidence
used
in
decision
making
needs
to
be
eval-
uated
for
quality.
We
don’t
take
HIPPOs
at
face
value.
Nor
do
we
assume
that
because
a
scientific
study
makes
a
claim
that
this
claim
is
actually
true.
A
critical
mindset
is
the
essence
of
evidence-based
decision
making
and
attention
to
quality
evidence
is
key
to
making
good
decisions.
Information
can
overwhelm
attention,
so
it
makes
sense
to
identify
what
information
is
worth
paying
attention
to.
Judgment
can
be
valuable
where
the
individuals
involved
have
years
of
experi-
ence
with
a
specific
domain
or
technical
problem
and
have
had
the
opportunity
get
feedback
on
their
judgment’s
accu-
racy
and
effectiveness.
Think
of
the
engineer
who
has
twenty
years
of
experience
building
bridges.
Expert
judg-
ment
stems
from
recognizing
patterns
held
in
memory
from
training
and
prior
experience,
one
reason
why
10,000
hours
of
practice
is
a
rule
of
thumb
for
becoming
an
expert.
Expert
opinion
tends
to
be
less
reliable
where
the
domain
is
broad,
success
harder
to
judge,
and
personal
experience
limited.
Here
think
of
the
manager
involved
in
two
organizational
change
projects
over
the
past
five
years.
Where
opportu-
nities
to
practice
and
get
accurate
feedback
are
limited,
expertise
is
much
more
difficult
to
acquire
from
experience
alone.
The
quality
of
organizational
data
depends
on
how
well
they
suit
the
purpose
for
which
they
are
used.
If
it’s
to
spot
a
crisis,
data
need
to
be
timely–— so
efforts
to
get
informed
early
reports
pay
off.
If
it’s
to
provide
insight
into
a
perfor-
mance
problem,
data
needs
to
provide
context–— incidence,
scope,
changes
over
time
and
comparisons
with
others
to
understand
magnitude
and
trends.
If
it’s
information
about
stakeholder
concerns,
data
need
to
be
representative
and
sufficiently
rich
to
be
interpretable.
The
quality
of
scientific
data
depend
on
the
nature
of
the
question
asked.
If
it’s
a
question
of
cause-effect
(what
works?),
controls
are
needed
like
comparison
groups
and
before/after
measures.
If
it’s
a
question
about
how
different
groups
might
respond
to
a
new
program,
a
cross-sectional
survey
can
provide
good
information
(how
do
they
react?).
In
general,
the
highest
quality
scientific
evidence
is
found
in
systematic
reviews
or
meta-analyses,
where
information
from
a
large
number
of
studies
can
help
cancel
out
the
limitations
of
single
studies.
An
added
benefit
exists
from
using
multiple
sources
of
information.
Arguments
based
on
experience
or
personal
values
can
seem
tainted
by
self-interest
and
make
other
participants
and
stakeholders
wary.
In
contrast,
using
scien-
tific
evidence,
organizational
data
and
stakeholder
perspec-
tives,
especially
from
users,
can
appear
more
objective
and
less
value-laden,
raising
the
odds
of
success
and
acceptance
of
the
decision.
All
decisions
ignore
some
evidence,
but
the
point
is
to
pay
attention
to
evidence
more
systematically
so
that
available
quality
evidence
is
used.
Implications
Organizational
decisions
have
better
odds
of
success
if
the
above
six
de-biasing
processes
are
used.
These
processes
are
synergistic
and
work
well
together.
In
combination
they
help
organizational
decision
makers
do
something
humans
often
6
D.M.
Rousseau/Organizational
Dynamics
49
(2020)
100756

find
tough
to
do:
confront
and
manage
incomplete
informa-
tion,
ambiguity,
and
uncertainty.
To
be
uncertain
means
to
be
unsure
because
some
pertinent
conditions
are
unknown.
Managers
typically
prefer
to
focus
on
concrete
evidence
and
distance
themselves
from
information
that
raises
doubt.
Using
organizational
practices
that
surface
biases
and
over-
come
them
provides
a
repair.
A
manager’s
own
self-serving
biases
can
lead
him
or
her
to
avoid
using
such
de-biasing
processes,
so
it
is
good
to
make
these
processes
routine
so
that
participants
can
practice.
In
doing
so,
the
organization
can
become
a
more
psychologically
safe
place
for
people
to
make
mindful
decisions.
Responding
appropriately
to
inevitable
uncertainties
is
an
important
capacity
in
contemporary
organizations.
The
de-
biasing
practices
described
here
build
this
capability
by
helping
decision
makers
approach
organizational
problems
with
an
open
mind,
be
politically
aware,
identify
appropri-
ate
goals
and
options,
and
search
for
and
appraise
relevant
evidence.
Using
the
set
of
six
practices
can
promote
positive
politics
in
an
organization
by
helping
make
decisions
more
integrative.
Such
practices
help
employees
feel
psychologi-
cal
safe,
and
in
turn,
improve
their
ability
to
constructively
de-bias
each
other.
These
six
practices
can
be
used
as
a
standalone
to
improve
decision
quality.
But
if
your
appetite
is
wetted
for
evidence-based
decision
processes,
consider
now
taking
this
capability
to
the
next
level.
Decision
Processes
Matter:
Three
Evidence-
Based
Decision
Processes
“Our
Age
of
Anxiety
is,
in
great
part,
the
result
of
trying
to
do
today's
job
with
yesterday's
tools
and
yesterday's
concepts.”
–Marshall
McLuhan
The
third
key
idea
is
that
decision
processes
matter
and
different
processes
are
suited
for
different
kinds
of
deci-
sions.
In
conventional
decision
making,
a
decision
process
often
is
a
set
of
actions
that
begins
with
identification
of
an
issue
or
need
and
results
in
an
action.
In
more
complex
decision
making,
a
process
can
be
iterative
“feedback
on
the
go,”
as
decision
makers
take
action
in
order
to
learn
what
happens
and
act
again
based
on
those
results.
In
this
section
we
discuss
deciding
how
to
decide.
This
meta-capability
involves
learning
the
decision
processes
appropriate
for
different
kinds
of
decisions
and
then
identifying
the
kind
of
the
decision
to
be
made.
Why
do
decision
processes
matter?
Organizational
deci-
sion
makers
often
can’t
rely
on
results
to
know
if
they’ve
made
a
“good”
decision.
Results
can
take
months
or
years
to
materialize.
In
the
meantime,
decision
makers
change
jobs
and
even
organizations.
This
lack
of
feedback
makes
learning
what
works
difficult.
It
makes
it
difficult
for
managers
and
other
organizational
decision
makers
to
learn
from
experi-
ence
alone.
Contrast
this
was
the
expertise
acquired
over
time
by
a
virtuoso
violinist
or
a
veteran
accountant
well-
versed
in
the
tax
code,
each
of
whom
gets
regular
feedback
on
their
performance.
The
absence
of
feedback
regarding
many
organizational
decisions
is
one
reason
why
experi-
enced
managers
aren’t
always
better
managers.
Fortunately,
research
by
Frank
Yates,
Paul
Nutt,
Karl
Weick
and
Kathy
Sutcliffe
among
others
shows
that
the
process
behind
a
decision
provides
a
reasonable
indicator
of
the
likely
quality
of
its
outcomes.
Decision
process
refers
to
the
steps
used
in
making
a
decision.
Using
the
right
process
is
important
in
order
to
make
good
use
of
the
information
in
hand
or
otherwise
accessible,
and
the
infor-
mation
that
could
be
created
through
action
and
experi-
mentation.
I
will
describe
three
evidence-based
decision
processes
that
correspond
to
three
commonplace
decision
situations.
There
are
large
bodies
of
scientific
research
for
each
deci-
sion
process.
Developing
the
capacity
to
use
the
right
deci-
sion
process
appropriately
is
part
of
making
good
decisions
(Table
3):
Routine
Decisions–— characterized
by
stability
and
clear
cause-effect
understanding.
The
repeated
conditions
characterizing
these
decisions
permit
effective
prac-
tices
to
be
identified
and
improved
on
over
time
(e.
g.,
how
best
to
hire
new
call
center
employees
or
conduct
regular
staff
meetings
for
effective
communi-
cation
and
team
building).
Non-Routine
Decisions–— complicated
situations
where
no
one
has
full
information,
but
relevant
information
already
exists
somewhere.
For
these
decisions,
a
well-
structured
decision
process
can
be
conducted
by
involving
knowledgeable
others
in
evidence
gathering
and
apprai-
sal,
developing
alternatives,
etc.
(e.g.,
whether
to
im-
plement
a
new
educational
program
or
re-locate
a
facility).
Truly
Novel
Decisions–—
where
critical
information
re-
quired
for
the
decision
does
not
exist
because
historical
evidence
is
irrelevant.
For
decisions
made
under
truly
novel
conditions,
evidence
must
be
generated
by
action,
learning
and
experimentation
(e.g.,
designing
home
health
devices
for
the
elderly
living
alone;
starting
a
business
in
an
emerging
market).
These
three
kinds
of
decisions
cover
a
wide
variety
of
organizational
problems,
opportunities
and
crises.
Although
they
don’t
cover
every
possible
decision,
many
other
deci-
sions
are
actually
combinations
of
these
three
types.
An
example
of
a
multi-type
decision
is
the
Canadian
health
system’s
implementation
of
the
then-new
nurse
practitioner
(NP)
role.
This
decision
included
all
three
types
of
decision
process.
Since
the
United
States
had
implemented
NPs
a
few
years
before,
a
fair
amount
of
technical
information
was
available
allowing
routine
and
non-routine
decision
making.
At
the
same
time,
the
day-to-day
implications
of
this
new
role
were
unknown
until a
task
force workingwith
the
first cohort
of
Canadian
NPs
began
experimenting
with
the
new
mode
of
practice,
the
learning
by
doing
characteristic
of
novel
situa-
tions.
As
this
example
suggests,
by
understanding
how
these
three
distinct
decision
processes
work,
we
build
capacity
to
better
manage
many
different
decisions.
Uncertainty
takes
several
different
forms.
Uncertainty
can
be
foreseeable
or
unforeseeable.
It’s
foreseeable
uncer-
D.M.
Rousseau/Organizational
Dynamics
49
(2020)
100756
7

tainty
that
we
cannot
know
what
next
year’s
gas
prices
will
be.
But
it’s
unforseeable
uncertainty
what,
where,
and
when
the
next
influenza
pandemic
will
be.
Uncertainty
can
be
technical
or
procedural
in
that
the
decision
maker
lacks
understanding
of
how
to
make
something
happen
(no
clear
cause-effect
connections),
for
example,
how
to
go
about
building
a
patient
information
database
that
serves
the
needs
of
physician
services,
the
billing
department,
and
medical
researchers.
Uncertainty
may
be
political
(ambig-
uous
goals
and
tenuous
relationships)
in
that
no
clear
course
of
action
is
known
that
all
key
stakeholders
will
accept,
as
in
the
case
of
a
police
department
seeking
better
community
relations.
Uncertainty
can
be
due
to
environmental
change,
which
may
vary
from
somewhat
predictable
to
totally
unpre-
dictable.
Environmental
change
can
involve
physical,
eco-
nomic,
technological
and
broader
social
forces,
sometimes
in
combination,
as
exemplified
by
the
multi-faceted
nature
of
climate
change.
Developing
the
skills
to
confront
uncer-
tainty
is
important
to
making
good
organizational
decisions.
People
have
feelings
about
uncertainty,
and
may
be
avoidant
or
curious.
Experience
with
alternative
decision
processes
can
make
dealing
with
uncertainty
easier
by
helping
decision
makers
respond
constructively
to
the
particular
uncertainty
they
face.
“Knowns”
refer
to
domain
knowledge
available
to
deci-
sion
makers.
They
can
be
technical,
where
the
situation
is
well-understood,
like
how
to
post
an
internal
position.
Or
knowns
can
be
political,
where
interests
and
differences
among
parties
are
well-established,
as
in
some
labor-man-
agement
disputes.
In
either
case,
use
of
domain
knowledge
can
improve
the
quality
of
the
decision.
In
contrast,
in
problems
with
many
“unknowns,”
critical
information
does
not
exist
and
has
to
be
created,
as
in
the
case
of
responses
to
hacking
or
exploring
business
opportunities
in
an
emerging
market.
Unknowns
are
addressed
through
learning
by
doing
and
experimentation.
Routine
Decisions
Involving
Known
Knowns
“Ask
whether
checklists
are
used;
if
the
answer
is
no
or
not
forthcoming,
choose
a
different
hospital.”
Gerd
Gigernezer
Table
3
Three
Kinds
of
Evidence-Based
Decision
Processes
Processes
Routine
Non-Routine
Novel
Uncertainty
Known
Knowns
Technical
rationality–— info
is
in
hand
Known
Unknowns
Missing
info
can
be
found
and
tests
can
be
run
Unknown
Unknown
No
historical
information
exists
so
must
learn
through
action
Exemplars
Onboarding
checklist.
Loan
guidelines.
Protocol
for
managing
staff
meetings.
Administrative
team
addresses
office
re-location.
Task
force
manages
r
a
restructuring.
Army
peacekeeping
in
Haiti.
Police-community
relationship
building.
Board
investigates
strategic
actions
regarding
climate
change
Key
actions
Follow
Appropriate
Decision
Process
Understand
how
work
is
done.
Identify
key
practices
usinglmultiple
evidence
sources.
Build
protocol
(formal
or
mental)
with
user
input
&
share
via
training.
Periodically
evaluate/
update/
redesign.
Identify
need
or
opportunity.
Diagnose
problem
to
solve.
Address
politics
&
stakeholders.
Search/generate
alternatives.
Evaluate
alternatives.
Develop
plan.
Execute.
Assess
&
feedback.
Novel
conditions:
Scenario
building.
Run
multiple
small
simultaneous
experiments.
Act
on
best
outcome,
evaluate,
learn.
(Action
learning
cycle.)
Use
After
Action
Reviews
to
sustain
learning.
Novel
event:
Quick
improvisation,
repeated
actions
in
response
to
feedback
until
acceptable
result
reached.
Success
Themes
Agile
Not
Rigid.
No
Good
Evidence!No
Guideline
Positive
Politics!
Psychological
Safety
&
Decision
Quality.
Reflection!Gains
Experiment/Improvise/
Learn.
Act!Check!Learn!Act!
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Learning
Master
each
process
to
improve
decision
outcomes
Periodically
evaluate
and
update
protocol.
Train
people
to
understand
its
principles
so
they
can
adapt
or
edit
it
appropriately.
Revisit
a
set
of
decisions
periodically.
Then,
reflect
upon
and
share
learnings.
Cause/effect
only
understood
after
the
fact,
if
ever.
Advance
practice
in
improvisation
and
using
AARs
improves
capacity
to
learn
by
doing.
8
D.M.
Rousseau/Organizational
Dynamics
49
(2020)
100756

Routine
decisions
involve
situations
where
the
causal
connections
are
clear:
If
you
do
A,
you
will
get
B.
These
decisions
often
entail
pulling
appropriate
responses
out
of
memory.
If
you
are
calling
a
meeting,
you
probably
have
a
mental
script
of
what
to
do:
Send
a
scheduling
email,
book
a
room,
and
lay
out
an
agenda.
Since
routine
decisions
are
based
on
past
information,
one
concern
from
an
evidence-
based
practice
perspective
is
overlooking
ways
to
improve.
Am
I
really
using
an
effective
way
of
organizing
and
running
a
meeting?
Three
important
reasons
for
routinizing
repeat
decisions
are
to
get
good
results
more
consistently,
find
ways
of
improving
results,
and
to
free
up
time
to
make
decisions
that
aren’t
routine
better.
A
key
aspect
of
making
effective
routine
decisions
is
to
gather
facts
on
how
the
decision
is
currently
being
made,
evaluate
its
outcomes,
and
then
redesign
the
process
per-
iodically
to
have
good
outcomes
more
consistently.
Since
Frederick
Ta y l o r,
we
have
known
that
to
improve
a
process
you
first
have
to
standardize
it.
If
you
run
every
meeting
in
an
ad
hoc
way
you
really
cannot
know
what
works
best.
How
exactly
the
routine
is
to
be
established
can
vary
consider-
ably
depending
on
user
experience
and
the
situations
deci-
sion
makers
face
in
applying
the
routine.
The
decision
could
be
made
using
a
mental
checklist
or
script
the
decision
maker
follows
or
a
physical
checklist,
protocol
or
even
an
algorithm.
A
variety
of
evidence
sources
help
in
creating
effective
routines
or
decision
protocols.
For
managerial
decisions,
we
first
need
to
find
out
what
managers
actually
are
doing.
We
find
out
what
is
typically
done,
what
is
inconsistently
done
and
what
important
factors
are
in
danger
of
being
over-
looked.
Next,
organizational
data
on
outcomes
of
these
decisions
can
provide
a
baseline
that
we
hope
to
improve
upon
by
creating
a
decision
protocol.
It
is
important
to
avoid
oversimplification
and
overstandardization.
Although
clear
technical
rationality
might
be
needed
in
a
cockpit,
it
can
be
too
rigid
for
stipulating
how
to
give
performance
feedback
to
employees
or
how
to
onboard
and
socialize
new
employees
with
very
different
kinds
of
education
and
experience.
The
goal
is
to
create
a
routinized
decision
protocol
that
is
agile
where
it
needs
to
be.
It
is
important
that
decision
makers
understand
the
principles
behind
the
protocol
or
can
go
off
script
if
the
situation
suggests
the
need
to.
Implementation
depends
on
the
right
balance
between
standardization
and
use
of
practitioner
judgment.
An
organization
that
gives
considerable
attention
to
rou-
tinization
of
decisions
is
the
US
Army.
It
does
so
since
turn-
over
is
inherent
as
missions
can
last
from
a
day
to
a
year,
involving
different
people
who
rotate
through.
Turnover
characterizes
many
organizations
but
unless
it
is
recognized
as
a
prevailing
condition,
organizations
often
make
little
effort
to
develop
good
decision
routines.
Importantly,
senior
leaders
need
to
act
as
visible
role
models
in
the
development
and
effective
use
of
routines.
To
adapt
routine
protocols
effectively,
high
performing
decision
makers
should
actually
review
or
even
rehearse
the
protocols
and
procedures
in
slow
times
in
order
to
refresh
their
memory
and
become
familiar
with
important
aspects
of
their
task
(e.g.,
where
equipment
is
located,
whom
to
notify).
Planning
for
the
tasks
performed
in
emergencies
allows
these
tasks
to
be
completed
faster,
offering
a
cushion
of
time
to
accomplish
the
essentials.
Periodically,
users
should
review
and
evaluate
decision
rou-
tines
or
protocols
for
their
continued
effectiveness
and
make
improvements
as
needed.
Accessibility
is
a
big
issue
in
using
decision
routines
or
protocols.
Can
a
protocol
really
be
followed
effectively
from
memory?
Military
aviation
inaugurated
use
of
checklists
when
the
B-17
was
adopted
since
this
plane
was
too
complex
to
fly
by
relying
on
a
pilot’s
memory
alone.
Where
memory
is
overburdened
or
inconsistencies
in
performance
have
ser-
ious
consequences,
decision
protocols
need
to
be
written
and
easily
accessible
(e.g.,
available
on
both
iPad
and
hard
copy).
Another
key
reason
for
using
decision
protocols
is
to
free
up
time
and
cognitive
effort
where
it
is
truly
needed:
in
making
decisions
that
aren’t
routine,
which
take
special
effort
to
make
them
well.
Non-Routine
Decisions
with
Known
Unknowns
Accepting
ambiguity
is
a
powerful
motivator
to
confront
and
then
deal
with
troubling
issues.
–Paul
C.
Nutt
Non-routine
decisions
are
the
kind
of
decision
for
which
managers
typically
are
trained
in
business
and
other
profes-
sional
schools.
Decision
analysis
is
the
overarching
frame-
work
schools
teach
for
making
important
non-routine
decisions.
But
many
recommended
processes
for
non-routine
decisions
(e.g.
multidimensional
approaches)
go
largely
unused.
In
this
context,
I
recommend
what
the
body
of
evidence
supports
in
making
non-routine
decisions:
the
use
of
the
six
de-biasing
practices
described
above.
Non-routine
decisions
involve
situations
new
to
the
deci-
sion
makers
for
which
relevant
evidence
exists
but
is
not
in
hand.
In
these
decisions,
the
critical
information
needed
must
first
be
identified.
To
do
so
necessitates
the
first
course
of
action
in
non-routine
decisions:
Understanding
the
pro-
blem
or
opportunity
that
motivates
the
decision.
Search
and
discovery
processes
are
essential
in
comprehend
the
pro-
blem
or
opportunity
in
order
to
identify
the
most
appropriate
decision
frame
(practice
#1).
The
political
implications
of
the
problem
or
opportunity
need
to
be
addressed,
including
who
benefits,
who
is
harmed,
whose
support
is
needed,
and
ways
of
making
the
decision
an
integrative
one
(practice
#2).
Multiple
alternatives
should
be
generated
(practice
#3)
and
evaluated
on
important
goals
and
success
criteria
(practice
#4).
All
serious
alternatives
should
be
considered
simulta-
neously
in
order
to
provide
fair
and
balanced
weighing
of
their
pros
and
cons.
Decision
makers
then
choose
the
best
option.
If
there
remain
contested
issues
regarding
alterna-
tives,
it
can
help
to
run
a
field
test
or
experiment
and
assess
its
results.
It
is
important
to
identify
the
stakeholders
rele-
vant
to
the
decision
(practice
#5)
and
involve
them
in
the
decision
process
where
possible
to
better
formulate
an
appropriate
decision
frame
and
understand
how
to
imple-
ment
alternatives
in
ways
that
can
meet
stakeholder
needs.
Throughout
this
process,
the
four
sources
of
evidence
(prac-
tice
#6)
should
be
considered
in
order
to
assess
what
is
known
and
what
needs
to
be
known
in
order
to
make
an
effective
decision.
The
next
step
is
developing
an
implementation
plan
and
then
monitoring
its
progress,
adjusting
the
plan
based
on
feedback
as
needed.
D.M.
Rousseau/Organizational
Dynamics
49
(2020)
100756
9

An
additional
way
to
improve
non-routine
decision
mak-
ing
is
to
conduct
periodic
after
action
reviews
(AARs).
AARs
can
revisit
a
recent
set
of
decisions
in
order
to
evaluate
their
effectiveness,
helping
decision
makers
reflection
on
the
processes
they
used,
assumptions
made,
and
the
implica-
tions
for
future
decisions.
Four
questions
are
asked:
1)
what
did
we
set
out
to
do?
2)
What
actually
happened?
3)
Why
did
it
happen?
4)
What
are
we
going
to
do
next
time?
Following
up
decisions
with
opportunities
for
reflection
can
make
lead
to
better
future
decisions.
Novel
Decisions
Involving
Unknown
Unknowns
“No
battle
plan
survives
the
first
hour.”
–Anonymous
Novel
decisions
involve
new
or
emergent
conditions
for
which
prior
experience
and
historical
knowledge
provide
little
useful
insight.
These
novel
decisions
can
involve
tech-
nical
uncertainty
where
effective
procedures
aren’t
yet
established
for
a
newly
recognized
problem,
like
how
to
help
elderly
people
with
health
problems
live
on
their
own,
or
how
to
deal
with
hacking
into
an
organization’s
website.
Or,
novel
decisions
can
involve
market
uncertainty
or
emer-
ging
technology,
like
how
best
to
use
social
media
to
reach
out
to
a
non-profit's
donor
population.
They
also
can
involve
political
uncertainty
where
no
plausible
solutions
are
known
that
are
endorsed
by
key
parties,
such
as
how
to
deal
with
community–— police
relationships
in
the
aftermath
of
a
con-
tested
civilian
shooting
by
an
officer.
Novel
situations
involve
considerable
ambiguity,
in
the
cues
that
signal
the
problem
or
in
the
options
available
to
respond.
Cues
can
be
so
vague
or
confusing
that
the
problem
cannot
be
identified.
The
refugee
crisis
in
the
Middle
East
and
Africa
is
in
part
a
function
of
a
multi-year
effects
of
drought
displacing
farming
peoples
who
headed
to
cities
in
search
of
resources,
a
problem
not
recognized
at
its
start
over
30
years
ago.
Or
if
the
problem
is
understood,
it
still
may
be
unclear
what
to
do.
An
illustration
of
this
was
the
1989
United
232
crash
in
Sioux
City,
Iowa,
where
all
flight
controls
were
lost
when
hydraulic
cables
were
severed
following
catastrophic
engine
failure.
In
a
classic
case
of
technical
uncertainty,
the
crew
had
to
figure
out
how
to
control
the
plane
under
conditions
that
had
never
occurred
before–— a
true
unknown
unknown.
Working
together
with
a
flight
trainer
who
happened
to
be
on
board,
the
crew
tried
a
variety
of
experiments
until
a
solution
emerged
using
alter-
native
thrusts
of
the
two
remaining
engines
in
order
to
steer.
In
such
complex
situations,
solutions
may
not
generalize
to
other
problems
but
the
process
used
does.
I
want
to
differentiate
two
kinds
of
unknown
unknowns.
The
first
is
complex
situations
where
the
unknowns
reflect
new
kinds
of
conditions
and
events,
whose
underlying
order
can
ultimately
be
understood.
The
second
are
chaotic
situa-
tions
where
there
may
be
more
unknowables.
Effectively
facing
the
first,
a
complex,
unknown
unknown
situation
necessitates
sensemaking
and
learning
by
doing.
Identifying
small
concrete
actions
that
probe
and
potentially
alter
the
environment
can
help
organizations
achieve
their
goals
despite
high
uncertainty.
Consider
how
the
movement
promoting
land
rights
for
indigeneous
people
managed
the
political
uncertainty
it
faced.
Its
leaders
noted
that
palm
oil
plantations
worldwidehave
encroached
on nativelands, threa-
ten
traditional
ways
of
life
and
undermine
both
economic
and
social
well-being.
Advocates
hit
on
the
idea
of
having
indige-
neous
people
from
three
continents
tour
Europe
and
meet
with
representatives
of
the
European
Union.
By
garnering
press
attention
and
making
visible
the
faces
of
people
affected
by
EU
policies,
the
result
was
a
dialing
back
of
subsidies
and
supports
for
palm
oil
production.
Incremental
activities
that
produce
visible
results
in
complex
situations
have
been
called
“small
wins.”
A
counter
example
where
decision
makers
didn’t
learn
from
exploratory
experiences,
the
Occupy
movement
failed
to
successfully
identify
a
corporate
target
upon
which
to
focus
its
efforts
in
the
aftermath
of
the
2008
crisis.
Another
example
of
addressing
truly
novel
unknown
unknowns
is
peacekeeping,
a
task
that
militaries
around
the
world
increasingly
perform.
In
the
1990s,
the
US
Army
began
systematic
efforts
to
learn
how
to
undertake
this
emergent
activity.
A
host
of
challenges
became
obvious. In the
aftermath
of
a
coup
in
Haiti,
the
US
Army
needed
to
develop
and
carry
out
proper
rules
of
engagement
with
a
civilian
population
and
deal
with
unrest,
terrorism
and
cultural
differences
all
while
trying
to
understand
what
could
be
achieved
realistically.
A
large
portion
of
the
Army’s
approach
to
learning
how
to
keep
the
peace
was
the
AAR.
At
the
end
of
an
activity,
at
the
end
of
day,
upon
completion
of
a
mission
or
a
project,
the
above
four
questions
were
asked.
The
reflection
process
that
results
serves
several
functions
including
sensemaking,
data
verifica-
tion
and
feedback
both
on
results
and
on
ways
to
improve
task
process.
(N.B.
This
reflection
needs
to
be
carefully
timed
and
managed sincetoo many
simultaneous
experiences
jeopardize
thoughtful
deliberation.)
In
the
second
case,
chaotic
situations
where
an
event
or
crisis
precipitates
decision
making,
these
“wild
cards,”
can
call
for
a
somewhat
different
sort
of
response.
Sudden
unex-
pected
external
events
with
immense
consequences
for
the
future,
like
9/11
or
the
collapse
of
worldwide
financial
mar-
kets
in
2008,
often
entail
unfolding
events
that
make
learning
more
difficult.
Such
decision
situations
require
resilience,that
is,
a
muddling
through
process
supported
by
experimentation,
quick
scenario
building,
acting
and
then
reflecting.
Though
wild
cards
are
unknowable,
the
processes
whereby
truly
novel
decisions
can
be
effectively
made
in
response
do
have
a
pattern,
which
can
be
the
focus
of
practice.
Consider
how
Gene
Kranz,
the
flight
director
of
Apollo
13,
regularly
conducted
“fire
drills”
for
his
staff,
that
is,
simu-
lated
crisis
situations
under
the
observation
of
trained
instructors.
When
the
oxygen
tank
on
Apollo13
malfunc-
tioned,
his
team
was
able
to
ultimately
improvise
a
fix
that
allowed
the
crew
to
survive
and
come
home.
In
crisis
situa-
tions,
where
multiple
decisions
must
be
made
in
an
evolving
situation,
it
appears
best
to
train
people
in
a
mix
of
slow
training
under
thoughtful
conditions
combined
with
more
rapid
training
under
realistic
conditions–— the
basis
of
effec-
tive
disaster
training.
The
little
decisions
that
ultimately
may
resolve
a
crisis
can
take
the
form
of
experiments.
Other
novel
decisions
precipitated
by
an
event
like
the
2008
global
financial
crisis
unfold
over
time
and
have
diverse
outcomes.
The
learning
processes
in
support
of
novel
deci-
sions
require
information
gathering
and
adjustment
after
surprising
events
occur,
since
advance
planning
or
anticipa-
tion
are
impossible.
Learning
can
take
the
form
of
small
trials
and
experiments,
as
in
the
case
of
strategies
that
10
D.M.
Rousseau/Organizational
Dynamics
49
(2020)
100756

ultimately
led
to
effective
treatments
for
AIDS).
One
plan
at
a
time
won't
cut
it:
multiple
scenarios
and
simultaneous
small
experiments
might
be
needed
to
explore
what
works.
By
allowing
these
tests
to
be
independent,
a
failed
experi-
ment
doesn’t
burn
down
the
house.
A
high
rate
of
failure
is
not
necessarily
bad
for
exploration.
Note
that
improvisations
need
not
be
limited
to
crisis
situations.
Improvisations
can
be
a
good
way
to
probe
novel
situations,
like
emerging
markets,
to
see
what
might
work.
Web-based
product
marketing,
for
example,
frequently
involves
small
variations
in
pricing,
display
and
bundling
of
products
in
order
to
see
what
customers
are
most
likely
to
buy.
In
truly
novel
situations,
it
is
important
to
resist
simplifying
one’s
understanding
of
the
problem
in
order
to
keep
updating
as
evidence
changes.
Unexpected
events
constitute
information
and
accelerating
feedback
via
impro-
visation
can
make
their
detection
earlier.
The
key
idea
is
to
treat
uncertainty
as
something
to
work
with
in
order
to
stimulate
new
ways
of
acting
and
understanding.
Caveats
The
decision
situations
I
describe
here
differ
from
real-time
dynamic
decisions
where
multiple
interdependent
decisions
are
made
as
in
the
case
of
operating
a
submarine
or
nuclear
power
plant.
Requisite
decision
practices
may
differ
from
typical
organizational
decisions.
CONCLUSIONS
It
is
possible
for
human
beings
to
become
more
comfor-
table
with
complexity
and
uncertainty
by
virtue
of
the
ways
decisions
are
made
in
organizations.
By
regularly
using
the
six
de-biasing
practices,
the
quality
of
organiza-
tional
decisions
can
get
better.
And
by
expanding
the
repertoire
of
decision
processes
used
in
our
organizations,
we
can
deploy
both
our
knowledge
and
agility
in
an
uncertain
world.
My
hope
is
that
readers
will
regularly
ask
themselves
certain
important
questions
about
how
decisions
are
made
in
their
organizations.
Are
we
spending
enough
time
understanding
the
issues
at
the
outset
of
our
decision
making?
Do
we
use
diverse
forms
of
good
quality
evidence?
Do
we
generate
enough
alternatives?
Are
sta-
keholders
effectively
included
and
considered?
How
well
do
we
learn
by
experiments
and
our
past
decisions?
When
we
have
good
evidence
of
what
works,
do
we
standardize
enough?
Effectively
addressing
these
questions
builds
the
capacity
to
make
decisions
and
manage
well
in
uncertain
times.
D.M.
Rousseau/Organizational
Dynamics
49
(2020)
100756
11

SELECTED
BIBLIOGRAPHY
The
work
of
many
decision
management
scholars
forms
the
basis
for
this
article.
(A
version
with
citations
is
available
from
the
author.)
Prominent
among
these
are
J.
Frank
Yates,
(Decision
management:
How
to
assure
better
decisions
in
your
company.
Wiley,
2003),
Paul
C.
Nutt,
(“Surprising
but
true:
Half
the
decisions
in
organizations
fail.”
The
Academy
of
Management
Executive,
1999,
13(4),
75—90),
Gerd
G.
Gigenerzer
(Risk
savvy:
How
to
make
good
decisions.
Penguin,
2015),
Karl
Weic k
and
Kathleen
Sut-
cliffe
(Managing
the
unexpected:
Resilient
performance
in
an
age
of
uncertainty.
Wiley,
2011),
Chip
Heath,
Richard
Larrick
and
Joshua
Klaymans
(“Cognitive
repairs:
How
organizational
practices
can
compensate
for
individual
shortcomings.”
Review
of
Organizational
Behavior,
1998,
20,
1—37.).
The
concept
of
known
knowns,
known
unknowns
and
unknown
unknowns
originated
in
the
US
Department
of
Defense
as
a
conceptual
framework
over
50
years
ago
and
forms
the
basis
for
the
Cynefin
model
developed
by
Dave
Snowdon
(see
D.
Snowdon
&
M.E
Boone,
“A
leader’s
framework
for
decision
making.”
Harvard
Busi-
ness
Review,
2007.)
Insights
into
uncertainty
stem
from
a
variety
of
studies,
most
prominently,
Lipshitz,
R.,
&
Strauss,
O.
(“Coping
with
uncertainty:
A
naturalistic
deci-
sion-making
analysis.”
Organizational
Behavior
and
Human
Decision
Processes,
1997,
69(2),
149—163).
The
fascinating
University
of
Chicago
study
on
thinking
in
a
foreign
language
is
by
Boaz
Keysar,
Sayuri
Hayakawa
and
Sun
Gyu
An
(“The
foreign-language
effect
thinking
in
a
foreign
tongue
reduces
decision
biases.”
Psychological
Science,
2012,
23,
661—668).
Denise
M.
Rousseau
(Ph.D.
UC
Berkeley)
is
an
organizational
psychologist
and
University
Professor
at
Carnegie
Mellon
University.
Two-time
winner
of
the
Academy
of
Management’s
George
Terry
Award
for
best
management
book,
for
Psychological
Contracts
in
Organizations:
Understanding
Written
and
Unwritten
Agreement
and
I-
Deals:
Idiosyncratic
Deals
Employees
Bargain
for
Themselves,
she
has
received
Distinguished
Scholarly
Contri-
bution
Awards
for
lifetime
achievement
from
both
the
Academy
of
Management
and
the
Society
for
Industrial/
Organizational
Psychology.
She
is
co-founder
of
the
Center
for
Evidence-Based
Management,
and
author
with
Eric
Barends
of
Evidence-Based
Management:
How
to
Make
Better
Organizational
Decisions
(Kogan
Page).
(Heinz
School
of
Public
Policy
and
Management
&
Tepper
School
of
Business,
Carnegie
Mellon
University,
Pittsburgh,
PA
15232,
USA.
Tel.:
+1
(412)
268
8470;
Fax:
+1
(412)
268-5338.
e-mail:
denise@cmu.edu).
12
D.M.
Rousseau/Organizational
Dynamics
49
(2020)
100756