Content uploaded by Alexander Conley
Author content
All content in this area was uploaded by Alexander Conley on Dec 15, 2015
Content may be subject to copyright.
Evaluating Collab orative Natural Resource
Managem e nt
ALEXANDER CONLEY
North Fork John Day Watershed Council
Monument, Oregon, USA
MARGARET A. MOOTE
Ecological Restoration Institute
Northern Arizona University
Flagstaff, Arizona, USA
In this article, we look at the evolving collaborative natural resource management
movement in the United States and discuss current calls to evaluate it. We then
explore approaches researchers have used to evaluate both specific efforts and the
broader movement. Evaluative criteria developed thus far by several researchers
show commonalities as well as differences. We argue that evaluation approaches will
necessarily vary with the evaluation’s intent, the type of collaborative effort being
evaluated, and the values of the evaluator. Evaluators need to consider and make
explicit their standards for comparison, criteria, and methods in order to clarify the
nature of an evaluation and facilitate the synthesis of findings.
Keywords collaboration, evaluation, natural resource management, participa-
tory decision making, process
Today, collaborative approaches to natural resource management are being broad ly
promoted as promising ways to de al with complex and contentious natural resource
issues. As collaborative efforts become more widespread and are incorporated into
official policies, both proponents and critics seek to evaluate these new approaches.
Are these new approaches all that they are held up to be? Do they really lead to
improved resource management? What can and cannot be reasonably expected of
them and what variables influence their effectiveness? Answering these questions and
others like them requires careful selection of methods to evaluat e the effectiveness of
collaborative efforts. In this article we give an overview of existing evaluation
approaches and methods and discuss the considerations that should be taken into
consideration by those conducting evaluations.
Received 2 October 2001; accepted 12 June 2002.
The authors thank three anonymous reviewers and the Udall Center for Studies in Public
Policy at the University of Arizona, where both authors worked during the initial development
of this article.
Address correspondence to Margaret A. Moote, Ecological Restoration Institute,
Northern Arizona University, PO Box 15018, Flagstaff, AZ 86011-5018, USA. E-mail:
ann.moote@nau.edu
371
Society and Natural Resources, 16:371–386, 2003
Copyright # 2003 Taylor & Francis
0894-1920/2003 $12.00 + .00
DOI: 10.1080/08941920390190032
Many terms have been coined to describe what we are calling collaborative
natural resources management. Collaborative efforts ha ve been referred to as
partnerships (Moote 1996; Williams and Ellefson 1997; Wondolleck and Yaffee
1994), consensus groups (Innes 1999), community-based collaboratives (Moote et al.
2000), and alternative problem-solving efforts (Kenney and Lord 1999). Colla-
borative approaches to natural resource management include watershed manage-
ment (Natural Resources Law Center 1996), collaborative conservation (Brick et al.
2000; Cestero 1999), community forestry (Brendler and Carey 1998), community-
based conservation (Western and Wright 1994), community-based ecosystem man-
agement (Gray et al. 2001), grass-roots ecosystem management (Weber 2000),
integrated environmental management (Born and Sonzogni 1995; Margerum 1999),
and community-based environmental protection (U.S. Environmental Protection
Agency 1997). Specific models have been developed, such as coordinated resource
management (Anderson and Baum 1988; Cleary and Phillippi 1993), and colla-
borative learning (Daniels and Walker 2000). While we recognize that these different
names are not interchangeable, the approaches they refer to do share several
common characteristics. In this article we use the term collaborative natural resources
management effort to refer to multiparty natural resourc e management projects,
programs, or decision-making processes using a participatory approach and explore
the range of evaluation approaches that have been applied to such efforts.
An idealized narrative of collaborative natural resource management has
emerged across the popular and academic literature. In it, collaboration is hailed as a
way to reduce conflict among stakeholders; build social capital; allow environm ental,
social, and economic issues to be addressed in tandem; and produce better decisions
(Bernard and Young 1997; Brick et al. 2000; Innes 1996; Jones 1996; Weber 2000).
Calls for more collaborative decision making in natural resource management can be
found in everything from the popular press (e.g., Krist 1998) to promot ional videos
(e.g., Bureau of Land Management 1995) to political speeches. Collaboration among
diverse interests has been promoted by groups ranging from conservative think tanks
(Harrington and Hartwell 1999) to critical theorists (e.g., Dryzek 1996). Numerous
handbooks give guidance on developing effective collaborative processes (Clark
1997; Cleary and Phillippi 1993; U.S. Environmental Protection Agency 1997;
Moote 1996; Paulson and Chamberlin 1998; Susskind et al. 1999), and analytical
treatises detail the steps in their development (Gray 1989; Selin and Chavez 1995).
Actual experience with collaborative efforts has grown exponentially over the
last 20 years, as collaborative responses to many resou rce-management challenges
have developed both independently and simultaneously across the United States
(Weber 2000). Hundreds of case studies describe specific efforts (e.g., Braxton Little
1997; Chisholm 1996; House 1999; KenCairn 1996; Mazaika 1999; Shelly 1998;
Smith 1999), and many readers will be aware of collaborative efforts in their own
regions. A number of studies catalogue some of these experiences (Bingham 1986;
Coughlin et al. 1999; Kenney et al. 2000; Kusel and Adler 2001; Natural Resources
Law Center 1996; Wondolleck and Yaffee 1994; 2000; Yaffee et al. 1996). Colla-
borative efforts are gaining considerable support: Many efforts have risen from the
grass roots and from state and federal governments, and nongovernmental organi-
zations and foundations are promoting collaborative efforts through a wide range of
programs and policies (Dukes and Firehock 2001; U.S. Environmental Protection
Agency 1997).
At the same time, collaborative approaches to resource management are
attracting vocal critics (e.g., Blumberg and Knuffke 1998; Coggins 1998; Coglianese
372 A. Conley and M. A. Moote
1999; McCloskey 1998; Southern Utah Wilderness Association 1994). National
interest groups claim that federal laws and the public interest are not adequately
considered in local decision-making efforts. Many environmental groups charge that
these efforts are co-opted by local economic development interests while industry
groups contend the opposite. People outside the ‘‘inner circle’’ sometimes charge that
their views are excluded, while agencies question whether successful collaborative
efforts are replicable in other communities. Participants in processes that fail to
achieve their desired outcomes may question the time and effort they invested.
Kenney (2000) provides an excellent overview of these criticisms, which are forcing
many people to rethink the idealized narrative of collaborative natural resource
management.
As collaborative natural resource management gains prominence, calls for
evaluations of both specific initiatives and the broader movement are becoming
common (Innes 1999; Innes and Booher 1999; Kellert et al. 2000; Kenney 1999;
Kenney 2000; Kenney et al. 2000; Leach et al. 2000; Moote et al. 2000; Selin et al.
2000). Collaborative groups themselves are initiating monitoring and self-evaluation
processes, often as part of a participatory approach to adaptive management.
Despite this widespread interest in using evaluation to learn from past experience,
discussions of different approaches to evaluation and the issues that they raise are
rare. There is a growing literature on evaluating collaborative efforts, but in most
cases the definitions of success used and the basis for the methods chosen are not
clear, limiting the generalizabil ity of the findings (Leach 2000, 92). In this article, we
describe different evaluation approaches and methods used for examining colla-
borative efforts; we then raise considerations that evaluators need to address, such as
the nor mative nature of evaluation and the comparability of results.
Evaluation Approaches
Why Evaluate?
While interest in evaluating collaborative conservation efforts is widespread, moti-
vations for evaluation vary. Participants in collaborative efforts want evaluations
that can help improve their efforts and meet their personal goals. Facilitators and
resource managers are looking for guidelines that help identify which approaches are
appropriate in different circumstances. Policymakers want informed evaluations that
help them formulate appropriate rules and regulations. Funders and interest groups
need to determine which collaborative efforts to support and what stance to take on
general policies promoting or inhibiting collaborative processes. Advocates want
proof of their success (Innes 1999), while critics want to demonstrate that their
concerns are well founded (e.g., Coglianese 1999). Academics are interested in
exploring how collaborative resource management affects society, and in testing
theoretical models on specific examples.
Ultimately, interest is fueled by the belief that effective evaluation can help: (1)
determine when the idealized narrative used to justify collaborative natural resource
management holds true, (2) address criticisms of these efforts, and (3) assess and
refine efforts to institutionalize a movement that has developed largely at the grass-
roots level. Between the critics and the current body of experience, even proponents
are coming to realize that collaborative approaches to natural resource management
can but do not always work and that at times failure comes at a heavy cost of time
and effort (and, perhaps more significantly, in social capital consumed rather than
Evaluating Collaboration 373
built). Ideally, evaluation will inform our understanding of both the potential and
the limits of collaborative natural resource management.
Who Evaluates?
Collaborative efforts are constantly being evaluated, formally and informally, by
those who participate in them, hope to learn from them, or are concerned about the
outcomes they determine. Yet many people wonder who is best placed to evaluate
these efforts. Some have called for neutral, third-party evaluations in order to
achieve reliable, unbiased results (Innes 1 999), while others—especially those directly
involved with collaborative efforts— emphasize the importance of participatory
evaluation, in which groups conduct self-evaluations, and=or the evaluator works
closely with those involved in and affected by a project or process (Jacks on and
Kassam 1999; Moote et al. 2000). Many participants in collaborative natural
resource management emphasize that evaluators must be intimately familiar with a
process, its history, and its context, and disparage evaluation from a distance (Moote
et al. 2000). At the same time, some worry that the interests of those directly involved
in a collaborative venture may reduce objectivity. For this reason, other evaluation
methodologies have been designe d to be non-invasive and readily conducted by an
outsider (d’Estree and Colby 2000). When determining how close to get to the col-
laborative effort under study, evaluators should consider what it is they plan to
evaluate. If the focus is on environmental changes, a neutral ‘‘outside’’ evaluator
may be preferred. For evaluations of process or human behavior al changes, how-
ever, some experts argue that the evaluator must become immersed in the situation
(Patton 1986).
What Is Evaluated?
Depending on the evaluator’s interests, evaluations may examine a variety of factors.
Typically, they focus on either characteristics of a process, such as inclusivenes s of
representation and decision-making methods, or outcomes. The earliest evaluations
of environmental media tion asked whether it resulted in cheaper, faster, fairer, more
innovative, and longer lasting agreements than those achieved through litigation.
Evaluations of these ‘‘agreement outcomes’’ produced mixed results (Bingham 1986;
Sipe 1998). In response, Buckle and Thomas-Buckle (1986) proposed that even
‘‘failed’’ mediations (where no formal agreement was reached) could have positive
and long-lasting outcomes, like increased understanding and improved relationships.
Proponents of collab orative natural resource management now commonly identify
such ‘‘social outcomes’’ as important evaluation criteria. Many observers contend
that whether or not a collaborative effort leads to improved environmental condi-
tions is the ultimate measure of its success (Kenney 1999; Snow 1998). Given the
goals of many collaborative efforts, changes in local economic development might be
another type of outcome to be assessed. Simultaneously evaluating all of these dif-
ferent types of outcomes is a daunting task that is rarely undertaken. Some eva-
luators be lieve that evaluating only one narrowly defined outcome at a time makes
analyses tighter and more consistent (d’Estree and Colby 2000). Others stress that all
outcomes of a collaborative effort must be considered together (Innes 1999).
Evaluation can also occur at many different scales. At the project level, a
watershed council may want to determine whether its members were able to complete
a specific stream-restoration project, the council or a third party may evaluate the
374 A. Conley and M. A. Moote
workings of the council as a whole, while a state agency may choose to assess the
combined impact of all the councils it funds. Evaluations can also occur at diffe rent
temporal scales: They may be ongoing, occurring in an iterative, adaptive fashion; or
they may occur at specific points in time, as in Innes’s (1999) distinction of mid-
course, end-of-process, and retrospective evaluations.
Evaluation Criteria
Any attempt at evaluation is based on comparing reality to a set of criteria.
Perhaps the simplest criterion put forth to assess collaborative efforts is the one used
by Williams and Ellefson (1997), who ‘‘defined a successful partnership as a group
able to attract and keep individuals engaged in partnership activities.’’ This defini-
tion of success has obvious shortcomings, such as the lack of clear correlation
between meeting attendance and tangible outcomes. The deeper one delves, the more
criteria one can identify, for each of the oft-cited benefits and criticisms of colla-
boration can easily be turned into criteria for evaluating specific collaborative
efforts. Thus, ‘‘Collaboration saves money’’ becomes ‘‘Did it save money?’’ and
‘‘Collaboration leads to local co-optation’’ becomes ‘‘Did it lead to local co-opta-
tion?’’ (See Kenney [2000] and Coughlin et al. [1999] for overviews of both benefits
and criticisms.)
Goals of an evaluation must be clearly defined in order to select appropriate
evaluation criteria and guide data collection. A recent study by the National
Academy of Public Administration (NAPA) points out the risk of not carefully
matching criteria to evaluation goals. When evaluating the effectiveness of U.S.
Environmental Protection Agency and state enforcement of environmental laws, the
National Academy of Public Administration found that the type of data available
(numbers of permits issued, inspections co nducted, enforcement actions initiated,
etc.) revealed very little about the programs’ effectiveness at improving environ-
mental conditions (NAPA 2001).
As interest in evaluating collaborative efforts has grown, several lists of eva-
luative criteria have been generated. Criteria lists have been developed for evaluating
environmental conflict resolution (d’Estree and Colby 2000), consensus-building
efforts (Innes 1999), participatory processes (Poisner 1996), watershed groups (Born
and Genskow 2000; Leach 2000), and integrated resource management (Bellamy et
al. 1999); for assisting foundations in evaluating which collaborative efforts to fund
(KenCairn 1998); and for helping environmental activists decide whether to support
a given effort (Blumberg 1999). There are also lists of criteria and indicators for
sustainable resource management (National Association of State Foresters 1997),
community development (e.g., Baun et al. 1996), and environmental, social, and
economic sustainability (Farrell and Hart 1998; Lead Partnership Group 2000).
Brunson (2000) identified criteria emphasized by participants in collaborative pro-
cesses. Despite the different perspectives and backgrounds of their authors, these lists
share many criteria an d indicators, including broad representation, consensus deci-
sion making, and feasible outcomes (Table 1).
The Normative Nature of Criteria Selection and Weighting
Commonalities among the various lists of indicators suggest that developing a
single, comprehensive, and broadly accepted set of criteria might be possible. Yet the
criteria relevan t to a given evaluation will always vary with the reasons for
the evaluation, the values and perspective of the evaluator, and the context and
Evaluating Collaboration 375
characteristics of the collaborative effort being evaluated. Even evaluators who
agreed to use a standard set of criteria would likely weight them differently and thus
come up with very different evaluations of the same case. How do we weigh capacity
building against results on the ground? Economic efficiency against equity issues?
Short-term results against long-term precedents?
Recognizing the normative nature of criteria weighting, d’Estree and Colby
(2000) have called for an objective evaluator to rank processes according to a
comprehensive set of criteria, creating a matrix to which users add their own weights
to come up with an overall evaluation of relative success. While conceptually
appealing, such a comprehensive evaluation would require an inordinate amount of
time and effort while remaining dependant on the evaluator’s judgements. Whatever
form an evaluation takes, researchers must acknowledge that evaluation is inherently
normative, and inevitably political, for it is a forum where the public image of a
collaborative effort is negotiated. Value judgments are the basis for de termining
what makes a process successful or uns uccessful, and different evaluato rs are likely
to judge the same process differently. Thus we do not advocate the development of a
comprehensive list of criteria. Instead, we stress that the criteria, weightings, and
methods used must be made clear if evaluations are to be fairly comp ared to each
other.
TABLE 1 Typical Evaluation Criteria
Process criteria Broadly shared vision
Clear, feasible goals
Diverse, inclusive participation
Participation by local government
Linkages to individuals and groups beyond
primary participants
Open, accessible, and transparent process
Clear, written plan
Consensus-based decision making
Decisions regarded as just
Consistent with existing laws and policies
Environmental outcome criteria Improved habitat
Land protected from development
Improved water quality
Changed land management practices
Biological diversity preserved
Soil and water resources conserved
Socioeconomic outcome criteria Relationships built or strengthened
Increased trust
Participants gained knowledge and
understanding
Increased employment
Improved capacity for dispute resolution
Changes in existing institutions or creation of
new institutions
Note. Sources: Blumberg (1999), Born and Genskow (2000), d’Estree and Colby (2000),
Innes (1999), KenCairn (1998), and Lead Partnership Group (2000).
376 A. Conley and M. A. Moote
Comparability of Cases
Evaluators also need to consider the differences between collaborative efforts
when selecting evaluation criteria. Despite their broad similarities, distinctions can
be made between different types of collaborative efforts. For instance, the terms
‘‘collaborative’’ and ‘‘community-based’’ are often used interchan geably, despite the
fact that they can refer to distinctly different activities. Community-based resource
management is management by local people (Brosius et al. 1997), while collaborative
refers to the involvement of multiple stakeholders (such as landowners, public
agencies, interested citizens, scientists, environmentalists, and other interest groups) .
An effort may be both community-based and collaborative, but this need not be so.
Another distinction may be drawn between a ‘‘mediation approach,’’ which focuses
on collaborative efforts convened to resolve specific, defined conflicts, and a ‘‘part-
nership approach,’’ which focuses on the development of longer term partnerships
that aim to promote ecological, econ omic, and social health within a defined region.
Because of differences such as these, evaluative questions, criteria, and methods
need to be carefully matched to the collaborative effort being evaluated. For
example, criteria based on the inclusion of all stakeholders may be essential in
evaluating a collaborative body that has significant decision-making power, but may
be unrealisti c—and even unfair—criteria to use in judging a community-based effort
that strives to give voice to disenfranchised interests in the context of a broader
pluralistic forum. Similarly, a facilitator’s neutrality may be an important criterion
when assessing a collaborative effort meant to mediate a specific controversy, but
may be irrelevant when assessing a partnership where the convener’s vision is a key
part of what keeps people working together. Each evaluator must carefully assess the
nature and context of an effort when choosing which evaluat ive criteria to apply.
Standards for Comparison
The standard against which a collaborative effort is evaluated similarly varies
depending on the evaluator’s values and goals. Should a collaborative effort be
evaluated against its own goals, against an ideal, or against another effort? The
answers to these questions in part determine the specific evaluation criteria that are
selected and the data collection and analysis methods used.
Comparing a Collaborative Effort With Its Goals
The most common form of evaluation focuses on whether and how collaborative
efforts meet their identified goals and objectives. Goal setting is an important activity
in many collaborative efforts, and many identify a range of social, economic, and
environmental goals. In this type of evaluation, outcomes are measured and com-
pared to targets identified in mission and goals statements and management plans.
Goal evaluation is popular with participants in adaptive management ventures
where goals and management applications are repeatedly reviewed and revised. Goal
evaluation is also popular among organizations that provide financial or technical
support to colla borative activities, and simple versions can be found in most every
annual report an d grant progress update.
Goal evaluations do have several limitations. First, they do not assess the
appropriateness of the goals and objectives themselves, the assumptions behind
them, or the process used to define them. Second, goal evaluation requires that a
collaborative effort have clearly defined and uncontested goals, which may not
Evaluating Collaboration 377
always be the case, especially in efforts that bring together diverse interests. As
Patton (1986) points out, identifying goals can be a complex task that ultimately
boils down to a question of whose goals should be used: a group’s publicly stated
goals, or its real goals? The funder’s goals, or the staff goals? Third, goal-based
evaluations run the risk of missing unanticipated (but important) outcomes (Patton
1986).
Comparing Multiple Efforts
Evaluators can choose to compare similar collaborative efforts and rank them
according to selected criteria in an effort to determine which are more successful and
why (for examples, see Carr et al. 1998; Duane 1997; Imperial and Hennessey 2000;
Moseley 1999; Paulson 1998; Williams and Ellefson 1997). Such comparisons can
show how variations in processes and in both social and ecological contexts result
in different outcomes; they also broaden our focus to include both successes and
failures.
When the collaborative effort being evaluated is a new decisionmaking
approach, it may be compared to different types of processes that are used for
similar purposes, such as different forms of planning used by the U.S. Forest Servic e
(Gericke and Sullivan 1994) or various forms of dispute resolution (Coglianese 1997;
d’Estree and Colby 2000; Sipe 1998). Such comparisons may be particularly useful to
policy makers deciding which approach is most appropriate for a given situations.
As noted above, it is important to clarify that the projects or programs being
evaluated share enough characteristics to make the comparison meaningful. Some
researchers have attempted to identify axes of differentiation (Coughlin et al. 1999)
and classify collaborative efforts (Blumenthal and Jannink 2000; Cestero 1999); such
classifications are useful to evaluations that compare multiple efforts. Developing a
systematic framework for understanding social institutions can also facilitate such
comparative evaluations (e.g., Imperial 1999; Kenney and Lord 1999; Ostrom et al.
1994). Another approach is to break a collaborative process into its component parts
and evaluate the parts separately. This method allows for comparison among col-
laborative efforts with like components, even if other aspects of the projects differ
(Patton 1986).
Comparing Collaborative Efforts to Theories
Taking a deductive rather than inductive approach, some researchers have compared
collaborative efforts to criteria derived from a theoretical construct such as com-
municative rationality (Duane 1997) or participatory democracy (C arr and Hal-
vorsen 2001; Moote et al. 1997). This type of comparison allows the evaluator to
look beyond the participants’ goals and examine broader concepts and questions. It
is built on the assumption that if the features called for by a theory are present,
beneficial outcomes will be forthcoming (e.g., Duane 1997; Innes 1996). However,
such assertions are only as good as the theory upon which they are based, and these
links between theory and outcomes are rarely proven and subject to controversy. For
example, according to many theories of collaboration, inclusion of all stakeholder
groups is essential. Yet few collaborative groups meet this standard, and many have
apparently achieved considerable success without it. Likewise, many theories
emphasize the need for consensus decision making (where no participant opposes a
group decision), yet many apparently successful collaborative efforts rely on dif-
ferent decision rules.
378 A. Conley and M. A. Moote
Inversely, evaluations themselves may also be used to test theories, as in
Coglianese’s (1999) use of an evaluatory case study to challenge the theoretical claim
that consensus decision making leads to better decisions. In these cases, the evaluator
investigates the strength of the association between a theory’s predictions and actual
outcomes. Such efforts to use evaluations to test theory will be most effective when
they adhere to princi ples of theory testing and experimental design such as trian-
gulation of methods and the use of refutable hypotheses.
Evaluations can also be used to constr uct theory. One interesting example of this
is Moseley’s (1999) use of case studies to develop the idea that certain kinds of social
and organizational capacity must already be in place in a given community if
institutional efforts to promote collaboration are to take root there. Others have
studied collaborative efforts that are identified a priori as ‘‘successes’’ in an attempt
to understand the factors that make that case a success, so that it can be replicated
elsewhere (e.g., Kusel and Adler 2001). At their heart, such studies strive to build
causal theories of successful colla boration.
Evaluation Methods
Traditionally, evaluators have used quasi-experimental methods and multivariate
statistical analyses to correlate outcomes with project characteristics or, ideally,
identify and test cause-and-effect relationships between project characteristics and
outcomes (e.g., Bingham and Felbinger 1989; Patton 1986). Such methods requ ire
large sample sizes of comparable entities and have difficulty accommodating the
complex and dynamic nature of collaborative efforts and their contexts (Chen
and Rossi 1987; Yin 1992). When such methods have been used to evaluate colla-
borative processes they have typically been based on structured surveys, as discussed
later.
Inductive and in-depth evaluation methods, particularly ethographic approa-
ches, have gained credibility over tim e because they allow for consideration of
complex interactions between variables and can be adapted when either external
variables or the internal process changes. Participant observation, focus groups and
workshops, document analysis, and interviews are typically used to generate the
‘‘rich’’ data favored in ethnographic evaluations.
Ultimately, the best method for evaluating a collaborative process will depend
on the questions being asked, the scale of the evaluation, and available resources.
Here, we categorize evaluation methods according to three broad classes: measuring
tangible outcomes, measuring participant perceptions, and participant observation.
While we address each separately, in many cases evaluators will want to use mult iple
methods and triangulate results to increase the validity of findings.
Measuring Tangible Outcomes
Outcome evaluation typicall y involv es comparing actual project or program out-
comes with desired outcomes. Documenting outcomes is easiest when they are
readily quantified, and where there is sufficient baseline information to allow reliable
comparisons over time or between cases. Measurable outcomes may be social,
economic, or behavioral, and typicall y include indicators such as include number of
acres treated, employment trends, numbers of lawsuits or appeals filed, and demand
for various public assistance programs. These may focus on fairly easily defined
Evaluating Collaboration 379
short-term outcomes, which facilitates the process of evaluation, or on longer term
outcomes like ecological health and commun ity well-being.
The Holy Grail for many is an evaluation showing that collaborative efforts
improve a landscape’s ecological health (Kenney et al. 2000). Evaluating biophysical
outcomes typically relies on sampling to measure progress toward clearly defined,
quantifiable objectives. Methods exist for sampl ing a wide array of natural resources
(Goldsmith 1991; Spellerberg 1991), but significant challenges exist in applying these
methods to evaluating collaborative natural resource management. The first is the
previously noted problem of identifying measurable goals. Second, the variability
inherent in ecological data, combined with the long time frame required for ecolo-
gical changes to occur, makes identifying trends difficult. This challenge is exacer-
bated in cases that lack long-term data from a carefully designed monitoring
program. Finally, making causal links between specific management activities and
ecological trends is often problematic, as it is difficult if not impossible to isolate
variables. In most cases such an evaluation is far in the future, and will only be
possible where baseline surveys exist and conditions are regularly monitored. Similar
challenges face those trying to evaluate socioeco nomic outcomes like community
well-being or economic sustainability.
Outcome evaluations are often seen as more objective than those based on
participant’s opinions (discussed later), but they give little insight into perceptual
factors like mutual learning, perceived fairness of the process or outcome, and
conflict abatement. Outcome evaluation has also been called a ‘‘black box’’ method,
because it often does not allow evaluators to determine which variables caused the
outcome (Patton 1986).
Measuring Participants’ Perceptions
Perhaps the most common data-gathering method used to evaluate collaborative
conservation is to ask participants about them. Typically, surveys or semistructured
interviews ask respondents to identify and assess an effort’s outcomes, the factors
that led to those outcomes, and the appropriateness of the processes used. The
simplest surveys focus on a single effort at one point in time (e.g., Daniels and Walker
1996; Harmon 1999), but surveys that look at numerous cases are becoming
increasingly common (e.g., Carr et al. 1998; Kenney et al. 2000; Leach 2000; Paulson
1998; Selin et al. 2000; Susskind et al. 2000; Williams and Ellefson 1997). Participant
evaluations are used to identify stakeholder attitudes, opinions, and relationships;
reduced conflicts between parties; increases in social cap ital; and other social changes.
Single-shot surveys and interviews have been criticized for failing to capture
changes in perspectives over time. Longitudinal studies can address this weakness by
surveying people before, during, and after they participate in a collaborative effort.
This measures both participants’ opinions about the process and its outcomes, the
way those opinions change over the course of the process, and, if adequate controls
are used, the degree to which the collaborative process is responsible for those
changes. In many cases, participants themselves may not have noticed or articulated
such changes.
Common variations on the participant-survey approach include group self-
assessment activities such as focus-group discussions, group ranking exercises to
rate outcomes and alternatives, developing flowcharts of project impacts, and
mapping both landscape features and abstract concepts (as in Venn diagrams of
relationships between different organizations) (Chambers 1997). These are often
380 A. Conley and M. A. Moote
combined into participatory evaluation workshops used for internal program
reviews, but have potential for other applications (Innes and Booher 1999; Jackson
and Kassam 1999).
Data based on people’s perceptions are often seen as less appropriate for mea-
suring tangible outcomes, due to their subjectivity and reliance on respon dant’s
memories. The participant survey approach has also been criticized because results
are limited to the perspectives of those who participate in the study. This is especially
problematic in studies looking at num erous cases where it is logistically necessary to
limit the number of informants per case. As Freeman House (1999) notes in his book
on collaboration in the Mattole watershed, ‘‘Talk to anyone who’s been involved in
the community endeavo r described here and you’ll hear a whole different set of
stories and, likely, a whole different interpretation of what they might mean.’’ Leach
(2000) found that assessments based on the perceptions of group coordinators dif-
fered substantially from those based on responses of all group members, and that
different groups of stakeholders could differ significantly in their assessments of the
same efforts, indicating that evaluators should strive to include the views of as wide a
cross section of participants as possible. Most surveys have also ignored the opinions
of parties who did not directly participate in a process but were affected by it. Where
resources allow, identifying and surveying affected nonparticipants (and lapsed
participants) may address this shortcoming.
Process Evaluation
Evaluations that ask process questions like how well a project or program is func-
tioning, how participants are recruited, or how decisions are made require that
evaluators ‘‘become intimately acquainted with the details’’ of a collaborative effort
(Patton 1986). In-depth interviews and participant observation are preferred process
evaluation methods, although systems analysis, an iterative method that involves
mapping interrelationships between components of the natural, social, and built
environment, also holds promise.
Participant observation is distinguis hed by the role of the evaluator, who has
extended contact with participants (on the order of months or years) (e.g., Duane
1997; Moote et al. 1997; Moseley 1999; Smith 1999). Such studies are often con-
ducted by participants in the collaborative process or researchers taking an
anthropological approach. Participant observation is favored by many evaluators
because it provides the richest data on both process and context characteristics and
permits in-depth analysis of the relationships linking process variables to outcomes.
Participant obs ervation is favore d in inductive research and is well suited to theory
building, but can be quite time-consuming.
Coordinating Research Efforts
In-depth case studies are favored by many researchers because they permit extensive
analysis of any number of context and process characteristics. Unfortunately, case
studies are time-consuming and evaluators typicall y tackle at most two or three at a
time. This limits the generalizability of their results. The widespread desire to address
broader questions indicates the obvious need for evaluations based on larger sam-
ples. These can be obtained through surveys of multiple cases or by conducting meta-
analysis of existing case studies.
Surveys of like cases can provide analyses with generalizable results, but they are
not always the best tool: They may fail to account for characteristics not included in
Evaluating Collaboration 381
the survey and, when applied to numerous cases, they are typically completed by at
best a few respondents for each case. Where typically the individual respondent is the
unit of analysis, most questions about collaborative efforts treat the gro up or effort
as the unit of analysis. This challenge can be addressed by surveying a representative
sample of participants and nonparticipants for each case, but doing so greatly
increases the logistical challenges. Some evaluators have used surveys in conjunction
with more in-depth analysis of a few of the cases surveyed (e.g., Paulson 1998), which
allows for more informed interpretation of the survey results.
For many evaluative questions, meta-analysis of existing case studies may prove
to be more effective. When data to address specific questions are not extant in
existing case studies, coordinated case studies conducted by research teams may be
needed to develop a sufficiently large sample. A surprising amount of evaluation
work—including hundreds of case studies—already exists, but it remains spread
through many disciplines with much buried in the gray literature. Meta-analyses can
synthesize findings from these studies, but legitimate meta-analysis requ ires com-
parable data. Here again we see the importance of making research questions, bases
for comparison, criteria and their wei ghtings, and research methods explicit. Meta-
analysis is most effective when the que stions asked are clearly defined and relatively
narrow so that they apply to specific components of cases rather that can readily be
broken out and compared. To date, few such meta-analyses exist, although Leach
and Pelkey (2001) and Kenney (2000) are beginnings.
Conclusion
As proponents of collaborative approaches to resource management, we are
unnerved by the ways in which these processes have been portrayed as a cure-all. We
are similarly troubled by knee-jerk criticisms of collaborative processes that are
based on an opposition to collaboration in principle rather than evaluation of
specific processes and outcomes. Thoughtful evaluation of the effectiveness of dif-
ferent collaborative processes is central to understanding what can and cannot be
expected of such processes and how they can be integrated with existing institutions.
Certain forms of evaluation will play key roles. Parti cipatory evaluations driven
by collaborative efforts themselves are needed to determine progress toward goals,
provide feedback to guide future actions, and identify larger scale issues that impact
specific efforts. Surveys, coordinated case studies, and meta-analyses can play an
important role in illuminating these larger scale issues and are best used to address
specific questions with broad import for policy-making and management. Building
networks connecting researchers, participants in collaborative efforts, policymakers
and critics will great ly facilitate identifying relevant research questions and applying
the results to management. Detailed case studies also have a role to play in devel-
oping theory about collab orative efforts and identifying specific issues and dynamics
that warrant further study. Despite their popularity, efforts to evaluate the ‘‘success’’
of collaborative approaches in general and to develop cookbooks for collaboration
are likely to be less useful.
Developing truly objective means of evaluating collaborative efforts is
impossible. This said, if evaluators make explicit their motives for an evaluation,
criteria used and their relative weightings, and data collection methods, we can
compare, synthesize, and learn from them. Such synthesis is the next step in
addressing the many questions being asked about collaborative natural resource
management.
382 A. Conley and M. A. Moote
References
Anderson, E. W., and R. C. Baum. 1988. How to do coordinated resource management
planning. J. Soil Water Conserv. 43(3):216–220.
Baun, R., B. Baker, and K. Johnson. 1996. Sustainable communities checklist. Seattle, WA:
University of Washington, Graduate School of Public Affairs, Northwest Policy Center.
Bellamy, J. A., G. T. McDonald, G. J. Syme, and J. E. Butterworth. 1999. Evaluating inte-
grated resource management. Society Nat. Resources 12:337–353.
Bernard, T., and J. Young. 1997. The ecology of hope: Communities collaborate for sustain-
ability. Gabriola Island, BC: New Society.
Bingham, G. 1986. Resolving environmental disputes: A decade of experience. Washington, DC:
Conservation Foundation.
Bingham, R. D., and C. L. Felbinger. 1989. Evaluation in practice: A methodological approach.
New York: Longman.
Blumberg, L. 1999. Preserving the public trust. Forum Appl. Res. Public Policy 14(2):89–93.
Blumberg, L., and D. Knuffke. 1998. Count us out: Why the Wilderness Society opposed the
Quincy Library Group legislation. Chron. Commun. 2(2):41–44.
Blumenthal, D., and J. L. Jannink. 2000. A classification of collaborative management
methods. Conserv. Ecol. 4(2):13.
Born, S. M., and K. D. Genskow. 2000. The watershed approach: An empirical assessment of
innovation in environmental management. Washington, DC: National Academy of Public
Administration.
Born, S. M., and W. C. Sonzogni. 1995. Integrated environmental management: Strengthening
the conceptualization. Environ. Manage. 19(2):167–181.
Braxton Little, J. 1997. The Feather River Alliance: Restoring creeks and communities in the
Sierra Nevada. Chron. Commun. 2(1):5–14.
Brendler, T., and H. Carey. 1998. Community forestry, defined. J. For. 96(3):21–23.
Brick, P. D., D. Snow, and S. B. Van de Wetering, eds. 2000. Across the great divide:
Explorations in collaborative conservation in the American West. Washington, DC: Island
Press.
Brosius, J. P., A. L. Tsing, and C. Zerner. 1997. Representing communities: Histories and
politics of community-based natural resource management. Society Nat. Resources
11:157–168.
Brunson, M. W. 2000. Observing vs. doing: A researcher learns about collaboration. Chron.
Commun. 4(2):47–52.
Buckle, L. G., and S. R. Thomas-Buckle. 1986. Placing environmental mediation in context:
Lessons from ‘‘failed’’ mediations. Environ. Impact Assess. Rev. 6(1):55–70.
Bureau of Land Management. 1995. If the mountain could speak: A story of collaboration.
Phoenix, AZ: Bureau of Land Management National Training Center. Video.
Carr, D. S., and K. Halvorsen. 2001. An evaluation of three democratic, community-based
approaches to citizen participation: Surveys, conversations with community groups, and
community dinners. Society Nat. Resources 14:107–127.
Carr, D. S., S. W. Selin, and M. A. Schuett. 1998. Managing public forests: Understanding the
role of collaborative planning. Environ. Manage. 22(5):767–776.
Cestero, B. 1999. Beyond the hundredth meeting: A field guide to collaborative conservation on
the West’s public lands. Tucson, AZ: Sonoran Institute.
Chambers, R. 1997. Whose reality counts? Putting the first last. London: Intermediate Tech-
nology.
Chen, H.-T., and P. H. Rossi. 1987. The theory-driven approach to validity. Eval. Program
Plan. 10:95–103.
Chisholm, G. 1996. Tough towns: The challenge of community-based conservation. In A wolf
in the garden: The land rights movement and the new environmental debate, eds. P. D. Brick
and R. M. Cawley, 279–292. Landham, MD: Rowan and Littlefield.
Evaluating Collaboration 383
Clark, J. 1997. Watershed partnerships: A strategic guide for local conservation efforts in the
West. Denver, CO: Western Governors’ Association.
Cleary, C. R., and D. Phillippi. 1993. Coordinated resource management: Guidelines for all who
participate. Denver, CO: Society for Range Management.
Coggins, G. C. 1998. Regulating federal natural resources: A summary case against devolved
collaboration. Ecol. Law Q. 25(4):602–610.
Coglianese, C. 1997. Assessing consensus: The promise and performance of negotiated rule-
making. Duke Law J. 46(6):1255–1340.
Coglianese, C. 1999. The limits of consensus. Environment 41(3):28–33.
Coughlin, C. W., M. L. Hoben, D. W. Manskopf, and S. W. Quesada. 1999. A systematic
assessment of collaborative resource management partnerships. Master’s project, School of
Natural Resources, University of Michigan, Ann Arbor.
Daniels, S. E., and G. B. Walker. 1996. Collaborative learning: Improving public deliberation
in ecosystem-based management. Environ. Impact Assess. Rev. 16:71–102.
Daniels, S. E., and G. B. Walker. 2000. Working through environmental policy conflicts: The
collaborative learning approach. New York: Praeger.
d’Estree, T. P., and B. G. Colby. 2000. Guidebook for analyzing success in environmental
conflict resolution cases. Fairfax, VA: Institute for Conflict Analysis and Resolution,
George Mason University.
Dryzek, J. 1996. Political and ecological communication. In Ecology and democracy, ed.
F. Matthews, 13–30. Portland, OR: Frank Cass.
Duane, T. P. 1997. Community participation in ecosystem management. Ecol. Law Q.
24(4):771–797.
Dukes, E. F., and K. Firehock. 2001. Collaboration: A guide for environmental advocates.
Charlottesville: University of Virginia, The Wilderness Society, and National Audubon
Society.
Farrell, A., and M. Hart. 1998. What does sustainability really mean? Environment 49(9):4–9,
26–31.
Gericke, K. L., and J. Sullivan. 1994. Public participation and appeals of Forest Service plans:
An empirical examination. Society Nat. Resources 7(2):125–135.
Goldsmith, F. B., ed. 1991. Monitoring for conservation and ecology. New York: Chapman and
Hall.
Gray, B. 1989. Collaborating: Finding common ground for multiparty problems. San Francisco,
CA: Jossey-Bass.
Gray, G. J., M. J. Enzer, and J. Kusel, eds. 2001. Understanding community based ecosystem
management in the United States. New York: Haworth Press.
Harmon, W. 1999. Montana group tries scorecard approach. Consensus 30(1):3, 7.
Harrington, M., and C. A. Hartwell. 1999. Rivers among us: Local watershed preservation and
resources management in the Western United States. Los Angeles, CA: Reason Public Policy
Institute.
House, F. 1999. Totem salmon: Life lessons from another species. Boston: Beacon Press.
Imperial, M. T. 1999. Analyzing institutional arrangements for ecosystem-based management.
Environ. Manage. 24:449–465.
Imperial, M. T., and T. Hennessey. 2000. Environmental governance in watersheds: The role of
collaboration. Paper read at 8th Biennial Conference of the International Association for
the Study of Common Property, 1 June, Bloomington, IN.
Innes, J. E. 1996. Planning through consensus building: A new view of the comprehensive
planning ideal. Am. Plan. Assoc. J. 62(4):460–472.
Innes, J. E. 1999. Evaluating consensus building. In The consensus building handbook:
A comprehensive guide to reaching agreement, ed. L. Susskind, S. McKearnan, and
J. Thomas-Larmer, 631–675. Thousand Oaks, CA: Sage.
Innes, J. E., and D. E. Booher. 1999. Consensus building and complex adaptive systems:
A framework for evaluating collaborative planning. Am. Plan. Assoc. J. 65(4):413–423.
384 A. Conley and M. A. Moote
Jackson, E. T., and Y. Kassam, eds. 1999. Knowledge shared: Participatory evaluation in
development cooperation. West Hartford, CT: Kumarian Press.
Jones, L. 1996. Howdy neighbor! As a last resort, Westerners start talking to each other. High
Country News 28(9):1, 6–8.
Kellert, S. R., J. N. Mehta, S. A. Ebbin, and L. L. Lichtenfeld. 2000. Community natural
resource management: Promise, rhetoric and reality. Society Nat. Resources 13:705–715.
KenCairn, B. 1996. Peril on common ground: The Applegate experiment. In A wolf in the
garden: The land rights movement and the new environmental debate, ed. P. D. Brick and
R. M. Cawley, 261–278. Lanham, MD: Rowan and Littlefield Publishers.
KenCairn, B. 1998. Criteria for evaluating community-based conservation=natural resources
partnership initiatives. In A report from Troutdale: Community-based strategies in forest
stewardship and sustainable economic development, 34–40. San Francisco, CA: Consultative
Group on Biological Diversity.
Kenney, D. S. 1999. Are community-based watershed groups really effective? Confronting the
thorny issue of measuring success. Chron. Commun. 3(2):33–37.
Kenney, D. S. 2000. Arguing about consensus: Examining the case against Western watershed
initiatives and other collaborative groups active in natural resources management. Boulder:
Natural Resources Law Center, University of Colorado.
Kenney, D. S., and W. B. Lord. 1999. Analysis of institutional innovation in the natural resources
and environmental realm. Boulder: Natural Resources Law Center, University of Colorado.
Kenney, D. S., S. T. McAllister, W. H. Caile, and J. S. Peckham. 2000. The new watershed
source book. Boulder: Natural Resources Law Center, University of Colorado.
Krist, J. 1998. Seeking common ground. Ventura County Star, 14–23 December.
Kusel, J., and E. Adler, eds. 2001. Forest communities, community forests: A collection of case
studies of community forestry. Taylorsville, CA: Forest Community Research.
Leach, W. D. 2000. Evaluating watershed partnerships in California: Theoretical and metho-
dological perspectives. PhD dissertation, Department of Ecology, University of California,
Davis, Davis.
Leach, W. D., and N. W. Pelkey. 2001. Making watershed partnerships work: A review of the
empirical literature. J. Water Resources Plan. Manage. 127(6):378–385.
Leach, W. D., N. W. Pelkey, and P. A. Sabatier. 2000. Conceptualizing and measuring success
in collaborative watershed partnerships. Paper read at 2000 Annual Meeting of the Amer-
ican Political Science Association, Washington, DC, 29 August.
Lead Partnership Group. 2000. The Lead Partnership Group identifies principles of com-
munity-based forestry. Lead Partnership Group Newslett. IV(1):1–2.
Margerum, R. D. 1999. Integrated environmental management: The foundations for suc-
cessful practice. Environ. Manage. 24(2):151–166.
Mazaika, R. 1999. The Grande Ronde model watershed program: A case study. Admin.
Theory Praxis 21(1):62–75.
McCloskey, M. 1998. Local communities and the management of public forests. Ecol. Law Q.
25(4):624–629.
Moote, A., A. Conley, K. Firehock, and F. Dukes. 2000. Assessing research needs: A summary
of a workshop on community-based collaboratives. Tucson: Udall Center for Studies in
Public Policy, University of Arizona.
Moote, M. A. 1996. The partnership handbook, vol. 2000. Tucson: Water Resources Center,
University of Arizona.
Moote, M. A., M. P. McClaran, and D. K. Chickering. 1997. Theory in practice:
Applying participatory democracy theory to public land planning. Environ. Manage.
21(6):877–889.
Moseley, C. 1999. New ideas, old institutions: Environment, community and state in the Pacific
Northwest. PhD dissertation, Department of Political Science, Yale University, New
Haven, CT.
National Academy of Public Administration. 2001. Evaluating environmental progress: How
EPA and the states can improve the quality of enforcement and compliance information.
Evaluating Collaboration 385
A report by a panel of the National Academy of Public Administration. June. http:==www.
napawash.org=pc_economy_environment=learning_innovations.html (accessed 2=26=2002).
National Association of State Foresters. 1997. Forests for a sustainable future: The use of
criteria and indicators in sustainable forest management. Washington, DC: National
Association of State Foresters.
Natural Resources Law Center. 1996. The watershed source book. Boulder: Natural Resources
Law Center, University of Colorado.
Ostrom, E., R. Gardner, and J. Walker. 1994. Rules, games and common-pool resources. Ann
Arbor: University of Michigan Press.
Patton, M. Q. 1986. Utilization-focused evaluation. Beverly Hills, CA: Sage.
Paulson, D. D. 1998. Collaborative management of public rangeland in Wyoming: Lessons in
co-management. Prof. Geographer 50(3):301–315.
Paulson, D. D., and K. M. Chamberlin. 1998. Guidelines and issues to consider in planning
a collaborative process. Laramie: Institute for Environment and Natural Resources,
University of Wyoming.
Poisner, J. 1996. Essays: A civic republican perspective on the National Environmental Policy
Act’s process for citizen participation. Environ. Law 26:53–94.
Selin, S., and D. Chavez. 1995. Developing a collaborative model for environmental planning
and management. Environ. Manage. 19(2):189–195.
Selin, S. W., M. A. Schuett, and D. Carr. 2000. Modeling stakeholder perceptions of colla-
borative initiative effectiveness. Society Nat. Resources 13:735–745.
Shelly, S. 1998. Making a difference on the ground: Colorado’s Ponderosa Pine Partnership
shows how it can be done. Chron. Commun. 3(1):37–39.
Sipe, N. G. 1998. An empirical analysis of environmental mediation. J. Am. Plan. Assoc.
64(3):275–285.
Smith, M. 1999. The Catron County Citizens’ Group: A case study in community colla-
boration. In The consensus building handbook, ed. L. Susskind, S. McKearnan and
J. Thomas-Larmer, 985–1009. Thousand Oaks, CA: Sage.
Snow, D. 1998. Some lines cast from troutdale. In A report from troutdale, 9–21. San Fran-
cisco, CA: Consultative Group on Biodiversity.
Southern Utah Wilderness Association. 1994. Why one advocacy group steers clear of con-
sensus efforts. High Country News 26(10).
Spellerberg, I. F. 1991. Monitoring ecological change. New York: Cambridge University Press.
Susskind, L., S. McKearnen, and J. Thomas-Larmer, eds. 1999. The consensus building
handbook. Thousand Oaks, CA: Sage.
Susskind, L., M. van der Wansem, and A. Ciccarelli. 2000. Mediating land use disputes: Pros
and cons. Cambridge, MA: Lincoln Institute of Land Policy.
U.S. Environmental Protection Agency. 1997. Community-based environmental protection:
A resource book for protecting ecosystems and communities. Washington, DC: U.S. EPA.
Weber, E. 2000. A new vanguard for the environment: Grass-roots ecosystem management as
a new environmental movement. Society Nat. Resources 13(3):237–259.
Western, D., and R. M. Wright, eds. 1994. Natural connections: Perspectives in community-
based conservation. Washington, DC: Island Press.
Williams, E. M., and P. V. Ellefson. 1997. Going into partnership to manage a landscape.
J. For. 95(5):29–33.
Wondolleck, J. M., and S. L. Yaffee. 1994. Building bridges across agency boundaries: In search
of excellence in the United States Forest Service. Ann Arbor: University of Michigan School
of Natural Resources and Environment.
Wondolleck, J. M., and S. L. Yaffee. 2000. Making collaboration work: Lessons from inno-
vation in natural resource management. Washington, DC: Island Press.
Yaffee, S. L., A. F. Phillips, I. C. Frentz, P. W. Hardy, S. M. Maleki, and B. E. Thorpe. 1996.
Ecosystem management in the United States. Washington, DC: Island Press.
Yin, R. K. 1992. The case study method as a tool for doing evaluation. Curr. Sociol.
40(1):121–137.
386 A. Conley and M. A. Moote
