Evaluating ecological outcomes from environmental stewardship initiatives: A comparative analysis of approaches

Abstract

Understanding the extent to which stewardship initiatives achieve objectives of enhanced ecological outcomes is important for enhancing effectiveness and efficiency of environmental management initiatives. Alternative approaches – community science, stakeholder perceptions, and remote sensing - are emerging in lieu of the conventional approach of collecting field data that present different benefits and drawbacks and to date have not been directly compared. This research compared the use of four approaches to evaluating ecological outcomes of a grassland restoration project on a 2 ha Niagara Parks Commission property in Ontario, Canada. We collected three levels of quantitative data, from general site assessments to species-specific data using standardized questionnaires and multi-spectral imagery from a remotely piloted aircraft system. We found that community scientists and stakeholders provided comparable general site assessments to the field data, but that as the assessments became more detailed, differences emerged. Further, remotely sensed data were assessed and provided a more positive site assessment than any other method. Experiences and knowledge of nature did not influence assessments by community scientists or stakeholders. Our findings show that for overall site assessments, community scientists and stakeholders may be able to provide a reasonably accurate assessment. If monitoring and evaluation needs (either research-based or practical) extend beyond a broad assessment, use of a field expert or multiple methods of data collection may be warranted.

Full text

Journal of Environmental Management 297 (2021) 113266
Available online 16 July 2021
0301-4797/© 2021 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
Evaluating ecological outcomes from environmental stewardship
initiatives: A comparative analysis of approaches
Julia Baird
a
,
b
,
*
, Ryan Plummer
a
, Marilyne Jollineau
a
, Gillian Dale
a
a
Environmental Sustainability Research Centre Brock University, St. Catharines, ON, L2S 3A1, Canada
b
Geography and Tourism Studies, Brock University, St. Catharines, ON, L2S 3A1, Canada
ARTICLE INFO
Keywords:
Environmental stewardship
Evaluation
Monitoring
Community science
Remote sensing
Perception
ABSTRACT
Understanding the extent to which stewardship initiatives achieve objectives of enhanced ecological outcomes is
important for enhancing effectiveness and efciency of environmental management initiatives. Alternative ap-
proaches – community science, stakeholder perceptions, and remote sensing - are emerging in lieu of the con-
ventional approach of collecting eld data that present different benets and drawbacks and to date have not
been directly compared. This research compared the use of four approaches to evaluating ecological outcomes of
a grassland restoration project on a 2 ha Niagara Parks Commission property in Ontario, Canada. We collected
three levels of quantitative data, from general site assessments to species-specic data using standardized
questionnaires and multi-spectral imagery from a remotely piloted aircraft system. We found that community
scientists and stakeholders provided comparable general site assessments to the eld data, but that as the as-
sessments became more detailed, differences emerged. Further, remotely sensed data were assessed and provided
a more positive site assessment than any other method. Experiences and knowledge of nature did not inuence
assessments by community scientists or stakeholders. Our ndings show that for overall site assessments,
community scientists and stakeholders may be able to provide a reasonably accurate assessment. If monitoring
and evaluation needs (either research-based or practical) extend beyond a broad assessment, use of a eld expert
or multiple methods of data collection may be warranted.
1. Introduction
Improving ecological conditions has long been the reason for envi-
ronmental management. Evaluating ecological outcomes from envi-
ronmental management initiatives, including the conservation of
biodiversity, water protection, environmental restoration, and other
objectives (Marttunen et al., 2019), is essential to gauge their effec-
tiveness. Evaluation provides information on the degree to which
anticipated results are achieved, and affords opportunities for learning,
adjustment, and adaptation (Plummer and Armitage, 2007; Carpenter
et al., 2009). Despite its importance and many benets, evaluation of
ecological outcomes is rare (Pullin and Knight, 2001; Hein et al., 2017;
McIntosh et al., 2018). Intensifying calls have been made for more and
better evaluation of ecological outcomes. Twenty years ago, Pullin and
Knight (2001) noted the consistent lack of monitoring and evaluation of
conservation actions as a problem. Ferraro and Pattanayak (2006) urged
those in the eld of conservation policy to not only evaluate ecological
outcomes more often, but to adopt state-of-the-art program evaluation
methods to better determine what works and when. Compelling argu-
ments have been made for evidence-based decision making (Carpenter
et al., 2009; Pullin et al., 2013) as well as evaluative frameworks for
environmental management that explicitly include ecological outcomes
(Plummer and Armitage, 2007). Statements that more and better eval-
uation of ecological outcomes are needed in research and practice also
continue to gain prominence in the environmental stewardship litera-
ture (e.g., Hein et al., 2017; Plummer et al., 2017a). Whereas universal
agreement is evident as to the need for better evaluation of ecological
outcomes, approaches to assessing them are varied and contested.
The success of environmental management interventions has
conventionally been assessed using quantitative eld data to evaluate
ecological outcomes (Conley and Moote, 2003; Ferraro and Pattanayak,
2006). This includes the assessment of collected eld data (e.g.,
evidence-based conservation; Pullin and Knight, 2001; Legge, 2015), use
of quantitative measures and counterfactuals (Ferraro and Pattanayak,
2006; Duncan and Reich, 2016), and the collection of baseline and
ongoing monitoring data (Leach et al., 2002; Likens and Lindenmayer,
* Corresponding author. Environmental Sustainability Research Centre Brock University, St. Catharines, ON, L2S 3A1, Canada.
E-mail address: jbaird@brocku.ca (J. Baird).
Contents lists available at ScienceDirect
Journal of Environmental Management
journal homepage: www.elsevier.com/locate/jenvman
https://doi.org/10.1016/j.jenvman.2021.113266
Received 12 March 2021; Received in revised form 25 June 2021; Accepted 9 July 2021

Journal of Environmental Management 297 (2021) 113266
2
2010). In general, eld data are considered more objective and accurate
in comparison to alternatives (Koontz and Thomas, 2006); however, in
many cases, the challenges associated with collecting eld data preclude
its use (Leach et al., 2002; Webb et al., 2004; Adam and Sandbrook,
2013). Some of the many challenges associated with the use of eld data
include time lags, issues with attribution, and the difculties associated
with trying to understand complex causal links and feedbacks (Margo-
luis et al., 2009; Biddle and Koontz, 2014; Carranza et al., 2014; Guerry
et al., 2015; Plummer et al., 2017a), particularly in the context of
emerging novel climate and ecosystem conditions (Margoluis et al.,
2009; Guerry et al., 2015). Additional challenges of cost (Margoluis
et al., 2009; Biddle and Koontz, 2014; Guerry et al., 2015; Plummer
et al., 2017a), technical expertise (Biddle and Koontz, 2014; Willis,
2015), accessibility (Wiens et al., 2009), incomplete and/or inadequate
data (Guerry et al., 2015; Plummer et al., 2017a), and scientic uncer-
tainty in outcome monitoring and modeling (Gillon et al., 2016)
frequently compound the situation.
Intensifying calls for evaluation have given rise to questions about
approaches to evaluate ecological outcomes (Conley and Moote, 2003;
Plummer et al., 2015). Whereas the collection of quantitative eld data
is the conventional approach to evaluation in natural resource man-
agement (Conley and Moote, 2003; Ferraro and Pattanayak, 2006),
alternative approaches to evaluating ecological outcomes from envi-
ronmental initiatives have emerged and large bodies of literature have
been developed around them, especially those emphasizing collabora-
tion and learning. In particular, community science and stakeholder
perceptions have been increasingly applied to measure ecological out-
comes (Conrad and Hilchey, 2011; Bennett, 2016; Gelcich and O’Keeffe,
2016). Additional alternatives come from technological advances. The
use of geospatial technologies, especially remote sensing, offers a tool
for this purpose (Wiens et al., 2009; Corbane et al., 2015; Zhou et al.,
2017; van lersel et al., 2018; Arasumani et al., 2021). These approaches:
community science, stakeholder perceptions, and the use of remote
sensing, while not universally employable depending on the context of
the environmental management initiative, represent three well-studied,
alternative approaches to the collection of eld data by an expert
assessor, and have not been subject to a direct comparison to date. A
description of each approach follows, with potential promises as well as
concerns highlighted.
1.1. Community science
Community science (often referred to as ‘citizen’ science
1
) is a
method that brings together scientic data collection with public
outreach, usually by engaging amateurs (Cohn, 2008; Devictor et al.,
2010). The use of community scientists provides the opportunity to
gather data on a larger geographic area, quickly and over a longer time
period, and with the additional benet of reduced costs, relative to more
traditional scientic research (Selin et al., 2000; Galloway et al., 2006;
Cohn, 2008; Lehtiniemi et al., 2020). For these reasons, and with the aid
of ever-improving technology (Dickinson et al., 2010), community sci-
ence continues to grow in popularity and expand in scope (Cohn, 2008;
Dickinson et al., 2010; Gardiner et al., 2012), especially in Canada and
the USA (Conrad and Hilchey, 2011). Community scientists have been
engaged in a range of data collection projects including Christmas Bird
Counts and e-bird, the UK Buttery Monitoring Scheme, and the Ap-
palachian Mountain Watch (Devictor et al., 2010).
Despite increasing popularity, community scientists’ ability to
accurately monitor and evaluate ecological conditions/outcomes is un-
certain (Conrad and Hilchey, 2011) and sometimes contested (Bedessem
and Ruphy, 2020). Studies testing the accuracy of community science
data collection have had varying results based on a number of factors
including the type of data being collected, the training volunteers
received, and the presence and type of data screening, among others
(Galloway et al., 2006). For example, while Delaney et al. (2008) found
that education level was a highly signicant predictor of a volunteer’s
ability to collect accurate data, Caneld et al. (2002), Galloway et al.
(2006), and Crall et al. (2011) found that volunteers performed as well
as experts in some cases. Concerns have also been expressed about the
use of community science as a way to advance a particular position on an
issue (e.g., by non-governmental organizations) (Bedessem and Ruphy,
2020). The potential of community science to be used instead of
expert-led quantitative eld data collection for the evaluation of
ecological outcomes of environmental stewardship is an open question,
as is how it compares to other alternative approaches.
1.2. Stakeholder perceptions
Stakeholder perceptions, or stakeholder evaluations (i.e., collecting
data from those familiar with environmental stewardship initiatives),
are often used as a proxy for measured outcomes (Selin et al., 2000;
Yasu´
e et al., 2010; Leleu et al., 2012; Bennett and Dearden, 2014;
Plummer et al., 2017b). The use of stakeholder evaluations has been
defended as a legitimate and effective means to evaluate ecological
outcomes of stewardship efforts including conservation initiatives and
policies (Bennett, 2016; Gelcich and O ’Keeffe, 2016), integrated coastal
management and marine protected areas (Webb et al., 2004; Gruby
et al., 2017), environmental governance (Armitage et al., 2017), and
adaptive co-management (Plummer et al., 2017a). Specically, stake-
holders who are considered knowledgeable about the outcomes of ini-
tiatives, based on participation in and learning from said initiatives, are
thought to be best situated to make these assessments (Webb et al.,
2004; Armitage et al., 2017; Plummer et al., 2017a). Webb et al. (2004)
argued that resource-accessing stakeholders may have a considerable
depth of knowledge about resource abundance, ecology, and manage-
ment. However, while perceptions are used and defended as a valid data
source for ecological outcomes, the ability of stakeholders to accurately
assess ecological conditions is disputed by others. Perceptions are sub-
jective (Conley and Moote, 2003; Bennett and Dearden, 2014), some-
times reliant on respondents’ memories (Conley and Moote, 2003), and
may be biased in multiple ways. For example, the halo effect refers to
articially inating environmental outcomes to make them seem more
positive because respondents have experienced positive social in-
teractions in a collaborative setting (Leach and Sabatier, 2005; Koontz
and Thomas, 2006; Mandarano, 2008). The opposite can also be true,
where perceptions of ecological outcomes can be negatively affected by
unfavourable experiences (Bennett and Dearden, 2014). Furthermore,
the stakeholder’s level of engagement may inuence perceptions of
outcomes, with greater frequency of engagement leading to more posi-
tive perceptions (Plummer et al., 2017a, 2017b). Although stakeholders
may be very knowledgeable, Lund et al. (2010) assert that there is no
guarantee of local consensus on that knowledge. For example, Gilchrist
et al. (2005) found that the accuracy of information provided by Inuit
people on marine bird population changes varied among interviewees
and that their quantitative assessments were generally inaccurate.
1.3. Remote sensing
Remote sensing has been dened as the “science and art of obtaining
information about an object, area, or phenomenon through the analysis
of data acquired by a device that is not in contact with the object, area,
or phenomenon under investigation” (Lillesand et al., 2015:1). It rep-
resents a useful tool for measuring ecological outcomes given that it can
be used to acquire accurate, reliable, non-invasive, and cost-effective
baseline ecological information that can be used to monitor changes
over time (Wiens et al., 2009; Alcaraz-Segura et al., 2009a; Townsend
et al., 2009; Corbane et al., 2015; Willis, 2015). Baseline ecological in-
formation that can be measured using remote sensing typically includes
1
Here we use the term ‘community science’ to include those who may not be
citizens but engage in these types of activities.
J. Baird et al.

Journal of Environmental Management 297 (2021) 113266
3
vegetation condition, type, phenology, structure, and biophysical pa-
rameters (Turner et al., 2003). This information is increasingly used to
track progress toward ecological outcomes of a range of stewardship
initiatives (Corbane et al., 2015; Zhou et al., 2017; van lersel et al., 2018;
Arasumani et al., 2021).
While permitting exible monitoring across a variety of spatial and
temporal scales, this tool also permits data acquisition in locations that
may otherwise be inaccessible because of their ownership status or
geographic, social, or political nature (Wiens et al., 2009). Challenges in
using the technology include the scale at which outcomes are being
evaluated, the lack of technical expertise available for image and data
interpretation, and the need for further validation of acquired data
through ground-based data (Turner et al., 2003; Macauley, 2006; Wiens
et al., 2009; Field et al., 2010; Mayer and Lopez, 2011). Despite these
challenges, remote-sensing data have proven to be very useful for
measuring ecological outcomes (Wiens et al., 2009; Corbane et al., 2015;
Zhou et al., 2017; Arasumani et al., 2021).
2. Objectives
The strengths and weaknesses associated with each of the ap-
proaches described above have provided the basis for lively scholarly
debate. Most signicant is deliberation as to the accuracy (e.g., Koontz
and Thomas, 2007; Likens and Lindenmayer, 2010), validity (e.g., Wiens
et al., 2009), and reliability (e.g., Tiner, 2004; Wiens et al., 2009; Ben-
nett, 2016) of such alternatives to quantitative eld data for evaluating
ecological outcomes. However, there are no studies that have used
empirical data to compare the accuracy of these approaches in a
comparative study at the same site. In response, this research critically
examines and compares four approaches to evaluating ecological out-
comes (quantitative eld data, stakeholder perceptions, community
science, and remote sensing) from an environmental initiative case
study of a grassland restoration project by the Niagara Parks Commis-
sion. We focus specically on their objectives of assessing vegetation
and presence of two bird species of interest.
3. Methods
3.1. Study site
Data collection took place at the Centennial Lilac Garden in Niagara
Falls, Canada managed by the Niagara Parks Commission (Fig. 1). In
2008, the Lilac Garden site was converted into a habitat for grassland-
dependent birds (see Fig. 2). This 2 ha (ha) habitat contains a mix of
native wildowers and prairie grasses of varying heights, and is
designed to support and protect threatened grassland-dependent bird
species, such as the Bobolink (Dolichonyx oryzivorus) and the Eastern
Meadowlark (Sturnella magna), as well as a variety of pollinator species
(Burant, pers. com.). The site is regularly maintained to reduce the
proliferation of invasive vegetation, and an assessment of the func-
tionality of the habitat, the quality of the observed vegetation, and the
number of bird and pollinator species present is conducted annually. As
such, the site provides the ideal setting for the current study.
3.2. Participants
A total of 36 participants, recruited from within local nature and
conservation groups, completed this study. Recruitment was conducted
using the research team’s and Niagara Parks Commission’s existing
connections to those with an interest in environmental issues in the
region (e.g., leaders and organizers of nature and birding groups) and
requesting broader circulation of a call for participation from those in-
dividuals to their networks. Participants were recruited from April 2019
to July 2019. All participants conrmed that they were 18 years of age
Fig. 1. Location of the study area (2 ha) and numbered sampling quadrats in the Centennial Lilac Garden in Niagara Falls, Canada. These quadrats are located
approximately 100 m apart.
J. Baird et al.

Journal of Environmental Management 297 (2021) 113266
4
or older, were uent in English, and were comfortable walking over
uneven terrain. Additionally, each participant provided written,
informed consent. All participants were entered in a draw for one of two
$500 gift cards to a local outdoor shop.
Prior to the data collection phase of the study, participants were
asked to self-report their level of expertise with a) bird identication and
b) native and invasive vegetation identication using a short survey
querying level and years of experience after they signed their consent
form. Based on their responses, 17 participants were classied as “ex-
perts” (i.e., at least 3 years of experience with bird and/or vegetation
identication), and 19 participants were classied as “non-experts”.
Participants were then placed into either the “community scientist”
group (N =20; 10 experts) or the “stakeholder” group (N =16; 7 ex-
perts) based on both their self-reported expertise and their availability
for the selected data collection dates. Each group contained a mix of
both experts and non-experts. The difference between these two groups
(community science and stakeholder perceptions) was in the amount of
training they received prior to their site assessments and the amount of
time and guidance they received when on-site.
In addition to the community scientist and stakeholder groups, eld
data was collected from an expert in vegetation and wildlife assessment.
The eld expert was hired by Niagara Parks to conduct their annual site
inventory and survey. In addition to providing us with their assessment
data, the eld expert agreed to complete a questionnaire about the site
(see below). As with the other groups, the expert provided written,
informed consent prior to participating. This study was approved by the
Human Research Ethics Board at Brock University and conducted in
accordance with Tri-Council ethical guidelines.
3.3. Stimuli and Procedure
This study compared four methods for evaluating the ecological
outcomes of the grassland habitat restoration project: eld expert survey
and inventory, community scientist assessments, stakeholder percep-
tions, and a remote-sensing analysis. Each of these methods are dis-
cussed in turn below. The expert survey/inventory was conducted rst
during the Bobolink and Eastern Meadowlark breeding season (late June
through early July 2019) in accordance with the guidelines set forth by
the Ministry of Natural Resources. The community scientist and stake-
holder data collection took place on separate days (two days apart)
approximately one week after the expert had completed their analysis
(July 8th and 10th, 2019, respectively). Finally, the remote-sensing
evaluation was conducted on August 9th, 2019, four weeks after
completion of the community scientist and stakeholder data collection
as a result of logistical delays. All data collection was completed be-
tween the hours of 6am and 10am, except for the remote-sensing data
which was collected under clear sky conditions between 10 a.m. and 2 p.
m.
3.3.1. Field expert survey/inventory
Since the initial development of the Lilac Garden grassland habitat
restoration project, an annual evaluation and inventory has been con-
ducted by a eld expert to assess the overall success of the project. This
evaluation and inventory is conducted at three separate occasions dur-
ing the Bobolink and Eastern Meadowlark breeding season and captures
detailed information about the health of these threatened bird species (e.
g., number of males and females spotted or heard, location, number of
nests, number of nestlings and young, etc.), as well as information about
the vegetation growth and cover at the site (e.g., type of native and non-
native species present, height and density of the growth, percentage of
soil exposed, etc.). The eld expert, in addition to their regular evalu-
ation of the site, completed the same observation and evaluation exer-
cise described in the community scientist section below at each of the
three marked quadrats (i.e., Parts 1 and 2 of the questionnaire). In
addition, they completed Part 3 of the questionnaire (described above)
to provide a measure of the overall health of the site.
3.3.2. Community scientist assessment
At the beginning of the data collection session, participants assigned
to the community scientist group were provided with a 30-min training
seminar to familiarize them with the study protocol and the types of
vegetation, birds, and pollinators they could expect to see at the site.
Given that one of the primary reasons for the habitat restoration project
was to increase the numbers of Bobolink and Eastern Meadowlark in the
area, participants were provided with a detailed description of the
characteristics of each bird species, including their calls. Participants
were then provided with a comprehensive guidebook that explained,
and provided images of, the identiable characteristics of the Bobolink
and Eastern Meadowlark, as well the native and invasive vegetation that
had previously been identied at the site. Additionally, participants
received access to several tools (e.g., tweezers, magnifying glasses,
binoculars, wireless tablets loaded with applications for bird [Merlin
Fig. 2. Site photos, from upper left clockwise: Milkweed plant; study site with participants assessing a quadrat; a taped 1 m quadrat.
J. Baird et al.

Journal of Environmental Management 297 (2021) 113266
5
Bird ID] and vegetation [PlantNet] identication) in order to aid with
their analysis of the site. Participants were also permitted to bring along
any personal tools that they felt might be useful, such as binoculars,
cameras, guidebooks, etc.
Following the training session, the participants were bussed to the
habitat restoration site. Three 1 m ×1 m quadrats were selected by the
researchers in locations approximately evenly distributed throughout
the site and marked off (see Fig. 1 for locations of each quadrat). The
selection of these quadrats and their size was based on those originally
identied by the eld expert, who used an aerial photo survey of the site
to select points representative of the range of vegetation at the site. Plot-
based vegetation surveys (1 m
2
) are often used to assess the type,
occurrence, and abundance/coverage of plants in grassland and other
vegetated environments (e.g., Lopatin et al., 2017). To reduce the
amount of damage to surrounding vegetation, and to provide partici-
pants with the opportunity to carefully explore the quadrats, the par-
ticipants were divided into teams. Each team was led by a member of the
research group to one of the three marked quadrats (assigned at random
but coordinated so that at all times only one team was at a quadrat) and
were instructed to explore the vegetation within the quadrat. Partici-
pants were verbally asked to ignore any vegetation where the stems
were not contained within the quadrat. In addition, participants were
asked to quietly observe the bird species in the surrounding area. During
this exercise, participants were instructed to ll out a questionnaire (see
Appendix A) about the species they had observed. Participants were
instructed to avoid interacting with each other during the assessment,
until they submitted their responses to the researchers.
Part 1 of the questionnaire asked participants to assess the presence
of Bobolink and Eastern Meadowlark from the viewpoint of the quadrat.
In addition, participants were asked to indicate any other bird species
that they had observed in the area by indicating both the species (or a
description if unknown) and the number of each species observed. Part 2
consisted of a list of 26 vegetation species (both native and non-native)
that had been identied at the site by the previous year’s eld expert.
Participants were instructed to indicate a) whether each species was
present within the quadrat, b) the percentage of the quadrat that was
comprised of each species (if present), and c) the average height (in
inches or centimeters) of each species within the quadrat (if present).
Participants were also asked to indicate what percentage of the quadrat
they were able to identify and to write down the names of any vegetation
that were not included in the list of 26.
After 20 min of observation, the teams were led to a new quadrat to
repeat the exercise. The data collection exercise was completed once
each team had had a chance to assess all three quadrats. At the
conclusion of the assessment exercise, participants were instructed to ll
out Part 3 of the questionnaire. This section asked them to rate a) the
bird diversity, b) the vegetation diversity, c) the vegetation density, and
d) the presence of invasive species using a 4-point Likert scale common
in ecological research (White et al., 2005) with the scale adapted from
ecological integrity monitoring methods in Canada’s national parks
(Parks Canada, 2011) (1 =poor; 4 =very good). Additionally, using the
same scale participants were asked to rate a) the overall condition of the
Bobolink and Eastern Meadowlark population, b) the overall condition
of the vegetation at the site, and c) the overall ecological health of the
site. Lastly, participants indicated the percentage of soil that was
exposed at the site (0–25% =completely covered by vegetation;
76–100% =completely exposed soil).
3.3.3. Stakeholder perceptions
‘Stakeholders’ were treated as if they were an interested site visitor,
mimicking to the extent possible the conditions under which, for
example, a birder or naturalist might experience the site. Since the
grassland restoration site was not visible from the road nor from any
major tourist attractions, and given the purpose of the site (to create
habitat for two specic bird species) we anticipate that an interested site
visitor would likely be someone with an interest in birds or nature.
Accordingly, while our recruitment method for the stakeholder group
did not mimic a usual visit, the visitors themselves were very likely
similar in type and how they might experience the site. Unlike the
community scientists, stakeholders were not provided with training or
materials related to the site. However, they did receive access to twee-
zers, magnifying glasses, and binoculars if they requested these tools
from the research team. In addition, participants were permitted to
bring along any personal tools that they felt might be useful, such as
binoculars, cameras, guidebooks, etc.
The participants were divided into teams once they arrived at the
site. Each team was led by a member of the research group to one of the
three marked quadrats. There, they were instructed to simply observe
both the bird species that were within view of the quadrat and the
vegetation within the marked quadrat (ignoring any vegetation where
stems were outside of it). Participants were instructed to avoid inter-
acting with each other during the assessment, until they submitted their
responses to the researchers. After 10 min of passive observation, the
teams were led to a new quadrat to repeat the exercise. After partici-
pants had an opportunity to passively observe each of the three quad-
rats, they were asked to provide their overall perceptions of the site
health by lling out Part 3 of the questionnaire described above. Addi-
tionally, participants were given the option to complete the vegetation
checklist from Part 2 of the community scientist questionnaire (for the
entire site, rather than separately for each quadrat as with the com-
munity scientists), and to list any bird species that they had observed
during the trip. While a casual visitor may not have specically focused
on quadrats in multiple locations in the site, this adjustment to
mimicking a real visit was needed for comparability among approaches.
3.3.4. Remote sensing
Multispectral ve-band MicaSense imagery with 5.3 cm spatial res-
olution were acquired by a remotely piloted aircraft system (RPAS) at an
altitude of 200 feet above ground level over the eld site on August 9,
2019. Radiometric calibration of these data was undertaken using the
default radiometric calibration model deployed by Pix4D Mapper Pro
(Pix4D Inc., Denver CO, USA; www.pix4d.com) and outlined by Mica-
Sense (MicaSense Inc., Seattle, WA, USA; www.micasense.com). This
model uses the radiometric calibration criterion formula provided by
MicaSense (2021). Recent assessments of the accuracy and reliability of
this sensor and calibration method found strong linear relationships
with laboratory and eld-based measurements of spectral reectance
(Cao et al., 2019). Using a global positioning system, ground control
points were subsequently acquired from the three quadrats simulta-
neously with an accuracy of 25 cm.
Once these data were preprocessed, a normalized difference vege-
tation index (NDVI) was calculated given that it has proven useful in
providing a quantitative measure of vegetation density and the overall
health of grassland vegetation (e.g., Xu et al., 2014; Zhou et al., 2017;
van lersel et al., 2018). NDVI is calculated from spectral reectance
values measured in the red and near-infrared regions of the EMS using
the formula (NDVI =NIR – red/NIR +red). Values for this index range
from - 1 (non-vegetated surfaces) to +1 (healthy vegetation). NDVI was
used in this study as it has been correlated with vegetation character-
istics, including such as the quantity, condition, percent cover, pro-
ductivity, and has been used as a descriptor of ecosystem function
(Asner, 1998; Alcaraz-Segura et al., 2009b). An unsupervised classi-
cation technique (i.e., k-means clustering) was also used in this study to
group unknown pixels with similar spectral reectance characteristics
into a number of unique spectral classes. Given that geolocated infor-
mation for individual plant species within each quadrat were not
available, this clustering technique was preferrable since it required
minimal input from the image analyst (Richards and Jia, 1999). Other
studies have also successfully used this clustering algorithm to identify
grassland vegetation types (e.g., Lopatin et al., 2017).
Using geolocated quadrat information identied for other partici-
pants in this study, two remote-sensing image-analysis techniques were
J. Baird et al.

Journal of Environmental Management 297 (2021) 113266
6
used to complete questions in Part 3 of the community scientist ques-
tionnaire (described above). Information derived from the application of
a NDVI was used to respond to questions related to vegetation density,
vegetation condition, overall vegetation and site health, and soil expo-
sure. An unsupervised k-means clustering algorithm (using ve bands of
multispectral RPAS data) was used to extract information on the di-
versity of vegetation at the site. The algorithm parameters (i.e., numbers
of clusters, threshold, and number of iterations) were varied, with
satisfactory results being attained using this technique. Given the coarse
spectral resolution of these data (similar to Landsat-7 band designa-
tions), the distinction between invasive and non-invasive plant species
could not be made.
4. Results
4.1. Overall means
The mean ratings and standard errors for the ecological health
measures can be found in Fig. 3a, b, and 3c as a function of group
(community scientist vs. stakeholder) and expertise (expert vs. non-
expert). Overall (combined across groups) bird population health rat-
ings ranged from 1 (poor) to 4 (very good), with a mean rating of 1.31
(SD =0.66). Overall (combined across groups) vegetation health ratings
ranged from 1 to 4, with a mean rating of 3.02 (SD =0.72). Finally,
overall (combined across groups) site health scores ranged from 1 to 4,
with a mean rating of 2.85 (SD =0.69). In addition to the mean ratings
for the community scientist and stakeholder groups, Fig. 3abc also dis-
plays the overall ratings from the eld expert and the ratings derived
from the remote-sensing data (where available). The eld expert rated
the bird population health as a 1 out of 5, overall vegetation health as a
3/5, and overall site health as 3/5. The remote-sensing data were used to
derive an overall vegetation health rating of 4/5, and an overall site
health rating of 4/5.
In addition to the individual measures of site health, we also asked
participants to assess the amount of soil coverage at the site. All in-
dividuals rated the soil coverage the same (0–25% of soil exposed), with
no deviations, thus this variable was not explored further.
We also measured how many different types of vegetation were
successfully identied by our groups, as compared to the eld expert
who identied 16 different types of vegetation across all three quadrats
(see Appendix B). The community scientists successfully identied be-
tween 2 and 9 different types of vegetation, with a mean of 5.76 (SD =
1.52). The stakeholders successfully identied between 0 and 10
different types of vegetation, with a mean of 3.34 (SD =2.71). Specic
vegetation type (species-level data) were not available using the remote-
sensing method.
4.2. Comparisons between community scientists and stakeholders
Next, we conducted a series of 2 (group) x 2 (expertise) factorial
ANOVAs in order to examine whether the scores on the various
ecological health ratings differed as a function of group (community
scientist vs. stakeholder) and expertise (expert vs. non-expert). There
was a signicant main effect of group (but not expertise) for the number
of vegetation species successfully identied, F (1,20) =2.40, p =.02,
η
ρ
2 =0.23, such that the community scientists successfully identied
more species of vegetation than the stakeholders. In addition, the rela-
tionship between group and overall vegetation health ratings
approached signicance, F (1,20) =4.07, p =.057,
η
ρ
2 =0.17, such that
the community scientists provided lower mean ratings of overall vege-
tation health as compared to the stakeholders. There were no other
signicant main effects of either group or expertise (all F’s <3.53), and
no interactions between group and expertise (all F’s <2.04).
4.3. Comparisons with eld data and remote sensing
To further explore how our four evaluation methods differed from
one another, we completed a series of one-sample t-tests with a Bon-
ferroni correction to compare a) the community scientists and stake-
holders to the eld data scores, and b) the community scientists and
stakeholders to the remote-sensing scores (where available). Only the
signicant relationships are discussed below (see Fig. 5 for a breakdown
of the relative strengths and weaknesses of each of the four methods).
Comparisons were also made between the number of vegetation
types identied by the eld expert and remote sensing, using the results
of the k-means clustering algorithm (where the classication parameters
were set to ve bands of data, eight clusters, and 20 iterations of the
algorithm). The total number of vegetation types within each quadrat
were ve (with three predominant vegetation types), eight (two pre-
dominant), and eight (four predominant). Quadrat 2 (Fig. 1) exhibited
the greatest diversity while quadrat 1 was the most homogeneous. The
identication of ve to eight vegetation types per quadrat, in addition to
their predominance within each quadrat, was consistent with observa-
tions made by the eld expert. The timing of remote-sensing data
acquisition, preprocessing, and subsequent data analyses, combined
with a lack of information about species location within each quadrat,
precluded validation of these results in the eld.
4.3.1. Community scientists
Community scientists (regardless of expertise) signicantly differed
from the eld expert in their ratings of bird diversity, t (17) =3.70, p =
.002, such that the community scientists provided higher diversity rat-
ings. They did not differ from the eld expert for any other ratings,
however. The mean community scientist ratings signicantly differed
from the remote-sensing rating for vegetation diversity, t (18) = − 6.08,
p <.001, and vegetation density, t (16) = − 4.39, p <.001, such that the
community scientists had a lower mean rating for both variables. In
addition, the community scientist ratings of both overall vegetation
health, t (18) = − 8.60, p <.001, and overall site health, t (18) = − 7.57,
p <.001, signicantly differed from the remote-sensing rating, such that
the community scientist ratings were lower.
4.3.2. Stakeholders
Stakeholders (regardless of expertise) signicantly differed from the
eld expert in their ratings of the presence of invasive species, t (12) =
−4.43, p =.001, such that the stakeholders rated the presence of inva-
sive species as signicantly lower than did the eld expert. They did not
differ from the eld expert for any other ratings.
As with the community scientists, the mean stakeholder ratings
signicantly differed from the remote sensing rating for vegetation di-
versity, t (15) = − 5.58, p <.001, and vegetation density, t (15) = − 3.47,
p =.003, such that the stakeholders reported a lower mean rating for
both variables. Lastly, the stakeholder ratings of both overall vegetation
health, t (14) = − 3.60, p =.003, and overall site health, t (14) = − 6.00,
p <.001, signicantly differed from the remote sensing rating, such that
the stakeholder ratings were lower.
4.4. Correlations among health variables
Next, we examined how the four sub-variables of ecological health (i.
e., bird diversity, vegetation diversity, vegetation density, and invasive
species) related to our three overall measures of ecological health (i.e.,
overall bird population health, overall vegetation health, and overall
site health). There were no signicant correlations between overall bird
population health and any other variable (all p’s >0.05). As expected,
both vegetation diversity (r =0.59, p <.001) and vegetation density (r
=0.53, p =.001) were signicantly correlated with overall vegetation
health. Finally, bird diversity (r =0.38, p =.03), vegetation diversity (r
=0.51, p =.002), and vegetation density (r =0.53, p =.001) were
signicantly correlated with overall site health. No other correlations
J. Baird et al.

Journal of Environmental Management 297 (2021) 113266
7
Fig. 3. Means and standard deviations for the bird
diversity and invasive species measures as a function
of group (community scientist; stakeholder; eld
expert) and expertise. Error bars represent the stan-
dard error for each condition mean. * =signicant
difference between means at p <.05. Fig. 3b. Means
and standard deviations for the vegetation diversity
and vegetation measures as a function of group
(community scientist; stakeholder; eld expert;
remote sensing) and expertise. Error bars represent
the standard error for each condition mean. * =sig-
nicant difference between means at p <.05. Fig. 3c.
Means and standard deviations for the overall bird
population health, overall vegetation health, and
overall site health measures as a function of group
(community scientist; stakeholder; eld expert;
remote sensing) and expertise. Error bars represent
the standard error for each condition mean. * =sig-
nicant difference between means at p <.05.
J. Baird et al.

Journal of Environmental Management 297 (2021) 113266
8
were signicant (all p’s >0.05).
4.5. Predicting ecological health
Two separate stepwise multiple regressions were performed in order
to examine which variables best predicted a) overall vegetation health
ratings, and b) overall site health rating. In the rst regression, all four
sub-variables of ecological health were entered as predictors, and
overall vegetation health was entered as the criterion. Overall, the
model explained a signicant 29.8% of the variance in overall vegeta-
tion health, R =0.546; F (1, 27) =11.38, p =.002, with vegetation
diversity emerging as the only signicant unique predictor (sr
2
=0.298).
In the second regression, all four sub-variables of ecological health were
entered as predictors, and overall site health was entered as the crite-
rion. Overall, the model explained a signicant 18.5% of the variance in
overall ecological health, R =0.430; F (1, 27) =6.18, p =.02, with
vegetation diversity emerging again as the only signicant unique pre-
dictor (sr
2
=0.185).
4.6. Relative costs of approaches
Finally, we examined the relative costs and time (as compared to the
costs/time for the eld expert, which was considered a single ‘unit’ of
cost’) for each of our methods in order to determine the costs-benets of
each approach (see Fig. 4). Costs for the community science and stake-
holder perceptions were high as a result of the research design requiring
group transportation rental, sustenance for participants, and purchasing
materials to support their assessment in sufcient quantity. The
remotely sensed data were collected and analyzed by a third party and
this cost may vary depending on the equipment (hardware and software)
and expertise available. Further, substantive time was required for study
planning and recruitment that would differ from practical application of
these methods. Overall, in this study, the community scientist and
stakeholder approaches were numerically similar with respect to both
costs and time, whereas the remote-sensing approach was more time-
effective but signicantly more expensive.
5. Discussion
The accuracy of the four approaches to site assessment showed
similarities and differences. All four approaches addressed three levels
of detail, ranging from overall site assessment to species-level. At the
broadest level (i.e., overall site health, overall bird health, and overall
vegetation health assessments), the eld expert, community scientists,
and stakeholders provided relatively similar assessments. While some
nuances to these results are evident, there were no signicant differ-
ences among assessment approaches. However, remote sensing provided
a more positive assessment than the other three approaches for overall
site and vegetation health. At the mid-level of detail for assessments –
with vegetation health divided into diversity and density and prevalence
of invasive species, and bird health divided into population density and
species diversity – there were more differences among approaches. Here,
community scientists and stakeholders still tended to provide similar
assessments, but there were some deviations between these approaches
and the assessment provided by the eld expert. Community scientists
tended to overestimate bird diversity and stakeholders tended to un-
derestimate the presence of invasive species, as compared to the eld
expert. For the variables that remotely sensed data could contribute
(vegetation diversity and density), these data were interpreted as
overestimations of the vegetation health in comparison to the assess-
ments from all other approaches. Finally, at the most granular (species)
level of vegetation and bird identication, no approach produced an
assessment similar to that of the eld expert.
There was substantial diversity in the ability of community scientists
and stakeholders to identify the same vegetation species as the eld
expert. While inter-observer variability is a common feature of data
collected from non-professionals (e.g., Abadie et al., 2008; Dickinson
et al., 2010), 10/16 correctly identied species was the closest from any
individual participant. The coarse spectral resolution of the remotely
sensed data (ve bands of multispectral data) could not provide an
assessment at this level, with dominant vegetation in the site impacting
the ability to distinguish other vegetation nearer to the ground as well as
the lag between other approaches and collection of remotely sensed data
(approximately 4 weeks). The lack of alignment between the eld expert
and other approaches signal that the latter are not particularly effective
at this granular level. Other studies comparing ndings of assessments at
different levels of granularity show similar ndings. For example, Crall
et al. (2011) found that trained volunteers (similar to community sci-
entists in our study) were less accurate than professionals at the
mid-level assessments, but equally accurate at the coarse assessment
level. Abadie et al.’s (2008) use of untrained volunteers to collect
mid-level assessment data on vegetation diversity similarly found the
need for some ‘naturalist skills’. Assessments at a ner level of detail,
however, show mixed results. For example, Galloway et al. (2006) found
that trained schoolchildren (grades 3–10) could accurately collect some
tree stand measures, but subjective assessments were signicantly
Fig. 4. Time and monetary costs for each method relative to the eld expert data method. One unit of time is equal to the overall time cost for the eld expert data
method. One unit of nancial cost is equal to the overall nancial cost for the eld expert data method.
J. Baird et al.

Journal of Environmental Management 297 (2021) 113266
9
different from professionals. Accordingly, our ndings align with and
extend previous ndings by considering three levels of granularity
(coarse, mid, and ne).
The assessments at the coarsest level (vegetation health and
ecological health) were predicted only, and strongly, by vegetation
density, explaining nearly 20–30% of the variance in assessments. This
indicates that it is only vegetation diversity – of the variables we
measured – that inuenced coarse site assessments. This nding is
particularly interesting and worth exploring in more detail, to under-
stand to what extent coarse site assessments could be improved through
training or guidance to focus on specic site attributes beyond vegeta-
tion density.
The characteristics of the approaches appear related to some of the
similarities and differences identied in the accuracy of the assessments.
There is a gradient of training that existed within the study design, from
the eld expert (with a high degree of ecological knowledge and
training, and specic expertise related to the site conditions) to the
community scientists (provided with training and additional informa-
tional materials) to the stakeholders (provided with no specic training
or information except what they chose to bring with them). The remote-
sensing approach does not align with this gradient, however. While the
analysts had little specic knowledge of the site, they hold a high degree
of expertise in interpreting and analyzing the data collected.
While the training gradient that was implemented throughout the
study design showed differences among approaches (particularly when
comparing community scientists to stakeholders), such that individuals
who were provided with more training and information were more ac-
curate when identifying vegetation species, the differences among in-
dividuals (those with expertise vs non-experts) showed no consistent,
meaningful differences in their assessments. This signals the potential
for even minimal training to support a common level of assessment
between those with pre-existing expertise and those without. Other
studies have demonstrated differences between experienced and novice
participants (e.g., Johnston et al., 2018; Ali et al., 2019). One possible
explanation of the difference in our ndings from others is that our
experienced participants were primarily experts in birding rather than
vegetation identication may have led to the lack of difference between
experts and non-experts in our study.
Importantly, the current case study tells us a that the usefulness of
the approach chosen for monitoring and evaluation of environmental
initiatives will depend on the objectives of the assessment. If a general
site assessment is needed, then the alternative approaches tested (i.e.,
community science, stakeholder perceptions, remote sensing) may be
sufcient, and indeed provide benets beyond the assessment itself. For
example, community science may provide social learning benets.
Employing stakeholder and site visitor perceptions may promote
cognitive learning (Land-Zandstra et al., 2015), shape attitudes towards
the subject of their assessment (Phillips et al., 2021), instill deeper
meaning to participants’ activities and hobbies, and enhance social
well-being (Bonney et al., 2015). If circumstances do not permit, or are
not ideal, for human entry into a site, remotely sensed data may provide
an alternative and provides the additional benet of a whole site
perspective which is otherwise very difcult to obtain (Kennedy et al.,
2009; Wiens et al., 2009; Corbane et al., 2015). Further, recent studies
suggest that remotely sensed data (including RPAS data) supplemented
with eld expertise provide complementary data for deriving environ-
mental information for evaluating ecological outcomes and reporting
conservation status in grassland environments (Corbane et al., 2015;
Arasumani et al., 2021).
The costs associated with these approaches may appear very high;
however, the costs are a function of the study design and may be sub-
stantially reduced in practice (e.g., stakeholder perceptions could be
passively collected through site visits with extremely minimal cost;
remotely sensed data could be accessed through publicly available, free
images). If assessments require specic detail at the species level (bird or
vegetative) present and their prevalence within a site, expert assessment
may be required and costs subject to the requirements. Other approaches
tended to show a wide range of assessment abilities (or require pro-
hibitively expensive hyperspectral instruments to collect data at this
level in the case of remote sensing).
Can non-eld expert assessments compare with a eld expert for the
purposes of accuracy, validity, and reliability (e.g., Koontz and Thomas,
2006; Wiens et al., 2009; Adams and Sandbrook, 2013; Bennett, 2016)?
The indications from this single case study suggest that, at a broad level,
there is evidence to support community science and stakeholder per-
ceptions as proxies for eld expert assessment. However, if needs are
such that specic details are required about a site (e.g., prevalence of
invasive species, presence, and density of a specic species) then
collaboration or coordination with associated experts appears impor-
tant. These ndings are not generalizable, but they are aligned with
ndings from other studies. For example, Edwards et al. (2018) suggest
that community scientists can collect meaningful data but that their
efforts should be augmented by professional engagement. Calls for using
multiple methods for data collection and validation are evident in the
remote sensing literature as well (e.g., Lausch et al., 2018).
There are limitations to this study that should be noted. First, the
timing at which assessments were conducted varied among approaches.
The community science, stakeholder, and eld expert approaches were
all collected within a two-week timeframe, whereas the remotely sensed
data were collected approximately four weeks later. We acknowledge
that the delay, particularly in collecting remotely sensed data, may have
had some impact on the results. Second, this is a single case and further
research in other contexts and with larger samples is needed. Third, the
focus of this site was on grassland habitat restoration for two bird species
Fig. 5. Relative strengths and weaknesses of each
evaluative approach. Each approach was subjectively
ranked relative to the others for each category (e.g.,
eld data had the greatest accuracy for use with bird
data, and thus was ranked #1, whereas remote
sensing had the poorest accuracy for bird data and
was thus ranked #4). Dark green circles represent the
best approach for a given category, followed by pale
green, then yellow, and nally orange (i.e., the
poorest approach for a given category). For the
specicity category, the left side of the circle repre-
sents the accuracy for gross estimates of ecological
health (e.g., overall health of site), whereas the right
side of the circle represents the accuracy for ne-
grained estimates of ecological health (e.g., presence
of specic vegetation). (For interpretation of the ref-
erences to colour in this gure legend, the reader is
referred to the Web version of this article.)
J. Baird et al.

Journal of Environmental Management 297 (2021) 113266
10
and thus represents a specic focus on a terrestrial, grassland landscape
in one location in southern Ontario.
6. Conclusions
This study examined how eld expert assessments of ecological
health of an environmental stewardship initiative compare to alterna-
tive approaches and represents the rst comparison of this kind. A
grassland habitat restoration project by the Niagara Parks Commission
provided an ideal site for this case study. Four approaches were
compared: eld expert, community science, stakeholder perceptions,
and remote sensing. Analysis of the remotely sensed data was signi-
cantly more positive (or optimistic) about the ecological health of the
site than other approaches given that a NDVI yielded pixel values that
typically exceeded 0.75, indicating that the majority of the study area
(over 90%) contained dense, healthy vegetation. It is important to note
that remote-sensing technologies provide access to information (i.e.,
near-infrared reectance) not available to the human eye. As a result,
the remote-sensing data may offer a more accurate assessment of overall
site condition.
Assessments were collected at three levels (overall site assessments,
mid-level assessments, and ne-grained specic level assessments). Ul-
timately, we found that eld experts, community scientists, and stake-
holders did not signicantly differ in their assessments at the coarsest
level (overall site assessment). However, as assessments increased in
detail (mid-level and ne-grained), differences among these three ap-
proaches became apparent. Participants’ prior experience in nature (e.
g., birding, active involvement in nature clubs) did not affect the nd-
ings; however, we anticipate that the focus of expertise specically on
birds and not vegetation may have played a role in this nding. Our
ndings provide evidence that for overall site assessments, community
scientists and stakeholders may be able to provide a relatively accurate
assessment. If monitoring and evaluation needs (either research-based
or practical) extend beyond a coarse assessment, a eld expert may
provide a more detailed assessment and multiple methods of data
collection may be warranted.
There are several future research directions for this line of inquiry.
Expansion of the study design to multiple sites is desirable to provide
additional data to assess the degree of alignment between eld experts
and other approaches to ecological site health assessments. Further,
assessments across different landscape types would be valuable (e.g.,
aquatic sites, forested landscapes). Third, although not assessed in the
current study, incorporation of the social benets of community-based
approaches would enhance future efforts to more fully understand
how the approaches that incorporate community participation have a
broader impact, and greater focus on the social sciences in citizen sci-
ence studies has been called for (Heiss and Matthes, 2017). Finally, we
recommend pursuing longitudinal studies of this type, where assess-
ments are conducted across multiple years to understand whether
assessment by individuals are relatively consistent in their alignment
with eld experts over time. Together, these research trajectories hold
much promise in identifying the reliability and accuracy of alternative
approaches to eld experts for monitoring and evaluating environ-
mental stewardship initiatives in practice and research design.
Author contribution
Julia Baird: Conceptualization, Methodology, Investigation, Re-
sources, Writing – original draft, Writing – review & editing, Supervi-
sion, Project administration, Funding acquisition; Ryan Plummer:
Conceptualization, Methodology, Writing – original draft, Writing –
review & editing, Funding acquisition; Marilyne Jollineau: Conceptu-
alization, Methodology, Formal analysis, Investigation, Data curation,
Resources, Writing – original draft, Writing – review & editing, Visual-
ization, Supervision, Funding acquisition; Gillian Dale: Methodology,
Investigation, Validation, Data curation, Writing – original draft,
Writing – review & editing, Visualization.
Declaration of competing interest
The authors declare that they have no known competing nancial
interests or personal relationships that could have appeared to inuence
the work reported in this paper.
Acknowledgements
We would like to thank Edward Anyan for producing the carto-
graphic illustrations included in this paper. We thank Seyi Obasi,
Samantha Witkowski, Angela Mallette, Todd Robinson, Nicole Rice,
Francisco Amaya, and Daniel Marshall for their research assistance.
Special thanks to Jacob Nederend at Deveron UAS for his assistance with
RPAS data collection and to Gordon Robert for his help in preprocessing
and analyzing these data. Finally, we appreciate research support from
Tom Staton, and gratefully acknowledge the in-kind contributions of the
Niagara Parks Commission, especially Corey Burant and Ellen Savoia,
through the Excellence in Environmental Stewardship partnership be-
tween the NPC and the Environmental Sustainability Research Centre,
Brock University. This work was supported by the Social Sciences and
Humanities Research Council of Canada [430-2018-01058] and Julia
Baird’s participation was supported, in part, by the Canada Research
Chairs program.
Appendix A. Supplementary data
Supplementary data to this article can be found online at https://doi.
org/10.1016/j.jenvman.2021.113266.
References
Abadie, J.C., Andrade, C., Machon, N., Porcher, E., 2008. On the use of parataxonomy in
biodiversity monitoring: a case study on wild ora. Biodivers. Conserv. 17,
3485–3500. https://doi.org/10.1007/s10531-008-9354-z.
Adams, W.M., Sandbrook, C., 2013. Conservation, evidence and policy. Fauna Flora Int.
47, 329–335. https://doi.org/10.1017/S0030605312001470.
Alcaraz-Segura, D., Cabello, J., Paruelo, J., 2009a. Baseline characterization of major
Iberian vegetation types based on the NDVI dynamics. Plant Ecol. 202, 13–29.
https://doi.org/10.1007/s11258-008-9555-2.
Alcaraz-Segura, D., Cabello, J., Paruelo, J.M., Delibes, M., 2009b. Use of descriptors of
ecosystem functioning for monitoring a national park network: a remote sensing
approach. Environ. Manag. 43, 38–48. https://doi.org/10.1007/s00267-008-9154-
y.
Ali, J.M., Noble, B.C., Nandi, I., Kolok, A.S., Bartelt-Hunt, S.L., 2019. Assessing the
accuracy of citizen scientist reported measurements for agrichemical contaminants.
Environ. Sci. Technol. 53 (10), 5633–5640. https://doi.org/10.1021/acs.
est.8b06707.
Arasumani, M., Bunyan, M., Robin, V., 2021. Opportunities and challenges in using
remote sensing for invasive tree species management, and in the identication of
restoration sites in tropical montane grasslands. J. Environ. Manag. 280, 111759.
https://doi.org/10.1016/j.jenvman.2020.111759.
Armitage, D., Dzyundzyak, A., Baird, J., Bodin, ¨
O., Plummer, R., Schultz, L., 2017. An
approach to assess learning conditions, effects and outcomes in environmental
governance. Environ Policy Gov. https://doi.org/10.1002/eet.1781.
Asner, G.P., 1998. Biophysical and biochemical sources of variability in canopy
reectance. Rem. Sens. Environ. 64, 234–253.
Bedessem, B., Ruphy, S., 2020. Citizen science and scientic objectivity: mapping out
epistemic risks and benets. Perspect. Sci. 28 (5), 630–654.
Bennett, N.J., Dearden, P., 2014. Why local people do not support conservation:
community perceptions of marine protected area livelihood impacts, governance and
management in Thailand. Mar. Pol. 44, 107–116. https://doi.org/10.1016/j.
marpol.2013.08.017.
Bennett, N.J., 2016. Using perceptions as evidence to improve conservation and
environmental management. Conserv. Biol. 30 (3), 582–592. https://doi.org/
10.1111/cobi.12681.
Biddle, J.C., Koontz, T.M., 2014. Goal specicity: a proxy measure for improvements in
environmental outcomes in collaborative governance. J. Environ. Manag. 145,
268–276. https://doi.org/10.1016/j.jenvman.2014.06.029.
Bonney, R., Phillips, T.B., Ballard, H.L., Enck, J.W., 2015. Can citizen science enhance
public understanding of science? Publ. Understand. Sci. 25 (1), 2–16. https://doi.
org/10.1177/0963662515607406.
Caneld, D.E., Brown, C.D., Bachmann, R.W., Hoyer, M.V., 2002. Volunteer lake
monitoring: testing the reliability of data collected by the Florida LAKEWATCH
J. Baird et al.

Journal of Environmental Management 297 (2021) 113266
11
Program. Lake Reservoir Manag. 18, 1–9. https://doi.org/10.1080/
07438140209353924.
Cao, S., Danielson, B., Clare, S., Koenig, S., Campos-Vargas, C., Sanchez-Azofeifa, A.,
2019. Radiometric calibration assessments for UAS-borne multispectral camera:
laboratory and eld protocols. ISPRS J. Photogrammetry Remote Sens. 149,
132–145. https://doi.org/10.1016/j.isprsjprs.2019.01.016.
Carpenter, S.R., Mooney, H.A., Agard, J., Capistrano, D., DeFries, R.S., Díaz, S., et al.,
2009. Science for managing ecosystem services: beyond the millennium ecosystem
Assessment. Source Proc Nat. Acad. Sci. United States Am. 106, 1305–1312.
Carranza, T., Manica, A., Kapos, V., Balmford, A., 2014. Mismatches between
conservation outcomes and management evaluation in protected areas: a case study
in the Brazilian Cerrado. Biol. Conserv. 173, 10–16. https://doi.org/10.1016/j.
biocon.2014.03.004.
Cohn, J.P., 2008. Citizen science: can volunteers do real research? Biosci. 58, 192–197.
https://doi.org/10.1641/B580303.
Conley, A., Moote, M.A., 2003. Evaluating collaborative natural resource management.
Soc. Nat. Resour. 16, 371–386. https://doi.org/10.1080/08941920390190032.
Conrad, C.C., Hilchey, K.G., 2011. A review of citizen science and community-based
environmental monitoring: issues and opportunities. Environ. Monit. Assess. 176,
273–291. https://doi.org/10.1007/s10661-010-1582-5.
Corbane, C., Lang, S., Pipkins, K., Alleaume, S., Deshayes, M., García Mill´
an, V.E.,
Strasser, T., Vanden Borre, J., Toon, S., Michael, F., 2015. Remote sensing for
mapping natural habitats and their conservation status – new opportunities and
challenges. Int. J. Appl. Earth Obs. Geoinf. 37, 7–16. https://doi.org/10.1016/j.
jag.2014.11.005.
Crall, A.W., Newman, G.J., Stohlgren, T.J., Holfelder, K.A., Graham, J., Waller, D.M.,
2011. Assessing citizen science data quality: an invasive species case study. Conserv.
Lett. 4, 433–442. https://doi.org/10.1111/j.1755-263X.2011.00196.x.
Delaney, D.G., Sperling, C.D., Adams, C.S., Leung, B., 2008. Marine invasive species:
validation of citizen science and implications for national monitoring networks. Biol.
Invasions 10, 117–128. https://doi.org/10.1007/s10530-007-9114-0.
Devictor, V., Whittaker, R.J., Beltrame, C., 2010. Beyond scarcity: citizen science
programmes as useful tools for conservation biogeography. Divers. Distrib. 16,
354–362. https://doi.org/10.1111/j.1472-4642.2009.00615.x.
Dickinson, J.L., Zuckerberg, B., Bonter, D.N., 2010. Citizen science as an ecological
research tool: challenges and benets. Annu. Rev. Ecol. Evol. Syst. 41, 149–172.
https://doi.org/10.1146/annurev-ecolsys-102209-144636.
Duncan, D., Reich, P., 2016. Controls and counterfactual information in agro-ecological
investment. In: Ansell, D., Gibson, F., Salt, D. (Eds.), Learning from Agri-
Environment Schemes in Australia. ANU Press, Acton, Australia, pp. 237–253.
Edwards, P.M., Shaloum, G., Bedell, D., 2018. A unique role for citizen science in
ecological restoration: a case study in streams. Restor. Ecol. 26, 29–35. https://doi.
org/10.1111/rec.12622.
Feld, C.K., Sousa, J.P., da Silva, P.M., Dawson, T.P., 2010. Indicators for biodiversity and
ecosystem services: towards an improved framework for ecosystems assessment.
Biodivers. Conserv. 19, 2895–2919. https://doi.org/10.1007/s10531-010-9875-0.
Ferraro, P.J., Pattanayak, S.K., 2006. Money for nothing? A call for empirical evaluation
of biodiversity conservation investments. PLoS Biol. 4, 482–488. https://doi.org/
10.1371/journal.pbio.0040105.
Galloway, A.W.E., Tudor, M.T., Matthew, W., Haegen, V., 2006. The reliability of citizen
science: a case study of Oregon White Oak Stand surveys. Wildl. Soc. Bull. 34,
1425–1429.
Gardiner, M.M., Allee, L.L., Brown, P., Losey, J.E., Roy, H.E., Smyth, R., 2012. Lessons
from lady beetles: accuracy of monitoring data from US and UK citizen-science
programs. Front. Ecol. Environ. 10, 471–476. https://doi.org/10.1890/110185.
Gelcich, S., O ’Keeffe, J., 2016. Emerging frontiers in perceptions research for aquatic
conservation. Aquat. Conserv. Mar. Freshw. Ecosyst. 26, 986–994. https://doi.org/
10.1002/aqc.2714.
Gilchrist, G., Mallory, M., Merkel, F., 2005. Can local ecological knowledge contribute to
wildlife management? Case studies of migratory birds. Ecol. Soc. 10, 20.
Gillon, S., Booth, E.G., Rissman, A.R., 2016. Shifting drivers and static baselines in
environmental governance: challenges for improving and proving water quality
outcomes. Reg. Environ. Change 16, 759–775. https://doi.org/10.1007/s10113-
015-0787-0.
Gruby, R.L., Fairbanks, L., Acton, L., Artis, E., Campbell, L.M., Gray, N.J., et al., 2017.
Conceptualizing social outcomes of large marine protected areas. Coast. Manag. 45,
416–435. https://doi.org/10.1080/08920753.2017.1373449.
Guerry, A.D., Polasky, S., Lubchenco, J., Chaplin-Kramer, R., Daily, G.C., Grifn, R.,
et al., 2015. Natural capital and ecosystem services informing decisions: from
promise to practice. Proc. Natl. Acad. Sci. Unit. States Am. 112, 7348–7355. https://
doi.org/10.1073/pnas.1503751112.
Hein, M.Y., Willis, B.L., Beeden, R., Birtles, A., 2017. The need for broader ecological and
socioeconomic tools to evaluate the effectiveness of coral restoration programs.
Restor. Ecol. 25, 873–883. https://doi.org/10.1111/rec.12580.
Heiss, R., Matthes, J., 2017. Citizen science in the social sciences: a call for more
evidence. Gaia 26 (1), 22–26. https://doi.org/10.14512/gaia.26.1.7.
Johnston, A., Fink, D., Hochachka, W.M., Kelling, S., 2018. Estimates of observer
expertise improve species distributions from citizen science data. Methods Ecol.
Evol. 9 (1), 88–97. https://doi.org/10.1111/2041-210X.12838.
Kennedy, R.E., Townsend, P.A., Gross, J.E., Cohen, W.B., Bolstad, P., Wang, Y.Q.,
Adams, P., 2009. Remote sensing change detection tools for natural resource
managers: understanding concepts and tradeoffs in the design of landscape
monitoring projects. Remote Sens. Environ. 113, 1382–1396.
Koontz, T.M., Thomas, C.W., 2006. What do we know and need to know about the
environmental outcomes of collaborative management? Publ. Adm. Rev. 66,
111–121.
Land-Zandstra, A.M., Devilee, J.L., Snik, F., Buurmeijer, F., van den Broek, J.M., 2015.
Citizen science on a smartphone: participants’ motivations and learning. Publ.
Understand. Sci. 25 (1), 45–60. https://doi.org/10.1177/0963662515602406.
Lausch, A., Bastian, O., Klotz, S., Leit˜
ao, P.J., Jung, A., Rocchini, D., Schaepman, M.E.,
Skidmore, A.K., Tischendorf, L., Knapp, S., 2018. Understanding and assessing
vegetation health by in situ species and remote-sensing approaches. Methods Ecol.
Evol. 9, 1799–1809. https://doi.org/10.1111/2041-210X.13025.
Leach, W.D., Pelkey, N.W., Sabatier, P.A., 2002. Stakeholder partnerships as
collaborative policymaking: evaluation criteria applied to watershed management in
California and Washington. J. Pol. Anal. Manag. 21, 645–670. https://doi.org/
10.1002/pam.10079.
Leach, W.D., Sabatier, P.A., 2005. To trust an adversary: integrating rational and
psychological models of collaborative policymaking. Am. Polit. Sci. Rev. 99,
491–503.
Legge, S., 2015. A plea for inserting evidence-based management into conservation
practice. Anim. Conserv. 18, 113–116. https://doi.org/10.1111/acv.12195.
Lehtiniemi, M., Outinen, O., Puntila-Dodd, R., 2020. Citizen science provides added
value in the monitoring for coastal non-indigenous species. J. Environ. Manag. 267,
110608. https://doi.org/10.1016/j.jenvman.2020.110608.
Leleu, K., Alban, F., Pelletier, D., Charbonnel, E., Letourneur, Y., Boudouresque, C.F.,
2012. Fishers’ perceptions as indicators of the performance of Marine Protected
Areas (MPAs). Mar. Pol. 36 (2), 414–422. https://doi.org/10.1016/j.
marpol.2011.06.002.
Likens, G., Lindenmayer, D., 2010. Effective Ecological Monitoring. Earthscan
Publishers, Collingwood, Victoria, Australia.
Lillesand, T.M., Kiefer, R.W., Chipman, J.W., 2015. Remote Sensing and Image
Interpretation, seventh ed. John Wiley and Sons Inc., New York, USA, ISBN 978-1-
118-34328-9, p. 736.
Lopatin, J., Fassnacht, F., Kattenborn, T., Schmidtlein, S., 2017. Mapping plant species in
mixed grassland communities using close range imaging spectroscopy. Rem. Sens.
Environ. 201, 12–23. https://doi.org/10.1016/j.rse.2017.08.031.
Lund, J.F., Balooni, K., Puri, L., 2010. Perception-based methods to evaluate
conservation impact in forests managed through popular participation. Ecol. Soc. 15
(3), 5.
Macauley, M.K., 2006. The value of information: measuring the contribution of space-
derived earth science data to resource management. Space Pol. 22, 274–282. https://
doi.org/10.1016/j.spacepol.2006.08.003.
Mandarano, L.A., 2008. Evaluating collaborative environmental planning outputs and
outcomes. J. Plann. Educ. Res. 27, 456–468. https://doi.org/10.1177/
0739456X08315888.
Margoluis, R., Stem, C.J., Salafsky, N., Brown, M., 2009. Design alternatives for
evaluating the impact of conservation projects. In: Birnbaum, M., Mickwitz, P.
(Eds.), Environmental Program and Policy Evaluation: Addressing Methodological
Challenges New Directions for Evaluation. 85–96. https://doi.org/10.1002/ev.
Marttunen, M., Weber, C., Åberg, U., Lienert, J., 2019. Identifying relevant objectives in
environmental management decisions: an application to a national monitoring
program for river restoration. Ecol. Indicat. 101, 851–866.
Mayer, A.L., Lopez, R.D., 2011. Use of remote sensing to support forest and wetlands
policies in the USA. Rem. Sens. 3, 1211–1233. https://doi.org/10.3390/rs3061211.
McIntosh, E.J., Chapman, S., Kearney, S.G., Williams, B., Althor, G., Thorn, J.P.R.,
Pressey, R.L., McKinnon, M.C., Grenyer, R., 2018. Absence of evidence for the
conservation outcomes of systematic conservation planning around the globe: a
systematic map. Environ. Evid. 7, 22. https://doi.org/10.1186/s13750-018-0134-2.
MicaSense, 2021 (last accessed May 12). RedEdge Camera Radiometric Calibration
Model. https://support.micasense.com/hc/en-us/articles/115000351194-RedEdge-
Camera-Radiometric-Calibration-Model.
Parks Canada, 2011. Consolidated Guidelines for Ecological Integrity Monitoring in
Canada’s National Parks. Protected Areas Establishment and Conservation Branch,
Parks Canada.
Phillips, T.B., Bailey, R.L., Martin, V., Faulkner-Grant, H., Bonter, D.N., 2021. The role of
citizen science in management of invasive avian species: what people think, know,
and do. J. Environ. Manag. 280, 111709. https://doi.org/10.1016/j.
jenvman.2020.111709.
Plummer, R., Armitage, D., 2007. A resilience-based framework for evaluating adaptive
co-management: linking ecology, economics and society in a complex world. Ecol.
Econ. 61, 62–74. https://doi.org/10.1016/j.ecolecon.2006.09.025.
Plummer, R., Baird, J., Dzyundzyak, A., Armitage, D., Bodin, ¨
O., Schultz, L., 2017b. Is
adaptive Co-management delivering? Examining relationships between
collaboration, learning and outcomes in UNESCO biosphere reserves. Ecol. Econ. The
Authors 140, 79–88. https://doi.org/10.1016/j.ecolecon.2017.04.028.
Plummer, R., Dzyundzyak, A., Baird, J., Bodin, ¨
O., Armitage, D., Schultz, L., 2017a. How
do environmental governance processes shape evaluation of outcomes by
stakeholders? A causal pathways approach. In: Belgrano, A. (Ed.), PLoS One, vol. 12.
Public Library of Science, e0185375. https://doi.org/10.1371/journal.
pone.0185375.
Pullin, A., Sutherland, W., Gardner, T., Kapos, V., 2013. Conservation priorities:
identifying need, taking action and evaluating success. In: Macdonald, D.W.,
Willis, K.J. (Eds.), Key Topics in Conservation Biology, rst ed., vol. 2. John Wiley
and Sons Inc, pp. 3–22. https://doi.org/10.1002/9781118520178.ch1.
Pullin, A.S., Knight, T.M., 2001. Effectiveness in conservation practice: pointers from
medicine and public health. Conserv. Biol. 15, 50–54.
Richards, J.A., Jia, X., 1999. Remote Sensing and Digital Image Analysis: an
Introduction, third ed. Springer-Verlag, New York, USA.
Selin, S.W., Schuett, M.A., Carr, D., 2000. Modeling stakeholder perceptions of
collaborative initiative effectiveness. Soc. Nat. Resour. 13, 735–745.
J. Baird et al.

Journal of Environmental Management 297 (2021) 113266
12
Tiner, Ralph, 2004. Tiner, Ralph W. “Remotely-sensed indicators for monitoring the
general condition of ”natural habitat" in watersheds: an application for Delaware’s
Nanticoke River watershed. Ecol. Indic. 4 (4), 227–243. https://doi.org/10.1016/j.
ecolind.2004.04.002.
Townsend, P.A., Lookingbill, T.R., Kingdon, C.C., Gardner, R.H., 2009. Spatial pattern
analysis for monitoring protected areas. Remote Sens. Environ. Elsevier Inc. 113,
1410–1420. https://doi.org/10.1016/j.rse.2008.05.023.
Turner, W., Spector, S., Gardiner, N., Fladeland, M., Sterlin, E., Steininger, M., 2003.
Remote sensing for biodiversity science and conservation. Trends Ecol. Evol. 18,
306–314.
van lersel, W., Straatsma, M., Addink, E., Middelkoop, H., 2018. Monitoring height and
greenness of non-woody oodplain vegetation with UAV time series. ISPRS J.
Photogrammetry Remote Sens. 141, 112–123. https://doi.org/10.1016/j.
isprsjprs.2018.04.011.
Webb, E.L., Maliao, R.J., Siar, S.V., 2004. Using local user perceptions to evaluate
outcomes of protected area management in the Sagay Marine Reserve, Philippines.
Environ. Conserv. 31, 138–148. https://doi.org/10.1017/S0376892904001377.
White, P.C., Jennings, N.V., Renwick, A.R., Barker, N.H., 2005. Questionnaires in
ecology: a review of past use and recommendations for best practice. J. Appl. Ecol.
42 (3), 421–430.
Wiens, J., Sutter, R., Anderson, M., Blanchard, J., Barnett, A., Aguilar-Amuchastegui, N.,
et al., 2009. Selecting and conserving lands for biodiversity: the role of remote
sensing. Remote Sens. Environ. Elsevier B.V. 113, 1370–1381. https://doi.org/
10.1016/j.rse.2008.06.020.
Willis, K.S., 2015. Remote sensing change detection for ecological monitoring in United
States protected areas. Biol. Conserv. 185, 233–242. https://doi.org/10.1016/j.
biocon.2014.12.006.
Xu, D., Guo, X., Li, Z., Yang, X., Yin, H., 2014. Measuring the dead component of mixed
grassland with Landsat imagery. Rem. Sens. Environ. 142, 33–43. https://doi.org/
10.1016/j.rse.2013.11.017.
Yasu´
e, M., Kaufman, L., Vincent, A.C.J., 2010. Assessing ecological changes in and
around marine reserves using community perceptions and biological surveys. Aquat.
Conserv. Mar. Freshw. Ecosyst. 20, 407–418. https://doi.org/10.1002/aqc.1090.
Zhou, W., Yang, H., Huang, L., Chen, C., Lin, X., Hu, Z., Li, J., 2017. Grassland
degradation remote sensing monitoring and driving factors quantitative assessment
in China from 1982 to 2010. Ecol. Indicat. 83, 303–313. https://doi.org/10.1016/j.
ecolind.2017.08.019.
J. Baird et al.