Strengthening Capacity and Equity in New England Evaluation (SCENE) Full Report

Abstract

This full report presents information about and results from the Strengthening Capacity and Equity in New England Evaluation study. The work explores whether and how evaluators promote equity in their evaluation practices within the New England region of the U.S.

Full text

Strengthening Capacity and Equity in New England Evaluation
(SCENE) Full Report
August 2021
Prepared for:
Barr Foundation &
New England area Evaluators
Prepared by:
Emily F. Gates, Assistant Professor
Eric Williamson & Joseph Madres, PhD Students
Kayla Benitez Alvarez, Undergraduate Student
Lynch School of Education & Human Development
Boston College
Jori N. Hall, Professor
University of Georgia, Athens
Suggested citation: Gates, E.F., Williamson, E., Madres, J., Benitez Alvarez, K., & Hall, J.N.
(2021). Strengthening Capacity and Equity in New England Evaluation (SCENE) Full Report.
[Unpublished report]. Boston College.

SCENE Report | August 2021 2
TABLE OF CONTENTS
Executive Summary ........................................................................................................................ 3
Findings....................................................................................................................................... 3
Recommendations ....................................................................................................................... 7
Introduction ..................................................................................................................................... 9
Research Team and Approach ........................................................................................................ 9
Study Questions, Design, and Methods ........................................................................................ 10
Questions................................................................................................................................... 10
Mixed Methods Design ............................................................................................................. 10
Questionnaire ............................................................................................................................ 11
Interviews .................................................................................................................................. 15
Integrated Analysis ..................................................................................................................... 2
Strengths and Limitations ............................................................................................................... 3
Results ............................................................................................................................................. 4
Evaluator Roles and Training ..................................................................................................... 4
Evaluator Demographics ............................................................................................................. 6
Evaluator Areas of Practice, Approaches, and Awareness of Professional Standards ............... 7
Evaluators’ Workplaces .............................................................................................................. 9
How Evaluators Define Equity ................................................................................................. 12
Whether and How Evaluators Center Equity ............................................................................ 14
Helpful and Hindering Factors for Centering Equity ............................................................... 27
Ways to Strengthen Capacity for Equity in Evaluation ............................................................ 31
Findings......................................................................................................................................... 35
Recommendations ......................................................................................................................... 39
References ..................................................................................................................................... 40

SCENE Report | August 2021 3
Executive Summary
This report presents results from the Strengthening Capacity and Equity in New England
Evaluation (SCENE) study. The SCENE study was conducted between August 2020-August
2021 by a research group at Boston College with funding from the Barr Foundation. The purpose
was three-fold: (1) characterize evaluation providers, evaluation practices, and evaluators
working in the New England area; (2) explore whether and how evaluators address inequities and
advance equity; and (3) identify ways to strengthen capacity and equity among evaluators within
the region. The study aims to inform regional evaluation practitioners and commissioners
interested in advancing equity in evaluation by providing an initial assessment that can inform
future capacity building and learning initiatives. Findings are particularly relevant to local
affiliates of the American Evaluation Association (AEA): Greater Boston Evaluators Network,
Vermont Evaluation Network, Connecticut Area Evaluators’ Network, and the Maine Evaluation
Society. The study also sought to address gaps in the evaluation literature regarding what equity
and equity-focused evaluation means and looks like in practice.
The study addressed five research questions:
1. What are the characteristics of participating evaluators (e.g., demographics, training) and
evaluation providers (e.g., size, services) in New England?
2. What does equity mean to evaluators in the context of their evaluation work?
3. To what extent and in what ways do participating evaluators center equity throughout
evaluations?
4. What helps and what hinders evaluators when working to center equity in evaluations?
5. What opportunities and needs are there to build capacity of evaluators and within
evaluation practice to center equity?
The study used a sequential, complementarity mixed method design. First, an online researcher-
developed questionnaire was administered to a census of AEA-affiliated evaluators in New
England and a snowball sample with 82 evaluators completing the questionnaire (approximate
response rate of 26%). The questionnaire addressed evaluators’ backgrounds and characteristics;
frequency of equity-oriented evaluation practices, on a scale from 1=never or rarely to 4=almost
always, and evaluation provider characteristics. Second, of those who indicated willingness to
participate in an interview on the questionnaire, 21 evaluators were purposely selected for
maximum variation. Interviews addressed evaluators’ backgrounds, whether and how evaluators
try to center equity during evaluation processes, and evaluators’ reflections on how to strengthen
capacity for equity-focused evaluation.
Findings
1. Evaluator Roles and Training. Most SCENE respondents are AEA members (72; 88%),
have masters (31; 38%) or doctoral degrees (36; 44%), and have an average of fifteen
years of evaluation experience (range 2-40). A majority of respondents have degrees in
education, public health, or psychology; work in senior-level researcher/evaluator roles
(60; 73%) with more than half as external evaluators (48; 59%) and about one-third
internal evaluators (29; 35%). Most reported learning their evaluation skills on-the-job
(71; 88%) followed by graduate-level university programs (61;75%) and professional
development workshops (52; 64%).

SCENE Report | August 2021 4
2. Evaluator Demographics. Nearly all SCENE respondents identify as white (76; 93%)
and mostly women (65; 80%) with about one-quarter between 30-39 years old, one-third
between 40-49 years old, and one-third between 50-64 years old. Few respondents
indicated belonging to racial/ethnic minority groups: Asian (4; 5%), Black/African
American (3; 4%), Hispanic/Latinx (2; 2%), and Multi-racial (1; 1%). Several identify as
men (13; 16%) and as gender non-binary (3;4%); about one-fifth (16; 20%) identify as
LGBQ+ and few (5; 6%) identify as having a disability.
3. Evaluator Areas of Practice. SCENE respondents practice evaluation in a variety of
areas including public health and/or health services (34; 43%), social services (30; 38%),
pre-K-12 education (29; 36%), and higher education (20; 25%). Most widely used
evaluation approaches include participatory evaluation (58; 71%), utilization-focused
evaluation (51; 62%) and program theory/theory-driven evaluation (47; 57%). Most
rarely or sometimes use AEA’s guiding principles, AEA’s cultural competence statement,
and evaluation standards in their work.
4. Evaluator Workplaces. SCENE respondents represent all states in New England, with
largest percentages from Massachusetts (35; 43%) and Vermont (16; 20%), and a
majority from community non-profits (34; 42%) and consulting or private sector work
(31; 38%). About half indicated they work for a small evaluation enterprise or are sole
proprietors (43; 52%) and about one-quarter work for medium sized firms (19; 23%).
Within their workplaces, about half of respondents (43; 52%) indicated having diversity,
equity and, inclusion (DEI) training and a DEI office or specific DEI-focused staff
members (43; 52%) within their workplace.
5. Defining Equity. Evaluators define equity within three areas: within the evaluation
process, as a focus when evaluating a program or intervention, and as an intended use or
influence of the evaluation.
a. Equity within an evaluation process includes minimizing evaluation team bias,
including beneficiaries in evaluation processes, and constructing multiculturally,
contextually valid knowledge.
b. Equity as a focus when evaluating an intervention includes examining reach and
accessibility of interventions, examining differential experiences of intended
beneficiaries, analyzing differential outcomes, and interrogating root causes of
needs/problems addressed by interventions.
c. Equity as intended use of an evaluation includes using evaluation to challenge and
alter historical and contemporary imbalances in power and decision-making,
providing professional training for intervention staff to expand who does
evaluations, and building evidence and guidance from evaluations to further
social equity and justice.
6. Centering Equity. SCENE respondents, on average, reported most often centering
equity during evaluation reporting and dissemination (m2.88) and throughout the
evaluation (m2.73), and data analysis (m2.67). Evaluators least often center equity in the

SCENE Report | August 2021 5
funding and contracting phase of the evaluation (m2.22) followed by the evaluation team
(m2.48).
7. Evaluation Teams. SCENE respondents reported most often self-examining cultural
identities (m2.98) and assumptions (m2.93). Respondents least often reported being led
by evaluators who identify as racial and/or ethnic minorities (m1.83) and having shared
lived experiences with the populations of focus (m2.05). Interviews revealed how some
evaluators try to enhance the diversity and expertise of evaluation teams by involving
community members and conducting equity reviews of evaluation plans and processes.
8. Throughout the Evaluation. Respondents most frequently highlight the strengths of the
intended beneficiaries of the evaluation (m3.09) and work to build trust with stakeholders
from minoritized or marginalized communities (m3.01). Respondents least frequently
work to mitigate power imbalances in the evaluation context (m2.45) and consult with
cultural brokers or translators to mediate between the cultures of stakeholders and the
evaluation team (m2.26).
9. Funding and Contracting. Most often, SCENE respondents have to work within
purposes and questions set by the funder/commissioner (m2.86) although they sometimes
try to prioritize RFPs focused on inclusion, diversity, and equity (m2.52). Interviewees
described centering equity in the funding and contracting phase in several ways: choose
RFPs carefully, make equity explicit in proposals, have up-front conversations, and
budget accordingly.
10. Evaluation Questions and Criteria. SCENE respondents often examine the underlying
problem an intervention seeks to address (m3.08). Respondents least often answer
questions about how an intervention addresses systemic drivers of inequity (m2.29),
include intended beneficiaries in evaluation decision-making (m2.28), and in defining
evaluative criteria (m2.29). Nearly all respondents evaluate effectiveness (73; 92%)
followed by most who examine the relevance of the intervention to the needs, culture,
interests, or circumstances of the intended beneficiaries (65; 82%). More than half of
respondents (46; 58%) examine equity of the intervention’s opportunities, experiences,
benefits, and/or results as an evaluation criterion within their evaluation work.
11. Data Collection. SCENE respondents often design and/or modify data collection to
culturally and/or linguistically suit the participants (m3.04). Interviewees expanded on
this by sharing examples of how they try to center equity during data collection, such as:
develop instruments with participant consultation and piloting; use data collectors with
shared identity/lived experience; conduct member checking; and consider access, reduce
burden, and maximize benefit for participants in data collection.
12. Data Analysis. When analyzing data, respondents reported often disaggregating data by
demographic differences (m3.36), looking for potential negative consequences (m3.10),
and looking for differential participant access and experiences (m3.00). Least often
respondents look at shifts in power to intended beneficiaries and communities (m2.21)
and re-distribution of resources (m2.19). Interviewees highlighted how they sometimes

SCENE Report | August 2021 6
disaggregate data and also how they struggle with this due to small sample sizes for some
minoritized groups and how they struggle to go beyond disaggregation to analyze
inequities more deeply and critically.
13. Reporting and Dissemination. Survey data indicate that a majority (54; 68%) often or
almost always present reports in formats accessible to intended beneficiaries and about
half (44; 55%) make evaluation results, in some form, publicly accessible. Some
interviewees discussed how they embed processes for shared sensemaking and
disseminate in multiple, accessible formats and across stakeholder groups. Other
interviewees struggle with collaborative sensemaking and accessible dissemination.
14. Helpful Factors. Interviewees discussed how factors within their teams, organizations
and wider society all can help to support equity-focused evaluation processes.
a. Within teams. Evaluators emphasized the importance of having members with
diverse cultural and linguistic expertise, inviting in community and new evaluator
voices, and fostering critical, self-reflective, and learning cultures.
b. Within evaluation organizations. Organizational helpful factors include having
missions and values aligned with equity, leadership support, internal diversity,
equity, and inclusion work, and dedicated staff time.
c. Within 2020 sociopolitical context. More broadly, evaluators reflected on how
public attention on issues of racial injustice in 2020 provided a context to further
advocate for equity-focused evaluation.
15. Hindering Factors. Interviewees identified how factors related to evaluation contracts,
evaluators’ racial identities, evaluation stakeholders, and the evaluation profession can all
hinder centering equity in evaluation.
a. Evaluation contracts. Within and across contracts, evaluators find that quick turn-
around time, limited budgets, restricted scopes of work, the need to secure and
keep contracts, stress and burnout, and constrained evaluator autonomy all work
against equity-focused evaluation.
b. Evaluators’ racial identities. Some interviewees talked about their own and
colleagues’ white racial identities as constraints that minimize seeing and
addressing inequity. Several interviewees of color shared examples of racial
ignorance and discrimination, including being treated as a token representative
and being expected to do more as evaluators of color which created inequities for
them as evaluators.
c. Evaluation stakeholders. In working with stakeholders, evaluators find differences
in what equity means, constraints on challenging inequities built-into programs,
and the lack of use of evaluation findings all to hinder equity.
d. Evaluation profession. Interviewees pointed to ways evaluation itself can
constrain equity in its professional culture and norms that privilege scientific
knowledge and objectivity, assumed evaluator role as experts, and a sense of
isolation, competition, and little sharing between evaluators and evaluation
organizations.

SCENE Report | August 2021 7
16. Strengthening Capacity. Suggestions for strengthening capacity for centering equity in
evaluation include enhancing evaluator skills, engaging evaluation stakeholders, and
changing how the broader evaluation field and profession to advance equity.
a. Enhancing and applying evaluator skills and responsibilities. Interviewees’
suggestions began with ways evaluators can build their own capacity by
participating in trainings about antiracism, racial justice, and intersectionality and
how these lenses can be applied to methods; documenting, self-assessing and
seeking critical feedback on evaluation processes; learning how to work together
across different cultural identities and ways of knowing; and critically applying
evaluation skills, such as situational analysis, to points to influence inequities
within the evaluation context. Finally, they urged evaluators to make the time and
space for addressing inequity.
b. Build professional learning among evaluators. Beyond individual evaluators and
evaluation teams, interviewees pointed to the need to build professional learning
by sharing practical guidance, concrete examples, and lessons from practice and
creating professional learning communities.
c. Engaging stakeholders. Evaluators suggested a need to educate funders and
engage new financial partners; bring on, compensate, and build capacity of local
and new partners; and clarify meanings of equity across a spectrum of
understanding among stakeholders.
d. Shifting the evaluation field. More broadly, evaluators raised concerns about and
suggested shifts needed within the evaluation profession, including interrogation
and change of dominant evaluation concepts and ways of working and advancing
field-level conversations, standards, and accountability to back evaluators up who
are working to promote equity.
Recommendations
1. Connect. Identify and build connections between evaluators and those in evaluation-
adjacent roles within New England. Expand beyond those affiliated with the American
Evaluation Association and local affiliates. Build inclusion and diversity by connecting
with those from underrepresented groups within the field and region.
2. Learn and Share. Support evaluation communities of practice and build in opportunities
for reflective practice and collective learning, within and across evaluation teams and
organizations. Raise awareness of existing guidance from the American Evaluation
Association, Equitable Evaluation Initiative, and other national and regional efforts to
promote culturally responsive and equitable evaluation. Translate frameworks into
practical guidance with case examples. Promote sharing between evaluators and across
evaluation teams and organizations of promising practices and lessons learned through
different efforts.
3. Deepen Understanding and Practice. Recognize that equity takes on different meanings
and that evaluators and stakeholders vary along a spectrum of understanding. Push
evaluators and support evaluators pushing stakeholders to deepen their understanding and
implications for changing practice. One area, in particular, is to emphasize how to

SCENE Report | August 2021 8
mitigate power imbalances between intended beneficiaries, evaluators, and funders
within decision-making about the evaluation and broader evaluation context.
4. Build Support. Given evaluators’ perceptions of constraints within contracts, funders,
and programmatic circumstances, identify ways that funders, clients, and program leaders
can support equity-focused evaluation work. This may mean hosting dialogues between
funders, evaluation practitioners, and intended beneficiary communities to critique and
reimagine evaluation practice. Creating templates, models, and examples for contracts
and client-evaluator relationships also could help evaluators push for the time and
resources needed.
5. Investigate. Within New England, some interviewees pointed to the predominance of
white supremacy culture within organizations and the social sector. They also pointed to
the default assumptions held by some evaluation team members and stakeholders about
the lack of racial, ethnic, and cultural diversity and, therefore, limited need for equity-
focused evaluation. Further research should examine evaluators’ racial identities in
relation to the predominantly white New England region and through the experiences of
evaluators of color.

SCENE Report | August 2021 9
Introduction
This report presents results from the Strengthening Capacity and Equity in New England
Evaluation (SCENE) study. The SCENE study was conducted between August 2020-August
2021 by a research group at Boston College with funding from the Barr Foundation. The purpose
was three-fold: (1) characterize evaluation providers, evaluation practices, and evaluators
working in the New England area; (2) explore whether and how evaluators address inequities and
advance equity; and (3) identify ways to strengthen capacity and equity among evaluators within
the region.
By providing an initial assessment to inform future capacity building and learning initiatives, the
study aims to inform regional evaluation practitioners and commissioners interested in advancing
equity in evaluation. Findings are particularly relevant to local affiliates of the American
Evaluation Association (AEA): Greater Boston Evaluators Network, Vermont Evaluation
Network, Connecticut Area Evaluators’ Network, and Maine Evaluation Society. The study also
sought to address gaps in the evaluation literature regarding what equity and equity-focused
evaluation mean and look like in practice. Study results will also be submitted for
publication/dissemination in academic journals.
Research Team and Approach
The study was led by Emily Gates, assistant professor of evaluation at Boston College whose
research examines the intersections of systems thinking, values, and equity in evaluation theory
and practice. Two doctoral students, Eric Williamson and Joseph Madres, worked as research
assistants. Eric primarily assisted with the questionnaire design, administration, and analysis and
Joe with the interviews and qualitative data analysis. Kayla Benitez Alvarez, an undergraduate
student, supported interview data collection and analysis. Jori N. Hall, professor at University of
Georgia and whose work focuses on culturally responsive approaches and qualitative
methodology, worked as a consultant reviewing study plans, processes, and providing feedback.
The team drew on principles of pragmatism (Hall, 2013) and collaborative and equity-oriented
research in the design and conduct of the study. We prioritized development of contextually
relevant and actionable knowledge. As a team, we met weekly throughout the study and
incorporated discussion of power dynamics and self-reflexivity to minimize potential that our
study could reproduce the inequities we sought to explore and help evaluators resist in their
professional practices. We also remained vigilant about identifying and addressing ethical and
methodological disagreements as they arose. The study was approved by Boston College’s
Institutional Review Board. All participants provided their informed consent.

SCENE Report | August 2021 10
Study Questions, Design, and Methods
Questions
The study addressed five research questions:
1. What are the characteristics of participating evaluators (e.g., demographics, training) and
evaluation providers (e.g., size, services) in New England?
2. What does equity mean to evaluators in the context of their evaluation work?
3. To what extent and in what ways do participating evaluators center equity throughout
evaluations?
4. What helps and what hinders evaluators when working to center equity in evaluations?
5. What opportunities and needs are there to build capacity of evaluators and within
evaluation practice to center equity?
Mixed Methods Design
The study used a sequential, complementarity mixed method design (Creswell & Plano-Clark,
2018) in which multiple methods are used to understand different aspects of or approaches to the
phenomenon of interest – addressing inequities and promoting equity in evaluation practice. This
design was also selected to allow the possibility of identifying unique aspects of equity beyond
those identified in the literature. We first developed and administered an online questionnaire
followed by semi-structured, individual interviews. Each method addressed all research
questions, allowing for an integrated analysis. Table 1 below shows the prominence of each
method in addressing each research question.
Table 1. SCENE Full Report - Study Design Matrix
Questions
Methods
Questionnaire
Interviews
(1) Characteristics of evaluators and evaluation
providers
X
x
(2) Meanings of equity
X
x
(3) Ways to center equity in evaluations
X
X
(4) Factors that help and hinder centering equity in
evaluation practice
x
X
(5) Opportunities and needs to build capacity
x
X
Note: Capitalized X indicates primary method and lowercase x indicates secondary method.

SCENE Report | August 2021 11
Questionnaire
This section describes questionnaire development; sample and administration; and data
processing and analysis.
Questionnaire Development. At the time of the study, there were no instruments assessing the
comprehensive set of constructs and items of interest. Therefore, we developed an original
instrument drawing on components of existing questionnaires and using a multi-step process to
establish face and construct validity.
● Background literature. First, we reviewed select literature, practical guidance, and existing
instruments relevant to culturally responsive and equitable evaluation practice as well as
surveying evaluators more generally about their practice (AEA, 2011; Andrews, Parekh, &
Peckoo, 2019; Center for Evaluation Innovation, 2017; Farrow & Morrison, 2019; Hood,
Hopson, & Kirkhart, 2015; Nielsen, Lemire, & Christie, 2018; Public Policy Associates,
2015; SPEC Associates, 2018; Westaby, Williams, Robinson, & Connors, 2019). Then, we
developed a matrix outlining the phases of evaluation practice, listed existing items for each
phase, and drafted original items for missing topics. We revised the set of items several times
to provide adequate coverage, use consistent language, and make items comprehensible.
● Expert reviews. Second, we conducted four expert reviews (Olson, 2010) with senior
evaluators specializing in culturally responsive and equitable evaluation. Reviewers rated the
effectiveness of and need to include each of the items regarding equitable evaluation practice.
These ratings were on a three-point scale: for quality, items were either “Strong” (3),
“Acceptable” (2), or “Needs Revision” (1); for item need, items were either “Definitely
Needed” (3), “Probably Needed” (2), or “Not Needed” (1). For any item that reviewers
indicated needed revision or was not needed, reviewers were asked to clarify why they gave
those ratings. Additionally, reviewers were asked if they thought additional items were
needed to ensure respondents gave feedback on each equity aspect of their evaluation
practice. Based on this feedback, we added items to fill gaps, omitted redundant items, and
clarified the language where necessary.
● Cognitive interviews. Third, we conducted cognitive interviews with six evaluators who were
similar to our target respondents but who would not be in our sample. During these cognitive
interviews, respondents verbalized their thought processes as they completed the instrument
and responded to additional semi-structured verbal probes about specific items and the
instrument as a whole (Peterson, Peterson, & Powell, 2017). We revised and finalized the
instrument.
This process resulted in an instrument with three sections addressing: evaluator backgrounds and
characteristics; equity-oriented evaluation practices; and evaluation provider characteristics.
Table 2 (p. 12) provides a brief description of each section, and Table 3 (p. 13) describes the
section on equity-focused evaluation practice grouped by phase.

SCENE Report | August 2021 12
Table 2. SCENE Full Report - Overview of Questionnaire
Topics
Variables
Evaluator
Background
and
Characteristics
● AEA and AEA affiliate membership
● Primary work sector
● Highest degree completed
● Field of highest degree
● Training: source of evaluation-related skills
● Years of evaluation work
● Role/job title
● Internal vs external evaluation role
● Racial/ethnic identity
● Sex and gender
● LGBQ+ identifying
● Age
● Language(s) other than English used in work
● Disabilities
● Demographic questions include write-in option
● Open-ended: Other domains of identity important to evaluation work
Evaluation
Practice
● Use of professional evaluation standards and guidelines
● Approaches/models for evaluations
● Equity-focused design and conduct of evaluations
● Criteria used in evaluations
● Open-ended: meanings of equity in context of evaluators’ work
● Open-ended: ways evaluation team successfully addresses inequities
and/or advances equity
● Open-ended: ways to develop capacity to address inequities and/or
advance equity in your work
Evaluation
Workplace
Characteristics
● Location
● US vs international work
● Sector of evaluation work
● Fields/areas of evaluation work
● Work for large/medium/small evaluation firm
● Percent of services that are evaluation-specific
● Services provided
● Offer diversity, equity, and/or inclusion training opportunities in
workplace; office and/or staff person; initiatives (if yes, describe for
each)

SCENE Report | August 2021 13
Table 3. SCENE Full Report - Equity-focused Evaluation Practice Scales
Phase/Scale
Sample Items
How often do you/does your team do each of the following?
Team
● Have shared lived experiences (e.g., life histories, cultural
perspectives) with the population(s) of focus in the evaluation.
● Conduct evaluations as part of a racially and/or ethnically diverse
team.
Throughout
● Work to mitigate power imbalances in the evaluation context.
● Challenge any assumptions that the intended beneficiaries lack the
ability to achieve because of their culture (i.e., deficit assumptions).
Funding and
contracting
● Prioritize requests for proposals that have purposes and/or questions
focused on inclusion, diversity, and/or equity.
● Negotiate with funding agencies to have purposes and/or questions
focused on inclusion, diversity, and/or equity.
Setting
questions and
criteria
● Include intended beneficiaries in decision making about the
evaluation purpose, approach, and/or questions.
● Include intended beneficiaries in selecting criteria/definitions of
success.
Data collection
● Design and/or modify data collection to be appropriate for the
culture(s) and language(s) of the people of whom the questions are
being asked.
● Use systems thinking and/or systems methods (e.g., social network
analysis, causal loop diagrams, agent-based modelling, critical
systems heuristics).
Data analysis
● Include intended beneficiaries in interpretation of data and sense-
making.
● When possible, disaggregate data by key demographic differences.
Reporting and
dissemination
● Present evaluation results in formats accessible to the intended
beneficiaries of the intervention.
● Make evaluation results, in some form, publicly accessible.
Questionnaire Sample and Administration. While we intended to understand the full
population of evaluators working in New England, there was no list and, therefore, no way to
randomly sample within this population. We used a combination of a census with snowball
sampling. We formally requested and obtained a list of AEA members who indicated living in a
state within New England and sent the survey invitation to this entire list (n=258). We also
invited those on the initial list to forward the invitation to colleagues practicing evaluation within
New England who may not be AEA members. Additionally, we sent the invitation out to two
national listservs, Center for Culturally Responsive Evaluation and Assessment and Annie E.

SCENE Report | August 2021 14
Casey’s Leaders in Equitable Evaluation and Diversity (LEEAD) network. Regionally, the
Greater Boston Evaluation Network and Vermont Evaluation Network also sent the invitation to
their members on our behalf. Finally, we posted the invitation on Twitter. 67 out of 258 on
AEA’s list completed the questionnaire for an initial response rate of 26%. An additional 15
responses came in through snowball sampling for a total of 82 responses. Table 4 below
compares select demographic characteristics of the survey sample with those of the AEA list of
evaluators to generate a rudimentary understanding of the representativeness of our survey
sample vis-a-vis AEA members who indicated living in the New England area. The
questionnaire was administered using Qualtrics and took approximately 15-20 minutes to
complete. It ran from November 24, 2020 to February 19, 2021. All participants who completed
a questionnaire received a $15 Amazon gift card.
Table 4. SCENE Full Report - Questionnaire Sample vs. Population of Interest
Item
n
(Sample %)
N
(Population %)
Race/Ethnicity
82*
(100%)*
258
(100%)1
White
76
(93%)
164
(64%)
Asian
4
(5%)
13
(4%)
Black/African American
3
(4%)
14
(5%)
Hispanic/Latinx
2
(2%)
7
(3%)
American Indian, Native American, or Alaskan
Native
-
-
2
(1%)
Multi-Racial
1
(1%)
20
(8%) (‘Other’)
(No response)
-
-
38
(15%)
Sex
82
(100%)
258
(100%)
Female
65
(79%)
170
(66%)
Male
14
(17%)
50
(19%)
Prefer not to answer/no answer
2
(2%)
38
(15%)
Intersex
1
(1%)
-
-
State
802
(100%)
258
(100%)
Massachusetts
35
(44%)
137
(53%)
Vermont
16
(20%)
29
(11%)
Connecticut
9
(11%)
29
(11%)
New Hampshire
9
(11%)
29
(11%)
Maine
8
(10%)
13
(5%)
Rhode Island
3
(4%)
21
(9%)
*Indicates a select-all-that-apply question, so % is out of 82 for each response option
1Multiple responses not allowed for the population information
2Two individuals did not supply the state of their primary office

SCENE Report | August 2021 15
Questionnaire Data Processing and Analysis. For quantitative data, one team member first
cleaned any accidental or fraudulent responses from the online Qualtrics responses. Partial
responses (i.e. questionnaire submissions with some items left incomplete) were included in the
analysis, as long as these respondents reached the end of the questionnaire. Then, he downloaded
all responses from Qualtrics in Excel (.xlsx) format. He split the consent information from the
survey data, created ID’s for the responses that were added to the consent and survey data Excel
documents, created a separate document with a key for the ID’s, and stored this separately from
the data. Below we explain the analytic procedures for quantitative and qualitative questionnaire
data.
● Quantitative Data. All Likert-style questions were coded in ascending order, with ‘almost
always’ (4), ‘often’ (3), ‘sometimes’ (2) and ‘never or rarely’ (1). The ‘unsure’, ‘outside my
scope’, and ‘not familiar with this’ options were coded nominally. Other questions were
coded according to the needs of the analysis. Data analysis took place in Excel, SPSS, and R.
Questions with nominal response options were counted, and frequencies for these items were
generated. For Likert questions, we generated descriptive statistics, including means,
standard deviations, and frequencies. Additionally, we cross-tabulated these items to explore
potential differences in frequency of equity-oriented evaluation practices by evaluator and/or
evaluation workplace characteristics. In addition to item-level analyses, we treated the five
sets of items within the equitable evaluation practice section of the questionnaire as scales
(see Table 3): evaluation team; contracting and funding; data collection; data analysis; and
reporting and dissemination. For these, we averaged respondents’ answers to each of the sub-
items in each phase/scale of equitable evaluation practice to generate summative scores for
each phase/scale (see pp. 15-26 for subitems in each phase/scale).
● Qualitative Data. For open-response questions, we used an inductive thematic analysis
approach (Braun & Clarke, 2006). For this approach, we first downloaded data into Excel
and, for each response, edited any minor grammatical issues and ensured anonymity. One
researcher reviewed all responses per question and developed an initial set of inductive
codes. The lead researcher reviewed the data and draft codes, organized the codes into
themes and revised the language. A third researcher then used the themes to code each
response, often coding one response into more than one theme. Using these, the team
developed thematic summary statements for each evaluation question and selected illustrative
quotations.
Interviews
This section describes (1) developing the interview protocol; (2) interview sampling and data
generation; and (3) data processing and analysis.
Protocol Development. To develop the protocol, we first discussed how the interview data
would expand on the questionnaire and what kinds of information we hoped to gather in relation
to each research question. We then drafted a protocol, piloted it with two interviewees similar to
our respondents but not in the sample, and revised the protocol. The final interview protocol
consisted of three sections:

SCENE Report | August 2021 16
● Evaluator background: information about interviewees (e.g. their identities and personal
values) and their current evaluation work (e.g. their organization, roles, typical evaluands,
methods/theories/frameworks used)
● Equity in evaluation practice: what the concept of in/equity means to interviewees, how they
and/or their teams try to address inequity and promote equity throughout evaluation phases
and activities (e.g. negotiating contracts, designing the evaluation, developing instruments),
and what helpful/hindering factors as well as tensions/dilemmas emerge when trying to
center equity in their practice.
● Reflections on strengthening capacity: what interviewees have learned through their
experiences, recommendations they have for building evaluator capacity, and questions they
have and areas in which they most want to continue learning
Sample and Data Generation. We used a combination of convenience and maximum variation
sampling (Patton, 2015). On the questionnaire, we asked respondents to indicate whether we
could contact them to invite them to participate in an interview. Of those who indicated yes on
this question (33), we utilized several criteria to prioritize and select those invited to interview:
● Race/ethnicity: maximum variation was sought; as there were disproportionately fewer in the
sample, evaluators of color were prioritized
● Sex/gender: maximum variation sought; as there were disproportionately fewer in the
sample, male- and non-binary-identifying evaluators were prioritized
● State: sought maximum variation from each state in the region; prioritized non-MA as fewer
in our sample
● LGBQ+: prioritized those who identify with this group
● Area of practice: sought maximum variation
Table 5 (p. 17) shows interviewees’ characteristics. We conducted a total of 21 semi-structured,
individual interviews (Patton, 2015) from January 12th to February 12th, 2021. All interviews
were conducted via Zoom video conferencing and lasted 50-60 minutes. Prior to each interview,
we sent the interviewee the protocol and encouraged them to reflect on their responses in
advance. For each interview, one member of the research team was the lead interviewer, asking
questions and probes, and another researcher took notes. During interviews, the interviewer
periodically restated interviewee’s responses as a form of member checking (Creswell & Poth,
2018).
Table 5. SCENE Full Report - Interviewee Sample
Item
n
(%)
Race/Ethnicity
21
(100%)
White
17
(81%)
Asian
2
(9%)
Black/African American
2
(10%)
Sex
21
(100%)
Female
15
(71%)
Male
4
(19%)
Prefer not to answer/no
answer
1
(5%)
Intersex
1
(5%)
State
21
(100%)
Massachusetts
11
(52%)
Maine
3
(14%)
New Hampshire
3
(14%)
Vermont
2
(10%)
Connecticut
1
(5%)
Rhode Island
1
(5%)
LGBQ Identification
21
(100%)
No
16
(76%)

SCENE Report | August 2021 16
Yes
5
(24%)
Evaluator Role
21
(100%)
External
10
(48%)
Internal
7
(33%)
A mix of both
3
(15%)
Item
n
(%)
Area of Practice
21*
(100%)*
Social Services
9
(43%)
Public Health and/or Health
Services
8
(38%)
Workforce development
6
(29%)
Pre-K-12 Education
6
(29%)
Higher Education
5
(24%)
International Development and
Aid
5
(24%)
Informal education (e.g., out-
of-school, museums)
4
(19%)
Organizational Development
and Learning
3
(14%)
Criminal Justice
3
(14%)
Public Policy and Analysis
2
(10%)
Something else
5
(24%)
Coastal Communities
1
(5%)
Environment, social
development
1
(5%)
Housing and homelessness
1
(5%)
Participatory grantmaking,
wealth building, small
businesses
1
(5%)
Special Education
1
(4%)
Position
21
(100%)
Senior-level evaluator
14
(67%)
Mid-level evaluator
3
(14%)
Junior-level evaluator
0
(0%)
Something else
4
(19%)
Assistant professor
1
(5%)
Director of Learning and
Evaluation (at a funder - a
community foundation)
1
(5%)
Independent evaluation
consultant
1
(5%)
Solo-evaluator so I feel
competent at the Mid-level,
but operate at the senior-
level
1
(4%)
Firm Size
21
(100%)
Large
6
(29%)
Mid-sized
4
(19%)
Small/Independent Evaluator
11
(52%)
Data Processing and Analysis. We audio recorded interviews and used an online AI
transcription service (sonix.ai) to generate written transcripts from the audio files. Team
members who led interviews reviewed the transcripts against the audio recordings of interviews
in which they had not participated. They conducted close reads while listening to the audio
recording, and cleaned the transcript of AI software mistakes. Following the accuracy check, the
interviewer downloaded the transcript and used a shared template to annotate the transcription.
This involved inserting headers and bolded questions to organize the interview according to the
protocol. We then shared the full, annotated transcript as a Google document with the
interviewee, requesting that they review and make any changes to the transcription—a second
form of member checking. After all interviewees approved their transcripts, we began a multi-
step coding process. The team coded iteratively to analyze the interview data using Dedoose.
This involved the following steps:
● Codebook development: The lead researcher reviewed a random selection of five transcripts
and developed an initial codebook composed of deductive codes identified from the research
*Indicates a select-all-that-apply question, so % is out of 21 for each response option

SCENE Report | August 2021 2
and interview questions, and inductive codes (Miles et. al, 2014). The two other researchers
piloted this codebook to code three interviews each. We met to discuss and revise each code,
add additional codes, and finalize the shared codebook (Smagorinsky, 2008).
● Coding, inter-coder reliability and agreement checks: Two researchers independently coded
one-quarter of the transcripts and then the lead researcher reviewed coded excerpts to
identify any discrepancies. The researchers discussed and resolved any discrepancies in the
already coded transcripts and moved to the next quarter of transcripts to continue coding.
This continued until all transcripts were coded or reviewed independently by two researchers.
● Thematic analysis: The lead researcher then analyzed sets of codes relevant to each research
question to identify patterns, focused on “explanatory or inferential codes, ones that identify
an emergent theme” (Miles et. al., 2014, p. 86). To do this, she used a constant comparative
method for excerpts within and between codes to develop a higher-level categorization of the
data (Corbin & Strauss, 2015), thus developing a thematic analysis for each research
question. She used an audit trail (Norwell et. al, 2017) to illustrate how she arrived at each
theme. A second researcher then reviewed the coded data and this analysis and minor
clarifications were made to the themes, which were written up along with exemplary quotes
from interviews and organized by research question.
Integrated Analysis
For this report, we conducted integrated analyses (Fetters et. al, 2013) across questionnaire and
interview data for the third research question on whether and how evaluators center equity. We
plan to conduct additional integration for the other research questions in future work. For
research question three (“To what extent and in what ways do participating evaluators center
equity throughout evaluations?”), we created a joint data display (Bazeley, 2012). This involved
creating tables with the core evaluation phases and the questionnaire results and interview
themes pertaining to each phase. This facilitated an examination of similarities and differences
by method and helped to examine how the interview data expanded on some questionnaire
results.

SCENE Report | August 2021 3
Strengths and Limitations
This is the first study of its kind to explore what equity means within the context of professional
evaluation practices. As such, the study sets out a line of inquiry that can be continued in future
work. Similarly, the exploratory nature of the work presents some limitations. Below, we
identify the strengths and limitations of our research project:
● Team Process: The team-based approach undertaken in this study incorporated multiple
perspectives of team members as well as external experts. One of the great advantages of
working as a team is the peer review process that is built in. We discussed all decisions
throughout the study design, conduct, and analysis and consistently had at least two
people conducting the work. This reduced individual researcher influence and bias.
● Samples: Given the lack of a comprehensive list of evaluators in New England, we were
unable to select a random sample and discern how well our sample represents the
population of interest. Given the homogeneity of the sample in terms of race/ethnicity,
we intentionally sought maximum variation of other characteristics (e.g., gender, area of
practice, state) in our interview sample.
● Instrument: It is both a strength and limitation of the study that we developed a new
instrument to examine evaluators’ backgrounds and equity within their practices.
Evidence for face and construct validity was generated by anchoring the items in the
literature and expert reviews. However, no psychometric testing was done on the
instrument.
● Data analysis: Given sample size and characteristics, our quantitative analyses were
limited to descriptive statistics. Strength of our qualitative interview data analysis process
was that it helped us stay close to the participants’ own conceptualizations and
interpretations of the phenomenon under study, while its primary weakness is that the
inductive process generated much richer data than could be reported succinctly and
thematically. As a research team, we made choices about what to highlight or overlook.
● Transferability: Ultimately, our study sample and design provide insights into patterns
and meanings that may be transferable to other regional evaluation contexts, evaluation
workplaces, and evaluators. It is up to readers and future research to discern the relevance
to their contexts and to future research to challenge, confirm, or expand on findings.

SCENE Report | August 2021 4
Results
This section presents the results organized by sixteen overarching findings, highlighted in blue
text below (pp. 4-34) and summarized in pp. 35-38.
Evaluator Roles and Training
1. Evaluator Roles. Most SCENE respondents are AEA members (72; 88%), have masters
(31; 38%) or doctoral degrees (36; 44%), and have an average of fifteen years of
evaluation experience (range 2-40). A majority of respondents have degrees in
education, public health, or psychology; work in senior-level researcher/evaluator roles
(60; 73%) with more than half as external evaluators (48; 59%) and about one-third
internal evaluators (29; 35%). Most reported learning their evaluation skills on-the-job
(71; 88%) followed by graduate-level university programs (61;75%) and professional
development workshops (52; 64%).
AEA and Affiliate Membership. Nearly all respondents (72; 88%) are AEA members, with 9
(11%) who are not, and one (1%) who is not sure. Nineteen respondents belong to the Greater
Boston Evaluation Network, 12 to the Vermont Evaluators’ Network, two to the Maine
Evaluation Society, and one to the Connecticut Area Program Evaluators’ Network. Three
respondents also belong to other AEA affiliates: one in New York City’s AEA Affiliate, and two
in the Washington Evaluators. There were also two respondents who belong to other affiliates.
One belongs to the Eastern Evaluation Research Society, and one to both the Blue Marble
Evaluation and International Development Evaluation Association.
Education. Thirty-six respondents (44%) have Doctoral degrees, 31 (38%) have Master’s
degrees, and 11 (13%) have professional degrees (e.g., M.S.W., J.D., M.D.). Three (4%) hold
Bachelor’s degrees as their highest degree, and one (1%) a High school degree or G.E.D. as their
highest degree. They hold degrees in a variety of disciplines. See Table 6.
Table 6. SCENE Full Report - Highest Educational Degree Disciplines
Item
Response
N
(%)
Highest Degree
Discipline
Other (Please specify):
23
(28%)
Education
18
(22%)
Public Health
14
(17%)
Psychology
10
(12%)
Public Policy
9
(11%)
Evaluation
6
(7%)
Social Work
5
(6%)
Public Administration
5
(6%)
Sociology
4
(5%)
Economics
2
(2%)
Educational Psychology
1
(1%)
Math or Statistics
1
(1%)
Non-Response
1
(1%)

SCENE Report | August 2021 5
Total
82
(100%)
Other Highest
Degree
Discipline:
Human Development and/or Family Studies
6
(26%)
Anthropology
3
(13%)
Environmental Studies
3
(13%)
International Affairs
3
(13%)
Community Planning
2
(9%)
Social Policy
2
(9%)
Counseling and School Psychology
1
(4%)
Law
1
(4%)
Masters of Research in Social Research
1
(4%)
Science and Technology Studies
1
(4%)
Total
23
(100%)
Mode for each item in bold.
Training. Most respondents (71; 88%) learned their evaluation skills on-the-job and many
learned through university degree-granting programs (61; 75%) and professional development
workshops (52; 64%). See Table 7.
Table 7. SCENE Full Report - Evaluation Training
Item
Response
N
(%)
Source of Evaluation
Skills:
On the job
71
(87%)
A university degree-granting Master’s or Doctoral
program
61
(74%)
Professional development workshops
52
(63%)
A self-learning source, such as online programs,
reading texts, asking others, etc.
46
(56%)
Webinars
37
(45%)
In-house training hosted by your organization
15
(18%)
On-site training hosted by an organization external to
yours
11
(13%)
Some other avenue (please specify):
7
(9%)
A university certificate program
5
(6%)
Non-Response
1
(1%)
Total
82
(100%)
Other Source of
Evaluation Skills:
Mentoring
4
(57%)
Teaching or taking graduate evaluation course
2
(29%)
Grantee of federal funding opportunities
1
(14%)
Total
7
(100%)
Mode for each item in bold.

SCENE Report | August 2021 6
Experience. On average, SCENE respondents reported about 15.3 years (S.D. 8.6 years) of
experience practicing evaluation, with a range from two to 40 years of experience.
Roles. Most SCENE respondents (60; 73%) work in a senior-level researcher/evaluator role,
followed by some respondents (13; 16%) in a mid-level researcher/evaluator role. One (1%)
worked in a junior-level researcher/evaluator role. Eight (10%) respondents identified some other
role level. In this group, three work as independent consultants, two manage evaluations (i.e., do
not have direct evaluation tasks). One works across levels, one teaches evaluation at a university,
and one used ‘extension work’ to describe their role and responsibilities.
More than half of respondents (48; 59%) work as external evaluators, while 29 (35%) work as
internal evaluators. Four (5%) work as both internal and external evaluators, and one (1%) does
not work as either an internal or external evaluator, but manages evaluators.
Evaluator Demographics
2. Evaluator Demographics. Nearly all SCENE respondents identify as white (76; 93%)
and mostly as women (65; 80%) with about one-quarter between 30-39 years old, one-
third between 40-49 years old, and one-third between 50-64 years old. Few respondents
indicated belonging to racial/ethnic minority groups: Asian (4; 5%), Black/African
American (3; 4%), Hispanic/Latinx (2; 2%), and Multi-racial (1; 1%). Several identify
as men (13; 16%) and as gender non-binary (3; 4%); about one-fifth (16; 20%) identify
as LGBQ+ and few (5; 6%) identify as having a disability.
Race/Ethnicity. Few respondents identify as belonging to racial/ethnic minoritized groups:
Asian (4; 5%), Black/African American (3; 4%), Hispanic/Latinx (2; 2%), and multi-racial (1;
1%). Most respondents (76; 93%) identify as White.
Sex. Most respondents identify as female (65; 79%), 14 (17%) as male, two (2%) preferred not
to answer, and one (1%) identifies as intersex.
Gender. Most respondents (65; 80%) identify as women, 13 (16%) as men, and three (4%) self-
identified as gender non-binary or gender queer in a write-in option.
LGBQ+. About one-fifth (16; 20%) identify as LGBQ+. Most respondents (66; 80%) do not
identify as LGBQ+.
Age Range. Most respondents (27; 33%) are between 40-49 years of age, followed by 50-64
years of age (26; 32%), and 30-39 years of age (21; 26%). Four (5%) are between the ages of 20-
29, and three (4%) are older than 65.
Disability Identification. Five (6%) identify as having a disability. Most respondents (77; 94%)
do not identify as having any type of disability. Of those five, two classify their disability as a
mental health disability, two as a learning disability, one as a physical disability, one as a
chronic health disability, and one as a neurological disability (respondents were able to select
multiple types of disability classifications).

SCENE Report | August 2021 7
Languages. Most respondents (56; 68%) use only English in their evaluation work. Additional
languages used most frequently include Spanish (14; 17%), Portuguese (3; 4%) and Haitian
Creole (3; 4%).
Additional Identifiers. When asked about additional dimensions of their identity salient to their
evaluation work, respondents listed the following: their age, experience level, first-generation
college attendee/graduate, immigrant status, mental health, parenthood, religion, past substance
use, socio-economic status, veteran status, and working at a woman-owned business.
Evaluator Areas of Practice, Approaches, and Awareness of Professional Standards
3. Evaluator Areas of Practice. SCENE respondents practice evaluation in a variety of
areas including public health and/or health services (34; 43%), social services (30;
38%), pre-K-12 education (29; 36%), and higher education (20; 25%). Most widely
used evaluation approaches include participatory evaluation (58; 71%), utilization-
focused evaluation (51; 62%) and program theory/theory-driven evaluation (47; 57%).
Most respondents rarely or sometimes use AEA’s guiding principles, AEA’s cultural
competence statement, and evaluation standards in their work.
Areas of Practice. Thirty-four (43%) respondents work in public health and/or health services,
followed by social services (30; 38%) and PK12 education (29; 36%). See Table 8.
Table 8. SCENE Full Report - Areas of Practice
Item
Response
N
(%)
Workplace Field
Public Health and/or Health Services
34
(41%)
Social Services
30
(37%)
Pre-K-12 Education
29
(35%)
Higher Education
20
(24%)
Organizational Development and Learning
16
(20%)
Workforce development
15
(18%)
Something else (please specify):
15
(18%)
Informal education (e.g., out-of-school, museums)
13
(16%)
International Development and Aid
10
(12%)
Public Policy and Analysis
10
(12%)
Criminal Justice
9
(11%)
Non-Response
2
(2%)
Total
82
(100%)
Other Workplace Field
Environmental Stewardship/Conservation
3
(20%)
Housing, community development
3
(20%)
Youth Development
2
(13%)
Special Education
1
(7%)
Scientific programs
1
(7%)
Religious organizations
1
(7%)

SCENE Report | August 2021 8
Participatory grantmaking, wealth building, small businesses
1
(7%)
Energy
1
(7%)
College access among k-12 students
1
(7%)
Coastal Communities
1
(7%)
Total
15
(100%)
Mode for each item in bold.
Evaluation Approaches and Models. Respondents most frequently use participatory evaluation
(58; 71%), followed by utilization-focused evaluation (51; 62%) and program theory/theory-
driven evaluation (47; 57%). Approaches not frequently used include responsive evaluation (11;
13%), social justice/transformative evaluation (7; 9%), and social return on investment (5; 6%).
See Table 9.
Table 9. SCENE Full Report- Evaluation Approaches and Models
Item
Response
N
(%)
Evaluation
Approach/Model
Participatory Evaluation
58
(71%)
Utilization-focused Evaluation
51
(62%)
Program Theory/Theory-driven Evaluation
47
(57%)
Developmental Evaluation
45
(55%)
Quasi-Experimental Approaches
41
(50%)
Equity-focused/Equitable Evaluation
36
(44%)
Culturally Responsive Evaluation
32
(39%)
Empowerment Evaluation
18
(22%)
Randomized Controlled Trials
16
(20%)
Responsive Evaluation
11
(13%)
Something else (please specify)
9
(11%)
Social Justice/Transformative Evaluation
7
(9%)
Social Return on Investment
5
(6%)
Unsure
3
(4%)
Total
82
(100%)
Other Approach/Model
Principles-Focused Evaluation
2
(2%)
Blue Marble Evaluation
1
(1%)
Collective Impact Evaluation
1
(1%)
Feminist Evaluation
1
(1%)
Improvement Science
1
(1%)
Lots of correlational and descriptive data
1
(1%)
Outcome-focused evaluation
1
(1%)
Various qualitative methods (process tracing, outcome
harvesting, positive deviance, etc.)
1
(1%)
Total
9
(11%)
Mode for each item in bold.

SCENE Report | August 2021 9
Awareness of Professional Standards. A majority of respondents indicated either never or
rarely or sometimes using AEA’s guiding principles, AEA’s cultural competence statement, and
evaluation standards in their work. See Table 10.
Table 10. SCENE Full Report - Awareness of Professional Standards
Item
Mean
(SD)
Never or
Rarely
(1)
Sometimes
(2)
Often
(3)
Almost
Always
(4)
Not
Familiar
With This
Unsure
American Evaluation
Association’s Guiding Principles
for Evaluators (n=82)
1.94
(0.93)
31
(38%)
30
(37%)
14
(17%)
6
(7%)
1
(1%)
0
(0%)
American Evaluation
Association’s Public Statement on
Cultural Competence (n=82)
1.66
(0.81)
37
(45%)
22
(27%)
9
(11%)
2
(2%)
11
(13%)
1
(1%)
Program Evaluation Standards
from the Joint Committee on
Standards for Educational
Evaluation (n=82)
1.44
(0.64)
41
(50%)
18
(22%)
5
(6%)
0
(0%)
17
(21%)
1
(1%)
Mode for each item in bold.
Evaluators’ Workplaces
4. Evaluators’ Workplaces. SCENE respondents represent all states in New England, with
largest percentages from Massachusetts (35; 44%) and Vermont (16; 20%), and a
majority from community non-profits (34; 43%) and consulting or private sector work
(31; 39%). About half indicated they work for a small evaluation enterprise or are sole
proprietors (43; 54%) and about one-quarter work for medium sized firms (19; 24%).
Within their workplaces, about half of respondents (43; 54%) indicated having
diversity, equity, and inclusion (DEI) training and a DEI office or specific DEI-focused
staff members (43; 54%) within their workplace.
Location. Prior to the pandemic, most respondents (35; 44%) worked in Massachusetts,
followed by Vermont (16; 20%), Connecticut (9; 11%), New Hampshire (9; 11%), Maine (8;
10%), and finally, Rhode Island (3; 4%). Most respondents (75; 94%) operate within the United
States, with 13 (16%) operating internationally.
Work Sector. Most respondents (34; 43%) work in community non-profits, followed by
consulting or private sector work (31; 39%). See Table 11 (p. 10).

SCENE Report | August 2021 10
Table 11. SCENE Full Report - Work Sector
Item
Response
N
(%)
Workplace Sector
Community Non-profit
34
(41%)
Consulting/Private Sector
31
(38%)
Academia/Higher Education
19
(23%)
Government (e.g., health department)
11
(13%)
Foundation or Funder
8
(10%)
Pre-K-12 Education
8
(10%)
Difficult to identify primary work sector - work is cross-
sector
8
(10%)
Something else (please specify) :
8
(10%)
Non-Response
3
(4%)
Total
82
(100%)
Other Workplace Sector
Public Health/Community Health/Healthcare
2
(25%)
Museum/informal education
2
(25%)
Public nonprofit but my work is funded by federal and state
grants
1
(13%)
International organization
1
(13%)
Extension
1
(13%)
Coastal Communities
1
(13%)
Total
8
(100%
Mode for each item in bold.
Size of Workplace. Most respondents (43; 54%) indicated they either work for a small
evaluation enterprise or are the sole proprietors. Nineteen (24%) work for a mid-sized enterprise,
with nine (11%) working for large evaluation enterprises.
Services. About half of respondents (36; 46%) indicated that less than 25% of the services they
provide are evaluation-specific. Twenty (25%) said that greater than 75% of the services they
provide are evaluation-specific, and the remaining respondents (25; 29%) report that 26-75% of
the services they provide are related to evaluation. Nearly all respondents (65; 84%) provide data
collection and analysis. Sixty-one (79%) provide both evaluation planning and reporting and
presentations. Forty-six (60%) provide evaluation capacity-building and/or technical assistance
to their clients. Three respondents specified that they provided all or most of these services
internally. See Table 12 (p. 11).

SCENE Report | August 2021 11
Table 12. SCENE Full Report - Evaluation Services
Item
Response
N
(%)
Evaluation
Services Provided:
Data collection & analysis
65
(79%)
Evaluation planning
61
(74%)
Reporting and presentations
61
(74%)
Evaluation capacity building and/or technical assistance
46
(56%)
Data visualization
40
(49%)
Setting up monitoring and evaluation systems
39
(48%)
Data management/Database management
35
(43%)
Providing evaluation training
24
(29%)
Program audits
11
(13%)
Other (please specify):
11
(13%)
Unsure
3
(4%)
Non-Response
5
(6%)
Total
82
(100%)
Other Evaluation
Services Provided:
Conducting these activities internally
3
(27%)
Wide range of other tasks associate with Blue Marble Evaluation
1
(9%)
We're a funder. We fund evaluation capacity building,
commission/manage/use evaluations, and I personally engage in
evaluative activities to understand our contribution to the
community
1
(9%)
Reviewing evaluation reports prepared by others
1
(9%)
Needs Assessments; strategic planning
1
(9%)
Improvement science implementation work, learning system
design and implementation and support
1
(9%)
Independent consultant
1
(9%)
Evaluation is a required component of our federal grant, so my
evaluation work ensures that the grant I work on continues to be
funded.
1
(9%)
Program theory development
1
(9%)
Total
11
(100%)
Mode for each item in bold.
Diversity, Equity, and Inclusion (DEI) Training. About half of respondents (43; 54%) indicate
that their workplace provides DEI training. These range from periodic DEI audits performed by
external groups, to training provided by DEI consultants or external groups, to working groups
between internal staff members.
DEI Office. Similarly, about half of respondents (43; 54%) have a DEI office or specific DEI-
related staff members in their workplace. Within this group, some have an Equity and Inclusion
Officer, others have specific DEI committees or teams, and others have offices under the same
organization as them, but not directly tied to their offices.

SCENE Report | August 2021 12
DEI Initiatives. Thirty-one respondents (39%) indicate that their workplace participates in DEI
initiatives. These initiatives range from working with a DEI consultant, to participation in the
Equitable Evaluation Initiative, to working with the AEA GEDI program. Others report that they
attempt to include program participants into each phase of their program evaluations.
Interviewees shared examples of building organizational, staff, and team capacity around
diversity, equity, and inclusion providing a fuller picture of questionnaire data.
We have instituted a team within the organization focused on DEI. So we have a chief diversity
officer and staff who report to them to kind of hold our organization more accountable to goals
in this space.
Our president, after the events of the spring and the murder of George Floyd and others,
launched a diversity, equity and inclusion task force in our [organization]. We now have two
pieces: this value statement and dedication of resources for an equity task force, and our
president saying, ‘I'm behind these things.’ That's been something to sort of hang our hat on and
to say, look, we're prioritizing these things; this is in our institutional values we need to live up
to.
My organization brought on a racial equity and racial justice consultant in the fall and did a
bunch of listening sessions and conversations.
How Evaluators Define Equity
This section reports results from an open-ended questionnaire item asking respondents to define
equity within the context of their work.
5. Defining Equity. Evaluators define equity within three areas: within the evaluation
process, as a focus when evaluating a program or intervention, and as an intended use
or influence of the evaluation.
a. Equity within an evaluation process includes minimizing evaluation team bias,
including beneficiaries in evaluation processes, and constructing multiculturally,
contextually valid knowledge.
Equity in our work means reflecting on the biases I bring to the work, examining evaluation
questions on the whole and disaggregated by race and gender at a minimum, and being clear
about actions required to achieve more equity based on the findings of any project.
Ensuring everyone has a voice in the evaluation, particularly those from historically
marginalized populations. This is done mostly during design, analysis and interpretation, and
reporting.
Being fair about how we collect, analyze, and report data and how we consider implications of
our research/evaluation such that it is inclusive, representative, and responsive to the
communities we're serving.

SCENE Report | August 2021 13
Documenting and understanding many of the underlying factors that are related to individuals’
cultures, beliefs, backgrounds and past experiences in a way that can add value to program
evaluations.
b. Equity as a focus when evaluating the intervention includes examining reach and
accessibility of interventions, examining differential experiences of intended
beneficiaries, analyzing differential outcomes, and interrogating root causes of
needs/problems addressed by interventions.
Equity means that everyone, regardless of where they live, their language, skin color, or any
other way they identify have equal access to resources and opportunities.
In our evaluations we are often examining how different groups perceive and experience in the
context of the study, but also in the larger educational context.
That the population and sub-populations in the assessment are all positively impacted by the
intervention under review.
Examining historical power imbalance and inappropriate resource allocations that have
disadvantaged some and advantaged other.
That interventions (and evaluation) examine the root causes and systemic drivers of inequities.
c. Equity as intended use of the evaluation includes challenging and expanding power
dynamics and decision-making processes, providing professional training for
intervention staff, and building evidence and guidance for social justice.
It means designing and distributing our educational products with those who have traditionally
been harmed by and excluded from STEM disciplines, with the goal of changing power dynamics
to shift away from traditional power structures for the benefit of those who have traditionally
been minoritized in STEM.
On a program level, it [equity] also means that the staff within a project have the ability to gain
professional growth whether that's about receiving training and gaining technical and "soft"
skills or the ability to "move up" in their position within the project.”
In this respect, equity means the ability for the recipients to receive the PD [professional
development] in a format & manner that meets their needs as an adult learner and then apply
the new knowledge/skills to improve education for children/students with disabilities.
I position evaluation as an instrument in the pursuit of a universal, high quality of life for all,
independent of racial identity, gender identity, income, ability status, etc. Evaluation *products*
contribute to identifying & developing interventions/knowledge/guiding principles (at both
individual & systems levels) that can get us further towards that end. Meanwhile, evaluation
*processes* empower & challenging existing power structures that exist and maintain &

SCENE Report | August 2021 14
exacerbate inequity (from evaluation stakeholder groups in planning, through
implementation/data collection, interpretation; to mixed-method designs, CRE, trauma-informed
methods).
Evaluation products contribute to identifying & developing interventions/knowledge/guiding
principles (at both individual & systems levels) that can get us further towards that end.
Whether and How Evaluators Center Equity
This section contains information on the extent to which respondents report conducting
evaluations in ways that are equitable. The section begins with an overview of the aggregate
mean scores for each phase in the order of a typical evaluation process. Then, we discuss each
phase in more detail and provide results for questionnaire items and qualitative interview data
relevant to each phase.
6. Centering Equity. SCENE respondents, on average, reported most often centering
equity during evaluation reporting and dissemination (m2.88) throughout the
evaluation (m2.73), and during data analysis (m2.67). Evaluators least often center
equity in the funding and contracting phase of the evaluation (m2.22) followed by the
evaluation team (m2.48).
On average, respondents engage in equity-related activity between ‘Sometimes’ (a score of 2)
and ‘Often’ (a score of 3) during each phase of an evaluation. Respondents rated efforts to center
equity highest for reporting and dissemination (m2.88) phase of the evaluation, and lowest for
funding and contracting (2.22) phase of the evaluation. See Table 13.
Table 13. SCENE Full Report - Phase Scores for Equitable Evaluation Practice
Phase
Mean (SD)
Team
2.48 (0.60)
Throughout
2.73 (0.61)
Funding & contracting
2.22 (0.56)
Setting questions & criteria
2.61 (0.63)
Data collection
2.64 (0.55)
Data analysis
2.67 (0.42)
Reporting & dissemination
2.88 (0.18)
Note: This tabled is ordered following the sequence of an evaluation and not from highest to lowest.
7. Evaluation Teams. SCENE respondents reported most often self-examining cultural
identities (m2.98) and assumptions (m2.93) and least often being led by evaluators who
identify as racial and/or ethnic minorities (m1.83) and having shared lived experiences
with the populations of focus (m2.05). Interviews revealed how some evaluators try to
enhance the diversity and expertise of evaluation teams by involving community
members and conducting equity reviews of evaluation plans and processes.
See Table 14 for full questionnaire results.

SCENE Report | August 2021 15
Table 14. SCENE Full Report - Evaluation Team
Item
Mean
(SD)
Never
or
Rarely
(1)
Sometimes
(2)
Often
(3)
Almost
Always
(4)
Outside the
Scope of
Our Work
Unsure
Critically reflect on how I/we relate
to others with different cultural
backgrounds, identities, and/or
perspectives than my/our own.
(n=82)
2.98
(0.80)
4
(5%)
15
(18%)
42
(51%)
21
(26%)
0
(0%)
0
(0%)
Identify and challenge my/our
personal biases and assumptions.
(n=82)
2.93
(0.78)
4
(5%)
16
(20%)
44
(54%)
18
(22%)
0
(0%)
0
(0%)
Critically reflect on how we/I share
power or alter existing power
imbalances between stakeholders.
(n=82)
2.72
(0.84)
3
(4%)
34
(41%)
27
(33%)
17
(21%)
0
(0%)
1
(1%)
Self-assess my/our privilege and
positioning within the evaluation
context. (n=82)
2.63
(0.84)
8
(10%)
25
(30%)
38
(46%)
11
(13%)
0
(0%)
0
(0%)
Conduct evaluations as part of a
racially and/or ethnically diverse
team. (n=82)
2.24
(1.00)
22
(27%)
29
(35%)
20
(24%)
11
(13%)
0
(0%)
0
(0%)
Have shared lived experiences (e.g.,
life histories, cultural perspectives)
with the population(s) of focus in
the evaluation. (n=82)
2.05
(0.74)
16
(20%)
49
(60%)
12
(15%)
4
(5%)
0
(0%)
1
(1%)
Conduct evaluations with leads
(PI's, supervisors) who identify as
racial and/or ethnic minorities.
(n=82)
1.83
(0.95)
37
(45%)
28
(34%)
9
(11%)
7
(9%)
0
(0%)
1
(1%)
Mode for each item in bold.
We identified the following themes among interviewees regarding their efforts to center equity
within evaluation teams, illustrated by select quotations:
• Try to Involve Team Members with Different Perspectives and with Similar Lived
Experiences as Intended Beneficiaries
…if I'm thinking about putting a team together, it's because I've learned that who is on your
evaluation team and what experiences they bring is really, really important. So, thinking about
who can I bring that will offer different perspectives and as close as perspectives to the
community that the work is to serve as possible.
…I have spent a lot of hours...just trying to build the network of people who are doing this work
from an equity perspective and who have expertise that's relevant to things that we're doing.

SCENE Report | August 2021 16
So, one of the things that we've started to do is expand the people involved in developing the
scope of work...And I think one of the important parts is seeing the evaluator as also a
stakeholder in that, particularly if you are seeking to hire an evaluator who has lived experience
with whatever intervention or evaluand that’s featured as part of the project. And having them as
a representative of equitable practice as well.
• Involve & Provide Incentives for Community Members to Participate as
Evaluators and within Evaluation Processes
There isn't this dichotomy of evaluators and lay folks that we use, I think the term that we've used
in the past in projects is community evaluators...creating opportunities to stipend community
members [to work as evaluators]. I really think that non-advisory bodies, involved bodies,
people who are actually doing some of the work and who are compensated for it. Big star next to
that, compensated for that is great whenever possible.
In terms of hiring local people that - I guess that's where a lot of the capacity building comes
into is training promising individuals to become experts in their different roles and whether it
will be research assistants or - and depending on their interest as well. Some people are just
doing it for the work and other people are very passionate about the work.
• Review Evaluation Plans and Processes to Minimize Bias and Inequity
A few years ago, I was talking to my manager and pushing for this work, and I suggested we do a
systematic audit of our practices with regards to equity. And at that point, we were taking a
culturally responsive evaluation lens to be able to look across what we do and sort of make
recommendations about where we can push forward.
8. Throughout the Evaluation. Respondents most frequently highlight the strengths of the
intended beneficiaries of the evaluation (m 3.09) and work to build trust with
stakeholders from minoritized or marginalized communities (m3.01). Respondents least
frequently work to mitigate power imbalances in the evaluation context (m2.45) and
consult with cultural brokers or translators to mediate between the cultures of
stakeholders and the evaluation team (m2.26).
See Table 15 for questionnaire results for items regarding equity throughout evaluation
processes. We do not include interview data in this section, as themes fit better in relation to
phases than throughout the process.
Table 15. SCENE Full Report - Throughout the Evaluation
Item
Mean
(SD)
Never
or
Rarely
(1)
Sometimes
(2)
Often
(3)
Almost
Always
(4)
Outside
the Scope
of Our
Work
Unsure

SCENE Report | August 2021 17
Highlight the strengths of the
intended beneficiaries of the
evaluation. (n=82)
3.09
(0.80)
2
(2%)
16
(20%)
34
(41%)
27
(33%)
0
(0%)
3
(4%)
Work to build trust with
stakeholders from minoritized or
marginalized communities. (n=82)
3.01
(0.86)
3
(4%)
20
(24%)
31
(38%)
27
(33%)
0
(0%)
1
(1%)
Minimize any potential for the
evaluation to exacerbate
disadvantage or inequity. (n=81)
2.96
(0.88)
3
(4%)
22
(27%)
27
(33%)
25
(31%)
1
(1%)
3
(4%)
Encourage intervention leaders
and/or staff to address root causes of
the problem(s) an intervention
targets. (n=81)
2.81
(0.90)
3
(4%)
31
(38%)
21
(26%)
22
(27%)
2
(2%)
2
(2%)
Conduct evaluation activities in the
languages that are
relevant/appropriate for the
community. (n=81)
2.66
(0.98)
10
(12%)
20
(25%)
25
(31%)
16
(20%)
8
(10%)
2
(2%)
Challenge any assumptions that the
intended beneficiaries lack the
ability to achieve because of their
culture (i.e., deficit assumptions).
(n=82)
2.65
(0.99)
9
(11%)
30
(37%)
20
(24%)
20
(24%)
1
(1%)
2
(2%)
Work to mitigate power imbalances
in the evaluation context. (n=82)
2.45
(0.86)
8
(10%)
38
(46%)
21
(26%)
11
(13%)
1
(1%)
3
(4%)
Consult with cultural brokers or
translators to mediate between the
cultures of stakeholders and the
evaluation team. (n=82)
2.26
(0.93)
17
(21%)
33
(40%)
19
(23%)
9
(11%)
2
(2%)
2
(2%)
Mode for each item in bold.
9. Funding and Contracting. Most often, SCENE respondents have to work within
purposes and questions set by the funder/commissioner (m2.86) although they try to
prioritize RFPs focused on inclusion, diversity, and equity (m2.52). Interviewees
described centering equity in the funding and contracting phase in several ways: choose
RFPs carefully, make equity explicit in proposals, have up-front conversations, and
budget accordingly.
See Table 16 for questionnaire results for items regarding equity during funding and contracting.

SCENE Report | August 2021 18
Table 16. SCENE Full Report - Funding and Contracting
Item
Mean
(SD)
Never
or
Rarely
(1)
Sometimes
(2)
Often
(3)
Almost
Always
(4)
Outside
the Scope
of Our
Work
Unsure
Have to work within specific
purposes and questions set by the
funder/commissioner. (n=80)
2.86
(0.87)
5
(6%)
18
(23%)
33
(41%)
18
(23%)
5
(6%)
1
(1%)
Prioritize requests for proposals that
have purposes and/or questions
focused on inclusion, diversity,
and/or equity. (n=80)
2.52
(1.02)
12
(15%)
22
(28%)
19
(24%)
14
(18%)
8
(10%)
5
(6%)
Have an adequate budget that
allows time and resources to
involve stakeholders in evaluations.
(n=80)
2.39
(0.90)
11
(14%)
33
(41%)
20
(25%)
10
(13%)
4
(5%)
2
(3%)
Have to work within a specific
design and methods set by the
funder/commissioner. (n=80)
2.15
(0.84)
17
(21%)
33
(41%)
20
(25%)
4
(5%)
5
(6%)
1
(1%)
Negotiate with funding agencies to
have purposes and/or questions
focused on inclusion, diversity,
and/or equity. (n=80)
2.09
(1.00)
22
(28%)
22
(28%)
14
(18%)
7
(9%)
9
(11%)
6
(8%)
Work with stakeholders that request
culturally responsive and/or equity-
focused evaluations. (n=79)
1.97
(0.74)
18
(23%)
42
(53%)
10
(13%)
3
(4%)
3
(4%)
3
(4%)
Mode for each item in bold.
We identified the following themes among interviewees regarding their efforts to center equity
during funding and contracting, illustrated by select quotations:
• When Possible, Select RFPs that Encourage Evaluators and Evaluations to Examine
Inequity. Consider Ways to Renegotiate Scopes of Work to Incorporate Equity.
So starting with applying for an RFP, thinking about the evaluation, how early on evaluation is
part of the work and even resisting within RFPs things that we know aren't aligned with our
values or responding in ways that are intentionally shaped.
It just happened to me recently...one of my colleagues sent me an RFP that I think was
evaluating services that are provided to women as they leave incarceration. And I read the RFP
and I said my first concern here is that they list various stakeholders that they want to see

SCENE Report | August 2021 19
involved in the evaluation, and they're not naming women that experience incarceration or have
been incarcerated...So that was kind of a red flag for me.
And so I think that was one of the things that the firm in California was really good at was they
started redoing how they did scopes of work. And they built in like a feedback loop so that you
could renegotiate the scope of work periodically because… we all determined that it needed to
be able to grow and breathe with the program, as programs change. And so that seemed better
able to both set their expectations that evaluations change over time and that we can revisit this.
• Make Equity Explicit in Your Work and Proposals.
One of those practices lately, for my team, has been working on grant proposals. It's at the
beginning of developing what our projects would be. And some of these have technically been
research studies. I think at the beginning, that's where we've been pushing a lot while we're
initially writing these evaluation plans for grant proposals. How can we advance culturally
responsive evaluation practices?
• Have Up-front Conversations with Clients About Values and Approach to
Evaluation.
So, wherever I can, I take what I would term a “ground-up approach” where I try to first and
foremost put a lot more effort into the planning phase of the evaluation… Have in-depth
conversations to try to understand the meaning behind those goals and impacts that they're
trying to evaluate and collaboratively design those tools with them.
• Propose Budgets that Provide the Time and Resources for Equity-focused
Evaluation, When Possible.
It's my job to figure out what is fair for our work and what can be a fair study. I would not take
on a study where I think I cannot do the work properly and also be fair to my practice. I won't
give you a watered-down study that doesn't really tell you what you need just because it's what
you have.
So, I've started to budget for more participation at the design phase, so that's just straight up
when I write a proposal now… There's money behind that. There's money for me to do at least a
few interviews with people who are being affected by the projects.
10. Evaluation Questions and Criteria. SCENE respondents often examine the underlying
problem an intervention seeks to address (m3.08), but much less often answer questions
about how an intervention addresses systemic drivers of inequity (m2.29) and least
often include intended beneficiaries in evaluation decision-making (m2.28) and in
defining evaluative criteria (m2.29). Nearly all respondents evaluate effectiveness (73;
92%) followed by most who examine the relevance of the intervention to the needs,
culture, interests, or circumstances of the intended beneficiaries (65; 82%). More than
half of respondents (46; 58%) examine equity of the intervention’s opportunities,

SCENE Report | August 2021 20
experiences, benefits, and/or results as an evaluation criterion within their evaluation
work.
See Table 17 for questionnaire results for items regarding equity during the framing of
evaluation questions and setting of evaluation criteria.
Table 17. SCENE Full Report - Evaluation Questions and Criteria
Item
Mean
(SD)
Never
or
Rarely
(1)
Sometimes
(2)
Often
(3)
Almost
Always
(4)
Outside
the Scope
of Our
Work
Unsure
Examine the problem the
intervention seeks to address and
how that problem is defined.
(n=81)
3.08
(0.90)
4
(5%)
17
(21%)
28
(35%)
31
(38%)
0
(0%)
1
(1%)
Include evaluation question(s)
about the outcomes and/or
impacts of an intervention on
different populations. (n=80)
3.05
(0.92)
4
(5%)
19
(24%)
25
(31%)
31
(39%)
1
(1%)
0
(0%)
Include evaluation question(s)
related to equity, inclusion, and/or
diversity. (n=81)
2.72
(0.88)
5
(6%)
31
(38%)
27
(33%)
18
(22%)
0
(0%)
0
(0%)
Negotiate criteria/definitions of
success across different
stakeholder groups. (n=81)
2.48
(0.86)
11
(14%)
27
(33%)
33
(41%)
8
(10%)
0
(0%)
2
(2%)
Include intended beneficiaries in
selecting criteria/definitions of
success. (n=81)
2.29
(0.90)
15
(19%)
33
(41%)
22
(27%)
8
(10%)
2
(2%)
1
(1%)
Include evaluation question(s)
about how an intervention impacts
or addresses underlying systemic
drivers of inequity. (n=81)
2.29
(1.01)
20
(25%)
26
(32%)
21
(26%)
11
(14%)
2
(2%)
1
(1%)
Include intended beneficiaries in
decision making about the
evaluation purpose, approach,
and/or questions. (n=81)
2.28
(0.88)
16
(20%)
31
(38%)
26
(32%)
6
(7%)
1
(1%)
1
(1%)
Mode for each item in bold.
SCENE respondents most frequently use effectiveness of the intervention at achieving desired
results, outcomes, or objectives (73; 92%) followed by relevance of the intervention to the needs,
culture, interests, or circumstances of the intended beneficiaries (65; 82%). More than half of
respondents (46; 58%) examine equity of the intervention’s opportunities, experiences, benefits,

SCENE Report | August 2021 21
and/or results as an evaluation criterion within their evaluation work. Notably, all respondents
use some criterion in their evaluations. Table 18 provides additional questionnaire results on the
criteria evaluators use. Those in italics related to equity.
Table 18. SCENE Full Report – Domains of Evaluation Criteria Used
Item
Response
N
(%)
Evaluation
Criteria
Effectiveness of the intervention at achieving desired results,
outcomes, or objectives
73
(89%)
Relevance of the intervention to the needs, culture, interests, or
circumstances of the intended beneficiaries
65
(79%)
Intended consequences/effects of the intervention
64
(78%)
Experience of intended beneficiaries during the intervention
62
(76%)
Design of the intervention in accordance with relevant theoretical
principles, best practices, standards, and/or laws
60
(73%)
Sustainability of the intervention to continue beyond the start-up
period or for the benefits to occur over longer time frame
56
(68%)
Alignment of the intervention with larger initiatives, related
interventions, funder aims, and/or interconnected problems
53
(65%)
Unintended consequences/side effects of the intervention
50
(61%)
Equity of the intervention's opportunities, experiences, benefits,
and/or results, with particular consideration to prioritizing
marginalized populations
46
(56%)
Replicability of the intervention to be duplicated or adapted to
another context
35
(43%)
Costs and resource use of the intervention
31
(38%)
Something else (please specify):
5
(6%)
Non-Response
3
(4%)
We do not use criteria in our evaluations
0
(0%)
Total
82
(100%)
Other
Evaluation
Criteria
Alignment between principles of the program/strategy and actual
practices
1
(20%)
Attribution (or not) of impact to intervention
1
(20%)
Satisfaction of participants in the evaluation process/outcomes
1
(20%)
Social & ecological considerations
1
(20%)
Utilization focus
1
(20%)
Total
5
(100%)
Mode for each item in bold. Italicized items pertain to equity.
In the interviews, few evaluators shared examples of incorporating equity into evaluation
questions and criteria. Several indicated that they work within the parameters of the questions
framed by clients and contracts and struggled with how to challenge and expand these to allow
for the examination of inequities and disparities. For instance, one evaluator noted:

SCENE Report | August 2021 22
…questions around equity come from good intentions, but they're always on the sideline. They're
always on the back burner because the questions from the funder take so much time and the data
management takes so much time that it's almost like your box of special projects that you want to
work on. And that's kind of where the equity work lives.
11. Data Collection. SCENE respondents often design and/or modify data collection to
culturally and/or linguistically suit the participants (m3.04). Interviewees expanded on
this by sharing examples of how they try to center equity during data collection, such
as: develop instruments with participant consultation and piloting; use data collectors
with shared identity/lived experience; conduct member checking; and consider access,
reduce burden, and maximize benefit for participants in data collection.
See Table 19 for questionnaire results for items regarding equity during data collection.
Table 19. SCENE Full Report - Data Collection
Item
Mean
(SD)
Never or
Rarely
(1)
Sometimes
(2)
Often
(3)
Almost
Always
(4)
Outside the
Scope of
Our Work
Unsure
Design and/or modify data
collection to be appropriate for the
culture(s) and language(s) of the
people of whom the questions are
being asked. (n=80)
3.04
(1.05)
7
(9%)
16
(20%)
15
(19%)
33
(41%)
5
(6%)
4
(5%)
Use systems thinking and/or
systems methods (e.g., social
network analysis, causal loop
diagrams, agent-based modelling,
critical systems heuristics). (n=79)
2.25
(1.09)
22
(28%)
27
(34%)
11
(14%)
15
(19%)
2
(3%)
2
(3%)
Mode for each item in bold.
We identified the following themes among interviewees regarding their efforts to center equity
during data collection:
• Invite Participant Input When Developing and Piloting Instruments
Whether it's designing the instruments, like designing the focus group protocols we…do
participatory design of the tool, even implementation of it. We try to avoid this big, dramatic top-
down thing.
We do definitely look at instrument development and may need to translate the survey items. In
the past, we have definitely translated survey instruments into Spanish.
• Involve People with Shared Identities and Lived Experience in Data Collection

SCENE Report | August 2021 23
In data collection and analysis, I mean it all kind of ties together like who you have on your
team, who's collecting the data. I’ve tried to strike a balance between people with similar lived
experience, but also trying to have diverse perspectives on the team, whether that's kind of like
racial diversity or economic diversity or cultural diversity. So it's a little bit of both, and that
kind of affects how you're collecting data and then in analysis.
• Consider Ways to Provide Access, Reduce Burden, and Fairly Incentivize People
Participating in Data Collection
I'm also thinking about who's doing the data collection. And what I think is basic practice now,
how we try to think about who we're collecting from: the timing of it. Do we need to offer
transportation? Do we need to think about how people are getting child care? Asking all those
sorts of relevant questions.
… we've tried to shift our model so it's less about reducing burden and more about maximizing
benefit. And there has been some interesting pushback around the, do we get their contact
information just for the sake of being able to share back later?
We provided cash, which I think is a very person-centered and respectful and dignified thing to
give someone when you're asking for their time. And then we also provided fruit items that the
program didn't typically have because they tried to utilize a lot of local produce.
I've only done this for foundations and hospitals that tend to have more money or don't have the
same restrictions. I haven't done this yet, for example, for like a government funded project, and
that might look different… but it's definitely about more time… and a bigger pot of money for
incentives.
12. Data Analysis. When analyzing data, respondents reported often disaggregating data
by demographic differences (m3.36), looking for potential negative consequences
(m3.10), and looking for differential participant access and experiences (m3.00). Least
often respondents look at shifts in power to intended beneficiaries and communities
(m2.21) and re-distribution of resources (m2.19). Interviewees highlighted how they
sometimes disaggregate data and also how they struggle with this due to small sample
sizes for some minoritized groups and how they struggle to go beyond disaggregation to
analyze inequities more deeply and critically.
See Table 20 for questionnaire results for items regarding equity during data analysis.
Table 20. SCENE Full Report - Data Analysis
Item
Mean
(SD)
Never or
Rarely
(1)
Sometimes
(2)
Often
(3)
Almost
Always
(4)
Outside the
Scope of
Our Work
Unsure
When possible, disaggregate data
by key demographic differences.
(n=79)
3.36
(0.81)
1
(1%)
13
(16%)
21
(27%)
43
(54%)
1
(1%)
0
(0%)

SCENE Report | August 2021 24
Look for potential negative
consequences of the intervention.
(n=79)
3.10
(0.91)
2
(3%)
22
(28%)
19
(24%)
34
(43%)
1
(1%)
1
(1%)
Look for disparities in access to
program services among subgroups
of intended beneficiaries. (n=80)
3.00
(0.98)
5
(6%)
21
(26%)
19
(24%)
31
(39%)
3
(4%)
1
(1%)
Examine differential experiences
of participants with/during the
intervention. (n=80)
2.99
(0.85)
2
(3%)
22
(28%)
29
(36%)
25
(31%)
0
(0%)
2
(3%)
Analyze how interactions between
race, ethnicity, class, gender,
sexual orientation, etc. influence
differential outcomes. (n=80)
2.84
(0.84)
2
(3%)
27
(34%)
27
(34%)
19
(24%)
4
(5%)
1
(1%)
Include intended beneficiaries in
interpretation of data and sense-
making. (n=79)
2.49
(0.99)
13
(16%)
29
(37%)
22
(28%)
15
(19%)
0
(0%)
0
(0%)
Look for potential negative
consequences of the evaluation.
(n=79)
2.49
(1.14)
19
(24%)
21
(27%)
16
(20%)
20
(25%)
1
(1%)
2
(3%)
Examine the extent to which
intended beneficiaries were
actively involved in the planning
and implementation of program
activities. (n=80)
2.41
(0.89)
12
(15%)
28
(35%)
27
(34%)
8
(10%)
4
(5%)
1
(1%)
Look for whether and/or how the
intervention contributes to
systemic and/or structural change
on a local, regional, or larger scale
(n=80)
2.39
(0.91)
11
(14%)
35
(44%)
19
(24%)
11
(14%)
4
(5%)
0
(0%)
Look for whether and/or how the
intervention shifts power to
intended beneficiaries and their
communities. (n=80)
2.21
(0.96)
17
(21%)
33
(41%)
12
(15%)
10
(13%)
7
(9%)
1
(1%)
Assess whether interventions re-
distribute resources to those most
marginalized and/or
disadvantaged. (n=80)
2.19
(0.97)
19
(24%)
26
(33%)
16
(20%)
8
(10%)
8
(10%)
3
(4%)
Mode for each item in bold.
We identified the following theme among interviewees regarding their efforts to center equity
during data analysis:
• While Evaluators Know to Disaggregate Data, They Encounter Challenges with
Small Sample Sizes, Limited Data Availability for Some Demographics (e.g., sexual
orientation), and Getting Beyond Representation to Disaggregate Outcome Data

SCENE Report | August 2021 25
So I would say that this is probably our least strong area in terms of data analyzing and that sort
of thing, I don't do a ton of it, but I hear a lot from my colleagues about, well, I had to repress
some of the data or oh, I couldn't look at it by X, Y, Z because the numbers were so small and
that we'll get back into the predominantly white New England.]
I would say the biggest gap that I've seen is that we do not collect sexual orientation data. We
collect gender identity data but not sexual orientation, which I think would be a big
organizational shift, and I don't know that the organization would be ready for it, but I think
that's something that's very interesting and it's an aspect of inequity that I think is really
important.
I would say [pause] there is little thought there. I feel like the analysis phase could be really
useful for looking at inequities if the tools are set up to analyze... but I think the tools aren't
really set up that way, and then the analysis phase doesn't bode well for looking at inequities.
And I don't think there's any key evaluation questions that ask... in a lot of situations, it's just,
what were the outcomes, but not any sub questions?
I think getting beyond the diversity of those you serve into equity of outcomes. If we're talking
about equity, we're not just talking about representation, we're talking about outcomes. I think
that's a big one. We haven't gotten all that far with that but I think we're learning there.
13. Reporting & Dissemination. Survey data indicate that a majority (54; 68%) often or
almost always present reports in formats accessible to intended beneficiaries and about
half (44; 55%) make evaluation results, in some form, publicly accessible. Some
interviewees discussed how they embed processes for shared sensemaking and
disseminate in multiple, accessible formats and across stakeholder groups. Other
interviewees struggle with collaborative sensemaking and accessible dissemination.
See Table 21 for questionnaire results for items regarding equity during reporting and
dissemination
Table 21. SCENE Full Report - Reporting and Dissemination
Item
Mean
(SD)
Never or
Rarely
(1)
Sometimes
(2)
Often
(3)
Almost
Always
(4)
Outside the
Scope of
Our Work
Unsure
Present evaluation results in
formats accessible to the
intended beneficiaries of the
intervention. (n=80)
3.04
(0.97)
5
(6%)
21
(26%)
20
(25%)
34
(43%)
0
(0%)
0
(0%)
Make evaluation results, in
some form, publicly
accessible. (n=79)
2.75
(0.93)
6
(8%)
27
(34%)
24
(30%)
20
(25%)
2
(3%)
0
(0%)
Mode for each item in bold.

SCENE Report | August 2021 26
We identified the following themes among interviewees regarding their efforts to center equity
during reporting and dissemination:
• Try to Embed Processes for Shared Interpretation and Discussion of Results with
Those Who Participated in the Evaluation
We did a needs assessment for supports and resources for people of color and women who own
small businesses. The evaluator…had a sensemaking session with community stakeholders. So
other business owners, lenders, banks, nonprofit development corporations, the foundation staff,
they came to this. We all had a joint sensemaking session.
We had a data party with all of the folks who had contributed to the study along the way. It was
optional, but people could join and it was like the first draft of the findings and they were into
different breakout rooms in Zoom and used a jam board to like basically the responses that I was
asking for was like, what are your key takeaways? What questions do you have and what
recommendations do you have for [project name] based on what we're seeing here?
One thing that we have done was taken the data and brought it back to the youth and said, this is
what we found. Does it sound right? And they'll say, oh, God, no. The youth obviously lied on
this question. There is way more pot use among us than what's in this. Twenty five percent or
whatever. So, we try as much as we can to kind of bring that data back to the population and say,
does this look right? And if not, why so?...
• Many Interviewees Provided Examples of Ways They Disseminate Evaluation
Results in Multiple Formats
.
We really make efforts to share back data, whether it's in flyers or digested ways of sharing the
data back. Sometimes it'll be in a community monthly meeting with the community partners and
residents…It's very collaborative.
Something that we want to continue doing in terms of dissemination...We do a lot of silly stuff
and hashtags and infographics and one-pagers and just trying to get folks excited, it really
depends on what it is and how actionable it is for people.
Something that I've been experimenting with colleagues is in the dissemination, bringing in other
voices who have nothing at all to do with the work to help kind of flatten that power dynamic. So,
who can comment on some of the questions that were raised but aren't necessarily working in
that same ecosystem? So it helps…they're speaking from a relative position of safety and power
because their livelihood is not dependent on resources from this funder. And it enables them to
say, oh, yeah, we think this is…heading in the right direction or we don't really think this is
what's happening here because of X, Y and Z.

SCENE Report | August 2021 27
Helpful and Hindering Factors for Centering Equity
Now that we’ve reviewed how evaluators address equity in the phases of an evaluation, this next
section explores the factors that help and hinder centering equity within evaluations, drawing
solely on themes from the interviews.
14. Helpful Factors. Interviewees discussed how factors within their teams, organizations
and wider society all can help to support equity-focused evaluation processes.
a. Within teams, evaluators emphasized the importance of having members with diverse
cultural and linguistic expertise, including community and new evaluator voices, and
fostering critical, self-reflective, and learning cultures.
And for instance, a grant that we're putting in next month, I'm super excited that we have a team
that is more than half people of color. It also includes a co-PI who has a disability. It has
multiple team members who identify as LGBTQ, and that we've been able to (pause) diversify
our group through those partnerships.
We were taking a culturally responsive evaluation lens to be able to look across what we do and
make recommendations about where we can push forward…. (A)ll of our staff members and also
people with whom we work in other departments went through a round of training…. we find
different spots in our work where we can advance these practices and we tested something out
that's new and documented it systematically.
b. Organizational helpful factors include having missions and values aligned with equity,
leadership support, internal diversity, equity, and inclusion work, and dedicated staff
time.
I know I've got the backing of everyone, all the leadership. Everyone up to the CEO is going to
be like yeah, whatever you need, do it. And they're not going to let us make excuses to not center
[equity].... I think knowing I've got that backing and support is huge.
...starting with our organizational culture overall….For instance, this past year, we had a task
force to develop values for our organization. And I was on that task force and worked really
hard to get the word equity in it after a lot of discussion that not everyone was comfortable with
it, but ultimately it is in there.
c. Public attention on issues of race and equity in 2020 provided a context to further
advocate for equity-focused evaluation.
Our [organization’s] president, after the events of the spring and the murder of George Floyd
and others, launched a diversity equity and inclusion task force in our organization….That's
been something to sort of hang our hat on and to say, look, we're prioritizing these things. This is
in our institutional values we need to live up to…. It's not me being the crazy advocate over
here… we've committed to these things publicly and we need to do them.

SCENE Report | August 2021 28
In the past year, everyone's talking about racism and inequity, that's helped. It's helped to just
bring it up as a normalized piece of conversation. And then because people want to be equitable
But I think this whole dual pandemic that we've just come through certainly is shedding a lot of
light on what we need to be doing as a society. And evaluators are part of that. So it's a lot
bigger than our little world here.
15. Hindering Factors. Interviewees identified how factors related to evaluation contracts,
evaluators’ racial identities, evaluation stakeholders, and the evaluation profession can
all hinder centering equity in evaluation.
a. Within and across contracts, evaluators find that quick turn-around time, limited
budgets, restricted scopes of work, the need to secure and keep contracts, stress and
burnout, and constrained evaluator autonomy all work against equity-focused
evaluation.
The often really ridiculously tight time frames that we're working in, very tight budgets limit.
You're constantly having to kind of narrow who gets to speak.
I think there also is probably a little bit of burnout, too, potentially when you're kind of finishing
up a project and granted, many of our programs just were ongoing. But I think you kind of get a
little bit burned out or maybe there's a timeline, especially if it's from a funder. And so you've got
to produce the last report by this date, get it to the fundraiser. And there's probably such a ramp
up and it's probably so stressful that afterwards you're just like, I've got to take a break and I
need to move on to something else or I need to pivot a little bit and I'll get back to that when I
can. And then you never get back to it...
And you know, the money, the amount of time that I need to spend on projects to pay my salary
versus the organizational work that we're doing internally, we've been pushing for a while to get
an external facilitator to work on anti-racism stuff at our organization. And the budget is a
barrier. We're getting there. But it wasn't built into the budget this year, so finding the money for
it.
We're small in budget, we're small in staff and organizational stability - it almost sometimes feels
like it's in conflict with spending a lot of time to do this good work. We can't turn down contracts
at this point in our institutional life, we can choose not to pursue opportunities and that feels just
like an opportunity cost. But if we're offered a job and it raises red flags, it is very difficult for us
to say no to it despite the red flags.
I would say in terms of some hindrances, I think scopes of work, like sometimes they're so small.
That they only have the budget to focus on a few things and broadening it is hard without us
going way beyond what we promised to do. And then some of the larger projects are so scripted
and it's just you have to do what they say you have to do and evaluate in the way they want you
to evaluate and measure what they say should be measured.
b. Some interviewees talked about their own and colleagues’ white racial identities as
constraints that minimize seeing and addressing inequity. Several interviewees of color
shared examples of racial ignorance and discrimination, including being treated as a

SCENE Report | August 2021 29
token representative and being expected to do more as evaluators of color which
created inequities for them as evaluators.
The first several quotations are from white evaluators.
I'm trying to figure out how to look at race and incorporate culturally responsive practices, but
sometimes that can be difficult and I think it's different because I'm a white person, and a lot of
the youth are non-white. And a lot of our staff are also white, so I think there is this idea of like,
OK, well, we don't have to look at race because... they're all youth of color, so I try to do some
things, but I also don't think I understand fully how to do those.
I would say that most of what kind of drives my challenges related to equity work is around the
intra-personal characteristics of individuals, either in my organization, in programming, just
kind of white folks in New England...So there's a lot of confusion and a lack of understanding, in
my experience, about how equity exists or doesn't exist in New England.
I'm in New Hampshire and Vermont, and when I do local evaluation work, my colleagues often
say well, I'm not sure we should look at race or other demographics because we're all white. So,
I keep trying to figure out what to say to that, that's not just like that's the end of the
conversation, but trying to think of well if a program only has…what you're saying is all white
participants, is there an equity issue in the program? Should we be examining that?...
I have questions around how do you do that when you're working with non-diverse populations,
because when I first read the email about your study, it was sort of like I wasn't going to do the
survey because I don't do- I'm working with Caucasians...all the Caucasians work with other
Caucasians. Like there's not even. They're not even thinking about anyone that doesn't look like
them, so how can I talk about equity and inclusion and diversity within these contexts?
These last two quotations are from evaluators of color.
What might hinder me is the idea that for those people who want representation, they may have
altruistic goals, but sometimes being the token hinders me. That you think that you're bringing
me into this project because the project deals with race stuff and you don't have caché to enter
those communities and therefore you're going to bring me onto your team so that I can enter
those communities, give you a pass basically.
Well, long story short, something a colleague said let me know he thought I wasn't pulling my
weight as a co-PI. And when I investigated a bit more about what he meant by that…he was
intimating that he thought I would do more because half of these students were students of color.
c. In working with stakeholders, evaluators find differences in what equity means,
constraints on challenging inequities built-into programs, and the lack of use of
evaluation findings all to hinder equity.
…I'm sure our [organization’s] president has read definitions of equity, but I'm not sure he
really understands or at least shares my definition of equity. We have a lot of people who are

SCENE Report | August 2021 30
very equality minded in our organization and think that equity is about equality…I would say
there is a big hindrance or stopping, as you put it, just not being on a shared page about what
equity is.
One thing that's been really interesting is that our president's perspective is that DE&I,
including equity, should be integrated into everything we do, which I think is useful in a lot of
ways. But it also means that it's never the focus of things that we do… But it's a real struggle to
continue to get it inserted in a prominent way in the work that we're doing.
Not only what are their perspectives, but because the dynamics are so power laden that as an
evaluator, I really struggle with the ability to even hear those perspectives. I even question
whether it's possible to create a scenario where I, as an evaluator working for the funder of their
funder, could ever hear genuine feedback from an organization because of the risks that they
experience.
… When you talk about equity, sometimes it's almost a futile effort. We're bringing in this cool
project about looking at osmosis through an iPad, you know, touch screen simulation from a
research university. That's so cool. But, you know what, that kid didn't have breakfast.
But this is the frustrating part for me. People will pay X thousands of dollars sometimes and
never look at those evaluation findings again, and sometimes they never leave the room. And as
an evaluator, you signed agreements, you signed a contract, I cannot make many of my
evaluation public. They're not for public consumption. But we walk away and you have to
wonder, to what extent did any of these really make it? And to a certain degree, that happens at
all stages, right?
d. Interviewees pointed to ways evaluation itself can constrain equity in its professional
culture and norms that privilege scientific knowledge and objectivity, assumed
evaluator role as experts, and a sense of isolation, competition, and little sharing
between evaluators and evaluation organizations.
I think we have to do some changes in our cultures of evaluation, of defining what a good
evaluation is in ways that place more value on the diversity of voices and challenging
perspectives that we bring in.
I remember working as a young evaluator and never being put as a P.I., even if I had experience
with a given population or an area or a methodology because I didn't have my Ph.D. yet. If I
experienced that as a cis white woman, I can only imagine what's happening for other folks who
are trying to break into and work in the field…. I think there's a lot of gatekeeping about who is,
who has the credentials or experience or whatever to undertake the traditional and
nontraditional aspects of evaluation.
Like we were talking there's the accidental evaluator, especially, you know, if it's been selected
to your job description and you're actually a program manager now, you have to do an
evaluation. So, I think that does get really, really lonely. You're trying to figure it out. You don't

SCENE Report | August 2021 31
know where to go because it's not your profession. To have the lone evaluator in an organization
there, they're an island of one. Yeah, they're lonely. They're isolated.
I think that's a huge barrier in this work. There's no need to reinvent the wheel, which is why we
get stuck in those evaluation methodologies where we did this so we'll take it and apply it to this
project without really thinking about it...Someone else has thought about this, but there's a
disconnect between evaluators. Just because of...I don't know if it's competition between
businesses, or just a lack of opportunities to share ideas. I don't need something formal, but I'm a
nerd, I like talking about this stuff with other evaluators... there's not a lot of opportunity outside
of my companies that I've worked for to talk and share ideas. I think that's a huge gap.
There's not really a space for that kind of dialogue currently. GBEN has round tables, but it's
like totally on me to say, hey, I want to do a roundtable discussion about this, which I think I
will. But there's just not a natural way for that to happen except for me to step up and say, this is
the topic I want to talk about. Yeah. Can you give me a month?... So I think that needs to be more
explicit, like I think there needs to be a more obvious place for those conversations to happen.
I think something that is a recommendation is that you work solo and I think a lot of other
evaluators work solo and there's not really an opportunity to share knowledge and push back on
each other's thinking. So it was almost like you're recommending having teams of practice, I
don't know how that would work, but maybe even outside of people's organizations, that would
be really challenging… It's difficult because it's almost like you need someone to work with you
on projects or feedback or there's going to be different models of doing some of this work where
maybe there's a consultant who double checks work.
Ways to Strengthen Capacity for Equity in Evaluation
This final section explores interviewees’ ideas for strengthening capacity for centering equity in
evaluation practice as well as the profession.
16. Strengthening Capacity. Suggestions for strengthening capacity for centering equity in
evaluation include enhancing evaluator skills, engaging evaluation stakeholders, and
changing how the broader evaluation field and profession to advance equity.
a. Enhancing and applying evaluator skills. Interviewees’ suggestions began with ways
evaluators can build their own capacity by participating in trainings about antiracism,
racial justice, and intersectionality and how these lenses can be applied to methods;
documenting, self-assessing and seeking critical feedback on evaluation processes;
learning how to work together across different cultural identities and ways of knowing;
and critically applying evaluation skills, such as situational analysis, to points to
influence inequities within the evaluation context. Finally, they urged evaluators to
make the time and space for addressing inequity.
And there is so much to learn that I think kind of just engaging in learning about equity and
specifically racial equity and racial injustice, I think is really just crucial for everybody and is a
necessary baseline in order to be able to incorporate it in the work. So I think I would definitely
encourage people to be doing their homework, even if it's not specific to evaluation.

SCENE Report | August 2021 32
Part of that work for white evaluators is understanding how to work with evaluators who have
different backgrounds than ours. And in a way that's, you know, not just being deferential, but
really being supportive colleagues and recognizing all of the issues that come into play when
you're working in spaces that are both kind of challenging, that you're expected to deliver work
that fits someone else's standards, that you work in ways that respect, internalized power
dynamics, externalized power dynamics, I think there's a whole layer of sensitivity and training
and understanding and insight that goes way beyond methodology.
I would really love honest feedback at all points in the process, I would love to learn more about
how other people have done the context thing…I would love to hear how other people have done
it, how other people are doing it, especially when they feel like they have a mandate to do it, like
what is a creative, crazy thing that people are trying.
Something that's new for me is really thinking about why and how quantitative methods have
been rooted in an understanding of statistics that has served sort of white supremacist purposes
and what we can do to use it in a more equitable way and or what should we use instead. And I
know some of this stuff isn't just about needing a better methodology, but needing a deeper
understanding. But I think we need to start somewhere.
I think something I've learned… is to make the time and the space for it, because otherwise other
folks might not, even if they're well intended…I think sometimes equitable practices and doing
things to the best of our ability to change systems almost feels threatening, but I think at the end
of the day, if you just try, it isn't as much, it just takes more conscious effort.
b. Build professional learning among evaluators. Beyond individual evaluators and
evaluation teams, interviewees pointed to the need to build professional learning by
sharing practical guidance, concrete examples, and lessons from practice and creating
professional learning communities.
I think in theory it's great. And you can try to do it as much as you can. But what does it really
look like in action and what does a good one look like? And what does a bad one look like? And
lessons learned from that? I think we're still fairly new in this type of approach. So I don't think
we have a lot of examples yet.
I think there needs to be like more concrete steps of what people are doing because everyone's
just trying to shove in these theories, even in applied webinars and more practitioner focused
things like I attended AEA's conference and it was great, but I think I was still hungry for
something more tangible.
I've attended a lot of webinars…really talking to folks and then implementing it. But how do you
navigate some of those trickier circumstances or are there any best practices that are more
tangible or actual instruments? I need something more concrete.

SCENE Report | August 2021 33
I think the biggest one would be the encouragement of evaluators to promote the VOPE
voluntary organization of professional organizers, evaluators, whatever the local affiliates of
AEA are, things like GBEN or Washington Evaluators.
I like the idea of educating each other because often these programs are conceptualized as
someone is going to tell someone else things to do or not. You know, that transfer of knowledge
rather than sharing. Evaluators do not have any kind of professional learning communities or
networks and how important is that right now to build?
a. Engaging stakeholders. Evaluators suggested a need to educate funders and engage
new financial partners; bring on, compensate, and build capacity of local and new
partners; and clarify meanings of equity across a spectrum of understanding among
stakeholders.
We need to engage in an education campaign among funders to get them to view outcomes as a
continuum rather than a pass/fail. And that if they want their programs to be accurately
evaluated, they cannot demand 100% success for their programs to be funded again. They
should also know that, if they are receiving reports that say that a program is 100% successful,
then they are falsified. Evidence shows that this is not how programs work best. Also, the
programs that do work tend to be those that address the multi-pronged causes and effects of, say,
poverty or poor health
Earlier, you asked about how I conceive of equity? Well, I know how I conceive of equity, but if I
ask that of every client I had now, I'd probably get five different definitions. And so is that the
piece you? Do I need my client to come over to my concept of equity or do I need to interrogate
based on their concept of equity?
b. Shifting the evaluation field. More broadly, evaluators raised concerns about and
suggested shifts needed within the evaluation profession, including interrogation
and change of dominant evaluation concepts and ways of working and advancing
field-level conversations, standards, and accountability to back evaluators up who
are working to promote equity.
So if it's baked into the fields to a place where right now the norm in the de facto state is, as I
mentioned, operating in this culture of white supremacy, how do we really and concretely turn
that on its head to where the de facto way of doing business is operating in a culture of anti-
racism? And I don't feel like we have good learnings or tools to be able to, like, really start to
make inroads there.
Evaluators need more power. We talked about issues with professionalization, but if we all agree
to a set of standards and ethics in which we don’t fall prey to those power imbalances that
pressure us to do work that doesn’t address inequity, that is underfunded, along timelines that
are inadequate to do our work well, evaluators in general would have more power. And if we
networked effectively and provided support for those standards on a group level, we could
approach evaluation negotiations with more confidence. Perhaps. I know AEA has been working
on ethics and prioritizing this work which I fully support, and I think they have the right

SCENE Report | August 2021 34
emphases, but I think it needs more “teeth” or enforcement, and I’m not sure the best way to do
that.

SCENE Report | August 2021 35
Findings
1. Evaluator Roles and Training. Most SCENE respondents are AEA members (72; 88%),
have masters (31; 38%) or doctoral degrees (36; 44%), and have an average of fifteen years
of evaluation experience (range 2-40). A majority of respondents have degrees in education,
public health, or psychology; work in senior-level researcher/evaluator roles (60; 73%) with
more than half as external evaluators (48; 59%) and about one-third internal evaluators (29;
35%). Most reported learning their evaluation skills on-the-job (71; 88%) followed by
graduate-level university programs (61;75%) and professional development workshops (52;
64%).
2. Evaluator Demographics. Nearly all SCENE respondents identify as white (76; 93%) and
mostly women (65; 80%) with about one-quarter between 30-39 years old, one-third between
40-49 years old, and one-third between 50-64 years old. Few respondents indicated
belonging to racial/ethnic minority groups: Asian (4; 5%), Black/African American (3; 4%),
Hispanic/Latinx (2; 2%), and Multi-racial (1; 1%). Several identify as men (13; 16%) and as
gender non-binary (3;4%); about one-fifth (16; 20%) identify as LGBQ+ and few (5; 6%)
identify as having a disability.
3. Evaluator Areas of Practice. SCENE respondents practice evaluation in a variety of areas
including public health and/or health services (34; 43%), social services (30; 38%), pre-K-12
education (29; 36%), and higher education (20; 25%). Most widely used evaluation
approaches include participatory evaluation (58; 71%), utilization-focused evaluation (51;
62%) and program theory/theory-driven evaluation (47; 57%). Most rarely or sometimes use
AEA’s guiding principles, AEA’s cultural competence statement, and evaluation standards in
their work.
4. Evaluator Workplaces. SCENE respondents represent all states in New England, with
largest percentages from Massachusetts (35; 44%) and Vermont (16; 20%), and a majority
from community non-profits (34; 43%) and consulting or private sector work (31; 39%).
About half indicated they work for a small evaluation enterprise or are sole proprietors (43;
54%) and about one-quarter work for medium sized firms (19; 24%). Within their
workplaces, about half of respondents (43; 54%) indicated having diversity, equity and,
inclusion (DEI) training and a DEI office or specific DEI-focused staff members (43; 54%)
within their workplace.
5. Defining Equity. Evaluators define equity within three areas: within the evaluation process,
as a focus when evaluating a program or intervention, and as an intended use or influence of
the evaluation.
a. Equity within an evaluation process includes minimizing evaluation team bias,
including beneficiaries in evaluation processes, and constructing multiculturally,
contextually valid knowledge.
b. Equity as a focus when evaluating an intervention includes examining reach and
accessibility of interventions, examining differential experiences of intended
beneficiaries, analyzing differential outcomes, and interrogating root causes of
needs/problems addressed by interventions.

SCENE Report | August 2021 36
c. Equity as intended use of an evaluation includes using evaluation to challenge and
alter historical and contemporary imbalances in power and decision-making,
providing professional training for intervention staff to expand who does evaluations,
and building evidence and guidance from evaluations to further social equity and
justice.
6. Centering Equity. SCENE respondents, on average, reported most often centering equity
during evaluation reporting and dissemination (m2.88) and throughout the evaluation
(m2.73), and data analysis (m2.67). Evaluators least often center equity in the funding and
contracting phase of the evaluation (m2.22) followed by the evaluation team (m2.48).
7. Evaluation Teams. SCENE respondents reported most often self-examining cultural
identities (m2.98) and assumptions (m2.93). Respondents least often reported being led by
evaluators who identify as racial and/or ethnic minorities (m1.83) and having shared lived
experiences with the populations of focus (m2.05). Interviews revealed how some evaluators
try to enhance the diversity and expertise of evaluation teams by involving community
members and conducting equity reviews of evaluation plans and processes.
8. Throughout the Evaluation. Respondents most frequently highlight the strengths of the
intended beneficiaries of the evaluation (m3.09) and work to build trust with stakeholders
from minoritized or marginalized communities (m3.01). Respondents least frequently work
to mitigate power imbalances in the evaluation context (m2.45) and consult with cultural
brokers or translators to mediate between the cultures of stakeholders and the evaluation
team (m2.26).
9. Funding and Contracting. Most often, SCENE respondents have to work within purposes
and questions set by the funder/commissioner (m2.86) although they sometimes try to
prioritize RFPs focused on inclusion, diversity, and equity (m2.52). Interviewees described
centering equity in the funding and contracting phase in several ways: choose RFPs carefully,
make equity explicit in proposals, have up-front conversations, and budget accordingly.
10. Evaluation Questions and Criteria. SCENE respondents often examine the underlying
problem an intervention seeks to address (m3.08). Respondents least often answer questions
about how an intervention addresses systemic drivers of inequity (m2.29), include intended
beneficiaries in evaluation decision-making (m2.28), and in defining evaluative criteria
(m2.29). Nearly all respondents evaluate effectiveness (73; 92%) followed by most who
examine the relevance of the intervention to the needs, culture, interests, or circumstances of
the intended beneficiaries (65; 82%). More than half of respondents (46; 58%) examine
equity of the intervention’s opportunities, experiences, benefits, and/or results as an
evaluation criterion within their evaluation work.
11. Data Collection. SCENE respondents often design and/or modify data collection to
culturally and/or linguistically suit the participants (m3.04). Interviewees expanded on this
by sharing examples of how they try to center equity during data collection, such as: develop
instruments with participant consultation and piloting; use data collectors with shared

SCENE Report | August 2021 37
identity/lived experience; conduct member checking; and consider access, reduce burden,
and maximize benefit for participants in data collection.
12. Data Analysis. When analyzing data, respondents reported often disaggregating data by
demographic differences (m3.36), looking for potential negative consequences (m3.10), and
looking for differential participant access and experiences (m3.00). Least often respondents
look at shifts in power to intended beneficiaries and communities (m2.21) and re-distribution
of resources (m2.19). Interviewees highlighted how they sometimes disaggregate data and
also how they struggle with this due to small sample sizes for some minoritized groups and
how they struggle to go beyond disaggregation to analyze inequities more deeply and
critically.
13. Reporting and Dissemination. Survey data indicate that a majority (54; 68%) often or
almost always present reports in formats accessible to intended beneficiaries and about half
(44; 55%) make evaluation results, in some form, publicly accessible. Some interviewees
discussed how they embed processes for shared sensemaking and disseminate in multiple,
accessible formats and across stakeholder groups. Other interviewees struggle with
collaborative sensemaking and accessible dissemination.
14. Helpful Factors. Interviewees discussed how factors within their teams, organizations and
wider society all can help to support equity-focused evaluation processes.
a. Within teams. Evaluators emphasized the importance of having members with diverse
cultural and linguistic expertise, inviting in community and new evaluator voices, and
fostering critical, self-reflective, and learning cultures.
b. Within evaluation organizations. Organizational helpful factors include having
missions and values aligned with equity, leadership support, internal diversity, equity,
and inclusion work, and dedicated staff time.
c. Within 2020 sociopolitical context. More broadly, evaluators reflected on how public
attention on issues of racial injustice in 2020 provided a context to further advocate
for equity-focused evaluation.
15. Hindering Factors. Interviewees identified how factors related to evaluation contracts,
evaluators’ racial identities, evaluation stakeholders, and the evaluation profession can all
hinder centering equity in evaluation.
a. Evaluation contracts. Within and across contracts, evaluators find that quick turn-
around time, limited budgets, restricted scopes of work, the need to secure and keep
contracts, stress and burnout, and constrained evaluator autonomy all work against
equity-focused evaluation.
b. Evaluators’ racial identities. Some interviewees talked about their own and
colleagues’ white racial identities as constraints that minimize seeing and addressing
inequity. Several interviewees of color shared examples of racial ignorance and
discrimination, including being treated as a token representative and being expected
to do more as evaluators of color which created inequities for them as evaluators.
c. Evaluation stakeholders. In working with stakeholders, evaluators find differences in
what equity means, constraints on challenging inequities built-into programs, and the
lack of use of evaluation findings all to hinder equity.

SCENE Report | August 2021 38
d. Evaluation profession. Interviewees pointed to ways evaluation itself can constrain
equity in its Professional culture and norms that privilege scientific knowledge and
objectivity, assumed evaluator role as experts, and a sense of isolation, competition,
and little sharing between evaluators and evaluation organizations.
16. Strengthening Capacity. Suggestions for strengthening capacity for centering equity in
evaluation include enhancing evaluator skills, engaging evaluation stakeholders, and
changing how the broader evaluation field and profession to advance equity.
a. Enhancing and applying evaluator skills and responsibilities. Interviewees’
suggestions began with ways evaluators can build their own capacity by participating
in trainings about antiracism, racial justice, and intersectionality and how these lenses
can be applied to methods; documenting, self-assessing and seeking critical feedback
on evaluation processes; learning how to work together across different cultural
identities and ways of knowing; and critically applying evaluation skills, such as
situational analysis, to points to influence inequities within the evaluation context.
Finally, they urged evaluators to make the time and space for addressing inequity.
b. Build professional learning among evaluators. Beyond individual evaluators and
evaluation teams, interviewees pointed to the need to build professional learning by
sharing practical guidance, concrete examples, and lessons from practice and creating
professional learning communities.
c. Engaging stakeholders. Evaluators suggested a need to educate funders and engage
new financial partners; bring on, compensate, and build capacity of local and new
partners; and clarify meanings of equity across a spectrum of understanding among
stakeholders.
d. Shifting the evaluation field. More broadly, evaluators raised concerns about and
suggested shifts needed within the evaluation profession, including interrogation and
change of dominant evaluation concepts and ways of working and advancing field-
level conversations, standards, and accountability to back evaluators up who are
working to promote equity.

SCENE Report | August 2021 39
Recommendations
1. Connect. Identify and build connections between evaluators and those in evaluation-adjacent
roles within New England. Expand beyond those affiliated with the American Evaluation
Association and local affiliates. Build inclusion and diversity by connecting with those from
underrepresented groups within the field and region.
2. Learn and Share. Support evaluation communities of practice and build in opportunities for
reflective practice and collective learning, within and across evaluation teams and
organizations. Raise awareness of existing guidance from the American Evaluation
Association, Equitable Evaluation Initiative, and other national and regional efforts to
promote culturally responsive and equitable evaluation. Translate frameworks into practical
guidance with case examples. Promote sharing between evaluators and across evaluation
teams and organizations of promising practices and lessons learned through different efforts.
3. Deepen Understanding and Practice. Recognize that equity takes on different meanings
and that evaluators and stakeholders vary along a spectrum of understanding. Push evaluators
and support evaluators pushing stakeholders to deepen their understanding and implications
for changing practice. One area, in particular, is to emphasize how to mitigate power
imbalances between intended beneficiaries, evaluators, and funders within decision-making
about the evaluation and broader evaluation context.
4. Build Support. Given evaluators’ perceptions of constraints within contracts, funders, and
programmatic circumstances, identify ways that funders, clients, and program leaders can
support equity-focused evaluation work. This may mean hosting dialogues between funders,
evaluation practitioners, and intended beneficiary communities to critique and reimagine
evaluation practice. Creating templates, models, and examples for contracts and client-
evaluator relationships also could help evaluators push for the time and resources needed.
5. Investigate. Within New England, some interviewees pointed to the predominance of white
supremacy culture within organizations and the social sector. They also pointed to the default
assumptions held by some evaluation team members and stakeholders about the lack of
racial, ethnic, and cultural diversity and, therefore, limited need for equity-focused
evaluation. Further research should examine evaluators’ racial identities in relation to the
predominantly white New England region and through the experiences of evaluators of color.

SCENE Report | August 2021 40
References
Note: All references consulted throughout the study. Not all are cited in this report.
Alkin, M. C., Vo, A. T., & Christie, C. A. (2012). The evaluator’s role in valuing: Who and with
whom. In G. Julnes (Ed.), Promoting valuation in the public interest: Informing policies
for judging value in evaluation. New Directions for Evaluation, 2012(133), 29–41.
https://doi.org/10.1002/ev.20004
American Evaluation Association. (2011). American Evaluation Association statement on
cultural competence in evaluation. Retrieved from eval.org/ccstatement.
American Evaluation Association. (2018). AEA-American Evaluation Association: Guiding
Principles for Evaluators. [online] https://www.eval.org/About/Guiding-Principles
Andrews, K., Parekh, J., & Peckoo, S. (2019). How to embed a racial and ethnic equity
perspective in research: Practical guidance for the research process.
https://www.childtrends.org/wp-
content/uploads/2019/09/RacialEthnicEquityPerspective_ChildTrends_October2019.pdf
Ball, R. A. (1977). Equitable evaluation through investigative sociology. Sociological Focus,
10(1), 1-14.
Bamberger, M., Raftree, L., & Olazabal, V. (2016). The role of new information and
communication technologies in equity-focused evaluation: Opportunities and challenges.
Evaluation, 22(2), 228-244.
Bauer, T., Fitzsimmons, K., & Sposato, D. (2019). The role of the practitioner in the next
generation of evidence. Project Evident.
https://www.projectevident.org/updates/2019/10/23/the-role-of-the-practitioner-in-the-
next-generation-of-evidence
Bazeley, P. (2012). Integrative Analysis Strategies for Mixed Data Sources. American
Behavioral Scientist, 56(6), 814–828. https://doi.org/10.1177/0002764211426330
Boyce, A. S. (2017). Lessons learned using a values-engaged approach to attend to culture,
diversity, and equity in a STEM program evaluation. Evaluation and Program Planning,
64, 33–43. https://doi.org/10.1016/j.evalprogplan.2017.05.018
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in
Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Buteau, E., Glickman, J., Loh, C., Coffman, J., & Beer, T. (2016). Benchmarking foundation
evaluation practices. Center for Effective Philanthropy & Center for Evaluation
Innovation. https://www.evaluationinnovation.org/wp-
content/uploads/2020/01/FinalBenchmarkingReport2020.pdf

SCENE Report | August 2021 41
Brown, M. A., & Di Lallo, S. (2020). Talking circles: A culturally responsive evaluation
practice. American Journal of Evaluation, 41(3), 367–383.
https://doi.org/10.1177/1098214019899164
Caldwell, L. D., & Bledsoe, K. L. (2019). Can social justice live in a house of structural racism?
A question for the field of evaluation. American Journal of Evaluation, 40(1), 6–18.
https://doi.org/10.1177/1098214018815772
Campbell, J. L., Quincy, C., Osserman, J., & Pedersen, O. K. (2013). Coding In-depth
Semistructured Interviews: Problems of Unitization and Intercoder Reliability and
Agreement. Sociological Methods & Research, 42(3), 294–320.
https://doi.org/10.1177/0049124113500475
Carden, F. (2013). Asking questions in the solution space: Methodological issues in evaluating
equity. Evaluation and Program Planning, 36(1), 213–217.
https://doi.org/10.1016/j.evalprogplan.2012.03.010
Center for Evaluation Innovation. (2017). Equitable Evaluation Framing Paper | Equitable
Evaluation Framing Paper. July. www.equitableeval.org
Change Elemental. (n.d.). Deep Equity. Change Elemental. Retrieved May 1, 2021, from
https://changeelemental.org/offerings/deep-equity/
Chelimsky, E. (2012). Valuing, evaluation methods, and the politicization of the evaluation
process. In G. Julnes (Ed.), Promoting valuation in the public interest: Informing policies
for judging value in evaluation. New Directions for Evaluation, 2012(133), 77–83.
https://doi.org/10.1002/ev.20008
Coleman, S., et al. (2017). Engaging community members with lived experience. SCALE 1.0
Synthesis Reports. Cambridge, Massachusetts: Institute for Healthcare Improvement.
Corbin, J. M., & Strauss, A. L. (2015). Basics of qualitative research: Techniques and
procedures for developing grounded theory (Fourth edition). SAGE.
Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research
(Third Edition). SAGE.
Creswell, J. W., & Poth, C. N. (2018). Qualitative inquiry & research design: Choosing among
five approaches (Fourth Edition). Sage.
Curren, R., Liu, N., Marsh, D., & Rose, K. (2016). Equitable development as a tool to advance
racial equity. The Local & Regional Government Alliance on Race and Equity. Retrieved
from http://www.racialequityalliance.org

SCENE Report | August 2021 42
Datta, L. (2011). Politics and evaluation: More than methodology. American Journal of
Evaluation, 32(2), 273–294. https://doi.org/10.1177/1098214011400060
Davidson, E. J. (2014). Evaluative Reasoning (No. 4; Impact Evaluation). UNICEF.
https://www.unicef-
irc.org/publications/pdf/brief%204%20evaluative%20reasoning_eng.pdf
de Vries, B., LeBlanc, A. J., Frost, D. M., Alston-Stepnitz, E., Stephenson, R., & Woodyatt, C.
R. (2017). The relationship timeline: A method for the study of shared lived experiences
in relational contexts. Advances in Life Course Research, 32, 55-64.
Dean-Coffey, J. (2018). What’s Race Got to Do With It? Equity and Philanthropic Evaluation
Practice. American Journal of Evaluation, 39(4), 527–542.
https://doi.org/10.1177/1098214018778533
Esterle, J., Kopell, M., & Strand, P. J. (2020, November 22). It’s All About Relationships:
Systems-Based Changemaking. Philanthropy for Active Civic Engagement (PACE) -
Medium. https://medium.com/office-of-citizen/its-all-about-relationships-systems-based-
changemaking-470207584bf4
Farrow, F., & Morrison, S. (2019). Placing Equity Concerns at the Center of Knowledge
Development. In Center for the Study of Social Policy.
Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013). Achieving Integration in Mixed Methods
Designs—Principles and Practices. Health Services Research, 48(6 Pt 2), 2134–2156.
https://doi.org/10.1111/1475-6773.12117
Forestieri, M. (2020). Equity Implications in Evaluating Development Aid: The Italian Case.
Journal of MultiDisciplinary Evaluation, 16(34), 65-90.
Fournier, D. M. (1995). Establishing evaluative conclusions: A distinction between general and
working logic. In D. M. Fournier (Ed.), Reasoning in Evaluation: Inferential Links and
Leaps. New Directions for Evaluation, 1995(68), 15–32. https://doi.org/10.1002/ev.1017
Garibay, C., & Teasdale, R. M. (2019). Equity and evaluation in informal STEM education. In
A. C. Fu, A. Kannan, & R. J. Shavelson (Eds.), Evaluation in Informal Science,
Technology, Engineering, and Mathematics Education. New Directions for Evaluation,
2019(161), 87– 106. https://doi.org/10.1002/ev.20352
Gates, E. F. (2018). Toward valuing with critical systems heuristics. American Journal of
Evaluation, 39(2), 201–220. https://doi.org/10.1177/1098214017703703
Goff, S. (2020). Visionary evaluation: Approaching Aboriginal ontological equity in water
management evaluation. Evaluation and Program Planning, 79.
https://doi.org/10.1016/j.evalprogplan.2019.101776

SCENE Report | August 2021 43
Goodchild, M. (2021). Relational systems thinking: That’s how change is going to come, from
our Earth mother. Journal of Awareness-Based Systems Change, 1(1), 75–103.
https://doi.org/10.47061/jabsc.v1i1.577
Greene, J. (2006). Evaluation, democracy, and social change. In I. F. Shaw, J. C. Greene, & M.
M. Mark (Eds.), The SAGE Handbook of Evaluation (pp. 118–140). SAGE Publications
Ltd.
Greene, J. C. (2007). Mixed methods in social inquiry (1st ed). Jossey-Bass.
Gullickson, A. M., & Hannum, K. M. (2019). Making values explicit in evaluation practice.
Evaluation Journal of Australia, 9(4), 162–178.
https://doi.org/10.1177/1035719X19893892
Hall, J.N. (2013). Pragmatism, Evidence, and Mixed Methods Evaluation. In D.M. Mertens & S.
Hesse-Biber (Eds.), Mixed Methods and Credibility of Evidence in Evaluation. New
Directions for Evaluation, 2013(138), 15-26. https://doi.org/10.1002/ev.20054
Hall, J. N. (2020). The Other Side of inequality: Using standpoint theories to examine the
privilege of the evaluation profession and individual evaluators. American Journal of
Evaluation, 41(1), 20-33.
Hargraves, M., Buckley, J., Urban, J. B., Linver, M. R., Chauveron, L. M., Samtani, S.,
Archibald, T., & Moorman, L. (2021). Resonance, stickiness, and the value propositions
of Evaluation Capacity Building: Key takeaways and future directions. In J. B. Urban, T.
Archibald, M. Hargraves, J. Buckley, C. Hebbard, M. R. Linver, W. M. Trochim (Eds.),
Relational Systems Evaluation. New Directions for Evaluation, 2021(169), 97– 116.
https://doi.org/10.1002/ev.20442
Hawn Nelson, A., Jenkins, D., Zanti, S., Katz, M., Berkowitz, E., Burnett, T. C., & Culhane, D.
(2020). A toolkit for centering racial equity throughout data integration. Actionable
Intelligence for Social Policy. University of Pennsylvania.
https://www.aisp.upenn.edu/centering-equity/
Heider, C. (2017). Rethinking evaluation: Have we had enough of R/E/E/I/S? Independent
Evaluation Group. World Bank Publications. Retrieved from: www.worldbank.org.
Hood, S. (2001). Nobody knows my name: In praise of African American evaluators who eere
responsive. In J. C. Greene & T. A. Amba (Eds.), Responsive Evaluation. New Directions
for Evaluation, 2001(92), 31–44. https://doi.org/10.1002
Hood, S., Hopson, R. K., & Kirkhart, K. E. (2015). Culturally Responsive Evaluation: Theory,
Practice, and Future Implications. In K. E. Newcomer, H. P. Hatry, & J. S. Wholey
(Eds.), Handbook of Practical Program Evaluation (4th ed., pp. 281–317). Jossey-Bass.

SCENE Report | August 2021 44
Hopson, R. K. (2003). Overview of multicultural and culturally competent program evaluation:
Issues, challenges and opportunities. Woodland Hills, CA: The California Endowment.
Hopson, R. K., Kirkhart, K. E., & Bledsoe, K. (2011). Decolonizing Evaluation in a Developing
World: Implications and Cautions for Equity-Focused Evaluation. In M. Segone (Ed.),
Evaluation for equitable development results (pp. 59–82).
Hwalek, M. A., & Straub, V. L. (2018). The small sellers of program evaluation services in the
United States. In S. B. Nielsen, S. Lemire, & C. A. Christie (Eds.), The Evaluation
Marketplace: Exploring the Evaluation Industry. New Directions for Evaluation,
2018(160), 125–143. https://doi.org/10.1002/ev.20340
Julnes, G. (2012). Managing valuation. In G. Julnes (Ed.), Promoting valuation in the public
interest: Informing policies for judging value in evaluation. New Directions for
Evaluation, 2012(133), 3–15. https://doi.org/10.1002/ev.20002
Kinarsky, A. R. (2018). The evaluation landscape: U.S. Foundation spending on evaluation. In S.
B. Nielsen, S. Lemire, & C. A. Christie (Eds.), The Evaluation Marketplace: Exploring
the Evaluation Industry. New Directions for Evaluation, 2018(160), 81–96.
https://doi.org/ 10.1002/ev.20342
Lemire, S., Fierro, L. A., Kinarsky, A. R., Fujita-Conrads, E., & Christie, C. A. (2018). The U.S.
federal evaluation market. In S. B. Nielsen, S. Lemire, & C. A. Christie (Eds.), The
Evaluation Marketplace: Exploring the Evaluation Industry. New Directions for
Evaluation, 2018(160), 63–80. https://doi.org/10.1002/ev.20343
Letiecq, B. L., & Bailey, S. J. (2004). Evaluating from the outside: Conducting cross-cultural
evaluation research on an American Indian reservation. Evaluation Review, 28(4), 342-
357.
Lo, F. & Espiritu, R. (2021). Evaluation is so White: Systemic wrongs reinforced by common
practices and how to start righting them. Funder & Evaluator Affinity Network.
Lucas, K., van Wee, B., & Maat, K. (2016). A method to evaluate equitable accessibility:
Combining ethical theories and accessibility-based approaches. Transportation, 43(3),
473–490. https://doi.org/10.1007/s11116-015-9585-2
Mathison, S. (2018). Does evaluation contribute to the public good? Evaluation, 24(1), 113–119.
https://doi.org/10.1177/1356389017749278
McArthur, J. (2016). Assessment for social justice: The role of assessment in achieving social
justice. Assessment & Evaluation in Higher Education, 41(7), 967–981.
https://doi.org/10.1080/02602938.2015.1053429
Miles, M. B., Huberman, A. M, & Saldaa, J. (2014). Qualitative data analysis: a methods
sourcebook (Third Edition). Sage.

SCENE Report | August 2021 45
Morris, M., & Jacobs, L. R. (2000). You Got a Problem With That?: Exploring Evaluators’
Disagreements about Ethics. Evaluation Review, 24(4), 384–406.
https://doi.org/10.1177/0193841X0002400403
Neal, J. W., Neal, Z. P., VanDyke, E., & Kornbluh, M. (2015). Expediting the Analysis of
Qualitative Data in Evaluation: A Procedure for the Rapid Identification of Themes From
Audio Recordings (RITA). American Journal of Evaluation, 36(1), 118–132.
https://doi.org/10.1177/1098214014536601
Nielsen, S. B., Lemire, S., & Christie, C. A. (2018). The evaluation marketplace and its industry.
In S. B. Nielsen, S. Lemire, & C. A. Christie (Eds.), The Evaluation Marketplace:
Exploring the Evaluation Industry. New Directions for Evaluation, 2018(160), 13–28.
https://doi.org/10.1002/ev.20344
Nowell, L. S., Norris, J. M., White, D. E., & Moules, N. J. (2017). Thematic Analysis: Striving
to Meet the Trustworthiness Criteria. International Journal of Qualitative Methods,
16(1), 1609406917733847. https://doi.org/10.1177/1609406917733847
Olson, K. (2010). An examination of questionnaire evaluation by expert reviewers. Field
Methods, 22(4), 295-318. https://dx.doi.org/0.1177/1525822X10379795
Ozeki, S., Coryn, C. L. S., & Schröter, D. C. (2019). Evaluation logic in practice: Findings from
two empirical investigations of American Evaluation Association members. Evaluation
and Program Planning, 76. https://doi.org/10.1016/j.evalprogplan.2019.101681
Patton, M. Q. (2015). Qualitative research & evaluation methods: Integrating theory and
practice (Fourth edition). Sage.
Peterson, C. H., Peterson, N. A., & Powell, K. G. (2017). Cognitive interviewing for item
development: Validity evidence based on content and response processes. Measurement
and Evaluation in Counseling and Development, 50(4), 217-223.
https://dx.doi.org/10.1080/07481756.2017.1339564
Public Policy Associates Inc. (2015). Considerations for Conducting Evaluation Using a
Culturally Responsive and Racial Equity Lens. http://publicpolicy.com/wp-
content/uploads/2017/04/PPA-Culturally-Responsive-Lens.pdf
Schwandt, T. A. & Gates, E. F. (2016). What Can Evaluation Do? An Agenda for Evaluation in
Service of an Equitable Society. In R. Picciotto & S. I. Donaldson (Eds.), Evaluation for
an Equitable Society (pp. 67–81). Information Age Publishing, Inc.
Smagorinsky, P. (2008). The method section as conceptual epicenter in constructing social
science research reports. Written Communication, 25(3), 389-411.

SCENE Report | August 2021 46
SPEC Associates. (2018). Survey of small sellers of evaluation services: print version
[Unpublished survey instrument].
Teasdale, R. M. (2021). Evaluative Criteria: An Integrated Model of Domains and Sources.
American Journal of Evaluation.
https://doi.org/https://doi.org/10.1177/1098214020955226
Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The Program
Evaluation Standards: A Guide for Evaluators and Evaluation Users. Sage.
Westaby, K. A., Williams, T. M., Robinson, N. N., & Connors, E. (2019). Being responsive: The
first assessment of Culturally Responsive Evaluation in Wisconsin: Findings from the
2017 survey. Milwaukee, WI: ¡Milwaukee Evaluation!, Inc.
Wothke, W. (1993). Nonpositive definite matrices in structural modeling. In K. A. Bollen & J. S.
Long (Eds.), Testing Structural Equation Models (pp. 256–293). SAGE.