Content uploaded by Rodney Hopson
Author content
All content in this area was uploaded by Rodney Hopson on Jul 11, 2020
Content may be subject to copyright.
CHAPTER TWELVE
CULTURALLY RESPONSIVE EVALUATION
Theory, Practice, and Future Implications∗,†
Stafford Hood, Rodney K. Hopson, Karen E. Kirkhart
In the last two decades, the evaluation literature reects increasing atten-
tion to culture and cultural contexts in the eld. A lion’s share of this lit-
erature has focused on culturally responsive evaluation (CRE) concepts and
frameworks.1Much less literature considers the practice, practical application,
or ways in which those in the eld maximize the use of such frameworks. As this
chapter will reveal, most of the current CRE literature discusses either theory
or practice; very few, if any, provide discussions of both theoretical and practical
applications of CRE.
As the practice of evaluation by non-prots, consultants, academics, and
the general public grows, the need to use CRE in evaluation practice has
increased because evaluators work in diverse cultural, contextual, and complex
communities in the United States and in many other parts of the world. In this
fourth edition of the Handbook, this chapter provides a core resource on the
history, theory, and application of CRE. This opportunity to bring CRE theory
and practice to a wider audience is set within an increasing global demand
for monitoring and evaluation of public programs and the requirements by
∗Acknowledgements: The authors thank Kathy Newcomer and two anonymous reviewers. Ad-
ditionally, authors credit Elizabeth Kahl and Kelly D. Lane for their assistance with the tech-
nical and graphic design support of Figure 12.2.
†This chapter reects a long-term collaboration among these authors, each of whom made
unique contributions to the conversation; therefore, the order of authorship is purposely
alphabetical.
281
Handbook of Practical Program Evaluation, Fourth Edition
By Kathryn E. Newcomer, Harry P. Hatry and Joseph S. Wholey
Copyright © 2015 by Kathryn E. Newcomer, Harry P. Hatryand Joseph S. Wholey
282 Handbook of Practical Program Evaluation
governments and international organizations to use evaluation, especially in
settings and communities that have traditionally been underserved, underrep-
resented, or marginalized.
The purpose of the chapter is threefold: to provide a historical record
of the development of CRE, to describe the theory that guides CRE practice,
and to demonstrate how practice applications inform and contribute to CRE
theory. The chapter begins with a summary and history of CRE, from its
inception in the evaluation literature to its current moment and use in train-
ing, professional development workshops, publications, and practice.
The second part of the chapter presents a framework used to distinguish
application of CRE in several dimensions of evaluation practice. Specically,
this section describes how core theoretical components of CRE provide a
framework to guide practice from the outset of an evaluation to its conclusion.
By integrating culturally responsive practices and applications throughout the
evaluation cycle, practitioners gain better practical knowledge in ways to use
CRE and how to provide more robust CRE learning in diverse cultural settings.
The third part of the chapter illustrates what CRE theory looks like in prac-
tice through the illustration of three practice applications published in the last
decade. These practice applications describe an increasingly complex world
of evaluation and show how the details of implementing CRE also build CRE
theory on the ground. This third section depicts distinct ways to think about
evaluation practice through a CRE theoretical framework and suggests that the
practical application of CRE in national and international settings is increas-
ingly timely and useful.
The fourth and nal section of the paper highlights ways in which CRE
challenges the evaluation profession to revisit basic premises such as validity,
rigor and responsibility. As such, the nal section provides implications and
considerations for future culturally responsive evaluators who intend to extend
practice even further.
Ultimately, the chapter lays out an afrmative statement on the bound-
aries of CRE in practical evaluation contexts and offers ways in which culturally
responsive evaluators in multiple settings can apply CRE practically and use-
fully. As demonstrated in the history of CRE, this chapter intends to serve both
as a reference point and a benchmark for further discussion and development
of CRE for years to come.
What Is CRE?
CRE is a holistic framework for centering evaluation in culture (Frierson,
Hood, Hughes, and Thomas, 2010). It rejects culture-free evaluation and
Culturally Responsive Evaluation 283
recognizes that culturally dened values and beliefs lie at the heart of any eval-
uative effort. Evaluation must be designed and carried out in a way that is cul-
turally responsive to these values and beliefs, many of which may be context-
specic. CRE advocates for the inclusion of culture and cultural context in
both evaluation theory and practice (Hood, 2014). Hopson (2009) expressed
it as follows:
CRE is a theoretical, conceptual and inherently political position that
includes the centrality of and [attunement] to culture in the theory and
practice of evaluation. That is, CRE recognizes that demographic,
sociopolitical, and contextual dimensions, locations, perspectives, and
characteristics of culture matter fundamentally in evaluation. (p. 431)
In examining the component parts of CRE, culture is understood as “a
cumulative body of learned and shared behavior, values, customs and beliefs
common to a particular group or society” (Frierson, Hood, and Hughes, 2002,
p. 63). Responsive “fundamentally means to attend substantively and politically
to issues of culture and race in evaluation practice” (Hood, 2001, p. 32). Evalu-
ation refers to the determination of merit, worth or value of a program, project
or other evaluand (Scriven, 1991). Thus, “an evaluation is culturally responsive
if it fully takes into account the culture of the program that is being evalu-
ated” (Frierson, Hood, and Hughes, 2002, p. 63) as well as “the needs and cul-
tural parameters of those who are being served relative to the implementation
of a program and its outcomes” (Hood and Hall, 2004, cited in Hood, 2014,
p. 114).
CRE gives particular attention to groups that have been historically
marginalized, seeking to bring balance and equity into the evaluation process.
Relevant theoretical roots include indigenous epistemologies, social advocacy
theories, and critical race theory (Hopson, 2009). CRE marries theories of cul-
turally responsive assessment and responsive evaluation to bring program eval-
uation into alignment with the lived experiences of stakeholders of color. As
the following section recounts, the historical foundations of CRE marry the-
ories of culturally responsive assessment and pedagogy with responsive evalu-
ation. As reected later in the chapter, the historical foundation of CRE sets
the record straight concerning the pioneers and legacy of CRE.
Pioneers in the Foundations of CRE
The historical foundation of CRE is largely framed in scholarship by Stafford
Hood, as well as the signicant contributions of others in the evaluation eld
FIGURE 12.1. FOUNDATIONAL INFLUENCES OF CRE.
Messick, 1994
1935 20152005199519851975196519551945
Lee, 1990
Gordon, 1995
Ladson-Billings, 1995a-b
Kirkhart, 1995
Hood, 1998a-b
Johnson, 1998
Messick, 1989
Hood, 2000
Hood, 2001
Stake, 1973
Madison, 1992
Jackson, 1938
Jackson, 1939
Jackson, 1940a-b
Hopson & Hood, 2005
Frierson, Hood & Hughes, 2002
Hood, 2009
Hopson, 2009
Frierson, Hood, Hughes,
& Thomas, 2010
Askew, Beverly, &
Jay, 2012
Frazier-Anderson,
Hood, & Hopson,
2012
Stake, 1987
284
Culturally Responsive Evaluation 285
in the last ten to fteen years. This section, as reected in Figure 12.1, summa-
rizes Hood’s inuences in culturally relevant pedagogy and culturally respon-
sive assessment, responsive evaluation, validity, and social justice as one initial
reference point.
The early roots of CRE began in education, specically in the work of Carol
Lee (1990) and Gloria Ladson-Billings (1995a-b) on culturally responsive ped-
agogy in conjunction with the work of Edmund Gordon (1995) and Sylvia
Johnson (1998) in educational assessment. Hood (1998a) extended this think-
ing from culturally responsive pedagogy to culturally responsive assessment,
and subsequently to culturally responsive evaluation. Interestingly, the bridge
from culturally responsive assessment to culturally responsive evaluation was
built within validity theory. Kirkhart’s (1995) conceptualization and articula-
tion of the construct multicultural validity in evaluation contributed signi-
cantly to Hood (1998a), extending his logic of cultural responsiveness from
pedagogy and educational assessment to evaluation. Hood’s initial thinking
on culturally responsive assessment had been inuenced by Messick’s (1989)
denition of validity and particularly Messick’s articulation of a consequential
basis of validity which emphasized “the salient role of both positive and nega-
tive consequences” in validation (Messick, 1994,p.13).WhenKirkhart(1995)
introduced the concept of multicultural validity, also building upon Messick’s
attention to consequences, Hood resonated with Kirkhart’s emphasis on social
justice and saw in it a bridge from culturally responsive assessment to cultur-
ally responsive evaluation. Hood’s (2000) commentary on “deliberative demo-
cratic evaluation” reects this transition, which was also supported by the work
of authors such as Madison (1992), who challenged evaluation to address race
and culture.
Hood rst used the term “culturally responsive evaluation” in his pre-
sentation at a May 1998 festschrift honoring Robert Stake and Stake’s initial
work in responsive evaluation (Stake, 1973/1987).2Hood’s (1998b) descrip-
tion of “responsive evaluation Amistad style” attached responsiveness explic-
itly to culture and cultural differences, emphasizing the importance of shared
lived experience between the evaluators/observers and persons intended to
be served and observed. Examples included culturally specic use of language
and non-verbal expression.3
Development of CRE continued through dialogue in a number of ways,
both through Hood’s work as co-founder of Arizona State University’s national
conference on Relevance of Assessment and Culture in Evaluation (RACE) in
2000 and his membership and leadership in two American Evaluation Associ-
ation (AEA) committees: the Diversity Committee and the Advisory Oversight
Committee of Building Diversity Initiative (BDI). As Hood (2014) reports:
286 Handbook of Practical Program Evaluation
The interface between the RACE conference and AEA Building Diversity
Initiative provided an “expanded space” for the conversations among
researchers, scholars, and practitioners about the role of culture and
cultural context in evaluation and assessment as well as the need to increase
the number of trained evaluators and assessment specialists of color.
(p. 113)
In the 2001 New Directions for Evaluation volume on Responsive Evaluation
(Greene and Abma, 2001), Hood explicitly infused Stake’s model (1973/1987)
with concerns for evaluation as a means of promoting equity and recognition of
scholars of color as evaluation forefathers.4Equity and equality are focal issues
in Hood (2001), bringing together concerns for racial equality with those of
responsive evaluation. Hood demonstrates in his Nobody Knows My Name (2001)
publication how four premises of responsive evaluation are visible in the work
of early African American evaluators, whose contributions have not been duly
recognized:
rIssues are the “advanced organizers” for evaluation study instead of objec-
tives or hypotheses.
rIssues are the structure for continuing discussions and data gathering
plan[s].
rHuman observers are best instruments.
rEvaluators should get their information in sufcient amounts from
numerous independent and credible sources so that it effectively rep-
resents the perceived status of the program, however, complex. (Stake,
1973/1987/1987, cited in Hood, 2001,p.38)
The work of Reid E. Jackson (1935,1936,1939.1940a-b) in the 1930’s and
1940’s would provide historical insight and clarity in the articulation of CRE. It
is important to note that not only did Jackson receive his Ph.D. in 1938 but also
that it was completed at Ohio State University, where Ralph Tyler marked the
Eight Year Study as an historic marker in the evaluation history timeline. Hood
(2001) had identied Reid E. Jackson as one of the earlier African American
pioneers in educational evaluation. It was Hopson and Hood (2005)who
connected the signicance of Jackson’s work as providing “one of the earliest
glimpses of culturally responsive evaluative judgments” (p. 96). Jackson’s
evaluations of segregated schooling for African Americans in Kentucky
(Jackson, 1935), Florida (Jackson, 1936), and particularly Alabama (Jackson,
1939,1940a-b), provide concrete examples of an evaluator designing and
implementing evaluations where culture was a central consideration.
Culturally Responsive Evaluation 287
Other signicant publications further rened the theoretical and ideolog-
ical roots of CRE. Table 12.1 summarizes the evolution of key points and prin-
ciples of CRE as articulated by the authors of core publications. It is a cumula-
tive list in the sense that characteristics introduced in earlier literature are not
repeated. For example, the notion of shared lived experience is a foundational
theme woven through all of the core literature on CRE; however, it appears in
Table 12.1 only where it was rst introduced in relation to CRE (Hood, 1998b).
From CRE Theory to CRE Practice
The theoretical parameters of CRE were translated into practice guidelines
by Frierson, Hood, and Hughes (2002) and Frierson, Hood, Hughes, and
Thomas (2010). These have been developed through workshop interactions
(for example, Hopson, 2013; Hopson and Casillas, 2014; Hopson and Kirkhart,
2012; Kirkhart and Hopson, 2010) and practice applications (for example, Jay,
Eatmon, and Frierson, 2005; King, Nielsen, and Colby, 2004; LaFrance and
Nichols, 2010; Manswell Butty, Reid, and LaPoint, 2004;Thomas,2004). While
CRE does not consist of a unique series of steps set apart from other evaluation
approaches, the details and distinction of CRE lie in how the stages of the eval-
uation are carried out. CRE is conducted in ways that create accurate, valid,
and culturally-grounded understanding of the evaluand. The nine procedural
stages outlined by Frierson, Hood, and Hughes (2002) and Frierson, Hood,
Hughes, and Thomas (2010) illustrate the practice of CRE. See Figure 12.2,
which depicts a guiding visual for incorporating the steps in the practice of
CRE.5
Preparing for the Evaluation
Evaluators must work hard in preparing to enter a community, neighborhood,
or organization; they have a responsibility to educate themselves. CRE requires
particular attention to the context in which an evaluation will be conducted.
This includes the history of the location, the program, and the people. What
are the stories of this community and its people, and who is telling them? CRE
evaluators are observant regarding communication and relational styles. How
does one respectfully enter this community? What dimensions of diversity are
most salient within this community and how is power distributed, both for-
mally and informally? What relationships are valued or privileged and what
relationships are discouraged or forbidden?
TABLE 12.1. KEY CHARACTERISTICS OF CULTURALLY
RESPONSIVE EVALUATION (CRE) FROM CORE LITERATURE.
Citation Core Characteristics of CRE
Hood, S.
(1998)
Importance of shared lived experience between observers and
observed
Emphasis on understanding a program as it functions in the
context of culturally diverse groups
Need for a greater number of trained African American
evaluators
Both language and cultural nuance may require
interpretation
Importance of bridging understanding between cultures
Hood, S.
(2001)
Recognizes the early work of African American scholars
Explicit attention to culture and race, “substantively and
politically”
Increased participation of African Americans and other
evaluators of color as a pragmatic necessity and moral
obligation
Broadens evidence to include qualitative as well as
quantitative data
Understanding as “vicarious experience”
Inclusion of multiple stakeholder perspectives
Social responsibility to address unequal opportunities and
resources
Frierson, H. T.,
Hood, S.,
and Hughes,
G. B. (2002)
Considers culture of the project or program as well as culture of
participants
Rejects “culture free” evaluation
Proposes evaluation strategies consonant with cultural
context
Racial/ethnic congruence of evaluators with setting does not
equate to cultural congruence or competence.
Addresses the epistemology of what will be accepted as
evidence
Evaluators must recognize their own cultural preferences
Represents all voices through a democratic process
Hood, S.
(2009)
Attention to power differentials among people and systems
Importance of historical and cultural antecedents
Social justice agenda
Evaluator understands own cultural values
Requires long-term investment of time to acquire necessary
skills and shared lived experiences
Use of a cultural liaison/language translator
Importance of how one enters relationships
Explicitly links CRE to validity
(Continued)
288
TABLE 12.1. KEY CHARACTERISTICS OF CULTURALLY
RESPONSIVE EVALUATION (CRE) FROM CORE LITERATURE.
(Continued)
Citation Core Characteristics of CRE
Hopson, R. K.
(2009)
Explicitly names white privilege
Challenges knowledge claims that delegitimize the lives,
values and abilities of people of color
Positions CRE as multidimensional, recognizing
demographic, sociopolitical and contextual characteristics of
culture
Warns against taking deficit perspectives that “evaluate
down”
Knowledge as situational and context-bound
Important to think multiculturally rather than monoculturally
Recognizes intergenerational and fictive kin relationships
Theoretical support from Indigenous frameworks and critical
race theory (CRT)
Frierson, H. T.,
Hood, S.,
Hughes, G.
B., and
Thomas, V.
G. (2010)
Positions CRE as a holistic framework, guiding the manner in
which an evaluation is planned and executed
Legitimizes culturally -specific knowledge and ways of
knowing
Links validity of evaluation and service to the public good
Expands context as totality of environment—geographic,
social, political, historical, economic and chronological
Recognizes both formal and informal positions of power or
authority
Understand and respects varying communication and
relational styles
Employ best practices of linguistic translation
Importance of establishing trust and ownership of evaluation
Mixed-method designs as more fully addressing complexities
of cultural diversity
Links procedural ethics and relational ethics to cultural
responsiveness, including risks to both individuals and
communities
Evaluator self-reflection and reflective adaptation
Askew, K.,
Beverly, M.
G., and Jay,
M. (2012)
Careful attention to assembling the evaluation team.
Draws theoretical support from collaborative evaluation
Enumerates CRE techniques (in comparison with
collaborative techniques)
Intentionally creates space and obtains permission to bring
up and respond to issues of race, power and privilege
Bidirectional exchange of cultural content and knowledge
between evaluator and stakeholder
(Continued)
289
290 Handbook of Practical Program Evaluation
TABLE 12.1. KEY CHARACTERISTICS OF CULTURALLY
RESPONSIVE EVALUATION (CRE) FROM CORE LITERATURE.
(Continued)
Citation Core Characteristics of CRE
Frazier-Anderson, P.,
Hood, S., and
Hopson, R. K.
(2012)
Provides a culturally specific example of CRE for work
with and benefit of African American communities,
taking an Afrocentric perspective
Differentiates culture from race
Comprehensive contextual analysis, including social
capital and civic capacity
Warns against perceiving one’s own culture as the only
one of value (cultural egoism)
Underscores importance of history (of oppression and
resilience)
Need to establish competence and credibility of
evaluation team in communities of color
Protect or prevent the exploitation of cultural minority
and economically disadvantaged stakeholders
Uses sankofa bird to frame an Afrocentric logic model
Inclusion of a CRE panel review of findings as a system
of checks and balances
FIGURE 12.2. CULTURALLY RESPONSIVE EVALUATION
FRAMEWORK.
2
Engage
stakeholders
3
Identify evaluation
purpose(s)
4
Frame the
right questions
5
Design the
evaluation
6
Select and adapt
instrumentation
7
Collect
the data
8
Analyze
the data
9
Disseminate and
use the results
1
Prepare for
the evaluation
Cultural
competence
C
u
l
t
u
r
a
l
l
y
R
e
s
p
o
n
s
i
v
e
E
v
a
l
u
a
t
i
o
n
Culturally Responsive Evaluation 291
As they inventory resources available to support evaluation, CRE evaluators
are mindful of ways in which culture offers rich opportunities in the evaluation
process and challenges traditional evaluation that omits attention to culture.
CRE evaluators are aware of their own cultural locations vis-`
a-vis the commu-
nity, including prior experiences, assumptions, and biases. These understand-
ings support the formation of an appropriate evaluation team. The collective
life experiences of CRE team members should promote genuine connection
with the local context. While this may include demographic similarities among
evaluators and community members, team composition does not reduce to a
simplistic “matching” exercise. Evaluation team members are required to have
an array of skills, competencies, and sensibilities, consistent with the Guid-
ing Principles of the evaluation profession (American Evaluation Association,
2004).
Engaging Stakeholders
Stakeholders are persons who are invested in a program or its evaluation by
virtue of their roles, values, or perceived gains or losses. Not all stakeholders
share the same investment; one person’s benet may come at another person’s
expense. CRE evaluators seek to develop a diverse stakeholder group, inclusive
of persons both directly and indirectly impacted by a program, representative
of the community and/or population of persons served by the program. To
create opportunities for conversations about equity and fairness, CRE evalua-
tors seek to include stakeholders of different status or with differing types of
power and resources.
CRE evaluators must work to model and cultivate a climate of trust and
respect among stakeholders. Toward this end, it is important that there be
meaningful roles and activities for stakeholder engagement; token represen-
tation is insufcient and disingenuous (Mathie and Greene, 1997). CRE evalu-
ators are guided in their interactions with stakeholders by the third edition of
The Program Evaluation Standards (Yarbrough, Shulha, Hopson, and Caruthers,
2011). Standards U2 (Attention to Stakeholders) and P1 (Responsive and
Inclusive Orientation) both speak to the importance of stakeholder relation-
ships in evaluation.6Stakeholders can educate evaluators on important history
and background, help dene the parameters of what is to be evaluated (the
evaluand), identify priority questions to be addressed by the evaluation, serve
as sources of information, and offer advice on other sources of evidence as well
as on strategies of information-gathering appropriate to context. Stakeholders
292 Handbook of Practical Program Evaluation
can also aid in the interpretation of data and the skillful, clear communication
of ndings.
Identifying the Purpose and Intent of the Evaluation
Both the preparation of the evaluators and the engagement of stakeholders
help rene the understanding of the evaluand, including the boundaries of
what will and will not be examined. But appreciating the purpose(s) of CRE
goes beyond specifying the evaluand. Is this evaluation required by funders to
demonstrate accountability? Is it called for by a local citizens’ group? Is it part
of routine oversight or is it intended to clarify and troubleshoot an apparent
problem? Is continuation, expansion, or reduction of program funding con-
tingent upon conducting this evaluation or upon the content of the results?
Is it intended to stimulate change and promote social justice? Because a given
evaluation may have more than one purpose and not all purposes are overtly
stated, evaluators must take time to understand different aspirations for the
evaluation and how it could benet the program and community. CRE evalua-
tors in particular must be attuned to how the avowed purposes of the evaluation
maintain or challenge current (im)balances of power and how social justice is
served by the envisioned evaluation.
Framing the Right Questions
A pivotal point in the evaluation is coming to agreement on what questions are
to be answered and how they should be prioritized. For contexts in which direct
questions are culturally inappropriate, this stage identies what it is that stake-
holders seek to learn about the program or community (LaFrance and Nichols,
2009). Both the focus and the wording of questions or statements of intention
are critical here in order to set the evaluation on the right path. Will the eval-
uation focus on community needs and strengths, on the daily operation of the
program, on appropriate and equitable use of resources, on progress toward
intended outcomes, or on overall effectiveness? CRE is particularly attentive
to the perspectives of program recipients and community in framing the ques-
tions (for example, Is the program operating in ways that respect local culture?
How well is the program connecting with the values, lifestyles, and worldviews
of its intended consumers? How are the burdens and benets of the program
distributed?)
The process of revising and rening evaluation questions establishes
critical dialogue among stakeholders in CRE. CRE evaluators work with stake-
holders to reect on nuances of meaning and how different expressions of
Culturally Responsive Evaluation 293
intent may limit or expand what can be learned from an evaluation. Translation
of ideas or terms may require the assistance of linguistic or language orthog-
raphy experts (LaFrance, Kirkhart, and Nichols, 2015). This stage may appear
tedious, but it is critical in establishing clear understandings and insuring that
the evaluation will address the concerns of diverse stakeholders, authentically
expressed. This includes reaching agreement on the most important questions
to be answered with the available resources.
Closely related to the framing of questions or statements of desired
learning is the matter of what will be accepted as trustworthy evidence in
formulating answers. Conversations among stakeholders may reveal differ-
ent perspectives on what “counts” as credible evidence. This is important
information as CRE evaluators seek to maintain balance among stakeholder
perspectives, moving into the design stage.
Designing the Evaluation
The design of a CRE evaluation is responsive to context; it is not dictated by
the CRE approach itself. CRE designs are congruent with the questions to be
answered/learnings desired, the evidence that is valued by stakeholders, and
the cultural values represented in the setting. These often include an extended
time frame in order to build the relationships necessary to establish trust.
An evaluation design typically maps out the sources of information that
will be accessed to gather information (including people, documents or other
archival sources, and databases), the time frames in which data will be col-
lected, and the means by which data will be collected and analyzed. Frierson,
Hood, and Hughes (2002) and Frierson, Hood, Hughes, and Thomas (2010)
discuss instrumentation separately in the next stage, but the design stage
explicitly frames the parameters of the evaluation. In CRE, mixed methods are
now recommended (Frierson, Hood, Hughes and Thomas, 2010; LaFrance
and Nichols, 2009); however, in early formulations of CRE, qualitative data
were privileged over quantitative to restore balance to a historically quantita-
tive enterprise (Hood, 2001). The descriptor, mixed methods refers not only
to the nature of the information and its collection but also to the underlying
epistemologies as well as the processes through which qualitative and quanti-
tative data are combined (Greene, Benjamin, and Goodyear, 2001).
A nal design consideration of particular relevance to CRE is the types of
understandings sought. Are these holistic understandings? Are comparisons
required among persons receiving services and those not yet connected to
services? In order to answer the priority questions, will it be important to
disaggregate the data by culturally relevant categories? These considerations
294 Handbook of Practical Program Evaluation
have implications for both selection and assignment of participants in the
evaluation.
Selecting and Adapting Instrumentation
A major concern in multicultural contexts is the validity of assessment tools
and instruments. Working in the eld of counseling psychology, Ridley, Tracy,
Pruitt-Stephens, Wimsatt, and Beard (2008) argue that “much of the conduct
of psychological assessment is culturally invalid and therefore an ethical prob-
lem” (p. 23). Similar concerns hold true for educational testing (Johnson,
Kirkhart, Madison, Noley, and Solano-Flores, 2008). When selecting instru-
ments for use in CRE, existing tools must be closely scrutinized for cultural
bias in both language and content. Norms based on other populations and
locations may be of little value in interpreting local scores. Instruments must
be validated for use in culturally-specic contexts. When translation is used,
it should follow best practices, addressing both semantic and content equiv-
alence. For example, Frierson, Hood, Hughes, and Thomas (2010) suggest a
combination of forward/backward translation (FBT), multiple forward trans-
lation (MFT) or translation by a committee (TBC). Single (forward) transla-
tion alone is never sufcient.
When appropriate existing instruments are not available or they cannot
be satisfactorily adapted, original instruments must be developed specically
for CRE. Such instrument development will need to be reected in both the
timeline and the expertise of the CRE team.
Collecting the Data
Beyond the tools or instruments themselves, the procedures surrounding their
use must also be responsive to cultural context. This applies equally to the
collection of qualitative and quantitative data. Similar to when entering the
community context as a whole, cultural protocols often dictate who the eval-
uator speaks to rst and who has authority to grant access to other sources of
information. Likewise, introducing oneself to individuals or groups holding
valuable information must follow a respectful, culturally appropriate protocol.
Time is required to establish trust and to ensure that participation is voluntary
and information freely shared.
CRE evaluators appreciate how their own experiences and cultural
locations affect what they can see or hear. Additionally, they recognize the
importance of self as instrument (Hood, 2001; Manswell Butty, Reid, and
LaPoint, 2004). Data collectors must be trained not only in correct use of
Culturally Responsive Evaluation 295
observation tools, interview schedules, and questionnaire administration, but
in cultural context and expression (written, oral, and nonverbal). Shared
lived experience between the evaluator/observer and the persons providing
information in CRE can anchor trustworthy communication and support valid
understandings.
Analyzing the Data
Data do not speak for themselves; they are given voice by those who interpret
them. Here again, understanding cultural context is necessary for accurate
interpretation. To achieve this, CRE evaluators go beyond members of their
own team. A cultural interpreter(s) may be needed to capture nuances of
meaning. Stakeholders can be involved as reviewers to assist in interpretation,
respond to drafts, and suggest alternate explanations.
CRE evaluators take an investigative approach to data analysis that goes
beyond simple description or calculation of main effects. Diversity within
groups can be examined by disaggregating data to explore, for example, how
programs may affect some community members more or differently than
others. Outliers can be studied to shed light on complexities or to challenge
simple explanations with disconrming information. Positive outliers—those
who succeed without programmatic interference/assistance, for example—
may be particularly helpful in appreciating resilience within a community.
Data can be scrutinized for evidence of unintended outcomes—positive or
negative. The existence of positive unintended outcomes can expand one’s
understanding of program benets, while negative unintended outcomes sug-
gest important caveats or cautions that must be considered to prevent harm.
Disseminating and Using the Results
This nal stage closes the circle of the CRE evaluation framework illustrated
in Figure 12.2 (Hopson and Kirkhart, 2014), often raising new questions that
begin another evaluation cycle. For CRE evaluators, this stage holds potential
for social betterment and positive change; therefore, it is extremely impor-
tant. Cultural responsiveness increases both the credibility and utility of eval-
uation results. Benet to community can be supported by inviting commu-
nity review and comment on ndings before wider dissemination. Community
review also requires that the communication mechanisms themselves are cul-
turally appropriate and respectful of cultural values and protocols. Knowledge
gained from the evaluation must be effectively communicated to a wide range
of diverse stakeholders; therefore, multiple, sometimes audience-specic,
296 Handbook of Practical Program Evaluation
communication formats and procedures will be needed. This stage promotes
use consistent with the purposes of CRE, emphasizing community benet, pos-
itive change, and social justice.
Taken together, the steps or components form the guiding theoretical
framework of CRE that centers evaluation in culture. Still, the core premise
of the chapter suggests that CRE theory informs practice and CRE practice
builds theory. The next section provides practice examples that illustrate how
theory is elaborated in local application.
Case Applications of CRE Theory and Practice
Applications and practices of CRE are emerging from the seminal practice
guidelines articulated by Frierson, Hood, and Hughes (2002) and Frierson,
Hood, Hughes, and Thomas (2010). This section describes three recent appli-
cations of CRE in evaluation literature and illustrates how CRE practice con-
tributes to and informs theoretical understandings of evaluation. The works
cited below (Bowen and Tillman, 2014; Manswell Butty, Reid, and LaPoint,
2004; and Ryan, Chandler, and Samuels, 2007) are not the only references
that dene CRE in the last ten years (see, for example, Askew, Beverly, and Jay,
2012; Chouinard, 2013, 2014; Greene, 2006; Samuels and Ryan, 2011), but they
are selected for their specic focus on CRE practice. In this section, we focus
on how practice eshes out the operational details of CRE theory, specic to
the context of the evaluation.
Lessons Learned from Evaluating the Struggle of Brazil’s Quilombos
(Bowen and Tillman, 2014).
Bowen and Tillman (2014) explore an under-examined area of CRE practice by
presenting lessons learned from the development, implementation, and analysis
of surveys used to evaluate the struggle of Brazil’s quilombos (former fugitive slave
communities) for land rights and livelihood. With a purpose of producing “useful
and culturally valid data on quilombos” (p. 4), the authors employ a mixed method
CRE approach to inform land-based research projects in Brazil.
In the development of the surveys, the authors describe ways in which they
lived and researched in quilombo communities previously, jointly participated in
everyday activities at the local level in the land-based economy of the area, and
conducted focus groups with locally elected associations in order to heighten their
sensitivity and responsiveness to the culture and context of the quilombo commu-
nities. The authors also recount how quilombo feedback was sought through the
Culturally Responsive Evaluation 297
entire planning stage of the survey development process, from electronic mail and
telephone conversations to reframing questions and reducing the length of the
survey to providing input on selection of communities for the study, remuneration
for survey enumerators, and suggested ways of disseminating results.
In the implementation of the surveys, the authors describe the use of teams of
enumerators to orally administer household and agriculture surveys, one team of
a multi-racial and gendered group of university students who had research experi-
ence in rural communities but lived in cities and one team of quilombola students
who were raised and resided in various communities in the study. Still, despite lan-
guage issues that were addressed, authors and team members had challenges with
survey implementation in regard to rephrasing or adding survey questions in the
field to allow for greater understanding of the local household and cultural context
and more comprehension among respondents as illustrated below:
Some survey questions were rephrased in the eld because neither the
enumerators nor the respondents easily understood them. For example, one of
the shortcomings of the household survey had to do with the denition and
boundaries of the “household.” According to the LSMS7household survey, which
informed our work, a residential denition of the household includes members
who have eaten and slept in the house at the time of interviewing for the last six
to twelve months or “normally.”‘ But the respondents did not easily comprehend
this denition because there were household members who did not reside (or
only sporadically) in the main residence but who contributed signicantly to the
expenses of the household and dependents from their income sources. (p. 9)
The authors illustrate where data analysis adjustments were made to attend to
cultural nuances of the quilombo communities surveyed. They offer three exam-
ples from their study in the areas of analysis of land measurement, crop yields and
marketed production, and wage labor versus self-employment, showing how they
contextualized results sensitive to the local conditions of the community. Occupa-
tions considered as self-employment in the communities carry different meanings
to international labor experts than they do to enumerators and respondents of the
survey administered, illustrative of larger challenges in rural labor survey design.
Case Study of Culturally Responsive School-Based Evaluation in
Southwestern U.S. (Ryan, Chandler, and Samuels, 2007)
Ryan, Chandler, and Samuels (2007) report on an instrumental, mixed-method
case study evaluation of a culturally responsive school-based federally funded ini-
tiative involving three urban public schools and one Navajo reservation school in
298 Handbook of Practical Program Evaluation
the southwestern United States that were not making “adequate yearly progress,”
the measurement defined by the No Child Left Behind Act (2002) to determine
how public schools and school districts perform on standardized tests in the United
States. By having teachers and principals participate in professional development
workshops designed to assist them with skills needed to conduct and design cultur-
ally responsive and school-based evaluation for their respective schools, the funded
project was designed to develop an empirically-based model for operationalizing
culture in evaluation and to teach schools how to develop evaluation capacity.
The authors describe data collection methods designed to provide triangula-
tion of interviews, focus groups with project team members, school and team lead-
ers, and a national consultant. Additionally, the authors were informed by docu-
ment analysis of evaluation plans and other relevant project information, including
video of school team forums.
Several findings inform the case study. First, as reported by Ryan, Chandler, and
Samuels (2007), the initial struggle with the meanings of culture by key school
team leaders resulted in developing data-based decision-making influenced by
nuanced notions of culture. Moving beyond initial recognition of ethnic and racial
diversity (only), changes in understanding culture and contextual factors led to
reconsidering their role in re-defining solutions to address their cultural realities.
The authors write:
Where schools may have initially balked at the concept of culture and evaluation,
progress was made as one school later began to disaggregate school
accountability data to target an underserved group of students and to provide
additional support to that subgroup to address areas of shortfall. According to
one participant, “teachers are starting to challenge the data. When teachers
begintheirgradelevelmeetings,theystartbyreviewingthedata...even
individual student [data].” This level of understanding was not apparent across
all participating schools, yet this kind of progress holds promise for schools doing
culturally responsive evaluation by being more inclusive in their discussions
about the meaning of data among key stakeholder groups. (p. 205)
As teachers began to think evaluatively, they began to have a better under-
standing of achievement and culture at their respective schools. Data collected
and analyzed indicate they showed adequate understanding about explaining the
context of the program, engaging stakeholders, determining the purpose of the
evaluation and designing a culturally oriented evaluation, but challenges were evi-
dent in instrument design and dissemination and utilization of results.
The authors suggest that structural and theoretical issues play an important
role in understanding the practical and logistical challenges with introducing the
notion of culture in school-based evaluation. For instance, the authors indicate
Culturally Responsive Evaluation 299
that values discussions and orientations are inevitable in applications and under-
standings of culture in school-based evaluation settings that tend to emphasize
top-down, bureaucratic structures and processes. Being inclusive is one thing, but
CRE evaluators must recognize the theoretical tensions in being inclusive and in
shifting power dynamics in schools.
Additionally, the authors raise questions about what should be expected
from novice, school-based evaluators who attempt to infuse culture in evaluation.
Should standards and expectations for internal school evaluation teams be the
same as standards and expectations for those who conduct external evaluations?
The authors also note that open discussions about culture can raise tensions and
that such conversations did not necessarily fit the bureaucratic, hierarchical struc-
tures and practices that existed among the four participating schools.
Successes and Challenges Evaluating an Urban School Talent
Development Intervention Program in the U.S.
(Manswell Butty, Reid, and LaPoint, 2004)
Manswell Butty, Reid, and LaPoint (2004) describe and analyze a Talent Devel-
opment (TD) school program in partnership with Howard University’s Center for
Research on Education of Students Placed at Risk (CRESPAR). The authors describe
a School-to-Career Transitions intervention that took place at a junior high school
in an urban northeastern part of the United States. It was designed to improve
the knowledge, attitudes, and practices of junior high school students related to
school-to-career opportunities in their transition from elementary to middle school,
middle to high school, and high school to post-secondary options through a variety
of learning activities. Specifically, the intervention was a Breakfast Club (including
interactive discussion groups and activities) that took place for one hour prior to
the start of the formal school day. Participants were seventeen ninth-grade stu-
dents who were expected to graduate during the academic year. Session evalu-
ations, self-assessments, and pre- and post-tests were collected during the eight
workshops that took place.
These authors provide a clear example of how the practice of CRE builds theory
on the ground by operationalizing general principles of the CRE framework at each
of the nine stages. Their work also illustrates how the stages overlap and repeat;
they are not distinctly separate, linear activities as illustrated below:
Stage 1. Preparing for the Evaluation. The TD evaluation was interwoven with the
TD intervention in the school, so evaluators met on multiple occasions with stake-
holders to understand the sociocultural context of that particular school. Evaluators
listened carefully to the perspectives of the principal, counselor, liaison, teachers,
and students and also reviewed student profiles to determine program goals and
aspirations. In a two-way exchange of information, evaluators brought relevant
300 Handbook of Practical Program Evaluation
findings from prior research to inform program development. Student, family, and
community factors were kept clearly in view as both the program and the evalua-
tion were tailored to fit the specific context of application.
Stage 2. Engaging Stakeholders. Stage 1 preparation included stakeholder
engagement, but in Stage 2, the authors gave particular attention to building solid
relationships with the school principal, liaison, and counselor, who were identified
as key stakeholders. The school liaison, was selected as the key point of contact
for evaluators, and the authors were explicit in identifying the personal character-
istics that made her ideal for this role. Her academic training, extensive experience
of over thirty years in public education, and perhaps most significantly, her gen-
uine commitment to the welfare of the students aligned her with the principles of
CRE. This created an atmosphere in which ideas and suggestions could be freely
exchanged and debated.
Stage 3. Identifying the Purpose and Intent of the Evaluation. This evaluation
was intended to serve both formative and summative purposes, and the evalua-
tion team laid these dual functions out clearly for the mutual understanding of
all stakeholders. The formative purpose was to document and describe program
operations, providing ongoing feedback to inform program staff so that they could
continue to develop and improve the program. Feedback from the students partic-
ipating in Breakfast Club “was used to fine-tune subsequent sessions to make them
more valuable and enjoyable” (Manswell Butty, Reid, and LaPoint, 2004,p.42).
The summative purpose was to determine whether the Breakfast Club achieved its
objectives, focusing on the direct effects on participants.
Stage 4. Framing the Right Questions. Following the CRE framework, the evalu-
ation included questions of concern to key stakeholders—school principal, liaison,
and counselor. Questions from the broader TD project level were adapted and
tailored to local context. To confirm that the right questions were being posed,
evaluators constructed a “data map” or visual matrix so that everyone was clear
how the evaluation questions related both to the Breakfast Club intervention and
to the evidence that was needed to answer each question.
Stage 5. Designing the Evaluation. This evaluation used mixed methods to pro-
mote both conceptual and methodological triangulation. Qualitative data included
interviews and written assessments and quantitative data included surveys and self-
assessments. Evaluators paid particular attention to both the preferred schedule
and method of data collection, noting that “many students preferred discussions
and other interactive activities rather than filling out surveys and self-assessments”
Manswell Butty, Reid, and LaPoint, 2004, p. 43). The career self-assessment survey
was administered to all ninth graders in the school, so the Breakfast Club partici-
pants were not singled out and comparisons could be made.
Stage 6. Selecting and Adapting Instrumentation. Many of the evaluation tools
were developed or adapted specifically for use with the Breakfast Club audiences.
Instruments were carefully selected with attention to their cultural sensitivity in
Culturally Responsive Evaluation 301
form, language, and content. The authors report that one of the standardized tools
was normed on majority populations, calling into question the validity of score
interpretation for students of color. The authors countered this by augmenting
that score with other outcome data that were more responsive to context.
Stage 7. Collecting Data. Overlapping instrumentation and data collection in
CRE is the recognition of self-as-instrument. Those who are collecting the data
must be attuned to the nuances of expression and communication specific to the
contexts of this program, school, and community. TD evaluators shared a racial
background with the stakeholders, entering the school with “an increased level
of sensitivity and awareness to the plight and lived experiences of the various
stakeholder groups” (Manswell Butty, Reid, and LaPoint, 2004,p.44).Theauthors
credited the (Stage 1) multiple meetings with stakeholders and participation in or
observation of school-related functions with helping evaluators be responsive to
context and culture.
Stage 8. Analyzing Data. The stakeholder conversations previously cited
extended into data interpretation. Stakeholders advised on how best to analyze
and interpret data to derive valid, contextualized meaning. Whole group data were
disaggregated to examine differences by gender and age. The age disaggregation
was especially relevant to this context, in which ninth-graders ranged in age from
fourteen to sixteen and were therefore at considerably different developmental
stages.
Stage 9. Disseminating and Using the Results. Evaluation findings were reported
to stakeholders in formats tailored to communicating effectively with each audi-
ence. Feedback to participants was delivered “in a student-friendly manner”
(p. 45). Results were explained to the principal, counselor, and liaison within the
context of the Breakfast Club program, so that practice implications were clear.
Successes as well as challenges were highlighted. Findings were also disseminated
to funders and other project staff outside of the immediate context of application.
This study is unique in enumerating practice contributions to CRE theory at
each stage of the framework. Overall, the authors also suggest that the culturally
responsive evaluation approach is labor-intensive but effective. In ensuring a team,
collaborative approach, responsive to aspects of culture and context at all stages of
the evaluation, and flexible enough to combine evaluation approaches for different
situations appropriate to the evaluand, the authors affirm that the approach “led
to an intervention and evaluation that benefited stakeholders and participants, as
evidenced by student and staff evaluations and positive student outcomes” (p. 45).
The three practice applications presented in this section illustrate the var-
ied integration of the CRE theoretical framework and, even more importantly,
how practice applications take a general framework and ll in context-specic
details on the ground. Similarly, the three cases provide examples of practice
in indigenous communities or communities of color and offer an opportunity
302 Handbook of Practical Program Evaluation
to dig deeper into matters that pertain to cultural context in evaluation and
ways in which culture is centered in evaluation practice.
The cases furthermore provide a clearer picture of the way theory informs
practice and practice informs theory. Ultimately, the articles show over a
decade how CRE practice happens in three distinct ways in international,
indigenous, and minoritized school and community contexts and how CRE
theory is deepened through practice in three different contexts.
Implications for the Profession
Whereas the previous sections examined how CRE theory is understood in its
historical context, provided an overarching framework for evaluation practice,
and described the ways in which CRE practice develops strategies that opera-
tionalize theory, this nal section addresses how CRE presses the eld itself to
revisit basic premises. In short, this nal section examines ways in which CRE
challenges the evaluation profession to expand its thinking and examine the
cultural location of core ideas such as validity, rigor, and the responsibilities of
the evaluator role.
Validity, Rigor, and CRE
One of the benets of centering evaluation in culture is that it pushes the
profession to examine and reect on respected standards of inquiry and to
see these in a new light. Consider three points regarding validity and rigor
congruent with CRE: (1) validity must be multicultural; (2) rigor should not
be equated with precision; and (3) rigor (and in turn, validity) is supported
by conceptual tools such as the Key Evaluation Checklist (Scriven, 2013)orA
Culture Checklist (Kirkhart, 2013a-b).
Concerns for validity have accompanied the development of CRE from
its earliest appearance as culturally responsive assessment (see, for example,
Boodoo, 1998; Hood, 1998a;Qualls,1998). In broad brush, validity marks
the correctness or trustworthiness of evidence-based understandings and
actions. But how should validity be understood in the context of CRE? Like
other theories, validity theory needs to be congruent with evaluation context,
so in the case of CRE, the concept of validity itself must be expanded and
repositioned to address the core characteristics of CRE listed in Table 12.1.
Validity must be understood as truly multicultural, open to perspectives
previously marginalized (Kirkhart, 1995), and it must be repositioned to
Culturally Responsive Evaluation 303
center it in culture (Kirkhart, 2013a)sothatall denitions of validity are
understood as culturally located.8
Kirkhart (1995;2005) has argued for a vision of validity that reects mul-
tiple cultural intersections. She uses the term multicultural validity not to spec-
ify a new type of validity but to suggest that validity is an expansive construct
that can be understood from multiple perspectives, including those histori-
cally marginalized. In repositioning validity in culture, Kirkhart (2013a)has
examined the perspectives from which validity is argued in feminist theory,
CRT, Indigenous epistemology, queer theory, disability studies, and aging stud-
ies, as well as measurement theory and research design. Justications have
been identied in ve categories at this writing; however, these understand-
ings continue to evolve (LaFrance, Kirkhart, and Nichols, 2015). Each of the
ve justications—methodological, relational, theoretical, experiential, and
consequential—is congruent with CRE (see Table 12.2 ). These justications
may stand alone or be used in combination to argue the validity of CRE. Con-
versely, when a justicatory perspective is ignored or violated, it may weaken
support for (threaten) validity (Kirkhart, 2011).
Rigor typically refers to compliance with strict standards of research
methodology (Johnson, Kirkhart, Madison, Noley, and Solano-Flores, 2008). It
is valued primarily because it supports methodological justications of validity,
but like validity, it requires an expanded conceptualization to make it useful to
CRE. While scientic rigor can serve several purposes that “advance under-
standing and ultimately advantage communities of color and other under-
represented groups” (Johnson, Kirkhart, Madison, Noley, and Solano-Flores,
2008, p. 200), narrow denitions of scientic rigor undermine validity. What
then does it mean to do rigorous evaluation that is culturally responsive? What
are the hallmarks of rigor for CRE? Are these specic to CRE or are they, simply
“good evaluation”?9
Nearly three decades after Lincoln and Guba (1986) cautioned that tra-
ditional criteria of rigor grounded in post-positivism were inadequate to the
task of evaluating the quality of all evaluation, we have a better sense of alter-
nate denitions of rigor and of criteria to achieve it. But whether one is work-
ing from a post-positivist or alternate paradigm such as CRE, rigorous inquiry
has been historically rule-driven, with strict standards to be met or bars to be
cleared. This understanding presents two challenges for rigor in CRE: equat-
ing rigor with precision and with xed, preordinate criteria.
The rst challenge emerges from implicitly associating rigor with preci-
sion. Precision values xed, often narrowly dened, boundaries that reect
positivist yearning for singular truths. Sharp denitions and exact specica-
tions are viewed as accurate and correct, while loose, holistic understandings
TABLE 12.2. JUSTIFICATIONS OF VALIDITY UNDER CULTURALLY
RESPONSIVE EVALUATION (CRE)a.
JustificationbDescription Applications in CRE
Methodological Validity is supported by the
cultural appropriateness
of epistemology and
method—measurement
tools, design
configurations, and
procedures of
information gathering,
analysis and
interpretation
Epistemology of persons
indigenous to the community
grounds the evaluation.
Measurement tools have been
developed for a particular
ethnic group and/or validated
for a context-specific use.
The sampling frame insures
inclusion of diverse cultural
perspectives appropriate to
the program being evaluated
and its context.
The study design employs a time
frame appropriate to the
cultural context.
Evaluation questions represent a
range of perspectives, values
and interest.
Relational Validity is supported by the
quality of the
relationships that
surround and infuse the
evaluation process
Evaluators respect local norms
and authority in entering the
community to undertake
evaluation.
Evaluators understand the
historical and spiritual
significance of the land and
the geographic location of
their work.
Evaluators take time to build
relationships and
understandings as part of the
early process of planning and
design development.
Evaluators reflect on their own
cultural positions and positions
of authority with respect to
other participants in the
evaluation process.
Meaningful roles are established
for stakeholder participation,
and barriers to full
participation are addressed.
(Continued)
304
TABLE 12.2. JUSTIFICATIONS OF VALIDITY UNDER CULTURALLY
RESPONSIVE EVALUATION (CRE)a.(Continued)
JustificationbDescription Applications in CRE
Theoretical Validity is supported by
invoking culturally
congruent theoretical
perspectives.
Evaluators frame their work in CRE
principles and practices, which in
turn are drawn from culturally
grounded social science theories.
When social science research is used
to develop program theory, it is
first examined with respect to its
multicultural validity and fit with
context.
Program theory is grounded in the
cultural traditions and beliefs of
program participants.
Validity theory itself is examined for
culturally-bound biases and
limitations.
Experiential Validity is supported by
the life experience of
participants.
Local citizens and program
consumers contribute their
wisdom to the evaluation process.
Evaluators reflect on their own
history and cultural positions,
seeking assumptions and “blind
spots.”
Evaluators employ a cultural guide
to increase their understanding
and appreciation of local culture.
Evaluative data are understood and
interpreted in terms of the
realities of the people they
represent.
Consequential Validity is supported by
the social
consequences of
understandings and
determinations of
value, and the actions
taken based upon
them.
History of evaluation in this
community is acknowledged and
addressed, especially if that
history is oppressive, exploitive.
Mechanisms are identified and
negotiated by which evaluation
and evaluators will give back to
the community.
Evaluation improves the ability of
the community to advance its
goals and meet the needs of its
members.
Evaluation promotes social justice.
aTable 2 is adapted from Hopson and Kirkhart (2012).
bJustifications of multicultural validity developed by Kirkhart (2005,2013a-b).
305
306 Handbook of Practical Program Evaluation
are denigrated as imprecise or incorrect. Such a narrow perspective on rigor
does not serve CRE well, however. To be responsive to culture and context, eval-
uation must include a broad vision, taking meaning not only from the minute
and precise, but also from the holistic and expansive, for example, the history
and worldview of the people who are stakeholders in the evaluation. Restrict-
ing the range of vision of evaluation methodology in the name of rigor under-
mines, rather than supports, valid understandings.
The second challenge is that preordinate criteria of rigor—those that are
specied at the outset—often do not match the world of practice. CRE is often
emergent in its design, grounded in its relationship with community, and uid
in its response to changing circumstances or resources. Rigor is not aban-
doned, but it may be more appropriately cast as criteria that guide evaluation
practice rather than as xed bars to be cleared.
The context-specic nature of CRE demands an understanding of rigor
that is also tted to context, providing guidance but not blocking the holis-
tic vantage point or the emergence of new understandings. A CRE-compatible
strategy is a non-linear, iterative, conceptual checklist such as Scriven’s (1991;
2013) Key Evaluation Checklist (KEC), which can guide rigorous CRE and sup-
port the validity of resulting understandings. Drawing on multicultural validity
theory, Kirkhart (2013a-b) proposed “A Culture Checklist” of nine conceptual
elements that can serve as hallmarks of rigor in CRE and beyond: history, loca-
tion, power, voice, relationship,10 time, return, plasticity, and reexivity (see
Table 12.3). Each concept links back to and supports one or more justications
of multicultural validity; hence the elements are intertwined, not independent
of one another. These represent concepts to be considered iteratively while
planning and implementing CRE. Used reexively, checklists such as these
can be used to keep the CRE evaluation on course and ag considerations and
activities that cannot be compromised. For any particular CRE application, it
may also be necessary to create a contextually-specic list of core considera-
tions that draws upon ideas and values central to that community (Kirkhart,
2013a).
Responsibility as a Core Principle of CRE
Central to the core principles of CRE (Table 12.1) as well as in the identity
of the CRE evaluator are responsibility and responsiveness. CRE evaluators
recognize the sense of “social responsibility” that requires the work to be
responsive to the community that is served. Hood (2001) asserted that African
American evaluators from the pre-Brown vs. Board of Education11 era acted
TABLE 12.3. A CULTURE CHECKLIST (ADAPTED FROM KIRKHART,
2013b, PP. 151–152).
Element Content Description Questions Raised
History History of place, people,
program (or other
evaluand), and
evaluation’s role.
Knowledge of cultural
heritages and traditions,
including their evolution
over time.
What is the story of this community?
What is the story of how this
program came to be in this place?
How has what is here today been
shaped by what came before it?
What is the history of evaluation in
this community or with this
program?
Location Cultural contexts and
affiliations of evaluators
and evaluand, including
theories, values,
meaning-making, and
worldviews. Recognizes
multiple cultural
intersections at
individual,
organizational, and
systems levels.
Geographic anchors of
culture in place.
What are the cultural identifications of
persons in this community and how
do these compare to those of the
program staff and of the evaluators?
What is valued here? How do people
understand their lives? What is the
geography of this place? How do
people relate to the land?
Power Understanding how
privilege is attached to
some cultural signifiers;
prejudice to others.
Attention to address
equity and social justice,
avoid perpetuating
condescension,
discrimination or
disparity.
Who holds power in various ways, and
what are the impacts of how power
is exercised? What are the formal,
legal, political, social and economic
sources of power? What are the
informal sources of power?
Connection Connections among the
evaluation, evaluand
and community.
Relating evaluation to
place, time and
Universe. Maintaining
accountability to
community with respect
and responsibility.
Establishing trust in
interpersonal
relationships.
How do members of the community
relate to one another, to the
program and its personnel, and to
the evaluators? How do the
evaluators relate to persons in the
program and in the community?
How does the evaluation relate to
the core values of the cultures,
community and context?
(Continued)
307
TABLE 12.3. A CULTURE CHECKLIST (ADAPTED FROM KIRKHART,
2013b, PP. 151–152). (Continued)
Element Content Description Questions Raised
Voice Addresses whose perspectives
are magnified and whose
are silenced. Maps inclusion
and exclusion or
marginalization. Includes
use of language, jargon,
and communicative
strategies.
Who participates in the planning,
design, and implementation of
the evaluation? Whose messages
are heard and heeded? Whose
methods of communication are
reflected in the languages and
expressions that are used to
discuss the evaluation process,
raise questions, interpret findings,
and communicate results?
Time Calls attention to rhythm,
pace and scheduling and to
the wide vision of past and
future. Encourages
evaluation to consider
longer impacts and
implications—positive or
negative.
How does the rhythm of this
evaluation fit the context? Is it
moving too fast? Too slowly? Has
it considered important outcomes
at various points in time? Will it
have the patience to watch
carefully for small changes? For
long-term consequences?
Return Supports reciprocity by
focusing attention on how
the evaluation and/or the
persons who conduct it
return benefit to the
evaluand and the
surrounding community.
Addresses returns both
during and after the
evaluation process.
Positions the evaluation as
non-exploitive.
How does evaluation advance the
goals of this community or serve
the needs of its people? Has the
benefit returned to community
compensated them fairly for their
time and attention and for any
disruption created by this
evaluation? In what ways are
persons better off? Have any been
harmed or disadvantaged?
Plasticity The ability to be molded, to
receive new information,
reorganize and change in
response to new
experiences, and evolve
new ideas and forms.
Applies both to the persons
who do evaluation and to
their designs, process and
products. Because culture is
fluid, not static, evaluation
must be responsive.
How is this evaluation changing in
response to local context? Are we
evaluators staying open to new
ideas or are we overly committed
to following a fixed plan or
timeline? What has surprised us
here that changes how we think
about evaluation? What have we
learned here that is new and/or
changes our understanding of
good evaluation?
(Continued)
308
Culturally Responsive Evaluation 309
TABLE 12.3. A CULTURE CHECKLIST (ADAPTED FROM KIRKHART,
2013b, PP. 151–152). (Continued)
Element Content Description Questions Raised
Reflexivity Applies the principles of
evaluation to one’s own
person and work, from
self-scrutiny to
metaevaluation.
Supports reflective
practice. Underscores
the importance of
metaevaluation.
What do I think I know in this context
and why? What do I know that I
don’t understand? What areas of
new learning must I watch for and
reflect upon? What do I need to let
go of or relearn, and how can I work
on that? What are the strengths and
limitations of this evaluation and
how it has addressed culture? How
strong are the arguments supporting
validity? What counterarguments
challenge validity?
on their “social responsibility” to address the inequities of segregated school-
ing by using their research, evaluation, and scholarly skills in the evaluation of
education systems in the South. The work of Reid E. Jackson was most notable
in this regard (Hopson and Hood, 2005).
Hood and Hall (2004), in their implementation of their NSF funded Rel-
evance of Culture in Evaluation Institute, along with the project’s advisory
board, devoted considerable thought to the characteristics of the “culturally
responsible evaluator.” First they made the distinction between being respon-
sive and being responsible. Being “responsible” is viewed as an active behavior
manifested in advocacy of social justice for those who had been traditionally
disenfranchised. To act on this responsibility requires one to be responsive by
being aware and recognizing the centrality of culture and cultural context in
our evaluative work and identifying the appropriate methods and tools that will
best serve the community. Culturally responsible evaluators are characterized as
those who:
rPrioritize and are responsive to the needs and cultural parameters of those
who are being served relative to the implementation of a program and its
outcomes,
rInvolve self in learning, engaging and appreciating the role of culture(s)
within the context of the evaluation,
rLearn to recognize dissonance within the evaluation context, for example,
between school and community groups being served, and
rAre committed to educating themselves, continuing to acquire training and
experience in working in diverse settings.
310 Handbook of Practical Program Evaluation
The centrality of responsibility and responsiveness to our conceptualiza-
tion of CRE is congruent with the work reported by the M¯
aori and other
indigenous members of our CRE family. It is particularly captured in the cultur-
ally specic evaluation of Kaupapa M¯
aori (that is, a M¯
aori way). Cram, Kennedy,
Paipa, Pipi, and Wehipeihana (2015) inform us that Kaupapa M¯
aori evaluation
is grounded in the discovery of the true M¯
aori Kaupapa to guide evaluators in
their determination of not only the right methods but also the right people to
undertake the evaluation. Ultimately, it isthe responsibility of M¯
aori to advance
aM
¯
aori way of evaluation. This culturally specic/responsive approach is con-
cretely illustrated by Paipa, Cram, Kennedy, and Pipi (2015)astheseM
¯
aori
evaluators utilize culturally responsive methods in a “family centered evalua-
tion” approach. In this case the M¯
aori evaluators act upon their responsibility
and accountability “to identify culturally relevant ways of working that make
sense to wh¯
anau [family] and align with wh¯
anau values with regard to kinship
and relational connections” (p. 329).
The work of the pre-Brown African American evaluators in the 1930s to
1954 (Hood, 2001) and work that is currently being reported by indigenous
evaluators such as the M¯
aori (Paipa, Cram, Kennedy, and Pipi, 2015)maysug-
gest a historical foundation connecting evaluators of color. This connection
is possibly found in their mutual recognition and use of culturally respon-
sive methods as they act upon their social responsibility to their communities
(Hood, 2001). Just as pre-Brown African American evaluators took responsi-
bility for being responsive in their own communities, there is a growing body
of work in indigenous communities to address these same matters of respon-
siveness; that is, to nd ways to use evaluations toward social and community
responsibility.
Conclusion
Culturally responsive evaluation marks a signicant advance in the ability of
the evaluation profession to address culture. It not only provides a valuable
framework for evaluation practice, but it challenges evaluators to reect on
power dynamics and sharpen their attention to social justice. This section sum-
marizes key points elaborated in the chapter.
CRE has a dened theory base and conceptual framework to guide prac-
tice. The theory base incorporates existing evaluation approaches and is inu-
enced by other culturally responsive notions in assessment and education. It
builds on a framework developed by Frierson, Hood, and Hughes (2002)and
Culturally Responsive Evaluation 311
Frierson, Hood, Hughes, and Thomas (2010), where cultural context is inte-
grated into evaluation practice and evaluation practice is centered in culture.
While there are many contributors to the development of CRE, its
historical development was inuenced by Stafford Hood’s funded research,
professional collaboration, and written work. From his Amistad paper to
collaborative work, he has encouraged and hopefully inspired others to
further rene the conceptualization of CRE as well as its applications.
CRE practice represents the fruits of this earlier conceptual work and it
contributes to local CRE theory. To understand and appreciate CRE practice
fully means understanding how it informs CRE and how CRE theory is ulti-
mately eshed out in the nuances and details of local context.
Future implications of CRE suggest that new approaches to core concepts
such as validity deserve more exploration. Additionally, CRE requires under-
standing and recognizing the importance of our responsibility as evaluators
and translating responsiveness in our practice. As illustrated in the chapter, the
future of CRE will be advanced through well documented practice examples
with rich detail (for example, Greene, 2015), combined with further reec-
tion on and articulation of alignments between CRE and other evaluation
approaches (for example, Askew, Beverly, and Jay, 2012). These advances coin-
cide with the need to develop increasingly sophisticated ways to center evalu-
ation in culture, both domestically and internationally. CRE stands poised to
contribute and we as members of the CRE community are collectively com-
pelled to use it as we act upon our responsibility to make a difference.
Notes
1. We carefully distinguish culturally responsive evaluation from other similar
approaches such as culturally competent evaluation or cross-cultural evaluation
(Chouinard, 2013; Chouinard, 2014; Chouinard and Cousins, 2009) which attend
to matters of culture in local and international settings but have distinct histories
and foci in evaluation.
2. Stake’s more recent work also addresses cultural pluralism as part of responsive
evaluation (Stake, 2004).
3. The title of this work references the historic Amistad trial, in which James Covey’s
role as “the portal between two conicting cultures” (Hood, 1998b, p. 108) was the
vehicle that made the defense of the Mende survivors culturally responsive. Hood
describes how Covey had been born and raised Mende but was subsequently cap-
tured and held as a slave. After being freed from a slave ship by a British naval vessel,
he learned to read and write in English and served as a sailor on a British brig of
war. Covey’s lived experience in both worlds made him essential to the understand-
ings in the trial. Similarly, Hood argues, African American evaluators play a key role
312 Handbook of Practical Program Evaluation
in the evaluation of educational programs that serve African American students by
deepening the understanding of a program, its value for participants, and potential
improvements needed to increase benets to culturally diverse groups.
4. The Nobody Knows My Name project takes its title from the 1961 collection of
essays by James Baldwin, Nobody Knows My Name; More Notes of a Native Son. It initially
focused on men of color who laid signicant intellectual groundwork in evaluation.
Scholarly contributions of women of color are a more recent addition. See, for
example, Frazier-Anderson and Bertrand Jones (2015).
5. Note that Figure 12.2 is referenced in Bledsoe and Donaldson (2015), but was devel-
oped earlier in the presentation by Kirkhart and Hopson (2010).
6. U2 Attention to Stakeholders. Evaluations should devote attention to the full range of
individuals and groups invested in the program and affected by its evaluation. P1
Responsive and Inclusive Orientation. Evaluations should be responsive to stakeholders
and their communities.
7. This refers to the World Bank’s Living Standards Measurement Survey (World
Bank, 2011, cited in Bowen and Tillman, 2014).
8. While there may be no direct linguistic translation for the Western, English-
language construct of validity, concerns for goodness, trustworthiness, and authen-
ticity emerge in different cultural contexts (LaFrance, Kirkhart, and Nichols, 2015).
9. Interestingly, similar questions were raised when Ladson-Billings proposed cultur-
ally responsive pedagogy—isn’t that just good teaching? (Ladson-Billings, 1995a).
10. Colleagues at the Aotearoa New Zealand Evaluation Association Conference in
Wellington (July 2014) suggested that relationship be retitled connection. This change
is reected in Table 12.3.
11. Brown v. Board of Education was the landmark U.S. Supreme Court decision that
found separate public schools for blacks and whites to be unconstitutional. Handed
down in 1954, it was considered a major victory of the civil rights movement and
led to the integration of American educational and public facilities in the South.
References
American Evaluation Association. Guiding Principles for Evaluators. www.eval.org/p/cm/ld/
d=51, 2004.
Askew, K., Beverly, M. G., and Jay, M. “Aligning Collaborative and Culturally Responsive
Evaluation Approaches.” Evaluation and Program Planning, 2012, 35(4), 552–557.
Bledsoe, K., and Donaldson, S. “Culturally Responsive Theory-Driven Evaluations.” In S.
Hood, R. K. Hopson, and H. Frierson (eds.), Continuing the Journey to Reposition Culture
and Cultural Context in Evaluation Theory and Practice (pp. 3–27). Greenwich, CT:
Information Age Publishing, Inc., 2015.
Boodoo, G. M. “Addressing Cultural Context in the Development of Performance-Based
Assessments and Computer Adaptive Testing: Preliminary Validity Considerations.”
Journal of Negro Education, 1998, 67(3), 211–219.
Bowen, M. L., and Tillman, A. S. “Developing Culturally Responsive Surveys. Lessons in
Development, Implementation, and Analysis from Brazil’s African Descent
Communities.” American Journal of Evaluation, 2014, 35(4), 507–524.
Culturally Responsive Evaluation 313
Chouinard, J. A. “The Case for Participatory Evaluation in an Era of Accountability.”
American Journal of Evaluation, 2013, 34(2), 237–253.
Chouinard, J. A. “Understanding Relationships in Culturally Complex Evaluation
Contexts.” Evaluation, 2014, 20(3), 332–347.
Chouinard, J. A., and Cousins, J. B. “A Review and Synthesis of Current Research on
Cross-Cultural Evaluation.” American Journal of Evaluation, 2009, 30(4), 457–494.
Cram, F., Kennedy, V., Paipa, K., Pipi, K., and Wehipeihana, N. “Being Culturally
Responsive Through Kaupapa M¯
aori Evaluation.” In S. Hood, R. K. Hopson, and H.
Frierson (eds.), Continuing the Journey to Reposition Culture and Cultural Context in
Evaluation Theory and Practice (pp. 289–231). Greenwich, CT: Information Age
Publishing, Inc., 2015.
Frazier-Anderson, P. N., and Bertrand Jones, T. “Analysis of Love My Children: Rose Butler
Browne’s Contributions to Culturally Responsive Evaluation.” In S. Hood, R. K.
Hopson, and H. Frierson (eds.), Continuing the Journey to Reposition Culture and Cultural
Context in Evaluation Theory and Practice (pp. 73–87). Greenwich, CT: Information Age
Publishing, Inc., 2015.
Frazier-Anderson, P., Hood, S., and Hopson, R. K. “Preliminary Consideration of an
African American Culturally Responsive Evaluation System.” In S. Lapan, M. Quartaroli,
and F. Riemer (eds.), Qualitative Research: An Introduction to Methods and Designs
(pp. 347–372). San Francisco, CA: Jossey-Bass, 2012.
Frierson, H. T., Hood, S., and Hughes, G. B. “Strategies That Address Culturally Responsive
Evaluation.” In J. Frechtling (ed.), The 2002 User-Friendly Handbook for Project Evaluation
(pp. 63–73). Arlington, VA: National Science Foundation, 2002.
Frierson, H. T., Hood, S., Hughes, G. B., and Thomas, V. G. “A Guide to Conducting
Culturally Responsive Evaluations.” In J. Frechtling (ed.), The 2010 User-Friendly
Handbook for Project Evaluation (pp. 75–96). Arlington, VA: National Science Foundation,
2010.
Gordon, E. W. “Toward an Equitable System of Educational Assessment.” Journal of Negro
Education, 1995, 64(3), 360–372.
Greene, J. C. “Evaluation, Democracy, and Social Change.” In I. F. Shaw, J. C. Greene, and
M. M. Mark (eds.), The SAGE Handbook of Evaluation (pp. 118–140). Thousand Oaks,
CA: Sage, 2006.
Greene, J. C. “Culture and Evaluation from a Transcultural Belvedere.” In S. Hood, R. K.
Hopson, & H. Frierson (eds.), Continuing the Journey to Reposition Culture and Cultural
Context in Evaluation Theory and Practice (pp. 91–107). Greenwich, CT: Information Age
Publishing, Inc., 2015.
Greene, J. C., and Abma, T. A. (eds.) (2001). Responsive Evaluation. New Directions for
Evaluation, 2001, 92.
Greene, J. C., Benjamin, L., and Goodyear, L. (2001). “The Merits of Mixing Methods in
Evaluation.” Evaluation, 2001, 7(1), 25–44.
Hood, S. “Culturally Responsive Performance-Based Assessment: Conceptual and
Psychometric Considerations.” Journal of Negro Education, 1998a, 67(3), 187–196.
Hood, S. “Responsive Evaluation Amistad Style: Perspectives of One African American
Evaluator.” In R. Davis (ed.), Proceedings of the Stake Symposium on Educational Evaluation
(pp. 101–112). Urbana-Champaign, IL: University of Illinois, 1998b.
Hood, S. “Commentary on Deliberative Democratic Evaluation.” In K. Ryan and L.
DeStefano (eds.), Evaluation as a Democratic Process: Promoting Inclusion, Dialogue, and
314 Handbook of Practical Program Evaluation
Deliberation. New Directions for Evaluation,No. 85, (pp. 77–84). San Francisco, CA:
Jossey-Bass, 2000.
Hood, S. “Nobody Knows My Name: In Praise of African American Evaluators Who Were
Responsive.” In J. C. Greene and T. A. Abma (eds.), Responsive Evaluation. New Directions
for Evaluation, 2001, 92, 31–44.
Hood, S. “Evaluation for and by Navajos: A Narrative Case of the Irrelevance of
Globalization.” In K. E. Ryan and J. B. Cousins (eds.), The SAGE International Handbook of
Educational Evaluation (pp. 447–463). Thousand Oaks, CA: Sage, 2009.
Hood, S. “How Will We Know It When We See It? A Critical Friend Perspective of the GEDI
Program and Its Legacy in Evaluation.” In P. Collins and R. K. Hopson (eds.), Building a
New Generation of Culturally Responsive Evaluators: Contributions of the American Evaluation
Association’s Graduate Education Diversity Internship Program, New Directions for Evaluation,
2014, 143, 109–121.
Hood, S., and Hall, M. Relevance of Culture in Evaluation Institute: Implementing and Empirically
Investigating Culturally Responsive Evaluation in Underperforming Schools. National Science
Foundation, Division of Research, Evaluation, and Communications, Directorate for
Education and Human Resources (Award #0438482), 2004.
Hood, S., and Hopson, R. K. “Evaluation Roots Reconsidered: Asa Hilliard, a Fallen Hero
in the ‘Nobody Knows My Name’ Project, and African Educational Excellence.” Review
of Educational Research September, 2008, 78(3), 410–426.
Hopson, R. K. “Reclaiming Knowledge at the Margins: Culturally Responsive Evaluation in
the Current Evaluation Moment.” In K. Ryan and J. B. Cousins (eds.), The SAGE
International Handbook of Educational Evaluation (pp. 429–446). Thousand Oaks, CA:
Sage, 2009.
Hopson, R. K. “Culturally Responsive Evaluation 101.” Workshop presented at the annual
meeting of the Aotearoa/New Zealand Evaluation Association (ANZEA), Auckland,
New Zealand, July 2013.
Hopson, R. K., and Casillas, W. D. “Culture Responsiveness in Applied Research and
Evaluation.” Workshop presented at the Claremont Graduate University, Claremont,
California, August 2014.
Hopson, R. K., and Hood, S. “An Untold Story in Evaluation Roots: Reid Jackson and His
Contributions Toward Culturally Responsive Evaluation at Three-Fourths Century.” In
S. Hood, R. K. Hopson, and H. T. Frierson (eds.), The Role of Culture and Cultural Context:
A Mandate for Inclusion, the Discovery of Truth, and Understanding in Evaluative Theory and
Practice. Greenwich, CT: Information Age Publishing, Inc., 2005.
Hopson, R. K., and Kirkhart, K. E. “Strengthening Evaluation Through Cultural Relevance
and Cultural Competence.” Workshop presented at the American Evaluation
Association/Centers for Disease Control 2012 Summer Evaluation Institute, Atlanta,
Georgia, June 2012.
Hopson, R. K., and Kirkhart, K. E. “Foundations of Culturally Responsive Evaluation
(CRE).” Workshop presented at the annual conference of CREA (Center for Culturally
Responsive Evaluation and Assessment), Oak Brook, Illinois, September 2014.
Jackson, R. E. “The Development and Present Status of Secondary Education for Negroes
in Kentucky.” The Journal of Negro Education, 1935, 4(2), 185–191.
Jackson, R. E. “Status of Education of the Negro in Florida, 1929–1934.” Opportunity, 1936,
14(11), 336–339.
Jackson, R. E. “Alabama County Training Schools.” School Review, 1939, 47, 683–694.
Culturally Responsive Evaluation 315
Jackson, R. E. “An Evaluation of Educational Opportunities for the Negro Adolescent in
Alabama, I.” Journal of Negro Education, 1940a, 9(1), 59–72.
Jackson, R. E. “A Evaluation of Educational Opportunities for the Negro Adolescent in
Alabama, II.” Journal of Negro Education, 1940b, 9(2), 200–207.
Jay, M., Eatmon, D., and Frierson, H. “Cultural Reections Stemming from the Evaluation
of an Undergraduate Research Program.” In S. Hood, R. K. Hopson, and H. T. Frierson
(eds.), The Role of Culture and Cultural Context: A Mandate for Inclusion, the Discovery of
Truth, and Understanding in Evaluative Theory and Practice (pp. 201–216). Greenwich, CT:
Information Age Publishing, Inc., 2005.
Johnson, S. T. “The Importance of Culture for Improving Education and Pedagogy. Journal
of Negro Education, 1998, 67(3), 181–183.
Johnson,E.C.,Kirkhart,K.E.,Madison,A.M.,Noley,G.B.,andSolano-Flores,G.“The
Impact of Narrow Views of Scientic Rigor on Evaluation Practices for
Underrepresented Groups.” In N. L. Smith and P. R. Brandon (eds.), Fundamental Issues
in Evaluation (pp. 197–218). New York: Guilford, 2008.
King, J. A., Nielsen, J. E., and Colby, J. “Lessons for Culturally Competent Evaluation from
the Study of a Multicultural Initiative.” In M. Thompson-Robinson, R. Hopson, and S.
SenGupta (eds.), In Search of Cultural Competence in Evaluation: Toward Principles and
Practices, New Directions for Evaluation, 2004, 102, 67–80.
Kirkhart, K. E. “Seeking Multicultural Validity: A Postcard from the Road.” Evaluation
Practice, 1995, 16(1), 1–12.
Kirkhart, K. E. “Through a Cultural Lens: Reections on Validity and Theory in
Evaluation.” In S. Hood, R. Hopson, and H. Frierson (eds.), The Role of Culture and
Cultural Context: A Mandate for Inclusion, the Discovery of Truth, and Understanding in
Evaluative Theory and Practice (pp. 21–39). Greenwich, CT: Information Age Publishing,
Inc., 2005.
Kirkhart, K. E. “Missing the Mark: Rethinking Validity Threats in Evaluation Practice.”
Paper presented at the 34th Annual Conference of the Eastern Evaluation Research
Society, Absecon, New Jersey, May 2011.
Kirkhart, K. E. “Repositioning Validity.” Plenary panel presented at the Inaugural
Conference of the Center for Culturally Responsive Evaluation and Assessment
(CREA), Chicago, Illinois, April 2013a.
Kirkhart, K. E. “Advancing Considerations of Culture and Validity: Honoring the Key
Evaluation Checklist.” In S. I. Donaldson (ed.), The Future of Evaluation in Society: A
Tribute to Michael Scriven (pp. 129–159). Greenwich, CT: Information Age Publishing,
Inc., 2013b.
Kirkhart, K. E., and Hopson, R. K. “Strengthening Evaluation Through Cultural Relevance
and Cultural Competence.” Workshop presented at the American Evaluation
Association/Centers for Disease Control 2010 Summer Evaluation Institute, Atlanta,
Georgia, June 2010.
Ladson-Billings, G. “But That’s Just Good Teaching: The Case for Culturally Relevant
Pedagogy.” Theory into Practice, 1995a, 34(3), 159–165.
Ladson-Billings, G. “Toward a Theory of Culturally Relevant Pedagogy.” American Education
al Research Journal, 1995b, 32(3), 465–491.
LaFrance, J., and Nichols, R. Indigenous Evaluation Framework: Telling Our Story in Our Place
and Time. Alexandria, VA: American Indian Higher Education Consortium (AIHEC),
2009.
316 Handbook of Practical Program Evaluation
LaFrance, J., and Nichols, R. “Reframing Evaluation: Dening an Indigenous Evaluation
Framework.” The Canadian Journal of Program Evaluation, 2010, 23(2), 13–31.
LaFrance, J., Kirkhart, K. E., and Nichols, R. (2015). “Cultural Views of Validity: A
Conversation.” In S. Hood, R. K. Hopson, and H. Frierson (eds.), Continuing the Journey
to Reposition Culture and Cultural Context in Evaluation Theory and Practice (pp. 49–72).
Greenwich, CT: Information Age Publishing, Inc., 2015.
Lee, C. “How Shall We Sing Our Sacred Song in a Strange Land? The Dilemma of
Double-Consciousness and the Complexities of an African-Centered Pedagogy.” Journal
of Education, 1990, 172(2), 45–61.
Lincoln, Y. S., and Guba, E. G. “But Is It Rigorous? Trustworthiness and Authenticity in
Naturalistic Evaluation.” New Directions for Evaluation, 1986, 30, 73–84.
Madison, A. M. (ed.). Minority Issues in Program Evaluation, New Directions for Program
Evaluation, Number 53. San Francisco, CA: Jossey-Bass, 1992.
Manswell Butty, J. L., Reid, M. D., and LaPoint, V. “A Culturally Responsive Evaluation
Approach Applied to the Talent Development School-to-Career Intervention Program.”
In V. G. Thomas and F. I. Stevens (eds.), Co-Constructing a Contextually Responsive
Evaluation Framework: The Talent Development Model of School Reform, New Directions for
Evaluation, 2004, 101, 37–47.
Mathie, A., and Greene, J. C. “Stakeholder Participation in Evaluation: How Important Is
Diversity?” Evaluation and Program Planning, 1997, 20(3), 279–285.
Messick, S. “Validity.” In R. L. Linn (ed.), Educational Measurement (pp. 13–103).
Washington, DC: American Council on Education and National Council on
Measurement in Education, 1989.
Messick, S. “The Interplay of Evidence and Consequences in the Validation of Performance
Assessments.” Educational Researcher, 1994, 23(2), 13–23.
No Child Left Behind Act of 2001, Pub. L. No. 107–110, §115, Stat. 1425, 2002.
Paipa, K., Cram, F., Kennedy, V., and Pipi, K. “Culturally Responsive Methods for Family
Centered Evaluation.” In S. Hood, R. K. Hopson, and H. Frierson (eds.), Continuing the
Journey to Reposition Culture and Cultural Context in Evaluation Theory and Practice (pp.
313–334). Greenwich, CT: Information Age Publishing, Inc., 2015.
Qualls, A. L. “Culturally Responsive Assessment: Development Strategies and Validity
Issues.” Journal of Negro Education, 1998, 67(3), 296–301.
Ridley, C. R., Tracy, M. L., Pruitt-Stephens, L., Wimsatt, M. K., and Beard, J. “Multicultural
Assessment Validity.” In L. A. Suzuki and J. G. Ponterotto (eds.), Handbook of
Multicultural Assessment: Clinical, Psychological and Educational Applications (3rd ed., pp.
22–33). Hoboken, NJ: Wiley, 2008.
Ryan, K. E., Chandler, M., and Samuels, M. “What Should School-Based Evaluation Look
Like?” Studies in Educational Evaluation, 2007, 33, 197–212.
Samuels, M., and Ryan, K. E. “Grounding Evaluations in Culture.” American Journal of
Evaluation, 2011, 32, 83–98.
Scriven, M. Evaluation Thesaurus (4th ed.). Thousand Oaks, CA: Sage, 1991.
Scriven, M. Key Evaluation Checklist edition of July 25, 2013. Downloaded from
http://michaelscriven.info/images/KEC 7.25.13.pdf, 2013.
Stake, R. E. “Program Evaluation, Particularly Responsive Evaluation.” In G. F. Madaus, M.
S. Scriven, and D. L. Stufebeam (eds.), Evaluation Models: Viewpoints on Education and
Human Services Evaluation (pp. 287–310). Boston, MA: Kluwer-Nijhoff, 1973/1987.
Stake, R. E. Standards-Based and Responsive Evaluation. Thousand Oaks, CA: Sage, 2004.
Culturally Responsive Evaluation 317
Thomas, V. G. “Building a Contextually Responsive Evaluation Framework: Lessons from
Working with Urban School Interventions.” In V. G. Thomas and F. I. Stevens (eds.),
Co-Constructing a Contextually Responsive Evaluation Framework: The Talent Development
Model of School Reform, New Directions for Evaluation, 2004, 101, 3–23.
Yarbrough, D. B., Shulha, L. M., Hopson, R. K., and Caruthers, F. A. The Program Evaluation
Standards: A Guide for Evaluators and Evaluation Users (3rd ed.). Thousand Oaks, CA:
Sage, 2011.