Conference PaperPDF Available

Reclaiming educational autonomy and minimizing measurement disjuncture through a culturally specific assessment development process

Authors:
  • Sul & Associates International

Abstract and Figures

Educational research findings often rely on evidence obtained through the use of standardized assessment instruments. This presentation draws a distinction between Western and Indigenous worldviews in noting that all assessment instruments are constructed within a particular worldview. Measurement disjuncture (Sul, 2019) refers to the misalignment that occurs when elements of an instrument development process from one worldview are applied to the instrument development process of another worldview. In manners that are both qualitative and quantitative, measurement disjuncture negatively affects the establishment of measurement validity. Further, as a result of measurement disjuncture, researchers are less likely to acknowledge that educational programming has had an impact when, in fact, it may have (Type II error). This presentation will focus on the development of culturally specific assessments, the Papakū Makawalu Competency Assessment and the Dibishgaademgak Anishinaabemowin (language) assessment, as a means to minimize measurement disjuncture.
Content may be subject to copyright.
© 2019 by Sul & Associates International.
Sul & Associates International is based in Petaluma, Sonoma County, California.
Please visit www.sulandassociates.com for more information.
Reclaiming educational autonomy and minimizing measurement disjuncture through a culturally
specific assessment development process
David A. Sul
Sul & Associates International
March 28, 2019
Paper presented at the Culturally Relevant Assessment and Evaluation Conference, Chicago, IL.
Contents
Background ..................................................................................................................................... 1
Purpose of the study ........................................................................................................................ 3
Theoretical framework .................................................................................................................... 5
Significance ................................................................................................................................... 13
Pending review of the literature .................................................................................................... 13
Pending research questions ........................................................................................................... 15
Conclusion .................................................................................................................................... 16
References ..................................................................................................................................... 18
Tables
Table 1. Progression of terminologies towards culturally specific assessment. ........................... 11
Table 2. Prior research conducted on Indigenous language assessments ..................................... 14
Figures
Figure 1. Location of all Anishinaabe Reservations/Reserves in North America, with diffusion
rings about communities speaking an Anishinaabe language. Cities with Anishinaabe population
also shown (Lippert, 2007). ............................................................................................................ 2
Figure 2. Alignment within and between the conceptual and operational elements of an
instrument development process. .................................................................................................... 4
Figure 3. Assessment applied across Western and Indigenous worldviews. ................................. 5
Figure 4. The impact of measurement disjuncture on the interpretation of quantitative research
design conclusions. ......................................................................................................................... 8
RUNNING HEAD: EDUCATIONAL AUTONOMY VIA CULTURALLY SPECIFIC
ASSESSMENT
© 2019 by Sul & Associates International.
Sul & Associates International is based in Petaluma, Sonoma County, California.
Please visit www.sulandassociates.com for more information.
Background
For generations, Indigenous peoples have utilized performance-based assessment
practices to determine how individuals could best contribute to the society (Bordeaux, 1995).
Adults observed children exhibiting varying degrees of skill in tasks such as “hunting, running,
consensus building, healing, and spiritual leadership” and those who demonstrated superior
performance were the ones who later led hunting parties, provided spiritual guidance, served as
orators for the people, and performed other necessary tasks for the group. To this day,
observation, assessment, and feedback practices remain present within Indigenous communities
and are used by parents, elders, teachers, master craftspeople, and ceremonial leaders. These
practices can play a critical role in language revitalization efforts.
Modern language revitalization efforts within North America arose from the voice of
Indigenous people seeking to preserve their languages, cultures, and ways of life. With support
from academic scholars concerned about the loss of Indigenous languages (Krauss, 1992),
revitalization efforts have become broad and expansive (Reyhner & Lockard, 2009). These
efforts revolve around the teaching and learning of these endangered languages and seek to
address the continued consequences of colonization, the effects of the residential school system,
and other inequities of history.
Anishinaabemowin, an Algonquian language, is spoken widely throughout Canada by
approximately 20,000 Anishinaabe people (Statistics Canada, 2017). In Canada,
Anishinaabemowin communities are found in southwestern Quebec, Ontario, southern Manitoba
and parts of southern Saskatchewan. In the United States, Anishinaabemowin communities exist
along the northern border from Montana to Michigan and as far south as Oklahoma (see Figure 1
EDUCATIONAL AUTONOMY VIA CULTURALLY SPECIFIC ASSESSMENT 2
below). Considered “endangered” in the United States, there are an estimated seven hundred
speakers of Anishinaabemowin across the United States (Hermes, Bang, & Marin, 2012).
Figure 1. Location of all Anishinaabe Reservations/Reserves in North America, with diffusion
rings about communities speaking an Anishinaabe language. Cities with Anishinaabe population
also shown (Lippert, 2007).
This research supports an Anishinaabemowin language assessment initiative sponsored
by Kenjgewin Teg, an educational institution located in M’Chigeeng on Mnidoo Mnising
(Manitoulin Island), Ontario, Canada. Kenjgewin Teg is governed by the United Chiefs and
Councils of Mnidoo Mnising who represent eight First Nations: Sagamok Anishnawbek First
Nation, Sheguiandah First Nation, Aundeck Omni Kaning First Nation, M’Chigeeng First
Nation, Zhiibaahaasing First Nation, Sheshegwaning First Nation, Whitefish River First Nation,
the Mamawmatawa Holistic Education Centre, and the Constance Lake First Nation. On October
14, 2011, these eight First Nations established the Anishinabek Language Declaration that
asserted their right to: “revitalize, use, develop and transmit to future generations their histories,
languages, oral traditions, philosophies, writing systems and literatures, and to designate and
EDUCATIONAL AUTONOMY VIA CULTURALLY SPECIFIC ASSESSMENT 3
retain their own names for communities, places and persons.” Included within the Anishnaabek
Language Declaration was the expectation that employees of Kenjgewin Teg will “provide all
work and service functions in their ancestral language by 2030" (United Chiefs and Councils of
Mnidoo Mnising, 2011). The eight First Nations sought the design and development an
Anishinaabemowin language assessment that could be used to support the Kenjgewin Teg in
meeting this long-term goal. The Kantaa-Anishinaabemi language assessment was created in
2014 to determine Kenjgewin Teg employees’ proficiency in Anishinaabemowin. It was based
on the Common European Framework of Reference for Languages (2001) and the 2012
proficiency guidelines established by the American Council on the Teaching of Foreign
Languages (2012).
In December 2017, Kenjgewin Teg contracted with the author to update the assessment to
align more with Anishinaabek principles of learning and overall worldview. The revised
assessment, known as the Dibishgaademgak Anishinaabemowin (measuring Anishinaabemowin)
is expected to address language learning domains that are deemed critical for Anishinaabemowin
language learning and provide a better understanding of the performance of language learners
than that obtained through the existing language assessment.
Purpose of the study
As with any other product of human activity, tests are cultural artifacts (Solano-Flores,
2011, p. 3) existing within a given worldview. Beyond that, the elements of the instrument
development process are prescribed by the cultural worldview under which they are presented.
The cultural validity of tests is the degree to which they address sociocultural influences such as
values, beliefs, experiences and epistemologies inherent within cultures as well as the
socioeconomic conditions under which cultural groups exist (Solano-Flores & Nelson-Barber,
EDUCATIONAL AUTONOMY VIA CULTURALLY SPECIFIC ASSESSMENT 4
2001). Assessment instruments are influenced by the worldview within which these structural
influences and conditions reside. Whether examined from a broad or narrow perspective, such
elements exist within a worldview that guides and influences the construction of educational
assessment instruments. From a closer perspective, educational instrument developers seek a
formal structure that maintains internal alignment both within and between the conceptual and
the operational elements of an instrument development process. The conceptual-operational
elements of the construct-framework, dimensions-components, elements-items, and stages-levels
components and their respective alignment orientations are represented in the figure below.
Figure 2. Alignment within and between the conceptual and operational elements of an
instrument development process.
From a broad perspective, educational assessment developers seek a formal structure that
maintains external alignment to such educational setting elements as the constructs of
knowledge, learning expectations, the educational framework, adopted curriculum, methods of
instruction, and forms of assessment. An issue of validity arises when assessment instruments are
developed within one worldview and applied inside of another. This issue is exacerbated when
EDUCATIONAL AUTONOMY VIA CULTURALLY SPECIFIC ASSESSMENT 5
assessment instruments are developed within Western worldviews and applied within an
Indigenous world view. This is summarized in the figure below.
Figure 3. Assessment applied across Western and Indigenous worldviews.
Proposed research project
The current study will focus on an instrument development process that maintains
internal and external alignment by being grounded within a specific worldview. It approaches
measurement from a stages- and performance-based perspective that aspires to align with
Indigenous notions of the transmission of knowledge. The purpose of this study is to determine
whether and to what degree the proposed assessment development process can produce an
Anishinaabemowin assessment that will provide a better understanding of the performance level
of Anishinaabemowin learners than that provided through the existing Kantaa-Anishinaabemi
assessment.
Theoretical framework
An instrument development process is comprised of an array of structural components
and is advanced through thoughtful consideration and decision-making. In the case where
EDUCATIONAL AUTONOMY VIA CULTURALLY SPECIFIC ASSESSMENT 6
assessment instruments are developed within Western worldviews and applied within an
Indigenous world view, since each element, consideration, and decision are influenced by the
worldview in which it exists, a multitude of opportunities exist for misalignment between the
two worldviews. This led to the posing of three fundamental questions about this form of
misalignment that exists within instrument development processes: What do we call this? Why is
this a problem? What do we do about it? This paper is offered in response to these three
questions.
What do we call this?
Measurement validity refers to the degree to which evidence and theory support the
interpretations of test scores for proposed uses of tests (American Educational Research
Association, American Psychological Association, & National Council on Measurement in
Education, 2014). Key elements of this definition are addressed by the terms “evidence,”
“theory,” “interpretations,” “scores,” “uses,” and “tests.” The meaning of these terms within the
very definition of measurement validity is grounded in and influenced by the worldview under
which the instrument development occurs.
Misalignment that is grounded in cultural and linguistic differences has been referred to
as “disjuncture” (Appadurai, 1996; Meek, 2010; Wyman, et al., 2010) or “discontinuity”
(Brown-Jeffy & Cooper, 2011; Bougie, Wright, and Taylor, 2003; Edwards, 2006; Meek, 2007).
Cultural discontinuity in school settings has been defined conceptually as “a school-based
behavioral process where the cultural value-based learning preferences and practices of many
ethnic minority students—those typically originating from home or parental socialization
activities—are discontinued at school” (Tyler et al., 2008). The cultural discontinuity hypothesis
posits that culturally based differences in the communication styles of minority students’ home
EDUCATIONAL AUTONOMY VIA CULTURALLY SPECIFIC ASSESSMENT 7
and the Anglo culture of the school lead to conflicts, misunderstandings, and, ultimately, failure
for those students (Ledlow, 1992). Cultural discontinuity arises for students when their personal
values clash with the ideals that shape their school system (Wiesner, 2006). Ladson-Billings
(1995) described the “discontinuity” problem as the gap between what students experience at
home and what they experience at school with respect to their interactions of speech and
language with teachers.
Measurement disjuncture is defined here as the misalignment that occurs when elements
of an instrument development process from one worldview are applied to the instrument
development process of another worldview. While measurement disjunctures can occur across
worldviews, environments or settings, this research will center on the measurement disjuncture
that exists across Western and Indigenous worldviews.
Why is this a problem?
When assessment instruments are developed within a Western worldview and are applied
within an Indigenous setting, measurement disjuncture results. Measurement disjuncture affects
the establishment of measurement validity and hence, the inferences made based on the scores
derived from such assessments. This is primarily due to the introduction of measurement error
caused by the misalignment. Measurement error introduced by measurement disjuncture
negatively affects the conclusions drawn from quantitative research designs. The figure below
presents a typical analysis of variance (ANOVA) table used to interpret the differences between
groups in a controlled quantitative study. Three elements of the table that are influenced by
measurement disjuncture are noted. In reference to the figure below, when the error term (a)
increases, the mean square error (b) increases causing the value of the F statistic (c) to decrease.
EDUCATIONAL AUTONOMY VIA CULTURALLY SPECIFIC ASSESSMENT 8
With a smaller than expected F statistic, researchers are less likely to acknowledge that the
treatment has had an impact when, in fact, it has. This represents a Type II error.
Figure 4. The impact of measurement disjuncture on the interpretation of quantitative
research design conclusions.
When measurement disjuncture exists within assessment instruments used by educational
researchers, the influence of interventions may end up being undervalued. In practical terms,
researchers evaluating programs to improve educational outcomes for Indigenous people through
the application of assessment instruments developed within a Western worldview may end up
undervaluing the influence of such programs.
What do we do about it?
While the term measurement disjuncture is presented here, attempts to both describe and
address the disjuncture within broader educational environments are not new. Au and Jordan
described as “culturally appropriate” the incorporation of “talk story” into a program of reading
instruction for Native Hawaiian students that improved upon expected scores on standardized
reading tests (Au & Jordan, 1981). Mohatt and Erickson (Mohatt & Erickson, 1981) used the
term “culturally congruent” to describe teachers’ use of interaction patterns that simulated Native
American students’ home cultural patterns to produce improved academic performance. Jordan
EDUCATIONAL AUTONOMY VIA CULTURALLY SPECIFIC ASSESSMENT 9
(1985) defined educational practices as “culturally compatible” when the culture of students is
used as a guide in choosing aspects of the educational program to maximize academically
desired behaviors and minimize undesired behaviors. Researchers beginning in the 1980s used
the term “culturally responsive education” to describe the language interactions of teachers with
linguistically diverse and Native American students (Cazden & Leggett, 1981; Erickson &
Mohatt, 1982). Erickson and Mohatt (1982) suggested their notion of culturally responsive
teaching could be seen as a beginning step for bridging the gap between home and school.
Ladson-Billings (1995) claimed the term culturally responsive represented a more expansive,
dynamic, and synergistic relationship between the culture of the school and that of the home and
greater community.
Ladson-Billings (1995) conducted a significant qualitative study on the teaching methods
of teachers who demonstrated consistent academic success with African American students. Her
work launched the movement towards the acknowledgement and identification of a “culturally
relevant pedagogy.” Ladson-Billings (1995), grounded in Black feminist thought, introduced the
theory of “culturally relevant pedagogy” to emphasize the significance of teaching to and
through the cultural strengths of ethnically diverse students. Ladson-Billings and Jordan argued
for the use of culturally relevant pedagogy to engage actively and motivate students from
ethnically diverse backgrounds to improve their academic achievement (Jordan, 1985; Ladson-
Billings, 1995). Ladson-Billings (1995) established three criteria for a culturally relevant
pedagogy that could be used to address the “discontinuity” problem: (a) an ability to develop
students academically; (b) a willingness to nurture and support cultural competence to help
students to maintain their cultural integrity while succeeding academically; and (c) the
development of a sociopolitical or critical consciousness. In a culturally relevant classroom, a
EDUCATIONAL AUTONOMY VIA CULTURALLY SPECIFIC ASSESSMENT 10
child’s culture is not only acknowledged but seen as a source of strength that can be utilized to
attain academic success. Sociopolitical consciousness has been described as an individual’s
ability to critically analyze the political, economic, and social forces shaping society and one’s
status in it (Seider et al., 2018). For the last definitional criterion, Ladson-Billings (1995)
borrowed from Freire and acknowledged that students must develop a broader sociopolitical
consciousness and the skills to critique the cultural norms, values, mores, and institutions that
produce and maintain social inequities (Freire, 1970). The development of a sociopolitical or
critical consciousness within students allows them to acknowledge and act on historical
circumstances that affect their current reality (Freire, 1970; Ladson-Billings, 1995).
Researchers in the field of program evaluation began to utilize the term “responsive
evaluation” in the early 1970s in reference to a focus on issues of practical importance to
program managers and developers (Stake, 2011). Stake (1973) sought to remove the emphasis
on static program objectives developed by those furthest from the delivery of program services
and stressed the importance of being responsive to situational realities in the management of
programs and to the reactions, concerns, and issues of participants. This represented a dramatic
departure from the emphasis on the use of evaluation plans that relied on preconceived notions of
program expectations. Stake (1973) believed that the ultimate test of the validity of an evaluation
is the extent to which it increases the audience’s understanding of the program. Stake’s (1973)
work led to the stream of responsive evaluation research and practices that exist today.
Drawing upon the lineage of research in responsive evaluation and culturally relevant
pedagogy, Hood (1998) argued that student learning is more effectively assessed through the use
of assessment approaches that are culturally responsive. Combining the ideas of Ladson-Billings
(1995) and Stake (1973), Hood (1998) promoted the development of “culturally responsive”
EDUCATIONAL AUTONOMY VIA CULTURALLY SPECIFIC ASSESSMENT 11
performance-based assessments as a means of achieving equity for students of color. Hood
(1998) noted that there were to be challenges and difficulties in the development of both
performance tasks and scoring criteria that would be “responsive to cultural differences and
adequately assess the content-related skills that are the focus of the assessment.”
“Culturally specific assessment” defined herein, represents a projection of Hood’s (1998)
culturally responsive assessment onto a named worldview through the addition of an additional
criterion: the assessment development process functions within a system of knowledge that exists
within a named worldview. Thus, the formal definition of culturally specific assessment that will
be utilized throughout this document is (a) assessment that supports the academic development
of students; (b) is inclusive of a willingness to nurture and support cultural competence; (c) aims
to support the development of a sociopolitical or critical consciousness within students; (d) is
focused on constructs and measures of importance to educational practitioners and other key
stakeholders; and (e) functions within a system of knowledge that exists within a named
worldview. The table below summarizes the progression of the terminologies towards this
definition of culturally specific assessment.
Table 1. Progression of terminologies towards culturally specific assessment.
Culturally
Description
Proponents
Responsive evaluation
Evaluation focuses is on issues of practical importance
to program managers and developers
(Stake, 1973)
Culturally appropriate
instruction
Early attempt to describe efforts to address the
discontinuity problem
(Au & Jordan, 1981)
Culturally congruent
instruction
Teachersuse of interaction patterns that simulated the
Native American studentshome cultural patterns
(Mohatt & Erickson, 1981)
Culturally responsive
education
Involves language interactions of teachers with
linguistically diverse students
(Cazden & Leggett, 1981;
Jordan, 1985; Mohatt &
Erickson, 1981)
EDUCATIONAL AUTONOMY VIA CULTURALLY SPECIFIC ASSESSMENT 12
Culturally
Description
Proponents
Culturally compatible
instruction
Culture of students is used as a guide in choosing
aspects of the educational program
(Vogt, Jordan, & Tharp, 1987)
Culturally relevant
pedagogy
(1) an ability to develop students academically; (2) a
willingness to nurture and support cultural competence;
and (3) the development of a sociopolitical or critical
consciousness.
(Ladson-Billings, 1995)
Culturally responsive
assessment
Supports the academic development of students;
inclusive of a willingness to nurture and support cultural
competence; aims to support the development of a
sociopolitical or critical consciousness; Focused on
constructs and measures of importance to educational
practitioners and other key stakeholders
(Hood, 1998)
Culturally specific
assessment
Supports the academic development of students;
Inclusive of a willingness to nurture and support cultural
competence; Aims to support the development of a
sociopolitical or critical consciousness; Focused on
constructs and measures of importance to educational
practitioners and other key stakeholders; Functions
within a system of knowledge that exists within a
named worldview
Sul (n.d.)
Throughout the transition of terminologies from culturally appropriate instruction to
culturally responsive assessment, researchers have focused their attention on the improvement of
academic performance of learners within educational settings or environments that are grounded
in the worldview of the dominant culture. The transition toward culturally specific assessment
described here represents an attempt to do the same within the worldview of cultures functioning
within a named worldview.
Having arrived at a formal definition of culturally specific assessment, it is offered here
as a potential solution to the problem of measurement disjuncture. In order to determine whether
the minimization of measurement disjuncture can be achieved through the employment of
culturally specific assessments, educational environments that meet the criteria for culturally
EDUCATIONAL AUTONOMY VIA CULTURALLY SPECIFIC ASSESSMENT 13
specific assessment are sought. Such environments do exist. Indigenous knowledge, inclusive of
both language and cultural knowledge and wisdom, are being promoted throughout Aotearoa
(New Zealand), Hawaiʻi, tribal communities within North America, and First Nations
communities within Canada.
Significance
Through this research, (a) a critical assessment-development process is sought (b) that
will support educational efforts of Indigenous people while (c) minimizing measurement
disjuncture thus (d) increasing the measurement validity of the resultant assessment scores.
The proposed assessment-development approach presents a radical departure from most
other instrument development processes. For example, many assessment developers rely on
factor analytic models to arrive at item classifications. The proposed process re-orients the
instrument development process towards one in which Indigenous stakeholders define domains
of knowledge and identify stages of learning within these domains. The elements of the
instrument development process are based on Indigenous stakeholders’ experiences and beliefs
about their own language, culture and knowledge systems.
Pending review of the literature
Language revitalization has been described as the set of actions that lead to the increase
of the numbers of speakers and domains of language use. This includes the actions of reversing
language shift (Fishman, 1991). Although a great amount of research on Indigenous language
revitalization exists, there is a limited research on the development of curricula, instructional
practices, and, in particular, assessments that operate from within an Indigenous cultural
worldview. Fifteen studies that focus on assessments developed in support of language
EDUCATIONAL AUTONOMY VIA CULTURALLY SPECIFIC ASSESSMENT 14
revitalization efforts have been identified. Each will be examined during the next stage of
research, the review of the literature. The degree to which each of the assessments described
adhere to the definitional components of culturally specific assessments described above will be
the focus of the review.
Table 2. Prior research conducted on Indigenous language assessments
Language assessment
1. Onön:dowaga: (Seneca)
2. Nsyilxcn and Interior Salish
3. Inuktitut and English Language Screening Tool
4. ANA ŌLELO Hawaiian language
5. American Indian Montana
6. Mohawk, New York State
7. Navajo Nation
8. Keresan communities, New Mexico
9. Hawaiian Oral Language Assessment (H-OLA)
10. Kaiapuni Assessment of Educational Outcomes
(KĀEO)
11. Kaiaka Reo Māori Language Proficiency in Writing
12. Cherokee Preschool Immersion Language
Assessment
13. Cree Language Assessment
14. Pathways to Creating Speakers of
Onkwehonwehneha at Six Nations
15. Assessing student profieciency in the Mohawk
language: “The Kanienʻkéha proficiency assessment”
EDUCATIONAL AUTONOMY VIA CULTURALLY SPECIFIC ASSESSMENT 15
Pending research questions
The current study will focus on an instrument development process that is grounded
within an Anishinaabek worldview. As such, a meta-analytic approach is applied in the
construction of the research questions that surround the proposed instrument development
process. The object of the research is not the individuals learning Anishinaabemowin. Rather, the
research will focus on the development of the Dibishgaademgak Anishinaabemowin, the
assessment instrument that will be the result of the proposed instrument development process.
For this reason, the pending research questions will revolve around (a) whether and to what
degree the Dibishgaademgak Anishinaabemowin assessment can be classified as culturally
specific; and (b) whether and to what degree the impact on measurement disjuncture is
minimized through the use of the Dibishgaademgak Anishinaabemowin assessment.
Culture loading refers to the degree to which a test is culturally specific (Hitchcock et al.,
2005; Reynolds & Ramsay, 2003). As the degree of culture loading within an instrument
increases, the elements of the assessment development process will carry enhanced meaning for
the target population. This also increases the degree of cultural bias when the instrument is
administered to people from other cultures. Cultural loading can be viewed along a continuum,
where an instrument may narrowly apply to a specific population (high cultural loading) or, on
the other end, may only generally touch upon specific cultural concerns. The degree of cultural
loading within an instrument development process will be approximated based on the degree to
which the assessments adhere to the five criteria described above.
The minimization of measurement disjuncture will be examined through qualitative
interviews with the developers and users of both the Kantaa-Anishinaabemi and
EDUCATIONAL AUTONOMY VIA CULTURALLY SPECIFIC ASSESSMENT 16
Dibishgaademgak Anishinaabemowin assessments. Interviews will focus on the perceived
validity of the assessments and will center on questions such as: Are the domains of
Anishinaabemowin learning being addressed and represented properly within the assessment?
Are the performance tasks aligned to the principles of Anishinaabemowin learning? Are the
performance task rating levels aligned to the principles of Anishinaabemowin learning?
Conclusion
As with other educational assessments, the Dibishgaademgak Anishinaabemowin exists
within a self-determined worldview. The proposed culturally specific instrument development
process is not a significant directional shift. Rather, the group identifying the system of
knowledge and naming the worldview is the significant directional shift. This research will
continue with a formal description and validation of the methodology used to develop the
Dibishgaademgak Anishinaabemowin. A long-term research agenda will focus on defining,
validating, and utilizing a culturally specific assessment-development process in support of
educational efforts of Indigenous people while minimizing measurement disjuncture thereby
increasing the measurement validity of the resultant assessment scores. To support this broader
agenda, outreach efforts to Indigenous communities seeking to develop culturally specific
assessments are being conducted.
The developers of the Dibishgaademgak Anishinaabemowin aspire to approach
measurement from a stages- and performance-based perspective that aligns well with
Anishinaabek notions of knowledge attainment. The clear articulation of the domains and stages
of learning from an Anishinaabek worldview will be by master speakers and teachers of
Anishinaabemowin. The following represent formal declarations of educational and assessment
EDUCATIONAL AUTONOMY VIA CULTURALLY SPECIFIC ASSESSMENT 17
autonomy that will ground the continued development of the Dibishgaademgak
Anishinaabemowin.
We assert the right to educate ourselves within our own worldview.
This is the declaration of our educational autonomy.
We assert the right to develop assessments within our own worldview.
This is the declaration of our assessment autonomy.
EDUCATIONAL AUTONOMY VIA CULTURALLY SPECIFIC ASSESSMENT 18
References
American Council for Teaching Foreign Languages. (2012). ACTFL proficiency guidelines 2012.
Alexandria, VA.
American Educational Research Association, American Psychological Association, & National
Council on Measurement in Education. (2014). Standards for educational and
psychological testing. Washington, DC: American Educational Research Association.
Au, K. H.-P., & Jordan, C. (1981). Teaching reading to Hawaiian children: Finding a culturally
appropriate solution. In H. Trueba, G. Guthrie, & K. H.-P. Au (Eds.), Culture and the
bilingual classroom: Studies in classroom ethnography (pp. 216–242). Rowley, MA:
Newbury House.
Bordeaux, R. (1995). Assessment for American Indian and Alaska native learners. In ERIC
Digest (Las Cruces, N.M.): EDO-RC-95-6. Charleston, WV: ERIC Clearinghouse on
Rural Education and Small Schools, Appalachia Educational Laboratory. edsgpr.
Borgia, M. (2009). Modifying assessment tools for Ganöhsesge:kha: Hë:nödeyë:stha a Seneca
culture-language school. In J. A. Reyhner & L. Lockard (Eds.), Indigenous Language
Revitalization: Encouragement, guidance & lessons learned (pp. 191–210). Flagstaff,
AZ: Northern Arizona University.
Cazden, C., & Leggett, E. (1981). Culturally responsive education: Recommendations for
achieving Lau remedies II. U.S. Department of Health, Education & Welfare, National
Institute of Education.
Council of Europe. (2001). Common European framework of reference for languages: Learning,
teaching, assessment. Press Syndicate of the University of Cambridge.
Dench, C., Cleave, P. L., Tagak, J., & Beddard, J. (2011). The development of an Inuktitut and
English language screening tool in Nunavut. Canadian Journal of Speech-Language
Pathology and Audiology, 35(2), 168–177.
Dupuis, J., & Abrams, E. (2017). Student science achievement and the integration of Indigenous
knowledge on standardized tests. Cultural Studies of Science Education, 12(3), 581–604.
https://doi.org/10.1007/s11422-016-9728-6
Edmonds, C. A. (2008). The reliability and validity of the māori language proficiency in writing
test: Kaiaka reo year eight (University of Hawai’i at Hilo). Retrieved from
https://search.proquest.com/docview/304835695
Erickson, F., & Mohatt, C. (1982). Cultural organization and participation structures in two
classrooms of Indian students. In Doing the ethnography of schooling (pp. 131–174).
New York, NY: Holt, Rinehart & Winston.
Freire, P. (1970). Pedagogy of the oppressed. New York, NY: Continuum.
EDUCATIONAL AUTONOMY VIA CULTURALLY SPECIFIC ASSESSMENT 19
Green, J. (2017). Pathways to creating Onkwehonwehnéha speakers at Six Nations of the Grand
River territory. Six Nations Polytechnic.
Hermes, M., Bang, M., & Marin, A. (2012). Designing Indigenous language revitalization.
Harvard Educational Review, 82, 381–402.
Hitchcock, J. H., Nastasi, B. K., Dai, D. Y., Newman, J., Jayasena, A., Bernstein-Moore, R., …
Varjas, K. (2005). Illustrating a mixed-method approach for validating culturally specific
constructs. Journal of School Psychology, 43(3), 259–278.
https://doi.org/10.1016/j.jsp.2005.04.007
Hood, S. (1998). Culturally responsive performance-based assessment: conceptual and
psychometric considerations. The Journal of Negro Education, 67(3), 187.
Housman, A., Dameg, K., Kobashigawa, M., & Brown, J. D. (2011). Report on the Hawaiian
oral language assessment (H-OLA) project. Second Language Studies, 29(2), 1–59.
Johnson, S. M. K. (2017). Breathing life into new speakers: Nsyilxcn and Tlingit sequenced
curriculum, direct acquisition, and assessments. Canadian Modern Language Review,
73(2), 109–132.
Jordan, C. (1985). Translating culture: From ethnographic information to educational program.
Anthropology & Education Quarterly, 16, 105–123.
Kahakalau, K. (2017). Developing an Indigenous proficiency scale. Cogent Education, 4(1).
Krauss, M. (1992). The world’s languages in crisis. Language, 68(1), 4–10.
Ladson-Billings, G. (1995). Toward a theory of culturally relevant pedagogy. American
Educational Research Journal, 32, 465.
Learning and Teaching Resources Branch Alberta Education. (2008). Cree language and
culture: 9-year program classroom assessment materials, grade 4. Edmonton, AB:
Alberta Education.
Ledlow, S. (1992). Is Cultural Discontinuity an Adequate Explanation for Dropping Out?
Journal of American Indian Education, 31(3), 21–36.
Lippert, C. J. (2007). Location of all Anishinaabe reservations/reserves in North America, with
diffusion rings about communities speaking an Anishinaabe language. Retrieved from
https://upload.wikimedia.org/wikipedia/commons/c/c1/Anishinaabewaki.jpg
McGroarty, M., Beck, A., & Butler, F. A. (1995). Policy issues in assessing Indigenous
languages: A Navajo case. Applied Linguistics, 16(3), 323–343.
Mohatt, G., & Erickson, F. (1981). Cultural differences in teaching styles in an Odawa school: A
sociolinguistic approach. Culture and the Bilingual Classroom: Studies in Classroom
Ethnography.
EDUCATIONAL AUTONOMY VIA CULTURALLY SPECIFIC ASSESSMENT 20
Montour, B. M. (2012). The Kanien’ke ́ha Proficiency Assessment. In First Nations lifelong
learning assessment report (pp. 22–27). Ontario, Canada: Indigenous Education
Coalition.
Peter, L., & Hirata-Edds, T. E. (2006). Using assessment to inform instruction in Cherokee
language revitalisation. International Journal of Bilingual Education and Bilingualism,
9(5), 643–658.
Reyhner, J. A., & Lockard, L. (Eds.). (2009). Indigenous language revitalization:
Encouragement, guidance & lessons learned. Flagstaff, AZ: Northern Arizona
University.
Reynolds, C. R., & Ramsay, M. C. (2003). Bias in Psychological Assessment: An Empirical
Review and Recommendations. In I. B. Weiner (Ed.), Handbook of Psychology.
https://doi.org/10.1002/0471264385.wei1004
Romaine, S. (2006). Planning for the survival of linguistic diversity. Language Policy, 5, 443–
475.
Seider, S., Graves, D., El-Amin, A., Soutter, M., Tamerat, J., Jennett, P., … Johannsen, J. (2018).
Developing sociopolitical consciousness of race and social class inequality in adolescents
attending progressive and no excuses urban secondary schools. Applied Developmental
Science, 22, 169–187.
Solano-Flores, G. (2011). Assessing the cultural validity of assessment practices: An
introduction. In M. del R. Basterra, E. Trumbull, & G. Solano-Flores (Eds.), Cultural
validity in assessment: Addressing linguistic and cultural diversity (pp. 3–21). New York,
NY: Routledge.
Solano-Flores, G., & Nelson-Barber, S. (2001). On the cultural validity of science assessments.
Journal of Research in Science Teaching, 38(5), 553–573.
https://doi.org/10.1002/tea.1018
Stake, R. (1973). Program evaluation particularly responsive evaluation. Presented at the New
Trends in Evaluation, Goteborg, Sweden.
Stake, R. (2011). Program evaluation particularly responsive evaluation. Journal of
MultiDisciplinary Evaluation, 7(15), 180–201.
Statistics Canada. (2017). Aboriginal languages in Canada, 2016 Census of population.
Retrieved from Aboriginal languages in Canada, 2016 Census of Population website:
https://www150.statcan.gc.ca/n1/pub/11-627-m/11-627-m2017035-eng.htm
Steele, C. (1978). Kan-ienkehaka Tiontenientenstakwa (Mohawk guidebook). Activities and
games. Mohawk Language Consortium.
EDUCATIONAL AUTONOMY VIA CULTURALLY SPECIFIC ASSESSMENT 21
Tyler, K. M., Uqdah, A. L., Dillihunt, M. L., Beatty-Hazelbaker, R., Conner, T., Gadson, N., …
Stevens, R. (2008). Cultural Discontinuity: Toward a Quantitative Investigation of a
Major Hypothesis in Education. Educational Researcher, 37(5), 280–297.
United Chiefs and Councils of Mnidoo Mnising. (2011). Anishinabek language declaration.
Retrieved from Anishinabek Language Declaration website:
http://www.uccmm.ca/anishinabek-language-declaration.html
Vogt, L. A., Jordan, C., & Tharp, R. G. (1987). Explaining school failure, producing school
success: Two cases. Anthropology & Education Quarterly, 18(4,), 276–286.
Westby, C., Moore, C., & Roman, R. (2002). Reinventing the Enemy’s Language: Developing
Narratives in Native American Children. Linguistics and Education, 13(2), 235–269.
https://doi.org/10.1016/S0898-5898(01)00063-8
Wiesner, J. L. (2006). School climate interventions for Native American students: Minimizing
cultural discontinuity in public schools. University of Wisconsin-Stout.
... Measurement disjuncture is the misalignment that occurs when elements of an instrument-development process from one worldview are applied to the instrument-development process of another worldview (Sul, 2019). Although measurement disjunctures can occur across multiple worldviews, this research will center on the measurement disjuncture that exists across Western and Indigenous worldviews. ...
... Hood (1998) noted that there were to be challenges and difficulties in the development of both performance tasks and scoring criteria that would be "responsive to cultural differences and adequately assess the contentrelated skills that are the focus of the assessment." Culturally specific assessment (Sul, 2019) represents an extension of Hood's (1998) culturally responsive assessment onto a named worldview through the addition of an additional criterion: the assessment development process functions within a system of knowledge that exists within a named worldview. Thus, the formal definition of culturally specific assessment that will be utilized throughout this document is (a) assessment that supports the academic development of students; (b) is inclusive of a willingness to nurture and support cultural competence; (c) aims to support the development of a sociopolitical or critical consciousness within students; (d) is focused on constructs and measures of importance to educational practitioners and other key stakeholders; and (e) functions within a MINIMIZING MEASUREMENT DISJUNCTURE 13 system of knowledge that exists within a named worldview. ...
... ( Ladson-Billings, 1995) Culturally responsive assessment Supports the academic development of students; inclusive of a willingness to nurture and support cultural competence; aims to support the development of a sociopolitical or critical consciousness; Focused on constructs and measures of importance to educational practitioners and other key stakeholders (Hood, 1998) Culturally specific assessment Supports the academic development of students; Inclusive of a willingness to nurture and support cultural competence; Aims to support the development of a sociopolitical or critical consciousness; Focused on constructs and measures of importance to educational practitioners and other key stakeholders; Functions within a system of knowledge that exists within a named worldview (Sul, 2019) Throughout the transition of terminologies from culturally appropriate instruction to culturally responsive assessment, researchers have focused their attention on the improvement of academic performance of learners within educational settings or environments that are grounded in the worldview of the dominant culture. The transition toward culturally specific assessment described here represents an attempt to do the same within the worldview of cultures functioning within a named worldview. ...
Conference Paper
Full-text available
Educational research findings often rely on evidence obtained through the use of standardized assessment instruments. This presentation draws a distinction between Western and Indigenous worldviews in noting that all assessment instruments are constructed within a particular worldview. Measurement disjuncture (Sul, 2019) refers to the misalignment that occurs when elements of an instrument development process from one worldview are applied to the instrument development process of another worldview. In manners that are both qualitative and quantitative, measurement disjuncture negatively affects the establishment of measurement validity. Further, as a result of measurement disjuncture, researchers are less likely to acknowledge that educational programming has had an impact when, in fact, it may have (Type II error). This paper focuses on the development of culturally specific assessments, the Papakū Makawalu Competency Assessment and the Dibishgaademgak Anishinaabemowin (language) assessment, as a means to minimize measurement disjuncture. The paper concludes with a declaration of educational and assessment autonomies.
... ( Ladson-Billings, 1995) Culturally responsive assessment Supports the academic development of students; inclusive of a willingness to nurture and support cultural competence; aims to support the development of a sociopolitical or critical consciousness; Focused on constructs and measures of importance to educational practitioners and other key stakeholders (Hood, 1998) Culturally specific assessment Supports the academic development of students; Inclusive of a willingness to nurture and support cultural competence; Aims to support the development of a sociopolitical or critical consciousness; Focused on constructs and measures of importance to educational practitioners and other key stakeholders; Functions within a system of knowledge that exists within a named worldview (Sul, 2019) Having arrived at a formal definition of culturally specific assessment, it is offered here as a potential solution to the problem of measurement disjuncture. In order to determine whether the minimization of measurement disjuncture can be achieved through the employment of culturally specific assessments, educational environments that meet the criteria for culturally specific assessment are sought. ...
Conference Paper
Educational research findings often rely on evidence obtained through the use of standardized assessment instruments. This presentation draws a distinction between Western and Indigenous worldviews in noting that all assessment instruments are constructed within a particular worldview. Measurement disjuncture (Sul, 2019) refers to the misalignment that occurs when elements of an instrument development process from one worldview are applied to the instrument development process of another worldview. In manners that are both qualitative and quantitative, measurement disjuncture negatively affects the establishment of measurement validity. Further, as a result of measurement disjuncture, researchers are less likely to acknowledge that educational programming has had an impact when, in fact, it may have (Type II error). This paper focuses on the development of culturally specific assessments, the Papakū Makawalu Competency Assessment and the Dibishgaademgak Anishinaabemowin (language) assessment, as a means to minimize measurement disjuncture. The paper concludes with a declaration of educational and assessment autonomies.
Article
Full-text available
With an increased interest in the revitalization of Indigenous languages and cultural practices worldwide, there is also an increased need to develop tools to support Indigenous language learners and instructors. The purpose of this article is to presents such a tool called ANA ‘ŌLELO, designed specifically to assess Hawaiian language proficiency. After US occupation of the islands in 1893, Native Hawaiians experienced a waning of cultural identity, traditional values and practices, and a near loss of the Hawaiian language in daily communication. To contribute to ongoing Hawaiian language revitalization efforts, Dr. Kahakalau developed ANA ‘ŌLELO, a unique scale that measures a learner’s proficiency of the Hawaiian language. This article explores ANA ‘ŌLELO and the benefits it holds for teachers, students, and researchers, through the use of standardized subsets of overall proficiency, conversational ability, and language protocol. The article also suggests that ANA ‘ŌLELO can have substantial beneficial impacts for other endangered Indigenous languages; and argues that native populations can and should develop their own language proficiency scales, unique to the culture and language of the population, and outside of the constraints of colonizing languages’ proficiency values. © 2017 The Author(s). This open access article is distributed under a Creative Commons Attribution (CC-BY) 4.0 license.
Article
Full-text available
In this article, we examine how American Indian students in Montana performed on standardized state science assessments when a small number of test items based upon traditional science knowledge from a cultural curriculum, “Indian Education for All”, were included. Montana is the first state in the US to mandate the use of a culturally relevant curriculum in all schools and to incorporate this curriculum into a portion of the standardized assessment items. This study compares White and American Indian student test scores on these particular test items to determine how White and American Indian students perform on culturally relevant test items compared to traditional standard science test items. The connections between student achievement on adapted culturally relevant science test items versus traditional items brings valuable insights to the fields of science education, research on student assessments, and Indigenous studies.
Article
Many Indigenous languages are critically endangered and faced with the urgent need to create parent-Aged advanced speakers. This goal requires sequenced curriculum, effective teaching methods, students being supported to spend more than 2,000 hours on task, and regular assessments. In response to this urgent need the author followed a proven direct acquisition method and curricular design developed for Nsyilxcn and Interior Salish languages and wrote two beginner Tlingit textbooks and their accompanying teaching manuals. The author piloted the Tlingit textbooks with a cohort and developed a filmed assessment process. This article shares results of filmed assessments for Nsyilxcn and Tlingit, implemented by beginner and intermediate speakers. Recommendations are made for Indigenous language revitalization, including assessment methods appropriate to critically endangered Indigenous languages and strategies to create advanced speakers. ©2017 The Canadian Modern Language Review/La Revue canadienne des langues vivantes.
Article
Sociopolitical consciousness refers to an individual’s ability to critically analyze the political, economic, and social forces shaping society and one’s status in it. A growing body of scholarship reports that high levels of sociopolitical consciousness are predictive in marginalized adolescents of a number of key outcomes including resilience and civic engagement. The present study explored the role that urban secondary schools can play in fostering adolescents’ sociopolitical consciousness through a longitudinal, mixed methods investigation of more than 400 adolescents attending “progressive” and “no excuses” charter high schools. Analyses revealed that, on average, students attending progressive high schools demonstrated sizeable shifts in their sociopolitical consciousness of racial inequality, and students attending no excuses high schools demonstrated sizeable shifts in their sociopolitical consciousness of social class inequality. Qualitative interviews with participating students offered insight into the curriculum, programming, and practices that these youth perceived as contributing to these differences in their sociopolitical consciousness.
Article
One of the challenges of providing speech and language pathology services to Indigenous communities is the lack of culturally appropriate screening and assessment tools. This paper describes the process used in the development of the Inuktitut and English Language Screening Tool intended for use in the Qikiqtani region of Nunavut, Canada. The project involved collaboration among teachers and the speech-language pathologist from Qikiqtani School Operations, a university faculty member, and several speech-language pathology students. The development of the instrument was informed by the social context and the features of the Indigenous language Inuktitut, and an understanding of ways to evaluate local language screening instruments.
Article
Language policy and assessment have rarely intersected, despite the potential value of accurate assessment in guiding programmatic decisions This lack of a close relation is particularly notable in decentralized educational systems like that of the US In making crucial decisions, local educational agencies generally rely on standardized instruments available almost exclusively in the dominant language—English Yet growing concerns related to cultural and linguistic pluralism as well as educational equity demand that skills in other native languages be recognized within the educational system This article describes some of the policy questions arising in the course of efforts to develop a test of Navajo comprehension for young children Using Cooper's (1989) framework for language planning as innovation and Ruiz's (1993) distinction between endoglossic and exoglossic language policies, we explain how the test development process has been similar to and different from the typical model of top-down test development by outside experts While the content of the test reflects the specific historical experience and current educational situation of the Navajo Nation, the procedures for test development as well as general issues related to using the test results may well be applicable to other indigenous language communities
Article
1. Except for the case of Eyak, which I can personally confirm, many of the statistics, large and small, in this article are but reports or estimates; I trust it will be obvious that any imprecision in the present figures should in no way detract from the basic point of their shocking significance. For North America and the Soviet North the figures for numbers of speakers come mainly from colleagues. For the numbers of languages and their speakers for the world generally, by far the best single source available that I am aware of is the Ethnologue (Grimes 1988), to which this paper refers below. 2. Note, however, that 187 languages comprise only a very small proportion of the world's languages, about 3%. For this and much of the following I am most indebted to Barbara and Joseph Grimes and their Ethnologue (1988), together with some late 1990 updates (personal communication). This work provides by far the most detailed worldwide survey of languages yet available, and it is also a project continuously being updated. In keeping with the estimated nature of statistics, I have generally rounded the Grimeses' figures. 3. The Grimeses' updated figures now include over 100 more very nearly extinct Australian languages listed in Wurm & Hattori 1981 but not in the 1988 Ethnologue. 4. Ken Hale wishes to point out that the figures attributed to him in Time magazine, September 23, 1991, are from Mike Krauss's presentation in the LSA Endangered Languages symposium of January, 1991. 5. As this goes to press, I note the article 'World of the Living Dead' (Natural history 9/91:30, 32-37) by the biologist Jared Diamond, who takes the Javanese bird situation as an example to illustrate his view, held by many biologists, that 'half of the world's species will be extinct or on the verge of extinction by the end of the next century'. Thus the enormity of the impending biological catastrophe may come much closer to matching that of the linguistic catastrophe than one might believe from the official endangered species listings. 6. As this goes to press, in addition to the political support of the federal Native American Languages Act of 1990 (described below by Watahomigie & Yamamoto), new federal legislation is proceeding that is to include appropriations: S. 1595, the Alaska Native Languages Preservation and Enhancement Act of 1991, introduced by Senator Murkowski of Alaska in July, 'to preserve and enhance the ability of Alaska Natives to speak and understand their native languages', passed by the Senate in November; and S. 2044, the Native American Languages Act of 1991, 'to assist Native Americans in assuring the survival and continuing vitality of their languages', introduced by Senator Inouye of Hawaii in November.