ArticlePDF Available

Quality and accountability in the out-of-school-time sector

Authors:

Abstract and Figures

In the fragmented out-of-school-time sector, defining and measuring quality in terms of staff behaviors at the point of service provides a common framework that can reduce obstacles to cross-sector and cross-program performance improvement efforts and streamline adoption of data-driven accountability policies. This chapter views the point of service, that is, the microsettings where adults and youth purposefully interact, as the critical unit of study because it is ubiquitous across out-of-school-time programs and because it is the place where key developmental experiences are intentionally delivered. However, because point-of-service behaviors are embedded within multilevel systems where managers set priorities and institutional incentives constrain innovation, effective quality interventions must contend with and attend to this broader policy environment. The Youth Program Quality Assessment (Youth PQA) is one of an emerging class of observational assessment tools that measure staff performances at the point of service and, depending on methodology of use, can help create the conditions that managers and youth workers need to accept, adopt, and sustain accountability initiatives. Observational assessment tools can be flexible enough to be used for program self-assessment (appropriate for low-stakes, non-normative learning purposes), external assessment (appropriate for higher stakes, normative comparisons, and performance accountability), and various hybrids that combine elements from each. We provide advice for decision makers regarding how to most effectively use the Youth PQA and similar measurement tools depending on the articulation of clear purposes for which accountability and improvement policies are enacted and effective sequencing of implementation.
Content may be subject to copyright.
The Youth Program Quality Assessment is one of
an emerging class of observational assessment tools
that measure staff performances at the point of ser-
vice and can help drive systemic approaches to
workforce development and accountability.
6
Quality and accountability in the
out-of-school-time sector
Charles Smith, Thomas J. Devaney,
Tom Akiva, Samantha A. Sugar
youth-serving organizations that operate outside the traditional
school day face increasing accountability pressures. According to
a recent tally, fourteen states are implementing quality account-
ability systems for subsidized child care and another thirty are
“exploring/designing or piloting” some type of quality improve-
ment process.1In addition, several national philanthropic organi-
zations and public agencies have invested heavily in after-school
quality improvement projects in cities and counties across the coun-
try, and numerous place-based projects are moving forward with
local resources.2Thus, more than ever before, theories of perfor-
mance measurement and innovative quality accountability policies
109
NEW DIRECTIONS FOR YOUTH DEVELOPMENT, NO. 121, SPRING 2009 © WILEY PERIODICALS, INC.
Published online in Wiley InterScience (www.interscience.wiley.com) DOI: 10.1002/yd.299
This article was supported by the William T. Grant Foundation, the Picower Founda-
tion, and the Robert Wood Johnson Foundation. Correspondence can be addressed to
Charles Smith at charles@cypq.org or David P. Weikart Center for Youth Program
Quality, 124 Pearl Street, Suite 601, Ypsilanti, MI 48198.
110 DEFINING AND MEASURING QUALITY IN YOUTH PROGRAMS
new directions for youth development DOI: 10.1002/yd
are being brought to bear within and across out-of-school-time
(OST) institutions.
Implicit in the calls to improve quality across the diverse uni-
verse of OST organizations is the notion that the quality of what
teachers and youth workers do when they interact with youth influ-
ences how young people engage with content and, more broadly,
supports developmental change on a range of cognitive, social-
emotional, health, and other outcomes. Thus, the point of ser-
vice—the microsetting where intentional interactions between
adults and youth occur—is increasingly viewed as the place where
the most important qualities of an OST program are enacted and
where youth outcomes can be most directly influenced. Traditional
accountability systems are not optimized to define, measure, or
support high-quality practices in microsettings; rather, they are
designed to measure and influence the inputs and outputs that
bound these settings.
In the after-school field, discomfort with the applicability of out-
comes-based policy has led to the development of alternative
accountability and improvement models that draw on understand-
ings of adult motivation, knowledge management, and performance
change.3Rather than holding professional staff “accountable”
for peak performances of children on standardized tests or for
population-level outcomes such as teen pregnancy or recidivism
rates, these models attempt to provide OST programs with a set of
metrics, methods, and policies they can use to:
Create learning communities focused on standards and profes-
sional learning
Measure and track changes in the quality of staff performances
at the point of service
Align professional development resources across programs
and policies
Although these approaches hold great promise for the OST
field, several questions about their practicality and ultimate
impact remain:
111
QUALITY AND ACCOUNTABILITY
new directions for youth development DOI: 10.1002/yd
How can point-of-service quality be defined in a way that makes
sense within and across the many organizational types that make
up the OST field?
Which assessment technologies (defined as an assessment tool
plus methods of use) can reliably measure and represent quanti-
ties of quality so defined?
How can the system-level momentum and incentives necessary
to pursue continuous quality improvement be generated and sus-
tained in a highly fragmented policy environment?
In this article, we provide a definition of point-of-service qual-
ity that is relevant to and measurable across a broad range of OST
settings; describe the Youth Program Quality Assessment and an
associated continuum of methods as a set of technologies designed
to serve different accountability purposes; and offer advice to OST
organizations and decision makers interested in using quality met-
rics as a foundation for innovative accountability policies.
Defining quality
In this opening section, we provide a detailed definition of point-of-
service quality and argue that this construct is most profitably located
and measured within, and compared across, OST program offerings
in which adults and youth deliberately interact. That is, we suggest
that a content-neutral quality construct, applied at the OST pro-
gram offering level, can support (1) measurement of quality across
the diverse universe of OST systems and sites and (2) systemic
implementation of quality accountability and improvement policies.
What do we mean by quality?
When we think about quality we start with a simple formula. Devel-
opmentally powerful environments provide (1) positive relationships
as a context for (2) a learning task or content that (3) increases in
complexity over time.4This rough formula is content neutral, that
is, it can be applied across all academic and nonacademic settings
112 DEFINING AND MEASURING QUALITY IN YOUTH PROGRAMS
new directions for youth development DOI: 10.1002/yd
where adults and youth purposefully interact, and thus it represents
a generic process for supporting positive developmental change.
The specific content that an OST organization chooses to pro-
vide is of critical importance. However, because OST organiza-
tions seek to influence youth development trajectories through a
variety of academic, enrichment, recreational, and other content-
laden activities, it is virtually impossible to define quality related
to particular content in a way that is relevant across the field. Con-
sequently, we define quality in terms of factors that are ubiquitous
across program types. Specifically, we define quality in terms of
staff behaviors and the nature of their interactions with youth. For
example, everyone agrees that relationships between staff and
youth are a critical characteristic of all high-quality youth settings
and that specific practices, such as welcoming participants, ensur-
ing inclusion, and employing conflict resolution techniques, influ-
ence those relationships regardless of setting type. These kinds of
practices represent the generic, content-neutral professional skills
that are necessary, for example, to create and maintain a positive
relational context and scaffold youth toward deep engagement
with content.
Our construct for quality is thus focused on discrete staff behav-
iors that represent best practice in youth work methods and reflect
important aspects of the research validated High/Scope active-
participatory approach to learning.5The elements described in Fig-
ure 6.1 are conceptualized as a set of staff practices that provide key
developmental experiences for youth and can be reliably observed.
A recent compendium of assessment tools suggests wide agreement
about specific elements of best practices in OST programs as
defined in Figure 1.6
Where do we look for quality?
Although there is general agreement about the basic elements of
high-quality practice in youth settings, there is less agreement and
even less discussion about where quality is located within those set-
tings. The “where” question, however, is critical. Answering it
reduces the scope of all possible things that can happen in an OST
new directions for youth development DOI: 10.1002/yd
Figure 6.1. Elements of program quality
114 DEFINING AND MEASURING QUALITY IN YOUTH PROGRAMS
new directions for youth development DOI: 10.1002/yd
program to the patterns of interaction and practice most likely to
support positive youth development.
Program offerings are the key microsystem in OST contexts
where access to powerful developmental and learning experiences
is available and thus the place where technology is optimized to
measure quality. In our definition of where, offerings consist of the
same staff, same youth, and same learning purpose delivered over
a sequence of sessions. For example, a mural painting workshop
that meets at a residential treatment center every Tuesday and Fri-
day is a program offering.
The language barrier
Program offerings are the common context in which quality is enacted.
However, the terminology used to define quality varies across places,
networks, and programs. Many different social science languages
include ideas related to positive youth development and are used to
support policies and programs with parallel intentions but different
names. Indeed, programs variously employ and mix the languages of
positive youth development, resiliency theory, asset building, social-
emotional learning, brain-based instruction, constructivist education,
community service, and others. These approaches are all designed to
optimize youth motivation and learning and include variations of sev-
eral basic elements: emotional and active learning supports, coopera-
tive learning methods, and higher-order cognitive engagement.
These competing languages about positive youth develop-
ment reinforce policy fragmentation and can serve as barriers
to cross-sector and cross-program improvement initiatives. The
key to overcoming these barriers is to conceptualize quality in
a way that can span these language divides. Defining quality
at the point of service in terms of distinct elements of content
neutral staff practices offers OST professionals a straightfor-
ward translation guide across different languages and program
sectors. Table 6.1 describes a diverse profile of programs in
twenty cities, counties, and states that planned to apply the
quality construct described in Figure 6.1 to create place-based
accountability and improvement systems as part of the Ready by
115
QUALITY AND ACCOUNTABILITY
new directions for youth development DOI: 10.1002/yd
Table 6.1. Diversity of program types in a place-based project
using the Youth PQA
Percentage
(N = 124)
Program’s primary purpose: What one or two of the following
best represent your program’s primary approach to working
with young people?
Thriving 21.8
Learning 40.3
Connecting 74.2
Working 16.1
Leading 33.1
Age level served: Which age level does your program primarily
serve?
Elementary 64.5
Middle school 68.5
High school 62.9
Post–high school 24.2
Organization type: Which type best represents your
organization?
Local nonprofit 57.3
Nationally affiliated nonprofit 25.0
Unit of city or county government 14.5
Local school organization 10.5
Program model: Which type best describes your program
model?
After-school (mix) 75.8
After-school (sports) 4.8
School-day insert 21.0
Residential 4.0
Court referred/juvenile justices 8.9
Alternative school 2.4
Note: The data presented were generated using a survey tool developed by the High/
Scope Educational Research Foundation and administered in October 2008 to 124
sites in twenty cities, counties, and states participating in a national project. Columns
total to more than 100 percent because respondents could select all that applied.
21 Quality Counts Initiative.7A content-neutral definition of
quality provides a common framework through which these
diverse programs can conceptualize and create system-level
quality improvement systems.
Taken together, our definitions of what quality is and where it can
be found form the integral elements of a larger point-of-service
theory.8This theory is grounded in the following assumptions:
116 DEFINING AND MEASURING QUALITY IN YOUTH PROGRAMS
new directions for youth development DOI: 10.1002/yd
OST participation can provide youth with access to powerful
development and learning experiences.
Access to these key experiences occurs during content-driven
offerings defined by continuity of staff, youth, and purpose.
Performances of staff that lead offerings are the key units of
quality in OST settings.
High-quality performances represent consistent approaches to
youth work and instruction over time within individual staff and
across different staff within the same organization.
Point-of-service performances cannot be sustainably improved
without explicit attention to and support from the organizations
and systems in which point-of-service interactions are embedded.
Assessing quality across diverse programs and purposes
In this section, we discuss a widely used quality assessment tool—
the Youth Program Quality Assessment (PQA)—focusing on sev-
eral methodologies for its use and the distinct accountability
purposes that these different methods serve. Note that although we
focus on a specific instrument, we believe that the logic of align-
ment between methods and purposes applies across an entire class
of observational tools that measure the quality of settings where
children and youth spend their time.
Youth Program Quality Assessment
Point-of-service theory defines and locates quality within OST
staff practices that occur within program offerings. As noted
above, however, the terms used to define and manage positive
youth development can differ substantially across places, net-
works, and programs. Although these distinct languages reflect
substantially similar ideas, in practice they serve as barriers to
cross-sector and cross-program improvement efforts. The key
to effectively measuring quality in this fragmented environment
is to define quality as a common, credible unit of study across pro-
grams and offerings.
117
QUALITY AND ACCOUNTABILITY
new directions for youth development DOI: 10.1002/yd
The Youth Program Quality Assessment (PQA) is an obser-
vational instrument designed to assess the quality of staff per-
formances during OST offerings.9The instrument requires an
observer to document practices employed by staff and then use
this written evidence to score multi-item rubrics, constituting
the scales represented in Figure 6.1. Relative to other observa-
tional approaches, the PQA is a lower-inference observational
metric since its items reflect counts of staff instructional behav-
iors. Youth PQA items are scored using a three-point scale: the
lowest level indicates the absence of a practice, the midlevel indi-
cates that a practice occurred but not all children had access to
it, and the highest level for each item indicates that the practice
was present and all youth participated. On average, the PQA
requires an observer to collect objective, anecdotal evidence for
one hour and one additional hour to score the instrument’s
multi-item rubrics.
Several samples of Youth PQA data produced by reliable exter-
nal raters have been subjected to psychometric evaluation. The evi-
dence from these evaluations indicates that data generated by the
instrument are both reliable (interrater and internal consistency)
and valid (concurrent and predictive validity) through association
with multiple other sources of observational and youth survey
data.10 One of the key strengths of the Youth PQA is that it pro-
duces data that is clearly indexed to staff practice.11 Another is the
utility or consequential validity of the data it provides for end users.
That is, the Youth PQA and its methods of use help youth work-
ers, program managers, and other stakeholders develop common
understandings of program quality and help build momentum for
improvement initiatives. Indeed, Youth PQA users report that the
process of using the instrument and reviewing the data it produces
leads to program change.12
Continuum of methods and purposes
The Youth PQA can be employed using a continuum of method-
ologies.13 As depicted in Table 6.2, different methods of data col-
lection serve different purposes and carry different strengths and
118 DEFINING AND MEASURING QUALITY IN YOUTH PROGRAMS
new directions for youth development DOI: 10.1002/yd
Table 6.2. Continuum of methods, purposes, and resources
for quality assessment
Lower-stakes program Higher-stakes
self-assessment Hybrid approaches external assessment
Site-based self- Trained, reliable asses- Trained, reliable
assessment teams. sors recruit site-based assessors not connected
self-assessment teams to the program.
Rough data to get staff to coproduce quality
thinking and discussing scores. Precise data for internal
program quality in the and external audiences
context of best practice. Rough and precise data for evaluation, moni-
comingled to produce toring, accountability,
Less time, less cost. more accurate scores improvement,
than self-assessment reporting.
Impact internal alone.
audiences. More time, more cost.
Process supports more
specific planning and Impact on internal and
staff development but external audiences.
not appropriate for eval-
uation or accountability.
Most expensive, poten-
tially highest learning
impact.
Impact on internal
audiences.
limitations. It is worth reiterating that the Youth PQAs primary
application has been in the context of quality accountability and
improvement policies for OST systems. By definition, account-
ability and improvement policies are designed to measure perfor-
mance on some set of behavioral norms and create incentives to
improve performances on those norms. However, such policies do
not always have the intended effect. In fact, some quality assess-
ment technologies may paradoxically push staff into counter-
productive accountability behaviors—resistance, avoidance, and
minimum compliance—that deliver little return on investment.14
In this article, our analysis of the various methods of deploying the
Youth PQA technology is designed to surface the strengths and
weaknesses of each method and highlight the impact of each
method on improvement behaviors. Note that case studies describ-
119
QUALITY AND ACCOUNTABILITY
new directions for youth development DOI: 10.1002/yd
ing the fit between external and self-assessment technologies and
system purposes have been described in greater detail elsewhere.15
In the external assessment data collection method, a single Youth
PQA is scored for a specific offering. Individual ratings can be
aggregated to represent the quality of an overall program, usually
using a sample drawn from all offerings at a site. External assessors
are trained to acceptable levels of reliability to ensure precision, a
process that takes up to two and a half days. Scores produced
through the external assessment process can be a source of perfor-
mance feedback to staff but can also serve higher-stakes purposes
such as public release, program monitoring, and accountability for
change over time or attainment of norms. In prior studies, we have
sampled either two or three unique offerings with different staff
and different learning purposes to produce a program-level qual-
ity score.
The primary limitation of the external assessment method is
expense associated with data collection and scoring. It takes an
average of two hours per observational rating plus logistics and
travel. However, when collected by data collectors of known relia-
bility, observational data can have powerful effects on site manage-
ment perceptions of staff skills, suggesting that the additional cost
of external observation may serve as a down payment toward more
effective interventions to increase workforce skills, program quality,
and ultimately youth development and learning.16
The program self-assessment data collection method is team
based and designed to build learning community and acceptance
for quality performance metrics at an OST site. To implement the
self-assessment method, a site manager attends training (six hours
of content, delivered live or online) and assembles a program
assessment and improvement team, typically consisting of site staff
but also frequently including volunteers, board members, the
school principal, and parents. Individuals from this team take turns
observing in fifteen- and twenty-minute increments during a selec-
tion of different program offerings (led by different staff ) at a site.
Then all anecdotal records are anonymously pooled and used to
score one Youth PQA Form A for the entire site.
120 DEFINING AND MEASURING QUALITY IN YOUTH PROGRAMS
new directions for youth development DOI: 10.1002/yd
This method has several strengths. First, the program self-
assessment process has been roundly endorsed by site managers as
a way to build communities of learners around program operation
and understanding of quality.17 For example, in a recent survey of
Michigan-based 21st Century Community Learning Center staff,
96.7 percent of respondents anonymously reported that using the
Youth PQA as a state-mandated self-assessment “was a good use of
time given the demands of the job” (N= 145). Second, we believe
that the program self-assessment process prepares teams to engage
with and adopt the external assessment process in more productive
ways.18 This is so because program self-assessment builds familiar-
ity among the staff with the metric that will eventually be used and
begins a learning process that staff find valuable, increasing demand
for more accurate data provided by external methods.19
From a research perspective, data produced in the program self-
assessment process are limited in two primary ways:
Data are generally biased in a positive direction; that is, scores pro-
duced by self-reports are likely to be inflated relative to those
that would be produced by external raters.
The flexibility of the process means that self-assessment does not
produce a consistent, definable unit of study, that is, a single staff
performance during a defined offering.
Consequently, it is not possible to make apples-to-apples compar-
isons between programs.
These limitations mean that program self-assessment data are not
appropriate for comparison or aggregation. Use of such data is thus
limited to immediate interpretation by the site team for purposes of
building common understanding around the content of the rubrics.
Hybrid methods are also possible, as described in Table 6.2.
This table represents one of the numerous ways to combine
external and self-assessment to meet local objectives and condi-
tions. Several years ago, the Maine 21st Century Community
Learning Centers program introduced an innovative account-
ability model that produced both more rigorous baseline scores
121
QUALITY AND ACCOUNTABILITY
new directions for youth development DOI: 10.1002/yd
than program self-assessment alone and built a learning commu-
nity among management staff by training site managers to high
levels of reliability and using them as external assessors in other
21st Century Community Learning Centers grantees.20 More
recently, the Rhode Island 21st Century Community Learning
Centers program trained its quality improvement coaches as
external raters. These coaches then recruited frontline staff to
join in both the process of observational evidence gathering and
scoring the Youth PQA.21 Again, these methods were adopted to
achieve specific purposes that balanced within-site learning and
measurement precision.
System-level implications of quality assessment
So far, we have argued that point-of-service theory, particularly as
embodied in the Youth PQA, is a useful lens through which to
measure and manage the quality of program offerings across the
diverse range of missions, funding, and regulatory streams and
client types that characterize the OST field. We have also suggested
a rough alignment between quality assessment technologies and
accountability purposes:
External assessment fits into an accountability context where scores
are made available for various types of comparison across set-
tings and over time and where external ratings are required
through program rules.
Program self-assessment fits into an accountability context where the
implementation of the team-based self-assessment process, whether
voluntary or mandated, is enacted for purposes of building a learn-
ing community around the content of the quality metrics.
At the heart of this alignment argument is the idea that innova-
tive accountability policies can be developed in ways that address
both POS performances (staff accountability for practices) and
organizational efforts to support workforce development and learn-
ing (system-level accountability for creating and sustaining profes-
sional learning communities).
122 DEFINING AND MEASURING QUALITY IN YOUTH PROGRAMS
new directions for youth development DOI: 10.1002/yd
Clearly the use of quality metrics at the point of service can have
an impact on management practices and organizational policies. By
finding moments of comparability across the great diversity of OST
programs, it is possible to generate system-level outcomes such as
greater coordination of resources across organizations and more
system-like approaches to workforce development. The Iowa Col-
laboration for Youth Development (ICYD), for example, conducted
a “quality snapshot” by collecting performance samples in thirty-
eight organizations representing thirteen different youth funding
streams in the state of Iowa. The snapshot was explicitly introduced
in order to “launch a program quality improvement system across
youth-serving programs funded by a variety of funding streams.”
Based on findings from the report, ICYD recommended that “state
agencies should continue to explore how best to meet the capacity
needs of their grantees, and, when feasible, other youth-serving
agencies not receiving state funding, as a way to improve interac-
tions and engagement of youth [two low-scoring domains of quality
from the snapshot report] . . .”22 This suggests that the deployment
of a uniform quality metric across the siloed education, human ser-
vice, juvenile justice, and community-based program settings that
are present in most cities and counties can initiate movement
toward shared vision, planning, and resource allocation at higher
levels of decision making. The Iowa example provides further sug-
gestive evidence that accountability and improvement technologies
like the Youth PQA can empower intermediary organizations to
facilitate this kind of movement toward greater collaboration across
agencies and across geography.
Implementation advice
There is ample evidence that a general definition of point-of-
service quality can be relevant and meaningful across a range of
OST settings and that observational assessment instruments like
the Youth PQA represent performance assessment technologies
that can provide a foundation for accountability and improvement
123
QUALITY AND ACCOUNTABILITY
new directions for youth development DOI: 10.1002/yd
policies. We now offer some more general advice based on our
experience as training and technical assistance providers to nearly
thirty active accountability and improvement projects.
Purposes
Purposes should drive decisions about how to use performance
metrics like the Youth PQA. In fact, in our experience, mis-
matches between purpose and methods are likely to lead to im-
plementation failure. Consequently it is imperative that systems
and organizations that wish to deploy a quality improvement sys-
tem clearly define their objectives (for example, high-stakes com-
parisons across time and settings versus low-stakes trust building
and professional development) and then select a method or meth-
ods of use.
Combined methods
Although there are relative strengths and weaknesses of the pro-
gram self- and external assessment methods when deployed sep-
arately, our experience suggests that there are great benefits to
deploying them sequentially as a means of building professional
learning community and achieving buy-in for future higher-
stakes accountability and improvement policies. In fact, the most
successful implementations of the Youth PQA have used program
self- and external observational assessments in ways that spur for-
mation of a team culture and generate demand for more objec-
tive external assessment of staff performance against quality
standards.23
This last point is critical. If systems and organizations do not
provide proper incentives to managers to prioritize and pursue
point-of-service quality initiatives, these efforts are likely to fade
or never gain momentum. In OST settings (as in other education
and human services organizations), managers set the priorities and
make the decisions that influence the adoption and use of quality
assessment and improvement practices. In Palm Beach County,
Florida, for example, qualitative and quantitative data collected as
part of a quality improvement system pilot suggest that low-stakes
124 DEFINING AND MEASURING QUALITY IN YOUTH PROGRAMS
new directions for youth development DOI: 10.1002/yd
deployment of program quality technology, coupled with substan-
tial technical support from an intermediary organization such as
Prime Time Palm Beach County, can produce measurable changes
in point-of-service quality and management practices, as well as
streamline adoption of accountability policies.24
Pilots and supports
Sequenced deployment of assessment methods can build trust
throughout stakeholders in a network and set the stage for higher-
stakes accountability and improvement initiatives. Our experience
suggests an additional strategy for ensuring the successful deploy-
ment of quality improvement initiatives using the Youth PQA or
similar tools: begin with a small-scale pilot demonstration. Demon-
stration pilots offer systems the opportunity to adjust technology
to context, demonstrate applicability across disparate program
types, and align training and technical assistance that supports
improvement. Furthermore, pilots allow word-of-mouth messages
that the quality accountability and improvement policy works to
spread, and they support the emergence of success stories and local
champions for the work. Nearly all of our place-based and statewide
projects have begun with pilot groups of ten to thirty programs
implementing the Youth PQA and various packages of supporting
training and technical assistance over six to eighteen months.
Conclusion
The Youth Program Quality Assessment is one of an emerging
class of observational assessment tools that measure staff perfor-
mances at the point of service and, depending on the methodology
of use, help create the conditions that managers and youth work-
ers need to accept, adopt, and sustain quality accountability and
improvement initiatives. Observational assessment tools are flexi-
ble enough to be used for program self-assessment (appropriate for
lower-stakes, nonnormative learning purposes), external assessment
125
QUALITY AND ACCOUNTABILITY
new directions for youth development DOI: 10.1002/yd
(appropriate for higher stakes, normative comparisons, and per-
formance accountability), as well as various hybrids that combine
elements from each. Innovative quality accountability and improve-
ment policies are emerging in numerous places and statewide sys-
tems across the country, applying uniform standards of quality and
professional performance at the point of service and increasing
higher-level collaboration of funders, regulators, and site-level
managers. These initiatives are often carried by intermediary orga-
nizations that can provide equal measures of disinterested objec-
tivity and careful support for successful point-of-service change.
We hope that the learning discussed in this article will contribute
to the ongoing development and adoption of innovative quality
accountability and improvement policies across the OST field.
Notes
1. Data compiled by the National Child Care Information Center as of
November 2006, cited in the 2007 annual conference presentation for the
National Association of Child Care Resource and Referral Agencies.
2. Both the Wallace Foundation and Robert Wood Johnson Foundation have
recently funded large-scale quality intervention efforts in cities, counties and states.
3. Smith, C., Akiva, T., Blazevski, J., Pelle, L., & Devaney, T. (2008). Final
report on the Palm Beach Quality Improvement System pilot: Model implementation
and program quality improvement in 38 after-school programs. Ypsilanti, MI:
High/Scope Educational Research Foundation; Smith, C., & Akiva, T. (2008).
Quality accountability: Improving fidelity of broad developmentally focused
interventions. In H. Yoshikawa & B. Shinn (Eds.), Transforming social settings:
Towards positive youth development. New York: Oxford University Press.
4. Bronfenbrenner, U. (1999). Environments in developmental perspec-
tive: Theoretical and operational models. In S. L. Friedman & T. D. Wachs
(Eds.), Measuring environment across the life span: Emerging methods and concepts
(pp. 3–28). Bethesda, MD: National Institutes of Health, Child Health and
Human Development Agency, Center for Research for Mothers and Children.
5. Ilfeld, E. M. (1996). Learning comes to life: An active learning program for
teens. Ypsilanti, MI: High/Scope Press; Oden, S., Kelly, M., & Weikart, D.
(1992). Changing the potential: Programs for talented disadvantaged youth.
Ypsilanti, MI: High/Scope Press; Smith, C. (2005a). Evidence of effectiveness for
training in the High/Scope Participatory Learning approach. Ypsilanti, MI:
High/Scope Educational Research Foundation. For more information, see
http://www.highscope.org.
6. Yohalem, N., & Wilson-Ahlstrom, A. (2007). Measuring youth program
quality: A guide to assessment tools. Washington, DC: Forum for Youth Investment.
126 DEFINING AND MEASURING QUALITY IN YOUTH PROGRAMS
new directions for youth development DOI: 10.1002/yd
7. Data from this table were drawn from pilot sites participating in the
Ready by 21, Quality Counts Initiative led by the Forum for Youth Invest-
ment and funded by the Robert Wood Johnson Foundation. For more infor-
mation, see www.forumforyouthinvestment.org.
8. Smith, C., Peck, S., Denault, A., Blazevski, J., & Akiva, T. (in review).
Quality at the point-of-service: Profiles of practice in afterschool settings. Manuscript
submitted for publication, January 8, 2009.
9. High/Scope Educational Research Foundation. (2005). Youth PQA
program quality assessment: Administration manual. Ypsilanti, MI: High/
Scope Press.; Smith, C., & Hohmann, C. (2005). Full findings from the
Youth PQA validation study. Ypsilanti, MI, High/Scope Educational
Research Foundation.
10. Smith & Hohmann. (2005); Blazevski, J., & Smith, C. (2007a). Inter-
rater reliability on the youth program quality assessment. Ypsilanti, MI:
High/Scope Educational Research Foundation; Blazevski, J., & Smith, C.
(2007b). After-school quality and school-day outcomes in Michigan’s 21st CCLC pro-
gram. Ypsilanti, MI: High/Scope Educational Research Foundation; Smith et
al. (2008).
11. A common challenge among observational measures is that all of the
scales tend to be highly correlated, resulting in an inability to assess any but
the most global dimensions of quality. We have tested theoretically relevant
subsets of items on the Youth PQA to achieve dimensionality of measurement
in replicated confirmatory analyses, suggesting that the measure can be used
to identify different aspects of staff performance quality in OST offerings. For
more information, see La Paro, K. M., Pianta, R. C., & Stuhlman, M. (2004).
Classroom assessment scoring system (CLASS): Findings from the prekinder-
garten year. Elementary School Journal, 104(5), 409–425; Perlman, M., & Zell-
man, G. L. (2004). Examining the psychometric properties of the Early
Childhood Environment Rating Scale-Revised (ECERS-R). Early Childhood
Research Quarterly, 19, 398–412. Also see Smith et al. (in review).
12. Smith & Hohmann. (2005); Spielberger, J., & Lockaby, T. (2008). Palm
Beach County’s Prime Time Initiative: Improving the quality of after-school programs.
Chicago: Chapin Hall Center for Children at the University of Chicago.
13. High/Scope Educational Research Foundation. (2005).
14. Smith & Akiva. (2008).
15. Smith, C., Akiva, T., Arrieux, D., & Jones, M. (2006). Improving qual-
ity at the point-of-service. In D. A. Blythe & J. A. Walker (Eds.), Rethinking
programs for youth in the middle years. San Francisco: Jossey-Bass; Smith, C.
(2005b). Findings from the self-assessment pilot in Michigan 21st Century Learn-
ing Centers. Ypsilanti, MI: High/Scope Educational Research Foundation.
16. Spielberger, J., & Lockaby, T. (2006). The Prime Time Initiative of Palm
Beach County, Florida—QIS development process evaluation: Year 2 report.
Chicago: Chapin Hall Center for Children at the University of Chicago; Spiel-
berger & Lockaby. (2008).
17. Smith. (2005b).
18. Smith et al. (2008).
19. Spielberger & Lockaby. (2006). Spielberger & Lockaby. (2008).
127
QUALITY AND ACCOUNTABILITY
new directions for youth development DOI: 10.1002/yd
20. Smith & Akiva. (2008).
21. Sugar, S., Devaney, T., & Smith, C. (2008). Results from the RIPQA qual-
ity improvement system: Quality standards implementation in 19 after-school pro-
grams. Ypsilanti, MI: Center for Youth Program Quality.
22. Sugar, S., Akiva, T., Wilson-Ahlstrom, A., Behrer, C., & Croll, A.
(2008). The Iowa Youth Program quality snapshot. Ypsilanti, MI: Center for
Youth Program Quality.
23. Smith et al. (2008); Spielberger & Lockaby. (2008).
24. Smith et al. (2008).
charles smith is the director of the David P. Weikart Center for Youth
Program Quality.
THOMAS J
.
DEVANEY
is the operations and knowledge manager at the
David P. Weikart Center for Youth Program Quality.
TOM AKIVA
is a doctoral student in the Combined Program in Education
and Psychology at the University of Michigan and a consultant with the
David P. Weikart Center for Youth Program Quality.
samantha a. sugar is a research assistant at the David P. Weikart Cen-
ter for Youth Program Quality.
... A collaborative and inclusive process has been demonstrated to increase buy-in from staff (Smith, Devaney, Akiva, & Sugar, 2009;Surr, 2012) and to contribute to positive outcomes in quality improvement efforts (Chaboyer et al., 2012). In addition, in a non-threatening environment, not only is feedback from the technical assistance team more likely to be received with an openness to change, the feedback is also bi-directional and reciprocal. ...
... Gathering and using data to inform sustained, high-quality program implementation represents a key component of the TCI QIFAP. In fact, a broad research literature supports the integration of a data-informed approach to ongoing quality improvement efforts (Derrick-Mills, 2015;Parand et al., 2012;Surr, 2012;Sanclimenti & Caceda-Castro, 2017;Smith et al., 2009). The data used for these purposes can come from a variety of sources including information gathered as part of routine documentation procedures (e.g. ...
... Having staff involved in the data gathering and review process promotes self-reflection of practices, identification of needed changes, and fosters buy-in to program quality improvement (Casey Family Programs and the National Child Welfare Resource Center for Organizational Improvement, 2005; Smith et al. 2009;Stuczynski & Kimmich, 2010). As such, both data gathering and interpretation are intentionally collaborative processes in the TCI QIFAP. ...
Article
Full-text available
Strategies for sustaining a program beyond initial implementation remain one of the most poorly understood aspects of high-quality program implementation. This paper describes the Quality Improvement and Fidelity Assessment Process (QIFAP), a program purveyor-agency partnership that uses a unique, multi-step method for supporting sustained implementation of the Therapeutic Crisis Intervention (TCI) system to manage crises in child serving organizations. It outlines the steps of the process and highlights how specific activities are linked to current knowledge and principles from implementation science. The QIFAP occurs over a period of about three months, during which time program developers and agency representatives conduct staff surveys, a two-day site visit, and fidelity assessments in order to gather information, discuss findings, and plan steps for improving the TCI system in the organization. The process is guided by principles that emphasize the importance of organization leadership, building relationships, co-learning, using an individualized approach, data informed decision making, acknowledging risk, and congruence. We describe ways in which the strategies and approaches within the QIFAP are rooted in implementation science literature. Thus, the model represents an illustration of how research-based knowledge can work in practice to support long-term, high-quality program implementation.
... High quality educational programs for children include developmentally supportive learning environments and a sequence of progressively more complex tasks. A definition of quality that includes developmentally appropriate learning environments (those that support increasingly complex tasks over time) grounded in positive relationships may apply to programs that serve a range of populations ( Smith et al. 2009).This could include programs that combine services for parents and children. For children, research consistently shows that developmentally supportive environments and scaffolding the complexity of tasks are associated with better academic outcomes ( Burchinal et al. 2016;Zaslow et al. 2010). ...
... Yet, another study found that structural features were not strongly correlated with quality for after-school programs, and that process features mattered more ( Smith et al. 2008). the content of activities and instruction, the level of engagement and interactions between students and staff, and the level of supportiveness and safety of the classroom environment ( Smith et al. 2009). A study of changes to after-school programs in one county found that programs in which staff focused on offering environmental supports for learning and peer interaction had higher quality (as measured by a diagnostic quality assessment tool) after their program improvement process than before ( Smith et al. 2008). ...
... learning environments and a sequence of progressively more complex tasks. A definition of quality that includes developmentally appropriate learning environments (those that support increasingly complex tasks over time) grounded in positive relationships may apply to programs that serve a range of populations ( Smith et al. 2009). Learning environments and relationships matter for adults. ...
... However, what seems unaddressed, or rather 'silent' (Arya 2022) in such work is the role of the undergraduate educator, activity facilitator, or mentor in these community contexts. The bulk of studies about educators or program leaders center on the design of program-related activities or student outcomes that include final project quality and assessment performance (e.g., Hinga and Mahoney 2010;Mahoney and Stattin 2000;Smith et al. 2009;Vandell et al. 2012;Kataoka and Vandell 2013). Can engagement thrive in a community context if those who are leading or facilitating activities are themselves not engaged? ...
... While we emphasized the goals of agency, co-learning, and belonging; we did not dictate how such qualities should look and sound like. Such flexible approaches may help to foster meaningful experiences (engagement) in afterschool programs, supporting and complementing approaches brought by other scholars (e.g., Hinga and Mahoney 2010;Larson 2000;Eccles and Gootman 2002;Posner and Vandell 1994;Shernoff and Vandell 2008;Pierce et al. 1999;Mahoney and Stattin 2000;Smith et al. 2009;Cano et al. 2021) who have studied what factors improve the quality of afterschool programs from different perspectives (e.g., youth experiences, curriculum, and staff and educator quality), especially when there are undergraduate students involved as educators or facilitators. The expressed perceptions and experiences of our undergraduate participants may help others striving to enhance engagement within a particular learning community. ...
Article
Full-text available
Our qualitative study is a deep exploration into the underexamined notion of engagement with respect to undergraduate students who took up the role of co-researchers within an afterschool program designed to engage young students about environmental issues and sustainability practices. This research program is based on a community-based literacies framework that addresses all members (program leaders, graduate students, undergraduate students, and young students) as co-learners. This study explores the largely unknown experiences of undergraduate students in informal learning contexts, which broadly center youth experiences. We took a critically framed approach (i.e., mindful of institutionalized inequities–known as a kind of silencing—for traditionally marginalized populations) in our analysis of interview responses from 11 undergraduate students involved in an afterschool environmental program for young students living in a Latinx neighborhood in central California. Our analysis involved a two-phase process that began with a general thematic exploration of transcribed interviews followed by in-depth, microlevel transcription of salient instances regarding community engagement. Responses suggest that the community-based context enabled a deep engagement founded on shared cultural practices, life experiences, and engaging in disciplinary projects (e.g., building and planting an edible garden). This study is a contribution to the long-needed insights into the importance of community engagement and leadership experiences during undergraduate learning, particularly for traditionally marginalized students. Findings may be informative to educators and researchers striving to transform college experiences for diverse student populations.
... Observation is one strategy that offers insight into the learning that can take place through making. This insight can be used as evidence for outside monitoring or evaluation as well as for internal program reflection or formative staff development (Smith et al., 2009). In educational settings, observation tools are well-established as a means for evaluating learning. ...
... Buldu, 2010). It also has utility as formative assessment for educators engaged in professional learning to align practice with staff quality measures (Smith et al., 2009). Finally, because observations are focused on what the learners are actually doing, data are grounded in learning processes afforded by the program. ...
Article
The maker movement has grown rapidly in educational contexts during the past decade. The field now needs valid, reliable measurement tools to capture learning that occurs in maker-based activities. In this study, facilitators and researchers from five organizations engaged in a codesign process to develop a tool measuring how learners seek and share resources while making. We describe our process of collaboratively developing, testing, and revising the Seek & Share Resources Tool and the coding protocol with strong validity. Two researchers coded 50 videos depicting learners ages 0-14 engaging in diverse maker-based activities yielding high inter-rater reliability for the tool. The Seek & Share Resources Tool offers a valid, reliable measure of learning behavior in making and our process may be replicated in future research of rigorous measurement tool design.
... The ability of youth programs to implement recognized best practices in their field is often expressed in terms of program quality. High quality programs include elements such as the performance and behavior of program staff; the existence of positive and supporting relationships; youth feeling a sense of belonging and developing self-efficacy; the availability of opportunities for active learning and the acquisition of new skills; and youth having the ability to make decisions, develop a sense of independence, and have a voice (Bowles & Brand, 2009;Durlak, et al., 2010;Grossman, Campbell, & Raley, 2007;Larson, Eccles, & Gootman, 2004;Larson, Rickman, Gibbons, & Walker, 2009;Sibthorp, Paisley & Gookin, 2007;Smith et al., 2009;Vandell, et al., 2007). In general, higher quality programs are those that are believed to produce increased outcome achievement, thus having a positive impact on the lives of youth (Garst, Browne & Bialeschki, 2011;Sheldon, Arbreton, Hopkins, & Grossman, 2010;Smith, Devaney, Akiva, & Sugar, 2009). ...
... High quality programs include elements such as the performance and behavior of program staff; the existence of positive and supporting relationships; youth feeling a sense of belonging and developing self-efficacy; the availability of opportunities for active learning and the acquisition of new skills; and youth having the ability to make decisions, develop a sense of independence, and have a voice (Bowles & Brand, 2009;Durlak, et al., 2010;Grossman, Campbell, & Raley, 2007;Larson, Eccles, & Gootman, 2004;Larson, Rickman, Gibbons, & Walker, 2009;Sibthorp, Paisley & Gookin, 2007;Smith et al., 2009;Vandell, et al., 2007). In general, higher quality programs are those that are believed to produce increased outcome achievement, thus having a positive impact on the lives of youth (Garst, Browne & Bialeschki, 2011;Sheldon, Arbreton, Hopkins, & Grossman, 2010;Smith, Devaney, Akiva, & Sugar, 2009). ...
Article
Full-text available
The purpose of this study was to pilot the use of the CPQA Camper Survey, a camper self-report survey to assess summer camp program quality. The survey is based on the best practices identified in the American Camp Association’s Camp Program Quality Assessment (CPQA) short form (American Camp Association, n.d.). Best practices are organized into 5 subscales on the CPQA: staff behavior; emotional safety; camper choice, planning, and reflection; learning at camp; and nature. The CPQA Camper Survey asked youth campers at 5 different overnight camps to report on their perceptions of how often camp program quality best practices occur at summer camp. Results from the surveys collected showed that for each of the 5 subscales, campers’ average camp program quality ratings tended to cluster around the upper end of the rating scales. Respondents’ answers were consistent across the questions comprising each subscale, which indicates that the questions in each subscale reliably measure the same construct or idea. The results of this study show that the CPQA Camper Survey is one tool that camp directors and administrators can use as part of their program improvement processes to assess the quality of their programs, and thus improve the quality of the camp experience.
... Keywords: youth mentoring, evidence-based practice, quality rating and improvement systems, organizational readiness for change, program quality improvement, qualitative methods a result, observational tools to assess the quality of youth program settings and point-of-service interactions have been developed as the basis for encouraging staff to form learning communities that focus on areas of program improvement and accountability (Smith et al., 2009;Yohalem & Wilson-Ahlstrom, 2010). Youth program quality improvement efforts typically involve first using such an assessment to evaluate program features and practices, then reviewing the assessment data to establish improvement goals, and finally engaging staff in targeted training and technical assistance to enhance performance (Smith et al., 2006;Spielberger et al., 2009;Yohalem et al., 2009). ...
... Another approach to quality is focusing on "coexistence and correspondence between staff practices and youth experience that is likely to produce positive developmental change" (Smith et al., 2010, p. 359). Indeed, focus on staff behavior and practice has been shown to elevate the quality of youth experience (Smith, Devaney, Akiva, & Sugar, 2009). ...
Article
Full-text available
Using themes (such as the Wild West or Survivor or Star Wars) in camp settings may enhance the quality of camp activities and youths’ overall camp experience. We evaluated the effect of theming camp experiences on the quality of subjective experiences of campers. Campers (N = 231) in 3 sessions of a residential 4-H camp participated in the study. One camp session was fully themed (all activities used tangible and intangible props, cues, and imaginary story contexts), a second was partially themed (intangible cues and stories only), and the third was not themed. Questionnaires measuring the quality of immediate subjective experiences (N=1,847) were completed following each of 8 activity sessions (e.g., climbing, fishing, swimming). Campers also completed a questionnaire for the purpose of overall camp evaluation at the end of their camp sessions. Activity-level data were analyzed using linear mixed modeling techniques. Ordinary least-squares regression was used to analyze campers’ overall camp experiences. Results at the activity level revealed significant theme-by-activity interaction effects. At the camp level, a hypothesized causal sequence linking theme to likelihood to recommend was supported.
... In the first section of this paper, we attempt to reframe the emerging QIS policies in the expanded learning field and their core components as performance-based accountability systems (PBAS), drawing on recent work from the Rand Corp. (Camm & Stecher, 2010;Stecher et al., 2010). The body of QIS policies for the expanded learning sector have received early treatments (Smith, Akiva, Devaney, & Sugar, 2009;Yohalem et al., 2012), but the field lacks a well-specified nomenclature and framework for describing and advancing the work. We extend the earlier PBAS framework as a design standard to guide development of PBAS in the expanded learning and other fields. ...
Technical Report
Full-text available
This paper includes: (1) An update to lower-stakes accountability theory and nomenclature with four case studies, (2) empirical evidence about reliability of quality ratings composed to represent a program site using longitudinal data from the Palm Beach county QIS, and (3) guidance about how to align benchmarks and incentives in the lower-stakes accountability designs.
Article
The existence of multiple purposes for site visits can explain some of the variation seen in the design and use of site visits in practice. In some instances, site visits are one element of a larger evaluation design where results are blended with data from a variety of sources. In other instances, site visits stand alone as the entire evaluation. This chapter discusses three case studies from the perspective of evaluation practitioners that are aligned with the typology presented in the previous chapter by Chapman Haynes, Murphy, and Patton. They illustrate three distinct conditions where site visits have been used in evaluation. The cases selected include two site visits that are part of a suite of numerous evaluation activities that comprise the evaluation design and one case where the site visit comprises the entire evaluation. The chapter concludes with five cross-cutting themes that form practical considerations for incorporating site visits into evaluation practice.
Article
Full-text available
Caring communities support the healthy growth of young people by fostering caring youth-staff relationships as well as a sense of connectedness to the people and norms within that setting. Out-of-school-time (OST) programs may be uniquely situated to serve as caring communities, particularly if staff are trained to facilitate caring activities and employ an ethic of care when interacting with youth. These processes can also be described as program design and staff implementation. Program design, which refers to the structured aspects of a program, differs from implementation, or the ways staff interacts with youth throughout the program, because design factors are typically robust to differences in individual staff members’ style. Implementation, on the other hand, varies with respect to the individual staff member. The purpose of this study was to examine the effects of a two-part staff training intervention focusing on program design and staff implementation on youths’ sense of caring community. Findings from this study and their implications for managers of OST programs are discussed.
Article
Full-text available
A unique observational data set was used to explore quality at the point of service in after-school programs. Staff practices in after-school settings were represented on a series of unidimensional scales closely indexed to staff behavior. In order to account for heterogeneity of staff performances, pattern-centered methods were used to construct profiles of common staff practices. Results revealed six pedagogy profiles that were classified in terms of three broad types of performances delivered by after-school staff: (1) positive youth development, (2) staff-centered, and (3) low-quality. Staff membership in these profiles was not related to youth-staff ratio. However, results revealed significant differences between the profiles on the content of the offering and the age of youth in the setting.
Article
Research on teacher-child relationships, classroom environments, and teaching practices provided the rationale for constructing a system for observing and assessing emotional and instructional elements of quality in early childhood educational environments: the Classroom Assessment Scoring System (CLASS). The CLASS provides a framework for observing key dimensions of classroom processes, such as emotional and instructional support, that contribute to quality of the classroom setting from preschool through third grade. This article provides information about the development, field testing, and use of this instrument in prekindergarten. Data from a national sample of 224 prekindergarten classrooms in 6 states are presented to provide reliability and validity information. The full range of the scale was used for the majority of ratings. Ratings reflected generally positive impressions of the classroom environment and teacher-child interactions. Factor scores from the CLASS were related to the Early Childhood Environmental Rating Scale (ECERS) total score and most strongly related to the ECERS interactions and language-reasoning subscales. Implications for policy and professional development from prekindergarten to third grade are discussed.
Article
The High/Scope Institute for IDEAS began in the early 1960s as a summer camp program and is now a dynamic learning program for teens that emphasizes working with them in an environment which supports emotional, social, and intellectual development. The High/Scope model for adolescent programs is based on the following principles: (1) adolescents need a safe psychological environment; (2) adolescents need a chance to make new beginnings; (3) adolescents need program activities designed to relate to their developmental stages; and (4) adolescents need to be given genuine responsibility. This book describes the High/Scope model in the following chapters: (1) "The High/Scope Institute for IDEAS Program--The Beginnings and Evolution," giving the history of the institute along with its philosophy and long- and short-term benefits; (2) "Who Are the Participants and What Do They Need?" focusing on developmental characteristics, tasks and developmental gains, and meeting adolescent needs; (3) "Creating a Supportive Context for Learning," outlining the program schedule and daily routine and also discussing psychological and physical safety, expectations, and adult participation; (4) "The Educational Approach," dealing with the five essential components of choice, active learning, plan-do-review, cooperative learning, and leadership development; (5) "Learning Potential Within the Institute Setting"; and (6) "Lessons Learned: Beyond the High/Scope Institute Model." (Contains 21 references.) (BGC)
Article
This report covers the second year of a 3-year process evaluation of the Prime Time Initiative of Palm Beach County, Florida, a system-building effort to strengthen the availability and quality of after-school programs in the county. During the past two decades, the after-school field has expanded enormously. This growth has occurred partly in response to increasing concern about developmental and achievement gaps between low-income children, especially those of ethnic minority backgrounds, and their more advantaged peers, although uncertainty about the role of after-school programs in closing those gaps remains. Opportunities to participate in constructive after-school activities are still more limited in low-income communities than in more affluent communities, and questions remain about the effects of after-school programs on children's development and academic achievement At the same time, as the field has evolved and grown, practitioners and policymakers are learning that programs are more likely to have effects when they address multiple developmental domains, are of high quality and led by professional staff, and engage children on a regular and sustained basis. Improving quality remains challenging, however, as the field is still beset by problems of unstable funding and staffing as well as difficulty developing realistic expectations and quality standards for a diverse array of providers. Intermediary organizations such as Prime Time can be a critical resource for bringing together diverse perspectives, advocating for and developing quality standards, and linking programs with needed supports and services, including professional development for staff, to meet those standards (e.g., Halpern, Spielberger, & Robb, 2001; Johnson, Rothstein, & Gajdosik, 2004; Yohalem, Wilson-Ahlstrom, & Yu, 2005). Chapin Hall's primary responsibilities in the second year of its evaluation were to observe both the continuing development of the program improvement system and the implementation of additional activities to develop a countywide system of supports and resources for out-of-school-time programs, and to provide feedback on progress. (Contains 1 table and 15 footnotes.) [This evaluation was produced with the support of Prime Time.]
Article
The psychometric properties of the revised Early Childhood Environment Rating Scale (ECERS-R) were examined using 202 Colorado child care centers. A factor analysis revealed that the ECERS-R does not measure seven distinct aspects of quality, as asserted by the developers of the ECERS-R, but instead measures one global aspect of quality. This suggests high redundancy among the 43 items in the ECERS-R. We selected subgroups of items randomly, by identifying easy-to-administer items, and by having practitioners select items they deemed to be especially important to child-care provider quality. Comparison of scores on these different subgroups of items to the overall ECERS-R scores revealed that the subgroups provided highly similar measures of quality. Relationships between scores based on each of the subgroups of items and the overall ECERS-R scores to staff credentials and staff-to-child ratios were comparable. Results suggest that much smaller subgroups of items provide information that is very similar to information generated by administering the full ECERS-R. Findings are interpreted in light of growing use of the ECERS-R in high-stakes settings.