Content uploaded by Ben Silliman
Author content
All content in this area was uploaded by Ben Silliman on Jul 13, 2016
Content may be subject to copyright.
Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=wmfr20
Download by: [129.130.251.16] Date: 22 September 2015, At: 08:17
Marriage & Family Review
ISSN: 0149-4929 (Print) 1540-9635 (Online) Journal homepage: http://www.tandfonline.com/loi/wmfr20
Youth Program Quality Survey: Youth Assessment
of Program Quality
Benjamin Silliman & Walter R. Schumm
To cite this article: Benjamin Silliman & Walter R. Schumm (2013) Youth Program Quality
Survey: Youth Assessment of Program Quality, Marriage & Family Review, 49:7, 647-670, DOI:
10.1080/01494929.2013.803010
To link to this article: http://dx.doi.org/10.1080/01494929.2013.803010
Published online: 23 Sep 2013.
Submit your article to this journal
Article views: 111
View related articles
647
Youth Program Quality Survey:
YouthAssessment of Program Quality
BENJAMIN SILLIMAN
Department of Youth, Family, and Community Sciences,
North Carolina State University, Raleigh, North Carolina, USA
WALTER R. SCHUMM
School of Family Studies and Human Services, Kansas State University,
Manhattan, Kansas, USA
The Youth Program Quality survey, a 24-item survey of youth par-
ticipant perceptions of program quality, based on program elements
identified by the National Research Council (NRC) and Institute of
Medicine, was developed and piloted with 614 younger teens (ages
10–13 years) and 486 older teens (ages 14–17 years) who attended
4-H camps and conferences. Evidence is presented for content and
construct validity. In addition, the overall instrument demonstrated
high reliability, as measured by Cronbach’s alpha, generally rang-
ing from .70 to .96, and moderate subscale reliability of .60 or
higher on four factors in the younger sample (
α
= .60 or higher on
four factors in the older sample). Results are discussed in terms of
goodness of fit to the National Research Council model, the signifi-
cance of youth voice in assessment of program quality, developmental
differences in perceptions of quality, and programming design.
Recommendations are offered for practice, research, and policy.
KEYWORDS adolescence, children, family life education, program
evaluation
INTRODUCTION
Program quality, particularly contextual features that facilitate support and
challenge, is the best predictor of positive youth development outcomes
Address correspondence to Benjamin Silliman, Box 7606, Department of Youth, Family,
and Community Sciences, North Carolina State University, Raleigh, NC 27695, USA. E-mail:
ben_silliman@ncsu.edu
Marriage & Family Review, 49:647–670, 2013
Copyright © Taylor & Francis Group, LLC
ISSN: 0149-4929 print/1540-9635 online
DOI: 10.1080/01494929.2013.803010
Downloaded by [] at 08:17 22 September 2015
648 B. Silliman and W. R. Schumm
such as preventing risks and building capabilities (Catalano, Berglund, Ryan,
Lonczak, & Hawkins, 2002; Durlak, Weissberg, & Padan, 2010; National
Research Council [NRC] & Institute of Medicine [IOM], 2002; Gambone, Klem,
& Connell, 2002; Mihalic, 2004; Yohalem & Wilson-Ahlstrom, 2010). These
features have been identified consistently in studies of research-based pro-
grams of extended length and likely apply to planned programs of any
length (e.g., one time, short term, and long term). Measures of quality are
often not included in research reports (Yohalem & Wilson-Ahlstrom, 2010)
and are less often assessed among community-based programs, perhaps due
to the limited funding and expertise or lack of sponsor demand for such
results (Eccles & Templeton, 2000; Kahn, Bronte-Tinkew, & Theokas, 2008;
Yohalem & Wilson-Ahlstrom, 2010). Moreover, published reports typically
reflect adult rather than youth participant perceptions of program quality
(Urban, 2008; Yohalem & Wilson-Ahlstrom, 2010). Thus, an accurate, inex-
pensive, youth-friendly measure of program quality might help program
leaders more effectively listen to youth participants and share youth percep-
tions of program quality with key stakeholders, including families with youth
considering or enrolled in such programs. In addition, demonstrating
program quality may augment accountability to and education of program
sponsors regarding “what makes a difference” in promoting program-specific
and broader developmental outcomes for children and adolescents.
This research report describes the development and testing of the Youth
Program Quality (YPQ) survey, a participant rating of experiences in youth
programs. YPQ focuses on the contextual features of youth programs shown
to promote positive youth development by the NRC and the IOM (2002).
These features include safety, support, social norms, social inclusion, skill-
building opportunities, self-efficacy, structure, and synergy with family and
community. The survey was developed to provide a reliable, research-based,
user-friendly tool to help youth leaders incorporate youth voice in assess-
ment of the features in the process of program development, monitoring,
and improvement.
Youth Program Paradigms
A diverse array of youth development programs serve millions of young
people from preschool through early adulthood, nationwide and internation-
ally (NRC & IOM, 2002; Mahoney, Larson, Eccles, & Lord, 2007). These pro-
grams operate on implicit or intentional goals that emphasize risk prevention
(Catalano, et al., 2002; Cicchetti & Rogosch, 2003) and/or positive youth
development (Lerner, etal., 2005; Roth & Brooks-Gunn, 2003; Theokas etal.,
2005). Prevention and developmental models reflect diverse theory and prac-
tice models with many parallel objectives and strategies but share the convic-
tion that “development can be redirected onto a positive course by changing
the pattern of person-to-context relations” (Schwartz, Pantin, Coatsworth, &
Szapocznik, 2007, p. 122).
Downloaded by [] at 08:17 22 September 2015
Youth Program Quality Survey 649
Both risk prevention and positive youth development approaches draw
on social ecology or developmental systems theories at the conceptual level
and use social learning, cognitive behavioral, and other theories at the pro-
grammatic level (NRC and IOM, 2002; Schwartz, etal., 2007). Both approaches
recognize that contextual features or processes in the program, program
outcomes, and the larger social environment are reciprocally related. In the
risk prevention model, for instance, a program offering physical and emo-
tional safety and support provides a respite from risks such as bullying and
violence and a context for learning conflict-resolution skills that might serve
as a protective factor in handling future threats. From a positive youth devel-
opment perspective, youth–adult partnerships grounded in adult support,
skill-learning opportunities, and norms of responsibility and service enable
youth to gain connection, character, competence, caring, confidence, and
eventually make contributions to their communities (Lerner etal., 2005).
Youth Program Quality
YPQ is widely regarded as the best predictor of risk prevention and positive
developmental outcomes (Catalano etal., 2002; Durlak etal., 2010; NRC &
IOM, 2002; Larson & Walker, 2009; Lerner etal., 2005; Yohalem & Wilson-
Ahlstrom, 2010). Program participation alone is not sufficient, and programs
low on qualities such as support or skill-building are unlikely to reduce
risks or promote positive outcomes (Larson, 2000; Moore & Hamilton, 2010).
Rather, the interactive climate within programs (Larson, 2000; Durlak etal.,
2010; Larson & Walker, 2009) as well as the ecology of sponsoring organiza-
tions and communities (Ramey & Rose-Krasnor, 2012; Edna McConnell
Clark Foundation, 2012) provide the grounding for long-term program
effectiveness.
YPQ is described in at least three ways. A comprehensive view of YPQ
(Edna McConnell Clark Foundation, 2012; Reisner, White, Russell, & Birmingham,
2004) includes the broader organizational capacity for management; engage-
ment of youth, families, and community partners; staff development; evi-
dence-based activities; evaluation; and sustainability, while offering a sup-
portive and challenging climate for youth. While acknowledging the
importance of a broader context, the organizational view of YPQ exempli-
fied in the High/Scope model (2012) focuses on organizational or site poli-
cies, expectations, and access, together with a point-of-service elements of
safety, support, interaction, and engagement indicators. Similarly, Huang and
Dietel (2011) identified key elements of good afterschool programs as (1)
goals, reflected in structured experiences; (2) leadership, characterized by
horizontal and vertical communication and collaboration; (3) capable and
caring staff; and (4) opportunities for skill-building and self-efficacy. Durlak
and DuPre (2008) linked quality of management and environment to other
dimensions of program fidelity such as dosage, participant responsiveness,
and staffing quality.
Downloaded by [] at 08:17 22 September 2015
650 B. Silliman and W. R. Schumm
A program-oriented view of YPQ, typical of most reports of program
quality, focuses on the environment and interactions “on the ground” within a
program (NRC & IOM, 2002; Gambone et al., 2002; Yohalem & Wilson-
Ahlstrom, 2010). Collating findings from two decades of prevention and enrich-
ment program research, the NRC and IOM (2002) identified eight contextual
features within programs that are most likely to promote positive developmen-
tal outcomes: (1) safety, physical, and emotional; (2) support from adults and
peers; (3) social norms, including respect and service; (4) social inclusion via
acceptance and belonging; (5) skill-building opportunities; (6) self-efficacy
and contribution to others; (7) structure appropriate to audience and setting;
and (8) synergy between youth programs and family, neighborhood, and com-
munity life. Subsequent research supported and expanded this description.
For instance, Roth and Brooks-Gunn (2003) identified quality components as
active citizenship, opportunities for skill-building and leadership, promotion of
wellness and future-time perspectives, fun activities, sustained youth–adult
relationships, and a sensitive and inclusive environment.
Lerner (2004) identified “The Big Three” features of effective youth-
serving programs as (1) positive and sustained relationships between youth
and adults, (2) activities that build important life skills, and (3) opportunities
to use life skills as participants and leaders. Similar findings are reflected in
research reviews on informal science (Friedman, 2008), residential camps
(Henderson, Bialeschki, Thurber, Schueler, & Marsh, 2007; Scanlon, 2006),
afterschool programs (Granger, Durlak, Yohalem, & Reisner, 2007), and civic
engagement (Michelson, Zaff, & Hair, 2002). Yohalem and Wilson-Ahlstrom
(2010) noted that recent research across teen and preteen programs under-
lines the roles of positive relationships, supportive climate, and consistent
engagement of youth as contributors to social and academic outcomes.
Sabatelli, Anderson, Kosutic, Sanderson, and Rubenfeld (2009) identified
four key concepts, namely, safety, support, challenging activities, and mean-
ingful relationships, from literature on quality. They created and critically
reviewed a 65-item questionnaire, and then conducted a pilot test with after-
school programs (serving 6th–12th graders) as the units of analysis, applying
exploratory factor analysis, which yielded three factors: supportive involve-
ment, challenge and involvement, and emotional well-being and safety. A
subsequent field test applying confirmatory factor analysis with a similar
audience identified several three- and two-factor models, depending on the
analytical approach. Process use was also applied to program improvement.
More recently, Tiffany, Exner-Cortens, and Eckenrode (2012) tested a
20-item measure with 12- to 18-year-old youth in urban afterschool programs
that yielded four key components of youth program participation: personal
development, voice and influence, safety and support, and community
engagement. Again, components identified resemble those featured in the
NRC-IOM report (2002), with emphasis on program input and community
involvement for older youth.
Downloaded by [] at 08:17 22 September 2015
Youth Program Quality Survey 651
Youth Voice in Program Evaluation
Program quality is best documented using multiple measures from multiple
sources over multiple points through the course of a program (Yohalem &
Wilson-Ahlstrom, 2010). Because youth voice is a critical element in the phi-
losophy and practice of positive youth development (Camino & Zeldin, 2002;
Hamilton, Hamilton, & Pittman, 2004), youth perspectives represent a critical
starting point in quality assessment. Moreover, participant readiness and
responsiveness is a critical indicator of program fidelity (Carroll etal., 2007).
Positive youth development and YPQ are most often defined and measured by
adults, whose views of programs vary somewhat from youth. For instance,
Urban (2008) found that although youth generally agreed with adult practitio-
ners, policymakers, and researchers on the most critical traits of positive youth
development programs (e.g., youth-friendly environments, opportunities for
leadership and life skills practice, balance of individuality and connectedness),
youth and researchers held different priorities on youth input and evaluation.
From a practical perspective, youth are the most authentic evaluators of
their experience. Youth leaders who know how to evoke and discern youth
feedback can be more effective in planning, monitoring, and improving pro-
gramming, thus aiding the achievement targeted outcomes. Youth and adults
offer complementary voices (Larson, Walker, & Pearce, 2004) that result in
more accurate and complete view of a program’s quality and impact. In the
absence of multiple measures, a brief but thorough participant survey pro-
vides an efficient, if not sufficient, measure of whether and how a program
is working. Although positive voices for program quality may not be inferred
as evidence for program outcomes, negative views might serve as formative
evidence for program fidelity and corroborative evidence that summative
outcomes rest on sound practices (Yohalem & Wilson-Ahlstrom, 2010).
This research reports on the development and testing of a measure of
YPQ, as perceived by preteen and teen participants in several 4-H camps
and conferences. Prior studies with smaller samples indicated high program
quality and instrument reliability as well as positive knowledge growth and
behavioral outcomes (Silliman, 2008; Silliman & Shutt, 2010). This research
incorporates a broader sample and then applies exploratory factor analysis
to determine salient elements of program quality for younger and older
adolescents.
METHODS
Samples
The YPQ survey was implemented over a 3-year period as part of routine
program evaluation for weeklong and 3-day citizenship and service con-
ferences and in weeklong resident camps sponsored by a 4-H program in
Downloaded by [] at 08:17 22 September 2015
652 B. Silliman and W. R. Schumm
the southeastern United States. 4-H is a nationwide youth development
organization serving 5- to 19-year-olds through programs in citizenship,
science, and healthy living, among other issues (4-H National Headquarters,
2012). Participation in the research study was voluntary, but 95% of par-
ents and youth completed a human subjects consent form previously
approved by the first author’s institutional review board. No parents or
youth refused participation, but 5% omitted the consent form in event
registration. Only data from consenting parents and youth were used in
this study. To reduce heterogeneity in age and development within groups,
participants younger than 10 years or older than 17 years were not included
in the analyses presented here.
In the younger sample (ages 14–17) of 614 participants, there were
more boys (52.0%) than girls (48.0%). Average age was 11.67 years (standard
deviation [SD] = 1.04), with more participants aged 12 years (30.5%) than
other ages (10 years, 16.3%; 11 years, 26.9%; 13 years, 26.4%). Average grade
level was 6.86 (SD = 0.86), with more participants in grade 7 (37.5%) than
other grades (grade 5, 4.2%; grade 6, 31.9%; grade 8, 26.4%). There were 542
missing cases on the “grade” item. Most participants had attended a 4-H
camp (90.1%) rather than a 4-H conference (9.9%). Most data were missing
for ethnicity (n = 556), but most of those who did respond were White
(84.5%) or Black (10.3%), with a few other responses (Asian, 3.4%; other,
1.7%). Data on ethnicity were only collected in 2008 at two sites; data on
grade level were collected only in 2007 and 2010 at three sites.
In the older sample (ages 14–17) of 486 participants, there were more
girls (66.5%) than boys (33.5%). Average age was 15.19 years (SD = 1.07),
with more participants aged 14 years (35.0%) than other ages (15 years,
26.5%; 16 years, 23.5%; 17 years, 15.0%). Average grade level was 10.27
(SD = 1.17), with more participants in grade 9 (28.8%) than other grades
(grade 7, 1.7%; grade 10, 27.1%; grade 11, 23.7%; grade 12, 18.6%). There
were 78 missing cases on the “grade” item. Most participants had attended a
4-H conference (82.1%) rather than a 4-H camp (17.9%). Most data were
missing for ethnicity (n = 409), but most of those who did respond were
White (71.4%) or Black (18.2%), with a few other responses (White, Hispanic,
1.3%; Hispanic, 6.5%; Asian, 1.3%; other, 1.3%).
Measures
Event registration forms provided demographic information used in group
comparisons. The YPQ was designed to evaluate participant perceptions of
experiences in short- and long-term youth programs. Three items were
developed for each of eight contextual features that promote positive youth
development (NRC & IOM, 2002), resulting in a 24-item survey (see Table 1
for items by trait). Survey items were reviewed for reading comprehension
by panels of youth and adult leaders and piloted with youth groups before
Downloaded by [] at 08:17 22 September 2015
Youth Program Quality Survey 653
TABLE 1 YPQ Survey with NRC-IOM (2002) Traits Identified in Brackets
Strongly
disagree Disagree Agree
Strongly
agree
[Safety]
1. I felt safe from being hurt or
injured………
ABCD
2. I was embarrassed or put down……… A B C D
3. Activities promoted healthy habits……… A B C D
[Support]
4. Adults listened to what I had to say……… A B C D
5. I felt comfortable going to adults for
advice………
ABCD
6. Other kids cared about me……… A B C D
[Social Norms]
7. Conflicts between people were a
problem………
ABCD
8. I learned to work with others as a
team………
ABCD
9. Serving others and volunteering was
important
ABCD
[Social Inclusion]
10. I felt like I didn’t belong……… A B C D
11. All kinds of kids were welcomed……… A B C D
12. I learned to accept differences in
others………
ABCD
[Skill-building Opportunities]
13. Activities taught me to develop a plan to
reach my goals………
ABCD
14. I was challenged to think and build
skills………
ABCD
15. There were opportunities to learn new
subjects
ABCD
[Self-efficacy]
16. I felt that I could make a difference……… A B C D
17. I was encouraged to take
responsibility………
ABCD
18. Perfect performance was more important
than learning from mistakes………
ABCD
[Structure]
19. Rules and expectations were clear……… A B C D
20. Discipline was not too strict, not too
loose………
ABCD
21. Activities were just right for my age……… A B C D
[Synergy]
22. I gained a broader view of 4-H……… A B C D
23. I gained a broader view of the world
beyond my community………
ABCD
24. Activities were related to issues in my
club, my family, my community………
ABCD
drafting a final edition. Youth feedback resulted in changing Likert-type
response options from “never–seldom–often–always” to a “strongly disagree–
strongly agree” continuum. Straightforward statements were preferred over
Downloaded by [] at 08:17 22 September 2015
654 B. Silliman and W. R. Schumm
negatively worded items, which youth and their leaders deemed too confus-
ing for inexperienced respondents. Thus, only four items were negatively
worded or reverse-coded.
Content validity was established on elements of program quality
described by the NRC and IOM report (2002), widely regarded as best prac-
tices in the field (Yohalem & Wilson-Ahlstrom, 2010). Although these internal
assets may be described with different labels, similar elements can be found
in youth programming at camps (Henderson etal., 2007), afterschool pro-
grams (Granger et al., 2007), and informal science programs (Friedman,
2008). A 1999 national study of program quality by the 4-H youth develop-
ment organization identified similar elements, although they were labeled
differently (Peterson, 2001).
Construct validity, the proposition that a measure accurately reflects
what it is intended to measure, was initially established through panel review
and pilot study in which youth and adult volunteers and professionals sup-
ported its face validity or perceived consistency with the construct of pro-
gram quality. Exploratory factor analysis was then conducted to evaluate
observed patterns in the data relative to NRC-IOM model constructs. Separate
analyses were conducted for younger and older adolescent samples consis-
tent with developmental differences in social cognition (Lerner, 2009). Results
of these analyses are reported below.
Procedures
Conference attendees completed paper forms, whereas camp participants
completed either paper forms or used clickers to respond to PowerPoint
slides read by a trained proctor. Monitoring and debriefing discussions with
program staff indicated that participants had few difficulties completing the
survey in either mode. If lower-literacy participants experienced difficulty
with the paper survey, the staff was instructed to read items aloud without
elaboration. About 10 participants required such assistance.
Analysis
Given the developmental differences between teens and preteens (Lerner,
2009), data from program participants 10–13 years of age and 14–17 years
of age were analyzed separately. The 24 items in the YPQ scale were sub-
jected to exploratory factor analysis, using maximum likelihood factoring,
extracting four to six factors in search of the best apparent solutions.
Factoring was conducted both with and without substitution for missing
data; because the solutions were similar, results for the larger sample with
mean substitution for missing data are presented here. A ratio of 20:1 for
subjects-to-items was exceeded, meeting the most strict subject-to-item
requirements recommended (Hair, Black, Babin, Anderson, & Tatham,
Downloaded by [] at 08:17 22 September 2015
Youth Program Quality Survey 655
TABLE 2 Varimax-Rotated Maximum Likelihood Factor Loadings for Items from the YPQ
Survey: Younger Youth Sample (Ages 10–13) (n = 610)
Items
Rotated factor loadings
12 3 4 5h2
5. I felt comfortable going to adults
for advice.
.71 .01 .09 .13 –.05 .54
4. Adults listened to what I had to
say.
.71 .13 .09 .13 .00 .55
8. I learned to work with others as a
team.
.54 .31 .22 –.03 –.15 .46
1. I felt safe from being hurt or
injured.
.52 .17 .03 .15 –.03 .32
6. Other kids cared about me. .51 .17 .16 .04 –.19 .36
11. All kinds of kids were welcomed. .44 .21 .17 .24 –.12 .34
9. Serving others and volunteering
was important.
.43 .28 .18 .15 –.16 .34
12. I learned to accept differences in
others.
.41 .34 .17 .19 –.21 .39
3. Activities promoted healthy habits. .38 .34 .22 .07 .03 .31
14. I was challenged to think and
build new skills.
.15 .56 .20 .18 .07 .41
13. Activities taught me to develop a
plan to reach my goals.
.26 .55 .29 .07 –.05 .47
15. There were opportunities to learn
new subjects.
.27 .47 .19 .25 –.02 .39
17. I was encouraged to take
responsibility.
.21 .36 .30 .13 –.13 .30
16. I felt that I could make a
difference.
.21 .34 .33 .15 –.05 .30
2006, p. 112; Osborne & Costello, 2004). Goldberg and Velicer (2006) indi-
cate that “samples in the 500–1,000 range are preferred” (p. 214) for factor
analyses.
Cronbach’s (1951) alpha, a “widely used estimate of reliability” (Zeller
& Carmines, 1980, p. 59), or internal consistency was used to assess survey
data for each contributing event and scales derived from the factor analysis.
SPSS (Norusis, 2006) was used for computing both the factor analyses and
the alphas.
RESULTS
Factor Analysis for Younger Youth (Preteen) Sample
Exploratory factor analyses were conducted on data from younger (ages 10–13)
and older (ages 14–17) youth. For younger youth, five factors were produced with
eigenvalues > 1.0 with eigenvalues between 1.05 and 6.66 (Table 2). The Kaiser-
Meyer-Olkin (KMO) measure of sampling adequacy was 0.91 with a Bartlett Test
(Continued)
Downloaded by [] at 08:17 22 September 2015
656 B. Silliman and W. R. Schumm
of Sphericity of 3,165.1, df = 276, p < .001. Extracted communalities ranged between
.11 and .60. Attempts to extract fewer factors yielded more residuals (>.05) from
the reproduced correlation matrix, 6% for four factors. Attempts to extract more
than five factors yielded minor factors with loadings on only one item, with 1%
residuals. Use of listwise deletion for missing data yielded similar factors with 6%
residuals (>.05). All items loaded at least 0.32 on at least one factor.
The first factor for the younger youth consisted of nine items related to
NRC-IOM Safety and Health (items 1, 3), Adult and Peer Support (items 4, 5,
6), Social Norms (items 8, 9), and Social Inclusion (items 11, 12) and was
labeled “Positive Emotional Climate” YPQ item numbers are listed after each
factor label for results of Younger Youth and Older Youth factor analyses.
The second factor, “Empowered Skill-building,” included three items from
NRC-IOM Skill-building (13, 14, 15) and two from NRC-IOM Self-efficacy (16,
17). The third factor, “Expanding Horizons,” included items from NRC-IOM
“Synergy” (22, 23, 24). The fourth factor included YPQ items 19, 20, and 21
related to “Structure” and was labeled accordingly. The fifth factor included
four items (2, 7, 10, and 18) from different NRC-I OM categories and was
labeled “Negative Experiences.”
23. I gained a broader view of the
world beyond my community
.13 .23 .71 .17 –.08 .60
22. I gained a broader view of 4-H. .14 .23 .62 .20 –.04 .50
24. Activities were related to issues in
my club, my family, and my
community.
.14 .21 .41 .16 .03 .26
20. Discipline was not too strict, not
too loose.
.16 .15 .17 .57 –.08 .40
21. Activities were just right for my
age.
.19 .14 .33 .49 –.09 .41
19. Rules and expectations were clear. .21 .28 .28 .42 –.06 .38
10. I felt like I didn’t belong. –.18 –.01 –.07 .01 .60 .39
2. I was embarrassed or put down. –.10 .00 –.02 –.03 .52 .29
7. Conflicts between people were a
problem.
–.07 –.00 –.10 –.05 .46 .23
18. Perfect performance was more
important than learning from
mistakes.
.05 –.03 .06 –.04 .32 .11
Eigenvalues 6.66 1.84 1.55 1.08 1.05
KMO sampling adequacy 0.91
Bartlett Test of Sphericity 3,169.1, p < .001 (df = 276)
Reproduced correlation matrix
residuals > .05
3.0%
Note. Primary factor loadings are in bold type. Eigenvalues are associated with unrotated factors only.
Communalities (h2) are extracted communalities.
TABLE 2 Continued
Items
Rotated factor loadings
12 3 4 5h2
Downloaded by [] at 08:17 22 September 2015
Youth Program Quality Survey 657
Factor Analysis for Older Youth (Teen) Sample
An identical factor analysis was conducted for the older sample. Use of an
eigenvalue = 1.0 criterion with maximum likelihood factor analysis yielded
five factors, with eigenvalues between 1.03 and 7.72 (Table 3). The KMO
measure of sampling adequacy was 0.93 with a Bartlett Test of Sphericity of
3,316.3, df = 276, p < .001. Extracted communalities ranged between .10 and
.66. Attempts to extract fewer factors yielded more residuals (>.05) from the
reproduced correlation matrix, 13% for four factors. Attempts to extract more
than five factors yielded minor factors with loadings on only one item, with
3% residuals. Use of listwise deletion for missing data yielded similar factors
with 8% residuals (>.05). All items loaded at least 0.30 on at least one factor,
but several apparent factors included items with loadings < .40. Item 20 was
the only item that did not load at least 0.33 on at least one factor.
The first factor loaded heavily on five items related to “Empowered
Skill-building” and included items from NRC-IOM traits of Skill learning oppor-
tunity (13, 14, 15), responsibility Social Norms (17), and developmentally appro-
priate Structure (17). Removing item 21 (Structure) from the scale only reduced
Cronbach alpha from .79 to .77. Only the items loading > .40 were retained
for the subscale derived from the four items with the heaviest loadings on the
first factor. The second factor loaded primarily on seven items and was labeled
“Positive Values” because the items seemed to focus on YPQ items for Social
Norms (8, 9), Social Inclusion (11, 12), Structure (19), and Safety (1, 3) related
to an inclusive environment in which service and teamwork were valued. The
third factor loaded on four items, including YPQ Self-Efficacy (16) and Synergy
(22, 23, 24) items, and was labeled “Expanding Horizons” because the items
referred to broadening views of 4-H and the world in context of family and
community. Item 16 related to Self-efficacy, also loaded to some extent on the
first and fourth factors but seemed to best support a higher Cronbach’s alpha
for this subscale. The fourth factor loaded on items 4 and 5 and was labeled
“Adult Support,” consistent with the YPQ item. The fifth factor related to
“Negative Experiences” loaded on five items consistent with NRC-IOM
descriptors, but the subscale generated from items 2, 6 (recoded), 7, 10, and
18 had an unacceptably low Cronbach’s alpha (.56).
The factor structures were somewhat similar for the two age groups.
Items 22, 23, and 24 formed a distinct factor; indeed, the third factor was
labeled “Expanding Horizons” in both groups, with item 16 (“Making a
Difference”) loading on the same factor for the older sample. Cronbach’s alpha
for this subscale was a bit higher (.78) for the older group than for the younger
group (.69). The fifth factor for both groups was labeled “Negative Experiences”
and included items 2, 7, 10, and 18 for both groups, although item 6 was
included for the older group. However, for both subscales, Cronbach’s alphas
were low (.56). The relatively low average scores for both Negative Experiences
subscales may suggest that few participants experienced rejection, conflict, or
Downloaded by [] at 08:17 22 September 2015
658 B. Silliman and W. R. Schumm
TABLE 3 Varimax-Rotated Maximum Likelihood Factor Loadings for Items from the YPQ
Survey: Older Youth Sample (Ages 14–17) (n = 486)
Items
Rotated factor loadings
12 3 4 5 h2
13. Activities taught me to develop a plan
to reach my goals.
.64 .15 .30 .16 .11 .55
14. I was challenged to think and build
new skills.
.58 .16 .22 .27 .05 .48
17. I was encouraged to take
responsibility.
.50 .14 .30 .18 .27 .46
15. There were opportunities to learn
new subjects.
.47 .33 .19 .18 .09 .41
21. Activities were just right for my age. .37 .32 .11 .23 .16 .33
12. I learned to accept differences in
others.
.08 .61 .28 .20 .28 .58
11. All kinds of kids were welcomed. .21 .57 .17 .16 .21 .47
9. Serving others and volunteering was
important.
.26 .45 .29 .16 .25 .44
19. Rules and expectations were clear. .26 .36 .09 .11 .19 .25
8. I learned to work with others as a
team.
.21 .35 .32 .28 .18 .39
1. I felt safe from being hurt or injured. .18 .35 .01 .24 .34 .33
3. Activities promoted healthy habits. .17 .33 .18 .33 .16 .30
23. I gained a broader view of the world
beyond my community.
.30 .15 .70 .14 .17 .66
22. I gained a broader view of 4-H. .23 .23 .59 .15 .06 .47
24. Activities were related to issues
in my club, my family, and my
community.
.41 .16 .50 .06 .10 .46
16. I felt that I could make a
difference.
.38 .23 .38 .30 .15 .46
4. Adults listened to what I had
to say.
.22 .17 .09 .66 .10 .52
5. I felt comfortable going to adults for
advice.
.19 .18 .14 .65 .26 .58
20. Discipline was not too strict, not too
loose.
.24 .20 .22 .31 .05 .25
10. I felt like I didn’t belong. −.16 −.14 −.07 −.01 −.59 .40
7. Conflicts between people were a
problem.
.08 −.11 −.15 −.18 −.46 .28
6. Other kids cared about me. .14 .23 .16 .29 .36 .31
18. Perfect performance was more
important than learning from
mistakes.
.06 −.08 −.15 −.10 −.35 .17
2. I was embarrassed or put down. −.07 −.08 .04 −.02 −.30 .10
Eigenvalues 7.72 1.58 1.20 1.07 1.03
KMO sampling adequacy 0.93
Bartlett Test of Sphericity 3,316.3, p < .001 (df = 276)
Reproduced correlation matrix
residuals >.05
5.0%
Note. Primary factor loadings are in bold type. Eigenvalues are associated with unrotated factors only.
Communalities (h2) are extracted communalities.
Downloaded by [] at 08:17 22 September 2015
Youth Program Quality Survey 659
unfair demands, making the items less relevant or meaningful for most partici-
pants. The second factor for the younger participants and first factor for the
older youth (labeled “Empowered Skill-building”) included the same four
items, 13, 14, 15, and 17, whereas item 16 was also included for the younger
sample. For both groups, Cronbach’s alphas were above .70.
For the younger group, items pertaining to both adult (4, 5) and group/
peer acceptance (1, 3, 8, 9, 11, 12) loaded on the first factor (“Positive
Emotional Climate”) together, whereas for the older group, those items formed
the second (1, 3, 8, 9, 11, 12, 19; “Positive Values”) and fourth (4, 5; “Adult
Support”) factors. For the younger sample, a fourth factor (items 19, 20, and
21; “Structure”) did not correspond to factors found for the older sample. For
the older sample, item 20 did not load strongly on any of the five factors.
Items 19 and 21 loaded, for the older sample, on the second and first factors,
respectively.
It appears that for the younger youth, the first factor of the YPQ survey
focused on their acceptance by adults and youth participants, whereas
those issues formed two distinct factors, acceptance by adults and peers,
separately, for the older youth. In terms of personal development
(“Empowered Skill-building”), gaining a broader view of the world
(“Expanding Horizons”), and perceived problems (“Negative Experiences”),
the younger and older participants seemed to report similar cognitive per-
ceptions of the scale items and their interrelatedness. Although most items
loaded primarily on one factor, a few showed relatively equal weights on
two factors. For the younger group, item 3 related to promotion of healthy
habits held about the same weight on “Positive Emotional Climate” (.38) and
“Empowered Skill-building” (.34). Loadings may reflect the role of health as
a context and target outcome or suggest that the item was too ambiguous
to discriminate well between those factors. Likewise, item 16, “I felt I could
make a difference,” had similar (although low) loadings for “Empowered
Skill-building” (.34) and “Expanded Horizons” (.33), a parallel with older
youth on the former, but connected to gaining a broader view of youth club
involvement.
For older youth, item 21 on age appropriateness loaded about equally
for “Empowering Skill-building” (.37) and “Positive Values” (.32), reflecting
the importance of developmental as well as social “fit” for achievement and
relational dimensions of youth programs. Item 24, reflecting the connec-
tion between 4-H activities and club, family, and community life, had high
loadings on both “Adult Support” (.50) and “Empowered Skill-building”
(.41), again reflecting the importance of context for affective and instru-
mental growth. A similar pattern is evident relative to self-efficacy on item
16, “I felt I could make a difference,” which might be seen as an element
on “Empowered Skill-building” (.38) and as a consequence of “Adult
Support” (.38).
Downloaded by [] at 08:17 22 September 2015
660 B. Silliman and W. R. Schumm
Finally, some of the factor analysis results for the younger children may
reflect a methods effect as the younger children’s factors tended to follow the
order of the questions: factor 1, items 3, 4, 5, 6, 8, 9, 11, and 12; factor 2,
items 13, 14, 15, 16, and 17; factor 3, items 22, 23, and 24; factor 4, items 19,
20, and 21; and factor 5, items 2, 7, 10, and 18. Thus, the structure of the
questionnaire may have influenced the responses of the younger 4-H mem-
bers more than it did for the older 4-H members. Table 6 summarizes those
items included in each factor for younger and older groups.
Reliability
Internal consistency as measured by Cronbach’s (1951) alpha for the entire
instrument (24 items) ranged between .70 and .88 for younger participants
and between .77 and .96 for older participants across each 4-H conference
or camp. Combining data from all camps and conferences, alphas were .85
and .82 for the entire instrument (24 items) for the younger and older sam-
ples, respectively. For the two samples, the alpha statistic, along with sub-
scale means and SDs, are presented in Tables 4 and 5.
Overall survey reliability (Cronbach, 1951) was calculated for each con-
ference or camp, with alpha coefficients ranging from .70 to .88 for younger
participants. Reliability of the overall instrument for teens (ages 14–17 years)
at 4-H camps/conferences ranged from
α
= .77 to .96. For the two samples,
reliability estimates (Cronbach, 1951) along with subscale means and stan-
dard deviations are presented in Tables 3 and 5.
TABLE 4 Characteristics for Subscales of the YPQ Survey: Younger Youth Sample
Subscale No. items nMean SD
α
Positive Emotional Climate 9 563 29.13 5.40 0.84
Empowered Skill-building 5 614 15.99 3.13 0.74
Expanding Horizons 3 502 9.31 2.23 0.69
Structure 4 508 9.82 2.08 0.64
Negative Experiences 4 501 7.49 2.58 0.56
TABLE 5 Characteristics for Subscales of the YPQ Survey: Older Youth Sample
Subscale No. items nMean SD
α
Empowered Skill-building 4 447 13.40 2.40 0.77
Positive Values 7 436 24.42 3.38 0.79
Expanding Horizons 4 443 13.39 2.45 0.78
Adult Support 2 474 6.49 1.46 0.71
Negative Experiences 5 440 8.54 2.59 0.56
Downloaded by [] at 08:17 22 September 2015
Youth Program Quality Survey 661
TABLE 6 Specific Items and NRC-IOM Traits for Subscales of the YPQ Survey
Subscale Item Numbers NRC-IOM Trait
younger youth sample
Positive Emotional Climate 1, 3 Safety
4, 5, 6 Support
8, 9 Social Norms
11, 12 Social Inclusion
Empowered Skill-building 13, 14, 15 Skill-building Opportunities
16, 17 Self-efficacy
Expanding Horizons 22, 23, 24 Synergy
Structure 19, 20, 21 Structure
Negative Experiences 2, 7, 10, 18 Negatively worded, recoded
older youth sample
Empowered Skill-building 13, 14, 15 Skill-building Opportunities
17 Social Norms
21 Structure
Positive Values 8, 9 Social Norms
11 Social Inclusion
19 Structure
1, 3 Safety
Expanding Horizons 16 Self-efficacy
22, 23, 24 Synergy
Adult Support 4, 5 Support
Negative Experiences 2, 6, 7, 10, 18 Negatively worded, recoded
DISCUSSION
General Comments
The 24-item YPQ survey was developed to provide a rapid and reliable indi-
cator of youth program quality keyed to participants’ perceptions of their
experience in the program. Internal validity was established for youth aged
10–17, through alignment with research-based best practices, review by youth
and adult leaders, and piloting with teens and preteens. The instrument was
used as part of routine postevent evaluation for several conferences and
camps sponsored by a 4-H program, with data from consenting youth and
parents entered into research analyses. Youth ages 10–17 completed survey
items without difficulty, and program staff found the results useful for pro-
gram accountability and improvement.
Exploratory factor analyses identified five factors with similar themes
for samples of younger youth (ages 10–13) and older youth (ages 14–17).
Scale reliability coefficients (.64 to .84 for younger;
α
= .71 to .79 for older)
were moderate-to-high (Cronbach, 1951) on four of five factor scales for
both groups. Both groups yielded one factor related to “Negative Experiences”
with unacceptably low reliability (
α
= .56). Not all youth program experi-
ences are positive (Larson, Hanson, & Moneta, 2006), but negatives may be
rare enough in this sample to decrease reliability. Alternatively, items might
not have been clear to all respondents. An abbreviated version of the YPC
Downloaded by [] at 08:17 22 September 2015
662 B. Silliman and W. R. Schumm
(nine items) used in the 4-H Science, Engineering, and Technology Youth
Engagement, Attitudes, and Knowledge Survey (LeFleur, Milke, & Sanzone,
2010) yielded an
α
= .83 reliability. Prior studies with subsets of preteens
(Silliman & Shutt, 2010) and teens (Silliman, 2008) found the overall survey
highly reliable (e.g.,
α
= .88 and .87, respectively). Singletary, Smith, and
Evans (2006) reported alphas between .76 and .87 for similar scales com-
pleted by adults. However, data in that study were not factor-analyzed.
Comments on Specific Groups
YOUNGER YOUTH (PRETEEN) SAMPLE
Exploratory factor analyses were conducted separately for younger and older
youth. Five factors emerged for the younger preteen group that differed from
the teen sample in the items and order of three factors. Factor 1, “Positive
Emotional Climate,” includes Social Norms (2), Inclusion (2), and Safety and
Health (2) items on the teen “Positive Values” factor along with both adult
and peer Support items (3). Both the priority and broader content of this
factor is consistent with the salience of identity and social adjustment issues
(e.g., being accepted and accepting others, cooperation in teams) for pre-
and early adolescents relative to older adolescents (Lerner, 2009). Although
belonging, safety, and teamwork are important for all youth, mature teens
tend to have a more secure identity, interactive skills, and life experience that
enables them to adjust to or even remove “speed bumps” that arise in social
groups (Lerner, 2009). This complex of elements in a single factor perhaps
illustrates younger youths’ tendency to view the social environment as a
whole, not distinguishing separate qualities such as belonging, service, or
support. Factor 2, “Empowered Skill-building” includes all but one item load-
ing on the similar factor for older youth. Likewise, Factor 3, “Expanding
Horizons,” mirrors the same factor of the older group. The Self-efficacy
“make a difference” item had similar loadings on Factors 2 and 3 that, like its
relation to Adult Support for the older sample, probably represent an infer-
ence about the effects of learning and broadening experiences. Perhaps this
breadth of application diminishes the distinctiveness of Self-efficacy as a
separate factor. Factor 4, “Structure,” coincides with the same NRC element,
and its emergence as a distinct element for younger youth may coincide with
their developmental needs for clearer rules, takes, and organization. Factor
5, “Negative Experiences,” may be an artifact of wording and coding or
reflection of less-than-ideal experiences that earlier analyses showed to be
atypical of the events as a whole (Silliman, 2008; Silliman & Shutt, 2010).
OLDER YOUTH (TEEN) SAMPLE
Analyses with older youth also identified five factors that were somewhat,
but not entirely, similar to the traits on which items were built. Factor 1,
Downloaded by [] at 08:17 22 September 2015
Youth Program Quality Survey 663
labeled “Empowered Skill-building,” focuses principally on Skill Opportunities
combined with a Social Norm item on responsibility and a Structure item on
age-appropriate activities that are consistent with program qualities that
foster achievement (Gambone etal., 2002; Larson etal., 2004). Skill content
such as “leadership” or “life skills” was not specified as with Urban (2008)
and others. Factor 2, “Positive Values,” includes two items each from Social
Norms, Inclusion, and Safety and one Structure item on clear rules. These
patterns show that teens distinguish a skill-related set from its social context,
which teens view as a complex of prosocial values related to personal growth
(e.g., learning to accept others), group interaction (e.g., safety, teamwork),
and interaction with the community (e.g., serving others). Thus, although
program qualities may be described distinctly—by experts or teens them-
selves—they are experienced holistically. Moreover, values of acceptance
and cooperation are common to both Social Norms and Inclusion and related
behaviors can foster a sense of Safety and consistent Structure. Perhaps the
“mix” of qualities may also shift from one program, setting, or audience to
another, as illustrated by Larson etal. (2006).
The third factor, “Expanding Horizons,” includes only Synergy items for
both groups, making it the most consistent with the NRC-IOM model. The
“broadening” and “in context” elements are reminiscent of autonomy and
connectedness themes evident in youth development and programming
research (Larson, 2000; Lerner, 2009; Urban, 2010). This connection might
also be an artifact of the conference focus on gaining vision and skills for
broader service. Factor 4, “Adult Support,” includes two items on adult roles
together with a Self-efficacy item on “confidence to make a difference,” per-
haps anticipating the consequences of support. However, the “difference”
item had the same loading for Factor 1 and seems to fit better as a correlate
of Skill-building. Encouragement and expectations elements of support
emerge in Factors 1 and 2, yet older youth identify less directive adult roles
of availability and trustworthiness as distinct qualities of their program expe-
rience. This pattern is consistent with previous research on agency (Larson,
2000) and appropriate roles of youth workers (Urban, 2008). As noted above,
“Negative Experiences” represents a “catch-all” of bad experiences and
reverse-worded or coded items.
YPQ priority factors for young adolescents (10–14 years), Supportive
Environment and Empowered Skill-building, are similar to factors identified
by the Youth Development Assessment Device (Sabatelli et al., 2009) in
which two thirds of respondents in both study samples were sixth to eighth
graders. A third factor, Emotional Well-being, included safety and belonging
items found in the YPQ Positive Emotional Climate. The greater number of
factors identified in the YPQ likely reflects the greater diversity of topics (8)
in the YPQ relative to the Youth Development Assessment Device (4). Factor
precedence for older youth more closely resembles participation factors
identified with older youth (12–18 years) by Tiffany etal. (2012) as their
Downloaded by [] at 08:17 22 September 2015
664 B. Silliman and W. R. Schumm
Personal Development factor resembles Empowered Skill-building, Safety
and Support parallels the YPQ Positive Values and Adult Support factors, and
Community Engagement describes the activities that led youth in the current
study to describe programs as Expanding Horizons on self and community.
The youth Voice and Influence factor identified by Tiffany etal. (2012) con-
tains items similar to those in Empowered Skill-building and Positive Values,
but specific items about influence were not asked in the YPQ.
Conclusions
For both younger and older youth, emergent factors broadly resemble scales
derived from the NRC-IOM model, clustering around skill opportunities,
social norms and values, and broadening experiences, with a weaker factor
related to negative experiences. Norm and skill factors emerged in a different
order for each group, with adult and peer support included with normative
behaviors in the view of younger youth. Factors unique to age groups include
Structure among younger youth and Adult Support among older youth.
These patterns are likely reflective of each group’s developmental needs and
capacities. Younger youth may be more dependent on a more explicit
Structure to regulate social and skill-building behavior, whereas older youth
may be more self-regulated, attuned to social norms, and/or involved in
leadership. In that context, Adult Support may be more salient for personal
growth and group dynamics.
Alternatively, age differences may partially reflect emphases in camps,
with a majority younger audience, and conferences, primarily attended by
older youth. Safety, Social Norms, and Self-efficacy did not emerge as sepa-
rate factors, perhaps because they represent broad themes with diverse appli-
cations (e.g., lack of threat or embarrassment, health habits; teamwork, con-
flict resolution, service; responsibility, doing your best, making a difference).
Participants may see these qualities as diffused throughout a program rather
than distinct, viewed as a whole rather as separate qualities. Qualities may
also overlap and complement each other. Qualities such as Structure, Safety,
Social Inclusion, and Social Norms may be most evident when not present
(e.g., disorganization, bullying) and “taken for granted” in smooth-running
programs. The quality of programs surveyed, evident in a high percentage of
positive ratings, moderate-to-high scale reliability, and the relative weakness
of the Negative Experiences factor may diminish the likelihood of detecting
such program qualities. Even when factors have similar items, however,
abbreviated statements such as “I felt safe” or “adults listened” may hold a dif-
ferent meaning or significance for a 10-year-old than for a 17-year-old.
Limitations
The YPQ survey, as a self-report measure, suffers the same limitations as
other self-report methodologies. Participants may have been motivated by
Downloaded by [] at 08:17 22 September 2015
Youth Program Quality Survey 665
social desirability to represent their youth program experiences more posi-
tively than they actually had been. Because most participants were surveyed
toward the end of their program experiences, responses may more accurately
reflect the later phases rather than the earlier phases of their experiences.
Because surveys were administered in groups, unknown biases associated
with the presence of other youth participants may have been present.
Random sampling of target and comparison groups across a greater diversity
of youth programs and over an extended time (e.g., annual assessment of
4-H experience) might yield more generalizable results. However, cumulative
perceptions are also subject to biases such as forgetting and over- or under-
generalization from experiences.
Future research should consider assessing the YPQ along with other mea-
sures to allow for testing of concurrent validity and the selection of contrasting
youth samples to permit assessment of predictive validity. In addition, a two-
step process for generating and testing quality factors using confirmatory as
well as exploratory factor analysis might yield richer, more authentic results.
Recommendations
The YPQ survey shows promise as a practice, research, and policy tool.
Recommendations for further development and use in those areas are offered
below.
RECOMMENDATIONS FOR PRACTICE
In practice, the YPQ appears to be a valid and reliable overall tool for
youth, ages 10–17, to give voice to that which they know best: their experi-
ences in a youth development program. The one-page survey may be
useful to gain rapid feedback from participants, particularly with use of
clickers or online surveys, regarding “how the program works for them.”
Findings of this study suggest that program leaders might expect different
patterns of response for younger and older youth. Leaders may follow up
with a qualitative discussion to gain a deeper, broader view of those expe-
riences. Engagement of youth voice enhances youth leadership (Larson
et al., 2006) and program improvement (Yohalem & Wilson-Alhstrom,
2010). Comparing youth views with staff or expert observations may yield
insights about program conditions or processes. Thus, development of sim-
ilar surveys or observation rubrics for program staff or expert evaluators,
volunteers, and parents would help provide a broader picture of program
quality. Evidence from YPQ cannot be construed as definitive evidence for
specified program outcomes. However, as a part of a formative evaluation
including other views on quality and tentative outcomes, participant feed-
back is a critical element for youth leaders to determine whether the pro-
gram is working as intended.
Downloaded by [] at 08:17 22 September 2015
666 B. Silliman and W. R. Schumm
As a planning tool, the YPQ provides a useful point of reference for
framing content and implementing curricula according to youth develop-
ment best practices. As a training tool, the YPQ has been used to help staff
understand key elements of program quality and to discuss scenarios in
which the NRC-IOM model and results of this research apply (Silliman &
Shutt, 2010). As an evaluation tool, YPQ input speaks to common qualities
of youth programs that may need to be supplemented by other tools to
examine experiences with more specialized components such as science
skills or leadership norms (Yohalem & Wilson-Ahlstrom, 2010).
Tools such as the YPQ are well adapted to process use (Sabatelli etal.,
2009) to inform program improvement, staff training, and marketing or educa-
tion with parents of prospective or active youth program participants, program
sponsors, or the community at large. In an environment of accountability and
comparative value, programs that demonstrate positive participant experi-
ences, especially when corroborated by external evaluations indicating quality,
distinguish themselves as a developmental asset and worthwhile investment.
RECOMMENDATIONS FOR RESEARCH
This research answered several questions about the validity and reliability of
YPQ while inviting additional questions for study. Although the NRC model
was largely affirmed by factor analysis with teen and preteen samples, differ-
ences with the model suggest intriguing conceptual issues for further research:
How well do context features predict diverse program outcomes? How are
these qualities correlated? Are such qualities distinguishable in an integrated
or holistic program? How well can teens or preteens perceive and distinguish
these qualities? Are these the same qualities perceived by program staff or
expert observers? Programmatic issues point to a different set of questions:
How do quality elements, or perceptions of them, vary across programs of
diverse intensity, frequency, duration, setting, audience, and format? How
should negative experiences be examined or factored into the description of
program qualities? How does use of YPQ enhance youth voice relative to
program engagement, perceptions of program quality, or outcome reports?
An equally intriguing set of methodological questions might be explored,
such as How can qualitative descriptions yield additional insight about teens’
and pre-teens’ experiences? How might items be improved or NRC-based
question sets be refined? Do other formats such as observations, games, or
focus groups provide more accurate or less intrusive approaches?
Several evaluation issues present interesting research questions: How can
the YPQ be used most effectively in evaluation planning, training, implemen-
tation, and formative or summative evaluation? How should youth and others’
perceptions of program quality be integrated or weighted? How can the YPQ
contribute to program improvement? How can evidence standards of feasibil-
ity, propriety, accuracy, utility, and accountability (Yarborough, Shultha,
Hopson, & Caruthers, 2011) be applied to measurement of program quality?
Downloaded by [] at 08:17 22 September 2015
Youth Program Quality Survey 667
Future research on the YPQ should address a wider audience, such as
youth from a broader range of ethnic and racial groups, youth participating
with organizations other than 4-H or programs of extended duration, or
youth programs other than those nested in a positive youth development
approach. Comparisons of youth responses by gender, ethnicity, or program
experience could benefit those interpreting as well as those developing pro-
grams. In addition, a thorough meta-analysis of research and practice across
age groups and venues (e.g., afterschool, youth clubs, camps) would help
update and integrate findings across the field.
Each step of progress in measuring program quality reminds practitio-
ners of the importance of quality programs for credible outcomes. Youth
organizations, funding agencies, and their range of stakeholders depend on
quality programs to produce valued results, yet program quality is more
often assumed than assessed. Quality assessment can corroborate quality
outcomes and enrich training, program development and implementation,
marketing and recruitment, and accountability to stakeholders. The YPQ
survey provides a simple yet effective tool that these organizations might put
to work to measure what they value. Finally, youth and family professionals
may want to consider a similar process of evaluating experiences of program
participants, including process use as debriefing with participants at the
close of a program or with staff immediately after the program.
RECOMMENDATIONS FOR POLICY
Development of a reliable, inexpensive, and easy-to-administer survey could
help local programs and extended youth organizations monitor program
quality and use data to plan, improve, and document benefits. Staff commit-
ted to positive youth development philosophy and practice have used the
tool to train staff, engage youth, and report on programs (Silliman & Shutt,
2010). Implementation of such an instrument across events, as annual program
assessments, and including parallel staff, parent, and/or expert assessments
might provide valuable cumulative and comparative data. Given current
practice, the greater challenge may be to gather program quality data and to
seek youth voice (Yohalem & Wilson-Alhstrom, 2010). Nevertheless, every
practical step that enables positive youth development can contribute to rais-
ing the expectations and effectiveness of youth organizations, funders, and
regulators who shape youth program policy.
REFERENCES
Camino, L., & Zeldin, S. (2002). From periphery to center: Pathways for youth civic
engagement in the day-to-day life of communities. Applied Developmental
Science, 6, 213–220.
Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain, S. (2007). A concep-
tual framework or implementation fidelity. Implementation, 2, 40–49.
Downloaded by [] at 08:17 22 September 2015
668 B. Silliman and W. R. Schumm
Catalano, R. F., Berglund, M. L., Ryan, J. A. M., Lonczak, H. S., & Hawkins, J. D. (2002).
Positive youth development in the United States: Research findings on evaluations
of positive youth development programs. Prevention & Treatment, 5. Retrieved
from http://journals.apa.org/prevention/volume5/pre0050015a.html
Cicchetti, D., & Rogosch, F. A. (2002). A developmental psychopathology perspective
on adolescence. Journal of Consulting and Clinical Psychology, 70, 6–20.
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests.
Psychometrika, 16, 297–334.
Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: Review of research on
the influence of implementation on program outcomes and the factors affecting
implementation. American Journal of Community Psychology, 41, 327–350.
Durlak, J. A., Weissberg, R. P., & Padan, M. (2010). A meta-analysis of after-school
programs that seek to promote personal and social skills in children and ado-
lescents. American Journal of Community Psychology, 45, 294–309.
Eccles, J. S., & Templeton, J. (2000). Community-based programs for youth: Lessons
learned from general developmental research and from experimental and
quasi-experimental evaluations. (Urban Seminar Series, Urban Health Initiative).
Cambridge, MA: Harvard University.
Edna McConnell Clark Foundation. (2012). How to assess organizational capacity.
Retrieved from www.emcf.org/evaluation/mckinsey_assessment_tool.htm
Friedman, A. (Ed.) (2008). Framework for evaluating impacts of informal science
education projects. Retrieved from http://insci.org
Gambone, M. A., Klem, A. M., & Connell, J. P. (2002). Finding out what matters for
youth: testing key links in a community action framework for youth develop-
ment. Island Heights, NJ: Youth Development Strategies, Inc.
Goldberg, L. R., & Velicer, W. F. (2006). Principles of exploratory factor analysis. In
S. Strack (Ed.), Differentiating normal and abnormal personality (2nd ed., pp.
209–237). New York, NY: Springer.
Granger, R., Durlak, J. A., Yohalem, N., & Reisner, E. (2007). Improving after-school
program quality. New York, NY: William T. Grant Foundation. Retrieved from
www.wtgrantfoundation.org
Hair, J. F., Jr., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2006). Multivariate
data analysis (6th ed.). Upper Saddle River, NJ: Pearson/Prentice Hall.
Hamilton, S. F., Hamilton, M. A., & Pittman, K. (2004). Principles for youth develop-
ment. In S. F. Hamilton & M. A. Hamilton (Eds.), The youth development hand-
book: Coming of age in American communities (pp. 3–22). Thousand Oaks, CA:
Sage.
Henderson, K. A., Bialeschki, M. D., Thurber, C., Schueler, W. L., & Marsh, P. (2007).
Components of camp experiences for positive youth development. Journal of
Youth Development, 3. Retrieved from wwwnae4ha.org/directory/jyd
High/Scope. (2012). Youth Program Quality Assessment (YPQA). Retrieved from
www.highscope.org/Content.asp?ContentId=308
Huang, D., & Dietel, R. (2011). Making afterschool programs better. (CRESST Policy
Brief). Los Angeles, CA: University of California. Retrieved from www.cse.ucla.
edu/products/policybriefs.php
Kahn, B. A., Bronte-Tinkew, J., & Theokas, C. (2008). How can I assess the quality of
my program? Tools for out-of-school time practitioners. Research-to-Results Brief
2008–10. Washington, DC: Child Trends. Retrieved from www.childtrends.org
Downloaded by [] at 08:17 22 September 2015
Youth Program Quality Survey 669
Larson, R. (2000). Toward a psychology of positive youth development. American
Psychologist, 55, 170–183.
Larson, R., Walker, K., & Pearce, N. (2004). A comparison of youth-driven and adult-
driven youth programs: Balancing inputs from youth and adults. Journal of
Community Psychology, 33, 57–74.
Larson, R. W., Hanson, D. M., & Moneta, G. (2006). Differing profiles of developmen-
tal experiences across types of organized youth activities. Developmental
Psychology, 42, 849–863.
Larson, R. W., & Walker, K. C. (2009). Dilemmas of practice: Challenges to program
quality encountered by youth program leaders. American Journal of Community
Psychology, 45, 338–349.
Lefleur, J., Milke, M., & Sanzone, J. (2010). 4-H Science, Engineering, and Technology
(SET): Youth engagement, attitudes, and knowledge study. Washington, DC:
Policy Studies Associates. Retrieved from www.4-h.org/about/youth-develop-
ment-research/science-program-research
Lerner, R., Lerner, J. A., and Colleagues. (2011). Positive development of youth: Report
of findings of the first seven years of the 4-H Study of Youth Development. Medford,
MA: Tufts University Institute for Applied Research on Youth Development.
Lerner, R. M. (2004). The good teen: Recruiting adolescents from the myths of storm
and stress. New York, NY: The Crown Publishing Group.
Lerner, R. M. (2009). The positive youth development perspective: Theoretical and
empirical bases of a strength-based approach to adolescent development. In C.
R. Snyder and S. J. Lopez (Eds.), Oxford handbook of positive psychology (2nd
ed., pp. 149–163). Oxford, UK: Oxford University Press.
Lerner, R. M., Lerner, J. V., Almerigi, J., Theokas, C., Phelps, E., Gestsdottir, S., … von
Eye, A. (2005). Positive youth development, participation in community youth
development programs, and community contributions of fifth grade adoles-
cents: Findings from the first wave of the 4-H Study of Positive Youth
Development. Journal of Early Adolescence, 25, 17–71.
Mahoney, J. L., Larson, R. W., Eccles, J., & Lord, H. (2007). Organized activities as
developmental contexts for children and adolescents. In J. L. Mahoney, R. W.
Larson, & J. S. Eccles (Eds.), Organized activities as contexts of development
(pp. 3–22). New York, NY: Psychology Press.
Michelson, E., Zaff, J. F., & Hair, E. C. (2002). Civic Engagement programs and youth
development: A synthesis. Washington, DC: Child Trends. Retrieved from www.
childtrends.org
Mihalic S. (2004). The importance of implementation fidelity. Emotional and
Behavioral Disorders in Youth, 4, 83–86, 99–105.
Moore, K. A., & Hamilton, K. (2010). How out of school time program quality is
related to adolescent outcomes. Research Brief. Publication 2010-19. Washington,
DC: Child Trends.
National 4-H Headquarters. (2012). About 4-H: History. Retrieved from www.4-h.org
National Research Council (NRC) and Institute of Medicine (IOM). (2002). Community
programs to promote youth development. In J. Eccles & J. A. Gootman (Eds.),
Board on Children, Youth and Families, Division of Behavioral and Social
Sciences and Education. Washington, DC: National Academy Press.
Norusis, M. (2006). SPSS 14.0 guide to data analysis. Upper Saddle River, NJ: Prentice
Hall.
Downloaded by [] at 08:17 22 September 2015
670 B. Silliman and W. R. Schumm
Osborne, J. W., & Costello, A. B. (2004). Sample size and subject to item ratio in
principal components analysis. Practical Assessment, Research, and Evaluation,
9. Retrieved from www.pareonline.net/getvn.asp?v=9&n=11
Peterson, W. (2001). National 4-H impact assessment project. Retrieved from www.
national4-hheadquarters.gov/about/impact
Ramey, H., & Rose-Krasnor, L. (2012). Contexts of structured youth activities and
positive youth development. Child Development Perspectives, 6, 85–91.
Reisner, E. R., White, R. N., Russell, C. A., & Birmingham, J. (2004). Building quality,
scale, and effectiveness in afterschool programs. A summary of the TASC study.
Policy Studies Associates. Retrieved from www.policystudies.com
Roth, J. K., & Brooks-Gunn, J. (2003). What exactly is a youth development program?
Answers from research and practice. Applied Developmental Science, 7, 94–111.
Roth, J. L., Malone, L. M., & Brooks-Gunn, J. (2010). Does the amount of participa-
tion in afterschool programs relate to developmental outcomes? A review of the
literature. American Journal of Community Psychology, 45, 310–324.
Sabatelli, R. M., Anderson, S. A., Kosutic, I., Sanderson, J., & Rubenfeld, S. (2009). A
validation study of the youth assessment device. Family Relations, 58, 361–372.
Scales, P. C., & Leffert, N. (2004). Developmental assets: A synthesis of the scientific
research (2nd ed.). Minneapolis, MN: Search Institute.
Scanlin, M. (2006). Inspirations: Developmental supports and opportunities of youths’
experiences at camp. Martinsville, IN: American Camp Association.
Schwartz, S. J., Pantin, H., Coatsworth, J. D., & Szapocznik, J. (2007). Addressing the
challenges and opportunities for today’s youth: toward an integrative model
and its implications for research and intervention. Journal of Primary Prevention,
28, 117–144.
Silliman, B. (2008). Taking Congress home: Effects of NC 4-H Congress on youth
behavior and interaction. Journal of Youth Development, 3, Article 0803PA001.
Silliman, B., & Shutt, R.E. (2010). Weaving evaluation into the fabric of youth devel-
opment. Journal of Youth Development, 5, Article 100503FA003.
Singletary, L., Smith, M., & Evans, W. P. (2006). Self-perceived 4-H leader competen-
cies and their relation to the skills youth learn through 4-H youth development
programs. Journal of Extension, 44, Article 4RIB2.
Theokas, C., Almergi, J. B., Lerner, R. M., Dowling, E. M., Benson, P. L., Scales, P. C.,
& von Eye, A. (2005). Conceptualizing and modeling individual and ecological
asset components of thriving in early adolescence. Journal of Early Adolescence,
25, 113–143.
Tiffany, J. S., Exner-Cortens. D., & Eckenrode, J. (2012). A new measure for assessing
youth program participation. Journal of Community Psychology, 40, 277–291.
Urban, J. B. (2008). Components and characteristics of youth development pro-
grams: The voices of youth-serving policy-makers, practitioners, researchers,
and youth. Applied Developmental Science, 12, 128–139.
Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The pro-
gram evaluation standards: A guide for evaluators and evaluation users (3rd
ed.). Thousand Oaks, CA: Sage.
Yohalem, N., & Wilson-Ahlstrom, A. (2010). Inside the black box: Assessing and
improving quality in youth programs. American Journal of Community
Psychology, 45, 350–357.
Zeller, R. A., & Carmines, E. G. (1980). Measurement in the social sciences: the link
between theory and action. New York, NY: Cambridge University Press.
Downloaded by [] at 08:17 22 September 2015
