Content uploaded by Curtis J. Mearns
Author content
All content in this area was uploaded by Curtis J. Mearns on Jun 04, 2014
Content may be subject to copyright.
Evaluation that works. Programs that work.
The Lograr Program Evaluation
A New Mexico Public Education Department Initiative
28 September 2011
Curtis J Mearns Ph.D.
8300 Carmel Ave. NE, Suite 601 | Albuquerque, NM 87122 | 505-828-0082
2
Lograr Program Evaluation
I. Introduction
The purpose of this evaluation is to document successes and challenges of Lograr as
implemented between May 2010 and June 2011. This task is complicated because the long-term
outcome of graduation was never intended to be met on the timeline of the grant. As most
educators are aware, the path to graduation after a series of failures or dropout events takes long
committed programming to re-engage students. This grant rather than help students graduate
directly, planned to support local efforts already in existence and to bolster community resources
that serve students needing assistance to graduate. The goal was to use ARRA money to help
existing community based organizations further develop effective graduation oriented
collaborations that would continue past the grant funding period.
An additional complication is that this sort of initiative is more about process than
outcome. The processes in this case varied from strong parent education efforts that empower
parents to support their own students, to fundamental educational efforts that provide direct
services to students. Long-term outcomes would be graduation, but the short-term measureable
outcomes were student connectedness to their school and their community. This evaluation will
address process and short term outcomes. Additionally, this report includes a section on cost.
II. Project Description
Lograr was funded by the American Recovery and Reinvestment Act. The funding was available
for only one fiscal year. This opportunity posed several challenges for the New Mexico Public
Education Department.
1. Without time for a needs assessment showing local specificity, it would be difficult to
develop an effective program to address student needs within their communities that
would also span numerous communities across the state.
2. A state agency would take far too long to develop a program, hire staff, and consummate
a program than the short window of opportunity for spending these funds.
3. There were not enough funds available to spread equitably around the state.
Program planners hypothesized that community organizations would behave like small
businesses that can plan and quickly implement services with close knowledge of their client’s
needs. Additionally, program staff selected an organizational networking model (e.g. Plastrik &
Taylor, 2006) that took advantage of local knowledge of needs and corresponding capacity to
meet those needs. ARRA funds awarded locally would then expand collaborative partnerships
and increase capacity of existing programs. However, local communities would have to contract
with Lograr to complete their scope of work thus allowing program staff to select communities
more likely to be successful.
Essentially, local organizations with established track records were funded to add partners to
their networks that increase the network’s ability to support students who face numerous barriers
to graduation. Such local organizations were thought to be able to:
1. Address local needs more skillfully (by knowing the needs and local resources)
2. Scale up operations more quickly than the PED and
3. Had the potential, through new partnerships, to develop sustainable increased capacity
after the grant funds expired.
3
Project designers elected to use short term measures that track improved student perceptions of
students school experiences since research (Davalos et.al., 1999) has shown a connection
between school perceptions and graduation.
Lograr – Self Described (with edits for readability)
The Lograr Institute was a statewide initiative that identified and worked with selected
communities with high Hispano/Latino student populations and correspondingly high drop-out
rates with the goal of eliminating the achievement gap for Hispano/Latino students. The purpose
of this project was to assist communities in capacity-building of families, students, school (Pre-
K-12 and post-secondary) leaders, school teachers and staff, community organizations, local,
state and federal government agencies, unions and community businesses to engage, navigate
and negotiate school and community resources in order to ensure the elimination of the
achievement gap. Program staff anticipated that this initiative would benefit and engage
approximately 2,000 families across New Mexico.
The Lograr Institute acted as an “umbrella” unit, for the larger Graduate New Mexico project.
Contractors across the state for Lograr were knowledgeable of local activities occurring with
respect to Graduate New Mexico and were positioned to act as “brokers” for those services by
directing families and communities to the right people and organizations best suited to support
their students graduation efforts. The Lograr Institute was to address the challenges faced by
New Mexico’s Hispano/Latino students by building capacity among local stakeholders in order
to reduce barriers to educational attainment and promote innovations and reforms that ensure
elimination of the achievement gaps.
Contractors for The Lograr Institute were to act as a resource and relationship broker. They were
responsible for the following:
Improving community engagement, volunteer recruitment.
o Community education;
o Developing business - corporate citizenship, apprenticeship, mentorship, tutoring
(volunteer and fund recruitment); and
Building community capacity;
Expanding successful programs,
Connecting students and families to resources (e.g. districts to resources, students to
districts and high schools),
Local organizations began operations after two waves of proposal review. PED reduced start up
lag to three and six months. When the PED announced proposal opportunities, local
organizations developed proposals, and the PED awarded contracts. Brief Project descriptions
can be found in Appendix A
III. Logic Model
Logic models concisely display program needs, implementation activities, and relations among
elements and outcomes (short and long term). Figure 01 show where Lograr fit into the larger
Graduate New Mexico scheme. Figure 02 shows the program elements of Lograr.
4
Figure 01. Graduate NM Logic Model
5
Figure 02. Lograr Logic Model
6
IV. Evaluation Questions
Evaluation questions focus the research on issues of primary importance to the program and
align with the logic model. See Appendix B for a table aligning the research questions with the
Scope of Work.
1. Did Lograr increase the number of organizations collaborating to support student
graduation?
2. What was the nature of these collaborations?
3. Did students increase their commitment to graduate? (awareness, knowledge, attitudes)
4. Were there other student benefits?
V. Methods
The short funding time frame prohibited pre-post evaluation methods for most measures. This
evaluation used Social Network Analysis methods with convenience sampling to address
evaluation questions associated with collaboration. Although convenience sampling has some
limitations, evaluators thought the short evaluation timeline would prohibit developing
relationships required for exhaustive sampling of Lograr contractor partners. Partners had no
obligation to respond since most collaborated with Lograr contractors without any remuneration.
This process required three rounds of data collection. The first was called a Resource Inventory
where Lograr contractors named the organizations they planned to collaborate with. The next
data collection round, Collaboration Characterization Instrument (CCI), used the contact
information from the previous round to ask Lograr contractors the nature of the relationship, they
currently had with those organizations mentioned in the Resource Inventory. This round also
allowed for updating current collaborators. The update led to round three of data collection, also
using the CCI, that went to the partners of Lograr Contractors. The CCI was developed to fill
some theoretical gaps that exist in the current collaboration literature. It offers a categorical
typology along with measures of collaborative timeframe and collaborative intensity. The CCI
contained additional retrospective questions allowing for modest estimates of pre-program
interactions.
Evaluators used the Hemmingway Measures of Adolescent Connectedness (HMAC) to
determine student engagement and a supplemental retrospective pretest that described conditions
before Lograr. The HMAC was designed for high school aged youth and was offered in both
Spanish and English. Some Lograr contractors programs allowed exhaustive sampling, while
others only allowed for sampling those attending at the time of survey administration.
Study limitations stem from the short project duration and the amount of depth of information
that can be gathered in a short period of time. Additionally, one site failed to respond to
evaluation requirements despite repeated effort to engage that site. Estimates described in this
report may underestimate actual impact if this site managed to carry out plans similar to other
sites. Finally, comparisons among sites may not be valid because of community differences.
VI. Results
Results are organized into four sections. The first describes the Lograr Contractors. The second
describes Lograr contractor collaborative efforts. Section three describes Student Engagement
while section four discusses unplanned outcomes.
7
a. Lograr Contractor Program Types
Lograr programs took 3 forms. The first sought to impact students directly. For example,
program staff may have looked for dropouts, re-engaged dropouts, found additional services for
dropouts, etc. The second program type would have engaged families to advocate for their
student. For example, staff may have provided families with lists of supporting organizations,
parent educational opportunities, or re-introductions to the original school setting.
Table 01. A table showing reporting sites and program type
Organization* Program Type
1. Aztec School District Student
2. BoCES -NMSU Student & Family
3. Elev8 - Albuquerque - Business Family
4. Elev8 - Anthony - Families Student & Family
5. Grants School District Student & Family
6. Rio Arriba County Student & Family
7. Robert F. Kennedy Charter High School Student & Family
8. Santa Fe Partnership for Communities & Schools Student & Family
9. Southwest Creations Collaborative Student
10. West Central Community Development Group Student
11. YDI GED Enhancement Student & Family
* Contracts 1 & 2 were operated by the same organization in different regions of the state. Although they began as
distinct efforts, success in one grant lead program staff to modify program practices to take advantage of lessons
learned. Some methods were extremely broad, making impact difficult to measure. Including youth or families
impacted by the broad effort would likely increase the numbers.
8
a. Lograr Contractor Collaborative Efforts
Figure 03 shows the collaborations existing before Lograr was funded based on CCI Partner responses. It is critical that these
collaborations existed in advance of the funding or the Lograr model may not have been as successful. Organizations 3 and 4 were the
same organization operating in different communities. As the program developed resource sharing from community to community
became common. The graphic omits this cross community resource sharing.
Figure 03. Pre Lograr Collaboration
* Labeled boxes in left margin without connections will appear as new connections in the next figure.
9
Figure 04 shows the collaboration after Lograr funded each program. Eight of ten contracting agencies increased the number of
collaborators working toward the Lograr goals. Additionally, interconnections among contractors strengthened in one community
(contractors 8, 9, 10). Overall there were 17 new connections, a 28% increase in collaborative partners.
Figure 04. Post Lograr
10
The CCI identifies the nature (types) of collaboration among partners and contractors. Table 02
shows the types of collaborations organizations identified themselves as. Table 02 shows results
from 2 perspectives, the Lograr contractor and the partners. Both contractors and partners
reported information sharing as the most common type of collaboration. Contractors indicated
that client referral as a common form of collaboration. Respondents described only a few
relationships as funding based.
Table 02. Types of Collaborative Relationships
Organizational Relationship Types Contractors
Describing
Partners % Partners
Describing
Contractors %
Early Networking, but No Work 5 10.9 0 0.0
Client Referral 13 28.3 1 4.3
Information Sharing 18 39.1 15 65.2
Funding Based 2 4.3 3 13.0
Work Sharing 6 13.0 2 8.7
Policy Development 0 0 0 0
Other 2 4.3 2 8.7
Total Responding 46 100 23 100
Additionally, the CCI demonstrates a collaboration time frame along the continuum shown in
Figure 05. The continuum is descriptive in nature and NOT prescriptive. A group designed as a
temporary taskforce cannot improve itself by aspiring for a longer relationship. A taskforce has
a different charge than some other collaboration timeframe. To change it would essentially
require a new charge and other changes is the formalization (intensity) of the relationship..
Figure 05. Collaboration Characterization Instrument’s Timeframe Continuum
11
Finally the CCI measures the Intensity of a collaboration by estimating the level of formalization
that the organizations have developed between themselves. There were four indicators of this
construct that were summed and divided by 4 to produce this indicator.
One organization provides TA to other (often, regularly, occasionally, rarely)
One organization can use funding incentives to change behavior of others (strongly agree,
agree, disagree, strongly dis-agree)
Our organization Receives client information (detailed form with documentation, without
documentation, general with documentation, general without documentation).
Our organization Sends client information (detailed form with documentation, without
documentation, general with documentation, general without documentation)
For the analyses that follow a factor analysis confirmed the variables and produced standard z
scores for each case. These were then adjusted for ease of interpretation.
Figures 06 shows collaboration type and the timeframe relationship (crudely) by varying the size
of the arrows; the larger the arrow, the longer the interaction timeframe. Note that the contractor
role is identified as “Multiple” as they would have multiple types of relationships. The arrows
indicate the magnitude of the magnitude of the timeframe in Figure 05 and of the intensity of the
interactions in Figure 06.
F
F
i
g
ure 06. Ti
m
m
eframe & Col
l
l
aboration T
yp
p
e
12
13
Figure 07 shows interaction type and intensity of interaction. The CCI conceptualizes intensity in terms of formalization of process as
a way to identify sustainable activity directed toward relationship maintenance.
Figure 08. Intensity Investment & Collaboration Type
14
Results for project level of community connectedness (CCI) of contractors are shown in Table 03
with CCI timeframe and intensity values and associated student and parent impact. Timeframe
scores range from 1 to 5, the higher the number the longer the relationship is planned to last.
Intensity scores range from 1 to 6, the higher the number the more formalized the relationship.
The table shows generally low levels of collaboration with organizations 4 and 9 showing
moderate levels of collaboration.
Table 03. Contractor Self Evaluation
Organization Community
Connectedness
Timeframe
Community
Connectedness
Intensity
N Students
Impacted N Parents
Impacted
1. Site A 1.6761,
(2, .511) 2.4248,
(2, .022) 874 772
2. Site B 2.2359,
(2, 1.303) 2.2180,
(2, .270 NP 74
3. Site C 2.8647,
(6, .676) 1.9646,
(6, .282) 19 NP
4. Site D 3.8076,
(9, .880) 2.9858,
(9, 1.752) 75 205
5. Site E 1.9448,
(7, .562) 1.7218,
(7, .175) 21 30
6. Site F 2.0786,
(7, .759) 2.4698,
(7, .664) 32 NP
7. Site G 2.3895,
(9, .758) 2.4349,
(9, .272) G = 211 326
8. Site H 2.8043,
(5, 2.804) 2.3621,
(2.362) 420: G =
122 240
9. Site I 3.4156,
(4, .746) 5.0735,
(4, .467) 175: G = 26 NP
10. Site J * * 107: G = 34 104
11. Site K 2.0008,
(12, .735) 1.5078,
(12, .414) 16 45
Totals 1950 1796
Timeframe and Intensity measures are reported using adjusted mean zscore, (number of participants, standard
deviation)
* Program design did not align with this data collection effort (i.e they had no partners)
NP = No Planned Impact.
NA = Participant never responded to evaluator communications.
G = Number of known graduates
It makes sense to ask if the longer timeframe and more intense partnerships lead to larger
impacts; however, no correlations of among those variables showed significant results. To
follow that question in more depth researchers developed a linear combination of time and
intensity and an additive impact variable. With these new variables added to the original set of
correlations, still no relationships were found (Table 04).
15
Table 04. Correlations of implementation features with impacts.
Community
Connectedness Measure N Students
Impacted N Parents
Impacted Total
Impact
Timeframe Pearson Correlation -0.248 -0.278 -0.268
Sig. (2-tailed) 0.489 0.437 0.455
N 10 10 10
Intensity Pearson Correlation 0.112 -0.0534 0.035
Sig. (2-tailed) 0.757 0.882 0.923
N 10 10 10
Combined
Connectedness
Pearson Correlation -.052 -.185 -.117
Sig. (2-tailed) .887 .609 .749
N 10 10 10
Results for project level of community connectedness (CCI) of partners are shown in Table 05
with CCI timeframe and intensity values and associated student and parent impact. This table
shows that partners of contractors characterized the partnership to be at low levels of interaction
as did the contractors; however, organizations 9 and 11 show moderate levels in terms of the
timeframe, while organizations 4, 6, 7, 9 show moderate levels of intensity (formalization).
Table 05. Partner Perspective
Organization Community
Connectedness
Timeframe
Community
Connectedness
Intensity
N Students
Impacted N Parents
Impacted
1. Site A NR NR 874 772
2. Site B NR NR 74
3. Site C 1.5425,
(2, .192) 2.1098,
(2, 1.238) 19 NP
4. Site D 2.3189,
(4, .474) 2.6377,
(4, .531) 75 205
5. Site E 2.0729,
(4, .962) 3.0979,
(4, .809) 21 30
6. Site F 2.2728,
(2, 2.083) 3.3759,
(2, .568) 32 NP
7. Site G 1.7441,
(4, 1.375) 2.9834,
(4, 1.047) G = 211 326
8. Site H 3.1570,
(4, .088) 2.4040,
(4, 1.135) 420: G =
122 240
9. Site I 2.4879,
(2, .045) 3.9141,
(2, 2.149) 175: G = 26 NP
10. Site J * * 107: G = 34 104
11. Site K 2.7411,
(2, 1.112) 1.9953,
(2, .216) 16 45
Totals
1950 1796
* Program design did not align with this data collection effort (i.e. they had no partners)
Timeframe & Intensity measures are reported using: adjusted mean z score, (N of participants, standard deviation)
NR= No Response, NA= Did not participate in evaluation, G = Number of known graduates
16
The table below shows a relationship between intensity (formalization) and total impact.
Although this is a relatively strong relationship in Table 06, it was not seen in its component
pieces, nor in the contractor data. While this may be a valid observation, more research would
be required to eliminate the possibility of a family wise statistical error.
Table 06. Intensity and Total Impact
Community
Connectedness Measure N Students
Impacted N Parents
Impacted Total
Impact
Time Pearson Correlation .523 .107 0.598
Sig. (2-tailed) .183 .801 0.117
N 8 8 8
Intensity Pearson Correlation .061 -.186 0.756
Sig. (2-tailed) .887 .658 0.030
N 8 8 8
Combined
Connectedness
Pearson Correlation .384 -.129 .151
Sig. (2-tailed) .347 .761 .721
N 8 8 8
Next the analysis asked if types of collaborations had longer timeframes or more intense
interactions. Statistical analyses show that, in general for both contractors and partners, funding
relationships were associated with longer timeframe and higher intensity relative to other types
of collaborative relationships. However these finding may not be reliable since the sample size
of the funding collaboration group was only five. For contractors, longer timeframe was
associated with performance dependency relative to client referral. Contractors involved in
client referral collaborations reported higher intensity than those collaborations involved in
information sharing. For partners, funding based relationships showed higher intensity than
information sharing.
The two perspectives, contractor and partner, provide important information. From a contractor
perspective that manages multiple relationships, they must develop processes that allow for
varying types of relationships and varying types of timeframes and intensity. Partners may have
fewer concerns about other partners, so intensity of the relationship may seem more onerous than
for the central coordinating organization. Each table in Table 07 shows the relationship from a
different perspective.
For contractors, funding based relationships generally had longer timeframes (formalization)
than client referral had shorter timeframes. Contractors also reported more intensity when there
were funding based relationships relative to other types of relationships. Additionally,
contractors reported more intense relationships within client referral relationships than for other
information sharing relationships.
17
Table 07. Collaboration Type Comparisons for Timeframe and Intensity by Perspective.
Contractor’s Perspective: Time Frame
Var 1 Mean 1 SD 1 N Var 2 Mean 2 SD 2 N sig
Funding
Based 3.36 1.01 5
Client
Referral 2.32 1.20 14 .045
Performance
Dependency 3.31 0.94 5
Client
Referral 2.32 1.20 14 .056
Contractor’s Perspective: Intensity
Var 1 Mean 1 SD 1 N Var 2 Mean 2 SD 2 N sig
Client
Referral 2.66 1.10 14
Information
Sharing 1.83 .72 23 .013
Funding
Based 4.65 1.14 5
Client
Referral 2.66 1.10 14 .000
Funding
Based 4.65 1.14 5
Information
Sharing 1.83 .72 23 .000
Funding
Based 4.65 1.14 5
Not
Currently
Collaborating 1.89 .32 3 .000
Funding
Based 4.65 1.14 5
Performance
Dependency 2.58 1.44 5 .001
Partners Perspective: Intensity
Var 1 Mean 1 SD 1 N Var 2 Mean 2 SD 2 N sig
Funding
Based 3.84 1.52 3
Information
Sharing 2.54 1.00 11 .059
Collaborations Summary
The Lograr plan depended o the ability of contractors to make use of existing local partner
resources and for contractors to expand the number of partners they have. Contractors increased
the total number of partners by 28% to 60. Additionally, the study showed that information
sharing and client referral were the most common types of partnerships and that they required
relatively shorter timeframes and low levels of formalization (intensity). Contractors scaled up
their local resources as state project staff planed. However, it may be that the collaborations had
no impact on student efforts to graduate. The remaining part of the report addresses student
impacts.
b. Student Commitment to Graduate
Table 08 shows many subscales indicating areas of student life that support their graduation and
connectedness to school. Each scale varies from one to five with 3 being a midpoint. The three
rows highlighted in green are essential to graduation; however, individual programs often sought
to impact familial support (in blue). The two rows in purple show students’ thoughts about their
future were slightly better than their thoughts about their present. It is interesting that students
thoughts about their neighborhood and reading remained low.
18
Table 08. Hemmingway Measures of Adolescent Connectedness
Subscale Mean
Neighborhood 2.8
Friends 3.7
Parents 3.6
Siblings 3.6
School 3.3
Peers 3.3
Teachers 3.3
Self-in-the-Present 3.5
Self-in-the-Future 3.6
Reading 2.8
Number taking survey 288
To evaluate the values in the table below, the top and bottom categories were compared using a
Chi2 with equal expected frequencies. Significant differences are shown in green when they
support the conclusion that Lograr activities improved student engagement. There were no
findings that Lograr hindered student engagement. One finding was significantly neutral (no
color). The robust pattern of results is best interpreted as positive program effects on student
attitude, awareness, and actions about their ability to graduate.
Self-reported student attendance improved while thoughts of dropout declined. Relationships
with adults and peers improved. This is particularly critical since research (Chavez et. al. 1994)
has shown that when students have friends who are graduating, they are more likely to graduate.
While increases in participating in school were offset by decreases, students reported increased
time spent reading and doing homework. Students felt their reading, homework, and
participation had improved as well. An overall measure of education activities (reading,
participation, homework) indicator asked students about their effort to reach graduation. This
measure showed the largest effect.
Table 09. Connectedness Supplement
Feelings About School More
Important Stayed
the same Less
Important
Percent N Percent N Percent N
Reading & Studying Feelings 56.1% 161 37.3% 107 6.6% 19
Participating in School Feelings 30.3% 87 57.5% 165 12.2% 35
Doing Homework Feelings 56.3% 162 39.6% 114 4.2% 12
19
Relationships Increased Stayed
the same Gotten
worse
Percent N Percent N Percent N
Relationship with friends 54.4% 156 42.9% 123 2.8% 8
Relationship with adults 37.2% 107 55.6% 160 7.3% 21
Miscellaneous Measures More
Frequent Stayed
the Same Less
Frequent
Percent N Percent N Percent N
Thoughts of drop out 10.4% 30 31.1% 90 58.5% 169
School Attendance 42.7% 123 46.2% 133 11.1% 32
Actions Increased Stayed
the same Decreased
Percent N Percent N Percent N
Reading Studying Time 38.8% 111 46.2% 132 15.0% 43
Participating in School Time 24.0% 69 51.0% 147 25.0% 72
Doing Homework Time 41.5% 120 44.3% 128 14.2% 41
My effort to reach graduation 70.1% 202 25.0% 72 4.9% 14
This general improvement in student perceptions of school aligns with research (Davalos et.al.
1999) showing that successful Hispanic students have higher perceptions of their school
experience than do dropouts. Many of the Lograr contractors sought to improve graduation
through improving these perceptions of school by teaching students and families about the
school context and by improving student social skills that support graduation from high school.
c. Benefits and Costs
Benefit to cost analysis depends a lot on the perspective that a reader brings to the analysis.
Table 10 shows various ways a reader may wish to examine student outcomes. Row one shows
unintended outcomes of the program from state project staff perspective. Although there may
have been hope that some students would graduate, it was designed with a longer term view.
Row one shows that the cost per graduate if the reader discounts all other program effects
documented above is $3347. Row two shows the number of students impacted by Lograr as
reported by the contractors with the cost of $312 per student. Row three shows the cost per
student if you take the number of self-reported students impacted and multiply that times .40.
The multiplier in row three is an estimate of the program effect size based on the average
difference between the most positive and most negative percent values of each of the
connectedness survey supplement questions with significant findings. Row three essentially
adjusts the number of students based on the estimate of the projects effect size with a final cost
of $781 per student. Row four shows the number of individuals impacted when adding
individual students and the number of families together. This method would expect that only one
student from each family was impacted. However, Mearns (1999) has found that programs that
address families not only impact target students but also siblings, so the potential impact of
family programs can exceed the number reported here when counting siblings.
20
Table 10. Benefit and Cost Analysis from various perspectives.
Evaluation Method N Students Impacted Cost per Student
1. Grads only* YDI GED = 211
Aztec =26
Rio Arriba = 122
Grants = 34
$609,108/393 =
$1549 per youth
2. Students impacted (Self
Report) 1950 $609,108/1950 =
$312 per youth
3. Students impacted (Est.
based on youth Survey) 1950 X .40 = 780 $609,108/780 =
$781 per youth
4. Students impacted including
family Ns 3746 $609,108/3746 =
$163 per youth
* This value is based on four sites whose design made collecting graduation data possible. Other
sites likely produced graduates, but data could not be collected systematically because of
program design.
VII. Conclusions
The following points address the original research questions.
1. Lograr increased the number of organizations collaborating to support student graduation
by adding 17 new organizations to the existing 43, a 40% increase.
2. The collaborative nature of partnership mostly entailed client referral and information
sharing.
3. Students increased their commitment to graduate as evidenced by self reported
improvements in actions like attendance, studying, homework completing and
participation.
4. Other student benefits included increased connectedness to school.
The most significant outcome was the number of graduates of the Lograr program. The program
design depended on the ability of local community resources to coordinate themselves to respond
to local student educational needs. At least three sites were so effective they produced a total of
182 graduates. Additionally, the program’s requirement for contractors to expand their referral
network likely produced enough student supports to re-establish successful student re-
engagement in school that will last past the end of the grant period.
21
References
Davalos, D. B., Chavez, E. L., and Guardiola, R. J. (1999) The Effects of Extracurricular
Activity, Ethnic Identification, and Perception of School on Student Dropout Rates, Hispanic
Journal of Behavioral Sciences. V 21, 61-77.
Chavez, E. L., Oetting, E. R., & Swaim, R. C. (1994), Dropout and delinquency: Mexican-
American and Caucasian non-Hispanic youth, Journal of Clinical Child Psychology. V 23, 47-
55.
Mearns, Curt (1999). Brief Family Counseling Intervention: Cibola Cluster, 1997-98, ERIC,
ED448374
Plastrik & Taylor (2006), Net Gains. downloaded from
http://www.impactalliance.org/ev_en.php?ID=44035_201&ID2=DO_TOPIC, on 8/16/2011.
22
Appendix A
Figure 09 shows a summary of key program features, early planning resources considered, and
final counts of collaboration partners. The discrepancy between what was planned and the final
count shows the amount of work required to develop final collaborative arrangements.
Beginning with a large number of potential collaborators and carefully developing relationships
means that some potential partners may not be good matches for the current task at hand. The
table only shows those partners electing to return survey. Overall there was a 29% response rate
for the Partner CCI, nearly three times that expected for a mail out survey.
Figure 09. Key program features and development of partnerships
Organization Contract Responsibilities (partial) Communit
y
Resources
Considered
Communit
y
Resources *
1. Site A Engage 500 families to help facilitate attainment
of their child's high school diploma's…..
Educate 1000 community members on closing
the achievement gap and strategies for success.
Recruit 10 - 15 community leaders and train on
educational issues to raise awareness of
educational expectations and to reduce the
achievement gap.
Hold 2 community meetings around student
success and institutional systemic barriers and
engage in community-driven solutions.
3 2
2. Site B Meet with at least 30 business and community
organizations.
Meet with at least 100 individual community
members.
Identify and develop 10-15 community members
to act as leaders on educational issues.
Reach out and educate 1500-2000 community
members on the educational achievement gap.
10 2
23
Organization Contract Responsibilities (partial) Communit
y
Resources
Considered
Communit
y
Resources *
3. Site C Host labor and education experts from the Sloan
Work and Family Research Network to build
community capacity on school engagement.
Create a manual for public & private sector
employers outlining the benefits of and steps to
take to provide parent involvement programming
and workplace flexibility for families.
Facilitate a 2 day training for 30 teachers,
counselors, and community based service
providers on resources and techniques for
integration college-track planning and attainment
for Hispano/Latino families.
12 6
4. Site D Organize parents and community members as
school educational partners and as trained leaders
of a broad based community organization with
the power to implement a cradle to career
strategy…..
increase student readiness, performance, and
school engagement.
Create high expectations for post high school
education.
Connect with APS School District school
administration for support and buy-in of Lograr
goals.
10 9
5. Site E Oversee the facilitation of an 8 module parent
workshop with 30 parents of middle school aged
youth in Las Cruses.
Oversee the facilitation of a youth leadership
program with 20 high school aged youth from
Las Cruses.
Oversee the facilitation of 5 cross cultural
learning and sharing dialogues with the first one
being with the youth participants only; parents,
school teachers, staff, and administrators,
community members, and business partners will
participate.
14 9
24
Organization Contract Responsibilities (partial) Communit
y
Resources
Considered
Communit
y
Resources *
6. Site F Conduct a Reach Out to Dropout Walk in the
West Mesa & Rio Grande school clusters.
Identify students that did not return in Fall 2011
and re-engage them and their families through
Graduate New Mexico.
Connect with APS School District school
administration for support and buy-in of Lograr
goals.
Increase the capacity of the support system within
the community by implementing strong ties
between schools, families and organizations
around school engagement.
10 7
7. Site G Enhance Services to meet needs of 80-100 newly
identified Graduate New Mexico students by
implementing 4 GED preparation classes.
Train 4 Parent allies/mentors to assist the students
navigate barriers to graduation.
Connect with APS School District school
administration for support and buy-in of Lograr
goals.
Increase the capacity of the support system within
the community by implementing strong ties
between schools, families and organizations
around school engagement.
16 7
8. Site H Engage 240 high-risk students to educate on the
financial value of staying in school and engaging
in towards educational achievement.
Work directly with 120 Carlos Vigil Middle
School students on In School Suspension, Out of
School Suspension, and demonstrating excessive
truancy.
Educate 800 [community] members on closing
the achievement gap and strategies for success.
6 4
9. Site I Enhance services to meet needs of 120 students
in credit recovery program to assist in gradation.
Increase advisement
Provide afterschool tutoring
Inform families about appropriate services
designed to help them graduate from high school
or reengage if they have dropped out.
4 4
25
Organization Contract Responsibilities (partial) Communit
y
Resources
Considered
Communit
y
Resources *
10. Site J Provide services to meet needs of 200 students in
credit recovery program.
Inform families about appropriate services
designed to help them graduate from high school
or reengage if they have dropped out.
0 0
11. Site K Assist 75 students and their families to identify
and access community resources, school policies
and education advocacy.
Provide case management services for 75
students.
Provide resources for 75 students and their
families.
Provide follow-up for 75 students and their
families.
15 12
Totals 100 62
* Contractor Perspective
**These 2 organizations provided services within their own organization. One conceptualized
departments as collaborators and responded accordingly.
Appendix B
Scope of Work and Research Question Alignment
Figure 10. Evaluation Measures/Methods/Instruments
Objective
Summary Evaluation
Measure/Method/Instrument Location in
Report Research Question
1. Build or
Develop
Mental Models
Documents describing each
primary partners’
relationship to Lograr and
their own partners
(DoView)?
Program
Description
Graphic provide
on page 5 of
this report.
2. Provide
Resources
Resource Inventory (to
primary Partners). A
request for partners to list
resources they offer will roll
out in 2 stages: 1) open
ended survey, 2) list form
with other responses
collected.
Results
(planned vs. self
reported)
RQ1. Did Lograr
increase the
number of
organizations
collaborating to
support student
graduation?
26
Objective
Summary Evaluation
Measure/Method/Instrument Location in
Report Research Question
3. Obtain
Community
Feedback
Primary Partners provide a list
of their partners (secondary
partners). Apex administers
Levels of Community
Linkage (LCL) survey as a
means of providing primary
partners with feedback
Results (Part of
table above –
add column of
partners -
Planned vs,
responding to
survey))
RQ2. What was
the nature of these
collaborations?
4. Improve
partner
relationships
Apex administers self
evaluation activity using
Levels of Community
Linkage with Primary
Partners
Results: Table
of Community
Collaboration
Measure
Site level results
sent to contractors.
5. Student
connectedness
Apex administers Measure
of Adolescent
Connectedness as an
indicator of ultimate
outcome. This measure
should be a better
indicator of success than
satisfaction alone.
Concurrently Apex will
gather satisfaction
information as well.
Results: RQ3. Did students
increase their
commitment to
graduate?
(awareness,
knowledge,
attitudes)
RQ4. Were there
other student
benefits?
