Content uploaded by Jay A. Mancini
Author content
All content in this area was uploaded by Jay A. Mancini
Content may be subject to copyright.
AIAEE 2004 Proceedings of the 20
th
Annual Conference Dublin, Ireland
Sustaining Community-based Programs:
Examination of Relationships Between Sustainability Factors and Program Results
Jay A. Mancini, Ph.D. and Lydia I. Marek, Ph.D.
Department of Human Development
Virginia Polytechnic Institute and State University
Blacksburg, VA 24061
(540) 231-9816 (office)
(540) 231-7012 (fax)
mancini@vt.edu
lmarek@vt.edu
This research was supported by the U.S. Department of Agriculture, CSREES,
Project #96-EYAR-1-3006 and Project #99-EXCA-3-0707.
Abstract
A conceptual framework for evaluating community-based program sustainability is
presented, and the empirical focus of the study is on the relationship between 7 sustainability
elements (leadership competence, effective collaboration, understanding the community,
demonstrating program results, strategic funding, staff involvement and integration, and
program responsivity) and 3 middle range program results (meeting needs of at risk families,
planning for program sustainability, and having confidence in project survival). Three
samples of program professionals are surveyed about their perceptions of their programs
(N=243, 58, and 55). Findings suggest all sustainability elements to some degree are related
to these program results, with effective collaboration and program responsivity being
relatively less important. Because there are many elements that contribute to aspects of
sustainability, planning for sustainability can be targeted in particular areas. This program
sustainability framework lends itself for use with community groups composed of program
professionals, evaluators, and other stakeholders. Our empirical results suggest that middle
range program results are affected by sustainability elements, which in turn are amenable to
manipulation.
560
AIAEE 2004 Proceedings of the 20
th
Annual Conference Dublin, Ireland
Introduction
Community-based programs are important in the service delivery system in many
communities and both public and private agencies provide resources to initiate and maintain
such programs. Yet a persistent gap in understanding program processes is the lack of
knowledge about how programs are sustained (Lerner, 1995). The significance of sustaining
programs was expressed by a service provider who said, “We have a responsibility to our
program recipients; they’ve had so many losses in their lives and for us to come in for a year
or two or three and give them hope, only to have the program go away, we’ve just caused
another loss and a further loss of hope in their lives” (Akerlund, 2000, p. 353).
Purpose and Objectives
The focus of this presentation is to examine the relationships between elements of a
program sustainability conceptual framework and these important middle range program
results: meeting needs of at risk families, confidence in community-based program survival,
and the timing of planning for sustainability.
Theoretical Base
There are three main sections in our logic model: elements that are assumed to lead to
sustainability, middle range program results, and an ultimate result of the program being
sustained (Figure 1). We assume that the sustainability elements lead to a set of desired
middle range program results, and it is also assumed that these, in turn, increase the chances
of a program being sustained (ultimate result). It is also assumed that these middle range
program results are directly related to the ultimate sustainability result. We also recognize
that the sustainability elements may be directly related to the ultimate result of a program
being sustained. Vectors in Figure 1 indicate those plausible relationships. We currently
examine the linkages between sustainability elements and selected middle range program
results.
561
AIAEE 2004 Proceedings of the 20
th
Annual Conference Dublin, Ireland
Figure 1: Model of Community-Based Program Sustainability
SUSTAINABILITY
ELEMENTS:
Leadership competence
Effective collaboration
Understanding the
community
Demonstrating program
results
Strategic funding
Staff involvement and
integration
Program responsivity
MIDDLE RANGE
PROGRAM
RESULTS:
Participant needs met
Confidence in
program survival
Effective
sustainability planning
Other program results
ULTIMATE
RESULT:
Sustainability
Defining Sustainability
Sustainability is the power or the capacity of programs to continuously respond to
identified community issues. A sustained program maintains a focus consonant with its
original goals and objectives, including the individuals, families, and communities it was
originally intended to serve. Within our definition are several facets, including the intensity
and level of programs and services that are offered and the flexibility a program demonstrates
with regard to specific services and programs. The key element of sustainability in our view
is retaining the original goal(s), in our case, supporting at risk children, youth, and families
by providing continued benefits, regardless of the particular activities that are delivered or
the format (institutionalization versus independence) in which they are delivered. It is more
important to sustain benefits to families and communities than to sustain program activities
per se.
Elements of Sustainability
The literature contains a number of perspectives on what is associated with
sustainability, which we term elements of sustainability. Our conceptualization is informed
by our research program begun in 1996 on community-based programs for at risk children,
youth, and families (Mancini & Marek, 2003; Mancini & Marek, 1998; Marek, Mancini, &
Brock, 1999).
Our framework contains seven major elements of sustainability: leadership
competence, effective collaboration, understanding the community, demonstrating program
results, strategic funding, staff involvement and integration, and program responsivity.
Leadership Competence
is the ability of leaders to establish goals, to develop clear
plans for program development, implementation, evaluation, and to be active in meeting
562
AIAEE 2004 Proceedings of the 20
th
Annual Conference Dublin, Ireland
those goals and managing those plans (Akerlund, 2000; Blythe, Tracy, Kotovsky, &
Gwatkin, 1992; Bossert, 1990; The Finance Project, 2002).
Effective Collaboration
pertains to the identification of relevant stakeholders who
understand and support program goals, who have clearly identified roles and responsibilities,
and who are actively involved in meeting program goals (Goodman & Steckler, 1989a,
1989b; Steckler & Goodman, 1989; Altman, Endres, Linzer, Lorig, Howard-Pitney, &
Rogers, 1991; O’Loughlin, Renaud, Richard, & Paradis, 1998; Ponzio, Peterson, Miller, &
Kinney, 1994).
Understanding the Community
entails having knowledge of community needs and
resources, having respect for community members, and involving key community members
in programs (Altman, et al, 1991; Mancini & Marek, 2003;Holder & Moore, 2000; Shediac-
Rizkallah & Bone, 1998; Pentz, 2000; Laken & Hutchins, 1995).
Demonstrating Program Results
is the evaluation of program processes and
outcomes using acceptable research methods, and informing stakeholders of the results of
those evaluations (Mancini, Marek, Byrne, & Huebner, in press; O’Laughlin et al, 1998; The
Finance Project, 2002; Holder & Moore, 2000; Laken & Hutchins, 1995).
Strategic Funding
includes having plans and resources in place to support current
program requirements as well as to ensure sufficient future fiscal support to meet ongoing
program goals. The need to be intentional and plan for continued funding includes an
analysis of short-term and long-term funding needs, developing a range of financing options,
and recognizing that program sustainability is enhanced when there is diversity in the forms
of support and the origins of support (Akerlund, 2000; The Finance Project, 2002; Laken &
Hutchins, 1995; Holder & Moore, 2000; Pentz, 2000).
Staff Involvement and Integration
is the inclusion of committed, qualified staff in
program design, implementation, evaluation, and decision-making. Supporting a program’s
mission and vision occurs more readily when staff feel that they are important components in
the organization, and in effect, make the organization their own (Holder & Moore, 2000;
O’Loughlin et al., 1998).
Program Responsivity
is the ability of a project to adapt programming to meet
changing community needs. Program responsivity is essential because of change that is
inherent in communities (Bamberger & Cheema, 1990; Goodman & Steckler, 1989a; The
Finance Project, 2002; Holder & Moore, 2000; Laken & Hutchins, 1995).
Middle Range Program Results
In our model we include middle range program results, those that may be
intermediate to a program actually being sustained. These results are closely aligned with a
program being sustained but since they are not end points, in some respects they are
objectives (short term) rather than goals (long term). In this study we have assessed these
middle range program results: continuing to provide and focus on the original program goals
(in our case, meeting the needs of at risk children, youth, and families), planning for
sustainability, and having confidence in project survival. We do not see these as the only
viable middle range program results, but they are examples of results that may be closely
associated with the ultimate sustainability of a program.
563
AIAEE 2004 Proceedings of the 20
th
Annual Conference Dublin, Ireland
Ultimate Result: A Sustained Program
Whether or not a program is sustained is the ultimate outcome in our model. We have
already noted our assumptions about how ultimate sustainability may be related to the
sustainability elements, and to the middle range program results. It is important to clarify
exactly what sustainability means and how it is measured. For example, Goodman and
associates (1993) found that number of years of existence in the organization was related to a
program becoming part of the organizational routines but unrelated to whether or not the
program was functioning to full capacity. In the former case it might be concluded that a
program was sustained but in the latter case it would not be considered sustained. In our
approach, either sustainability elements or middle range program results may be directly
related to the ultimate sustainability result. What is ultimately important is whether a
program is sustaining benefits to families and communities.
Methods, Procedures, and Data Sources
We use three samples to examine the relationships between the sustainability
elements and the three middle range program results. Data from sample 1 (N=243) were
collected in the spring of 2001 at an annual meeting of USDA’s Children, Youth and
Families at Risk (CYFAR) Initiative. Human development and family life professionals who
work at local, regional, and national levels of program development and evaluation
voluntarily completed a structured survey. Sample 2 data were collected in 2002 with staff
of 58 community-based programs that were funded under USDA’s Youth at Risk Program
Initiative. Study 3 data were collected in 2003 with staff of 55 community-based programs
that were funded under USDA’s State Strengthening Program Initiative.
Measurement
The Program Sustainability Index (PSI) is used to assess sustainability elements, and
includes 53 items. Items are grouped in the seven conceptual framework elements as follows
(number of items in parentheses): Leadership competence (7), Effective collaboration (12),
Understanding the community (9), Demonstrating program results (7), Strategic funding (5),
Staff involvement and integration (10), and Program responsivity (3). These seven elements
were determined through a series of earlier studies wherein interviews were held with over
100 community program personnel between 1996 and 1998. The results informed a survey
that focused on a wide variety of areas thought to influence sustainability; it was
implemented from 1999 through 2003 to program personnel from 153 active projects across
the United States and its territories. Further, in 1998 over 4,000 program professionals who
participated in a study of organizational change were asked to define program sustainability
and then to indicate their view on key elements of sustainability (Betts, Peterson, Marczak, &
Richmond, 2001). Based on these qualitative and quantitative studies, seven elements were
identified as consistently contributing to program sustainability. Please contact us for
additional information on the PSI.
Three middle range program results items were included in all three studies. The first
question asked, “To what extent does this project’s current programming meet the needs of at
risk children, youth, and families?” Response choices were “Not at all”, “Somewhat”,
564
AIAEE 2004 Proceedings of the 20
th
Annual Conference Dublin, Ireland
“Moderately”, and “Fully”. Given the program affiliations of the respondents, asking about
meeting needs of at risk children, youth, and families is an important aspect of whether a
program is targeting an appropriate community audience and fulfilling the original goal(s) for
the program. A second question asked, “When did program leaders begin actively planning
for this project’s survival (post-funding)?” Response choices were: during the initial
program proposal phase, during the first, second, third, fourth, or fifth year of funding, and
after the initial funding period had ended. We assume that the timing of sustainability
planning is an important aspect of whether sustainability actually occurs, and that planning is
related to sustainability elements such as leadership competence, strategic funding, and so
on. The third question asked, “How confident are you that your project will still be active in
five years?” Response choices were “Not at all”, “Somewhat”, and “Very confident”. Our
assumption is that being confident in the future of a program is affected by such elements as
effective collaboration, demonstrating program results, and so on, and may be a precursor of
ultimate sustainability because this confidence is grounded in what is actually happening
with a program.
Results
Our examination of the relationships between the sustainability elements and the
middle range program outcomes is addressed by using bi-variate correlations (one-tailed test
of significance). Correlation effect sizes range from .02 to .48, with most being moderate;
60% of the correlations are statistically significant (Table 1). Leadership Competence and
Understanding the Community correlate most frequently with the three program results (for
example, meeting at-risk needs), and Effective Collaboration and Program Responsivity
correlate least frequently. Across the 3 studies, there is concurrence with regard to (1) the
relationships that Leadership Competence, Strategic Funding, and Staff Involvement and
Integration have with having confidence in program survival, (2) the relationships that
Leadership Competence and Understanding the Community have with planning earlier for
sustainability, and (3) the relationship that Demonstrating Program Results has with meeting
needs of at-risk families. Higher scores on these sustainability factors are associated with
meeting at-risk needs, with having confidence in the future of the program, and with focusing
on sustainability planning.
565
AIAEE 2004 Proceedings of the 20
th
Annual Conference Dublin, Ireland
Table 1. Correlations between Sustainability Factors and Program Results
Meeting At-Risk
Needs
Planning Process Confidence in Program
Survival
Factors
1
2 3 1 2 3 1 2 3
Leadership
Competence
18** 22 19 22** 46** 26* 20** 43** 48**
Effective
Collaboration
12* 14 08 08 19 25* 05 09 28*
Understanding the
Community
29** 31* 02 39** 29* 32* 20** 12 33**
Demonstrating
Program Results
22** 32* 23* 13* 19 24* 11 20 30*
Strategic Funding 17** 05 17 24** 37** 07 18** 42** 44**
Staff Involvement
and Integration
19** 25* 17 19** 22 04 16* 26* 36**
Program
Responsivity
12* 04 10 05 39** 17 10 13 43**
*p < .05. **p < .01. Note: Decimal points are deleted. Sample sizes for Studies 1, 2, and 3
are, respectively, 243, 58, and 55.
Conclusions, Implications, and Recommendations
Our framework was conceived from the nexus of qualitative and quantitative
research, the extant literature, and our experience as social and behavioral scientists, all
elements that are important for the development of program theory (Connell & Kubisch,
1998). Our study provides substantive information about how these sustainability elements
are linked to middle range program results, in this case meeting the needs of at risk children,
youth, and families, the sustainability planning process, and the degree of confidence in
program survival. We first note that the correlations are modest though statistically
significant. Even though the correlation magnitude is less than desired, the direction of the
coefficients is consistent with expectations. To varying degrees among these samples of
respondents, and for the programs that they are reporting on, all dimensions of sustainability
are important for accomplishing primary program goals involving meeting needs of at risk
families, and in an expected direction. The relationships between the sustainability elements
and the sustainability planning process measure were also in the expected direction. This
finding is consistent with our advice to program professionals that suggests the significance
of early sustainability planning for program success. We also expected that the sustainability
566
AIAEE 2004 Proceedings of the 20
th
Annual Conference Dublin, Ireland
elements would covary positively with having confidence in the future of the program. This
expectation was also supported.
Generally speaking, across the studies, leadership competence and understanding the
community are most consistently related to the middle range program results, followed by
demonstrating program results, strategic funding, and staff involvement and integration.
Effective collaboration and program responsivity are related substantially less. More
specifically, if we look at which sustainability elements are more important for meeting
needs of at risk families, it appears that understanding the community, demonstrating
program results, and staff involvement are more consistently related; for planning for
sustainability, it appears that leadership competence and understanding the community are
more important; and for having confidence in project survival, it appears that leadership
competence, strategic funding, and staff involvement and integration are more consistently
related.
From Theory and Research to Professional Practice
For program professionals the findings of this study suggest a roadmap for being
intentional about sustainability efforts by virtue of the development and implementation of a
sustainability plan. The need for intentionality is particularly important in light of research
that discusses how early sustainability planning is in itself an important step towards
sustaining programs (Goodman & Steckler, 1987-88; Laken & Hutchins, 1995; Shediac-
Rizkallah & Bone, 1998) and the research that argues the need for being ethically responsible
to continue programs once they are started, particularly for those in the neediest communities
(Akerlund, 2000; Goodman & Steckler, 1987-88). These sustainability elements provide not
only important information about what and who is necessary for sustaining programs but also
the means for program leaders, staff, and other key stakeholders to work together in the
development of a sustainability plan. The major benefit for program professionals in
focusing on sustainability elements is that they become focal points for program teams to
develop, implement, and monitor a sustainability plan. Key stakeholders can be brought
together to focus discussions on critical aspects of sustainability which then determines what
mechanisms should be activated as programs are being developed, implemented, and
monitored. We have discovered that each of these sustainability elements is important in
education and training on program sustainability (Marek & Mancini, 2001).
Developing a sustainability plan is an entry-level activity rather than an endpoint.
Once a program is implemented, the sustainability elements can be used to monitor program
supports and to continue to gauge sustainability itself. Using our framework as a monitoring
tool helps to continually appraise and prioritize the sustainability process, including strengths
and gaps. Intentional efforts toward sustaining programs are dynamic and evolving, much as
programming efforts are in general. Sustaining programs is a process that benefits from
continual monitoring and adaptation in order to meet individual, family, program, and
community needs. Findings from this three-sample study suggest that important precursors to
program sustainability are substantially related to a range of sustainability elements that can
be manipulated by program professionals.
Our own work with program professionals has used the Program Sustainability Index
and its sustainability elements as an interactive sustainability assessment process to assist
program teams in this planning (Marek & Mancini, 2001). In the trainings we have
567
AIAEE 2004 Proceedings of the 20
th
Annual Conference Dublin, Ireland
conducted, we transposed the PSI into a program sustainability checklist and requested each
team member to complete it. We have taken a type of performance indicator approach by
asking workshop respondents to respond to each item by using three levels of responses in
assessing their program (in effect, treating each item as a program goal and than asking if the
goal is “Fully,” “Somewhat/partially,” or “Barely/not at all” met). After this individual
assessment is completed, team members share their own perceptions of how their program is
responding to each of the elements (based on the items for each element). Workshop
attendees have reported that although they may have spoken of the need to sustain their
programming efforts, they never did so in as specified and detailed way as they did using the
PSI. This process allows the team to reach consensus on their key gaps of sustainability
efforts and then decide, as a group, what needs to be done, when it needs to be done, how it
will be done, and who will be doing what to intentionally plan for and implement a
sustainability plan. Through asking and answering these questions, the framework of a
sustainability action plan emerges. For example, if we consider items related to effective
collaboration, a program team may find that they have not yet included local decision-makers
or representatives from all appropriate community services agencies. By determining this
gap, discussion ensues as to who would be appropriate to include on the program team, how
will they be approached and invited to participate, who will do so, and when will this occur.
Findings from the current analysis suggest that middle range program results are affected by
sustainability elements, which can be objectives of training at the community level.
References
Akerlund, K.M. (2000). Prevention program sustainability: The state’s perspective.
Journal of Community Psychology, 28, 353-362.
Altman, D. G., Endres, J., Linzer, J., Lorig, K., Howard-Pitney, B., & Rogers, T.
(1991). Obstacles to and future goals of ten comprehensive community health
promotion projects. Journal of Community Health, 16, 299-314.
Bamberger, M. & Cheema, S. (1990). Economic Development Institute for The World
Bank. Case studies of project sustainability: Implications for policy and operations
from Asian experience. The World Bank: Washington.
Betts, S. C., Peterson, D. J., Marczak, M. S. & Richmond, L. S. (2001). System-wide
evaluation: Taking the pulse of a national organization serving children, youth,
and families at risk. Children's Services: Social Policy, Research, and Practice, 4,
87-101.
Blythe, B. J., Tracy, E. M., Kotovsky, A., & Gwatkin, S. (1992). Organizational supports to
sustain intensive family preservation programs. Families in Society: The Journal of
Contemporary Human Services, 73, 463-470.
Bossert, T. J. (1990). Can they get along without us? Sustainability of donor
supported health projects in Central America and Africa. Social Science Medicine, 30,
1015-1023.
Connell, J.P., & Kubisch, A.C. (1998). Applying a theory of change approach to the
evaluation of comprehensive community initiatives: Progress, prospects, and
problems. In Fulbright-Anderson, K., Kubisch, A.C., & Connell, J.P. (Eds.),
New approaches to evaluating community initiatives, Volume 2: Theory,
Measurement, and analysis (pp. 15-44). Washington, DC: Aspen Institute.
568
AIAEE 2004 Proceedings of the 20
th
Annual Conference Dublin, Ireland
Goodman, R. M. and Steckler A. B. (1989a). A framework for assessing program
institutionalization. Knowledge in Society: the International Journal of Knowledge
Transfer, 2, 57-71.
Goodman, R. M. & Steckler, A. (1989b). A model for the institutionalization of health
promotion programs. Family & Community Health, 11, 63-78.
Goodman, R. M. & Steckler, A. B. (1987-1988). The life and death of a health
promotion program: An institutionalization case study. International Quarterly of
Community Health Education, 8, 5-21.
Goodman, R.M., McLeroy, K.R., Steckler, A.B., & Hoyle, R.H. (1993). Development of
level of institutionalization scales for health promotion programs. Health Education
Quarterly, 20, 161-178.
Holder, H. D. & Moore, R. S. (2000). Institutionalization of community action projects
to reduce alcohol use and related problems: Systematic facilitators. Substance Use &
Misuse, 35, 75-86.
Laken, M. P. & Hutchins, E. (1995). Building and sustaining systems of care for substance-
using pregnant women and their infants: Lessons learned. Arlington, VA:
National Center for Education in Maternal and Child Health.
Lerner, R. (1995). America’s youth in crisis. Thousand Oaks, CA: Sage.
Mancini, J.A., & Marek, L.I. (March 2003). Sustaining community-based programs for
families: Program development implications from longitudinal research. Research
lecture presented at the annual meeting of the Children, Youth, and Families At risk
Initiative, U.S. Department of Agriculture, Minneapolis, MN.
Mancini, J.A., & Marek, L.I. (1998). Patterns of project survival and organizational
support: The national youth at risk program sustainability study. (Virginia
Cooperative Extension Publication 350-800). Retrieved from Virginia Cooperative
Extension Web site, September 6, 2003: http://www.ext.vt.edu/pubs/family/350-
800/350-800.html
Mancini, J.A., Marek, L.I., Byrne, R., & Huebner, A.J. (in press). Community-based
program research: Context, program readiness, and evaluation usefulness. Journal of
Community Practice.
Marek, L.I., & Mancini, J.A. (March 2001). Sustaining your community-based program:
Planning for Success. Workshop presented at the annual meeting of the USDA
Children, Youth, and Families at Risk Initiative, San Diego, CA.
Marek, L.I., Mancini, J.A., & Brock, D.J. (1999). Continuity, success, and survival of
community-based projects: The national youth at risk program sustainability study.
(Virginia Cooperative Extension Publication 350-801). Retrieved September 6, 2003,
from Virginia Cooperative Extension Web site:
http://www.ext.vt.edu/pubs/family/350-801/350-801.html
O’Loughlin, J., Renaud, L., Richard, L., Gomez, L. S., & Paradis, G. (1998). Correlates of
the sustainability of community-based heart health promotion interventions.
Preventive Medicine, 27, 702-712.
Pentz, M.A. (2000). Institutionalizing community-based prevention through policy
change. Journal of Community Psychology, 28, 257-270.
569
AIAEE 2004 Proceedings of the 20
th
Annual Conference Dublin, Ireland
570
Ponzio, R. C., Peterson, K.D., Miller, J.P., & Kinney, M.B. (1994). A program
portfolio/panel review evaluation of 4-H sponsored community-based, social
action projects for at risk youth. Journal of Research and Development in Education,
28, 55-65.
Shediac, M. C. & Bone, L. R. (1998). Planning for the sustainability of community-based
health programs: Conceptual frameworks and future directions for research, practice
and policy. Health Education Research, 13, 87-108.
Steckler, A. & Goodman, R. M. (1989). How to institutionalize health promotion programs.
American Journal of Health Promotion, 3, 34-44.
The Finance Project (2002). Sustaining comprehensive community initiatives: Key
elements for success. Retrieved June 21, 2003, from The Finance Project Web
site:
http://www.financeproject.org
