Evaluation and program planning

Publisher: Elsevier

Journal description

Current impact factor: 0.89

Impact Factor Rankings

Additional details

5-year impact 0.99
Cited half-life 0.00
Immediacy index 0.07
Eigenfactor 0.00
Article influence 0.34
ISSN 1873-7870

Publisher details


  • Pre-print
    • Author can archive a pre-print version
  • Post-print
    • Author can archive a post-print version
  • Conditions
    • Pre-print allowed on any website or open access repository
    • Voluntary deposit by author of authors post-print allowed on authors' personal website, arXiv.org or institutions open scholarly website including Institutional Repository, without embargo, where there is not a policy or mandate
    • Deposit due to Funding Body, Institutional and Governmental policy or mandate only allowed where separate agreement between repository and the publisher exists.
    • Permitted deposit due to Funding Body, Institutional and Governmental policy or mandate, may be required to comply with embargo periods of 12 months to 48 months .
    • Set statement to accompany deposit
    • Published source must be acknowledged
    • Must link to journal home page or articles' DOI
    • Publisher's version/PDF cannot be used
    • Articles in some journals can be made Open Access on payment of additional charge
    • NIH Authors articles will be submitted to PubMed Central after 12 months
    • Publisher last contacted on 18/10/2013
  • Classification
    ​ green

Publications in this journal

  • [Show abstract] [Hide abstract]
    ABSTRACT: Evolutionary theory, developmental systems theory, and evolutionary epistemology provide deep theoretical foundations for understanding programs, their development over time, and the role of evaluation. This paper relates core concepts from these powerful bodies of theory to program evaluation. Evolutionary Evaluation is operationalized in terms of program and evaluation evolutionary phases, which are in turn aligned with multiple types of validity. The model of Evolutionary Evaluation incorporates Chen's conceptualization of bottom-up versus top-down program development. The resulting framework has important implications for many program management and evaluation issues. The paper illustrates how an Evolutionary Evaluation perspective can illuminate important controversies in evaluation using the example of the appropriate role of randomized controlled trials that encourages a rethinking of “evidence-based programs”. From an Evolutionary Evaluation perspective, prevailing interpretations of rigor and mandates for evidence-based programs pose significant challenges to program evolution. This perspective also illuminates the consequences of misalignment between program and evaluation phases; the importance of supporting both researcher-derived and practitioner-derived programs; and the need for variation and evolutionary phase diversity within portfolios of programs.
    Evaluation and program planning 08/2014;
  • [Show abstract] [Hide abstract]
    ABSTRACT: The use of online resources to reduce the attrition of program participants in longitudinal studies is examined. Higher-risk individuals, those involved in illegal activities, and females with last name changes are typically more difficult to locate. The effectiveness of using online resources for these participants is addressed. These resources include social networking sites, people-finder search engines, telephone and address directories, judicial records, and death records. The strengths and limitations of these resources are presented and discussed. Longitudinal studies using these resources are examined to evaluate their successful follow-up rates. The results of these studies indicate that participant characteristics are more important to successful follow-up than the length of time since participation or sample size. The use of multiple online sites increased follow-up rates, especially for those who are typically difficult to locate. The variables and websites to consider are discussed, and six lessons learned are offered. The prospective use of online participant involvement is especially important for successful longitudinal evaluation and program planning.
    Evaluation and program planning 08/2014; 45:82-89.
  • [Show abstract] [Hide abstract]
    ABSTRACT: Workers who incur permanent impairments or have ongoing medical restrictions due to injuries or illnesses sustained at work may require support from vocational rehabilitation programs in order to return to work. Vocational rehabilitation programs implemented within workers’ compensation settings are costly, and effective service delivery has proven challenging. The Vocational Improvement Project, a 5.5-year pilot program beginning in 2008, introduced major changes to the Washington State workers’ compensation-based vocational rehabilitation program. In the evaluation of this pilot program, set within a large complex system characterized by competing stakeholder interests, we assessed effects on system efficiency and employment outcomes for injured workers. While descriptive in nature, this evaluation provided evidence that several of the intended outcomes were attained, including: (1) fewer repeat referrals, (2) fewer delays, (3) increased choice for workers, and (4) establishment of statewide partnerships to improve worker outcomes. There remains substantial room for further improvement. Retraining plan completion rates remain under 60% and only half of workers earned any wages within two years of completing their retraining plan. Ongoing communication with stakeholders was critical to the successful conduct and policy impact of this evaluation, which culminated in a 3-year extension of the pilot program through June 2016.
    Evaluation and program planning 06/2014;
  • [Show abstract] [Hide abstract]
    ABSTRACT: Complexity theory has increasingly been discussed and applied within evaluation literature over the past decade. This article reviews the discussion and use of complexity theory within academic journal literature. The aim is to identify the issues to be considered when applying complexity theory to evaluation. Reviewing 46 articles, two groups of themes are identified. The first group considers implications of applying complexity theory concepts for defining evaluation purpose, scope and units of analysis. The second group of themes consider methodology and method. Results provide a starting point for a configuration of an evaluation approach consistent with complexity theory, whilst also identifying a number of design considerations to be resolved within evaluation planning.
    Evaluation and program planning 04/2014; 45C:119-126.
  • [Show abstract] [Hide abstract]
    ABSTRACT: There is a need to improve girls' physical activity (PA) in afterschool programs as girls' PA levels are consistently lower than boys'. An evidence-based professional development framework, the 5 Ms, has been effective in helping staff to improve PA in both girls and boys but further improvements in girls' PA are needed. Little is known about staff's perceptions of using PA promotion strategies to promote girls' PA. Therefore, the purpose of this study was to explore staff perceptions of the use of evidence-based PA promotion strategies for promoting PA in girls. Semi-structured interviews were conducted with staff from three community-based afterschool programs located within a school setting (n=18). Data were analyzed using the process of immersion/crystallization. A majority of staff had some knowledge of PA promotion strategies but few staff consistently utilized these strategies and a majority felt several strategies were unnecessary (i.e., having a PA policy). Newer staff reported depending on senior staff to promote PA in girls. Overall, findings suggest that staff's perceptions may impact their use of PA promotions strategies. The results of this study will contribute to the enhancement of an existing staff training framework (the 5 Ms) to improve girls' PA in afterschool programs.
    Evaluation and program planning 04/2014; 45C:102-109.
  • [Show abstract] [Hide abstract]
    ABSTRACT: Program theory-driven evaluation science (PTDES) provides a useful framework for uncovering the mechanisms responsible for positive change resulting from participation in youth development (YD) programs. Yet it is difficult to find examples of PTDES that capture the complexity of such experiences. This article offers a much-needed example of PTDES applied to Project K, a youth development program with adventure, service-learning and mentoring components. Findings from eight program staff focus groups, 351 youth participants' comments, four key program documents, and results from six previous Project K research projects were integrated to produce a theory of change for the program. A direct logic analysis was then conducted to assess the plausibility of the proposed theory against relevant research literature. This demonstrated that Project K incorporates many of the best practice principles discussed in the literature that covers the three components of the program. The contributions of this theory-building process to organizational learning and development are discussed.
    Evaluation and program planning 04/2014; 45C:61-70.
  • [Show abstract] [Hide abstract]
    ABSTRACT: Community-based health promotion is poorly theorised and lacks an agreed evidence-base. This paper examines characteristics of community-based health promotion and the challenges they present to evaluation. A review of health promotion evaluation leads to an exploration of more recent approaches, drawing on ideas from complexity theory and developmental evaluation. A reflexive analysis of three program evaluations previously undertaken as an evaluation consultant is used to develop a conceptual model to help in the design and conduct of health promotion evaluation. The model is further explored by applying it retrospectively to one evaluation. Findings suggest that the context-contingent nature of health promotion programs; turbulence in the community context and players; multiple stakeholders, goals and strategies; and uncertainty of outcomes all contribute to the complexity of interventions. Bringing together insights from developmental evaluation and complexity theory can help to address some evaluation challenges. The proposed model emphasises recognising and responding to changing contexts and emerging outcomes, providing rapid feedback and facilitating reflexive practice. This will enable the evaluator to gain a better understanding of the influence of context and other implementation factors in a complex setting. Use of the model should contribute to building cumulative evidence and knowledge in order to identify the principles of health promotion effectiveness that may be transferable to new situations.
    Evaluation and program planning 04/2014; 45C:71-81.
  • [Show abstract] [Hide abstract]
    ABSTRACT: The work described in this article focuses primarily on how human service organizations can use an evidence-based, self-assessment approach to organization evaluation to facilitate continuous quality improvement and organization change. Real-life examples are presented, strengths and challenges discussed, and future conceptual and measurement issues identified.
    Evaluation and program planning 03/2014; 45C:110-118.
  • [Show abstract] [Hide abstract]
    ABSTRACT: Little is known about the factors associated with use of employment services among homeless youth. Social network characteristics have been known to be influential in motivating people's decision to seek services. Traditional theoretical frameworks applied to studies of service use emphasize individual factors over social contexts and interactions. Using key social network, social capital, and social influence theories, this paper developed an integrated theoretical framework that capture the social network processes that act as barriers or facilitators of use of employment services by homeless youth, and understand empirically, the salience of each of these constructs in influencing the use of employment services among homeless youth. We used the "Event based-approach" strategy to recruit a sample of 136 homeless youth at one drop-in agency serving homeless youth in Los Angeles, California in 2008. The participants were queried regarding their individual and network characteristics. Data were entered into NetDraw 2.090 and the spring embedder routine was used to generate the network visualizations. Logistic regression was used to assess the influence of the network characteristics on use of employment services. The study findings suggest that social capital is more significant in understanding why homeless youth use employment services, relative to network structure and network influence. In particular, bonding and bridging social capital were found to have differential effects on use of employment services among this population. The results from this study provide specific directions for interventions aimed to increase use of employment services among homeless youth.
    Evaluation and program planning 03/2014; 45C:90-101.
  • [Show abstract] [Hide abstract]
    ABSTRACT: There is increasing recognition that performance evaluation of national as well as foreign public economic development programs should emphasize on their implementation rather than focus solely on their outcomes. This paper presents a framework for comparing the delivery or administrative efficiency of public spending programs at local level, based on a novel application of Data Envelopment Analysis (DEA). It is focused on the European Union context and specifically on the Greek LEADER operational program. The results of this application indicate that DEA, in contrast to traditional performance metrics, is an insightful tool in revealing administrative inefficiencies in program delivery by capturing the operational and scale components of performance while taking into account the complex mix of tasks and interventions carried out by its operators. The proposed technique may be easily integrated into a program evaluation scheme and may be utilized as a tool to support rational decision-making and program improvement.
    Evaluation and program planning 03/2014; 45C:140-150.
  • [Show abstract] [Hide abstract]
    ABSTRACT: This paper describes a mixed methods evaluation of partner agency collaboration within a system of care implemented from 2010 to 2012 in a historically underserved minority community in Houston, Texas. The first section describes the project and the framework for evaluating partner agency collaboration. The second section describes the evaluation methods and presents the baseline and follow-up results of the evaluation. The third section includes a discussion of the evaluation findings, the conclusion, and the lessons learned.
    Evaluation and program planning 03/2014; 45C:50-60.
  • [Show abstract] [Hide abstract]
    ABSTRACT: This study analyzed the planning process (summer 2008 to fall 2009) of a Montreal project that offers housing and community follow-up to homeless people with mental disorders, with or without substance abuse disorders. With the help of the Advocacy Coalition Framework (ACF), advocacy groups that were able to navigate a complex intervention implementation process were identified. In all, 25 people involved in the Montreal At Home/Chez Soi project were surveyed through interviews (n=18) and a discussion group (n=7). Participant observations and documentation (minutes and correspondence) were also used for the analysis. The start-up phase of the At Home/Chez may be broken down into three separate periods qualified respectively as "honeymoon;" "clash of cultures;" and "acceptance & commitment". In each of the planning phases of the At Home/Chez Soi project in Montreal, at least two advocacy coalitions were in confrontation about their specific belief systems concerning solutions to address the recurring homelessness social problem, while a third, more moderate one contributed in rallying most key actors under specified secondary aspects. The study confirms the importance of policy brokers in achieving compromises acceptable to all advocacy coalitions.
    Evaluation and program planning 03/2014; 45C:42-49.
  • [Show abstract] [Hide abstract]
    ABSTRACT: According to demographic trends, the ratio of senior citizens in the overall population of the Republic of Serbia is rising. This generates the need to create socially acceptable and economically sustainable models for the protection of the elderly. The goal of this paper is to stress the necessity of analyzing and evaluating the efficiency of social protection services aimed at senior citizens. The first part of the paper underlines the need for economic analysis of these services; while the second part features the analysis of the two most frequently provided services for the elderly in Serbia: admission to social protection institutions and home care for senior citizens. Based on the research results, the paper also provides a comparative overview of the efficiency of the services mentioned. This overview clearly confirms that both services prove to be economically justifiable from a social perspective; nevertheless, it also indicates that the cost of home care per user is considerably lower than the cost of putting a senior citizen into a nursing home. After presenting and discussing the results of the studies, the paper also offers recommendations aimed at enhancing the development and sustainability of the social protection system for the elderly in Serbia.
    Evaluation and program planning 03/2014; 45C:9-21.
  • [Show abstract] [Hide abstract]
    ABSTRACT: The transition to parenthood involves many stressors that can have implications for the couple relationship as well as the developmental environment of the child. Scholars and policymakers have recognized the potential for interventions that can help couples navigate these stressors to improve parenting and coparenting strategies. Such evidence-based programs are scarcely available, however, and little is known about the resources necessary to carry out these programs. This study examines the costs and resources necessary to implement Family Foundations, a program that addresses the multifaceted issues facing first-time parents through a series of pre- and post-natal classes. Costs were determined using a 6-step analytic process and are based on the first implementation of the program carried out through a five-year demonstration project. This assessment demonstrates how overall costs change across years as new cohorts of families are introduced, and how cost breakdowns differ by category as needs shift from training group leaders to sustaining program services. Information from this cost analysis helps clarify how the program could be made more efficient in subsequent implementations. We also consider how results may be used in future research examining economic benefits of participation in the program.
    Evaluation and program planning 02/2014; 44C:59-67.
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Corporate community initiatives (CCI) are often established via cross-sector partnerships with nonprofit agencies to address critical social problems. While there is a growing body of literature exploring the effectiveness and social impact of these partnerships, there is a limited evaluative research on the implementation and execution processes of CCIs. In this paper, we examined the implementation and operational processes in the delivery of a professional sport organization's CCI initiative using program theory evaluation. The findings showed discrepancies between the associate organization and the implementers regarding understanding and fulfilling responsibilities with performing certain aspects (maintaining accurate records and program marketing) of the service delivery protocol. Despite program stakeholders being satisfied overall with the program delivery, contradictions between program stakeholders' satisfaction in the quality of program delivery was found in critical components (marketing and communications) of the service delivery. We conclude that ongoing evaluations are necessary to pinpoint the catalyst of the discrepancies along with all partners valuing process evaluation in addition to outcome evaluation.
    Evaluation and program planning 01/2014; 44C:36-47.