Evaluation and Program Planning

Published by Elsevier
Online ISSN: 0149-7189
Publications
Article
The continued need for improvement within a 'system of care' is essential as the need for mental health services by those 'youth' within the child welfare system continually grows. This article outlines the statewide reform of New Jersey's Children's Behavioral Healthcare System, which began in 2000, as well as including the recommendations of the University of South Florida as part of their 'Final Report: Independent Assessment of New Jersey's Child Behavioral Health Services' on continued changes within the system of care. Successful outcomes have resulted from this welfare reform initiative, which include most notably the significant caseload decrease of the Division of Youth and Family Services (DYFS) and the creation of a new cabinet entity, the Department of Children and Families (DCF). This article specifically outlines systemic recommendations to best serve the target population of 'transitional' youth between the ages of 16 to 18+ years utilizing interagency cooperation based upon 'theory of change' and Total Clinical Outcomes Management (TCOM) strategies.
 
Article
Longitudinal studies offer significant advantages in rendering data commensurate with the complexity of human development. However, incomplete enrollment and attrition over time can introduce bias. Furthermore, there is a scarcity of evaluative information on cohorts in developing countries. This paper documents various strategies adopted to minimize loss to follow up and describes a retrospective analysis of a small group of families who were missed during initial enrollment and through several subsequent rounds of data collection of the Birth to Twenty (BT20) birth cohort in Soweto-Johannesburg, South Africa that began in 1990. A purposive case study approach was used, and 10 of the 119 families missed at enrollment were interviewed to investigate why these families were not enrolled into the study. The findings demonstrate that high mobility, both within urban areas and between urban and rural areas, are a major challenge for longitudinal studies in densely populated urban areas. In addition, enrollment was also affected by individuals changing their names, largely motivated to facilitate access to employment under Apartheid, as well as varying motivations for participating in research. Longitudinal studies in the developing country context must be mindful of the political, social and economic climate that influences enrollment and ongoing cohort maintenance.
 
Article
People invited to participate in an evaluation process will inevitably come from a variety of personal backgrounds and hold different views based on their own lived experience. However, evaluators are in a privileged position because they have access to information from a wide range of sources and can play an important role in helping stakeholders to hear and appreciate one another's opinions and ideas. Indeed, in some cases a difference in perspective can be utilised by an evaluator to engage key stakeholders in fruitful discussion that can add value to the evaluation outcome. In other instances the evaluator finds that the task of facilitating positive interaction between multiple stakeholders is just ‘an uphill battle’ and so conflict, rather than consensus, occurs as the evaluation findings emerge and are debated.
 
Article
California's voter-initiated Proposition 36 offers non-violent drug offenders community-based treatment as an alternative to incarceration or probation without treatment. This article reports short-term treatment outcomes subsequent to this major shift in drug policy. Data are from 1104 individuals randomly selected from all Proposition 36 participants assessed for treatment in five California counties during 2004. The overall study sample was 30% female, 51% white, 18% Black, 24% Hispanic, and 7% other racial/ethnic groups. The mean+/-SD age was 37+/-10 years. Counties varied considerably in participant characteristics, treatment service intensity, treatment duration, urine testing, and employment and recidivism outcomes, but not in drug use at 3-month follow-up. Controlling for county, logistic regression analysis showed that drug abstinence was predicted by gender (female), employment at baseline (full or part-time), residential (vs. outpatient) stay, low psychiatric severity, frequent urine testing by treatment facility, and more days in treatment. Recidivism was predicted only by shorter treatment duration. Employment predictors included age (younger), gender (male), baseline employment, and lower psychiatric severity. The study findings support drug testing to monitor abstinence and highlight the need to address employment and psychiatric problems among Proposition 36 participants.
 
Article
Operative for nearly a decade, California's voter-initiated Proposition 36 program offers many offenders community-based substance abuse treatment in lieu of likely incarceration. Research has documented program successes and plans for replication have proliferated, yet very little is known about how the Proposition 36 program works or practices for achieving optimal program outcomes. In this article, we identify policies and practices that key stakeholders perceive to be most responsible for the successful delivery of court-supervised substance abuse treatment to offenders under Proposition 36. Data was collected via focus groups conducted with 59 county stakeholders in six high-performing counties during 2009. Discussion was informed by seven empirical indicators of program performance and outcomes and was focused on identifying and describing elements contributing to success. Program success was primarily attributed to four strategies, those that: (1) fostered program engagement, monitored participant progress, and sustained cooperation among participants; (2) cultivated buy-in among key stakeholders; (3) capitalized on the role of the court and the judge; and (4) created a setting which promoted a high-quality treatment system, utilization of existing resources, and broad financial and political support for the program. Goals and practices for implementing each strategy are discussed. Findings provide a "promising practices" resource for Proposition 36 program evaluation and improvement and inform the design and study of other similar types of collaborative justice treatment efforts.
 
Article
Federal legislation regarding health care in the U.S. has increased rapidly in the past few years. A major law with potential far-reaching effects was enacted as a result of increasing legislation and rising health care costs. This law,The National Health Planning and Resources Development Act,has created a network of over 200 local, mostly nongovernment units, called health systems agencies. These agencies are responsible for areawide health planning, plan implementation, review and approval of federal health care expenditures for local programs, and facilities review. They will affect health and mental health programs at the local level. The article is directed to local health and mental health care providers who will, of necessity, deal directly with the local HSA's.
 
Article
Health and psychosocial service needs that may be co-morbid with opioid addiction may impede the success of drug treatment among patients attending methadone maintenance treatment programs (MMTPs). This longitudinal panel study investigates whether receipt of services from one or more helping professionals outside of the MMTP confers a benefit for drug treatment outcomes among a random sample of male MMTP patients (N=356). Each participant was interviewed 3 times, with 6 months between each interview. Since this observational study did not employ random assignment, propensity score matching was employed to strengthen causal validity of effect estimates. Results support hypotheses that receiving additional off-site services has significant beneficial effects in increasing the likelihood of abstaining from cocaine, heroin, and any illicit drug use over both the ensuing 6- and 12-month time periods. These findings indicate that receipt of additional medical and/or psychosocial services enhances the efficacy of methadone treatment in increasing abstinence from illicit drug use.
 
Article
Although the organizational structures and operating procedures of state substance abuse prevention systems vary substantially across states, there is scant empirical research regarding approaches for rigorous assessment of system attributes and which attributes are most conducive to overall effectiveness. As one component of the national cross-site evaluation of the SPF State Incentive Grant Program (SPF SIG), an instrument was developed to assess state substance abuse prevention system infrastructure in order to measure infrastructure change and examine the role of state infrastructure in achieving prevention-related outcomes. In this paper we describe the development of this instrument and summarize findings from its baseline administration. As expected, states and territories were found to vary substantially with respect seven key characteristics, or domains, of state prevention infrastructure. Across the six domains that were assessed using numeric ratings, states scored highest on data systems and lowest on strategic planning. Positive intercorrelations were observed among these domains, indicating that states with high capacity on one domain generally have relatively high capacity on other domains as well. The findings also suggest that state prevention infrastructure development is linked to both funding from state government and the presence of a state interagency coordinating body with decision-making authority. The methodology and baseline findings presented will be used to inform the ongoing national cross-site evaluation of the SPF SIG and may provide useful information to guide further research on state substance abuse prevention infrastructure.
 
Article
This paper examines ways in which the level of participation in intensive outreach contributes to entry into substance abuse treatment. It was hypothesized that the number of outreach contacts to substance abusing women predicts the likelihood of their entry into drug abuse treatment. As such women often have multiple vulnerabilities that impact upon their needs and readiness for seeking treatment, we also examine the relationships among readiness to seek assistance, the women's level of involvement with the outreach project, and admission to treatment. A sample of 665 women who participated in an enhanced outreach and treatment readiness preparation program was tracked to examine patterns of referral and entry into substance abuse treatment. The number of outreach contacts was inversely related to receiving referrals to substance abuse treatment, as well as to the completion of those referrals (for a subsample of 551 women who had received referrals). In addition, victims of domestic violence were 50% more likely to receive a referral to drug abuse treatment, but of the women who received such a referral, the lack of involvement in a violent relationship predicted entry into treatment. Findings are discussed in terms of their implications for designing and implementing outreach and treatment programs for women with multiple therapeutic issues, particularly substance abuse and domestic violence.
 
Article
The Substance Abuse Mental Health Services Administration has promoted HIV testing and counseling as an evidence-based practice. Nevertheless, adoption of HIV testing in substance abuse treatment programs has been slow. This article describes the experience of a substance abuse treatment agency where, following participation in a clinical trial, the agency implemented an HIV testing and counseling program. During the trial, a post-trial pilot, and early implementation the agency identified challenges and developed strategies to overcome barriers to adoption of the intervention. Their experience may be instructive for other treatment providers seeking to implement an HIV testing program. Lessons learned encompassed the observed acceptability of testing and counseling to clients, the importance of a "champion" and staff buy-in, the necessity of multiple levels of community and agency support and collaboration, the ability to streamline staff training, the need for a clear chain of command, the need to develop program specific strategies, and the requirement for sufficient funding. An examination of costs indicated that some staff time may not be adequately reimbursed by funding sources for activities such as adapting the intervention, start-up training, ongoing supervision and quality assurance, and overhead costs.
 
Article
This paper highlights the value of utilizing a participatory evaluation approach when working with community agencies receiving federal funding for prevention and intervention services. Drawing from our experience as evaluators of a SAMHSA-funded substance abuse, HIV and Hepatitis prevention program targeting homeless young adults, we describe the importance of and strategies for creating a participatory evaluation partnership with program implementers. By participatory evaluation we mean the active involvement of program implementers in defining the evaluation, developing instruments, collecting data, discussing findings, and disseminating results. There are a number of challenges faced when using this approach with federally funded programs that require the use of standardized measurement tools and data collection procedures. Strategies we used to strike a balance between federal requirements and local needs are presented. By increasing the understanding of and participation in the evaluation process, program implementers have greater support for data collection requirements and are appreciably more interested in learning from the evaluation data. This approach has helped to build the capacity of a program and stimulated new possibilities for learning, growing, and ultimately improving the services offered to those the program strives to reach.
 
Article
Recent years have seen an increased interest in developing culturally and linguistically responsive systems of care in substance abuse treatment in the United States. This study examines the extent to which external and internal organizational pressures contributed to the degree of adoption of culturally and linguistically responsive practices in the nation's outpatient substance abuse treatment system early in the period of development of this system of care. Findings show that a higher degree of adoption of culturally competent practices was most likely in treatment programs with high dependence on external funding and regulation. Internally, programs with a larger number of professionals were associated with the lowest degree of adoption, while managers' cultural sensitivity contributed significantly to a high degree of adoption of these responsive practices. Considering the passage of recent legislation enforcing the use of cultural and linguistic competence in health care, implications of these baseline findings on early adoption patterns are discussed for future research and health care policy evaluation.
 
Article
Evaluation researchers have been broadening the traditional client input--"black box" treatment--client outcome paradigm that has guided many evaluations of mental health, substance abuse treatment, and other intervention programs. The points of expansion are in the areas of treatment implementation and treatment processes, as well as "extratreatment" influences on treatment selection, duration, and, especially, outcome. This review illustrates the application of environmental assessment procedures--particularly social climate measures--in four aspects of evaluation research suggested by the more comprehensive model: (a) evaluating treatment implementation; (b) exploring treatment processes; (c) identifying extratreatment influences on client posttreatment functioning; and (d) operationalizing outcome variables. Conceptual and methodological issues raised by these applications are discussed, and the benefits to be derived from an expanded model of evaluation research--especially the greater potential for program improvement--are considered.
 
Article
This paper describes a methodology used to overcome the difficulties in evaluating and improving prevention programs. The methodology was applied in this case to a drug abuse prevention program. The methodology includes using health-oriented rather than pathology-oriented outcome measures, random assignment to groups, multiple settings, multiple outcome measures, goal-free evaluation, and the recording of process variables. The first year results showed program effectiveness in one setting and not in the other. Process analyses pointed toward improvements that could be made. Those improvements were implemented and led to greater program effectiveness in the second year.
 
Article
The client-provider relationship is increasingly evaluated as an active ingredient in the delivery of substance abuse treatment services. This study examines gender differences in client-provider relationship as an important treatment ingredient affecting retention in treatment and reduced post-treatment substance use. The study uses data collected for the National Treatment Improvement Evaluation Study (NTIES), a prospective, cohort study of U.S. substance abuse treatment programs and their clients. Data on individual characteristics were collected at the pre-treatment interview; on client-provider relationship and services received at treatment exit; and on post-treatment drug use at 12 months post-treatment. The analytic sample consists of 3027 clients from 59 service delivery units (1922 men and 1105 women). Structural equation modeling (SEM) was used to assess the structural relations and causal connections between relationship and service variables and treatment outcome variables. Results indicate that a positive client-provider relationship is related directly to longer duration and reduced post-treatment drug use for the total sample and for men analyzed separately. For women, a positive client-provider relationship was related directly to treatment duration and only indirectly to reduced post-treatment drug use. The findings point to the significance of including client-provider relationship in service delivery models--both as a therapeutic element as well as an element facilitative of matching services to specific client needs.
 
Article
State public health authorities are critical to the successful implementation of science based addiction treatment practices by community-based providers. The literature to date, however, lacks examples of state level policy strategies that promote evidence-based practices (EBPs). This mixed-methods study documents changes in two critical state-to-provider strategies aimed at accelerating use of evidence-based practices: purchasing levers (financial incentives and mechanisms) and policy or regulatory levers. A sample of 51 state representatives was interviewed. Single State Authorities for substance abuse treatment (SSAs) that fund providers directly or through managed care were significantly more likely to have contracts that required or encouraged evidence-based interventions, as compared to SSAs that fund providers indirectly through sub-state entities. Policy levers included EBP-related legislation, language in rules and regulations, and evidence-based criteria in state plans and standards. These differences in state policy are likely to result in significant state level variations regarding both the extent to which EBPs are implemented by community-based treatment providers and the quality of implementation.
 
Article
Substance abuse is a frequent and complicating feature of homelessness. Barriers to service access have prevented homeless individuals from receiving substance abuse care. Outreach to homeless persons is a mechanism for identifying homeless substance abusers and engaging them in treatment. This study describes a program of substance abuse treatment conducted by a medical care provider for homeless persons in Baltimore, Maryland and compares characteristics of outreach recipients to those of walk-in clients. The study also examines the differences in drug abuse pathology and selected treatment outcomes among homeless and non-homeless clients and the extent to which homeless persons are unidentified in the drug abuse treatment reporting system of the state. Homelessness is about 5.5 times more prevalent in the clientele of drug abuse treatment programs than is reflected in the statewide substance abuse management information system of Maryland. Composite scores on the Addiction Severity Index for homeless individuals are significantly higher (reflecting more serious problems) on every measure in the interview, compared to non-homeless individuals. Except for residential treatment settings, homeless persons demonstrate a shorter length-of-stay in substance abuse treatment than housed clients. Our outreach effort successfully located its targets. Outreach clients reported significantly higher levels of substance abuse than walk-in clients. In addition 42.3% of outreach contacts became service recipients. These findings indicate that outreach can be a successful method of targeting and engaging a segment of homeless substance abusers who are otherwise difficult to engage in treatment.
 
Article
Longitudinal studies require high follow-up rates in order to maintain statistical power, reduce bias, and enhance the generalizability of results. This study reports on locating and survey completion for a 10-year follow-up of the Focus on Families project, an investigation of 130 families headed by parents who were enrolled in methadone treatment for opiate addiction. Despite having no contact with participants in the study for at least 10 years, the project successfully located nearly 99% of parent participants and 98% of their children. Twenty-four percent of the parents and one child had died before the follow-up. Of the surviving sample, 91% of parents and 86% of the children completed the follow-up interview. Multiple techniques were used to locate study participants, including internet searches, researching court and public records, collaborating with government and service agencies, and contacting family and social networks. For more than half of the sample, costly efforts were required to locate individual participants.
 
Article
The process of liberalization and globalization of Indian economy has brought new opportunities and challenges in all areas of human endeavor including education. Educational institutions have to adopt new strategies to make best use of the opportunities and counter the challenges. One of these challenges is how to assess the performance of academic programs based on multiple criteria. Keeping this in view, this paper attempts to evaluate the performance efficiencies of 19 academic departments of IIT Roorkee (India) through data envelopment analysis (DEA) technique. The technique has been used to assess the performance of academic institutions in a number of countries like USA, UK, Australia, etc. But we are using it first time in Indian context to the best of our knowledge. Applying DEA models, we calculate technical, pure technical and scale efficiencies and identify the reference sets for inefficient departments. Input and output projections are also suggested for inefficient departments to reach the frontier. Overall performance, research performance and teaching performance are assessed separately using sensitivity analysis.
 
Article
This paper overviews the use of simple measures of acceptability in mental health facilities, centering on measures of consumer satisfaction. Specific guidelines for methodology are offered.
 
Article
Web-based e-learning is not restricted by time or place and can provide teachers with a learning environment that is flexible and convenient, enabling them to efficiently learn, quickly develop their professional expertise, and advance professionally. Many research reports on web-based e-learning have neglected the role of the teacher's perspective in the acceptance of using web-based e-learning systems for in-service education. We distributed questionnaires to 402 junior high school teachers in central Taiwan. This study used the Technology Acceptance Model (TAM) as our theoretical foundation and employed the Structure Equation Model (SEM) to examine factors that influenced intentions to use in-service training conducted through web-based e-learning. The results showed that motivation to use and Internet self-efficacy were significantly positively associated with behavioral intentions regarding the use of web-based e-learning for in-service training through the factors of perceived usefulness and perceived ease of use. The factor of computer anxiety had a significantly negative effect on behavioral intentions toward web-based e-learning in-service training through the factor of perceived ease of use. Perceived usefulness and motivation to use were the primary reasons for the acceptance by junior high school teachers of web-based e-learning systems for in-service training.
 
Article
Conflict of interest refers to a set of conditions in which professional judgment concerning the validity of research might be influenced by a secondary competing interest. The competing interest that has received most attention in the literature addressing the prevalence and effects of such conflicts on the practice of empirical research has been that of financial relationships between investigators and research sponsors. The potential for conflicts of interest to arise in the evaluation of drug prevention programs was raised by Moskowitz in this journal in 1993, but to date there has been no attempt made to estimate the scope of this problem. The present study addressed this issue using a sample of "model" school-based drug and violence prevention interventions by first, identifying the types or relationships that exist between program developers and program distributors, and, second, by assessing how many of the evaluations of these programs published in peer-reviewed journals had been conducted by the developers of the programs compared to independent evaluation teams. The data presented indicate that there are relatively few published evaluations that do not involve program developers and that there are few instances in which there is complete separation between the program developer and program distributor. Using the open systems model of the Institute of Medicine Committee on Research Integrity as a framework, it is argued that the culture and norms of the program developer and those of the program evaluator are fundamentally distinct and therefore failure to separate these roles produces high potential for conflict of interest to arise.
 
Article
In 1997, UNAIDS decided to implement Drug Access Initiatives (DAI) in four different pilot-countries. We studied the implementation of the DAI in Chile as part of the evaluation program conducted by the 'Agence Nationale de Recherche sur le SIDA' (ANRS/France). The objective was to understand how the politico-organizational dynamic influenced the implementation process of the DAI. Approximately 50 semi-directed interviews and observation activities were conducted with the actors who participated in the implementation of the DAI or who played a role in the HIV/AIDS context. The program theory models were established and their evolution analyzed. This article offers an original analysis of an international HIV/AIDS drug access program that was put in place at a time when such programs were seen as a priority by international and governmental institutions. It also offers some insights for the creation of international projects that will be locally implemented.
 
Article
Despite recognition of the central role that service accessibility (and availability) should assume within a system of care, the definition proposed in the feature article of this special issue does not identify specific factors that systems of care must take into account in order to serve diverse children with serious emotional disturbance and their families in a culturally competent manner. This paper provides support for inclusion of the concepts of access and availability in the system of care definition and presents important factors that systems of care must take into account in order to increase access in a culturally competent manner, a core system of care value.
 
Article
In this short paper I reflect on the application of a program logic modeling approach to visualizing evaluation theory as it relates to practical participatory evaluation (PPE). The work of Hansen, Alkin and associates presented in this volume is well-thought out, carefully done, rigorous, and important. I found that their application of the approach resonated well with my understanding of PPE with a few notable exceptions. Specifically, the essence of the partnership aspect of the approach was underemphasized as was stakeholders' role in augmenting the credibility of evaluation findings and PPEs contribution to conceptual use. I then reminisce about the development and evolution of the participatory evaluation framework that we have been working with for so many years. I raise the point that we think of it as a theoretical framework to bound inquiry, not as an evaluation theory per se, which raises an interesting question: when does a theoretical framework become a theory?
 
Article
In this case study, we detail and analyze how the Tobacco Control Evaluation Center (TCEC), an evaluation technical assistance center that serves approximately 100 local tobacco control organizations in California, endeavors to build capacity among the state-funded local providers it serves by using evaluation capacity building activities with an utilization-focused evaluation framework. We call this a "blended approach" and describe these methods. Satisfaction and demand for TCEC services are documented to provide measurements for evaluation capacity building. Final evaluation report scores from two intervention cycles (2004-2007 and 2007-2010) submitted to the California Health Department, Tobacco Control Division are also assessed and compared. These measures demonstrate an increase in evaluation capacity by local projects under TCEC's purview.
 
Article
In the past few years, "accountability" for public mental health programs has become differentiated in the minds of not only program evaluators, but also program managers and funders, including state and local-level legislators. Increasingly, these officials are becoming concerned with more than just the numbers and targets of services delivered, and the cost involved, and are looking for evidence of positive outcome or impact on clients to justify program implementation and maintenance. This represents a significant move beyond the two accountability models that most recently seemed to be the focus of most formal accountability efforts--performance measurement and quality assurance. Pressures for implementing these two alternatives seem to have been reduced somewhat by the new federal Administration, but even prior to its advent there had been a rapid escalation in awareness of and concern for client outcome measurement among important audiences, including state and local mental health policy-makers and the U.S. Congress. This presents a major new opportunity and challenge for program evaluators at this new accountability focus continues to gather momentum.
 
Article
Findings, issues, and lessons learned about program evaluation are examined from a national qualitative study of 15 faith-based human service programs targeting those in need in urban areas. Using a grounded theory design, five properties emerge as part of the evaluation network: (1) philosophy of accountability, (2) legitimacy, (3) evaluation design, (4) feedback loop, and (5) barriers to evaluation. While funders expect measurable outcomes to evaluate service effectiveness, respondents acknowledge other competing expectations of multiple constituents in religious and secular communities. What emerges is an excellent example of managing multiple program evaluation demands in programs that are particularly facile at process evaluation in the interest of quality service and relationship building. The article concludes with important lessons learned about the process of program evaluation.
 
Article
Emerging perspectives of control in human service delivery organizations are facilitated by rational accountability formulations whose tenets and assumptions remain basically unquestioned in organizational and accounting literature. This paper critiques the underpinnings of rational accountability formulations and their appropriateness for human service delivery organizations. In turn, a non-rational metaphor of the role of management accounting information is proposed which recognizes uncertainty for what it is rather than assuming the very certainties that cannot be found in the delivery of human services.
 
Article
A study of standard-setting efforts in accounting and auditing is reported. The study reveals four major areas of concern in a professional standard-setting effort: (1) issues related to the rationale for setting standards, (2) issues related to the standard-setting board and its support structure, (3) issues related to the content of standards and rules for generating them, and (4) issues that deal with how standards are put to use. Principles derived from the study of accounting and auditing are provided to illuminate and assess standard-setting efforts in evaluation.
 
Article
Base-line data from a community-based HIV outreach effort serving Puerto Rican injection drug users (IDUs) in Massachusetts identified that approximately half of their clients were born on the mainland and half on the island. Logistic regression methods examined the relationship between place of birth, primary language spoken, primary residence of family and needle sharing for a sample of 200 Puerto Rican IDUs residing in Massachusetts. Focus groups were used to interpret quantitative findings. A logistic regression model indicated that Puerto Rican IDUs born on mainland USA were 2.1 times more likely to share needles than IDUs born in Puerto Rico, after controlling for gender, age, education, drug overdose, incarceration history and psychiatric status. Also, Puerto Rican IDUs who were older had overdosed on drugs in the past year, had been incarcerated in their lifetime, and were homeless were significantly more likely to report having shared needles in the past 6 months compared to their counterparts. Focus group interviews with Puerto Rican outreach workers and individuals in recovery suggested that differences in needle sharing by mainland versus island born IDUs may be due to cultural differences in interpretation of the interview questions. Researchers examining HIV risk behaviors among culturally diverse substance abusers need to conduct more mixed-method studies to identify if different cultural groups understand quantitative measures differently. Incarceration may be a significant risk factor in the continued spread of HIV among IDUs and expanded HIV prevention efforts need to be developed that specifically target this high-risk group.
 
Article
Benefit-cost analysis was conducted to estimate the increase in earnings, increased tax revenues, value of less crime, and reductions in welfare costs attributable to nationwide implementation of rapid assessment, a promising intervention for raising student achievement in math and reading. Results suggest that social benefits would exceed total social costs by a ratio of 28. Fiscal benefits to the federal government would exceed costs to the federal treasury by a ratio of 93. Social benefits would exceed costs to each state treasury by a ratio no lower than 286, and fiscal benefits would exceed costs to each state treasury by a ratio no lower than 5, for all but two state treasuries. Sensitivity analyses suggest that the findings are robust to a 5-fold change in the underlying parameters.
 
Article
Information and communication technologies hold much promise for use in education in developing countries. This study reports on an evaluation conducted on the introduction of computers in the delivery of the mathematics curriculum in one of the provinces of South Africa. Although the request was for an outcome evaluation very early in the implementation of the program, it was tailored in such a way as to fulfill a more formative role. Despite substantial variability in implementation, and in most cases with very weak exposure of the learners to the intervention, sufficient evidence emerged to indicate that this mode of curriculum delivery may be effective. Improvement in mathematics performance was related to a range of variables: some concerned classroom teaching practices, some referred to social differences between the learners, and some to the specific intervention. The strongest of these predictors in the sample was the strength of the intervention: the more time learners spent on using the software to study mathematics, the more improvement they showed from 1 year to the next in their performance in the subject.
 
Article
This paper evaluates the effectiveness of the introduction of a Writing Center at a university, which aims at improving students' scientific writing abilities. In order to deal with the presumed limited utility of student feedback surveys for evaluating the effectiveness of educational programs, we use students' actual learning outcomes as our quality measure. Based on this objective measure, different statistical evaluation methods established in the labor market treatment literature are applied. We present and discuss the validity of these methods to evaluate educational programs and compare the results of these approaches to implications obtained using corresponding student surveys. Although almost all students reported the writing courses to be helpful, we find no significant effect of course participation on students' grades. This result highlights the need for institutions not to rely solely on student course evaluations for evidence-based policy decisions.
 
Article
This article outlines the development and implementation of a cost-effective approach to client and program evaluation. The indices presented summarize a client's acquisition of behavioral skills (Skill Acquisition Index) and progression along a continuum of becoming less dependent and more productive (Client Movement Index). Data are presented summarizing how the two non-monetary outcome measures have been used in formative program evaluation. The paper also discusses some of the problems involved in implementing cost-effective analyses in rehabilitation programs and suggests specific strategies to overcome the problems.
 
Article
This article compares and contrasts the evaluation activities described in Practical Participatory Evaluation (Cousins & Whitmore, 1998), Values-engaged Evaluation (Greene, 2005), and Emergent Realist Evaluation (Mark, Henry, & Julnes, 1998). Using the logic models developed to depict each of the three evaluation theories (Hansen, Alkin, & Wallace, this issue) as a starting point, both quantitative and qualitative analysis techniques are employed to discuss the similarities and differences across the practice prescriptions. The approaches are then described according to Miller's (2010) standards for empirical examinations of evaluation theory. Specifically, I offer speculation about their operational specificity and feasibility in practice. I argue that none of the models is completely specific, or wholly unique, and they all present challenges of adaptation into the field. However, the models each offer varying degrees of guidance and unique elements through their prescriptions.
 
Article
Unlabelled: Healthy People 2020 states ethnic health disparities are a priority for the US. Although considerable national statistics document ethnic-related health disparities, information specific to rural areas is scarce and does not provide direction for implementing chronic disease prevention programming. Therefore, the purpose of our project was to use the Hispanic Health Needs Assessment (HHNA), a tool designed by the National Alliance for Hispanic Health (NAHH), in culturally diverse, rural Southwest Kansas. Our focus areas included: access to healthcare, heart disease, diabetes, overweight, nutrition, and physical activity. Methods: The assessment followed six steps: (1) developing the assessment team, (2) data gathering using community member surveys, existing statistics and community leader interviews, (3) assembling the findings, (4) formulating recommendations for action at individual, institutional, community and policy levels, (5) sharing findings and program planning, and (6) sharing findings with NAHH. We identified several challenges collecting health related data in rural communities, but overall, the HHNA was a comprehensive and useful tool for guiding a community level health assessment. Conclusion: This process has provided our community partners with locally relevant statistics regarding the current status of health, health behaviors, and perceived community needs to inform resource allocation, program planning and applications for new funding initiatives.
 
Article
This paper presents work using case studies as a source of data to see if we could extrapolate from the specific to the general particularly with regard to understanding what constitutes effective practice in taking action on SDHI and as a way of enabling policy makers to make better use of knowledge within the case studies and as a way of better understanding what works, in what context and why. Case studies are important to evaluators in that they are relatively straightforward to undertake and because those involved in implementing an intervention are usually keen to profile the intervention. A checklist described in this paper will enable policy advisers and evaluators to quickly review a case study and right away see if it contains enough information to assist in the development of policy options for reducing socially determined health inequalities.
 
Article
This article aims to provide more insight into advantages and challenges of conducting action research with the intention to improve health care. Action research is distinctive in the sense that the researcher has a dual role as both researcher and implementer of the program studied. The article is based on two Norwegian action-oriented studies that aimed to explore the potential role of online self-help groups for breast cancer patients and adolescents with mentally ill parents respectively. We argue that action research can contribute both to the generation of knowledge, as well as a greater sense of ownership to the program among those who are intended to use it. Nonetheless, a potential conflict between the researcher's pursuit of data, and ethical considerations became apparent in the contexts studied here. Bearing these challenges in mind, we still conclude that action research offers an important contribution for the further development of health care services.
 
Article
This paper provides a foundation for understanding the importance of shared system of care definitions that encourage and support family involvement at every level of the system's development and implementation. The author articulates the necessity of family involvement in system of care efforts to assure effectiveness and sustainability. In addition to presenting a general overview of family involvement in system of care initiatives, the author further states the necessity of clarifying definitions and expectations of real-life family involvement for all stakeholders.
 
A simple linear theory of action for the teacher professional learning component of the Math Science Partnership. 
The first attempt at a graphical theory of action for the teacher professional learning component of the Math Science Partnership (adapted from Weiss, 1998).
RM-MSMSP program logic model relating teacher professional learning activities and outcomes. 
An organizational model for distinguishing between and integrating research and evaluation activities. 
Article
Theory-based evaluation (TBE) is an evaluation method that shows how a program will work under certain conditions and has been supported as a viable, evidence-based option in cases where randomized trials or high-quality quasi-experiments are not feasible. Despite the model's widely accepted theoretical appeal there are few examples of its well-implemented use, probably due to time and money limitations necessary for planning and a confusion over the definitions between research and evaluation functions and roles. In this paper, we describe the development of a theory-based evaluation design in a Math and Science Partnership (MSP) research project funded by the U.S. National Science Foundation (NSF). Through this work we developed an organizational model distinguishing between and integrating evaluation and research functions, explicating personnel roles and responsibilities, and highlighting connections between research and evaluation work. Although the research and evaluation components operated on independent budgeting, staffing, and implementation activities, we were able to combine datasets across activities to allow us to assess the integrity of the program theory, not just the hypothesized connections within it. This model has since been used for proposal development and has been invaluable as it creates a research and evaluation plan that is seamless from the beginning.
 
Article
This study proposes a psychological analysis of the relationships between people and their residential environment in two neighbourhoods in Rome, within the theoretical framework of place theory. The analysis was aimed at getting indications for neighbourhood improvement, which can lead to residential satisfaction and neighbourhood attachment. We considered both constructs as the result of the relationships between the physical attributes of the environment, the cognitive perceptions and the affective appraisals of residents, and the activities they carry out. The role of socio-demographic and residential variables was also considered. Theoretical implications of results and indications for neighbourhood improvement are discussed. Residential satisfaction and neighbourhood attachment have a different pattern of predictors, emerging from all the dimensions of analysis we considered. Using hierarchical linear models, cognitive, affective and behavioural variables emerged as significant first-level predictors of both criteria, and physical attributes were found to be significant second-level predictors. In addition, the joint analysis of objective neighbourhood features and residents' experience within a place-specific framework showed to be an effective approach to identify relevant domains for neighbourhood improvement. Commercial and leisure facilities can contribute to make the neighbourhood more lively; building density and green areas have inverse effects on the prevalence of social activities.
 
Article
Learning-by-doing and adaptive management require careful monitoring and evaluation of the outcomes of environmental policies and programs under implementation. Selecting relevant indicators is difficult, especially when monitoring over a longer period of time. Further challenges arise when policies are developed as a collaborative effort among multiple actors. This paper discusses an approach to design frameworks for long-term monitoring and evaluation in multi-actor systems. It uses Dynamic Actor Network Analysis (DANA) as an actor-sensitive method to reconstruct program theories. This is combined with elements of assumption-based planning to identify critical assumptions and associated indicators to incorporate the dynamic aspects related to long-term monitoring. An application of this approach is described for a case of water management in the Netherlands. Here, mapping multiple perspectives and identifying critical assumptions helped to broaden the scope of monitoring in important ways. Identifying associated indicators and expectations on their development in response to policy implementation proved more difficult. From this case, it can be concluded that the approach is feasible, useful, but also demanding. However, with continuing trends of networked governance and adaptive management, additional efforts to reflect these trends in monitoring and evaluation, through this and similar approaches, are needed.
 
Top-cited authors
C. Clifford Attkisson
  • University of California, San Francisco (UCSF)
Tuan Nguyen
  • The Harris Center
William Michael Trochim
  • Cornell University
John E Ware
  • John Ware Research Group
Norman Weatherby
  • Middle Tennessee State University