In entering a new millennium, it is a good time for evaluators to critically appraise their program evaluation approaches and decide which ones are most worthy of continued application and further development. It is equally important to decide which approaches are best abandoned. In this spirit, this monograph identifies and assesses twenty-two approaches often employed to evaluate programs. These approaches, in varying degrees, are unique and cover most program evaluation efforts. Two of the approaches, reflecting the political realities of evaluation, are often used illegitimately to falsely characterize a program's value and are labeled pseudo-evaluations. The remaining twenty approaches are typically used legitimately to judge programs and are divided into questions/methods-oriented approaches, improvement/accountability approaches, and social agenda/advocacy approaches. The best and most applicable of the program evaluation approaches appear to be Client-Centered/Responsive, Utilization-Focused, Decision/Accountability, Consumer-Oriented, Constructivist, Case Study, Outcome/Value-Added Assessment, and Accreditation, with the new Deliberative Democratic approach showing promise. The approaches judged indefensible or least useful were Politically Controlled, Public Relations, Accountability (especially payment by results), Clarification Hearing, and Program Theory-Based. The rest including Objectives-Based, Experimental Studies, Management Information Systems, Criticism and Connoisseurship, Mixed Methods, Benefit-Cost analysis, Performance Testing, and Objective Testing Programs were judged to have restricted though beneficial use in program evaluation. All legitimate approaches are enhanced when keyed to and assessed against professional standards for evaluations.
Information technology is often a key factor in delivering public services well and at the lowest possible cost. This chapter presents five best practices in information technology performance management and measurement. These practices could be usefully applied in other support functions, such as financial management or human resources management.
The federal evaluation of forty-one Community Integrated Service Systems projects is described, focusing on issues in conducting a national, cross-site evaluation and the findings of the implementation study of the first two years of the initiative.
This chapter summarizes the results of the 2004 Independent Consulting Topical Interest Group Industry Survey. Respondents provided information on demographic characteristics, consulting experience, services, methods, business structures, and income.
The Robert Wood Johnson Foundation sponsors the Fighting Back initiative, in which communities across the country receive funding to combat alcohol and drug use. The evaluators of this initiative have tried to present an evaluation that is responsive to both national policy agendas and local community contextual differences, and thus offer important insight for both levels.
As illustrated in the Bunche–Da Vinci context, a value-engaged approach to evaluation emphasizes responsiveness to the particularities of the context, inclusion of and engagement with multiple stakeholder perspectives and experiences, and attention to the social and relational dimensions of evaluation practice.
This chapter describes an evaluation of a teen dating violence prevention media campaign, including evaluation design and results, and the challenges that arose during the evaluation process. It makes recommendations for future evaluations of mass media campaigns that target adolescents.
This chapter describes the framework and implementation of a program accountability system in a statewide initiative (South Carolina First Steps to School Readiness), which was developed (1) to enable practitioners to provide evaluation information required by legislative mandate and (2) to develop the capacity of practitioners to systematically plan their program, implement with quality, and self-evaluate. The components of this program are reflected in its name: Planning, Implementation, and Evaluation (PIE). The chapter describes PIE in relation to program theory, usage, and efforts to mainstream.
This chapter examines issues in managing the politics of evaluation by considering the context in which an evaluation occurs and by maximizing both evaluators' independence from and their responsiveness to stakeholders.
This chapter uses cluster analysis to distinguish three groups of leavers of Temporary Assistance to Needy Families in terms of their self-reported well-being after exit, verifies these clusters, and then uses classification tree analysis to isolate some of the factors that may be responsible for these differences in post-exit outcomes.
Although the authorizing environment may dictate the nature and use of evaluations, state and locally conducted evaluation provides a distinct lens for understanding the conditions affecting program implementation and the outcomes of programs under study. The learning that transpires at the state and local government levels through evaluation can motivate stakeholders to use evaluation to influence the policy agenda.
The profound commitments of a participatory action researcher to engagement with practice and to collaborative inquiry with service providers and service users are historically traced and reflectively analyzed.
Appreciative Inquiry was used to highlight the successes of a donor-supported project of working with victims of trauma in an environment of civil war and high security risk, and thereby also to honor the work of dedicated staff who often face extremely difficult situations.
This chapter describes the adaptation of the case study method to assessing increasingly complex, comprehensive reform initiatives that highlight the blurring of the boundaries between phenomenon and context and the concurrence of multiple interventions. Completed studies of two education reform programs illustrate the ongoing challenges of identifying, measuring, and analyzing large-scale reforms at multiple levels and across sites.
The editors make note of several trends and implications arising from the discussions in the earlier chapters, in particular, the need to recognize the complexity of evaluation contexts involving multiple sponsors and stakeholders. The significance of the multifaceted and complex environment for the conduct of evaluation in state and local government settings is noted, a summary of the historical context is provided, and a discussion of the importance of maintaining evaluator independence and objectivity when addressing the diverse needs of sponsors and stakeholders is included.
This chapter describes methodological and logistical issues associated with the design, development, and management of the Marijuana Treatment Project, a multisite, randomized clinical trial investigating the effectiveness of brief interventions for individuals who are dependent on marijuana.
The Center for Substance Abuse Treatment Methamphetamine Treatment Project, with seven sites and more than a thousand participants, is an evaluation of a manualized psychosocial outpatient approach for the treatment of methamphetamine-dependent individuals.
This chapter examines implications from the application of a transformative lens and the concepts of cultural competency to increase our understanding of how evaluation can contribute to the goal of improving STEM outcomes for underrepresented groups.
Few ethical issues seem to arouse the passion generated by discussions of whether or when evaluators should be advocates, adversaries, or neutral in light of the social goals that programs seek to achieve. The arguments for and against evaluation advocacy are examined in terms of the American Evaluation Association's “Guiding Principles for Evaluators,” more recent statements on advocacy and neutrality, and one aspect of practice: “closeness” to intended beneficiaries. The conclusion? It is time for a revised definition of Guiding Principle C (Integrity/Honesty), one that more effectively reflects our common ground and permits better articulation of standards.
This chapter describes how Appreciative Inquiry methods were used to focus the evaluation of an international nonprofit organization's Africa-based center, develop interview guides and a questionnaire, engender trust, and maximize utilization of results.
An operational plan for monitoring and evaluation with a detailed budget is an essential step in moving from an indicator set to a functioning monitoring and evaluation system. Experience suggests that this will happen only with appropriate funding incentives.
The essential place of race and culture in the meanings of responsive evaluation are argued through (1) a historical accounting of the significant but inexplicably unknown contributions of early African American evaluators and (2) the unassailable warrant for contemporary responsiveness to our “long silenced cultures of color.”
A description of the complex geographical and cultural context of the African Evaluation Guidelines, with special emphasis on the role of the African Evaluation Association, is followed by a discussion of the development of the guidelines, their implementation, and use.
Taking a historical view of selected program evaluation approaches, this chapter articulates a personal lens and journey of understanding how culturally responsive evaluation is geared toward effectiveness, benefits, and outcomes of programs designed to serve the less powerful.
In building on each agency's strengths, we must identify incentives and opportunities for collaboration, with the fundamental consensus that working together in a harmonized manner is better than going it alone.
Recent events that have encouraged performance measurement in local governments and local private nonprofit organizations are described in this chapter. The author discusses some obstacles managers face in measuring outcomes at the local level, and he offers successful uses of outcomes reporting to address citizen concerns.
Using Christie's research as an example, the authors describe a variety of forms that a more evidence-based approach to evaluation theory could take and offer some suggestions to help increase the amount and impact of evidence in evaluation theory.
This chapter critically examines the appropriateness of a comprehensive performance measurement system, based on standard performance indicators, to agricultural and rural development programs in poor countries.