New Directions for Evaluation

Published by Wiley
Online ISSN: 1534-875X
Publications
Chapter
Logic models are critical in helping evaluators understand the design and implementation of programs and advance the practice of case study approaches to evaluation.
 
Article
In entering a new millennium, it is a good time for evaluators to critically appraise their program evaluation approaches and decide which ones are most worthy of continued application and further development. It is equally important to decide which approaches are best abandoned. In this spirit, this monograph identifies and assesses twenty-two approaches often employed to evaluate programs. These approaches, in varying degrees, are unique and cover most program evaluation efforts. Two of the approaches, reflecting the political realities of evaluation, are often used illegitimately to falsely characterize a program's value and are labeled pseudo-evaluations. The remaining twenty approaches are typically used legitimately to judge programs and are divided into questions/methods-oriented approaches, improvement/accountability approaches, and social agenda/advocacy approaches. The best and most applicable of the program evaluation approaches appear to be Client-Centered/Responsive, Utilization-Focused, Decision/Accountability, Consumer-Oriented, Constructivist, Case Study, Outcome/Value-Added Assessment, and Accreditation, with the new Deliberative Democratic approach showing promise. The approaches judged indefensible or least useful were Politically Controlled, Public Relations, Accountability (especially payment by results), Clarification Hearing, and Program Theory-Based. The rest including Objectives-Based, Experimental Studies, Management Information Systems, Criticism and Connoisseurship, Mixed Methods, Benefit-Cost analysis, Performance Testing, and Objective Testing Programs were judged to have restricted though beneficial use in program evaluation. All legitimate approaches are enhanced when keyed to and assessed against professional standards for evaluations.
 
Article
Information technology is often a key factor in delivering public services well and at the lowest possible cost. This chapter presents five best practices in information technology performance management and measurement. These practices could be usefully applied in other support functions, such as financial management or human resources management.
 
Article
The federal evaluation of forty-one Community Integrated Service Systems projects is described, focusing on issues in conducting a national, cross-site evaluation and the findings of the implementation study of the first two years of the initiative.
 
Article
The authors’ case study narrative is about the seven-year process of building an ECB structure at this nationwide, not-for-profit health organization.
 
Article
This chapter summarizes the results of the 2004 Independent Consulting Topical Interest Group Industry Survey. Respondents provided information on demographic characteristics, consulting experience, services, methods, business structures, and income.
 
Article
Aotearoa New Zealand is a small bicultural island nation in the Pacific with a multicultural population, a context that poses unique challenges for new evaluators, as it does for our more experienced counterparts. As emerging evaluators, the key challenge we have faced is navigating relationships, at the interface between program funders and culturally diverse communities, with integrity, and to ensure meaningful, credible, and valid evaluation findings. Drawing on their early experiences, the authors discuss their discoveries on sailing through relationships toward evaluation that is “good.” © Wiley Periodicals, Inc., and the American Evaluation Association
 
Article
The Robert Wood Johnson Foundation sponsors the Fighting Back initiative, in which communities across the country receive funding to combat alcohol and drug use. The evaluators of this initiative have tried to present an evaluation that is responsive to both national policy agendas and local community contextual differences, and thus offer important insight for both levels.
 
Article
The evaluator in this chapter describes potential evaluation capacity-building activities in contrast to the specifics of an evaluation design.
 
Article
As illustrated in the Bunche–Da Vinci context, a value-engaged approach to evaluation emphasizes responsiveness to the particularities of the context, inclusion of and engagement with multiple stakeholder perspectives and experiences, and attention to the social and relational dimensions of evaluation practice.
 
Article
The Bunche–Da Vinci case presents a situation at Bunche Elementary School that the four theorists were asked to address in their evaluation designs.
 
Article
This chapter describes an evaluation of a teen dating violence prevention media campaign, including evaluation design and results, and the challenges that arose during the evaluation process. It makes recommendations for future evaluations of mass media campaigns that target adolescents.
 
Article
The authors review the unique characteristics of nonprofit organizations and explain how nonprofit organizations play a prominent role in providing today's public services. The authors describe how the field of professional evaluation developed in response to government performance-based management initiatives, noting that recent efforts at the federal level—Government Performance and Results Act (GPRA) and the Program Assessment Rating Tool (PART) in particular—have “revived” program evaluation and brought outcomes measurement into the forefront of public and nonprofit management. The chapter concludes with a reflection on the last 30 years, noting key important developments, milestones, and trends in the field, including calls for more effective philanthropy, an increase in accessible evaluation training manuals and tools, and organized efforts to build the evaluation capacity of nonprofit organizations. © Wiley Periodicals, Inc.
 
Article
The impact of No Child Left Behind (NCLB) is usually understood in relation to schools and districts, but the legislation has also affected community-based organizations that operate school-linked programs. This case study of an afterschool program in California demonstrates how educational accountability systems that emphasize students' academic achievement and scientifically based research prompted evaluators to modify evaluation questions, methods, and analytic techniques. The external demands of NCLB transformed the evaluation to support the relevance and value of this community-based program within the evolving framework of accountability for the school. © Wiley Periodicals, Inc.
 
Article
The authors write about evaluation, testing, and research and their relation to policy, planning, and program in the Jefferson County Public Schools (JCPS) in Louisville, Kentucky. The authors focus on evaluation and testing for accountability and on managing the unit for this purpose. In detail they show the many evaluation demands from both inside and outside JCPS, from the State of Kentucky to No Child Left Behind. Their everyday work context is active, often tumultuous. Managing evaluation in this context is a form of juggling, and the authors succeed in part because they “have seen most of it before.” Also contributing to their effectiveness are the leaders' managing styles. This case study gives a good glimpse of the everyday life in a school district evaluation shop. © Wiley Periodicals, Inc.
 
Article
A district evaluator in a large Texas district examines new challenges arising since implementation of No Child Left Behind, relating to (1) navigating competing requirements in state and federal accountability systems; (2) evaluating effectiveness of sanctions districts are required to address; (3) using scientifically based research (SBR) to select effective programs and interventions; and (4) initiating SBR given high student mobility, inefficient data-management systems, and competing priorities of local schools. This chapter details these challenges for district-level evaluators and highlights how they can implement processes that strike a balance between supporting decision making and conducting rigorous evaluation. © Wiley Periodicals, Inc.
 
Article
This chapter describes the framework and implementation of a program accountability system in a statewide initiative (South Carolina First Steps to School Readiness), which was developed (1) to enable practitioners to provide evaluation information required by legislative mandate and (2) to develop the capacity of practitioners to systematically plan their program, implement with quality, and self-evaluate. The components of this program are reflected in its name: Planning, Implementation, and Evaluation (PIE). The chapter describes PIE in relation to program theory, usage, and efforts to mainstream.
 
Article
This chapter examines issues in managing the politics of evaluation by considering the context in which an evaluation occurs and by maximizing both evaluators' independence from and their responsiveness to stakeholders.
 
Article
This chapter uses cluster analysis to distinguish three groups of leavers of Temporary Assistance to Needy Families in terms of their self-reported well-being after exit, verifies these clusters, and then uses classification tree analysis to isolate some of the factors that may be responsible for these differences in post-exit outcomes.
 
Article
Although the authorizing environment may dictate the nature and use of evaluations, state and locally conducted evaluation provides a distinct lens for understanding the conditions affecting program implementation and the outcomes of programs under study. The learning that transpires at the state and local government levels through evaluation can motivate stakeholders to use evaluation to influence the policy agenda.
 
Article
The profound commitments of a participatory action researcher to engagement with practice and to collaborative inquiry with service providers and service users are historically traced and reflectively analyzed.
 
Article
The authors analyze themes across the four case studies and show how each element of their conceptual definition of ECB is illuminated by case study details.
 
Article
A conception of evaluation as learning focuses attention on the critical inquiry cycle that incorporates use throughout the evaluation process.
 
Article
This chapter describes a concrete process that stakeholders can use to make predictions about the future performance of programs in local contexts. Within the field of evaluation, the discussion of validity as it relates to outcome evaluation seems to be focused largely on questions of internal validity (Did it work?) with less emphasis on external validity (Will it work?). However, recent debates about the credibility of evaluation evidence have called attention to how evaluations can inform predictions about future performance. Using this as a starting point, we expand upon the traditional framework regarding external validity that is closely associated with Donald Campbell. The result is a process for making predictions and taking action that is collaborative, systematic, feasible, and transparent. © Wiley Periodicals, Inc., and the American Evaluation Association.
 
Article
This case study of one middle school focuses on improving teachers' skills in datadriven decision making through analysis of student work and their own professional practice. The expectation that schools will make adequate yearly progress has pushed evaluation practice down to the teacher level, where teachers are asked to analyze and disaggregate standardized test scores to facilitate instructional decision making that will lead to increased student achievement. The authors analyze this change in relation to No Child Left Behind and to the literature on evaluation capacity building within schools. © Wiley Periodicals, Inc.
 
Article
Appreciative Inquiry was used to highlight the successes of a donor-supported project of working with victims of trauma in an environment of civil war and high security risk, and thereby also to honor the work of dedicated staff who often face extremely difficult situations.
 
Article
This chapter describes the adaptation of the case study method to assessing increasingly complex, comprehensive reform initiatives that highlight the blurring of the boundaries between phenomenon and context and the concurrence of multiple interventions. Completed studies of two education reform programs illustrate the ongoing challenges of identifying, measuring, and analyzing large-scale reforms at multiple levels and across sites.
 
Article
The editors make note of several trends and implications arising from the discussions in the earlier chapters, in particular, the need to recognize the complexity of evaluation contexts involving multiple sponsors and stakeholders. The significance of the multifaceted and complex environment for the conduct of evaluation in state and local government settings is noted, a summary of the historical context is provided, and a discussion of the importance of maintaining evaluator independence and objectivity when addressing the diverse needs of sponsors and stakeholders is included.
 
Article
This chapter describes potential drawbacks of using intent-to-treat (ITT) analyses to examine intervention effects and presents several additional analytic methods as alternatives to ITT.
 
Article
This chapter presents a macroeconomic framework for evaluating the effects of policy-based adjustment programs, with an illustration from a recent World Bank evaluation.
 
Article
This chapter describes methodological and logistical issues associated with the design, development, and management of the Marijuana Treatment Project, a multisite, randomized clinical trial investigating the effectiveness of brief interventions for individuals who are dependent on marijuana.
 
Article
The Center for Substance Abuse Treatment Methamphetamine Treatment Project, with seven sites and more than a thousand participants, is an evaluation of a manualized psychosocial outpatient approach for the treatment of methamphetamine-dependent individuals.
 
Article
In 1999, the National Science Foundation (NSF) awarded funds to the Evaluation Center at Western Michigan University to conduct an external evaluation of the Advanced Technological Education (ATE) program. ATE, a federally mandated program designed to increase the number and quality of skilled technicians in the U.S. workforce, has funded over 346 projects and centers across the nation. This case study describes the relationship between project-level involvement in the ATE program evaluation and the use and influence of the evaluation on project primary investigators and evaluators. Although this large, multisite program evaluation employed numerous evaluative data-collection and dissemination techniques, project leaders and evaluators associated the program evaluation primarily with an annual Web-based survey. The NSF's expectation that projects would complete the annual survey contributed to feelings of involvement and, in many cases, promoted use and impact. © Wiley Periodicals, Inc., and the American Evaluation Association.
 
Article
This chapter provides a brief synthesis of the examples described in the four case study chapters, linking them to the framework outlined in Chapter One and the design types offered in Chapter Two.
 
Article
National programs for AIDS need the capacity to conduct their own monitoring and evaluation, not only for monitoring the epidemic and reporting to donors, but also for improving the programs.
 
Article
This chapter examines implications from the application of a transformative lens and the concepts of cultural competency to increase our understanding of how evaluation can contribute to the goal of improving STEM outcomes for underrepresented groups.
 
Article
Few ethical issues seem to arouse the passion generated by discussions of whether or when evaluators should be advocates, adversaries, or neutral in light of the social goals that programs seek to achieve. The arguments for and against evaluation advocacy are examined in terms of the American Evaluation Association's “Guiding Principles for Evaluators,” more recent statements on advocacy and neutrality, and one aspect of practice: “closeness” to intended beneficiaries. The conclusion? It is time for a revised definition of Guiding Principle C (Integrity/Honesty), one that more effectively reflects our common ground and permits better articulation of standards.
 
Article
This chapter describes how Appreciative Inquiry methods were used to focus the evaluation of an international nonprofit organization's Africa-based center, develop interview guides and a questionnaire, engender trust, and maximize utilization of results.
 
Article
An operational plan for monitoring and evaluation with a detailed budget is an essential step in moving from an indicator set to a functioning monitoring and evaluation system. Experience suggests that this will happen only with appropriate funding incentives.
 
Article
The essential place of race and culture in the meanings of responsive evaluation are argued through (1) a historical accounting of the significant but inexplicably unknown contributions of early African American evaluators and (2) the unassailable warrant for contemporary responsiveness to our “long silenced cultures of color.”
 
Article
A description of the complex geographical and cultural context of the African Evaluation Guidelines, with special emphasis on the role of the African Evaluation Association, is followed by a discussion of the development of the guidelines, their implementation, and use.
 
Article
Taking a historical view of selected program evaluation approaches, this chapter articulates a personal lens and journey of understanding how culturally responsive evaluation is geared toward effectiveness, benefits, and outcomes of programs designed to serve the less powerful.
 
Article
In building on each agency's strengths, we must identify incentives and opportunities for collaboration, with the fundamental consensus that working together in a harmonized manner is better than going it alone.
 
Article
Recent events that have encouraged performance measurement in local governments and local private nonprofit organizations are described in this chapter. The author discusses some obstacles managers face in measuring outcomes at the local level, and he offers successful uses of outcomes reporting to address citizen concerns.
 
Article
This chapter describes the experience of an auditor's office in providing practical solutions to a long-term problem involving county jails.
 
Article
The authors focus on the growing international momentum for interagency or joint evaluations of humanitarian response. The Interagency Health and Nutrition Evaluation (IHE) initiative, established in 2003, is described and analyzed in this chapter. The aims of IHE are described, as are the five components in the IHE framework of analysis: outcomes, service performance, policy and planning, risks to health and nutrition, and humanitarian context. This chapter focuses on lessons learned and identifies options for institutionalizing IHEs. © Wiley Periodicals, Inc., and the American Evaluation Association.
 
Article
Using Christie's research as an example, the authors describe a variety of forms that a more evidence-based approach to evaluation theory could take and offer some suggestions to help increase the amount and impact of evidence in evaluation theory.
 
Article
Reflections on participatory evaluation with young people are offered.
 
Article
This chapter critically examines the appropriateness of a comprehensive performance measurement system, based on standard performance indicators, to agricultural and rural development programs in poor countries.
 
Article
Based on their personal experience and reflections, the authors describe and analyze the monitoring and evaluation system employed by Katrina Aid Today (KAT), a program created by a consortium of partner agencies to provide disaster recovery case management services throughout the United States. In 2005, Hurricane Katrina devastated communities along the U.S. Gulf Coast and displaced scores of people on a scale previously unknown in the United States. The authors' reflections on the KAT model provide suggestions for future evaluations of disaster case management. The need for flexibility in disaster recovery, monitoring and evaluation program design, critical aspects for implementing and adapting an interagency monitoring and evaluation system, evolving interagency data collection, developing outcome measures, and emphasizing program evaluation in disaster recovery are discussed. © Wiley Periodicals, Inc., and the American Evaluation Association.
 
Top-cited authors
Jennifer Greene
  • University of Illinois, Urbana-Champaign
J. Bradley Cousins
  • University of Ottawa
Elizabeth Whitmore
  • Carleton University
Thomas Schwandt
  • University of Illinois, Urbana-Champaign
Patricia Rogers
  • BetterEvaluation