No full-text available
To read the full-text of this research,
you can request a copy directly from the author.
Donaldson, S.I. (2005). Using program theory-driven evaluation science to crack the Da Vinci Code. New Directions for Evaluation, 106, 65-84.
Abstract
The evaluator in this chapter provides a realistic account of the actions he would take to provide external evaluation services using the program theory– driven evaluation science approach.
... A "logic model" in Stage 3 ( Figure 2) is a tool to facilitate adoption, implementation, evaluation, and use of a new curriculum. It is a causal path diagram that shows how different components and variables associated with a new curriculum or educational program would likely interrelate, leading to expected outcomes in the targeted population (Donaldson, 2007). As shown in Figure 2, assumptions about the new curriculum, expected processes, and how trainee outcomes will be manifested can be mapped with a logic model. ...
... There could be more than one way to conceptualize logic models for this purpose. We adopted a systemsinformed approach (Donaldson, 2007;Stufflebeam, 2004). Such logic models help in guiding various stakeholders toward more effective implementation of new programs. ...
... Because of the historically strong emphasis these institutions place on workforce development, the potential for trainees to find internship and job-placement opportunities through them was higher. These two products were intended to inform both formative and summative evaluations of the curriculum as the program was taken to scale in field settings (Chen, 1990;Donaldson, 2007). ...
The Health Information Technology for Economic and Clinical Health (HITECH) initiative of the 2009 American Recovery and Reinvestment Act in the United States was intended to promote meaningful use of electronic health records (EHRs). This article reports on a comprehensive, three-stage model employed to develop, validate, and facilitate regional implementation of a health care information technology curriculum for workforce development as part of that coordinated national effort. Building on needs assessed at the national level, the the stages involved: (a) curriculum design, (b) assuring quality of curricular products through validation and revision, and (c) design of a systems-based, curriculum implementation and evaluation protocol. The objective of the project was to prepare health care professionals with competencies necessary to implement EHRs meaningfully, thereby improving patient care. We produced content-validated and usable versions of curriculum goal frameworks, student learning outcomes, instructional materials, multiple-choice tests, and performance assessments for 40 HITECH units across four curriculum components: (a) Public Health IT, (b) Vendor Specific Systems, (c) Usability and Human Factors, and (d) Training and Instructional Design. The three-stage approach is an innovative, replicable, and stakeholder-oriented method that fills a gap in the health care literature for curriculum design.
... Over the decades a consistent starting point for TDE evaluators has been to begin by conducting a situational analysis, by clearly defining the problem and its context (Cole, 1999;Donaldson, 2005;McLaughlin & Jordan, 1999;Renger & Titcomb, 2002;Rosas, 2005). This is necessary to begin to define the program logic, which describes the linkages between the program theory, the inputs, activities, and outcomes (Leeuw, 2003). ...
... In the absence of time and resources the use of existing documentation (e.g., minutes, lesson plans, mission statements, etc.-hereinafter referred to as source documentation) could prove useful for generating a program theory. Making the program theory explicit has been referred to as cracking the Da Vinci code (Donaldson, 2005). Building on this analogy, the process of establishing and determining a program's intended theory which is encrypted within the source documentation requires some reverse engineering. ...
... To the author's knowledge, they have not been explicitly categorized in the literature as TDE approaches. However, it was reasoned that since each approach begins by using methods to develop a program theory, the necessary first step of TDE (Donaldson, 2005;McLaughlin & Jordan, 1999), they could be classified as TDE approaches. ...
Making the program theory explicit is an essential first step in Theory Driven Evaluation (TDE). Once explicit, the program logic can be established making necessary links between the program theory, activities, and outcomes. Despite its importance evaluators often encounter situations where the program theory is not explicitly stated. Under such circumstances evaluators require alternatives to generate a program theory with limited time and resources. Using source documentation (e.g., lesson plans, mission statements) to develop program theory is discussed in the evaluation literature as a viable alternative when time and resources do not permit a priori program theory development. Unfortunately, the evaluation literature is devoid of methodology illustrating how to translate source documentation into an explicitly stated program theory. The article describes the steps in using source documentation to develop and verify a program theory and illustrates the application of these steps. It concludes with a discussion about the feasibility and limitations of this methodology.
... Program theories are the crux of theory-driven forms of evaluation and are typically represented as graphical diagrams that specify relationships among programmatic actions, outcomes, and other factors , although they also may be expressed in tabular, narrative, or other forms. Such representations vary widely in their complexity and level of detail (Chen, 1990Chen, , 2005aChen, , 2005bChen, , 2005c Donaldson, 2007; Frechtling, 2007; Funnel, 1997; Gugiu & Rodriguez-Campos, 2007; McLaughlin & Jordan, 1999; Patton, 2008; Rogers, 2000 Rogers, , 2008 W. K. Kellogg Foundation, 2000; Wyatt Knowlton & Phillips, 2008). A typical, linear program theory model is shown inFigure 1. ...
... Real models are, of course, often much more complex, but the essential message is generally the same. The elements used to describe or represent a program theory often (but not always) include inputs, activities, and outputs, which in combination loosely form a program process theory, and initial outcomes (sometimes called short-term, proximal, or immediate outcomes), intermediate outcomes (sometimes called medial outcomes), and long-term outcomes (sometimes called distal outcomes or impacts), which are intended to represent a program impact theory, or some variation of these (Donaldson, 2007; Donaldson & Lipsey, 2006; Lipsey, Rossi, & Freeman, 2004; Patton, 2008). Inputs include various types of resources necessary to implement a program (e.g., human,Figure 1. Linear program theory model. ...
... Most importantly, a program theory should be plausible (i.e., having the outward appearance of truth, reason, or credibility) and stipulate the cause-and-effect sequence through which actions are presumed to produce long-term outcomes or benefits (Donaldson & Lipsey, 2006; Lipsey, 1993). Donaldson (2001 Donaldson ( , 2007) has described four potential sources of program theory. These include prior theory and research, implicit theories of those close to the program, observations of the program in operation, and exploratory research to test critical assumptions in regard to a presumed program theory. ...
Although the general conceptual basis appeared far earlier, theory-driven evaluation came to prominence only a few decades ago with the appearance of Chen's 1990 book Theory-Driven Evaluations. Since that time, the approach has attracted many supporters as well as detractors. In this paper, 45 cases of theory-driven evaluations, published over a twenty-year period, are systematically examined to ascertain how closely theory-driven evaluation practices comport with the key tenants of theory-driven evaluation as described and prescribed by prominent theoretical writers. Evidence derived from this review to repudiate or substantiate many of the claims put forth both by critics of and advocates for theory-driven forms of evaluation are presented and an agenda for future research on the approach is recommended. Evaluation theories describe and prescribe what evaluators do or should do when conducting evalua-tions. They specify such things as evaluation purposes, users, and uses, who participates in the eva-luation process and to what extent, general activities or strategies, method choices, and roles and responsibilities of the evaluator, among others (Fournier, 1995; Smith, 1993). Largely, such theories are normative in origin and have been derived from practice rather than theories that are put into practice (Chelimsky, 1998). Stimulated by Miller and Campbell's (2006) review of empowerment evaluation and Christie's (2003) research on the practice–theory relationship in evaluation, the authors of this review sought to replicate certain aspects of Miller and Campbell's study except that the phenomenon under investigation was theory-driven evaluation practice. Therefore, this inquiry also is intended to contribute to the scarcity of systematically derived knowledge about evaluation practice by investigating whether ''theoretical prescriptions and real-world practices do or do not align'' (Miller & Campbell, 2006, p. 297).
... Hall and O'Day [11] introduced the causal chain approach to evaluate road safety countermeasures that is a foundational construct to theory-driven program evaluation [9,[12][13][14][15][16][17]. However, despite the strengths of some road safety evaluations that align with previous and current thinking on program evaluation, road safety evaluations are still being published that have not followed standard conceptualization and methodology outlined in numerous program evaluation textbooks, journal articles and Web-based handbooks [2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20]. However, conceptual and methodological challenges inherent in many evaluations of road safety countermeasures can affect causal attribution. ...
... Although the use of a theory-driven approach with inclusion of process evaluation is not necessary for the conduct of road safety evaluations, seminal program evaluation textbooks, journal articles and Web-based handbooks pontificate on the importance of theory-driven evaluations with the use of causal chains and/or logic models to articulate hypothesized intermediate steps reflecting causal pathways, and of the inclusion of process evaluations (e.g., [2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20]). Most theory-driven evaluation approaches share three fundamental characteristics: (a) clearly present the theory of an intervention by describing the expected relationships among inputs, mediating processes, and short-and long-term outcomes; (b) measure all of the hypothesized constructs in the theory; and (c) analyze the data to assess the extent to which the hypothesized relationships actually occurred [76]. ...
Researchers have a long history in the conduct of evaluations of road safety countermeasures. However, despite the strengths of some evaluative road safety evaluations that align with previous and current thinking on program evaluation, few published road safety evaluations have followed standard conceptualization and methodology outlined in numerous program evaluation textbooks, journal articles and Web-based handbooks. However, conceptual and methodological challenges inherent in many evaluations of road safety countermeasures can affect causal attribution. Valid determination of causal attribution is enhanced by use of relevant theory or hypotheses on the putative mechanisms or pathways of change and by the use of a process evaluation to assess the actual implementation process. This article provides a detailed description of the constructs of causal chain, program logic models and process evaluation. This article provides an example of how these standard methods of theory-driven evaluation can improve the interpretation of outcomes and enhance causal attribution of a road safety countermeasure.
... Programme theory refers to the assumptions or ideas of stakeholders about how the programme works (Bickman, 1987), or how it will bring about change. Very often, programme theories are modest theories (Donaldson, 2007). These programme theories may also be implausible. ...
... The evaluations presented in this special edition utilised a theory-driven evaluation approach (Donaldson, 2007). A theory-driven evaluation approach requires a good http://www.sajhrm.co.za ...
Orientation: Programme evaluation is a transdiscipline, which examines whether a programme has merit or not. A programme is a coherent set of activities aimed at bringing about a change in people or their circumstances.
Research purpose: The purpose of this special edition is to introduce readers to the evaluation of human resource (HR) programmes.
Motivation for the study: There are few comprehensive evaluations of HR programmes despite many publications on functional efficiency measures of HR (i.e. measures of cost, time, quantity, error and quality).
Research design, approach and method: This article provides a value chain for HR activities and introduces the reader to programme theory-driven evaluation.
Main findings: In summarising all of the contributions in this edition, one of the main findings was the lack of programme evaluation experience within HR functions and the difficulty this posed for the evaluators.
Practical/managerial implications: This introductory article presents answers to two simple questions: What does HR do? and, What is programme evaluation? These answers will enable practitioners to understand what programme evaluators mean when we say that programme evaluation seeks to determine the merit of a programme.
Contribution/value-add: The main contribution of this introductory article is to set the scene for the HR evaluations that follow. It alerts the reader to the rich theory contribution in HR literature and how to apply this in a theory-driven evaluation.
... The first step in the process is for stakeholders to surface a program theory. A great deal has been written about how to do this (Chen, 1990; Donaldson, 2007; Knowlton & Phillips, 2009) and how a program theory should reflect a shared understanding of which outcomes stakeholders hope to achieve and how the various characteristics of the program, participants, and setting are expected to interact to bring about those outcomes (see Donaldson, 2001 Donaldson, , 2007 Donaldson & Gooler, 2003). With their program theory in hand, stakeholders review evaluations of candidate programs. ...
... The first step in the process is for stakeholders to surface a program theory. A great deal has been written about how to do this (Chen, 1990; Donaldson, 2007; Knowlton & Phillips, 2009) and how a program theory should reflect a shared understanding of which outcomes stakeholders hope to achieve and how the various characteristics of the program, participants, and setting are expected to interact to bring about those outcomes (see Donaldson, 2001 Donaldson, , 2007 Donaldson & Gooler, 2003). With their program theory in hand, stakeholders review evaluations of candidate programs. ...
This chapter describes a concrete process that stakeholders can use to make predictions about the future performance of programs in local contexts. Within the field of evaluation, the discussion of validity as it relates to outcome evaluation seems to be focused largely on questions of internal validity (Did it work?) with less emphasis on external validity (Will it work?). However, recent debates about the credibility of evaluation evidence have called attention to how evaluations can inform predictions about future performance. Using this as a starting point, we expand upon the traditional framework regarding external validity that is closely associated with Donald Campbell. The result is a process for making predictions and taking action that is collaborative, systematic, feasible, and transparent .
... Alignment is defined as placing or bringing critical system and organization functions into a logical sequence of activities for the purposes of description, explanation, and systematic analysis (Schalock & Verdugo, 2012). Program logic models articulate the operative relationships among a system or program's input, throughput, and output components (Donaldson, 2007; Gugui & Rodriquez-Campos, 2007; Millar, Simeone, & Carnevale, 2001). A program logic model allows one to determine the critical indicators that are used to conduct discrepancy analysis as a basis for policy development, and sequential analysis as a basis for systems change. ...
... p. 192), " Horizontal alignment is not only about reconstituting the performance context (e.g., strategy, goals, accountability, and protocols), but is also about reshaping relationships. " The analytic procedure involves sequential analysis, which is a commonly used method in systems analysis and change (Donaldson, 2007; Morzinski & Montagnini, 2002 ). The procedure involves multiple stakeholders using a sequential analysis matrix to evaluate the logical sequence between two components of a program logic model. ...
The authors describe a conceptual and measurement framework that provides a template for guiding policy development and systems change. The framework is built on the concepts of vertical and horizontal alignment, system‐level processes, and organization‐level practices. Application of the framework can structure the thinking and analytic activities of systems and organization‐level personnel and can help them identify significant disconnects between and among system‐level processes and organization‐level practices. Once the disconnects are identified, information can be used to develop policies that enhance the alignment or congruence between system‐level processes and organization‐level practices, or implement systems changes that enhance the logical sequence of input and throughput, input and output, and/or throughput and output program logical components. The article concludes with a discussion of a number of change‐oriented principles (vision of the future, simple communication, leadership, empowering, immediate feedback and reinforcement, organization's culture) and guidelines (pace the change process, reallocate and restructure resources, alignment process, partnerships, continuous quality improvement framework, system and organization‐level outputs, innovation, performance‐based evaluation and management system) that address significant challenges faced today by intellectual and developmental disabilities' organizations and systems.
... The first step in the process is for stakeholders to surface a program theory. A great deal has been written about how to do this (Chen, 1990; Donaldson, 2007; Knowlton & Phillips, 2009) and how a program theory should reflect a shared understanding of which outcomes stakeholders hope to achieve and how the various characteristics of the program, participants, and setting are expected to interact to bring about those outcomes (see Donaldson, 2001 Donaldson, , 2007 Donaldson & Gooler, 2003). With their program theory in hand, stakeholders review evaluations of candidate programs. ...
... The first step in the process is for stakeholders to surface a program theory. A great deal has been written about how to do this (Chen, 1990; Donaldson, 2007; Knowlton & Phillips, 2009) and how a program theory should reflect a shared understanding of which outcomes stakeholders hope to achieve and how the various characteristics of the program, participants, and setting are expected to interact to bring about those outcomes (see Donaldson, 2001 Donaldson, , 2007 Donaldson & Gooler, 2003). With their program theory in hand, stakeholders review evaluations of candidate programs. ...
This chapter describes a concrete process that stakeholders can use to make predictions about the future performance of programs in local contexts. Within the field of evaluation, the discussion of validity as it relates to outcome evaluation seems to be focused largely on questions of internal validity (Did it work?) with less emphasis on external validity (Will it work?). However, recent debates about the credibility of evaluation evidence have called attention to how evaluations can inform predictions about future performance. Using this as a starting point, we expand upon the traditional framework regarding external validity that is closely associated with Donald Campbell. The result is a process for making predictions and taking action that is collaborative, systematic, feasible, and transparent. © Wiley Periodicals, Inc., and the American Evaluation Association.
... This study is anchored upon program or logic theory that explains the set of cause-and-effect relationships through which a program is implemented and the outcomes it seeks to affect. This theory also clarifies how an intervention (a project, a program, a policy, or a strategy) is acknowledged to facilitate and contribute to a chain of results that produce the intended long-term impacts [6,15,56]. ...
Rapid population growth has increased the global demand for food. However, some studies have revealed that more than one-third of the global food production is lost during postharvest operations along the food supply chain. Managerial and technical limitations such as a lack of proper storage facilities, poor handling practices and, associated knowledge are among the main causes of food losses particularly in Africa. Maize is among the most important staple food and cash crops across sub-Saharan Africa. In Rwanda, most of the maize produced incurs losses during storage. Hermetic storage technologies (HST) have been proven to be effective in the control of post-harvest storage losses in maize. However, their adoption in maize storage has been low with farmers continuing to use less effective technologies. Therefore, this study assessed the impact of hermetic maize storage technologies adoption, on the income of 301 randomly selected smallholder maize farmers in Gatsibo District of Rwanda, using the endogenous switching regression (ESR). The results revealed that household size, post-harvest training, access to credit, distance to input provider, and the household head's experience in maize production were the major factors influencing farmers' decision to adopt HST. The occupation of the household head, number of plots reserved for other crops, training, household size, age of the household head, and household maize self-sufficiency goal significantly influenced income for both HST adopters and non-adopters. Overall, the adoption of HST had a positive and significant impact on income from stored maize, among those who adopted it. The study recommends that the government of Rwanda and other stakeholders should support the dissemination of HST to facilitate access. Thus, increased access to institutional support services such as post-harvest training, credit access, and agricultural input supply, should be a major part of efforts aimed at promoting the effective use of hermetic maize storage technologies among smallholder maize farmers in Rwanda.
... PT offers two potential benefits for researchers. It can articulate the theory for how programme interventions are to work and deliver outcomes (Davidoff et al. 2015) and can also be used for evaluation of programme interventions (Lipsey and Cordray 2000;Donaldson 2005;Goicolea et al. 2015). This is particularly important, as PT evaluations can capture cultural, individual and team influences on interventions through acknowledging the complexity of the context and its inherent subtleties (Albaek 1989;Dahler-Larsen 2001;Goicolea et al. 2015). ...
Lean in healthcare has continued to attract practitioner and research interest over the past two decades yet lacks evidence of sustained application. This longitudinal case study examines Lean healthcare interventions through the lens of Programme Theory, to provide an understanding of the design and implementation of a Lean programme. Through a qualitative single case study of an NHS organisation implementing Lean, content analysis of 70 reports from a 6-year period was undertaken and supported by interview data from 12 respondents involved in Lean initiatives. Through Programme Theory, we identify contextual moderators, internal and external, informing the design and development of Lean efforts. The key to successful and sustained Lean efforts is staff acceptance of the Programme Theory for delivering and maintaining outcomes. This has been identified as a key social determinant for Lean sustainability. Results from the research provide practical implications for designing a programme that accommodates resource challenges while incorporating multiple contextual drivers. The case study highlights an organisation’s Lean journey and the best practices that evolved; however, further longitudinal investigations are required to validate the findings.
... The theory underpinning this study is the program or logic that explains the set of cause-and-effect relationships through which a program is thought to work and the outcomes it seeks to affect. This theory also clarifies how an intervention (a project, a program, a policy, a strategy) is acknowledged to facilitate and contribute to a chain of results that produce the intended impacts (Bickman, 2000;Rogers et al., 2000;Donaldson, 2005). ...
Rapid population growth has increased the global demand for food. However, some studies have revealed that more than one-third of the global food production is lost during postharvest operations along the food supply chain. Managerial and technical limitations such as a lack of proper storage facilities, poor handling practices and, associated knowledge are among the main causes of food losses particularly in Africa. Maize is among the most important staple food and cash crops across sub-Saharan Africa. In Rwanda, most of the maize produced incurs losses during storage. Hermetic storage technologies (HST) have been proven to be effective in the control of post-harvest storage losses in maize. However, their adoption in maize storage has been low with farmers continuing to use less effective technologies. Therefore, this study assessed the impact of
hermetic maize storage technologies adoption, on the income of 301 randomly selected smallholder maize farmers in Gatsibo District of Rwanda, using the endogenous switching regression
(ESR). The results revealed that household size, post-harvest training, access to credit, distance to input provider, and the household head’s experience in maize production were the major factors influencing farmers’ decision to adopt HST. The occupation of the household head, number of
plots reserved for other crops, training, household size, age of the household head, and household maize self-sufficiency goal significantly influenced income for both HST adopters and nonadopters. Overall, the adoption of HST had a positive and significant impact on income from stored maize, among those who adopted it. The study recommends that the government of
Rwanda and other stakeholders should support the dissemination of HST to facilitate access. Thus, increased access to institutional support services such as post-harvest training, credit access,
and agricultural input supply, should be a major part of efforts aimed at promoting the effective use of hermetic maize storage technologies among smallholder maize farmers in Rwanda.
... It may be possible to develop SOPs cheaper and quicker using source documentation (Donaldson, 2005;Leeuw, 2003;Renger, 2011). For example, some aspects of treating cardiac arrest, like the two minute CPR cycle, are well documented (Xanthos, Bassiakou, Koudouna & Papadimitriou, 2009). ...
This article shares methods used to evaluate system cascading failures. A cascading failure occurs when a problem is passed from one subsystem to a downstream subsystem creating a domino effect that undermines system efficiency and effectiveness. First, the basics of system evaluation theory (SET) are reviewed. Then drawing on different examples from the evaluation of emergency response systems the article describes how a) standard operating procedures (SOPs) can be used to locate possible system cascading failure trigger points, and b) mock exercises and secondary data are used to evaluate these trigger points. The discussion highlights the need to expand SET's conceptualization to include within subsystem cascading failures in addition to between subsystem cascading failures. The extent to which program evaluation methods can be adapted for use in system evaluation is also discussed.
... Program theory, also known as program logic (Funnell, 1997), theory-based evaluation (Weiss, 1998), theory-driven evaluation (Chen, 1990), or program-theory-driven evaluation science (Donaldson, 2005), explains how an intervention in the form of a program can be expected to lead to an intended result. Program theory can be used as a conceptual framework to bring together existing evidence from multiple interventions and evaluations. ...
Corporate accelerators are organizational devices designed to bring together innovative new ventures and startups with specialist knowledge and creativity with the experience and funding of established companies. The main goal for the use of an accelerator program by an established corporation is to open up the innovation process and actively profit from the innovative capacity of new ventures. However, until now there has been little empirical investigation of the characteristics of this particular model of “open innovation”. This paper outlines the key features of corporate accelerator programs and presents empirical data on their characteristics. The existing literature generally reflects positive results from this form of innovation but displays a lack of empirically and theoretically grounded research of how accelerator programs work. Using a standard, holistic taxonomy for programs, covering such components as strategy, resources, roles and structure, we analyzed and assessed data from stakeholders engaged in this approach to innovation in order to understand the expectations and reasoning behind these programs.
... The Impact Pathway Analysis or IPA (Boru Douthwaite et al. 2003;B. Douthwaite et al. 2007) is inspired from the program-theory (PT), which is often referred as theory of change, theory-based evaluation (Weiss 1998), intervention logic (Vanheukelen 1997), or program theory-driven evaluation science (Donaldson 2005;Coryn et al. 2011). Program-theory (PT) refers to a set of possibilities for developing a comprehensive impact logic model of the program with the aim to guide evaluation of an intervention (Rogers et al. 2000). ...
This paper attempts to assess the impacts of science-based research and innovation program (ISRIP) of farmer's transition to organic rice in Camargue. We applied a methods-mix revolving around the Impact Pathway Analysis (IPA) with the goal of shedding light on complex mechanisms that occur along innovation processes and ex-post reconstructing the impact pathway of the " basket of innovations " which was identified as the main innovation in the theory of change of the program. The analysis allows identification of the specific role of the research. We demonstrate that the French National Institute of Research Agriculture (INRA) contributed to change in supporting growers through experimental trials, discussions and leaflets even though the rice producers underlined that their trials played a more important role to find relevant techniques. Farmers also reported other driving factors like the selling price of organic rice or CAP subsidies for conversion and maintenance of land for organic rice. Finally, the analysis emphasises that the research system is not the central node in the innovation pathway, although it does play a substantial role.
... Building on this, the scope and focus of theory-based evaluations, both in short-term and longterm, project-specific evaluations, should be considered within the context of systems change. The recent use of theory-based evaluation in development programs, investigating and testing causal links in a project's or program's impact theory, attempts to address this by explicitly stating assumptions in proposing relationships between program interventions and outputs and development outcomes and impacts (Coryn et al., 2011;Donaldson, 2007). ...
This article presents discussion and recommendations on approaches to retrospectively evaluating development interventions in the long term through a systems lens. It is based on experiences from the implementation of an 18-month study to investigate the impact of development interventions on economic and social change over a 40-year period in the Koshi Hills region of Nepal. A multi-disciplinary team used a mixed-methods approach to data collection and analysis. A theory-based analytical approach was utilized to produce narratives of plausible cause and effect and identify key drivers of change within the context of cumulative and interconnected impacts of multiple programs and factors. This article responds to increasing interest in development evaluation to look beyond intervention-specific impact to broader determinants of change to assist with intervention planning. It is the authors’ hope that this will stimulate debate and progress in the use of high quality research to inform future development work.
... At the heart of much of this work is a ''logic model'' or results chain in which organizational inputs (e.g., knowledge, equipment, and financial resources) are used to support activities or processes for the production of goods and services (e.g., food, shelter, health services, schooling, job training, etc.) that in turn result in the delivery of outputs to a target beneficiary population (typically measured in terms of the number of people reached within that target population and immediate benefits to them). These short-term outputs are expected, over time, to lead to improved outcomes in the lives of beneficiaries typically measured in terms of longterm benefits (e.g., increased incomes, health, social integration, or quality of life) (Bickman, 1987; Chen & Rossi, 1983; Donaldson, 2007; Ebrahim & Rangan, 2014; Liket, Rey-Garcia, & Maas, 2014; Weiss, 1972). The distinction between outputs and outcomes has also been described in terms of proximal and distal goals, particularly in the field of health, with the former referring to delivery of services or goods and short-term changes in patient behavior, and the latter to longer-term improvements in health that arise from achieving a combination of distal goals (Brenner, Curbow, & Legro, 1995; Seijts & Latham, 2001; Singh-Manoux, Clarke, & Marmot, 2002 ). ...
We examine the challenges of governance and accountability facing organizations that pursue a social mission through the use of market mechanisms. We call such organizations socioeconomic hybrids because they combine market and social welfare logics in their activities. We distinguish between two ideal types of hybrid organizations — differentiated and integrated — and we elaborate on the persistent accountability dilemmas faced by such organizations: achieving the dual objectives of profit-making and social mission, and addressing the demands of multiple constituencies. We then theorize about the roles of governing boards in addressing these challenges in order to maintain hybridity. We offer three sets of propositions on the role of boards in: a) monitoring the relationship between social and commercial activities; b) developing appropriate control strategies for monitoring performance; and, c) enacting meaningful forms of downward accountability to beneficiaries. We propose that the mechanisms for addressing these challenges differ between integrated and differentiated hybrid organizations.
... A TBE is an extremely useful tool as it is sufficiently flexible for evaluators to identify and prioritize evaluation criteria and therefore tailor the evaluation methodology to answer the specific evaluation questions (Donaldson & Gooler, 2003). The approach is method neutral as it argues that quantitative, qualitative or mixed method designs are neither superior nor applicable in every evaluation situation (Donaldson, 2005), but instead it is a case of horses-for-courses. ...
Theory-based evaluation (TBE) is an effectiveness assessment technique that critically analyses the theory underlying an intervention. Whilst its use has been widely reported in the area of social programmes, it is less applied in the field of energy and climate change policy evaluations. This paper reports a recent study that has evaluated the effectiveness of the national biofuel policy (NBP) for the transport sector in Malaysia by adapting a TBE approach. Three evaluation criteria were derived from the official goals of the NBP, those are (i) improve sustainability and environmental friendliness, (ii) reduce fossil fuel dependency, and (iii) enhance stakeholders' welfare. The policy theory underlying the NBP has been reconstructed through critical examination of the policy and regulatory documents followed by a rigorous appraisal of the causal link within the policy theory through the application of scientific knowledge. This study has identified several weaknesses in the policy framework that may engender the policy to be ineffective. Experiences with the use of a TBE approach for policy evaluations are also shared in this report.
Copyright © 2015 Elsevier Ltd. All rights reserved.
... An evaluation approach that can be used to evaluate any situation whether through formative evaluation or summative evaluation, is the approach that uses theory-based evaluation. This is because this approach can describe the function of causal mechanisms that can affect the project's success or failure (Donaldson & Gooler, 2003; Cho & White, 2005; Donaldson, 2007). Theory-based evaluation has many examples from research carried out abroad, but on the issue of education reform policy implementation, from the researcher's search of the literature, there is little clear evidence. ...
One of the problems found in the first decade of education reform in Thailand was practitioner's misunderstanding due to confused guidelines on educational management issued by the central authority. The purpose of this research is to develop a program theory using three approaches. A deductive approach focuses on stated policies and previous research. An inductive approach builds from observing the intervention in action. A mental model approach focuses on working with stakeholders to articulate their tacit understandings of how the intervention works. Data were collected through a variety of means: questionnaires sent to a sample of teachers and administrators, review of documents, and unstructured interviews with stakeholders. Using the method of structural equation modeling to analyze data related to the education reform policy implementation program, this study demonstrates how evaluation practitioners can combine three approaches. The study concludes with a discussion of methodological issues related to developing and testing models of program theory.
... At the heart of much of this work is a ''logic model'' or results chain in which organizational inputs (e.g., knowledge, equipment, and financial resources) are used to support activities or processes for the production of goods and services (e.g., food, shelter, health services, schooling, job training, etc.) that in turn result in the delivery of outputs to a target beneficiary population (typically measured in terms of the number of people reached within that target population and immediate benefits to them). These short-term outputs are expected, over time, to lead to improved outcomes in the lives of beneficiaries typically measured in terms of longterm benefits (e.g., increased incomes, health, social integration, or quality of life) (Bickman, 1987; Chen & Rossi, 1983; Donaldson, 2007; Ebrahim & Rangan, 2014; Liket, Rey-Garcia, & Maas, 2014; Weiss, 1972). The distinction between outputs and outcomes has also been described in terms of proximal and distal goals, particularly in the field of health, with the former referring to delivery of services or goods and short-term changes in patient behavior, and the latter to longer-term improvements in health that arise from achieving a combination of distal goals (Brenner, Curbow, & Legro, 1995; Seijts & Latham, 2001; Singh-Manoux, Clarke, & Marmot, 2002 ). ...
... Although the exact definition of theory in evaluation is debated, it is evident that program theories offer a logical framework for program evaluations (Chen, 2003;Weiss, 1997). Program theories not only offer structure for measurement and data collection, but provide a basis for expectation and explanation of program processes, outcomes, and their connections (e.g., Donaldson, 2005). The use of theory to drive evaluations also represents the application of a standard to which basic social science is held, and is contributing to the maturity of evaluation as a discipline. ...
The force of fast-paced technological growth and globalization influence today's organizational environments. Successful organizations have become more nimble and responsive to these environmental realities, while simultaneously managing increasingly complex informational needs. The success of program evaluations will depend on evaluators' abilities to leverage future technologies to produce and disseminate knowledge in an accessible and actionable form. The authors propose that the roles and responsibilities of the evaluator will shift as current evaluation approaches are adapted to make effective use of these technological tools. Specifically, the authors discuss evaluators' future roles as collaborators with design-science practitioners, filterers and translators of information, and customizers of knowledge dissemination. © Wiley Periodicals, Inc., and the American Evaluation Association
... In programme evaluation terms, this is called programme theory, where 'theory' simply refers to a set of beliefs (or assumptions) that underlie the programme's activities. It typically consists of two aspects: impact theory and process theory (Donaldson, 2007; Rossi et al., 2009). Impact theory refers to a programme's cause-and-effect sequence – what is supposed to happen when the programme is delivered to the beneficiaries. ...
The Working for Water programme of the South African Department of Environmental Affairs has as its major objective the eradication of invasive alien plant species. However, it also has a social development component, which aims at the promotion of small business and entrepreneurship development. This paper explores the socio-economic rationale of one component, the Contractor Development Model. It does so via an examination of the programme's assumptions, and the development of its impact theory. The study revealed a shortcoming in monitoring data for the programme, and a relative lack of assessment of the effectiveness of its activities. It is suggested that the selection criteria for contractors be reexamined, and that consideration be given to two additional elements that may strengthen the effectiveness of the training: mentoring and networking.
... Our approach is based in a strategy know as 'programme theory evaluation', a tradition that is commonplace in the study of social interventions (Chen and Rossi, 1980;Coryn et al., 2010) but less widely used in bio-medical evaluation (Pawson and Sridharan, 2010;Funnell and Rogers, 2011). This approach is also known as 'theory-based' or 'theory-driven' evaluation (Atsbury and Leeuw, 2010;Donaldson, 2007;Leeuw, 2003;Rogers et al., 2000;Pawson and Tilley, 1997;Weiss, 1995Weiss, , 1997. Put simply, research designs begin by eliciting the ideas, the planning or the 'programme theory' that has gone into the making of an intervention and then move to a period of sustained empirical investigation whose purpose is to gauge the extent to which the theories are met in practice. ...
Purpose - Organ donation and transplantation services represent a microcosm of modern healthcare organisations. They are complex adaptive systems. They face perpetual problems of matching supply and demand. They operate under fierce time and resource constraints. And yet they have received relatively little attention from a systems perspective. This paper considers some of the fundamental issues in evaluating, improving and policy reform in such complex systems.
Design/methodology/approach - The article advocates an approach based on programme theory evaluation.
Findings - The paper explains how the death to donation to transplantation process depends on the accumulation of series of embedded, institutional sub-processes. Evaluators need to be concerned with this whole system rather than with its discrete parts or sectors. Policy makers may expect disappointment if they seek to improve donation rates by applying nudges or administrative reforms at a single point in the implementation chain.
Originality/value - These services represent concentrated, perfect storms of complexity and the paper offers guidance to practitioners with bio-medical backgrounds on how such services might be evaluated and improved. For the methodological audience the paper caters for the burgeoning interest in programme theory evaluation while illustrating the design phase of this research strategy.
... This should equip the research team to know whether trial outcomes can be attributed to the intended intervention or if there was implementation failure [22]. The intended implementation, consisting of inputs, process, and outputs—elsewhere termed 'implementation theory' [23], 'process theory' [35], or 'action model' [31]—will be described elsewhere, including the rationale for the design. The implementation will be assessed in terms of the aspects detailed inTable 2, following Saunders et al. [36], for each of the PRIME training intervention components (Health Centre Management , Fever Case Management, and Patient-Centered Services). ...
Despite significant investments into health improvement programmes in Uganda, health indicators and access to healthcare remain poor across the country. The PRIME trial aims to evaluate the impact of a complex intervention delivered in public health centres on health outcomes of children and management of malaria in rural Uganda. The intervention consists of four components: Health Centre Management; Fever Case Management; Patient- Centered Services; and support for supplies of malaria diagnostics and antimalarial drugs.
The PROCESS study will use mixed methods to evaluate the processes, mechanisms of change, and context of the PRIME intervention by addressing five objectives. First, to develop a comprehensive logic model of the intervention, articulating the project's hypothesised pathways to trial outcomes. Second, to evaluate the implementation of the intervention, including health worker training, health centre management tools, and the supply of artemether-lumefantrine (AL) and rapid diagnostic tests (RDTs) for malaria. Third, to understand mechanisms of change of the intervention components, including testing hypotheses and interpreting realities of the intervention, including resistance, in context. Fourth, to develop a contextual record over time of factors that may have affected implementation of the intervention, mechanisms of change, and trial outcomes, including factors at population, health centre and district levels. Fifth, to capture broader expected and unexpected impacts of the intervention and trial activities among community members, health centre workers, and private providers. Methods will include intervention logic mapping, questionnaires, recorded consultations, in-depth interviews, focus group discussions, and contextual data documentation.
The findings of this PROCESS study will be interpreted alongside the PRIME trial results. This will enable a greater ability to generalise the findings of the main trial. The investigators will attempt to assess which methods are most informative in such evaluations of complex interventions in low-resource settings.Trial registration: Clinicaltrials.gov NCT01024426.
... Programme theory, variously referred to as programme theory, programme logic (Funnell, 1997), theory-based evaluation or theory of change (Weiss, 1995(Weiss, , 1998, theory-driven evaluation (Chen, 1990), theory-of-action (Schorr, 1997), intervention logic (Nagarajan and Vanheukelen, 1997), impact pathway analysis (Douthwaite et al., 2003b), and programme theory-driven evaluation science (Donaldson, 2005) refers to a variety of ways of developing a causal modal linking programme inputs and activities to a chain of intended or observed outcomes, and then using this model to guide the evaluation . In this article, the term 'logic model' is used to refer to the summarized theory of how the intervention works (usually in diagrammatic form) and 'programme theory evaluation' is used for the process of developing a logic model and using this in some way in an evaluation. ...
This article proposes ways to use programme theory for evaluating aspects of programmes that are complicated or complex. It argues that there are useful distinctions to be drawn between aspects that are complicated and those that are complex, and provides examples of programme theory evaluations that have usefully represented and address both of these. While complexity has been defined in varied ways in previous discussions of evaluation theory and practice, this article draws on Glouberman and Zimmerman's conceptualization of the differences between what is complicated (multiple components) and what is complex (emergent). Complicated programme theory may be used to represent interventions with multiple components, multiple agencies, multiple simultaneous causal strands and/or multiple alternative causal strands. Complex programme theory may be used to represent recursive causality (with reinforcing loops), disproportionate relationships (where at critical levels, a small change can make a big difference — a `tipping point') and emergent outcomes.
... This conceptualization of education as future oriented is only functional within the traditional, disciplinary based schooling system. In examining education's role within an environmental organization, the mission of the organization would suggest that education, outreach, and communications efforts help achieve short and long-term conservation outcomes (Donaldson, 2005). As the educational philosophy under which an organization operates directly shapes the nature of its educational programs (Petress, 2003), the environmental education programs in agencies and organizations growing from the dominant philosophy of education in our society would likely default to a future focus rather than the desired short term focus. ...
Critical consideration of the role of environmental education in meeting conservation outcomes is increasingly necessary for environmental agencies and organizations. Evaluation can help move organizations to alignment between educational goals and organizational mission. Moving theory-driven evaluation into mission-based program theory, this chapter examines the ways in which educational goals can and should be linked to conservation outcomes for an agency or organization.
The paper presents the concept of 'program theory' as a way of conceptualizing key factors, presumably contributing to success, of a language program. It then considers two possible systemic structures advanced historically and eventually introduced into the literature of language teaching, that could be drawn on in conceptualizing or searching for program theory: open systems theory and activity theory. Finally, a pilot investigation underway of the Korean Language Flagship program is briefly and partially reported, which suggests a number of putative key features contributing to the program's success.
The paper presents the concept of 'program theory' as a way of conceptualizing key factors, presumably contributing to success, of a language program. It then considers two possible systemic structures advanced historically and eventually introduced into the literature of language teaching, that could be drawn on in conceptualizing or searching for program theory: open systems theory and activity theory. Finally, a pilot investigation underway of the Korean Language Flagship program is briefly and partially reported, which suggests a number of putative key features contributing to the program's success.
This article examines the relationship between systems thinking concepts and the logic model. Two notable shortcomings of the logic model are illustrated: depicting the program theory linearly and failing to place the program in context. Both issues lead to an artificial depiction of reality making evaluation findings difficult to interpret. Systems thinking concepts are defined and how they address these shortcomings is discussed. This article then demonstrates how many systems thinking concepts are evident in the logic model; although in a limited way. This article concludes by noting the importance of using system concepts when answering evaluation questions related to system dynamics and interrelationships.
This contribution to the journal examines program clarification for evaluation purposes. It traces the development of this approach over the past three decades, during which the terms evaluability assessment, program theory and program logic have been applied in turn. This is followed by an extensive list of resources that either discuss one of the terms generally or describe applications to a range of program areas.
Rigorous evidence of ‘what works’ to improve health care is in demand, but methods for the development of interventions have not been scrutinized in the same ways as methods for evaluation. This paper presents and examines intervention development processes of eight malaria health care interventions in East and West Africa. A case study approach was used to draw out experiences and insights from multidisciplinary teams who undertook to design and evaluate these studies. Four steps appeared necessary for intervention design: (1) definition of scope, with reference to evaluation possibilities; (2) research to inform design, including evidence and theory reviews and empirical formative research; (3) intervention design, including consideration and selection of approaches and development of activities and materials; (4) refining and finalizing the intervention, incorporating piloting and pretesting. Alongside these steps, projects produced theories, explicitly or implicitly, about (i) intended pathways of change and (ii) how their intervention would be implemented.The work required to design interventions that meet and contribute to current standards of evidence should not be underestimated. Furthermore, the process should be recognized not only as technical, but the result of micro and macro social, political and economic contexts, which should be acknowledged and documented in order to infer generalizability. Reporting of interventions should go beyond descriptions of final intervention components or techniques, to encompass the development process. The role that evaluation possibilities play in intervention design should be brought to the fore in debates over health care improvement.
The main goal of the paper that is called "Comparison of Selected Evaluation Approaches" is to show the variety of how evaluators could solve the evaluation tasks. Results of the performed analysis present how uneven the outcomes of evaluation could be when different approach is taken. The article turns attention toward the basics of evaluative thinking by opening the range of alternative methodological and conceptual frameworks that could be used in the course of intervention program evaluation.
Evaluation is defined with use of several definitions that put emphasis on methodological aspects of evaluative inquiry. In detail, there are four different approaches compared: goal-based, goal-free, constructivists and theory-driven. Each of the approaches is briefly introduced, context of its dissemination is explained and it is also described how the approach relates to other approaches. Useful component of the analysis is an example that highlights the main distinctions among the four approaches and results that each of the approaches can offer.
This action research dissertation study was undertaken to establish the foundation of a comprehensive evaluation component for the Turn-It-Around (TIA) workshop intervention program at Arizona State University (ASU), and was delivered in the form of a program development consultation. The study’s intent was to enhance the ASU Counseling Service’s departmental capacity to evaluate one of its important clinical services. The outcomes of this study included multiple assessments of TIA’s evaluability and the fidelity of its implementation to its program design. The study products include a well-articulated program theory comprised of program goals, learning objectives, a detailed description of program activities, a logic model, and theoretical construct checklist documents articulating the behavioral science theory underlying the TIA intervention. In addition, instruments tailored to the Turn-It-Around intervention that are suitable for assessing program outcomes were developed and are implementation ready. TIA’s clinical stakeholders were interviewed following the generation and delivery of the products and instruments mentioned above to determine whether they found the study’s processes and products to be worthwhile and useful. In general, the clinicians reported that they were very satisfied with the benefits and outcomes of the program development consultation. As an action research dissertation, this study generated useful and usable collateral materials in the form of reports, documents, and models. These products are now at the disposal of TIA’s institutional stakeholders for use in day-to-day business activities such as training new facilitators and liaisons, and giving presentations that describe the usefulness of TIA as an intervention. Beyond the documents generated to form a program evaluation infrastructure for Turn-It-Around, the processes involved in crafting the documents served to engage relevant stakeholders in a cycle of action research that enriched and solidified their understandings of TIA and furnished them with insight into their counterparts’ thinking about the intervention and its potential to benefit the college students they are responsible for helping. Consistent with the intent of action research, the processes involved in accomplishing the objectives of this study surfaced new topics and questions that will be useful in subsequent cycles of program improvement.
The author argues that evaluators must forge a strong bridge between basic social science and evaluation. Realizing this potential will involve holding evaluation to a high standard, through advancing cutting-edge empirical methods, balancing demands for accountability, and providing direction for basic social science. Approaches such as theory-based evaluation and natural-variation research designs can demonstrate the importance of evaluation, and help evaluators to ensure that traffic of ideas between basic social science and evaluation runs efficiently in both directions. © Wiley Periodicals, Inc., and the American Evaluation Association
Die Qualität von Dienstleistungen im Gesundheitsbereich ist nicht nur für Laien, sondern oft auch für ExpertInnen schwer einzuordnen. Ein Prüfsystem basierend auf Qualitätskriterien kann eine objektive und transparente Beurteilungsgrundlage für die Qualität von verschiedenen Angeboten und AnbieterInnen auf dem vielfaltigen Markt der Stress- und Burnoutprävention bieten. Aufbauend auf dem aktuellen Forschungsstand wurde in einem interdisziplinaren ExpertInnenteam ein Kriterienkatalog erarbeitet, der hochwertige gesundheitspräventive Angebote von weniger professionellen Angeboten differenzieren soll. Bei der Entwicklung wurde eine konzept- bzw. theoriebasierte Vorgehensweise gewählt, wobei die dabei erarbeiteten Indikatoren der Struktur-, Konzept-, Prozess- und Ergebnisqualität zugeordnet werden können. In einer ersten Analyse wurde der erarbeitete Qualitätskriterienkatalog geprüft. Das Ergebnis zeigt, dass die entwickelten Qualitätskriterien in der Lage sind, AnbieterInnen von gesundheitspräventiven Programmen zu vergleichen, zu differenzieren und zu bewerten.
This article demonstrates the potential of program theory evaluation (PTE) in evaluating sport league reforms through an analysis of the Union Cycliste International's (UCI) competition reform in professional road cycling. It is asserted that PTE can yield different insights in comparison with other evaluation models about why a sport league reform succeeds or fails. With respect to the UCI competition reform, previous assertions that the failure of the reform was caused exclusively by team-related problems are rejected. In contrast, it is asserted that the reform was unsuccessful due to problems related to both teams and race organizers: (1) the unwillingness of race organizers to participate in the competition; (2) implementation failure of the start duty of ProTour teams; and (3) the dependency of professional cycling on European races and teams.
Wat levert sociaal werk op? Om daarachter te komen is evaluatieonderzoek noodzakelijk. Weten wat werkt biedt inzicht in uiteenlopende vormen van evaluatie die passen bij de sociale sector. Slimme vormen van onderzoek die te vaak onbenut blijven. De auteurs maken gebruik van waargebeurde voorbeelden uit de dagelijkse praktijk van maatschappelijke ondersteuning in zorg en welzijn. Uit de voorbeelden blijkt dat samenwerking tussen onderzoekers en professionals bijdraagt aan praktisch bruikbare kennis over de doelen, de werkzame mechanismen en de effecten van de interventie. In de conclusie plaatsen de redacteuren de diverse bijdragen in een breder perspectief. Wat is de waarde van evaluatieonderzoek? Hoe kan kennis over wat werkt in het dagelijks handelen van professionals worden geïntegreerd? En hoe verhouden de opbrengsten van onderzoek zich tot politieke en morele keuzes? Deze publicatie biedt inspiratie en handvatten voor professionals, onderzoekers, beleidsmakers, docenten en (master) studenten social work die samen willen werken aan het antwoord op de vraag wat werkt in de sociale sector. Met bijdragen van onder meer Trudie Knijn, Radboud Engbersen, Erik Snel, Harry Michon en Anne Goossensen.
Current debates on impact evaluation have addressed the question ‘what works and what doesn’t?’ mainly focussing on methodology failures when providing evidence of impact. In order to answer that question, this article contrasts different approaches to evaluation in terms of the way they address different kinds of possible failures. First, there is more to be debated than simply methodological failures: there are also programme theory failures and implementations failures. Moreover, not all methodological failures are a simple matter of selection bias. Second, the article reviews issues that have recently been raised within different approaches relative to each failure. For programme theory failure, it is a matter of complexity and providing rival explanations; for implementation failure: how to use guidelines, and how to take context into account; and for methodology failure: how to move from internal to external validity, and to syntheses, within the framework of ‘situational responsiveness’. All these issues disclose a terrain for potential exchange between the protagonists of different approaches to impact evaluation.
Program Theory-Driven Evaluation Science fills the gap between 21st century literature on evaluation and what is happening in practice. it features detailed examples of how evaluations actually unfold in practice to develop people, programs, and organizations. Commonly accepted strategies for practicing evaluation are outlined, followed by comprehensive accounts of how those strategies have played out in the face of the complexities and challenges of “real world” settings. in so doing, the book illustrates the authentic challenges of implementing an evaluation approach in practice.
This paper will review the ways in which the effectiveness of counterterrorism measures and policies has been assessed and argue that they suffer largely from the same difficulties. For many of the chosen indicators it is not clear whether they represent what they are supposed to represent and whether shifts in the scores can be attributed to counterterrorism. After having discussed these problems, a different way of measuring counterterrorism effectiveness is outlined. It rests on the assumption that counterterrorism should be broken down into separate components that should all be evaluated separately. For all components, a causal chain from cause (measure) to effect (shift in indicator) should be formulated to solve the meaning and attribution problems.
This paper offers a framework for using a systems orientation and "foundational theory" to enhance theory-driven evaluations and logic models. The framework guides the process of identifying and explaining operative relationships and perspectives within human service program systems. Self-Determination Theory exemplifies how a foundational theory can be used to support the framework in a wide range of program evaluations. Two examples illustrate how applications of the framework have improved the evaluators' abilities to observe and explain program effect. In both exemplars improvements involved addressing and organizing into a single logic model heretofore seemingly disparate evaluation issues regarding valuing (by whose values); the role of organizational and program context; and evaluation anxiety and utilization.
The summer 2005 issue of New Directions for Evaluation, “Theorists’ Models in Action”, edited by Marvin C. Alkin and Christina A. Christie is, in part, inspired by the “Radnor Middle School” case study conducted in the early 1980s (Brandt, 1981). In the Radnor case study notable theorists including Scriven, Stake, Eisner, and Popham were asked to explain how they would evaluate Radnor’s humanities curriculum. In this issue Alkin and Christie asked four contemporary theorists—Jennifer C. Greene, Gary T. Henry, Stewart I. Donaldson, and Jean A. King—to describe how they would evaluate the case of the “Bunche-Da Vinci Learning Partnership Academy.” The Bunche-Da Vinci Learning Partnership Academy is essentially a “unique partnership between the [school] district and a nonprofit educational company specializing in innovative school interventions for low-performing students” (Eisenberg, Winters, & Alkin, 2005, p. 5).
Shows how theory-driven program evaluation promises to close the gap between the state of affairs in social programming, and the author's vision of a more desirable state where most social programs are (1) well-designed and based on sound theory and research, (2) implemented with high fidelity, (3) evaluated in a manner that minimizes the chances of design sensitivity and validity errors, (4) evaluated in a way that empowers the stakeholder to use the findings to continuously improve their efforts, and (5) evaluated so that cumulative knowledge and wisdom about social programming is advanced. This chapter provides evidence that theory-driven evaluation has gained considerable momentum, and is, or is quickly becoming, considered the "state-of-the-art" approach in many social program domains. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Excessive evaluation anxiety (XEA) can be a destructive phenomenon in modern program eval-uation. Some of the negative consequences include: Lack of access to important information and data; compliance and cooperation problems; false reporting; effects on bias and validity; and reduced utilization of evaluation findings. If left alone, XEA can lead stakeholders to behave in ways that destroy the credibility of evaluation findings and evaluators. The purpose of this paper is to examine the sources and signs of XEA in program evaluation, and to provide practicing evaluators with strategies to prevent and manage this common problem. This example of how more than technical skills are required to conduct high quality program evaluations illustrates the need for and begins a broader discussion of the psychology of evaluation.
While there is little disagreement about the need for, and value of, program evaluation, there remain major disagreements in the field about best practices. An exchange about the role of randomized control trials in program evaluation practice in educational settings set the stage for the 2004 Claremont Debate. In an effort to seek a deeper understanding of the current dispute, and to possibly discover a middleground or productive resolution, Claremont Graduate University hosted a debate between representatives from both sides.
Program Theory-Driven Evaluation Science fills the gap between 21st century literature on evaluation and what is happening in practice. it features detailed examples of how evaluations actually unfold in practice to develop people, programs, and organizations. Commonly accepted strategies for practicing evaluation are outlined, followed by comprehensive accounts of how those strategies have played out in the face of the complexities and challenges of “real world” settings. in so doing, the book illustrates the authentic challenges of implementing an evaluation approach in practice.
Presented here is the version of the American Evaluation Association Guiding Principles for Evaluators that was approved and copyrighted by the AEA board of directors and subsequently adopted by vote of the AEA membership.
The application of mixed methods under the framework of theory-driven evaluations can minimize the potential tension and conflict of mixing qualitative and quantitative methods, as well as compensate for each method's weaknesses. Mixed methods should not be applied indiscriminantly, however, but rather contingently under particular conditions as described in this chapter.
Despite the burgeoning literature explicating the benefits of theory-driven program development and evaluation, there remains a strong need for practical advice, written insights and experiences, and examples from evaluation practice illustrating how to implement this approach. The purpose of this paper is to move the field closer to a concrete understanding of the strengths, limitations, and challenges of implementing theory-driven program development and evaluation in modern human service organizations. This is accomplished by describing the evaluation process, resulting program theories, and lessons learned from the evaluation a five year, $20 million statewide Work and Health Initiative.
Centers for Disease Control Program Evaluation Framework
- Centers
Centers for Disease Control. Centers for Disease Control Program Evaluation Framework. Atlanta: Centers for Disease Control, 1999
Evaluation in the New Millennium: The Transdisciplinary Vision Evaluating Social Programs and Problems: Visions for the New Millennium
- M Scriven
Scriven, M. " Evaluation in the New Millennium: The Transdisciplinary Vision. " In S. I. Donaldson and M. Scriven (eds.), Evaluating Social Programs and Problems: Visions for the New Millennium. Mahwah, N.J.: Erlbaum, 2003. Shadish, W. R., Cook, T. D., and Campbell, D. T. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin, 2002.
Roles for Theory in Evaluation Practice
- S I Donaldson
- M W Lipsey
Donaldson, S. I., and Lipsey, M. W. "Roles for Theory in Evaluation Practice." In I. Shaw,
J. Greene, and M. Mark (eds.), Handbook of Evaluation. Thousand Oaks, Calif.: Sage,
forthcoming.
Handbook of Program Development for Health Behavior Research
- S Donaldson
Donaldson, S. I. "Mediator and Moderator Analysis in Program Development." In S.
Sussman (ed.), Handbook of Program Development for Health Behavior Research.
Thousand Oaks, Calif.: Sage, 2001b.
Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness
- H T Chen
Chen, H. T. Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness. Thousand Oaks, Calif.: Sage, 2005.
- P H Rossi
- M W Lipsey
Rossi, P. H., Lipsey, M. W., and Freeman, H. E. Evaluation: A Systematic Approach. (7th
ed.) Thousand Oaks, Calif.: Sage, 2004.
On Theory‐Based Evaluation: Winning Friends and Influencing People
- Weiss C. H.
Weiss, C. H. "On Theory-Based Evaluation: Winning Friends and Influencing People."
Evaluation Exchange, 2004b, 9(4), 1-5.
Applying Mixed Methods Under the Framework of Theory-Driven Evaluations Advances in Mixed-Method Evaluation: The Challenges and Benefits of Integrating Diverse Paradigms
- H Chen
Chen, H. T. " Applying Mixed Methods Under the Framework of Theory-Driven
Evaluations. " In J. C. Greene and V. J. Caracelli (eds.), Advances in Mixed-Method
Evaluation: The Challenges and Benefits of Integrating Diverse Paradigms. New
Directions for Evaluation, no. 74. San Francisco: Jossey-Bass, 1997.
Theory-Driven Program Evaluation in the New Millennium
- S I Donaldson
Donaldson, S. I. "Theory-Driven Program Evaluation in the New Millennium." In S. I.
Donaldson and M. Scriven (eds.), Evaluating Social Programs and Problems: Visions for
the New Millennium. Mahwah, N.J.: Erlbaum, 2003.
Evaluating Social Programs and Problems: Visions for the New Millennium
- S I Donaldson
Donaldson, S. I., and Scriven, M. (eds.). Evaluating Social Programs and Problems: Visions
for the New Millennium. Mahwah, N.J.: Erlbaum, 2003.
Evaluation in the New Millennium: The Transdisciplinary Vision
- M Scriven
Scriven, M. "Evaluation in the New Millennium: The Transdisciplinary Vision." In S. I.
Donaldson and M. Scriven (eds.), Evaluating Social Programs and Problems: Visions for
the New Millennium. Mahwah, N.J.: Erlbaum, 2003.
- Alkin
- Fitzpatrick
- Chen
- Donaldson
- Donaldson S. I.