No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Structures et fonctions des systèmes d’évaluation complexes : comparaison de six pays d’Europe centrale et orientale
Abstract
La pratique de l’évaluation est essentielle pour la responsabilisation et l’apprentissage des administrations qui mettent en œuvre des politiques complexes. Cet article explore les relations entre les structures des systèmes d’évaluation et leurs fonctions. Les conclusions sont basées sur une analyse comparative de six systèmes nationaux chargés d’évaluer la politique de cohésion de l’Union européenne. L’étude identifie trois types de structures de système d’évaluation : centralisées avec une seule unité d’évaluation, décentralisées avec un organe de coordination et décentralisées sans organe de coordination. Ces systèmes diffèrent en termes d’orientation thématique des évaluations et d’utilisateurs ciblés. Les systèmes décentralisés se concentrent sur les utilisateurs internes des connaissances et produisent principalement des études opérationnelles ; leur fonction principale est l’apprentissage orienté vers l’intérieur pour une mise en œuvre harmonieuse du programme. Les systèmes centralisés remplissent une fonction plus stratégique, et tiennent compte du public externe et de la responsabilité externe des effets. Remarques à l’intention des praticiens Les praticiens qui conçoivent des systèmes d’évaluation multi-organisationnels doivent garder à l’esprit que leur structure et leurs fonctions sont interdépendantes. Si l’on vise à la fois l’imputabilité et l’apprentissage, le système d’évaluation a besoin d’un degré minimum de décentralisation d’une part, et de la présence d’un organe de coordination actif et indépendant d’autre part.
ResearchGate has not been able to resolve any citations for this publication.
Public managers require different types of knowledge to run programs successfully. This includes knowledge about the context, operational know-how, knowledge about the effects, and causal mechanisms. This knowledge comes from different sources, and evaluation studies are just one of them.
This article takes the perspective of knowledge users. It explores to what extent evaluation is a useful source of knowledge for public managers of cohesion policy. Findings are based on an extensive study of 116 Polish institutions: surveys with 945 program managers, followed by 78 interviews with key policy actors. The article concludes that: (a) utility of evaluation studies, in comparison to other sources of knowledge, is limited, (b) evaluation reports are used to some extent as a source of knowledge on effects and mechanisms, however, (c) "effects" are shallowly interpreted as smooth money spending, not socio-economic change.
In conclusion this article offers practical ideas on what evaluation practitioners could do to make evaluation more useful for knowledge users in policy implementation.
Evaluations do not take place in a vacuum. Evaluation systems are embedded within organizations; they shape and are shaped by organizational norms, processes, and behaviors. In International Organizations, evaluation systems are ubiquitous. Yet, little is known about how they “function,” namely how they are used, how they contribute to organizational performance, and how they influence actors’ behaviors. These are empirical questions that cannot be solved without a robust theoretical grounding, which is currently absent from the existing evaluation literature. This article seeks to bridge some of the identified gaps by weaving together insights from evaluation theory and international organization sociology into a unifying framework of factors. The article then demonstrates how the framework can be used to empirically study the relative power and dysfunction of evaluation systems within International Organizations. A forthcoming connected contribution will illustrate such empirical inquiry through the case of the World Bank’s project-level evaluation system.
All of the East-Central European countries have been diverging from the European Union (EU) mainstream in recent years, but Hungary most of all. This paper offers a country study of Hungary, focusing on both internal and external political transformations and on the “de-democratization” and “de-Europeanization” process as a serious divergence from mainstream EU developments due to the socio-economic and political crises of the past quarter-century. Hungary has become a “defective” or “Potemkin” democracy: since the 2010 elections the formal institutions of democracy have been nothing more than a façade for nondemocratic, authoritarian rule.
To date, the role of evaluation in parliaments has hardly been analyzed. This is a surprise as members of parliament are stakeholders par excellence, who could have an interest in evaluation. But do they? Through a systematic analysis of written questions and interrogations in Germany and Flanders we investigate MPs’ attention to evaluations. The following aspects are analyzed, from a comparative perspective: the content of questions on evaluation, the political profile of the MPs posing the questions, the share of questions on evaluation, and the distribution of questions over political parties, policy areas and over time.
Points for practitioners The role and use of evaluation in the democratic process might be enhanced. On the one side members of parliament might wish to be better informed on evaluation methods as well as on possibilities for using evaluation results for accountability and learning purposes. On the other side evaluators might strive for strengthening the demand for evaluation in the policy-making process and not only public administration.
A criticism of evaluation practice in regional policy in Central and Eastern Europe (CEE) countries is that it stresses its function in terms of accountability and formal reporting on progress, rather than its role in policy or institutional learning. This article aims to test this critique through the use of ‘system thinking’ and focuses on the case of regional policy in Poland, where evaluation practice has developed dynamically since 2000. It contributes to the literature by developing an analytical framework that sets out the role of evaluation in regional policy learning and identifies key factors and mechanisms that determine the development of an effective evaluation system for regional policy.
The analysis draws on all Polish evaluation studies conducted between 1999 and 2010, interviews with civil servants and experts involved in the implementation of cohesion policy programmes, as well as secondary data from earlier studies. The essay identifies key mechanisms and factors that determine the main function of evaluation as a learning tool that produces and utilises knowledge in the decision-making process. These main elements are: growth in funding of the policy; stability of institutions; motivators; and system architecture.
FULL ARTICLE CAN BE DOWNLOADED from this link: http://www.tandfonline.com/eprint/NKDh7jesfiybdDrdqY9c/full
This article examines the status of historical legacies in debates on the reform of public administration in East Central Europe. It identifies limitations of existing accounts and derives three dimensions for the further development of legacy explanations of administrative reform in East Central Europe. First, legacy arguments tend to zoom in on the negative effects of the communist past. Yet there is not one but many legacies that matter for post-communist reforms and these many legacies have to be carefully distinguished and conceptualized. Second, legacy explanations tend to search for broad similarities between the administrative past and the present set-up of East Central European administrations in order to demonstrate the importance of the legacy. The identification of similarities is, however, not sufficient for the identification of legacy effects. Instead, the article argues in favour of the identification of causal mechanisms of legacification to explain recent administrative developments in East Central Europe. Finally, the article draws attention to the interaction of legacy effects with other determinants of administrative reform such as European integration and political parties.
Points for practitioners
This article addresses primarily policy-makers who deal with the reform of public administration in Central and Eastern Europe. It addresses the issue of how administrative traditions and, generally, historical legacies affect the design of administrative reforms and the successful implementation of reforms. Conventional wisdom concentrates on the negative effects of the communist-type administration on contemporary reform in Central and Eastern Europe. This article advances a more differentiated perspective on the impact of historical legacies. It argues that communist administrations evolved over time and differed considerably across countries. The administrative experience of other historical periods further interacts with the communist legacy of the past. The article also identifies various mechanisms that help to ‘transport’ the legacy of the past into the contemporary administrative reform context. For administrative policy-makers this approach implies that they cannot take for granted that the effect of the communist legacy is identical across countries and they cannot even assume that the communist administration will be long-lasting after transition. Instead, it is recommended that the specifics of local administrative traditions and the kind of mechanisms that produce legacy effects in the context of contemporary reform efforts be examined more closely.
This article tries to describe Japanese npm (New Public Management) from two opposite poles: from the national legislative framework and from an experimental example in local government.
Since the late 1990s, although npm was developed from Anglo-Saxon experiences, it has been implemented in Japan at the national level in a unique manner. The crisis in public finance, the urgent need for public sector reform and political instability led to two extreme options: self-reform by the bureaucracy itself; and citizen empowerment resulting inpressure on the bureaucracy. While the second one has been struggling to obtain public consensus, expertise for its practice and institutionalization, the first has resulted, to a certain extent, in the reorganization and restructuring of administrative institutions andin the establishment of both a legal framework and an operational system for measuring performance and evaluating policy.
This article is intended to contribute to the understanding of evaluation use in the context of policy making and governance. It does this, first, by developing a framework that sets out the prerequisites of six possible functions of an evaluation ‘management response system’; and second, by analysing three aid organizations’ management response systems in relation to this framework. The prerequisites of the different functions are theoretically derived. The analysis finds that response systems have contributed to organizational legitimacy and achieved many of their intended functions, but only to an extent. Two factors – the system design, and top managers’ support for the system – were found to be critical for how these systems worked. The article also discusses whether a management response system meets the needs of public organizations and stakeholders operating in multi-actor policy making.
Although use is a core construct in the field of evaluation, neither the change processes through which evaluation affects attitudes, beliefs, and actions, nor the interim outcomes that lie between the evaluation and its ultimate goal—social betterment—have been sufficiently developed. We draw a number of these change mechanisms, such as justification, persuasion, and policy diffusion, from the social science research literature, and organize them into a framework that has three levels: individual, interpersonal, and collective. We illustrate how these change processes can be linked together to form “pathways” or working hypotheses that link evaluation processes to outcomes that move us along the road toward the goal of social betterment. In addition, we join with Kirkhart (2000) in moving beyond use, to focus our thinking on evaluation influence. Influence, combined with the set of mechanisms and interim outcomes presented here, offers a better way for thinking about, communicating, and adding to the evidence base about the consequences of evaluation and the relationship of evaluation to social betterment.
Articles should deal with topics applicable to the broad field of program evaluation. Articles may deal with evaluation practice or theory, but if the latter, implications for practicing evaluators should be clearly identified. Examples of contributions include, but are not limited to, reviews of new developments in evaluation and descriptions of a current evaluation effort, problem, or technique. Results of empirical evaluation studies will be published only if the methods and/or findings have clear utility to other evaluation practitioners. Manuscripts should include appropriate references and normally should not exceed 10 double-spaced type- written pages in length; longer articles will occasionally be published, but only where their importance to AJE readers is judged to be quite high. ABSTRACT Evaluation has been beset with serious divisions, including the paradigm wars and the seeming segmentation of evaluation prac- tice into distinct evaluation theories and approaches. In this paper, we describe key aspects of an integrative framework that may help evaluators move beyond such divisions. We offer a new scheme for categorizing evaluation methods within four inquiry modes, which are "families" or clusters of methods: description, classification, causal analysis, and values inquiry. In addition, we briefly describe a set of alternative evaluation purposes. We argue that, together with a form of realist philosophy, the framework of inquiry modes and evaluation purposes (1) provides a common lexicon for evaluators, which may help the field in moving beyond past divisions, and (2) offers a useful approach to evaluation planning.
This article examines key institutional drivers that may contribute to improving public sector efficiency and focuses on one of them in more detail: performance information and its role and use in the budget process (“performance budgeting”).
Background: Hundreds of evaluators visit the Claremont Colleges in southern California each year to discuss a wide range of topics related to improving the quality of evaluation practice. Debates between thought leaders in the field have been one of the most popular and informative ways to advance understanding about how best to practice evaluation in contemporary times. Purpose: The purpose of this article is to provide a written transcript of the 2009 Claremont Evaluation Debates. The first debate is between Michael Quinn Patton and Michael Scriven on the promise and pitfalls of utilization-focused evaluation. The second debate is between David Fetterman, Michael Quinn Patton, and Michael Scriven on the promise and pitfalls of empowerment evaluation. Setting: The debates occurred at the Claremont Graduate University on August 23-24, 2009. Several hundred evaluators from around the world also viewed and participated in the debates via a live webcast. Intervention: Not applicable. Research Design: Not applicable. Data Collection and Analysis: Not applicable. Findings: Not applicable.
How does the design of evaluation systems affect the different ways of using the results of evaluations? This article offers a conceptual model that outlines three ‘ideal’ types of evaluation systems. It is a heuristic tool for opening up the ‘black box’ of evaluation systems and assessing their qualitative differences in terms of types of ‘owners’ of evaluations, questions asked, methods deployed, answers provided and avenues for use of evaluative knowledge. We apply the model to study the case of the Lithuanian evaluation system. In contrast to the expectations of some of the previously developed models, it does use evaluation results, and we aim to understand why the generated evidence is more often used in some areas rather than others.
The impact of European Cohesion Policy in different contexts. Regional Studies. Cohesion Policy, an important pillar of the European Union, has always been closely scrutinized and subject to debate because of the size of the budget and supranational role of the European Commission. Recent research has acknowledged that the impact of Cohesion Policy is far from uniform; academic interest has shifted away from attempts to assess its ‘total impact’ towards an emphasis on the ‘conditioning factors’ that explain where, when and how policy is effective. This provides insights that can contribute to policy design. The five papers in this thematic issue contribute to this research agenda by showing in what contexts and under what conditions Cohesion Policy can be more effective.
This article puts institutional economics concepts to work to help identify the proper role of evaluation in organizations and circumvent key obstacles to evaluation use. Looking at the role of evaluation in bureaucracies through an economics lens has its limitations. But addressing ‘the rules of the evaluation game’ helps to complement currently dominant approaches that concentrate on evaluation quality and practices. In concert with systems thinking, the neo-institutional economics perspective provides useful pointers for the design of evaluation governance configurations geared to organizational learning and accountability.
Evaluation units, located within public institutions, are important actors responsible for the production and dissemination of evaluative knowledge in complex programming and institutional settings. The current evaluation literature does not adequately explain their role in fostering better evaluation use. The article offers an empirically tested framework for the analysis of the role of evaluation units as knowledge brokers. It is based on a systematic, interdisciplinary literature review and empirical research on evaluation units in Poland within the context of the European Union Cohesion Policy, with complementary evidence from the US federal government and international organizations. In the proposed framework, evaluation units are to perform six types of brokering activities: identifying knowledge users’ needs, acquiring credible knowledge, feeding it to users, building networks between producers and users, accumulating knowledge over time and promoting an evidence-based culture. This framework transforms evaluation units from mere buyers of expertise and producers of isolated reports into animators of reflexive social learning that steer streams of knowledge to decision makers.
Evaluations may perform a key role in political systems as they provide a basis for parliaments to hold their executives accountable. This is equally the case in the European Union. Yet, several factors may work against the usage of European Union evaluations for accountability purposes. Members of the European Parliament work under great time pressure and executives may have little incentives to produce high-quality evaluations. This article therefore addresses the question of to what extent and when Members of the European Parliament use ex post legislative evaluations. We present an analysis of 220 evaluations, studying how many were referred to in parliamentary questions. Our main finding is that 16% of the evaluations are followed up through questions. However, the parliamentary questions hardly serve accountability aims. Members of the European Parliament mostly use evaluations for agenda-setting purposes. The main variable explaining differences in the usage of evaluations is the level of conflict between the European Parliament and Commission during the legislative process.
Points for practitioners
This article studies the usage of ex post evaluations of European Union legislation by Members of the European Parliament for accountability purposes by analysing European Parliament questions. It shows that MEPs ask different types of questions, referring to ex post evaluations. Most of the questions reveal forward-looking rather than backward-looking motives, aimed at agenda-setting and policy change instead of accountability. It concludes that variance in parliamentary questions about the follow-up of evaluation outcomes can be explained by the level of conflict between the European Parliament and Commission during the legislative stage.
This article is about evaluation use. It focuses on the well-known paradox that evaluation is undertaken to improve policy, but in fact rarely does so. Additionally, the article also finds that justificatory uses of evaluation do not fit with evaluation's objective of policy improvement and social betterment. The article explains why the paradox exists and suggests applying organizational institutional theory to explain evaluation use. The key argument is that in order to explain all types of evaluation uses, including non-use and justificatory uses, the focus needs to be on the evaluating organization and its conditioning factors, rather than the evaluation itself.
The purpose of this research is to analyse and characterize the practices of French local
authorities in the field of performance management. It sets out to do so, as a first step,
by putting forward a theoretical analytical framework of the different models of local
performance management. It then goes on to draw on this framework and test it
empirically in order to understand the nature of the French model of local performance
management. The results of our research demonstrate the existence of a French, ‘performance
administration’ type model evident in the trends common to French local
authorities with regard to the instrumental and methodological content and the methods
of steering public performance.
This paper is a systematic analysis of the comparative method. Its emphasis is on both the limitations of the method and the ways in which, despite these limitations, it can be used to maximum advantage.
The comparative method is defined and analyzed in terms of its similarities and differences vis-à-vis the experimental and statistical methods. The principal difficulty facing the comparative method is that it must generalize on the basis of relatively few empirical cases. Four specific ways in which this difficulty may be resolved are discussed and illustrated: (1) increasing the number of cases as much as possible by means of longitudinal extension and a global range of analysis, (2) reducing the property space of the analysis, (3) focusing the comparative analysis on “comparable” cases (e.g., by means of area, diachronic, or intranation comparisons), and (4) focusing on the key variables.
It is argued that the case study method is closely related to the comparative method. Six types of case studies (the atheoretical, interpretative, hypothesis-generating, theory-confirming, theory-infirming, and deviant case analyses) are distinguished, and their theoretical value is analyzed.
http://eur.sagepub.com/content/21/4/355.full.pdf+html?rss=1
Researchers have for some time been interested in the relationship between the strategy and structure of an organization. In this article the authors discuss the most widely-held view on the nature of this relationship, and then suggest an alternative explanation. For them strategy, structure, and environment are closely linked. Whereas men may build the structure of an organization, in practice it is this very structure which later constrains the strategic choices they may make.
I seek to explain the disappointing performance of the field of comparative politics addressing the three basic questions: Why compare? What is comparable? and How? I also challenge the view that the methodology of comparison is pretty well known and established. Hosts of unsettled issues remain, while a growing cause of frustration and failure is the undetected proliferation of `cat-dogs' (or worse), that is, nonexistent aggregates which are bound to defy, on account of their non-comparable characteristics, any and all attempts at law-like generalizations. The bottom line is that the comparative endeavor suffers from loss of purpose.
The search for increased effectiveness of development aid and improved accountability mechanisms has led donors to place more emphasis on evaluation as the tool to discern what works, when and why in development assistance. Yet, as the attention to evaluation grows, an analysis of the evolution of evaluation systems and their adaptation to new demands has yet to be carried out. This article attempts to address this issue. By drawing mainly on data contained in the peer reviews of the development cooperation programmes of the members of the OECD Development Assistance Committee (DAC), this article discusses some of the challenges aid agencies' evaluation systems are currently facing, in terms of new demands and increasing pressures; capacity constraints; use of evaluation results; institutional position within the aid agency; and relationships to internal and external stakeholders and constituencies.
Scholars have long addressed the question of how to improve the usefulness of evaluation in the public sector. This article describes the role of evaluation within the Swiss Federation in order to investigate the conditions of evaluation use. The article is based on an examination of several offices in the federal administration. Evaluation within the Swiss federal administration shows great variety among the different units with regard to both the general understanding and the implementation of evaluation. These provide good examples for studying the relationship of purpose, utilization and institutional design in evaluation. Furthermore, decisions are often taken in the absence of basic considerations regarding the purpose or utility of the evaluation, and this substantially affects the subsequent institutional design of the evaluation. The authors recommend a clearer functional differentiation between different forms of evaluations as means for improving the evaluation measures currently used in and by the Swiss federal administration.
The relative importance of strategy and structure lies at the heart of the strategic management field, yet little is known about the associated performance implications of such changes. The purpose of this study was to investigate the financial performance impact of strategy and structure changes. We conducted an examination of 259 firms over a 36-year period using a quasi-experimental design. No significant change–performance relationships were found for tandem changes (i.e. when both strategy and structure were changed). But, we found that singular structure changes were associated with the highest performance. This finding was consistent with the theory: though strategy is important, proper deployment of firm uniqueness is paramount to performance enhancement.
The paper provides a critical assessment of the evaluation of European Union Cohesion policy, focusing on the current regulatory framework, and the difficulties this poses for achieving rigorous and useful evaluation outputs. The paper argues that the evaluation framework for Cohesion policy is limited to three core purposes: accountability, improved planning, and quality and performance, but that it would benefit from widening this to include other functions. The decentralization of evaluation to the Member States means the evaluation of Cohesion policy relies on the presence of a pre-existing evaluation culture and skills base in the regions. Further, obstacles to effective evaluation arise from the lack of data comparability, rigidity of time-scales and a focus on performance approaches.
What makes a classic? Identifying and revisiting the classics of public policy and administration
- S Balla
- M Lodge
- E Page
Evaluation in Multi-Actor Policy Processes
- Van Der Meer Frans-Bauke