No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Evaluation units as knowledge brokers: Testing and calibrating an innovative framework
Abstract
Evaluation units, located within public institutions, are important actors responsible for the production and dissemination of evaluative knowledge in complex programming and institutional settings. The current evaluation literature does not adequately explain their role in fostering better evaluation use. The article offers an empirically tested framework for the analysis of the role of evaluation units as knowledge brokers. It is based on a systematic, interdisciplinary literature review and empirical research on evaluation units in Poland within the context of the European Union Cohesion Policy, with complementary evidence from the US federal government and international organizations. In the proposed framework, evaluation units are to perform six types of brokering activities: identifying knowledge users’ needs, acquiring credible knowledge, feeding it to users, building networks between producers and users, accumulating knowledge over time and promoting an evidence-based culture. This framework transforms evaluation units from mere buyers of expertise and producers of isolated reports into animators of reflexive social learning that steer streams of knowledge to decision makers.
... In this same vein, other empirical studies have shown that analytical policy work, in general, occurs associated with other work, like those of a "relational" type, such as intergovernmental negotiation, public consultations, translation, and even democratization functions (Meltsner, 1976;Colebatch, Hoppe and Noordegraaf, 2010;Kohoutek, Nekola and Novotný, 2013;Olejniczak, Raimondo and Kupiec, 2016). The possible permeability of diverse sources of knowledge brought in by the different actors participating in policy with whom the bureaucracy interacts in its work must be acknowledged (Cairney, 2019). ...
... For the discussion in this chapter, we are interested in capabilities in their analytical dimension, which refers more specifically to "knowledge acquisition and its use in the processes developed in policy" (Howlett, 2009, p. 162). The specialized literature takes the analytical capacity of both bureaucrats and public organizations as a fundamental condition for enabling the flow of intelligence about and for policies to policy decision-makers (Olejniczak, Raimondo and Kupiec, 2016). ...
... Concerning analytical capabilities, by observing recommendations from the literature (Olejniczak, Raimondo and Kupiec, 2016;Wu, Ramesh and Howlett, 2015;Elgin and Weible, 2013;Pattyn and Brans, 2015), we sought to analyze both the effects of capacities accumulated at the individual level of bureaucrats and the impacts of capacities accumulated at the level of direct administration organizations, as presented in table 3. As for the individual level, analytical capacities were represented by the educational background, learned skills, and previous bureaucrats' experience, seeking to capture the analytical resources from formal knowledge and the analytical resources from tacit knowledge. ...
O uso de evidências como subsídio à atuação governamental não é tema novo no debate sobre a produção e legitimação da ação do Estado. Nas últimas décadas, no entanto, o movimento das políticas públicas baseadas em evidências (PPBEs) tem intensificado a defesa de que mais e melhores evidências sejam produzidas como instrumentos capazes de orientar a produção de políticas públicas. Esta publicação almejou ilustrar essa multiplicidade de recursos para produção de inferências com contribuições que mobilizam tanto estudos qualitativos como quantitativos ou mistos, além de experimentais. Ao longo da publicação, cada capítulo adota métodos distintos, como é de se esperar a partir dos variados objetos de análise.
... Por consequência, há uma variação relevante nas técnicas utilizadas, no formato de interação com atores internos e externos, além das diferentes questões (issues) de políticas em que esses profissionais se envolvem. Cenário similar a esse, apontando uma diversidade nos tipos e formatos de inserção na máquina pública, são observados em contextos tão diversos como o do governo federal canadense (Wellstead e Stedman, 2010), da República Tcheca (Veselý, 2014) Nessa mesma linha, outros estudos empíricos têm demonstrado que o trabalho analítico na política pública, em geral, ocorre associado a outros trabalhos, como os de tipo "relacional", tais como as funções de negociação intergovernamental, consultas públicas, tradução e até mesmo de democratização (Meltsner, 1976;Colebatch, Hoppe e Noordegraaf, 2010;Kohoutek, Nekola e Novotný, 2013;Olejniczak, Raimondo e Kupiec, 2016). Sendo assim, há que se reconhecer a possível permeabilidade de diversas fontes de conhecimento que são trazidas pelos diferentes atores que participam da política pública (Colebatch, Hoppe e Noordegraaf, 2010) com os quais a burocracia interage em seu trabalho (Cairney, 2019). ...
... Para a discussão deste capítulo, interessa-nos as capacidades em sua dimensão analítica remete mais especificamente "à aquisição de conhecimento e sua utilização nos processos desenvolvidos nas políticas públicas" (Howlett, 2009, p. 162). A literatura especializada toma a capacidade analítica tanto dos burocratas como das organizações públicas como condicionante fundamental para viabilizar o fluxo da inteligência sobre e para as políticas públicas para as instâncias decisórias das políticas públicas (Olejniczak, Raimondo e Kupiec, 2016). ...
... No que se refere às capacidades analíticas, observando-se recomendações da literatura (Olejniczak, Raimondo e Kupiec, 2016;Wu, Ramesh e Howlett, 2015;Elgin e Weible, 2013;Pattyn e Brans, 2015), buscamos analisar tanto os efeitos das capacidades acumuladas no nível individual dos burocratas como os efeitos das capacidades acumuladas no nível das organizações da administração direta, conforme apresentado na tabela 3. Quanto ao nível individual, as capacidades analíticas foram representadas pelo conjunto de formação, habilidades apreendidas e experiência prévia dos burocratas, buscando captar os recursos analíticos provindos dos conhecimentos formais, bem como os recursos analíticos provindos dos conhecimentos tácitos. ...
... The neutral assessment reports may also describe other findings, including evaluation capacity, as described in Goldman and al. (2019), as well as concrete examples of evaluation utilization. The contribution of evaluation to organizational decision-making, most notably, can be described in various ways, including: a) highlighting key evaluation recommendations and how they were implemented (Bourgeois and Whynot 2018a), b) the extent to which evaluation reports contribute to the accumulated body of evidence (Ginsburg and Rhett 2003;Olejniczak et al. 2016), c) the extent to which evaluations respond to the information demands of managers and other key stakeholders (Bourgeois and Whynot, 2018b;Bundi 2016;Perrin 2015), and d) the role that evaluators play internally in fostering evaluation use. ...
... This can be done by highlighting instances of instrumental or conceptual uses (Johnson et al. 2009), which describe how evaluation results and recommendations were implemented to improve programs or to support organization-wide decision-making. Neutral assessments should also comment on whether evaluations contribute to the accumulated body of evidence or policymaking (Andersen 2019;Ginsburg and Rhett 2003;Olejniczak, Raimondo and Kupiec, 2016), and on the role that evaluators have played in fostering evaluation use (Olejniczak, Raimondo and Kupiec 2016). Such evidence would serve not only to demonstrate the departmental function's value internally, but could contribute to creating a body of evidence on governmental results over time, as was advocated by the evaluators of the 2009 Policy on Evaluation. ...
... This can be done by highlighting instances of instrumental or conceptual uses (Johnson et al. 2009), which describe how evaluation results and recommendations were implemented to improve programs or to support organization-wide decision-making. Neutral assessments should also comment on whether evaluations contribute to the accumulated body of evidence or policymaking (Andersen 2019;Ginsburg and Rhett 2003;Olejniczak, Raimondo and Kupiec, 2016), and on the role that evaluators have played in fostering evaluation use (Olejniczak, Raimondo and Kupiec 2016). Such evidence would serve not only to demonstrate the departmental function's value internally, but could contribute to creating a body of evidence on governmental results over time, as was advocated by the evaluators of the 2009 Policy on Evaluation. ...
Most Canadian federal government departments and agencies are required to conduct a “neutral assessment” of their evaluation function every five years. Such assessments should focus on the capacity of the evaluation units to produce high‐quality evaluation reports which serve the needs of organizational decision‐makers. Our study sought to analyze neutral assessment reports produced over a ten‐year period. Overall, our review suggests that the neutral assessments consistently conform with central‐agency requirements as well as indicate evolving evaluation capacity and utilization. We provide recommendations for improving the function and its impact. La plupart des ministères et organismes du gouvernement fédéral canadien sont tenus d'effectuer un « examen neutre » de leur fonction d'évaluation tous les cinq ans. De tels examens devraient être axés sur la capacité des directions d'évaluation à produire des rapports d'évaluation de haute qualité qui répondent aux besoins des décideurs organisationnels. Le but de notre étude était d’analyser des rapports d'examens neutres effectués sur une durée de dix ans. Dans l'ensemble, notre analyse indique que les examens neutres sont invariablement conformes aux exigences des organismes centraux et indique aussi un progrès au niveau de la capacité et de l'utilisation de l'évaluation. Nous proposons des recommandations pour améliorer la fonction et ses résultats.
... A capacity-building strategy focuses on bolstering the skills, understanding, or selfreliance of those involved in the process of evidence-informed decision making (discussed in 48.6% of articles, see Figure 2; Ward et al, 2009;Bornbaum et al, 2015). Specifically, this strategy includes activities to build knowledge users' capacity to appraise the validity, trustworthiness, or methods used to produce research evidence (for example, Robeson et al, 2008;Dobbins et al, 2009;Armstrong et al, 2013;Olejniczak et al, 2016). It also includes technical assistance designed to build knowledge users' capacity to implement evidence-based programmes or practices (Jacobs et al, 2005;Lemos et al, 2012;Sharples and Sheard, 2015). ...
... BIBS can broaden or expand the range of options decision makers consider (Duncan et al, 2020), for example by 'navigating options and choices for strategic policy and planning ' (Choi et al, 2005: 635). BIBS can also focus attention by making the case for considering particular research (Olejniczak et al, 2016) and supporting the role of research in decision making by defending scientific expertise and articulating the scientific basis for sources of information (De Pryck and Wanneau, 2017). BIBS advise decision makers in 'accessing, appraising, adapting and applying findings' from research (Mallidou et al, 2018). ...
... Additionally, expertise in research and policy was described as important for identifying opportunities to bring research into policy conversations and for bridging communities of knowledge producers and users (for example, Campbell et al, 2011;Cooper, 2014;Cvitanovic et al, 2016;Hering, 2016;Cranley et al, 2017;Bednarek et al, 2018;Topp et al, 2018). Finally, BIBS' knowledge of change and decision-making processes were described as instrumental to their success in facilitating the use of research in practice and policy (for example, Lomas, 2007;Robeson et al, 2008;Cvitanovic et al, 2016;Olejniczak et al, 2016). ...
Background
Brokers, intermediaries, and boundary spanners (BIBS) bridge research and policy or practice, and can elevate the role of evidence in decision making. However, there is limited integration of the literature across different sectors to understand the strategies that BIBS use, the skills needed to carry out these strategies, and the expected outcomes of these strategies.
Aims and objectives
In this review, we characterise the strategies, skills, and outcomes of BIBS across the literature in education, environmental, health and other relevant sectors.
Methods
We included 185 conceptual and review papers written in English that included descriptions or conceptualisations of BIBS in the context of knowledge transfer or research use in the education, environmental, health, or other relevant sectors (for example, social services, international development). For each included paper, we extracted and coded information on sector, BIBS strategies, skills, and outcomes.
Findings
Our review revealed five strategies used by BIBS that were emphasised in the literature. Specifically, 79.5% of papers mentioned facilitating relationships, 75.7% mentioned disseminating evidence, 56.8% mentioned finding alignment, 48.6% mentioned capacity building, and 37.3% mentioned advising decisions as strategies used by BIBS. Additionally, papers described skills and expected outcomes that were common across these strategies as well as those that were unique to specific strategies.
Discussion and conclusions
We discuss implications of these findings for understanding how BIBS interface with knowledge users and producers as well as directions for future research on BIBS and the professionalisation of BIBS roles.
... A knowledge broker is the individual who connects information or creates new ways of using existing knowledge [10,46,48] Knowledge brokers act as bridges between different communities of practice and facilitate interaction. Knowledge brokers also facilitate the transfer of knowledge among organizational units [38,41]. The ability to develop relationships among colleagues is emphasized in virtual environments where physical connections are missing. ...
... Being able to identify knowledge brokers based on their messages in an ESM gives possibilities to learn about knowledge brokering and helps organizations improve knowledge sharing and collaboration further by using knowledge brokers as mediators. This paper contributes to a larger body of literature on knowledge brokering [14,32,38] and knowledge creation [16,35] by taking a communicative perspective on the strategic use of knowledge brokering. ...
... Our specific interest is on knowledge brokers-those knowledge workers who used social media widely to create, disseminate and share organizational knowledge. The importance of knowledge brokers has been recognized in prior studies in management and science [34,38]. Our research helps develop a better and more nuanced understanding of how the communicative actions of organizational members in a collaborative network may help in developing effective practices of knowledge brokering. ...
... The momentum of economic-physical intensity measures is due to a number of reasons, among which a remarkable support that comes from think tanks, public organisations, consulting groups and academics focused on energy and climate change policy development. Those actors play a pivotal role as bridges between scientific and policy-maker communities (Kauffeld-Monz and Fritsch, 2013;Olejniczak et al. 2016;Nutley et al., 2007). The ways intensity indicators are framed and perceived in the public debates generated by these intermediaries are important because they might influence policy making, favouring or dismissing certain political agendas or enabling/disabling the development of better sets of indicators designed to assess how well countries are addressing climate change and resource efficiency policies. ...
... Given the potential influence that knowledge intermediaries might have on policy making, the risk of acritical (or superficial) use of intensity indicators is that of presenting the policy maker with a set of imperfect or even biased tools to develop effective actions to address climate change. Additionally, the very specific selection of such intermediaries as unit of analysis contributes to shed light on the role of those knowledge brokers, whereas most of literature is focused on knowledge producers or users neglecting intermediaries (Olejniczak et al., 2016). ...
... In this scenario, an important role is played by think tanks, consulting groups, funders, mass media, analyst or political advisers, educators, gurus, lobbies and other intermediaries that act as bridges between the scientific community, society and the policy maker (Nutley et al., 2007;Sebba, 2013;Olejniczak et al. 2016). Those intermediaries break into in a highly complex environment faced by policy-makers and society; situation requiring advanced answers that only are available at the forefront of science and the intersection of multi-disciplinary knowledge (Scholl, 2015). ...
In this paper we analyse how energy and carbon intensity indicators, that have huge popularity among policy makers, are framed in the discourse of think tanks, consulting groups and other stakeholders. What emerges from the analysis of public documents, reports and policy briefs is that intensity indicators are often framed uncritically and unreflectively. Our analysis highlights three fundamental themes that emerge from public debates around intensity indicator. First, ‘efficiency’ and ‘intensity’ – often framed in terms of productivity – are used equivalently. Second, intensity is perceived as a measure of win-win sustainability in which economic growth can be decoupled from energy consumption without a substantial restructuring of the productive system. Third, the analysis suggests that energy and carbon intensity indicators are attractive for policy makers – especially in the so-called developing world – because they can be used to design political targets (e.g. Paris agreement) without questioning the right to economic growth of powerful emerging economies like the BRICS. The paper shows the strong limitations of intensity indicators and their potential do misguide policymakers.
... This diversity of types and forms of insertion in the government machinery, is observed in contexts as diverse as the Netherlands (Hoppe and Jeliazkova, 2006), the Canadian federal government (Wellstead and Stedman, 2010), the Czech Republic (Veselý, 2014), the Philippines (Saguin et al, 2018), and Brazil (Filgueiras et al, 2020). Other empirical studies (Meltsner, 1976;Colebatch et al, 2010;Kohoutek et al, 2013;Olejniczak et al, 2016) show that the analytical type of policy work is generally associated with other types of work, such as the relational type (for example, intergovernmental negotiations, public consultations, translations and even democratisation). Therefore, there seems to be a mix of different sources of knowledge brought by different public policy actors (Colebatch et al, 2010) which interact with bureaucracy (Cairney, 2019). ...
... Based on the literature (Elgin and Weible, 2013;Wu et al, 2015;Pattyn and Brans, 2015;Olejniczak et al, 2016), we analyse both the effects of accumulated analytical capacities at the bureaucrats' individual level and at the organisational level of government agencies, as shown in Table 4. 11 To measure individual analytical capacity, we use the bureaucrat's level of education and acquired skills, 12 and years of experience in policymaking, seeking to capture the analytical resources coming from formal and tacit knowledge. As Ouitmet et al (2009) argue, prior knowledge and skills would determine the ability of individuals to recognise the value of, acquire, evaluate, and use different sources of knowledge. ...
The evidence-based policy (EBP) movement argues for policy actors, especially public officials, to use scientific evidence on what works to improve public policies. Empirical research in Anglo-Saxon countries shows that public officials do not use scientific knowledge that widely, often preferring other sources of information, such as news media, public opinion and peers. What about countries with low influence of EBP, what informs policy here? Using data from a large-n survey with Brazilian federal bureaucrats we uncover associations between sources of information and factors shaping their preferences, such as policy work and policy capacities. We find that in a civil law system such as the Brazilian administration homemade sources rule: there is a prevalence of use of government sources, especially among bureaucrats performing analytical and oversight tasks, and those in higher positions. Academic sources are associated with higher analytical capacity (of the individual and organisation), but not with any particular policy sector. By investigating an important yet often neglected issue in EBP – the role of different types of information and how they inform policy – this article contributes to the literatures on policy work and policy capacity, especially given its empirical focus on Brazil.
... Prior studies on knowledge brokering have emphasized the role of knowledge brokers as bridges between research and practice or as translators of specialized knowledge or as collaboration facilitators [5,6,7]. A network perspective on knowledge brokers highlights their role in bridging gaps between individuals, groups or clusters, also known as structural holes [8]. ...
... In our study, we consider knowledge brokers as a type of actor who links people in the network. Knowledge brokers are traditionally seen as bridges, translators, and facilitators in knowledge networks [6,7]. The other two affordances, editability and persistence, provide tools for correcting misinformation and, as messages are continuously accessible, support asynchronous communication. ...
... Brokers need to operate because decision makers and researchers are driven by different imperatives and time frames using different languages and practices (Caplan, 1979;Clark & Holmes, 2010;Wehrens, Bekker, & Bal, 2011). Literature on evidence-informed policies points to "knowledge brokering" as a promising strategy for bridging this know-do gap between research and decisionmaking process (Dobbins et al., 2009;Frost et al., 2012;Lomas, 2007;Meyer, 2010;Olejniczak et al., 2016). ...
... Building networks with producers and users of knowledge Figure 2. Concept Map B: Knowledge broker activities for evidence use.Note. Adapted fromOlejniczak, Raimondo, and Kupiec (2016). ...
Knowledge brokering is a promising practice for addressing the challenge of using research evidence, including evaluation findings, in policy implementation. For public policy practitioners, it means playing the role of an intermediary who steers the flow of knowledge between producers (researchers) and users (decision makers). It requires a set of specific skills that can be learnt most effectively by experience. The article explains how to develop knowledge brokering skills through experiential learning in a risk-free environment. It reports on the application of an innovative learning method—a game-based workshop. The article introduces the conceptual framework for designing game-based learning. Then, it demonstrates how this framework was applied in practice of teaching knowledge brokering. In conclusion, article discusses the first lessons of game application in training public policy practitioners. It concludes that game-based workshop is a promising method for learning about knowledge brokering strategies that increase evaluation use.
... Recent literature on evidence use in public policy argues that bringing credible and rigorous evidence to decision makers is not sufficient; the evidence needs to be 'brokered' (Olejniczak et al. 2016). That is because decision makers and researchers are driven by different imperatives and time frames, using different language. ...
... That is because decision makers and researchers are driven by different imperatives and time frames, using different language. Studies point to "knowledge brokering" as an effective way of addressing this challenge (Meyer, 2010;Olejniczak et al. 2016). ...
In 2015, the Regional Studies Association (RSA) celebrated 50 years and one of the keywords of this (hi)story is “impactful”. Having an impact requires to be able to bring research-based knowledge to policymakers moving out of the academic ‘Ivory Tower’ and engaging in policy debate, thus acknowledging the societal role of research. Yet, this requires changing the traditional approach to research, academic conferences, publications and scientific workshops.
This volume aims to tackle the challenge of a research-policy dialogue addressing the case of the Cohesion Policy (CP), the most important EU regional policy and, probably, the most complex policy in the world for economic, social, geographical, cultural, administrative and legal reasons. This challenge is even more relevant in this period when Europe is showing difficulties to fully recover from the financial and economic crisis that started in 2008, and the following political crisis that culminated with ‘Brexit’. In this context, researchers are called to be ‘impactful’, proposing research-based policy lessons that can feed the political debate.
In a period of crisis, research-policy dialogue is needed; however, how to make it happen is not easy. This volume adopts a ‘policy learning’ perspective proposing sixteen different research-based policy lessons to contribute to the debate on policy learning in the case of the EU Cohesion Policy, and beyond.
... First, they are essential for carrying out evaluation studies. Second, they stimulate policy learning via information sharing with relevant experts and stakeholders (Maybin, 2015;Olejniczak et al., 2016Olejniczak et al., , 2017. The internal capacity to use knowledge in the policy process interacts with the external pressure on policymakers to be accountable, especially for "relevant" policy under scrutiny (Raudla et al., 2018;Rimkutė, 2015;Schrefler, 2010). ...
The European Union (EU), especially in the context of Cohesion Policy (CP), has played a crucial role in developing and promoting policy evaluation practices across its Member States. Evaluation systems across the Member States have been established to assess CP investments. Remarkably, the use of evaluation research and its contribution to stimulating policy learning has remained a “black box.” To address this issue, this article aims to develop a novel framework centered around four conditions for evaluation‐based policy learning, namely: (1) policy relevance, (2) resources and organizational settings, (3) quality of evaluation, and (4) evaluation culture. These conditions are retrieved from the existing literature on policy evaluation and applied to the six‐country cases across the EU. The findings suggest how loosening the formal EU evaluation requirements could affect policy learning in the Member States.
... Our understanding of the system of knowledge brokering was based on a systematic literature review and earlier studies on evaluation use (Olejniczak et al., 2016). The game loop followed typical decisions required for the evaluation execution (i.e., decide if you are addressing a knowledge need, choose an appropriate research design, identify the main user of the study, and pick methods for delivering knowledge). ...
... The evaluation tradition has focused on improving ways and forms of communication with potential users of knowledge. This includes studies on understanding the credibility of information in the eyes of the beholder (Miller, 2014), effective data visualisation (Azzam et al, 2013) and broader knowledge brokering strategies (Olejniczak et al, 2016). To this, labs could add their organisational structures that facilitate the utilisation of their products. ...
The chapter explores the potential benefits to public policy of combining traditional evaluative inquiry with insights developed dynamically in policy labs. Twenty leading labs from five continents are critically analysed through a literature review as well as policy and programme evaluation practices, assessing the extent to which the purpose, structures and processes used in policy labs address three challenges: (1) establishing the causality and value of public interventions, (2) explaining mechanisms of change, and (3) utilising research findings in public policy. The chapter concludes that creating synergies between evaluation inquiry and policy labs can improve the design and implementation of public policy and programmes.
... Díky zahraničním zkušenostem (viz např. Høydal 2020; Nutley et al. 2007;Olejniczak a Kupiec 2016;Olejniczak et al. 2020) víme, že potenciálními bariérami v přenosu vědění může být nízká rigoróznost výzkumných designů a malá opora evaluační zprávy v datech, nízká praktická využitelnost závěrů, špatné načasování a konečně příliš akademický jazyk, který je pro rozhodovatele málo srozumitelný. ...
... The evaluation tradition has focused on improving ways and forms of communication with potential users of knowledge. This includes studies on understanding the credibility of information in the eyes of the beholder (Miller, 2014), effective data visualisation (Azzam et al, 2013) and broader knowledge brokering strategies (Olejniczak et al, 2016). To this, labs could add their organisational structures that facilitate the utilisation of their products. ...
... The evaluation tradition has focused on improving ways and forms of communication with potential users of knowledge. This includes studies on understanding the credibility of information in the eyes of the beholder (Miller, 2014), effective data visualisation (Azzam et al, 2013) and broader knowledge brokering strategies (Olejniczak et al, 2016). To this, labs could add their organisational structures that facilitate the utilisation of their products. ...
... The evaluation tradition has focused on improving ways and forms of communication with potential users of knowledge. This includes studies on understanding the credibility of information in the eyes of the beholder (Miller, 2014), effective data visualisation (Azzam et al, 2013) and broader knowledge brokering strategies (Olejniczak et al, 2016). To this, labs could add their organisational structures that facilitate the utilisation of their products. ...
... Evaluation uptake at this level has only recently emerged in our work with WVCTSI, and we believe it has largely been enabled by our developing a context where Level 3 was encouraged and valued. Though the reality of evaluation use is not well known in these types of large complex multi-site studies (Olejniczak et al., 2016), it is reasonable that this be considered an extension of the Utility standards, especially U3, Negotiated Purposes, and U6, Meaningful Processes and Products, as stakeholders move beyond (i.e., transcend) contributing evaluative thinking to requesting and initiating new evaluative efforts. ...
Background: The Program Evaluation Standards (PES) can be considered established criteria for high quality evaluations. We emphasize PES Utility Standards and evaluation capacity building as we strive for meaningful application of our work in the real world.
Purpose: We focused our methodology on understanding how stakeholders discussed utility and how their perceptions related to our evaluation work aligned with PES Utility Standards.
Setting: The West Virginia Clinical Translational Science Institute (WVCTSI) is a statewide multi-institutional entity for which we conduct tracking and evaluation since 2012.
Intervention: Sustained collaborative engagement of evaluation stakeholders with the goal of increasing their utilization of evaluation products and evaluative thinking.
Research Design: Case study.
Data Collection and Analysis: We interviewed five key stakeholders. Themes developed from analysis of PES Utility standard coding of interview data informed document analysis. Interview and document analysis were used to develop themes and illustrative examples, as well as to develop and describe a five-level Evaluation Uptake Scale.
Findings: We describe shifts in initiation, use, and internalization of evaluative thinking by non-evaluation personnel that prompted development and application of an Evaluation Uptake Scale to capture increased evaluation capacity among stakeholders over time. We discuss how focus on PES Utility and evaluation capacity building facilitated such shifts and their implications for maximizing utility of evaluation activity in large complex programmatic evaluations.
Keywords: Program evaluation standards, evaluation utility, evaluation capacity building.
... Po czwarte, wiele przykładów ze świata pokazuje, że wskazane byłoby rozróżnienie w kontekście ram kompetencyjnych różnych ról odgrywanych w procesie ewaluacji. Najbardziej konserwatywne rozwiązanie, opcja minimum, to wydzielenie kompetencji wykonawcy ewaluacji i zamawiającego (pracowników jednostek ewaluacyjnych, których można też nazywać i postrzegać jako brokerów wiedzy) [Olejniczak et al., 2016]), ale jak wskazano w artykule możliwe są też dużo bardziej rozbudowane podejścia. Do rozważenia jest też opcja uznawania kompetencji na poziomie podstawowym i zaawansowanym. ...
... Osiągnięcie efektu w postaci poprawy jakości ewaluacji nie jest możliwe bez eliminacji niekompetentnych ewaluatorów z rynku, ewentualnie skłonienia ich do podniesienia kompetencji.Po czwarte, wiele przykładów ze świata pokazuje, że wskazane byłoby rozróżnienie w kontekście ram kompetencyjnych różnych ról odgrywanych w procesie ewaluacji. Najbardziej konserwatywne rozwiązanie, opcja minimum, to wydzielenie kompetencji wykonawcy ewaluacji i zamawiającego (pracowników jednostek ewalu-acyjnych, których można też nazywać i postrzegać jako brokerów wiedzy)[Olejniczak et al., 2016]), ale jak wskazano w artykule możliwe są też dużo bardziej rozbudowane podejścia. Do rozważenia jest też opcja uznawania kompetencji na poziomie podstawowym i zaawansowanym.Po piąte, zbudowanie poparcia w środowisku dla promowanego rozwiązania będzie łatwiejsze, jeśli uda się przedstawić przekonującą logikę interwencji łączącą mechanizm potwierdzania kwalifikacji z oczekiwanymi efektami. ...
Artykuł ten jest krytyczną refleksją nad inicjatywą Polskiego Towarzystwa Ewaluacyjnego na rzecz stworzenia ram kwalifikacji ewaluatora. Bezpośrednim celem jest wskazanie warunków sukcesu działań na rzecz profesjonalizacji ewaluacji poprzez certyfikację kompetencji i ocena, na ile inicjatywa PTE te warunki spełniała. Podstawą rozważań był systematyczny przegląd literatury dotyczącej certyfikacji, potwierdzania kwalifikacji ewaluatora i ich wpływu na jakość i wykorzystanie ewaluacji. Przegląd doświadczeń i polemik prowadzonych w środowiskach ewaluatorów innych krajów sugeruje, że skuteczność mechanizmu potwierdzania kwalifikacji ewaluatora wymaga m.in.: silnego, wiarygodnego lidera, zrównoważenia wymogów grandparentingu i decertyfikacji, rozróżnienia w kontekście ram kompetencyjnych co najmniej ról – zamawiającego i wykonującego ewaluację, wskazania przekonującego związku między potwierdzaniem kwalifikacji i jakością ewaluacji oraz zrezygnowaniu z pomysłu certyfikacji na rzecz dobrowolnego potwierdzania kwalifikacji.
... Por otro lado, los productores de conocimiento son entendidos como aquellos actores que poseen información relevante para la formulación de las políticas, similar a lo expuesto por Maloney (1994). Por último, los agentes de conocimiento son considerados como aquellos actores gubernamentales o externos que tienen como objetivo transformar la información recogida durante los procesos de desarrollo de política en orientaciones para la toma de decisiones (Olejniczak, 2016). El uso de agentes de conocimiento ha sido sugerido como "una estrategia promisoria para cerrar la brecha del saber-hacer entre la investigación y los procesos de decisión" (Olejniczak, 2017, pp. 2) en los procesos de formulación de políticas públicas. ...
La competitividad es un concepto asociado al desempeño de las empresas y los territorios en los mercados. Sus planteamientos han sido adoptados frecuentemente para la orientación de políticas públicas, si bien en el ámbito académico persisten discusiones sobre su pertinencia conceptual y utilidad práctica. El presente trabajo busca analizar el proceso de formulación de políticas públicas locales de competitividad (PPLC) a la luz de una revisión reflexiva de diversas discusiones teóricas relacionadas, en contraste con un estudio de caso sobre la construcción de una PPLC en la ciudad de Manizales, perteneciente a la región andina de Colombia. Se espera que las observaciones, argumentos y conclusiones que se presentan en este estudio puedan servir como base analítica para la realización de futuros ejercicios de construcción y deliberación política, al igual que como aporte a los amplios campos teóricos de la competitividad y las políticas públicas.
Competitiveness is a concept associated with the performance of companies and territories in the markets. Approaches related with said concept have frequently been adopted in the development of public policies, although there are still ongoing discussions about its conceptual relevance and practical utility in the academic sphere. This paper seeks to analyze the process of formulation of local competitiveness public policies (LCPP) through the contrasting of a review of various related theoretical discussions, in contrast to a case study on the construction of a LCPP in the city of Manizales, in Andean region of Colombia. The observations, arguments and conclusions presented in this study aim at providing analytical basis for the realization of future exercises of policy construction and political deliberation, as well as contributing to the broad theoretical fields of competitiveness and public policy.
... In addition, research on KB is more recent and less developed than research on boundary spanners and gatekeepers (Paraponaris et al., 2015). Looking into the context of public administration, KBs identify and target knowledge, assist knowledge creation, translate and adapt knowledge, negotiate and convince, outreach and network, promote capacity building and use knowledge by integrating internal and external knowledge (Olejniczak et al., 2016). ...
You can reach my book, 'Digital Marketing' via the following link:
https://www.morebooks.shop/store/gb/book/digital-marketing/isbn/978-613-8-96557-2
... Studies which analyze the context in which bureaucrats act through the concept of policy work, that is the work that they perform in carrying out public policies, demonstrate that in practice analytical work generally occurs in association with other forms of work, such as negotiation, communication, translation and even the democratization of state actions (Colebatch, Hoppe & Noordegraaf, 2010;Meltsner, 1976;Olejniczak, Raimondo & Kupiec, 2016). From this perspective, the various types of work involved in carrying out public policy demand a variety of abilities and resources, including those of an informational nature (Howlett & Wellstead, 2011;Wu, Ramesh & Howlett, 2015). ...
Policy-making is a complex process involving stimuli and conditioning. Understanding the sources of information bureaucrats and public organizations use is essential to characterize policy-making and discuss the opportunities and limits of Evidence-Based Public Policies (EBPP) in the Brazilian context. The different sources of information available to bureaucrats in the country are investigated, such as the state's internal and external sources and scientific and experiential sources. This paper presents the results of a survey with 2,180 officials of the direct federal administration conducted at the end of 2019. The findings indicate variation in the types of information bureaucrats mobilized depending on policy area and the type of policy work they perform. O uso de múltiplas fontes de informação nas políticas públicas: um olhar sobre a burocracia federal brasileira A produção de políticas públicas é um processo complexo que envolve um conjunto de estímulos e condicionantes. Compreender o que, de fato, tem informado os burocratas e as organizações públicas é um elemento essencial para caracterizar a produção de políticas públicas, assim como para a discussão das potencialidades e dos limites das Políticas Públicas Baseadas em Evidências (PPBE) no contexto brasileiro. Neste estudo, investigamos as diferentes fontes de informação disponíveis aos burocratas brasileiros; dentre as quais, não somente as científicas, mas também as experienciais, as internas e as externas ao Estado. Esta pesquisa traz resultados de um survey respondido por 2.180 servidores da Administração Federal Direta no final de 2019. Os resultados indicam que há variação na utilização de fontes de informação a serem mobilizadas pelos burocratas de acordo com o tipo de trabalho desempenhado e, também, com a área de política pública na qual atuam. Palavras-chave: políticas públicas baseadas em evidências; burocracia; produção de política pública; trabalho na política pública. El uso de múltiples fuentes de información en las políticas públicas: una mirada a la burocracia federal brasileña La elaboración de políticas públicas es un proceso complejo que implica un conjunto de estímulos y limitaciones. Comprender lo que de hecho ha informado a los burócratas y las organizaciones públicas es un elemento esencial para caracterizar la elaboración de políticas, así como para discutir las potencialidades y los límites de las políticas públicas basadas en la evidencia (PPBE) en el contexto brasileño. Se investigan las diferentes fuentes de información de que disponen los burócratas brasileños, incluyendo las científicas, pero también las experienciales, internas y externas al Estado. Esta investigación trae los resultados de una encuesta contestada por 2.180 funcionarios de la administración federal directa a finales de 2019. Los resultados indican que hay variaciones en la elección de los tipos de información que deben movilizar los burócratas en función del tipo de trabajo realizado, así como del ámbito de la política pública en el que operan. Palabras clave: políticas públicas basadas en la evidencia; burocracia; elaboración de política pública; trabajo en las políticas públicas.
... The acquisition of evaluative knowledge is often performed by distinct bodies named evaluation units. They identify knowledge needs, conduct studies and then feed knowledge to the intended users (Olejniczak et al., 2016). User-units and evaluation units might be from the same organization but when they are not we are dealing with a multiorganizational evaluation system. ...
Evaluation practice is vital for the accountability and learning of administrations implementing complex policies. This article explores the relationships between the structures of the evaluation systems and their functions. The findings are based on a comparative analysis of six national systems executing evaluation of the European Union Cohesion Policy. The study identifies three types of evaluation system structure: centralized with a single evaluation unit, decentralized with a coordinating body and decentralized without a coordinating body. These systems differ in terms of the thematic focus of evaluations and the targeted users. Decentralized systems focus on internal users of knowledge and produce mostly operational studies; their primary function is inward-oriented learning about smooth programme implementation. Centralized systems fulfil a more strategic function, recognizing the external audience and external accountability for effects.
Points for practitioners
Practitioners who design multi-organizational evaluation systems should bear in mind that their structure and functions are interrelated. If both accountability and learning are desired, the evaluation system needs at least a minimum degree of decentralization on the one hand and the presence of an active and independent coordination body on the other.
... This debate has a long and vivid tradition; starting from the early stages of CP (Bachtler & Michie, 1995;Mceldouney, 1991), it has proficiently continued over policy cycles (Bachtler & Wren, 2006) and is continuously being stimulated (Naldini, 2018). The discussion has revolved around several aspects of the CP monitoring and evaluation system, indicatively: (i) the role of evaluation units and monitoring committees (Cartwright & Batory, 2012;Olejniczak, Raimondo, & Kupiec, 2016), (ii) the indicators' misuse (Nigohosyan & Vutsova, 2018) and development (Barca & McCann, 2011;Barca, 2009;Masana & Fernández, 2019;Marchante & Ortega, 2010), (iii) practical lessons in the field (Gaffey, 2013;Lion & Martini, 2006;Perrin, 2011), (iv) the systems' fundamentals (Batterbury, 2006;Perrin, 2011;Saunders, 2011), (v) how it has evolved over time (Gaffey, 2013;Naldini, 2018), and (vi) its formative impact on organisations (Wojtowicz & Kupiec, 2018). ...
European Union’s Cohesion Policy aims to foster development and reduce disparities among regions by redistributing more than one-third of the European budget. Given the policy’s importance and complexity, an elaborated monitoring and evaluation system has been established. While attention has been dedicated to evaluating policy impact, the monitoring of inputs (i.e., allocated financial resources) has been limited to the control of financial dimensions (i.e., funds’ absorption rate). As the implementation process entails a sequence of steps, this research explores whether financial proxies alone are adequate to monitor the policy inputs. To test this hypothesis, a system dynamics model is built. Simulations highlight that the absorption rate captures shocks that might occur during the inputs’ expenditure with significant delay. To that end, we elaborate three novel operative monitoring indicators (i.e., funds’ demand, funds’ offer, procedural efficiency), which may overcome the financial indicators’ mono-dimensionality and time lags’ limitations.
... In addition, research on KB is more recent and less developed than research on boundary spanners and gatekeepers (Paraponaris et al., 2015). Looking into the context of public administration, KBs identify and target knowledge, assist knowledge creation, translate and adapt knowledge, negotiate and convince, outreach and network, promote capacity building and use knowledge by integrating internal and external knowledge (Olejniczak et al., 2016). ...
Purpose
This study aims to understand if and how European digital innovation hubs (DIHs) filling the role of knowledge brokers (KBs) can support the digital transformation (DX) of small and medium-sized enterprises (SMEs) by triggering open innovation (OI) practices.
Design/methodology/approach
After presenting a conceptual model of reference, a survey and a subsequent in-depth interview were conducted to capture evidence from Italian DIHs. These structures were selected for their growing importance, as confirmed by the National Plan for Industry 4.0.
Findings
The findings highlight that Italian DIHs act not only as KBs but also as knowledge sources that give rise to a digital imprinting process that is able to shape the DX of SMEs.
Originality/value
Research on knowledge sharing and OI has mainly focused on large firms. The study covers the gaps identified in the literature by considering the role of KBs in enabling SMEs to embrace DX.
... In 2015, evaluation units in the Polish national government started to search for ways to address the gap between the increasing production of evaluation studies and the limited use of their findings. A set of comparative studies had identified knowledge brokering as a promising practice for addressing the challenge of using evidence to inform policy design and implementation (Olejniczak et al., 2016). Knowledge Brokers are typically government staff who help policy makers and other potential evidence users develop the questions they would like addressed and then ensure that evaluation studies and data answer the knowledge needs. ...
Evaluation professionals need to be nimble and innovative in their approaches in order to be relevant and provide useful evidence to decision-makers, stakeholders, and society in the crowded public policy landscape. In this article, we offer serious games as a method that can be employed by evaluators to address three persisting challenges in current evaluation practice: inclusion of stakeholders, understanding of causal mechanisms, and utilization of evaluation findings. We provide a framework that distinguishes among games along two crucial aspects of evaluation inquiry - its function and the nature of the evaluand. We offer examples of successfully implemented games in each set of the four arenas we delineate: teaching knowables, testing retention, crash-testing mechanisms, and exploring systems. We explain how games can be employed to promote learning about and among stakeholders, and to collect valuable intelligence about the operations of programs and policies.
... First, federal agencies need the focus provided by an agency-level (or bureau-level in large departments) knowledge broker to leverage the expanding new sorts of knowledge provided by administrative data as well as externally produced big data (Olejniczak, Raimondo, and Kupiec 2016). Knowledge brokers are technically skilled professionals who work with agency leadership to build both the motivation and the infrastructure to help leaders and senior managers develop relevant questions about an organization's programs and policies; oversee the collection and generation of (or secure access to, if already collected by other agencies) data to address questions; ensure data are protected, analyzed, and interpreted for users; and provide relevant insights from the evidence to inform strategy formulation as well as management, that is, they orchestrate both the demand for and supply of evidence. ...
Performance management in government is at a crossroads. The advent of big data and advances in technological and analytical tools have provided opportunities to measure and track a wider variety of internal and external indicators on a more timely basis. Public leaders require new vision and capacity to design and manage knowledge‐building systems. This article provides two tools: (1) a more comprehensive open systems performance management framework and (2) a model of leadership needed to orchestrate such systems—knowledge brokers who orchestrate the demand for and supply of evidence. This article recommends that public agencies strategically build evidence to better track measures of the effects of governmental actions on public value and their intended and unintended consequences on the ecosystem; articulate, measure, and test the assumptions built into their operating models; and learn from routine monitoring of the dynamic environment in which their organizations strive to achieve their missions.
... The evaluation tradition has focused on improving ways and forms of communication with potential users of knowledge. This includes studies on understanding the credibility of information in the eyes of the beholder (Miller, 2014), effective data visualisation (Azzam et al, 2013) and broader knowledge brokering strategies (Olejniczak et al, 2016). To this, labs could add their organisational structures that facilitate the utilisation of their products. ...
The article explores the potential benefits to public policy of combining traditional evaluative inquiry with insights developed dynamically in policy labs. Twenty leading labs from five continents are critically analysed through a literature review as well as policy and programme evaluation practices, assessing the extent to which the purpose, structures and processes used in policy labs address three challenges: (1) establishing the causality and value of public interventions, (2) explaining mechanisms of change, and (3) utilising research findings in public policy. The article concludes that creating synergies between evaluation inquiry and policy labs can improve the design and implementation of public policy and programmes.
... Current literature on internal evaluation units reveals little about how this can be achieved in practice. This is affirmed by Olejniczak, Raimondo, and Kupiec (2016 ) who state, "Th e current evaluation literature does not explain well the reality of evaluation use in complex program and institutional settings" (p. 169). ...
While there is an abundance of literature on evaluation use, there has been little discussion regarding internal evaluators’ role in promoting evaluation use. Evaluation can be undervalued if context is not taken into consideration. Evaluation literacy is needed to make evaluation more appropriate, understandable, and accessible, particularly in non-government organizations (NGOs) where there is a growing focus on demonstrable outcomes. Evaluation literacy refers to an individual’s understanding and knowledge of evaluation and is an essential component of embedding evaluation into organizational culture. In recognition of the value of the internal perspective, a small exploratory exercise was undertaken to reveal internal evaluator roles and ways of engaging with colleagues around evaluation. Th e exercise examined a key question: What is the role of evaluation literacy in internal evaluation in the non-government sector? Three Australian auto-narrative examples from internal evaluators highlight evaluation literacy and locate it among the multiplicity of roles required for optimal evaluation uptake. Analysis of the narratives revealed the underlying issues affecting evaluation use in NGOs and the skills needed to motivate and enable others to access, understand, and use evaluation information. Responding to the call for expanded research into internal evaluation from a practice perspective, the authors hope that the findings will stimulate a wider conversation and further advance understanding of evaluation literacy.
... The literature largely describes three main brokering models: 1) single brokers (Robeson et al, 2008;Ward et al, 2012;Urquhart et al, 2011); 2) a cohort of brokers from the same profession deployed across similar organisations (Moore et al, 2017;Rivard et al, 2010;Bruce and O'Callaghan, 2016); or 3) a quasi-/fully independent agency with either an explicit or added remit for brokering (Frost et al, 2012;Olejniczak et al, 2016;van Kammen et al, 2006). However, a fourth, newly-emerging model is of brokers from different disciplines entering into two-way secondments, with each working in isolation, rather than together (Uneke et al, 2017;O'Donoughue-Jenkins and Anstley, 2017). ...
Introduction The Bristol Knowledge Mobilisation (KM) Team was an unusual collective brokering model, consisting of a multi-professional team of four managers and three academics embedded in both local healthcare policymaking (aka commissioning) and academic primary care. They aimed to encourage ‘research-informed commissioning’ and ‘commissioning-informed research’. This paper covers context, structure, processes, advantages, challenges and impact.
Methods
Data sources from brokers included personal logs, reflective essays, exit interviews and a team workshop. These were analysed inductively using constant comparison. To obtain critical distance, three external evaluations were conducted, using interviews, observations and documentation.
Results
Stable, solvent organisations; senior involvement with good inter-professional relationships; secure funding; and networks of engaged allies in host organisations supported the brokers. Essential elements were two-way embedding, ‘buddying up’, team leadership, brokers’ interpersonal skills, and two-year, part-time contracts. By working collectively, the brokers fostered cross-community interactions and modelled collaborative behaviour, drawing on each other’s ‘insider’ knowledge, networks and experience. Challenges included too many taskmasters, unrealistic expectations and work overload. However, team-brokering provided a safe space to be vulnerable, share learning, and build confidence. As host organisations benefitted most from embedded brokers, both communities noted changes in attitude, knowledge, skills and confidence. The team were more successful in fostering ‘commissioning-informed research’ with co-produced research grants than ‘research-informed commissioning’.
Conclusion
Although still difficult, the collective support and comradery of an embedded, two-way, multi-professional team made encouraging interactions, and therefore brokering, easier. A team approach modelled collaborative behaviour and created a critical mass to affect cultural change.
key messages
An embedded two-way collective team makes brokering easier.
It provides a safe space for brokers to be vulnerable, share learning and navigate problems.
Team-brokering models collaborative behaviour and creates a critical mass.
But many brokering challenges still exist.
... Therefore, knowledge dynamics cannot simply be explained in terms of production and use, and new types of actors have been emerging to influence knowledge dynamics. Second, brokers are a specific type of actor with a particular raison d'eˆtre, working in-between actors with different, and sometimes even diverging goals (Meyer and Kearnes, 2013;Olejniczak et al., 2016;Osborne, 2004). Finally, knowledge requires trust, repeated contacts, and includes many tacit, unwritten and informal dimensions, in addition to the codified and explicit information that can be traded between actors. ...
Knowledge brokers have emerged as a new type of actors shaping scientific production, influencing science–policy relationships, and thereby contributing to regional competitiveness. Yet, the spatial dimension of these knowledge brokers has received little attention. Using Framework Programme participations in European cities, we analyse and discuss the location strategy of knowledge brokers, highlighting the importance of co-location with the funding source. Our findings show that knowledge brokers are clustered in Brussels, and not elsewhere, to be closer to the European Commission in order to access strategic, informal and tacit information, while contributing to the construction of transnational R&D networks. While this ‘local buzz’ has positive side effects on the regional innovation system of Brussels; knowledge brokers emerge as a new type of spatially clustered actors shaping the distribution of EU funding for ‘European knowledge pipelines’.
... This is how the described results have been used by Polish government. This holistic approach corresponds well with recent development in literature and practice that indicate transformation of government analysts into "knowledge brokers" (Olejniczak, Raimondo, Kupiec, 2016). ...
This chapter focuses on the analytical capabilities of individuals in central government public organizations. Drawing upon the results of a large study undertaken for the Polish government that aimed to identify and improve the analytical capacity of central government personnel across all policy domains, the chapter provides a definition and a multi-dimensional typology of analysts in government. The typology is accompanied by a practical toolbox for identifying and grouping analysts, as tested in the Polish government.
Publikacja jest praktycznym przewodnikiem po procesie ewaluacji dla administracji publicznej. Opracowanie to jest w dużej mierze aktualizacją poradnika, który został opracowany w ramach Krajowej Jednostki Oceny (obecnie Krajowej Jednostki Ewaluacji) w 2012r. Choć fundamentalne zasady i istota ewaluacji nie zmieniły się w ciągu ostatnich 10 lat, światowa i szczególnie polska praktyka ewaluacji rozwijała się w tym okresie niezwykle dynamicznie. Nowe podejścia, metody, wyzwania, zmieniające się postrzeganie tego, po co ewaluacja jest realizowana i jak ją wykorzystywać w kontekście zarządzania w sektorze publicznym, to tylko niektóre elementy, które uzasadniały aktualizację. Swoistym novum prezentowanego poradnika jest również szeroka gama uzupełniających się perspektyw. Wśród autorów poszczególnych rozdziałów znajdują się przedstawiciele organizacji zamawiających i wykorzystujących ewaluację, wykonawców, jak i akademików zajmujących się ewaluacją naukowo.
La pratique de l’évaluation est essentielle pour la responsabilisation et l’apprentissage des administrations qui mettent en œuvre des politiques complexes. Cet article explore les relations entre les structures des systèmes d’évaluation et leurs fonctions. Les conclusions sont basées sur une analyse comparative de six systèmes nationaux chargés d’évaluer la politique de cohésion de l’Union européenne. L’étude identifie trois types de structures de système d’évaluation : centralisées avec une seule unité d’évaluation, décentralisées avec un organe de coordination et décentralisées sans organe de coordination. Ces systèmes diffèrent en termes d’orientation thématique des évaluations et d’utilisateurs ciblés. Les systèmes décentralisés se concentrent sur les utilisateurs internes des connaissances et produisent principalement des études opérationnelles ; leur fonction principale est l’apprentissage orienté vers l’intérieur pour une mise en œuvre harmonieuse du programme. Les systèmes centralisés remplissent une fonction plus stratégique, et tiennent compte du public externe et de la responsabilité externe des effets. Remarques à l’intention des praticiens Les praticiens qui conçoivent des systèmes d’évaluation multi-organisationnels doivent garder à l’esprit que leur structure et leurs fonctions sont interdépendantes. Si l’on vise à la fois l’imputabilité et l’apprentissage, le système d’évaluation a besoin d’un degré minimum de décentralisation d’une part, et de la présence d’un organe de coordination actif et indépendant d’autre part.
The majority of the projects funded under the European Research Framework Programme (FP) require a collaborative effort in order to tackle complex societal challenges. The underlying assumption is that collaborative networks boost research productivity and the attainment of the goals set out by the European Commission's agenda. In general, the production of scientific knowledge increasingly takes place within networks and consortia with a wide diversity and interlinkages of disciplines, expertise, backgrounds and approaches. This relational dimension has become even more important in the Horizon Europe FP, which has moved from a project-focused to a mission-oriented approach whereby a portfolio of projects is managed in coordination in order to stimulate cross-learning and experimentation. Starting from key concepts and theories of evaluation research, including ten years of experience for the Italian National Research Council (CNR) team as an internal evaluator, we are proposing a new approach, namely a Relations-based Evaluation (RE) approach, that analyses, assesses and builds relationships, both internal and external, with the ultimate aim of supporting collaborative research and innovation (R&I) projects. The framework originated within the context of EU projects and deals with the specific case of consortia funded under the EU FPs. However, it could be easily applied to other types of research consortia and project networks with similar features.
Numerous studies have examined the impact of evaluation quality on the use of evaluation, yet the findings are still inconclusive and only derived from examples in countries with a well-established evaluation culture. This article examines the relationship between evaluation quality and the instrumental use of evaluation in administrations with a low maturity of evaluation culture using Poland as a case study. The research method employs quantitative analysis of datasets regarding evaluation quality and recommendation use from the Polish cohesion policy evaluation system.
The use of evaluation findings was measured at two stages: acceptance of recommendations and actual implementation. The results show that the impact of evaluation quality on the use of evaluation is limited and sometimes puzzling. While perceived utility increases the odds of accepting recommendations, the quality of analysis and conclusions reduces it. The actual implementation of recommendations is supported by the robustness of methodology, quality of the analysis and the clarity of the final report. Data reliability and achievement of the study objectives have no significant impact on either the decision to accept or implement recommendations. The problem of symbolic use is discussed as a potential explanation for the observed results.
Public managers require different types of knowledge to run programs successfully. This includes knowledge about the context, operational know-how, knowledge about the effects, and causal mechanisms. This knowledge comes from different sources, and evaluation studies are just one of them.
This article takes the perspective of knowledge users. It explores to what extent evaluation is a useful source of knowledge for public managers of cohesion policy. Findings are based on an extensive study of 116 Polish institutions: surveys with 945 program managers, followed by 78 interviews with key policy actors. The article concludes that: (a) utility of evaluation studies, in comparison to other sources of knowledge, is limited, (b) evaluation reports are used to some extent as a source of knowledge on effects and mechanisms, however, (c) "effects" are shallowly interpreted as smooth money spending, not socio-economic change.
In conclusion this article offers practical ideas on what evaluation practitioners could do to make evaluation more useful for knowledge users in policy implementation.
The paper provides a summary on the role of family services in promoting child well-being,
and then reviews the policy issues at all levels of the family service delivery systems. At
the government level, the paper emphasizes the need to fostering collaboration between
different government bodies, and to ensure adequate funding for early intervention and
preventative services. At service delivery level, the main identified issues include getting a
better integration between delivery organisations, building capacities to adapt evidence
based interventions, sharing tools to facilitate service implementation, training practitioners with the necessary skills, ensuring that service delivery fits within the local context, and engaging families in services.
A specific feature of the countries of Central and Eastern Europe is that they lack the tradition of creating public policies using methods similar to those applied in mature democracies. And the consequences of this lack of tradition remain visible today, notwithstanding the fact that communism fell over 25 years ago now. For the potential for programming actions (policy capacity), as well as for implementing policies effectively and evaluating them, remains limited. The author presents the results of one of the few studies to be carried out on public policy in Poland. The aim has been to test a research hypothesis pertaining to the circumstances in which public policy in regard to an emergent problem is formulated. The creation of such a policy was in fact found to depend on a constellation of three factors: a significant threat to stability under a given policy, decision-makers who think that legislative tools for action are enough to achieve the desired result, and an emergent dominant stakeholder who achieves a decisive influence over the decision-making process.
The claim that evidence-based policy produces better outcomes has gained increasing support over the last three decades. Knowledge brokering is seen as a way to achieve improve policy making and governments worldwide are investing significant resources in knowledge brokering initiatives. It is therefore important to understand the range of these activities and to investigate whether and how they facilitate evidence-based policy. This article critically reviews the extant literature on knowledge brokering. It identifies six important limitations: the existence of multiple definitions of knowledge brokering; a lack of theory based empirical analysis; a neglect of knowledge brokering organisations; insufficient research on knowledge brokering in social policy; limited analysis of impact and effectiveness; and a lack of attention to the role played by politics. The paper proposes an agenda for future research that bridges disciplinary boundaries in order to address these gaps and contribute new insights into the politics of evidence use.
his is a follow-up study to Reid et al (2017) which considered the barriers and facilitators of getting knowledge into policy when using a knowledge brokering approach. The previous study analysed the use of strategies to reduce barriers to the use of evidence in mental health strategy planning in Scotland using outcome frameworks. The main facilitators highlighted were the importance of local champions, cooperation within partnership networks, and national-level support. The barriers were local implementation cultures, local time pressures, perceived complexities of the framework, and timeliness of the framework. The present article details the results of a follow-up qualitative evaluation of the sustainability of the mental health improvement outcomes framework with local planners. There is a dearth of literature which focuses on the sustainability of outcome frameworks and the findings of this study suggest that the barriers highlighted by Reid et al (2017) remain acute issues. However, there are further aspects for learning for knowledge brokers themselves in terms of national and local relations and the wider challenges and opportunities relating to network governance and policy reform agendas.
Pomimo mnogości prowadzonych badań i analiz ich faktyczne wykorzystanie w projektowaniu i wdrażaniu polityki publicznej jest dość ograniczone. Najnowsze badania wskazują, że skuteczną strategią wzmacniania wykorzystania wyników badań w praktyce polityki publicznej jest brokering wiedzy. Artykuł przedstawia użycie innowacji dydaktycznej umożliwiającej nauczanie brokeringu wiedzy poprzez praktykę – za pomocą szkolenia opartego na grze symulacyjnej. Dotychczasowe doświadczenia z zastosowania gry Brokerzy wiedzy do nauczania analityków polityki publicznej z Polski, Stanów Zjednoczonych i Kanady potwierdzają, że gra pomaga w: (1) zrozumieniu roli wyników badań i analiz w polityce publicznej, (2) opanowaniu sześciu kluczowych umiejętności brokera wiedzy oraz (3) zrozumieniu ograniczeń brokera we wpływaniu na procesy decyzyjne. Instytucje administracji publicznej mogą wykorzystać Brokerów wiedzy do praktycznego kształcenia swoich kadr analitycznych i podnoszenia swojego organizacyjnego potencjału do prowadzenia polityki publicznej opartej na dowodach.
This article aims at discussing the values and principles of the so called AGILE management (AGILE methodologies) in the context of public policy making. In particular, an effort is put into answering the question on the possibility of including the AGILE values and principles in the conduct of policy making. Known from the IT sector, AGILE methodologies have been lately drawing more attention of public policy makers in the world. This approach seems to have a big advantage over hitherto applied modes of public operation, as it allows for a quicker adaptation of policies to changes of the environment. At the same time however, it seems to pose much bigger challenges on policy making bodies, as it requires a strong capacity for regulating diversity throughout the policy making processes in real time. *** W artykule prowadzona jest dyskusja nad wartościami i zasadami tzw. " zwinnego " zarządzania (AGILE methodologies) w kontekście polityki publicznej. Następuje próba odpowiedzi na pytanie o możliwość włączenia tych wartości i zasad do procesu realizacji polityki publicznej. Znane z sektora informatycznego metody przyciągają coraz większą uwagę praktyków zaangażowanych w prowadzenie polityki publicznej na świecie. Wydaje się, że podejście to ma dużą przewagę nad dotychczas stosowanymi metodami prowa-dzenia polityki publicznej, ponieważ pozwala na o wiele szybsze dostosowanie polityki do pojawiających się zmian otoczenia. Jednocześnie jednak, od instytucji prowadzących politykę publiczną w podejściu tym wymaga się na bieżąco silnej zdolności do regulo-wania zróżnicowania zachodzącego w trakcie realizacji polityki. Słowa kluczowe: rządzenie, zwinne rządzenie, inkrementalizm 006_SzPP 3_2016.indb 41
Public policies need research results in order to effectively address the complex socioeconomic challenges (so-called evidence-based policies). However there is a clear gap between producing scientific expertise and using it in public decision-making. This “know-do” gap is common in all policy areas. Knowledge brokering is a new and promising practice for tackling the challenge of evidence use. It means that selected civil servants play the role of intermediaries who steer the flow of knowledge between its producers (experts and researchers) and users (decision-makers and public managers). Knowledge brokering requires a specific combination of skills that can be learned effectively only by experience. However this is very challenging in the public sector. Experiential learning requires learning from own actions – often own mistakes, while public institutions tend to avoid risk and are naturally concerned with the costs of potential errors. Therefore, a special approach is required to teach civil servants.
This chapter addresses the question of how to develop knowledge brokering skills for civil servants working in analytical units. It reports on the application of a simulation game to teach civil servants through experiential learning in a risk-free environment. The chapter (1) introduces the concept of knowledge brokering, (2) shows how it was translated into a game design and applied in the teaching process of civil servants, and (3) reflects on further improvement. It concludes that serious game simulation is a promising tool for teaching knowledge brokering to public policy practitioners.
This paper suggests a new theoretical approach to studying intermediary institutions, particularly intermediary institutions
at the science–policy nexus. Intermediary institutions that mediate between science and politics have often been approached
from the perspective of ‘boundary organizations’. But this model, incorporating assumptions of principal–agent theory, is
not suitable for capturing case studies (of intermediary institutions) in ambiguous governance contexts. Indeed, there is
a lack of systemic studies of the material implications that the rise of new governance actors has, for example for governing
emerging technologies. Therefore, we need a new theoretical vocabulary to grasp how intermediary institutions emerge and how
they work in practice. This paper addresses this gap by analysing the shortcomings of a nanotechnology observatory project.
Drawing on both actor network theory and political discourse theory, the paper outlines a three-fold framework based on the
concept of logics that shifts the focus of research toward an analysis of context-based case studies of intermediaries.
A criticism of evaluation practice in regional policy in Central and Eastern Europe (CEE) countries is that it stresses its function in terms of accountability and formal reporting on progress, rather than its role in policy or institutional learning. This article aims to test this critique through the use of ‘system thinking’ and focuses on the case of regional policy in Poland, where evaluation practice has developed dynamically since 2000. It contributes to the literature by developing an analytical framework that sets out the role of evaluation in regional policy learning and identifies key factors and mechanisms that determine the development of an effective evaluation system for regional policy.
The analysis draws on all Polish evaluation studies conducted between 1999 and 2010, interviews with civil servants and experts involved in the implementation of cohesion policy programmes, as well as secondary data from earlier studies. The essay identifies key mechanisms and factors that determine the main function of evaluation as a learning tool that produces and utilises knowledge in the decision-making process. These main elements are: growth in funding of the policy; stability of institutions; motivators; and system architecture.
FULL ARTICLE CAN BE DOWNLOADED from this link: http://www.tandfonline.com/eprint/NKDh7jesfiybdDrdqY9c/full
Increasingly, in the world of evaluation, `systems of evaluation' have been developed.This article outlines four criteria that help characterize such systems. One criterion is the existence of a distinctive epistemological perspective; another is that, in order to be labelled a system, evaluation activities are carried out by evaluators within organizational structures and institutions and not only (or largely) by `lonely' or sole-trader evaluators. Permanence is the third criterion and the fourth is that there is a focus on the intended use of results of evaluations. Examples of systems are the performance-monitoring system, the `experimentalist' system and the evaluation-accreditation system. Several problematic aspects of these systems are described, making it relevant to study them. One of these is the danger that evaluation systems breed (new) evaluation systems.Another problem is that these systems may produce largely routinized information relevant for day-to-day practices and single-loop learning processes, but which is of little relevance for fundamental reassessments of policies and programmes.
The implementation of clinical practice guidelines in rehabilitation practice is often troublesome and incomplete. An intervention to enhance the implementation of guidelines is the knowledge transfer program built around the activities of a knowledge broker (KB).This study investigates the use of KBs to implement guideline recommendations for intensive therapy and physical activity for patients post-stroke in 22 stroke units in hospitals and rehabilitation centers in The Netherlands.
This study includes a quantitative evaluation with a non controlled pre-post intervention design and a mixed methods process evaluation. From each stroke unit, an enterprising nurse and therapist will be recruited and trained as KB. The KB will work for one year on the implementation of the guideline recommendations in their team. To evaluate the effectiveness of the KB, a questionnaires will be administered to patients, health professionals and KBs at baseline (T0) and after one year (T1). Furthermore, semi structured interviews with 5 KBs will be performed at T1.The primary outcome of this implementation project is the support health professionals give patients to exercise and be physically active, as reported by patients and health professionals themselves. The support immediately after the intervention is compared with the support at the start of the intervention.Additionally we will explore the influence of socio-demographic characteristics of health professionals and determinants identified in the Theory of Planned Behavior (intention, attitude, subjective norm and perceived behavioral control) on the change of supportive behavior of health professionals. Finally, KBs will be complete a questionnaire on their own psychological and social demographic characteristics and on organizational conditions needed for health-care improvement such as time, workforce, sponsoring and support from management.
With this study we will gain insight in when and why knowledge brokers seem to be effective. Also we will identify determinants that predict which health professionals are susceptible to change their behavior. This study will provide guidance how to implement guidelines and will help to improve stroke rehabilitation services.
In this paper we address the way boundary organizations can accommodate tensions in the science – politics interface. Literature on boundary organizations suggests that this type of organization can provide stability in science – politics interaction, but how these organizations function over a longer period of time is not a point of theoretical or empirical attention. We study a boundary organization, the Netherlands Environmental Assessment Agency (MNP), by analyzing the ideas that guided the foundation of the MNP and by analyzing two cases in which the MNP advised Dutch policy makers. In both cases the MNP had to adjust its boundary orientation because of changes in its institutional context. These findings show that the dynamics involved in boundary organizations should be included in academic research. We conclude by discussing two conceptual frameworks that may help to capture these dynamics: the notion of ‘learning organizations’ and a typology of roles of experts in politics. Keywords: science–politics, boundary organizations, contextual dynamics, Netherlands Environmental Assessment Agency (MNP)
This paper reviews empirical research on the use of evaluation from 1986 to 2005 using Cousins and Leithwood’s 1986 framework for categorizing empirical studies of evaluation use conducted since that time. The literature review located 41 empirical studies of evaluation use conducted between 1986 and 2005 that met minimum quality standards. The Cousins and Leithwood framework allowed a comparison over time. After initially grouping these studies according to Cousins and Leithwood’s two categories and twelve characteristics, one additional category and one new characteristic were added to their framework. The new category is stakeholder involvement, and the new characteristic is evaluator competence (under the category of evaluation implementation). Findings point to the importance of stakeholder involvement in facilitating evaluation use and suggest that engagement, interaction, and communication between evaluation clients and evaluators is critical to the meaningful use of evaluations.
This paper presents a model of innovation, knowledge brokering, that explains how some organizations are able to routinely innovate by recombining their past knowledge in new ways. While existing theories of organizational learning and innovation are useful, the links between them are crucial for understanding how existing knowledge becomes the raw materials from which individuals in organizations construct innovative solutions. This model develops these links by grounding processes of learning and innovation in the larger social context within which they occur. Using a microsociological perspective, this article draws together research spanning levels of analysis to explain innovation as the dissembling and reassembling of extant ideas, artifacts, and people. Previous research has suggested that firms spanning multiple domains may innovate by moving ideas from where they are known to where they are not, in the process creating new combinations of existing ideas. This paper more fully develops this process by linking the cognitive, social, and structural activities it comprises. Knowledge brokering involves exploiting the preconditions for innovation that reside within the larger social structure by bridging multiple domains, learning about the resources within those domains, linking that knowledge to new situations, and finally building new networks around the innovations that emerge from the process. This article also considers the origins of knowledge brokers as firms committed to this innovation strategy, the structural and cultural supports for the knowledge brokering process, and several obstacles to the process that these firms experience. Finally, I discuss the implications of this model for further research on innovation and learning, and the implications for other organizations seeking to establish their own capabilities for brokering knowledge.
The importance of using research evidence in decision making at the policy level has been increasingly recognized. However, knowledge brokering to engage researchers and policymakers in government and non-government organizations is challenging. This paper describes and evaluates the knowledge exchange processes employed by the Translational Research on Obesity Prevention in Communities (TROPIC) project that was conducted from July 2009 to April 2012 in Fiji. TROPIC aimed to enhance: the evidence-informed decision making skills of policy developers; and awareness and utilization of local and other obesity-related evidence to develop policies that could potentially improve the nation’s food and physical activity environments. The specific research question was: Can a knowledge brokering approach advance evidence-informed policy development to improve eating and physical activity environments in Fiji.
<br /
The past decade has witnessed widespread interest in the development of policy and practice that is better informed by evidence. Enthusiasm has, however, been tempered by recognition of the difficulties of devising effective strategies to ensure that evidence is integrated into policy and utilized in practice. There is already a rich but diverse and widely dispersed literature that can be drawn upon to inform such strategies. This article offers a guide to this literature by focusing on six main interrelated concerns: (1) the types of knowledge relevant to understanding research utilization/evidence-based practice (RU/EBP) implementation; (2) the ways in which research knowledge is utilized; (3) models of the process of utilization; (4) the conceptual frameworks that enable us to understand the process of RU/EBP implementation; (5) the main ways of intervening to increase evidence uptake and the effectiveness of these; (6) different ways of conceptualizing what RU/EBP means in practice.
Investigators of the influence of evaluations on policy decisions have noted three main routes to influence: instrumental, conceptual, and political/symbolic. This study, an inquiry into the effect of evaluations of the Drug Abuse Resistance Education (D.A.R.E.) program, found a fourth main way that evaluations exert an influence: imposed use. The Safe and Drug Free Schools office of the U.S. Department of Education obliged districts to select a program that met its “Principles of Effectiveness,” which most districts construed to mean that the program had to be on the department’s approved list. Because results of D.A.R.E. evaluations repeatedly showed that D.A.R.E.’s effectiveness on knowledge and attitudes was neither sustained nor led to lower use of drugs, D.A.R.E. did not make the “lists.” Hence, many districts dropped or scaled back D.A.R.E. This kind of imposed use is likely to become more common when government agencies make greater demands for accountability.
The strategic use of science in regional policy-making forums often assumes collaborative interactions between stakeholders. However, other types of stakeholder interactions are possible. This paper uses the ecology of games to frame an investigation into stakeholder participation in the policy networks for regional climate change planning for South East Queensland, Australia. We tracked organisational participation in policy forums between 2008 and 2012. We then used a novel bipartite network theoretical approach to identify participation by different types of organisations across shared multiple forums, which we argue prefaces: cooperation, collaboration, support or advocacy. Network analysis was then combined with semi-structured interviews to access how scientific information was utilised across the regional network. Our results suggest that stakeholder interactions were predominately used to advocate for organisational agendas. Advocacy artificially narrows the scope of possible policy options and represents a biased, selective use of information. While advocacy is an important part of policy process, as a counter balance, explicit efforts are needed to recurrently expand the scope of policy options.
The process of knowledge brokering in the agricultural sector, where it is generally called agricultural extension, has been studied since the 1950s. While agricultural extension initially employed research push models, it gradually moved towards research pull and collaborative research models. The current agricultural innovation systems perspective goes beyond seeing research as the main input to change and innovation, and recognises that innovation emerges from the complex interactions among multiple actors and is about fostering combined technical, social and institutional change. As a result of adopting this
innovation systems perspective, extension is refocusing to go beyond enhancing research uptake, and engaging in systemic facilitation or what has been called ‘innovation brokering’. Innovation brokering is about performing several linkage building and facilitation activities in innovation systems, creating an enabling context for effective policy formulation and implementation, development and innovation. Conclusions are that an innovation systems perspective also has relevance for sectors other than agriculture, which implies that in these sectors knowledge brokering as enhancing research uptake and use should be complemented with broader innovation brokering activities.
Purpose
This study aims to conceptualise the role of knowledge transfer offices (KTOs) as knowledge brokers (KBs) and identify which factors are most significantly related with their performance for supporting public‐private research organizations (PROs), testing the authors' hypothesis for the Spanish case.
Design/methodology/approach
An empirical analysis is conducted based on data from RedOTRI 2008 annual report about 63 Spanish KTOs. A multiple lineal regression model is carried on each of the selected variables representative of KTOs' performance (number of priority patents, revenues from industry collaboration and number of spin‐offs) in order to establish possible relationships with some factors related to the knowledge process that characterize KTOs' activity.
Findings
A theoretical framework conceptualizing the KTOs' role as knowledge brokers is suggested. Factors positively influencing KTOs' performance are PRO's total annual expenses, the type of PRO, the KTO age, the existence of a science park, the explicit regulation of intellectual property rights, the number of specialized full‐time staff of the KTO and the availability of a patent stock.
Practical implications
The practical implication is the identification of those critical factors for the day‐to‐day operation of Spanish KTOs in their different ways of transferring knowledge, drawing managerial and organisational practices that may improve their performance.
Originality/value
This paper provides two original contributions for literature on knowledge transfer: a theoretical framework for the conceptualisation of KTOs as KBs, and the categorisation and further analysis of factors closely related to the performance of KTOs. A set of managerial implications for a better improvement of such institutions is presented.
Despite some recognition of the role of destination marketing organisations (DMOs) in crisis management, limited attention has focused on the role of DMOs in crisis events, and in particular their role in managing knowledge across diverse stakeholder groups and domains. This theoretical paper attempts to address this deficiency by synthesising knowledge management and tourism crisis management literature, to outline the potential role of DMOs in managing knowledge across boundaries during crises. Carlile's [(2004). Transferring, translating, and transforming: An integrative framework for managing knowledge across boundaries. Organization Science, 15(5), 555–568] work on boundary spanning is used to consider potential organisational and management issues for DMOs dealing with crisis events and how they should be managed. This paper argues that because of the role and nature of DMOs, they should play an important role as knowledge spanners/brokers to transfer, translate and transform knowledge to stakeholders. The paper concludes with future research avenues related to knowledge management, DMOs and crises.
The theme of this book is about using evaluation to best effect. It draws on the experience and rich examples of internal evaluation units to share information about the different measures they use to improve the use of evaluations. The authors argue that while a great deal of advice about evaluation utilisation has already been written, it is generally the work of "outsiders". There is a real lack of information coming from the experienced "insiders" who commission, manage or even carry out evaluations in their organisations. The book offers a wealth of practical knowledge which will help readers ensure that evaluation does not just sit on the shelf, but becomes a valuable resource for adding value to their organisations.
"Use of evaluation" used to mean the use of results for making program decisions. Now we are aware of its larger dimensions. Many aspects of evaluation can be used-ideas and insights, signals that are sent by the very fact of evaluating, the processes of learning through cooperation with evaluation, the choice of outcome indicators that herald success, even the study design. "Use" encompasses a broad array of effects by multiple classes of users. The paper highlights the importance of considering further potential collective users-the program organization, client groups, and civil society.
This book presents a solid, research-based conceptual framework that demystifies organizational learning and bridges the gap between theory and practice. Using an integrative approach, authors Raanan Lipshitz, Victor Friedman and Micha Popper provide practitioners and researchers with tools for understanding organizational learning under real-world conditions.
Introduction Public service organisations are preoccupied with understanding how good performance can be achieved: what matters is what works. But delivering high-quality services requires a far wider array of evidence than just that on effectiveness – we need, for example, knowledge about the scale, source and structuring of social problems; practical ‘know-how’ to support effective programme implementation in local contexts; and insights into the relationships between values and policy directions. Research can make an important contribution to the development of public services and policy programmes, and it can enrich debates about the nature of social problems and what works in addressing them. However, such positive research impacts are far from routine, and the impact of research is not always positive. Negative impacts may, for example, arise in situations where tentative or highly specific findings are seized upon too readily or applied too widely. Despite this, the overzealous use of research is not normally considered to be the main problem. Quite the opposite; researchers and others are often disappointed that clear findings are overlooked or ignored when decisions are made about the direction and delivery of services. This view is supported by many studies that have found that practice often lags behind the best available evidence about what works and that it may remain out of step for quite some time.
This paper explores the increasingly prominent role of research brokering organizations (RBOs) in strengthening connections between education research, policy and practice across Canada. This paper is organized in three sections. First, it provides a literature review of research mediation-exploring terminology, models and empirical work (albeit sparse) across health, business, education sectors. The second section provides three conceptual contributions to the field: RBOs' roles in knowledge mobilization occurring in the white space of broader public service systems, a typology of Canadian RBOs that exist in education using four broad categories: governmental, not-for-profit, for-profit and membership RBOs, and a knowledge brokering framework along seven dimensions: mission, resources, staff roles, political affiliation, autonomy, message and linkages. The third section provides empirical data about the frequency and types of RBOs that exist across Canada.
This article investigates the European Union’s evaluation system and its conduciveness to evaluation use. Taking the European Commission’s LIFE programme as its case, the article makes an empirical contribution to an emerging focus in the literature on the importance of organization and institutions when analyzing evaluation use. By focusing on the European Union’s evaluation system the article finds that evaluation use mainly takes place in the European Commission and less so in the European Parliament and the European Council. The main explanatory factors enabling evaluation use relate to the system’s formalization of evaluation implementation and use; these factors ensure evaluation quality, timeliness and capacity in the Commission. At the same time, however, the system’s formalization also impedes evaluation use, reducing the direct influence of evaluations on policy-making and effectively ‘de-politicizing’ programme evaluations and largely limiting their use to the level of programme management.
This article is about evaluation use. It focuses on the well-known paradox that evaluation is undertaken to improve policy, but in fact rarely does so. Additionally, the article also finds that justificatory uses of evaluation do not fit with evaluation's objective of policy improvement and social betterment. The article explains why the paradox exists and suggests applying organizational institutional theory to explain evaluation use. The key argument is that in order to explain all types of evaluation uses, including non-use and justificatory uses, the focus needs to be on the evaluating organization and its conditioning factors, rather than the evaluation itself.
Our study examines brokering of situated knowledge within an organizational context, characterized by professional hierarchy. We examine how professional affiliation and associated power differentials impact upon knowledge brokering at the individual and group levels within an organization. Our empirical case, which combines social network analysis and qualitative fieldwork, is set in healthcare with a focus upon integration of management, psychosocial and clinical component knowledge domains deemed necessary for treatment of a long-term condition. Our study shows that peer-to-peer knowledge brokering, which is framed by professional hierarchy, remains pervasive with respect to medical knowledge brokering. However, social structures might be mediated through developing architectural knowledge, reflected in both formal and informal organizational routines and schema, which engenders community tendencies that transcend professional hierarchy, so that knowledge brokering is more widely distributed to benefit patients.
This paper explores some of the difficulties in accounting for the value of knowledge intermediation, focusing on two types of knowledge inter-mediators: science shops and knowledge brokers. We analyse the way in which actors themselves 'situate', and thus value, their work and practices. Drawing upon theorisations by Antoine Hennion and Bruno Latour, who have distinguished between intermediaries and mediation, we show that actors mobilise two non-mutually exclusive, coexisting repertoires: a 'two-world' repertoire and a repertoire of 'exploration'. In the two-world repertoire, knowledge inter-mediators are seen as actors placed between two (pre-existing) worlds: as a result, their task becomes connecting these two worlds. The repertoire of exploration refers to the fact that inter-mediators evolve in uncertain worlds, their role consisting of exploring and performing new worlds including their own place and role within these. We argue that the tensions between these two repertoires are a potential resource for theory and practice.
Significant progress has been made in recent years to improve the quality of the evaluation of international aid. Increasingly, this includes an interest in improving the way evaluations are used to improve policies and programmes. However, the prevalence of symbolic use – a phenomenon
that is often mentioned but rarely studied – reflects an uncomfortable gap that continues to emerge between evaluation practice and rhetoric. Drawing on literature from several contexts, this paper argues that efforts to improve evaluation use in this complex sector must be, post-Busan,
more firmly rooted in evidence describing how its principals and agents actually behave.
This special section analyses the practices and the performativity of intermediaries. Rather than conceiving intermediaries as agents that passively transfer knowledge and objects between the worlds of science, policy and the economy, the focus is on how they mobilise, reframe and structure expertise and policy imperatives. The papers demonstrate that intermediaries come to: collectively explore new worlds and ventures; perform, define and constitute new scientific fields; and actively constitute logics such as scientification or forecasting in the development of techno-logical regulation. In this way, the papers that comprise this special section contribute to a performative understanding of the practices engaged in intermediation that extend and challenge documentary and ideographic modes of analysis that dominate current scholarhip.
Purpose
– The strategic management literature lacks a comprehensive explanation as to why seemingly similar business models in the same industry perform differently. This paper strives to explain this phenomenon.
Design/methodology/approach
– The model is conceptualized and accompanied by a case study on the airline industry to explain knowledge brokerage that creates value from the effective utilization of knowledge resources acquired from intra‐ and inter‐firm environments.
Findings
– The model explains a cyclical view of business model flexibility in which the knowledge‐based resource accumulation of the business model is spread across the intra‐ and inter‐firm environments. Knowledge brokerage strategies from the inter‐ and intra‐firm environments result in improved performance of the business model. The flexibility that the business model acquires is determined by how efficiently resource accumulation is aligned with its external environment.
Originality/value
– The paper effectively integrates the concepts of knowledge brokerage and business models from a resource accumulation‐based view and simultaneously arrives at the performance heterogeneity of seemingly similar business models within the same industry. It has performance implications for firms that start out without any distinct resources of their own, or that use an imitated business model, to attain better performance through business model evolution aligned with successful knowledge brokerage strategies. It adds to the resource accumulation literature by explaining how resources can be effectively acquired to create value.
Purpose
– This study aims to use organizational identification, organizational culture and safety culture as the intervening variables between safety mission statements and safety behavior to survey and model the process of brokering knowledge from the top strategy makers to the workers on assignments. A pilot empirical research initiative was launched to determine the linkage between safety missions and safety behavior in the airline industry.
Design/methodology/approach
– First, descriptive statistics and independent‐sample t‐tests were used to evaluate the mean, standard deviation (SD) and the relationship between the safety mission statement, organizational identification, organizational culture, safety culture and safety behavior and the respondent's years of working for the sampled airline. In addition, Pearson correlation analysis was used to test the relationships between the five factors. Finally, path analysis was used to examine the direct effects and indirect effects between individual factors.
Findings
– The findings show that organizational identification and organizational culture are two important intervening variables between the safety mission statement (knowledge brokering) and safety behavior. The results of Pearson correlation analysis indicate that five factors are highly correlated with each other, especially the safety mission statement and organizational identification; organizational culture and safety culture; organizational identification and organizational culture; and safety culture and safety behavior. Besides, the safety mission statement has a negative direct effect on the pilot's safety behavior.
Originality/value
– To understand the process of using the safety mission statement to change safety behavior can significantly increase the benefits of brokering knowledge. Furthermore, this study has provided an overview of the flight crews' perceptions on how safety is managed in the aviation industry. Based on the findings, it is concluded that organizations in the various sectors of the airline could do better in managing safety, and in brokering the safety knowledge in the industry. At another level, this study provides an opportunity to explore the validity of causal relationships among different areas.
The Gulf of California hosts astounding biodiversity that supports numerous economic activities in the region. These activities, and emerging threats, are placing pressure on the region's ecosystems. Government and civil society are working to address threats through several conservation and management mechanisms. Nevertheless, the use and incorporation of scientific information—a key component for creating effective and durable management—is still deficient. This article presents the concept of science integration and discusses the findings of a study that assesses the regional landscape, existing institutional arrangements, and capacity for using science to inform policy and management decisions. It also explores the current use of science within fisheries policy and management and the capacity of the National Network of Information and Research of Fisheries and Aquaculture (RENIIPA) and the State Fisheries and Aquaculture Councils, two mechanisms in the region. Finally, it shares lessons learned and offers recommendations on how the region can strengthen science-based decision-making. Results indicate that while there are some actors in the Gulf of California producing relevant science, there is varying capacity of intermediary groups connecting producers with users of science, or mechanisms in place to ensure that science is being utilized in decision-making processes. Moreover, despite having a well-developed landscape of producers and intermediaries and mechanisms in place for fisheries management in the region, effective science integration is not occurring.
Purpose:
To conduct a systematic review of the literature related to the use of knowledge brokers within paediatric rehabilitation, and specifically to determine (1) how knowledge brokers are defined and used in paediatric rehabilitation and (2) whether knowledge brokers in paediatric rehabilitation have demonstrably improved the performance of health care providers or organizations.
Methods:
The MEDLINE, CINAHL, EMBASE, and AMED databases were systematically searched to identify studies relating to knowledge brokers or knowledge brokering within paediatric rehabilitation, with no restriction on the study design or primary aim. Following review of titles and abstracts, those studies identified as potentially relevant were assessed based on the inclusion criteria that they: (1) examined some aspect of knowledge brokers/brokering in paediatric rehabilitation; (2) included sufficient descriptive detail on how knowledge brokers/brokering were used; and(3) were peer-reviewed and published in English.
Results:
Of 1513 articles retrieved, 4 met the inclusion criteria, 3 of which referenced the same knowledge broker initiative. Two papers used mixed methods, one qualitative methodology, and one case presentation. Because of the different methods used in the included studies, the findings are presented in a narrative summary.
Conclusions:
This study provides an overview of the limited understanding of knowledge brokers within paediatric rehabilitation. Knowledge broker initiatives introduced within paediatric rehabilitation have been anchored in different theoretical frameworks, and no conclusions can be drawn as to the optimum combination of knowledge brokering activities and methods, nor about optimal duration, for sustained results.
It has been suggested that in order to address current complex problems, scientists have to take on new roles that link the production and use of knowledge. However, what exactly these new roles mean is often not clearly identified. This paper contributes to this by discussing relevant literature related to different roles of science in society and by presenting the results of a study on the perspectives and activities of knowledge brokering. We identify three knowledge brokering repertoires: supplying, bridging and facilitating, which differ with respect to whether they maintain clear boundaries between knowledge production and use or set out to blur them. Based on our findings, we conclude that new roles of science not only afford, but may even strengthen traditional disciplinary scientific ideals. Thus, it is crucial to look beyond the intentions and ideals of new roles of science to how they are taken up in practice.
But what experience and history teach us is this, that nations and governments have never learned anything from history. (G. W. F. Hegel, 1837, cited in Feyerabend 1978)
Contemporary literature on policy evaluation challenges the 'traditional', rational-objectivist model of policy evaluation. Instead, an argumentative- subjectivist approach is forwarded, conceiving of policy-making as an ongoing dialogue, in which both governmental and societal actors contest their views on policy issues by exchanging arguments. It is argued that, through constructive argumentation, policy actors, networks or advocacy coalitions may arrive at moral judgements on policy issues and, hopefully, at 'better' policies and ways of delivering those policies. The paradigm shift from the rational-objectivist model to the argumentative-subjectivist approach has implications for the way policy evaluation is studied as a means of institutionalized policy-oriented learning; the searching process of improving and perfecting public policy and its underlying normative assumptions through the detection and correction of perceived imperfections. Policy- oriented learning can be studied from a cybernetic control, a cognitive development and a social-constructivist perspective. Within policy-oriented argumentation and negotiation—the discursive processes that constitute the roots of policy-oriented learning—there may (still) be a need for methodologically sound assessments of the cost-effectiveness or 'quality' of policy measures. Accepting this premise, we advance an integrated learning strategy, in which the 'traditional', rational-objectivist role of evaluating institutions may well serve to complement more argumentative-oriented perspectives.
Urbanisation is recognised as a major pressure on coastal biodiversity. Increasing risks of flooding and erosion associated with future climate change indicate that new hard infrastructure will have to continue to be built – and existing structures upgraded – in areas of high social and economic value. Ecological enhancement involves undertaking management interventions at the design stage to improve the ecological potential of these structures, or to improve the ecological value of existing structures. Whilst scientific research into ecological enhancement methods and designs is growing, there has been limited discussion of the non-science drivers and mechanisms by which ecological enhancements can be successfully implemented in coastal infrastructure projects.We explore the science–policy–practice interfaces of the ecological enhancement of hard coastal structures from three perspectives. First, we outline the growing number of European and UK policies and legislative instruments that are increasing the need to consider ecological enhancement in coastal developments. These serve as a facilitative tool for making enhancement projects happen, constituting a significant ‘policy push’ for research and application in this area. Second, we examine the role of people in influencing the uptake of ecological enhancements. The critical role of ‘knowledge brokers’ and the need for effective and sustained collaboration between a range of groups and individuals to get research approved operational trials off the ground is discussed. Third, we examine where in the typical planning, design and build process current enhancement projects have been embedded, serving to illustrate how the science can be used in practice.
Brokering knowledge is a fast growing innovative and important research theme in the project management environment. The current paper analyzes and classifies the research on knowledge brokering and knowledge transfer in project management published in the leading journals over the last decade. An array of classifications was implemented on the articles in order to identify patterns and themes of interest. The findings indicate that this field of research is rapidly developing, mainly in the engineering and information technology sectors. It was revealed that many studies are based on qualitative research methods and that research is focused on understanding knowledge transfer between individuals rather than groups. Contemporary issues of study include developing tools for knowledge transferring, understanding the unique characteristics of knowledge transfer in global projects, and discussing the social aspect of brokering knowledge. These subjects are probably expected to gain research attention in the following years.
Constructive and collaborative planning theory has exposed the perceived limitations of public participation in impact assessment. At strategic levels of assessment the established norm can be misleading and practice is illusive. For example, debates on SEA effectiveness recognize insufficiencies, but are often based on questionable premises. The authors of this paper argue that public participation in strategic assessment requires new forms of information and engagement, consistent with the complexity of the issues at these levels and that strategic assessments can act as knowledge brokerage instruments with the potential to generate more participative environments and attitudes. The paper explores barriers and limitations, as well as the role of knowledge brokerage in stimulating the engagement of the public, through learning-oriented processes and responsibility sharing in more participative models of governance. The paper concludes with a discussion on building and inter-change of knowledge, towards creative solutions to identified problems, stimulating learning processes, largely beyond simple information transfer mechanisms through consultative processes. The paper argues fundamentally for the need to conceive strategic assessments as learning platforms and design knowledge brokerage opportunities explicitly as a means to enhance learning processes and power sharing in IA.
In the context of climate change, resource limitations and other drivers, there is growing international acceptance that conventional technocratic approaches to planning urban water systems are inadequate to deliver the services society requires. Instead, scholars and practitioners are calling for a shift to an adaptive approach that increases a system's sustainability and resilience. This shift is significant, requiring transitions in the way urban water systems are planned, designed and managed. However, there is limited understanding of how strategic initiatives can be deliberately managed and coordinated to reform mainstream policy and practice. This paper aims to develop a strategic program for this purpose. It draws on strategy literature to develop a scope and logic for a general program that can address challenges for long-term urban infrastructure management related to path-dependencies, the direction of transformative change, system complexity and future uncertainty. The content of a normative transition scenario, developed in participatory workshops by water practitioners in Melbourne, is then presented, focusing on the transition to a “water sensitive city”. The scenario comprises a problem definition, vision and strategies, which provide lessons for contextualizing the strategic program for the specific purpose of enabling transformative change in urban water systems. These lessons are synthesized in strategy goals and planning processes that form the design base of a strategic program. With tailoring for local contexts, the strategic program can provide operational guidance for planners, designers and decision-makers in strategically planning and managing initiatives to facilitate sustainability transitions in urban water systems.
Knowledge brokers are people or organizations that move knowledge around and create connections between researchers and their various audiences. This commentary reviews some of the literature on knowledge brokering and lays out some thoughts on how to analyze and theorize this practice. Discussing the invisibility and interstitiality of knowledge brokers, the author argues that social scientists need to analyze more thoroughly their practices, the brokering devices they use, and the benefits and drawbacks of their double peripherality. The author also argues that knowledge brokers do not only move knowledge, but they also produce a new kind of knowledge: brokered knowledge.
Debates about biotechnology continue to be polarized despite its potential to improve the living standards of the poor in Sub‐Saharan Africa. In the backdrop of this polarized scenario, this paper asked, is there a place for brokers in bringing about a productive debate that is pro‐development? The paper argued that if potential intermediaries are analyzed from the perspective of understanding their role and stakeholding in the regulatory change process, this may help breakout the current polarized anti‐ and pro‐biotechnology debates and thereby focus on how to enable productive biotechnology development. Informed by insights from innovation brokering, the functions of brokers in biotechnology regulation are analyzed through the lens of organizations involved in agricultural biotechnology debates in Kenya. The analysis found that policy brokering function attracts varying opportunities and challenges appropriate for informing relevant policy. The paper drew lessons from Kenya's experience to inform a productive policy brokering model for biotechnology regulation.
One important objective of introductory courses in public administration is to sensitize students to the difference between two concepts: substantive rationality and political rationality. Both types of rationality play an important role in policy processes. Yet, although the difference is straightforward in theory, and is addressed and well-illustrated in most standard textbooks on public administration, students seem to have difficulty internalizing it. This article reports on our findings from a role-playing game designed to make students experience the difference between policy making as a process of rational design and policy making as a process of political negotiation. We conducted an experiment involving a large group of students enrolled in a first year, one-semester course, and a control group of students who enrolled in the same course 1 year later. The former were tested four times (start of the course, immediately before and after playing the game, and 3 months later) and the latter two times (at the start of the course and at the exam) for their understanding of how policy making—as-rational-design and policy making—as-political-negotiation differ on seven characteristics. Comparison of test results obtained before and after the role-play indicates a positive learning effect for some characteristics, and a negative learning effect for others.
Solar flares are very complex electromagnetic phenomena. Particles are accelerated to high velocities and a variery of physical processes happen inside and outside flares. These processes can be studied by a large number of techniques from Earth and from space. The aim is to discover the physics behind solar flares.
This goal is complicated because information about the flare mechanism can be obtained mainly in an indirect way by studying the secondary effects. This book provides three stages of the solution of the solar flare problem. Chapter one describes the connection between observational data and theoretical concepts; it is stressed that next to investigating flares, the related non-stationary large-scale phenomena must be studied as well. The second chapter deals with secondary physical processes, in particular the study of high-temperature plasma dynamics during impulsive heating by accelerated particles, heat fluxes and X-ray emission. The last chapter presents a model built on the knowledge of the two previous chapters and it constructs a theory on non-neutral high-temperature turbulent-current reconnecting layers.
Audience: Solar physicists, plasma physicists, high-energy particle physicists.
Purpose
– Team concepts do not necessarily include both of the following important facts: implementing team work leads to shifts in traditional supervisory relations; and these shifts bring about a need to ensure that teams have the requisite skills. The lack of either of them may cause empowerment to get unspecified interpretations and breadth. This paper aims to stress the need of specified empowerment both as a part of skill structure and as a part of the concept of a suitable team.
Design/methodology/approach
– In this paper two well‐known concepts of management and leadership – the idea of Katz and the division constructed by Kotter – are addressed as tools to open the backgrounds of teams. The paper constructs a background for empirical investigations.
Findings
– It is useful to assure the skill structure especially during and after shifts within changes which happen in team organizations. The division into management and leadership tasks, combined with the idea of skills needed at several management levels: offers a fruitful background to address the need of different skills in teams. Also finds that it is argued how unspecified empowerment can establish an obstacle to the success with teams.
Originality/value
– This paper contributes to developing tools for considering explanations for failures of teams.
This paper reviews the literature on organizational learning. Organizational learning is viewed as routine-based, history-dependent, and target-oriented. Organizations are seen as learning by encoding inferences from history into routines that guide behavior. Within this perspective on organizational learning, topics covered include how organizations learn from direct experience, how organizations learn from the experience of others, and how organizations develop conceptual frameworks or paradigms for interpreting that experience. The section on organizational memory discusses how organizations encode, store, and retrieve the lessons of history despite the turnover of personnel and the passage of time. Organizational learning is further complicated by the ecological structure of the simultaneously adapting behavior of other organizations, and by an endogenously changing environment. The final section discusses the limitations as well as the possibilities of organizational learning as a form of intelligence.
No theory or model of organizational learning has widespread acceptance. This paper clarifies the distinction between organizational learning and organizational adaptation and shows that change does not necessarily imply learning. There are different levels of learning, each having a different impact on the strategic management of the firm.