ArticlePDF Available

Building a new tool to evaluate networks and multi-stakeholder governance systems

Authors:
  • Spatial Foresight GmbH

Abstract and Figures

Networks are increasingly used in public policy to tackle ever more complex societal challenges. Multi-stakeholder governance systems have been set up to strengthen regional innovation systems, to fight the consequences of climate change and to tackle social change. However, multiple challenges mean that evaluating these networks and governance systems is complex and requires specific evaluation approaches. Learning from the strengths and weaknesses of existing approaches, this article presents a new tool (GOCAPASS) which enables evaluators to assess governance systems and networks involving multiple stakeholders. It has already been tested in regional innovation policy and governance of EU macro-regional strategies. Practical testing demonstrated its usefulness as a tool for evaluation and policy learning, though some weaknesses still need to be resolved.
Content may be subject to copyright.
A preview of the PDF is not available
... Yet systemic transformation is a complex endeavour requiring multiple stakeholders to act simultaneously. Multi-stakeholder platforms (MSPs) have been promoted as a form of collaborative governance arrangement designed to deliver collective solutions to complex challenges at scale (Breeman et al., 2015;Dentoni et al., 2012;Haarich, 2018;Hermans et al., 2017). However, questions remain as to how effective MSPs are as change agents contributing to sustainability. ...
... Multi-stakeholder platforms (MSPs) offer an alternative solution based on facilitating long-term engagement between actors that cut across traditional boundaries (Hermans et al., 2017). Although there is no single accepted definition of MSPs, in this paper, they are understood to be governance arrangements that work with actor interests across public, private and community boundaries to solve complex and systemic problems which participants believe cannot be solved by acting alone (Breeman et al., 2015;Haarich, 2018;Hermans et al., 2017). MSPs are broader than 'partnerships', although catalyzing multiple partnerships may be part of their modus operandi (Pfisterer & Brouwer, 2020). ...
... In all cases, however, the studies rely on respondents' perceptions to judge whether changes can be attributed to the MSP. While respondents can reasonable assess how MSP activities have led to outputs (Haarich 2018;Provan & Milward, 2001), they may have limited understanding of how these have contributed, or not, to higher level outcomes, given complex dynamics in the agrifood system and multiple factors leading to results. ...
Article
Multi-stakeholder platforms (MSPs) have proved popular as network governance approaches to align business, government, NGOs and other stakeholders in dealing with complex sustainability challenges in the agri-food sector. However, secretariats, members, stakeholders and funders rarely possess good information regarding MSP effectiveness, leaving them in the dark regarding how best to allocate resources and expectations towards different MSPs or MSP activities. This paper explores the feasibility of generating better and more timely evidence of the effectiveness of MSPs as agents of agri-food system transformation. It is based on the findings of action research with two agri-food MSPs: Bonsucro and the Farm to Market Alliance (FtMA), which used a theory-based methodology to understand progress towards intended results. Through critical questioning of their theory of change alongside multiple sources of evidence, the approach generated new insights regarding MSP effectiveness. The findings also offer lessons on three critical challenges in assessing network governance forms like MSPs: causal inference, goal attainment, and the potential for scale. The findings have implications for improving MSPs’ own monitoring in order to adaptively manage and improve their effectiveness, and for generating meaningful evidence of MSP contribution to more sustainable food systems.
... Referencing Mu & Jong (2016), Uyl & Driessen (2015), Stewart (2006) and Haarich (2018), Alkhamis et al. stated, "resolving social problems is challenging because successful policy integration can be achieved only through goal alignment strategies for all stakeholders so all actors will perform as one system (2019, p. 3496)."The UN has acknowledged the urban planning challenges many countries face, leading to inadequate access to public services, infrastructure and technologies. ...
... Under the governance umbrella, different quality management and excellence models have been applied by organizations and governments to evaluate service delivery performance and excellence, but the results remain unsatisfactory while addressing the wicked problems (Bovaired & Loffler, 2002. Public governance evaluation is used in literature and practice to resolve wicked problems and improve complex fields' performance with multiple sectors and stakeholders and high uncertainty, like transit oriented developments, protected areas and environmental fields (Antunes et al. 2009;Mu & Jong 2016;Haarich 2018). ...
Article
This paper responds to the concern of lack of public governance indicators for Accessible and Inclusive Urban Development (AIUD) despite agreements in the literature and in practice to develop field-specific governance indicators to resolve wicked problems. AIUD is known as a multidisciplinary and multi-sectoral field requiring various sectors' involvement. The resources needed to resolve this issue are scattered across diverse sectors. Furthermore, accessibility for and inclusion of disabled and elderly people is considered a global challenge. International organizations like the United Nations (UN), the Inclusion International (II) and the World Bank (WB) have addressed this concern. The UN prioritized this vital issue in its robust Sustainable Development Goals (SDGs) (2015-2030). AIUD significantly impacted social inclusion, human welfare and economic development warrants developing governance indicators. This facilitates decision-making, international performance comparisons, benchmarking, assessing governance quality and adopting corrective measures which improve the service delivery system. This study endeavours to develop a set of governance indicators for AIUD in the UAE through a systematic literature review and the inputs gleaned from a group of experts by administering a questionnaire survey.
... Evaluation methods designed to tackle the complexity of interventions are yet to be widely adopted . This is largely due to the difficulties evaluators encounter in selecting and applying appropriate methods for assessing complex systems (Caffrey & Munro, 2017;Haarich, 2018;Picciotto, 2016). Complexity in evaluation refers to the unpredictable and non-linear interactions between actions and their outcomes (Glouberman & Zimmerman, 2004). ...
Article
The use of complexity-based evaluation methods remains relatively underexplored in the field of evaluation. While increasingly employed to assess complex interventions, Contribution Analysis (CA) continues to suffer from a lack of operationalization. In this article, we propose enhancing the implementation of CA by leveraging Participatory Systems Mapping as a tool to delineate and scrutinize contribution narratives. The research has a dual objective. It aims to experiment with the recently introduced Participatory Systems Mapping method in the field of evaluation while providing a tool for examining contribution narratives. The combination of CA and Participatory Systems Mapping was tested by evaluating a youth employability project. This approach enabled a collaborative and in-depth examination of impact stories and alternative explanations. It proved valuable not only in managing the complexity of the intervention but also in strengthening the implementation of CA.
... Territorial food governance requires collaborative (Siddiki, 2015;Clark, 2019), networked (Lever et al., 2019;Ovaska et al., 2021), or multistakeholder governance arrangements (Haarich, 2018;Alliance of Bioversity et al., 2021) capable of engaging with the various food systems actors in policy-and decision-making. ...
Article
Full-text available
Transition towards sustainable food systems demands a change to integrated or territorial policy approaches and boundary-spanning governance arrangements. Territorial approaches provide an effective framework to address food systems transformation at a scale where it can be tackled with the active participation of all stakeholders. The paper brings the territorial governance approach to the food governance debate and introduces the concept of participatory multi-stakeholder food platforms as arrangements to implement territorial food governance. The paper investigates the implementation of territorial food governance in Tuscany (Italy) at local level and explores the emerging attempt to scaling-up local food governance by developing a regional participatory multi-stakeholder food platform: the Regional Food Roundtable of Tuscany.
... Then the content was used to open-code the potential NG characteristics. For instance Haarich (2018) noted some NG characteristics one of which is "Rules and behavior: Network and governance system performance depends not only on the quantifiable activities of the network but also on underlying conditions and intangible elements that make collaboration possible, such as trust, motivation, common norms and values." As a result, the NG characteristic 'rules and behavior' was obtained. ...
Conference Paper
The typical structure of the healthcare sector involves (specialist) intertwined practices co-occurring in formal or informal networks. These practices must answer to the concerns and needs of all related stakeholders. Multimorbidity and the need to share knowledge for scientific development are among the driving factors for collaboration in healthcare. To establish and keep up a permanent collaborative link, it takes effort and understanding of the network characteristics that must be governed. It is not hard to find practices of Network Governance (NG) in a variety of industries. Still, there is a lack of insight in this subject, including knowledge on how to establish and maintain an effective healthcare network. Consequently, this study's research question is: How is network governance organized in the healthcare sector? A systematic literature study was performed to select 80 NG articles. Based on these publications the characteristics of NG are made explicit. The findings demonstrate that combinations of governance style (relational versus contractual governance) and governance structure (lead versus shared governance) lead to different network dynamics. Furthermore, the results show that in order to comprehend how networks in the healthcare sector emerge and can be regulated, it is vital to understand the current network type. Additionally, it informs us of the governing factors.
... To precisely measure governance is difficult and "there are no silver bullets in measuring governance" (Kaufmann & Kraay, 2007, p. 3). Scholars discuss the importance of the measurement (Buduru & Pal, 2010), the various definitions of 'good governance' (Patton & Director, 2008;Rotberg, 2014), the difficulties that may be encountered when measuring governance (Haarich, 2018;Kaufmann & Kraay, 2007), and approaches to measure governance (Anten, 2009;Ehler, 2003;Rauschmayer et al., 2009). ...
Article
Full-text available
In the past decade, Public Rental Housing (PRH) has become the program of providing affordable rental housing to low- and middle-income households in China. Even though descriptions of the governance results are numerous, the previous studies are not underpinned by a theoretical foundation from a governance perspective, nor have they empirically examined whether PRH governance works on the ground. This explorative and empirical paper aims to fill this gap of an outcome-oriented evaluation of the impacts of governance as perceived by the final user. Central government formulated the objective for PRH governance as maintaining stability in the society. Whether the tenants perceive the goal of social stability as achieved was measured along three governance outcome dimensions: satisfaction with housing quality, satisfaction with housing quantity, and willingness to communicate with the government about PRH governance. Data were collected from questionnaires to PRH-tenants in Chongqing, the most important pilot city of PRH provision in China. The findings show that the perceived governance outcomes were quite mixed as tenants were moderately satisfied with PRH housing quantity, but less satisfied with housing quality, and thought they could relatively easily communicate with local government. In view of these mixed outcomes, to strengthen the effectiveness of PRH governance in the eyes of the tenants, this study suggests local governments: (1) to rethink and redevelop the performance evaluation; (2) to rethink the relations with non-governmental actors and organise a monitoring system that will assist in optimizing housing quality; and (3) to facilitate tenants’ communication with local government.
... This network guidance is generally referred to as 'meta-governance' (Sørensen and Torfing, 2009) and a key role is often attributed to national or regional government authorities that may relinquish some direct authority over implementation while retaining important coordination functions (Marinetto, 2003). However, practical knowledge on what makes 'meta-governance' efficient is fragmentary (Haarich, 2018). Research indicates a positive relationship between the networking capacity and experience of partners in implementation of shared projects and the design and delivery of more complex and high-quality projects. ...
Article
Full-text available
Network-based arrangements are increasingly used for policy implementation. Arguments are made for the potential benefits of this approach, but evidence of their efficiency is inconclusive. Recent methodological innovations, incorporating social network analysis, are building the evidence base. This article contributes to a growing area of study by exploring the relationship between the efficient implementation of European Union cohesion policy projects, and the characteristics of the networks involved in the implementation process. The research combined quantitative analysis of implementation of projects in Scotland in 2007–2013 and semi-structured interviews. The research finds that the involvement of many partners can have a negative impact on implementation. Rather, the strategic position of key actors in the network is important for efficient implementation. The results stress the importance of network governance and the role of key agents as nodes that bridge structural gaps and facilitate exchange of knowledge and resources.
... However, in working with a partner organisation a new dimension will be introduced, reflecting the need for mutual learning between Q Lab and the partner organisation, and between both and wider organisations. The capacity of the partnership to perform well is likely to be observable in four dimensions: decision-making capacity; managerial capacity; networking and cooperation capacity; and system engagement and alignment (Haarich 2018). This requires, in addition to building on this evaluation, a more topic-specific evaluation of the partnership. ...
Preprint
Full-text available
Accessibility and Inclusion are challenging issues for decision-makers due to its impact on equity and quality of life. This research presents a novel governance evaluation model (GEM) for Accessible and Inclusive Urban Development (AIUD) to develop quantitative approach by applying Analytical Network Process (ANP). Experts feedback was sought to first: rank priorities of governance principles (GP’s), second: determine interdependencies of GI’s with respect to GP’s, third: determine interdependencies of indicators with respect to each other and finally scores for public governance. Therefore, the model was used to evaluate public governance in Sharjah Emirate by four agencies representing public sector. AIUD GEM results showed that the most important governance principle at AIUD in the UAE is (Transparency and Accountability). While the highest rated governance indicator is (representation of persons with disabilities and their families). The model implementation in Sharjah, demonstrated organizations agreement on rating (political commitment and legitimacy) as the highest in Sharjah public governance. The final outcome established reliable tool to support decision making in resource allocation, prioritizing intervention actions in addition to determining areas of improvement needed among various stakeholders and sectors. It is this gap or niche in the field of AIUD that this research endeavoured to fill by formulating a bespoke GEM for AIUD in the UAE, with the potential for adaptation in other contexts around the world. The model formulated in this research is a holistic monitoring and evaluation tool that engages stakeholders from various sectors, uses indicators for which data collection is less time- and effort-consuming, and allows for continuous and easy monitoring by decision-makers. This research is a timely response to a trans-disciplinary subject which is a global concern to decision- makers due to its significant impact on sustainable development. Due to the impact of AIUD on social inclusion, human welfare and economic development, the development of GI’s facilitates decision-making through stakeholder’s involvement in order to allow benchmarking and comparison of performance. The model would enable the UAE to remain a role model in the field of AIUD and continue to influence knowledge and practice in this field globally. This research is the first published research that utilized MCDM tools to develop public GEM considering holistic approach in the context of AIUD. Therefore, it is forming solid base for academics and decision-makers to utilize this model to promote public governance performance not only in the UAE but also internationally.
Article
This paper responds to the concern with the lack of public governance indicators for Accessible and Inclusive Urban Developments (AIUDs) despite agreements in the literature and in practice to develop field-specific governance indicators to resolve intricate problems. AIUD is known as a multi-disciplinary and multi-sectorial field requiring various sectors’ involvement. The resources needed to resolve this issue are scattered across diverse sectors. Furthermore, accessibility for and inclusion of disabled and elderly people is considered a global challenge. International organizations like the United Nations (UN), the Inclusion International (II) and the World Bank (WB) have addressed this concern. The UN prioritized this vital issue in its robust Sustainable Development Goals (SDGs) (2015-2030). AIUDs significantly impacted social inclusion, human welfare and economic development warrants developing governance indicators. This facilitates decision making, international performance comparisons, benchmarking, assessing governance quality and adopting corrective measures which improve the service delivery system. This study endeavours to develop a set of governance indicators for AIDUs in the UAE through a systematic literature review and the inputs gleaned from a group of experts by administering a questionnaire survey.
Conference Paper
Full-text available
Innovative and competitive regional development processes involve tangible and intangible assets. Hard factors such as infrastructure, etc. can usually be identified and measured quite easily. On the other hand, intangible factors, such as cultural and organizational (networking, leadership, trust, cooperation, etc.) assets needed to promote innovation, still challenge researchers and policy makers. Thus the increasing interest by funding entities (e.g. European Commission, Development Banks) and policy makers in finding ways to measure impacts at the cultural/organizational level in the innovation systems. To assess the quality and non-quantifiable impact of these factors is a complex task. Although there are some successful examples of how to measure intangible assets, most approaches are based on complex scientific research, requiring time-consuming field work and analysis, thus rarely applied in real capacity development programmes. The challenge is to find a methodological approach to measure intangible assets that does not ignore the complexity of the innovations system process at a regional level, but provide at the same time a simple and handy enough tool to be used in the assessment of innovation systems that will provide the meaningful feedback that the decision-making process requires. This article presents the Governance Capacities Assessment tool (GOCAPASS), a new method that has been tested during the implementation of the first Regional Innovation Strategy (according to the RIS methodology) in the Region of Aysén in Chile. The case of Aysén is also interesting from the point of view of promoting new types of innovation (non-technological, social, eco-innovation, innovation for sustainability), as the development paradigm of the region is based on its natural resources and on sustainable development rather than on industrial growth.
Article
Full-text available
Wicked problems present major challenges for evaluation as they cross the boundaries between countries, policy domains, organizations, and scientific disciplines. They force evaluators to study large networks in order to trace extended chains of cause and effect. However, beyond this category of ‘regular’ wicked problems that cross boundaries, there is another class of even more complex problems that truly transcend boundaries. Problems such as terrorism, the global financial crisis, and climate change transcend distinctions between cause and effect, local and global problems, fact and impressions. These issues represent a class of transboundary wicked problems that are not about local challenges and uncertainty, but about globally connected events and ambiguity. We draw on the literature on wickedness, ambiguity, and transboundary issues, to formulate a transboundary evaluation approach capable of assessing the governance of these extremely complex problems. The usefulness of this perspective is demonstrated through our evaluation of the Dutch national counterterrorism strategy 2011–2015. We detail: (a) the evaluation framework which provided focus in a diffuse transboundary field; (b) the evaluation principles which guided us as evaluators; and (c) the evaluation conditions which allowed us to engage and empower the actors being evaluated. These components can together bolster the relevance and competence of evaluators in a complex world.
Article
Full-text available
There is growing interest in evaluating policy implementation in ways that grapple with the complexity of the process. This article offers an example of using systems methodology to explore how the child protection policy in child contact centres has functioned in practice. Rather than just asking the traditional evaluation question “is it working?” this study sought to understand how the policy was working and how it was interpreted as it interacted with other systems, producing conflicts, local variation and emergent effects. It illustrates how the systems concepts of ‘emergence’, ‘local rationality’, ‘socio-technical systems’ and ‘feedback for learning’ can contribute new knowledge and understanding to a complex policy evaluation problem.
Article
Full-text available
This paper discusses the evaluation of networks of community-based, mostly publicly funded health, human service, and public welfare organizations. Consistent with pressures to perform effectively by a broad range of stakeholders, we argue that networks must be evaluated at three levels of analysis; community, network, and organization/participant levels. These three levels of analysis are closely related but each has its own set of effectiveness criteria that must be considered. We outline these criteria, discussing how effectiveness at one level of network analysis may or may not match effectiveness criteria at another level. We also discuss the importance of key stakeholders and how multiple constituents may be satisfied through the formation and maintenance of service delivery networks.
Book
Climate change is a cause for concern both globally and locally. In order for it to be tackled holistically, its governance is an important topic needing scientific and practical consideration. Climate change governance is an emerging area, and one which is closely related to state and public administrative systems and the behaviour of private actors, including the business sector, as well as the civil society and non-governmental organisations. Questions of climate change governance deal both with mitigation and adaptation whilst at the same time trying to devise effective ways of managing the consequences of these measures across the different sectors. Many books have been produced on general matters related to climate change, such as climate modelling, temperature variations, sea level rise, but, to date, very few publications have addressed the political, economic and social elements of climate change and their links with governance. This book will address this gap. Furthermore, a particular feature of this book is that it not only presents different perspectives on climate change governance, but it also introduces theoretical approaches and brings these together with practical examples which show how main principles may be implemented in practice.
Book
The concept of sustainable development is a source of inspiration for many, who see it as a call to cooperative action. In practice, however, policies intended to further this goal often generate conflicts of interest. The ensuing disputes occur among governmental organizations, but disputes also arise between public authorities, private interest groups, and the environmental movement. In the opinion of the authors, the fact that environmental policy can provoke such conflict may be attributed largely to decision making procedures in our society. Accordingly, the authors are convinced that a new approach to managing environmental disputes is needed in order to deal effectively with environmental problems. Indeed, this book presents a viable alternative, which is called network management.