Barbara Befani's research while affiliated with University of Surrey and other places

Publications (26)

Article
Full-text available
As policy makers require more rigorous assessments for the strength of evidence in Theory-Based evaluations, Bayesian logic is attracting increasing interest; however, the estimation of probabilities that this logic (almost) inevitably requires presents challenges. Probabilities can be estimated on the basis of empirical frequencies, but such data...
Book
Full-text available
This guide is intended as a quick reference to evaluation approaches and methods. Its aim is to provide easily accessible and jargon-light descriptions of a broad yet select set of methodological approaches and methods used in evaluations in the field of international development (and beyond). The guide provides concise descriptions of established...
Article
Full-text available
This article discusses several practical issues arising with the application of diagnostic principles to theory-based evaluation (e.g. with Process Tracing and Bayesian Updating). It is structured around three iterative application steps, focusing mostly on the third. While covering different ways evaluators fall victims to confirmation bias and co...
Article
Full-text available
This chapter argues that the credibility of causal mechanisms can be greatly increased by formulating them as statements that are both empirically falsifiable and empirically confirmable. Whether statements can be so depends on the potential availability of the relevant evidence (e.g., no evidence exists that can prove or disprove the existence of...
Article
This article discusses the integration of a diagnostic lens in qualitative or mixed methods evaluations, arguing that this will improve quality: in particular, it will improve transparency, credibility and reliability of evaluation findings.We start by unpacking the notion of evaluation quality and pointing out the typical weaknesses of qualitative...
Article
Full-text available
While contribution analysis provides a step-by-step approach to verify whether and why an intervention is a contributory factor to development impact, most contribution analysis studies do not quantify the 'share of contribution' that can be attributed to a particular support intervention. Commissioners of evaluations, however, often want to unders...
Article
Full-text available
Qualitative comparative analysis (QCA) is gaining ground in evaluation circles, but the number of applications is still limited. In this article, we consider the challenges that can emerge during a QCA evaluation by drawing on our experience of conducting one in the field of development cooperation. For each stage of the evaluation process, we syst...
Research
Full-text available
In October 2014, the board of Uganda Wildlife Authority approved a policy change to increase the mountain gorilla tourist permit fee from US$5 to US$10 per permit. Focusing on the policy change, this evaluation interrogates the role and influence of Uganda Poverty and Conservation Group (U-PCLG) who appeared to have accelerated and shaped the UWA’s...
Article
Commissioners of impact evaluation often place great emphasis on assessing the contribution made by a particular intervention in achieving one or more outcomes, commonly referred to as a ‘contribution claim’. Current theory-based approaches fail to provide evaluators with guidance on how to collect data and assess how strongly or weakly such data s...
Technical Report
Full-text available
Development actors facing pressure to provide more rigorous assessments of their impact on policy and practice need new methods to deliver them. There is now a broad consensus that the traditional counterfactual analysis leading to the assessment of the net effect of an intervention is incapable of capturing the complexity of factors at play in any...
Article
This IDS Bulletin is the second of two that follow an Institute of Development Studies event seeking to define an agenda for research and practice of development impact evaluation. It focuses on exploring the potential of systems ideas and complexity concepts to meet the increasingly complex challenges of an increasingly ambitious development agend...
Article
This IDS Bulletin is the first of two special issues presenting contributions from the event ‘Impact Innovation and Learning: Towards a Research and Practice Agenda for the Future’, organised by IDS in March 2013. The initiative, as well as these two issues, represent a ‘rallying cry’ for impact evaluation to rise to the challenges of a post-MDG/po...
Article
Full-text available
This article proposes a combination of a popular evaluation approach, contribution analysis (CA), with an emerging method for causal inference, process tracing (PT). Both are grounded in generative causality and take a probabilistic approach to the interpretation of evidence. The combined approach is tested on the evaluation of the contribution of...
Article
This article argues that Qualitative Comparative Analysis can be a useful method in case-based evaluations for two reasons: a) it is aimed at causal inference and explanation, leading to theory development; b) it is strong on external validity and generalization, allowing for theory testing and refinement. After a brief introduction to QCA, the spe...
Article
This article argues that evaluation-specific logic, which involves the identification of relevant criteria and performance standards for the evaluand (considered as a whole or in its single dimensions), can benefit from research on the causal chains leading from programme inputs to the final outcomes; and on the contexts and mechanisms responsible,...
Article
Full-text available
International relations research has regarded networks as a particular mode of organization, distinguished from markets or state hierarchies. In contrast, network analysis permits the investigation and measurement of network structures---emergent properties of persistent patterns of relations among agents that can define, enable, and constrain thos...
Article
This article pursues two aims, a major and a minor one. The major one is to read some important results in the sociological literature on suicide, namely those obtained by Durkheim in 1897, Giddens in 1966 and Pope and Danigelis in 1981, as Context–Mechanism–Outcome configurations. The realist interpretation of the theories of suicide is deemed to...
Article
This article presents an innovative evaluation approach jointly inspired by the two renowned, theory-oriented, Pawson and Tilley' realistic evaluation and Carol Weiss's theory-based evaluation. First, the two approaches' main features are reviewed, with special regard to those which are preserved in the unified model. Secondly, their theoretical di...

Citations

... C'est notamment ici que l'on retrouve les approches d'évaluation d'impact, qui visent à établir un lien de causalité entre 4. C'est d'autant plus important si, comme le soutient MQ Patton, « les gens recrutent probablement des évaluateurs parce qu'ils partagent des valeurs communes plutôt que pour les outils qu'ils proposent. » (cité dans Williams, 2018, p. 142). 5. Pour des listes de méthodes, voir Befani (2020) et Vaessen et al. (2020). Pour un aperçu des approches et méthodes en français, voir Quadrant Conseil (2017). ...
... Therefore, combining QCA with explanatory approaches can provide insight into the actual mechanisms 4 which lead to the outcome. Examples are process tracing, contribution analysis and realistic evaluation (Befani, 2020). Using both configurational and mechanistic approaches provides more accurate picture of the investigated reality than stand-alone QCA and better answers to the research questions . ...
... For our purposes, we would need to calculate the conditional Sensitivity values (see Supplemental Table S4) and the conditional Type I Error values (Supplemental Table S5). It is possible but potentially quite cumbersome to calculate these conditional probabilities for each piece of evidence and each theory; as we mention above in option 2a, a relatively acceptable shortcut is to consider the whole package as one single piece of evidence and calculate two single values to feed into the Bayes formula (D'Errico et al., 2017). ...
... Evaluation has developed a range of methods to draw out causal relationships that are distinct from established tools like randomised controlled trials (RCTs) (Befani and O'Donnell, 2016;Stern et al., 2012), which is often seen as the gold standard but requires that the environment be controlled. Diagnostic Theory-Based Evaluation, which applies Bayesian Updating to Theory-Based Evaluation, has been proposed as a promising methodology to account for the uncertainty inherent in testing causal theories or mechanisms (Befani, 2020a). Historically, when Bayesian principles have been applied to theory-based evaluation, various labels have been used: Bayesian Process Tracing (Fairfield and Charman, 2017), Process Tracing with Bayesian Updating , and Contribution Tracing (Befani and Stedman-Bryce, 2017). ...
... Both Process Tracing and Realist Evaluation are strong methodologies for reasoning and credibility because they are capable of developing precise mechanistic explanations (Stern et al., 2012;White and Philips, 2012). Contribution analysis, for example, takes advantage of Bayesian reasoning and evidence tests within Process Tracing and has built these explicitly into its approach in some cases (Befani and Mayne, 2014;Ton et al., 2019). ...
... Guy-Peters and Fontaine (2020) continue that based on this idea, QCA is similar to most-similar and most-different systems design, as they are also based on the concepts of necessity and sufficiency. QCA can produce empirically well-grounded, context-sensitive evidence about policy instruments (Pattyn et al. 2019), such as IPA when looking at policy models as conceptual systems. ...
... At the same time, not everything can be measured; this highlights the need for narratives that can complement and support quantitative information. Bringing both qualitative and quantitative information to bear remains a substantial challenge (Vogel, 2012), and impact assessment methodology needs to evolve considerably to address social processes and outcomes in robust ways; it is not yet up to the task (Befani et al., 2016). ...
... The subject of the training varied and included data aggregation, 27 new data entry form 28 and utilisation of new digital systems. [29][30][31][32][33][34][35][36][37][38][39][40][41][42][43][44][45] The effect of enhanced training with longer training period did not seem to exceed the effect of short-term training in improving accuracy and completeness ( figure 5 and 6), this is consistent with findings by Rowe et al. 46 The use of mHealth solutions alone was only effective in improving timeliness, using mHealth interventions in reducing the time lag between collection and usage points had an impact on improving timeliness. Conversely, it did not show an effect in preventing input errors at the point of collection and thus was not enough to ensure data accuracy. ...
... Mobile phone interventions, including short message service (SMS) appointment reminders, have shown to be effective for improving immunization coverage in low and middle-income countries. [14][15][16] In 2012, HKI began supporting the government of Côte d'Ivoire in the use of mobile phone messaging to increase immunization coverage, given that only 40% of children 12 months of age were estimated to be fully vaccinated in the country. 17 A pilot study was conducted in Korhogo district that involved providing SMS reminders to mothers prior to each routine child immunization and newly-introduced VAS visit. ...
... This demonstrates a logical flow depicted by a cause-effect link mostly labelled as activity, output, outcome, and impact. According to Befani and Stedman-Bryce (2017), the results chain framework is further underpinned by the impact evaluation theory. ...