ASIRPA: A comprehensive theory-based approach to assessing the societal impacts of a research organization
Abstract
ASIRPA is an original and comprehensive approach for assessing the socio-economic impact of public-sector research organizations
through case studies. The cases are theory-based, selected to characterize the diversity of the broader impacts, and standardized
so as to allow the scaling-up of the analysis of impact to the level of the organization. ASIRPA is founded on well-tried
tools, and the added value of the approach lies in the adaptation and combination of these tools to design a comprehensive
approach, which has been tested in a real situation and proven to be robust, credible, and implementable.
... These two domains were further enriched by recording the quality descriptors mentioned by both ISS providers and beneficiaries, as used in the methodological approach of Joly et al. (2015) and following an abductive approach. ...
... The research community is increasingly recognizing the need to reconsider altmetrics concerning impact (Spaapen & van Drooge 2011;Joly et al. 2015;Morton 2015). Rather than functioning as direct indicators of impact, recent studies suggest that altmetrics are better understood as tools for analysing how research engages with society and how knowledge circulates beyond academic boundaries (Haustein et al. 2016;Ravenscroft et al. 2017). ...
This study examines the use of evidence in policymaking by analysing a range of journal and article attributes, as well as online engagement metrics. It employs a large-scale citation analysis of nearly 150,000 articles covering diverse policy topics. The findings highlight that scholarly citations exert the strongest positive influence on policy citations. Articles from journals with a higher citation impact and larger Mendeley readership are cited more frequently in policy documents. Other online engagements, such as news and blog mentions, also boost policy citations, while mentions on social media X have a negative effect. The finding that highly cited and widely read papers are also frequently referenced in policy documents likely reflects the perception among policymakers that such research is more trustworthy. In contrast, papers that derive their influence primarily from social media tend to be cited less often in policy contexts.
... Ce modèle vise ainsi à créer des synergies entre recherche académique, formation des ingénieurs agronomes et innovation industrielle. (Barret et al., 2017;Joly et al., 2015). La méthodologie combine des approches qualitatives et quantitatives (de Reviers et al., 2018) pour fournir une compréhension approfondie des changements induits par les activités de la Chaire. ...
... Nevertheless, this was without a theoretical basis (Ravenscroft et al. 2017). The scholarly assessment literature distinguishes between the scientific impact within academia and the broader societal impact according to Spaapen & van Drooge (2011), Joly et al. (2015), and Morton (2015). Despite the initial excitement about altmetrics as a way to gauge societal impact due to the emphasis of funding agencies, the field now acknowledges the necessity for a new strategy. ...
This study explores the connection between patent citations and scientific publications across six fields: Biochemistry, Genetics, Pharmacology, Engineering, Mathematics, and Physics. Analysing 117,590 papers from 2014 to 2023, the research emphasises how publication year, open access (OA) status, and discipline influence patent citations. Openly accessible papers, particularly those in hybrid OA journals or green OA repositories, are significantly more likely to be cited in patents, seven times more than those mentioned in blogs, and over twice as likely compared to older publications. However, papers with policy-related references are less frequently cited, indicating that patents may prioritise commercially viable innovations over those addressing societal challenges. Disciplinary differences reveal distinct innovation patterns across sectors. While academic visibility via blogs or platforms like Mendeley increases within scholarly circles, these have limited impact on patent citations. The study also finds that increased funding, possibly tied to applied research trends and fully open access journals, negatively affects patent citations. Social media presence and the number of authors have minimal impact. These findings highlight the complex factors shaping the integration of scientific research into technological innovations.
... To achieve this, we employ a case study approach in which we consider each H2020 project as a bounded system in which societal impact is implemented and managed. In doing so, our work complements well-known societal impact assessment methodologies such as ASIRPA (Joly et al. 2015), Contribution Mapping (Kok and Schuit 2012), and SIAMPI (Spaapen and Van Drooge 2011) which assess impacts, or research activities and interactions between researchers and stakeholders that precede these impacts. ...
This paper explores the management of societal impact in the context of Horizon 2020 (H2020) Social Sciences and Humanities (SSH) collaborative projects. Given the increasing demand for societal relevance in European-funded research, this study aims to (a) understand how societal impact is managed and (b1) identify the driving forces and (b2) barriers that influence impact management. By conducting semi-structured interviews with ten project coordinators from different European SSH projects, selected through purposive sampling procedures, the collected qualitative data were analysed using Atlas.ti. The thematic analysis revealed six themes: effective framework, stakeholder involvement; appropriate dissemination, competence, diversity and experienced members, limited post-project impact resources; and ineffective planning. This paper contributes to the discourse on project management by proposing a refined evaluation framework for societal impact that recognises different management strategies and considers the need for strategic planning and resource allocation to achieve sustainable societal impact. Our findings advocate for policies that promote greater transparency and inclusiveness in project evaluation, ultimately aiming to better align research outcomes with European policy priorities and societal challenges.
This study explores the connection between patent citations and scientific publications across six fields: Biochemistry, Genetics, Pharmacology, Engineering, Mathematics, and Physics. Analysing 117,590 papers from 2014 to 2023, the research emphasizes how publication year, open access (OA) status, and discipline influence patent citations. Openly accessible papers, particularly those in hybrid OA journals or green OA repositories, are significantly more likely to be cited in patents, seven times more than those mentioned in blogs, and over twice as likely compared to older publications. However, papers with policy-related references are less frequently cited, indicating that patents may prioritize commercially viable innovations over those addressing societal challenges. Disciplinary differences reveal distinct innovation patterns across sectors. While academic visibility via blogs or platforms like Mendeley increases within scholarly circles, these have a limited impact on patent citations. The study also finds that increased funding, possibly tied to applied research trends and fully open access journals, negatively affects patent citations. Social media presence and the number of authors have minimal impact. These findings highlight the complex factors shaping the integration of scientific research into technological innovations.
Understanding the use and users of social media in the context of scientific communication is essential. Analyzing and identifying different types of public engagement provides valuable insights into how academics and broader society interact with scientific outputs. This study takes quantum physics papers as samples and employs machine learning, scientometric methods, and regression analysis to unravel the interaction patterns and characteristics of various users. The user identification results indicate that, in addition to academics, several professional groups actively participate in disseminating scientific papers on Twitter. Using the density peak clustering method, the study profiles users and finds that those who play key roles in the dissemination network exhibit distinct interactive features in their social media behavior. Examining interaction patterns, including mentions, retweets, and quotes, reveals that users display homogeneity in the mentioning pattern, while in retweets and quotes, they are more inclined to engage with the academic category. Furthermore, the sentiment orientation, media richness, and dialogic loop elements of scientific tweets have varying influences on user engagement, with user types serving as significant moderators. By integrating Altmetrics metrics, this study contributes to the literature by providing empirical evidence that the involvement of diverse stakeholders enhances the dissemination of academic achievements on social media. These findings improve our understanding of the societal impact of publications and shed light on the interactions between science and society.
The changing culture of public administration involves accountability for results and outcomes. This article suggests that performance measurement can address such attribution questions. Contribution analysis has a major role to play in helping managers, researchers, and policymakers to arrive at conclusions about the contribution their program has made to particular outcomes. The article describes the steps necessary to produce a credible contribution story.
This article analyses the emergence of problem-oriented, so-called Mode 2 research as a supplement to disciplinary-oriented
Mode 1 research. Developments in research are compared with research policy developments, focusing on the use of performance
indicator tools from new public management. Two examples from Denmark and the UK describe how this thinking has gained ground
in the overall management of the allocation of basic funds for university research. Based on the analysis of examples and
research policy developments, the article argues that existing indicators are lagging behind and need to be updated as they
do not adequately reflect the politically desired Mode 2 features of the public research system. Consequently, there is a
need to develop and implement new Mode 2 indicators for measuring and managing public research. In conclusion, the article
suggests possible new indicators of Mode 2-oriented research. These indicators need to be further developed and can be seen
as an agenda for further research.
There is a growing emphasis on the importance of research having demonstrable public benefit. Measurements of the impacts of research are therefore needed. We applied a modified impact assessment process that builds on best practice to 5 years (2003-2007) of intervention research funded by Australia's National Health and Medical Research Council to determine if these studies had post-research real-world policy and practice impacts.
We used a mixed method sequential methodology whereby chief investigators of eligible intervention studies who completed two surveys and an interview were included in our final sample (n = 50), on which we conducted post-research impact assessments. Data from the surveys and interviews were triangulated with additional information obtained from documentary analysis to develop comprehensive case studies. These case studies were then summarized and the reported impacts were scored by an expert panel using criteria for four impact dimensions: corroboration; attribution, reach, and importance.
Nineteen (38%) of the cases in our final sample were found to have had policy and practice impacts, with an even distribution of high, medium, and low impact scores. While the tool facilitated a rigorous and explicit criterion-based assessment of post-research impacts, it was not always possible to obtain evidence using documentary analysis to corroborate the impacts reported in chief investigator interviews.
While policy and practice is ideally informed by reviews of evidence, some intervention research can and does have real world impacts that can be attributed to single studies. We recommend impact assessments apply explicit criteria to consider the corroboration, attribution, reach, and importance of reported impacts on policy and practice. Impact assessments should also allow sufficient time between impact data collection and completion of the original research and include mechanisms to obtain end-user input to corroborate claims and reduce biases that result from seeking information from researchers only.
Une approche multidimensionnelle de la mesure des effets de la recherche publique agronomique : le cas de l’INRA
Consultable sur Internet : http://www.nesta.org.uk/publications/working_papers/assets/features/the_impact_of_direct_support_to_rnd_and_innovation_in_firms
A substantial investment is made each year in research to support environmental policies. Understanding the impact of this
research is important from a number of perspectives. What remains unclear is how such evaluations may be undertaken, particularly
as very little current practice is captured in the literature. This paper reports on a set of 10 exploratory case studies
of environmental research impact assessment in practice. Most of the impact evaluations identified have multiple objectives
and used a combination of research methods. Challenges include establishing attribution, the timing of an evaluation, how
to capture the duration of research impact, checking the reliability of information from key informant interviews and identifying
methods for capturing as many impacts as possible. Best and Holmes' (2010) framework is used to consider the status of the
case-study organisations in progressing from first generation linear models of knowledge to action to the more recently advocated
systems models.
Though the US National Science Foundation introduced a broader impacts criterion to their merit review process in 1997, policy evaluations remain still scarce. Reactions from different scientific fields varied. This paper aims to quantitatively compare the proposed broader impacts of 360 funded abstracts from biology, engineering, and mathematical/physical sciences. Specifically, it considers whether or not certain fields are more likely to propose certain types of broader impacts activities, whether women principal investigators are more likely to propose broader impacts, and the effect of grant size. This study demonstrates that cultural differences exist between scientific fields and also supports existing policy recommendations that encourage the creation of organizations and partnerships at university level to allow scientists to more easily participate in activities with broader impacts. Emphasizing broader impacts activities may also attract a more diverse scientific workforce, as many individuals do not pursue science because of a perceived lack of impact. © The Author 2012. Published by Oxford University Press. All rights reserved.
This paper characterizes the various impact patterns generated by an agricultural public research organization (PRO), namely INRA (National Institute for Agronomic Research). We define an impact pattern as the combination of specific research outputs with specific actors that generates various types of impact. The analysis is based on information related to more than a thousand INRA innovations for which research outputs, beneficiaries, and impacts, have been codified. A classification based on the Partitioning Around Medoids (PAM) method is used to identify the seven main impact patterns.
There are two patterns that correspond to traditional INRA interventions to foster agricultural sector competitiveness; two that are related to innovations in health and economic issues; and two that have impacts on the conservation of natural resources. The seventh involves scientific advice related to public policy decisions. The research outputs and beneficiaries differ across these impact patterns. For example, those with economic impacts are more related to the agricultural sectors while impact patterns in the area of health affect industrial firms.