ArticlePDF Available

The 'Payback Framework' explained

Authors:

Abstract

The Payback Framework, originally developed to examine the ‘impact’ or ‘payback’ of health services research, is explained. The Payback Framework is a research tool used to facilitate data collection and cross-case analysis by providing a common structure and so ensuring cognate information is recorded. It consists of a logic model representation of the complete research process, and a series of categories to classify the individual paybacks from research. Its multi-dimensional categorisation of benefits from research starts with more traditional academic benefits of knowledge production and research capacity-building, and then extends to wider benefits to society.
Research Evaluation September 2011 0958-2029/11/03000-13 US$12.00 © Beech Tree Publishing 2011
1
Research Evaluation, 20(3), September 2011, pages 000–000
DOI: 10.3152/095820211X13118583635756; http://www.ingentaconnect.com/content/beech/rev
The ‘Payback Framework’ explained
Claire Donovan and Stephen Hanney
The Payback Framework, originally developed to examine the ‘impact’ or ‘payback’ of health services
research, is explained. The Payback Framework is a research tool used to facilitate data collection and
cross-case analysis by providing a common structure and so ensuring cognate information is recorded.
It consists of a logic model representation of the complete research process, and a series of categories
to classify the individual paybacks from research. Its multi-dimensional categorisation of benefits from
research starts with more traditional academic benefits of knowledge production and research capacity-
building, and then extends to wider benefits to society.
HE PAYBACK FRAMEWORK was original-
ly developed by Martin Buxton and Stephen
Hanney at the Health Economics Research
Group (HERG) at Brunel University, UK, to exam-
ine the ‘impact’ or ‘payback’ of health services re-
search (Buxton and Hanney, 1994; 1996). It was
further developed in studies of research funded by
the National Health Service (NHS) (Buxton and
Hanney, 1998), and subsequently extended in col-
laboration with RAND Europe to also examine basic
and early clinical biomedical research (Hanney et al,
2004; Wooding et al, 2005).
The Payback Framework consists of two ele-
ments: a logic model representation of the complete
research processes (for the purposes of research im-
pact evaluation), and a series of categories to classi-
fy the individual paybacks from research. The
framework has undergone some development and
revision, partly to reflect the perspectives of various
research funders who have commissioned studies
organised using the framework. Nevertheless, the
basic Payback Framework still retains most of its
original structure and elements.
The logic model is presented in Figure 1. It con-
sists of seven stages (0–6) and two interfaces be-
tween the research system and the wider political,
professional and economic environment.
The model facilitates analysis of the ‘story’ of a
research idea from initial inception (Stage 0) through
the research process (Stage 2) into dissemination
(Interface B) and on towards its impact on society,
potentially reaching the final outcomes of health and
economic benefits (Stage 6). Depending on the type
of research funding being considered, Stage 0 might
represent two rather different forms of topic identifi-
cation. It could be undertaken by researchers inter-
nally within the scientific community and be aimed
at addressing particular scientific imperatives or un-
answered questions. Alternatively, the topic identifi-
cation could involve, at least partially, the wider
environment and include policy-makers, healthcare
professionals, patient representatives, etc. (Buxton
and Hanney, 1996; Hanney et al, 2007).
The framework is a research tool to facilitate data
collection (by informing surveys, interview sched-
ules and documentary analysis) and cross-case analy-
sis by providing a common structure for each case
study, thereby ensuring cognate information for each
study is recorded in the same place. The model con-
tains numerous feedback loops and so is not meant
to imply that the research process is linear.
The multi-dimensional categorisation of benefits
from health research starts with more traditional
academic benefits of knowledge production and re-
search capacity-building. But the next three catego-
ries constitute wider benefits to society. Apart from
the first category, the others have various sub-
categories as illustrated in Table 1. There has been a
widening of the scope of some categories of bene-
fits, for example, the ‘Benefits from informing poli-
cy and product development’ category has expanded
T
Claire Donovan is Reader and Stephen Hanney is Professorial
Research Fellow at the Health Economics Research Group
(HERG), Brunel University, Uxbridge, UB8 3PH, UK; Email:
claire.donovan@brunel.ac.uk; Tel: +44 (0)1895 267651.
The ‘Payback Framework’ explained
Research Evaluation September 2011
2
to give more emphasis to product development. This
widening is partly a consequence of the expansion of
the types of research to which the Payback Frame-
work has been applied, especially to basic and early
clinical research.
While it is not completely possible to tie the cate-
gories of benefits to specific stages of the model, it
is possible to identify broad correlations that show
where the categories of impacts are most likely to be
found in the logic model: in this instance the
‘Knowledge’ and ‘Benefits to future research and
research use’ categories together are generally the
primary outputs from research; the ‘Benefits from
informing policy and product development’ category
relates to the secondary outputs; and the categories
for ‘Health and health sector benefits’ and ‘Broader
economic benefits’, respectively, are generally the
final outcomes.
While the Payback Framework was originally
developed to examine the ‘impact’ or ‘payback’
of healthcare research, it has subsequently been
adapted to assess the impact of research in other are-
as such as the social sciences (Wooding et al, 2007;
Klautzer et al, 2011) and the humanities (Levitt et
al, 2010).
References
Buxton, Martin and Stephen Hanney 1994. Assessing Payback
from Department of Health Research and Development: Pre-
liminary Report. Volume 1: The Main Report. HERG Research
Report, No. 19. Uxbridge: HERG, Brunel University.
Buxton, Martin and Stephen Hanney 1996. How can payback
from health services research be assessed? Journal of Health
Service Research and Policy, 1(1), 35–43.
Buxton, Martin and Stephen Hanney 1997. Assessing Payback
from Department of Health Research and Development:
Figure 1. The logic model of the Payback Framework
Source: Hanney et al (2004)
Table 1. Example of the multi-dimensional categorisation of paybacks of the Payback Framework
Category Definition
1. Knowledge Journal articles; conference presentations; books; book chapters; research reports
2. Benefits to future research and
research use
Better targeting of future research
Development of research skills, personnel and overall research capacity
A critical capacity to absorb and utilise appropriately existing research including that from overseas
Staff development and educational benefits
3. Benefits from informing policy
and product development
Improved information bases for political and executive decisions
Other political benefits from undertaking research
Development of pharmaceutical products and therapeutic techniques
4. Health and health sector benefits
Improved health
Cost reduction in delivery of existing services
Qualitative improvements in the process of delivery
Improved equity in service delivery
5. Broader economic benefits
Wider economic benefits from commercial exploitation of innovations arising from R&D
Economic benefits from a healthy workforce and reduction in working days lost
Source: Adapted from Buxton and Hanney (1994, 1996, 1997) and Wooding et al (2004)
The ‘Payback Framework’ explained
Research Evaluation September 2011
3
Second Report. Volume 1: The Main Report. HERG Research
Report, No. 24. Uxbridge: HERG, Brunel University.
Buxton, Martin and Stephen Hanney 1998. Evaluating the NHS
R&D programme: will the programme give value for money?
Journal of the Royal Society of Medicine, 91(suppl 35): 2–6.
Hanney, Stephen, Jonathan Grant, Steven Wooding and Martin
Buxton 2004. Proposed methods for reviewing the outcomes
of research: the impact of funding by the UK’s Arthritis Re-
search Campaign. Health Research Policy and Systems, 2(4).
Hanney Stephen, Martin Buxton, Colin Green, Diane Coulson and
James Raftery 2007. An assessment of the impact of the NHS
Health Technology Assessment Programme. Health Technol-
ogy Assessment, 11(53).
Klautzer, Lisa, Stephen Hanney, Edward Nason, Jennifer Rubin,
Jonathan Grant and Steven Wooding 2011. Assessing policy
and practice impacts of social science research: the applica-
tion of the Payback Framework to assess the Future of Work
programme. Research Evaluation, 20(3), xxx–xxx.
Levitt, Ruth, Claire Celia, Stephanie Diepeveen, Siobhan Ni
Chonaill, Lila Rabinovich and Jan Tiessen 2010. Assessing
the Impact of Arts and Humanities Research at the University
of Cambridge. Report prepared for the University of Cam-
bridge and the Arts and Humanities Research Council. Cam-
bridge: RAND Europe. <http://www.rand.org/pubs/technical
_reports/TR816/>, last accessed 15 July 2011.
Wooding, Steve, Steve Hanney, Martin Buxton and Jonathan
Grant 2004. The Returns from Arthritis Research Volume 1:
Approach, Analysis and Recommendations. Cambridge:
RAND Europe. <http://www.rand.org/pubs/monographs/2004/
RAND_MG251.pdf>, last accessed 15 July 2011.
Wooding, Steven, Stephen Hanney, Martin Buxton and Jonathan
Grant 2005. Payback arising from research funding: evaluation
of the Arthritis Research Campaign. Rheumatology (Oxford),
44(9), 1145–1156.
Wooding, Steven, Edward Nason, Lisa Klautzer, Jennifer Rubin,
Stephen Hanney and Jonathan Grant 2007. Policy and Prac-
tice Impacts of Research Funded by the Economic and Social
Research Council: a Case Study of the Future of Work Pro-
gramme, Approach and Analysis. Santa Monica, CA: The
RAND Corporation. <www.esrc.ac.uk/_images/Case_Study_
of_the_Future_of_Work_Programme_Volume_2_tcm8-4563.
pdf>, last accessed 15 July 2011.
... An emerging stream of literature has focused on interaction between researchers and stakeholders, taking slightly different perspectives. Examples are multidimensional models focused on the social impact of research, like the Payback model (Donovan and Hanney 2011) and Public Value Mapping (Bozeman and Sarewitz 2011). Other models aim to uncover the contribution of interactions, such as the Social Impact Assessment Methods through Productive Interactions (SIAMPI, Spaapen and van Drooge 2011) and Contribution Mapping (Kok and Schuit 2012). ...
... It consists of a logic model of the research processes, and various categories of research paybacks and anticipated impacts. The Payback Framework recognizes and explicitly makes room for a plurality of types of impact, and has been applied in a range of different contexts both within and beyond healthcare (Donovan and Hanney 2011). The framework is relevant for PBR because PBR involves different contexts, types of impacts, and ways of doing research. ...
... Payback Framework (Donovan and Hanney 2011) Facilitate data collection and crosscase analysis by providing a common structure and consists of two elements: ...
Article
Full-text available
This study explores the evaluation of research pathways of self-management health innovations from discovery to implementation in the context of practice-based research. The aim is to understand how a new process model for evaluating practice-based research provides insights into the implementation success of innovations. Data were collected from nine research projects in the Netherlands. Through document analysis and semi-structured interviews, we analysed how the projects start, evolve, and contribute to the healthcare practice. Building on previous research evaluation approaches to monitor knowledge utilization, we developed a Research Pathway Model. The model’s process character enables us to include and evaluate the incremental work required throughout the lifespan of an innovation project and it helps to foreground that innovation continues during implementation in real-life settings. We found that in each research project, pathways are followed that include activities to explore a new solution, deliver a prototype and contribute to theory. Only three projects explored the solution in real life and included activities to create the necessary changes for the solutions to be adopted. These three projects were associated with successful implementation. The exploration of the solution in a real-life environment in which users test a prototype in their own context seems to be a necessary research activity for the successful implementation of self-management health innovations.
... The literature review indicated that societal impact arose as the result of a range of knowledge transfer and exchange (KTE) processes between researchers, knowledge brokers, and users (Mitton et al. 2007;Dybdal, Nielsen, and Lemire 2010;Banzi et al. 2011). KTE models that focus on knowledge transfer processes typically use linear logic models with a number of phases and feedback loops Gillen 2010;Van Eerd et al. 2011;Schulte et al. 2017;Downes, Novicki, and Howard 2019;Felknor et al. 2019), such as the Payback Framework (Donovan and Hanney 2011). Some KTE models focus more on exchange processes and include interactions with stakeholders (Van Eerd, Moser, and Saunders 2021), spiral development processes (Goldenhar et al. 2001;Morrow, Goreham, and Ross 2017), or layers of knowledge interaction and competence building (Graham et al. 2006;Flaspohler et al. 2012;Kislov et al. 2014). ...
... Although feedback, loops, and similar demand pull-related processes are acknowledged in our logic model, the instrument does not measure such processes explicitly. The instrument relies on the assumption that feedback loops improve research and societal impact in sequences involving dissemination and use of knowledge (Goldenhar et al. 2001;Donovan and Hanney 2011). Other methodologies may more appropriately dissect such processes (Guthrie et al. 2013;Pedersen, Gronvad, and Hvidtfeldt 2020). ...
Article
Research funders and policymakers increasingly focus on societal benefits of their investments in research. Research institutions thus face increasing pressure to demonstrate their societal impact to prove their legitimacy and worth. To this end, research institutions need reliable, quantitative methods to measure societal impact. This article describes the development and test of an instrument to quantitatively measure societal impact of applied research at research institution and program levels. It demonstrates the successful validation of the instrument in the multidisciplinary field of occupational health and safety. The instrument, the Societal Impact Instrument: Occupational Health and Safety Research (SII:OHSR), produces an aggregate measure of societal impact for the research institution as a whole and subscales for each research program. The SII:OHSR instrument is built on a process model of knowledge translation and exchange. It has been developed in the context of multidisciplinary occupational health and safety research. The instrument is constructed as a generalized and context-independent tool that can be relocated to other research domains and languages. To the best of our knowledge, it is the first singular instrument that quantitatively measures societal impact. It is therefore highly pertinent for the research evaluation field.
... Depuis les années 1990, de nombreux frameworks ont été développés issus de différents domaines (Bornmann, 2013) : le Payback Framework (Donovan & Hanney, 2011), les interactions productives de la méthodologie SIAMPI (Molas-Gallart & Tang, 2011), ASPIRA , le REF (Watermeyer, 2016), Research Quality Framework australien (RQF) (Madison, 2007) ou encore des méthodes mixtes ad hoc (Bloch, Sørensen, et al., 2014). Chacun d'entre eux est associé à une littérature, principalement issue de la science policy, qui en discute les conditions d'application et les limites. ...
... Le premier insiste sur le lien entre la recherche menée et les bénéficies qui en résultent, à partir d'une conception linéaire inputs / outputs. L'exemple d'une approche de l'impact dans une perspective utilitariste est donné par le Payback Framework, initialement développé pour évaluer des unités de recherche dans le domaine médical (Donovan & Hanney, 2011). Il relie un modèle du processus de recherche aux bénéfices attendus, autant dans le domaine académique que dans différents secteur de la société comme les retours en termes de santé publique ou d'économie. ...
Thesis
L'Agence Nationale de la Recherche, créée en 2005, contribue à rapprocher le système de recherche français du modèle anglo-saxon, dans lequel les agences de financement jouent un rôle déterminant dans la conduite des politiques scientifiques. À partir d'une enquête sur l'agence et de l’étude des programmes de financement en génomique végétale et en chimie durable, cette thèse analyse le rôle de l'ANR dans la transformation du mode d'allocation des financements. Elle s’appuie sur des données issues d'entretiens auprès de chercheurs, d’un traitement statistique des projets déposés et de l'analyse de sources primaires et secondaires sur l’activité de l'ANR. Nous montrons notamment que l'ANR constitue une organisation intermédiaire partiellement autonome dont le métier a considérablement évolué depuis sa création en 2005. La carrière de l'agence témoigne des luttes de définition dont elle fait l'objet. Le gouvernement par projet pratiqué par l'ANR a contribué à généraliser la pratique des appels à projets compétitifs en France, qui favorise la différenciation des situations locales, la fragmentation des communautés de recherche et le changement du rapport des chercheurs aux financements. Plus généralement, nous montrons que la multiplication des appels à projets compétitifs conduit à établir ce que nous qualifions une économie de la recherche sur projets. Dans une telle économie, l'enjeu central pour les chercheurs devient le contrôle de la traduction des activités de recherche locales sous la forme générique de « projet » afin de mener leur activité.
... In this vein, proposals such as the Payback Framework differentiate outputs from outcomes and impacts (Donovan and Hanney 2011), while others focus on productive interactions (Spaapen and Van Drooge 2011). Additionally, approaches based on impact narratives have been developed and implemented, for example, in the UK Research Excellence Framework (Derrick 2018;Hellström and Hellström 2017). ...
Article
Full-text available
Technology development and innovation are fundamentally different from scientific research. However, in many circumstances, they are evaluated jointly and by the same processes. In these cases, peer review—the most usual procedure for evaluating research—is also applied to the evaluation of technological products and innovation activities. This can lead to unfair results and end up discouraging the involvement of researchers in these fields. This paper analyzes the evaluation processes in Uruguay's National System of Researchers. In this system, all members' activities, both scientific and technological, are evaluated by peer committees. Based on documentary analysis and semi-structured interviews, the difficulties faced by evaluators in assessing technology products are explored. The article highlights the persistence of a linear conception of the link between science and technology and describes the obstacles to assimilate the particularities of technological activities. Refereed publications are presented as the only uncontested product. Other types of output are reviewed with suspicion. This study emphasizes the need for specific mechanisms to evaluate technological production within academic careers.
Article
Research funded by the National Institute for Health Research Public Health Research Programme is being undertaken in a complex system which brings opportunities and challenges for researchers to maximise the impact of their research. This study seeks to better understand the facilitators, challenges and barriers to research impact and knowledge mobilisation from the perspective of UK public health researchers. A qualitative study using semi-structured interviews, informed by the Payback Framework, with public health researchers who held a research award with the National Institute for Health Research Public Health Research programme up to March 2017 was conducted. Following a thematic analysis, three strongly interlinked themes were extracted from the data and three key factors were highlighted as important for facilitating knowledge mobilisation and impact in UK public health research: (1) Public health researcher’s perception of the purpose of the research (2) Approaches to undertaking Knowledge mobilisation activities (3) The complex nature of public health research in the wider research context. These have been reflected onto the Payback framework. Public health researchers can maximise the likelihood for impact by being aware of the context in which they are undertaking research, using different methods, and employing several strategies to take advantage of opportunities. There is a need to support researchers with knowledge mobilisation activities and for funders to identify their expectations of the impact resulting from research. Our findings have relevance to public health researchers and funders interested in increasing the benefit that research brings to society.
Article
Full-text available
This paper presents a framework to understand the impact of scientific knowledge on the policy-making process, focusing on the conceptual impact. We note the continuing dissatisfaction with the quality and effects of science-policy interactions in both theory and practice. We critique the current literature’s emphasis on the efforts of scientists to generate policy impact, because it neglects the role of ‘user’ policymaking organisations. The framework offered in the paper develops an argument about the essential role of institutional conditions of policy ‘users’ for scientific knowledge to achieve impact. The framework is informed by the reflexive institutionalist and the neo-institutionalist theoretical approaches. Its main contribution is in outlining the intra- and inter-organisational conditions of policymaking organisations, along with personal characteristics of individual policy officials that influence the likelihood of scientific knowledge to generate conceptual impact. We also offer an operationalisation of the framework. The wider relevance of the paper is in moving the focus from the activities of scientists and the incentive structure in scientific organisations to the policy user side.
Article
Цитируемость все больше используется в качестве показателя производительности в научной политике и внутри исследовательской системы. Как правило, предполагается, что цитируемость свидетельствует о влиянии исследования или его качества. Что подтверждает эти предположения и как цитируемость соотносится с качеством исследований? Эти и подобные вопросы изучаются на протяжении десятилетий наукометрических исследований. Предоставляется обзор некоторых основных актуальных вопросов, включающих теории цитируемости, трактовку и обоснованность использования цитируемости как измерения результативности. Качество исследований является многоаспектным понятием, в котором достоверность/правильность, оригинальность, научная ценность, а также общественная ценность общепринято воспринимаются ключевыми характеристиками. Изучается то, как цитируемость может затрагивать подобные разнообразные измерения качества исследований. Утверждается, что цитируемость отражает аспекты, касающиеся научного влияния и релевантности, но с определенными ограничениями. С другой стороны, нет ни одного свидетельства, подтверждающего, что цитируемость отражает другие ключевые величины качества исследований. Следовательно, рост использования показателей цитируемости в оценке исследований и финансирования может снижать внимание к этим иным величинам качества исследований, таким как надежность/достоверность, оригинальность и общественная ценность.
Preprint
Full-text available
Landfill post-closure with contaminant concentration in soil below permissible limit assessed at limited spot does not represent the contamination issue. Assessment limit to professionals also does not gives a potential of change to practice constant assessment to a wider context of assessor - citizen living nearby - as a collaborative effort to sustain a safe environment. Therefore sizeable, qualitative, and cost-effective analysis of the concentrations of contaminants is needed and this work recommends kriging assessment and the logical impact pathway framework as factors of change in landfill aftercare management. The kriging framework is developed utilising lead (Pb) and chromium (Cr) data from inductively coupled plasma mass spectrometry (ICP-MS) analysis. The development of the kriging framework is conducted based on the observation of censored data from ICP-MS analysis. The estimation analysis involves the analysis of ordinary kriging with regression analysis, showing the interpolation of spatial correlation and regression error. Hence, ordinary kriging with regression of the variable of interest, i.e., Pb, using the data of the explanatory variable, i.e., Cr, is inappropriate. Further investigation with the utilisation of guess-field kriging analysis hypothetically exposed a potential contaminated area using an existing but limited number of explanatory variables; although, guess-field kriging may possibly result immense uncertainty at the area where the explanatory variable does not exist. Besides, this work anticipated outcomes in societal impact and sustainability practices from the proposed kriging framework by recommending a logical impact pathway. The development of the kriging framework and impact pathway reassure the necessary actions to be executed by responsible parties and act as the stimulus of a wider spectrum of improvement initiatives to oversee real issues, such as the time of occurrence, and to prevent negative impacts on the environment and humans.
Article
In recent years, the productive interactions approach has gained increasing prominence as a foundation for studying the socioeconomic impact of science. So far, however, it has been deployed primarily as a heuristic for analyses in the context of social sciences research at universities. The majority of impact studies of basic research infrastructures, in contrast, have remained focused on these facilities' long-term contribution to science, their role as economic agents or the learning effects induced through their collaboration with suppliers. This paper demonstrates that the array of productive interactions at basic research infrastructures can be substantially wider. First, it identifies a number of productive interactions relevant for this context based on workshops with diverse infrastructures' managements. Beyond qualified procurement, these include on-demand experiments, scientific collaboration with industrial partners, the provision of datasets, the development of technical equipment and external communication. Second, our survey of the DESY synchrotron's users reveals that productive interactions with external partners are prevalent and tend to be embedded in particular project types. Among such projects, we find those most associated with concrete impact that involve external partners in a formal manner.
Article
Systematic evaluation of the results of research activity funded by the state, including analysis of the impact of the results of research upon the development of the economy and society, has become common practice in governance of the science sector. It pertains to acquisition of evidence-based data and feedback necessary for decision making on the effectiveness of the existing governance techniques and selection of the methods (including legal) for their improvement. The exceptional complexity and dynamic change of the object of analysis (science, production of scientific knowledge) encourages to constantly search for new approaches worldwide for acquiring qualitative and accurate evaluations of the results of scientific research. From such perspective, systematic monitoring and analysis of the relevant foreign experience is advantageous for the theory and practice of governing scientific development, as it allows taking into account the mistakes and achievements of other countries with regards to development and improvement of their evaluation systems. Analysis is conducted on a range of systems (mainly European) and approaches towards evaluation of the contribution of sciences to socioeconomic development. The author reveals the peculiarities and flaws of the evaluation system under study. This article is first to demonstrate that the vast problematic field associated with the assessment of the contribution of sciences, should be viewed within the framework of the theory of state audit, which distinguishes between external and internal systems of monitoring. The conclusion is made that the peculiarities of functionality of science do not allow demarcating the contours of internal and external audit of the results of “scientific production” without compromising the quality of the acquired conclusions. The improvement of evaluation systems is a continuous process, associated with the co-evolution of science and scientific policy.
Article
Full-text available
There is increasing pressure for research funders to demonstrate, and seek to maximize, the payback from the research they fund. This report, prepared for and funded by the Arthritis Research Campaign (ARC), presents the results of an evaluation of 16 research grants awarded by ARC in the early 1990s. The main objective was to develop a system for evaluating arthritis research, with a view to allowing arc to stimulate and manage the exploitation of research advances so that they translate into outcomes of practical benefit to people with arthritis. The report presents a framework that conceptualizes the relationship between research inputs, process, output and outcomes. Using this framework, we catalogue a diverse range of research output and outcomes arising from these 16 grants and make a series of quantitative and qualitative assessments comparing, for example, payback from project grants versus programme grants. In conclusion, we make six observations: --There is a diversity of research payback. --The researcher is the key driver of research translation. --Short, focused project grants seem to provide value for money. --Intended and unintended flexibility in funding is used advantageously. --Referees' contributions to the peer-review process are of variable benefit. --The payback framework could be operationalized and embedded by ARC. The companion Volume 2 is a collection of the case studies. These case studies all follow a similar format based on the conceptual model and provide a rich and detailed narrative on the payback of each research grant.
Article
Full-text available
Background External and internal factors are increasingly encouraging research funding bodies to demonstrate the outcomes of their research. Traditional methods of assessing research are still important, but can be merged into broader multi-dimensional categorisations of research benefits. The onus has hitherto been on public sector funding bodies, but in the UK the role of medical charities in funding research is particularly important and the Arthritis Research Campaign, the leading medical charity in its field in the UK, commissioned a study to identify the outcomes from research that it funds. This article describes the methods to be used. Methods A case study approach will enable narratives to be told, illuminating how research funded in the early 1990s was (or was not) translated into practice. Each study will be organised using a common structure, which, with careful selection of cases, should enable cross-case analysis to illustrate the strengths of different modes and categories of research. Three main interdependent methods will be used: documentary and literature review; semi-structured interviews; and bibliometric analysis. The evaluative framework for organising the studies was previously used for assessing the benefits from health services research. Here, it has been specifically amended for a medical charity that funds a wide range of research and is concerned to develop the careers of researchers. It was further refined in three pilot studies. The framework has two main elements. First, a multi-dimensional categorisation of benefits going from the knowledge produced in peer reviewed journal articles through to the health and potential economic gain. The second element is a logic model, which, with various stages, should provide a way of organising the studies. The stock of knowledge is important: much research, especially basic, will feed into it and influence further research rather than directly lead to health gains. The cross-case analysis will look for factors associated with outcomes. Conclusions The pilots confirmed the applicability of the methods for a full study which should assist the Arthritis Research Campaign to demonstrate the outcomes from its funding, and provide it with evidence to inform its own policies.
Article
Full-text available
Using a structured evaluation framework to systematically review and document the outputs and outcomes of research funded by the Arthritis Research Campaign in the early 1990s. To illustrate the strengths and weaknesses of different modes of research funding. The payback framework was applied to 16 case studies of research grants funded in the early 1990s. Case study methodology included bibliometric analysis, literature and archival document review and key informant interviews. A range of research paybacks was identified from the 16 research grants. The payback included 302 peer-reviewed papers, postgraduate training and career development, including 28 PhD/MDs, research informing recommendations in clinical guidelines, improved quality of life for people with RA and the reduction of the likelihood of recurrent miscarriage for women with antiphospholipid syndrome. The payback arising from project grants appeared to be similar to that arising from other modes of funding that were better resourced. There is a wide diversity of research payback. Short focused project grants seem to provide value for money.
Article
Full-text available
To consider how the impact of the NHS Health Technology Assessment (HTA) Programme should be measured. To determine what models are available and their strengths and weaknesses. To assess the impact of the first 10 years of the NHS HTA programme from its inception in 1993 to June 2003 and to identify the factors associated with HTA research that are making an impact. Main electronic databases from 1990 to June 2005. The documentation of the National Coordinating Centre for Health Technology Assessment (NCCHTA). Questionnaires to eligible researchers. Interviews with lead investigators. Case study documentation. A literature review of research programmes was carried out, the work of the NCCHTA was reviewed, lead researchers were surveyed and 16 detailed case studies were undertaken. Each case study was written up using the payback framework. A cross-case analysis informed the analysis of factors associated with achieving payback. Each case study was scored for impact before and after the interview to assess the gain in information due to the interview. The draft write-up of each study was checked with each respondent for accuracy and changed if necessary. The literature review identified a highly diverse literature but confirmed that the 'payback' framework pioneered by Buxton and Hanney was the most widely used and most appropriate model available. The review also confirmed that impact on knowledge generation was more easily quantified than that on policy, behaviour or especially health gain. The review of the included studies indicated a higher level of impact on policy than is often assumed to occur. The survey showed that data pertinent to payback exist and can be collected. The completed questionnaires showed that the HTA Programme had considerable impact in terms of publications, dissemination, policy and behaviour. It also showed, as expected, that different parts of the Programme had different impacts. The Technology Assessment Reports (TARs) for the National Institute for Health and Clinical Excellence (NICE) had the clearest impact on policy in the form of NICE guidance. Mean publications per project were 2.93 (1.98 excluding the monographs), above the level reported for other programmes. The case studies revealed the large diversity in the levels and forms of impacts and the ways in which they arise. All the NICE TARs and more than half of the other case studies had some impact on policy making at the national level whether through NICE, the National Screening Committee, the National Service Frameworks, professional bodies or the Department of Health. This underlines the importance of having a customer or 'receptor' body. A few case studies had very considerable impact in terms of knowledge production and in informing national and international policies. In some of these the principal investigator had prior expertise and/or a research record in the topic. The case studies confirmed the questionnaire responses but also showed how some projects led to further research. This study concluded that the HTA Programme has had considerable impact in terms of knowledge generation and perceived impact on policy and to some extent on practice. This high impact may have resulted partly from the HTA Programme's objectives, in that topics tend to be of relevance to the NHS and have policy customers. The required use of scientific methods, notably systematic reviews and trials, coupled with strict peer reviewing, may have helped projects publish in high-quality peer-reviewed journals. Further research should cover more detailed, comprehensive case studies, as well as enhancement of the 'payback framework'. A project that collated health research impact studies in an ongoing manner and analysed them in a consistent fashion would also be valuable.
Article
The UK Economic and Social Research Council funded exploratory evaluation studies to assess the wider impacts on society of various examples of its research. The Payback Framework is a conceptual approach previously used to evaluate impacts from health research. We tested its applicability to social sciences by using an adapted version to assess the impacts of the Future of Work (FoW) programme. We undertook key informant interviews, a programme-wide survey, user interviews and four case studies of selected projects. The FoW programme had significant impacts on knowledge, research and career development. While some principal investigators (PIs) could identify specific impacts of their research, PIs generally thought they had influenced policy in an incremental way and informed the policy debate. The study suggests progress can be made in applying an adapted version of the framework to the social sciences. However, some impacts may be inaccessible to evaluation, and some evaluations may occur too early or too late to capture the impact of research on a constantly changing policy environment.
Article
This project for the University of Cambridge and the Arts and Humanities Research Council (AHRC) assesses the impacts of arts and humanities research at the University of Cambridge. Evidence from interviews, a survey of research staff and detailed case studies indicates that these disciplines already have a broad range of impacts. Many of these can be observed and described, while others are harder to define. This existing level of impact by the arts and humanities merits wider recognition, though continued efforts by the University and the AHRC remain worthwhile to ensure that it is maintained and, where possible, increased. The study used and adapted the "Payback Framework", which other universities can use to assess arts and humanities research impact. Benefits of research include: (1) Academic impacts: knowledge creation that increases understanding, challenges existing understanding or establishes new research trends; also the creation of resources for further research; (2) Policy impacts: research that informs new or revised policies (local, national or international), such as school curricula or professional guidelines, or that influences policy makers to make informed decisions; and (3) Impacts on practice: changes in professional behaviour such as shifts in legal interpretation and judgements, informed by research. Wider societal and economic impacts: public knowledge creation, preservation of heritage including objects, buildings and languages at risk, leisure and entertainment such as editions of literary works, theatrical productions; economic impacts such income from fees and grants, revenues from publications and exhibitions; and by training productive individuals whose activities are commercially competitive. Appended are: (1) List of interviewees; (2) Interview protocol for University of Cambridge interviewees; (3) Interview protocol for external interviews; and (4) Survey questions. (Contains 21 figures and 5 tables and 11 footnotes.) [The research described in this report was prepared for the University of Cambridge and the Arts and Humanities Research Council.]
Article
Throughout the world there is a growing recognition that health care should be research-led. This strengthens the requirement for expenditure on health services research to be justified by demonstrating the benefits it produces. However, payback from health research and development is a complex concept and little used term. Five main categories of payback can be identified: Knowledge; research benefits; political and administrative benefits; health sector benefits; and broader economic benefits. Various models of research utilization together with previous assessments of payback from research helped in the development of a new conceptual model of how and where payback may occur. The model combines an input-output perspective with an examination of the permeable interfaces between research and its environment. The model characterizes research projects in terms of Inputs, Processes, and Primary Outputs. The last consist of knowledge and research benefits. There are two interfaces between the project and its environment. The first (Project Specification, Selection and Commissioning) is the link with Research Needs Assessment. The second (Dissemination) should lead to Secondary Outputs (which are policy or administrative decisions), and usually Applications (which take the form of behavioural changes), from which Impacts or Final Outcomes result. It is at this final stage that health and wider economic benefits can be measured. A series of case studies were used to assess the feasibility both of applying the model and the payback categorization. The paper draws various conclusions from the case studies and identifies a range of issues for further work.
The Returns from Arthritis Research
  • Wooding
  • Steve Steve
  • Martin Hanney
  • Jonathan Grant Buxton
Wooding, Steve, Steve Hanney, Martin Buxton and Jonathan Grant 2004. The Returns from Arthritis Research Volume 1: Approach, Analysis and Recommendations. Cambridge: RAND Europe. <http://www.rand.org/pubs/monographs/2004/ RAND_MG251.pdf>, last accessed 15 July 2011.
Policy and Prac-tice Impacts of Research Funded by the Economic and Social Research Council: a Case Study of the Future of Work Pro-gramme, Approach and Analysis
  • Wooding
  • Edward Steven
  • Lisa Nason
  • Jennifer Rubin Klautzer
  • Stephen Hanney
Wooding, Steven, Edward Nason, Lisa Klautzer, Jennifer Rubin, Stephen Hanney and Jonathan Grant 2007. Policy and Prac-tice Impacts of Research Funded by the Economic and Social Research Council: a Case Study of the Future of Work Pro-gramme, Approach and Analysis. Santa Monica, CA: The RAND Corporation. <www.esrc.ac.uk/_images/Case_Study_ of_the_Future_of_Work_Programme_Volume_2_tcm8-4563. pdf>, last accessed 15 July 2011.