ArticlePDF Available

The 'Payback Framework' explained



The Payback Framework, originally developed to examine the ‘impact’ or ‘payback’ of health services research, is explained. The Payback Framework is a research tool used to facilitate data collection and cross-case analysis by providing a common structure and so ensuring cognate information is recorded. It consists of a logic model representation of the complete research process, and a series of categories to classify the individual paybacks from research. Its multi-dimensional categorisation of benefits from research starts with more traditional academic benefits of knowledge production and research capacity-building, and then extends to wider benefits to society.
Research Evaluation September 2011 0958-2029/11/03000-13 US$12.00 © Beech Tree Publishing 2011
Research Evaluation, 20(3), September 2011, pages 000–000
DOI: 10.3152/095820211X13118583635756;
The ‘Payback Framework’ explained
Claire Donovan and Stephen Hanney
The Payback Framework, originally developed to examine the ‘impact’ or ‘payback’ of health services
research, is explained. The Payback Framework is a research tool used to facilitate data collection and
cross-case analysis by providing a common structure and so ensuring cognate information is recorded.
It consists of a logic model representation of the complete research process, and a series of categories
to classify the individual paybacks from research. Its multi-dimensional categorisation of benefits from
research starts with more traditional academic benefits of knowledge production and research capacity-
building, and then extends to wider benefits to society.
ly developed by Martin Buxton and Stephen
Hanney at the Health Economics Research
Group (HERG) at Brunel University, UK, to exam-
ine the ‘impact’ or ‘payback’ of health services re-
search (Buxton and Hanney, 1994; 1996). It was
further developed in studies of research funded by
the National Health Service (NHS) (Buxton and
Hanney, 1998), and subsequently extended in col-
laboration with RAND Europe to also examine basic
and early clinical biomedical research (Hanney et al,
2004; Wooding et al, 2005).
The Payback Framework consists of two ele-
ments: a logic model representation of the complete
research processes (for the purposes of research im-
pact evaluation), and a series of categories to classi-
fy the individual paybacks from research. The
framework has undergone some development and
revision, partly to reflect the perspectives of various
research funders who have commissioned studies
organised using the framework. Nevertheless, the
basic Payback Framework still retains most of its
original structure and elements.
The logic model is presented in Figure 1. It con-
sists of seven stages (0–6) and two interfaces be-
tween the research system and the wider political,
professional and economic environment.
The model facilitates analysis of the ‘story’ of a
research idea from initial inception (Stage 0) through
the research process (Stage 2) into dissemination
(Interface B) and on towards its impact on society,
potentially reaching the final outcomes of health and
economic benefits (Stage 6). Depending on the type
of research funding being considered, Stage 0 might
represent two rather different forms of topic identifi-
cation. It could be undertaken by researchers inter-
nally within the scientific community and be aimed
at addressing particular scientific imperatives or un-
answered questions. Alternatively, the topic identifi-
cation could involve, at least partially, the wider
environment and include policy-makers, healthcare
professionals, patient representatives, etc. (Buxton
and Hanney, 1996; Hanney et al, 2007).
The framework is a research tool to facilitate data
collection (by informing surveys, interview sched-
ules and documentary analysis) and cross-case analy-
sis by providing a common structure for each case
study, thereby ensuring cognate information for each
study is recorded in the same place. The model con-
tains numerous feedback loops and so is not meant
to imply that the research process is linear.
The multi-dimensional categorisation of benefits
from health research starts with more traditional
academic benefits of knowledge production and re-
search capacity-building. But the next three catego-
ries constitute wider benefits to society. Apart from
the first category, the others have various sub-
categories as illustrated in Table 1. There has been a
widening of the scope of some categories of bene-
fits, for example, the ‘Benefits from informing poli-
cy and product development’ category has expanded
Claire Donovan is Reader and Stephen Hanney is Professorial
Research Fellow at the Health Economics Research Group
(HERG), Brunel University, Uxbridge, UB8 3PH, UK; Email:; Tel: +44 (0)1895 267651.
The ‘Payback Framework’ explained
Research Evaluation September 2011
to give more emphasis to product development. This
widening is partly a consequence of the expansion of
the types of research to which the Payback Frame-
work has been applied, especially to basic and early
clinical research.
While it is not completely possible to tie the cate-
gories of benefits to specific stages of the model, it
is possible to identify broad correlations that show
where the categories of impacts are most likely to be
found in the logic model: in this instance the
‘Knowledge’ and ‘Benefits to future research and
research use’ categories together are generally the
primary outputs from research; the ‘Benefits from
informing policy and product development’ category
relates to the secondary outputs; and the categories
for ‘Health and health sector benefits’ and ‘Broader
economic benefits’, respectively, are generally the
final outcomes.
While the Payback Framework was originally
developed to examine the ‘impact’ or ‘payback’
of healthcare research, it has subsequently been
adapted to assess the impact of research in other are-
as such as the social sciences (Wooding et al, 2007;
Klautzer et al, 2011) and the humanities (Levitt et
al, 2010).
Buxton, Martin and Stephen Hanney 1994. Assessing Payback
from Department of Health Research and Development: Pre-
liminary Report. Volume 1: The Main Report. HERG Research
Report, No. 19. Uxbridge: HERG, Brunel University.
Buxton, Martin and Stephen Hanney 1996. How can payback
from health services research be assessed? Journal of Health
Service Research and Policy, 1(1), 35–43.
Buxton, Martin and Stephen Hanney 1997. Assessing Payback
from Department of Health Research and Development:
Figure 1. The logic model of the Payback Framework
Source: Hanney et al (2004)
Table 1. Example of the multi-dimensional categorisation of paybacks of the Payback Framework
Category Definition
1. Knowledge Journal articles; conference presentations; books; book chapters; research reports
2. Benefits to future research and
research use
Better targeting of future research
Development of research skills, personnel and overall research capacity
A critical capacity to absorb and utilise appropriately existing research including that from overseas
Staff development and educational benefits
3. Benefits from informing policy
and product development
Improved information bases for political and executive decisions
Other political benefits from undertaking research
Development of pharmaceutical products and therapeutic techniques
4. Health and health sector benefits
Improved health
Cost reduction in delivery of existing services
Qualitative improvements in the process of delivery
Improved equity in service delivery
5. Broader economic benefits
Wider economic benefits from commercial exploitation of innovations arising from R&D
Economic benefits from a healthy workforce and reduction in working days lost
Source: Adapted from Buxton and Hanney (1994, 1996, 1997) and Wooding et al (2004)
The ‘Payback Framework’ explained
Research Evaluation September 2011
Second Report. Volume 1: The Main Report. HERG Research
Report, No. 24. Uxbridge: HERG, Brunel University.
Buxton, Martin and Stephen Hanney 1998. Evaluating the NHS
R&D programme: will the programme give value for money?
Journal of the Royal Society of Medicine, 91(suppl 35): 2–6.
Hanney, Stephen, Jonathan Grant, Steven Wooding and Martin
Buxton 2004. Proposed methods for reviewing the outcomes
of research: the impact of funding by the UK’s Arthritis Re-
search Campaign. Health Research Policy and Systems, 2(4).
Hanney Stephen, Martin Buxton, Colin Green, Diane Coulson and
James Raftery 2007. An assessment of the impact of the NHS
Health Technology Assessment Programme. Health Technol-
ogy Assessment, 11(53).
Klautzer, Lisa, Stephen Hanney, Edward Nason, Jennifer Rubin,
Jonathan Grant and Steven Wooding 2011. Assessing policy
and practice impacts of social science research: the applica-
tion of the Payback Framework to assess the Future of Work
programme. Research Evaluation, 20(3), xxx–xxx.
Levitt, Ruth, Claire Celia, Stephanie Diepeveen, Siobhan Ni
Chonaill, Lila Rabinovich and Jan Tiessen 2010. Assessing
the Impact of Arts and Humanities Research at the University
of Cambridge. Report prepared for the University of Cam-
bridge and the Arts and Humanities Research Council. Cam-
bridge: RAND Europe. <
_reports/TR816/>, last accessed 15 July 2011.
Wooding, Steve, Steve Hanney, Martin Buxton and Jonathan
Grant 2004. The Returns from Arthritis Research Volume 1:
Approach, Analysis and Recommendations. Cambridge:
RAND Europe. <
RAND_MG251.pdf>, last accessed 15 July 2011.
Wooding, Steven, Stephen Hanney, Martin Buxton and Jonathan
Grant 2005. Payback arising from research funding: evaluation
of the Arthritis Research Campaign. Rheumatology (Oxford),
44(9), 1145–1156.
Wooding, Steven, Edward Nason, Lisa Klautzer, Jennifer Rubin,
Stephen Hanney and Jonathan Grant 2007. Policy and Prac-
tice Impacts of Research Funded by the Economic and Social
Research Council: a Case Study of the Future of Work Pro-
gramme, Approach and Analysis. Santa Monica, CA: The
RAND Corporation. <
pdf>, last accessed 15 July 2011.
... They define the stages in the process, the relationships between them and the feedback loops and the contributions of different actors (including users) at the different stages. The 'payback model' (Buxton and Hanney 1996;Donovan and Hanney 2011) was developed to assess the return to biomedical research investments and has been used extensively in this and other sectors. It 'attempts to combine elements of the rational, linear model with a more interactive approach' (Buxton and Hanney 1996: 38). ...
Full-text available
Impact assessment research has developed theory-based approaches to trace the societal impact of scientific research. Impact assessment typically starts from the perspective of a research investment, organization, or project. Research users, non-academic actors involved in knowledge production, translation, and application, are well represented in many of these approaches. Researcher users are usually positioned as contributors to research, recipients of research outputs, or beneficiaries of research-driven outcomes. This paper argues that impact assessment would benefit from a more comprehensive understanding and analysis of research valorization processes from the user perspective. The first half of the paper reviews key impact assessment literature to identify how research users are positioned and portrayed in relation to valorization processes. In the second half of the paper, we use the results of this review to propose a set of principles to guide a systematic approach to constructing user perspectives on research impact. We suggest four concepts for operationalization of this approach. The paper concludes that the addition of a more comprehensive research user perspective on research valorization would complement and enhance existing impact assessment approaches.
... Penfield (2014) recognises it as one of the most often applied approaches for impact assessment. The model uses healthcare field and includes an impact assessment of academic results and benefits for the society (Donovan & Hanney, 2011). As described by Hanney and Gonzalez-Block (2011), the payback framework model systematically links research to the benefits thereof and can be examined at two levels. ...
Full-text available
This book provides analysis of current trends in research evaluation worldwide and compares the research assessment and innovation ecosystems in Austria, Bulgaria, the Czech Republic, Hungary, Lithuania, the Netherlands, Poland and Slovenia. It argues that in each country the research assessment system is interdependent with the national innovation system and the overall institutional governance/enforcement. The lead author, Albena Vutsova, has over 20 years of experience in research assessment both in Bulgaria and at European institutions (incl. JRC) and has been a member of the European Network for Research Evaluation in the Social Sciences and the Humanities. The three authors work at Sofia University, Bulgaria.
... It consists of a logical model representing the complete research process, and a set of categories used to classify individual paybacks from a given research program. Its multidimensional classification of the benefits starts with the more traditional academic fallout in terms of knowledge production and research capacity-building, then extends to broader benefits to society [8]. These categories move "downstream" from research through translation to society, and can capture outputs of interest to different audiences. ...
Full-text available
Background: research in the health sector is an activity associated with an uncertain return on investment. This study aims to outline a multidimensional framework for assessing the imact of research funded by public (or mixed public and private) sponsors. Methods: a focus group representing different stakeholders was created to develop a sense of purpose that would be shared by the intended beneficiaries of the research assessment process, and others with a direct or indirect interest in a program’s implementation. Results: a framework was designed that takes a holistic approach to the impact of research in which not only technical, but also economic, social and political dimensions are interrelated with the results achieved for the population and health services. Conclusions: the framework reflected the views of different stakeholders, favoring the development of the positive, proactive relationships essential to orienting the process of transforming the evidence emerging from research into action for health.
... Our starting point it to use the 'Payback Framework', which has been developed for examining the impact of health research. 66 It offers a multidimensional categorisation of benefits ranging from more traditional academic benefits of knowledge production to wider benefits to society, but it may need some adaptation to this context. ...
Full-text available
Introduction Research–practice partnerships (RPPs) are long-term collaborations between research and practice that aim to conduct research that can be used to make practice-based improvements. They intentionally bring together diverse experience in decision making and seek to shift power dynamics so that all partners have a say. The Creating Care Partnerships project aims to explore whether the RPP approach developed within the US educational context can be successfully applied to the English care home context. The project involves a programme of codesign, implementation and evaluation within three case study sites. This protocol set outs the aims, research design and governance of the evaluation. Methods and analysis The evaluation takes a theory-based approach to explore how, why and in what circumstances RPPs in the care home context contribute to enhancing research and research use in local care homes and informing wider improvement efforts. A mixed-methods design will be used for each case study, including semistructured interviews, observations of RPP events and meetings, an online survey, activity diary and review of local data and documents. Data collection will proceed in waves, with the theory of change (ToC) being continually refined and used to guide further data collection and analysis. Insights will be drawn using Contribution Analysis, Realist Evaluation and systems perspectives to assess the contribution made by the case study sites to achieving outcomes and the influence of contextual factors. Economic consequences will be identified through the ToC, using a narrative economic analysis to assess costs, consequences and value for money. Ethics and dissemination The study has undergone ethics review by HRA Research Ethics Committee. It does not pose major ethical issues. A final report will be published and articles will be submitted to international journals.
... The side merit of these cases is the provision of harmonised, detailed and comparable data on SI (see Bornmann et al., 2019;Heyeres et al., 2019). In this perspective, the 'Payback Framework' has been recognised as the first and most relevant protocol for data collection (Bornmann, 2013;Donovan & Hanney, 2011;Hanney et al., 2013;Newson et al., 2018;Rivera et al., 2017;Sivertsen & Meijer, 2020;Tahamtan & Bornmann, 2020). ...
The discussion on the ‘societal impact of university research’ has emerged within the recent decade with different utilisation from the British REF impact stories to the EU Framework Programmes’ criteria. The fundamental idea is that (university) research should contribute to addressing societal challenges, such as climate change, and not just pursue ‘scientific excellence’. Several approaches have been developed, from case studies of the ‘productive interactions’ that co-create knowledge with societal stakeholders to the so-called ‘altmetrics’ tracking online dissemination of research outputs. Nonetheless, these experiences do not always point in the same direction, and the notion of ‘societal impact’ seems still unclearly defined. Given the growing policy pressure on universities to address societal challenges, this paper reviews the scientific literature on the so-called ‘societal impact’ of university research. By querying three scientific archives (Web of Science, Scopus and ScienceDirect), 135 relevant scientific publications are selected and discussed, looking for conceptualisations, monitoring methods and success factors. The review maps a value-laden discussion arguing for longer-term, multi-dimensional perspectives on university research impacts beyond bibliometric indicators. Findings highlight a progressive shift from ‘attribution’, i.e., looking for causal relationships between research and societal changes, to 'contribution', acknowledging researchers' efforts to engage with societal challenges.
Academia focuses on the interplay of Higher Education Institutions and external stakeholders. In this context, academia is concerned with the societal impact and impact created in interactions with external stakeholders; the latter is often referred to as impact co-creation. There is agreement that the related processes leading to an impact are complex and multi-dimensional. However, academics disagree on how the ultimate, wider impact of research should be measured. This study seeks to conceptualize societal impact through the lens of value co-creation, arguing that societal impact is best conceptualized as the uptake of research. Based on this, we developed a generic research impact assessment framework to facilitate evaluations and enable cross-sector learning. This study contributes to academia by providing an overarching understanding of impact creation, including wider research impact, and offers the perspective that any research project involving stakeholders to a certain extent, also entails co-production.
Purpose To evaluate the dissemination and implementation impacts of a rehabilitation intervention. Methods Systematic evaluation of data sources including academic publishing metrics, publications, and surveys was used to describe the dissemination and implementation impact of the graded repetitive arm supplementary program (GRASP). Three categories in the Payback Framework were evaluated: knowledge production and dissemination, benefits to future research and research use, and real-world uptake and implementation. Results In the Knowledge production and dissemination category, seven publications, authored by the GRASP research team, were associated with the GRASP, and there were approximately 17,000 download counts of GRASP manuals from the website from 120 countries. In the Benefits to future research and research use category, 15 studies and 8 registered clinical trials, authored by researchers outside of the GRASP team, have used GRASP as an intervention. In the real-world uptake and implementation category, GRASP has informed recommendations in 2 clinical guidelines and 20 review papers, and had high implementation uptake (e.g., 35% [53/154] of UK therapists surveyed had used GRASP; 95% [649/681] who downloaded GRASP had used it). More than 75% of those who had used GRASP identified that GRASP provides more intensity in upper extremity rehabilitation, is evidence-based and easy to implement, and the equipment and manual are easy to obtain. Conclusion The Payback Framework is useful to evaluate the dissemination and implementation impacts of a rehabilitation intervention. GRASP has been implemented extensively in clinical practice and community in a relatively short time since it has been developed.
Full-text available
The book traces the main stages of third mission and social impact assessment, in the context of the (recent) history of higher education and research assessment in Italy, starting with a review of the transformation of the relationship between the University and the society. Several social drives are leading the university to a series of transformations, at the national, European and international level. The requirement to use transparency criteria and the associated returns of research spending (in accountability) has become a fundamental issue in informing government, business, citizens and society on the achieved results. The introduction of social impact assessment frameworks is also advancing research agendas towards socially relevant domains, to provide solutions in international competitiveness, social welfare, sustainability and other grand challenges. A terminological and conceptual shift is occurring from a traditional concept of technology transfer and third mission towards a broader meaning of knowledge exchange and co-creation between universities and extra-academic actors and the impact generated. However, this shift towards a transdisciplinary, trans-epistemic and inclusive evaluation framework poses new challenges to capture this complexity, requiring the adoption of new methods and tools presented in this book. The international literature and debate on practices are vital combined with trial-and-error approaches, community involvement, and targeted pilot studies. Beyond controversies, resistances, and easily contrived enthusiasms, this is what this volume tries to investigate.
Background and objective: Little information is available on how to assess the impact of research studies conducted in government hospitals in Latin America and specifically in Mexico. We aimed to determine the returns on investment of the research projects that were carried out in the Hospital General "Dr. Manuel Gea Gonzalez" (HGMGG), a general university hospital located in Mexico City, using a categorization model. Methods: We conducted a study including bibliometric analyses of publications associated with all research studies performed during the period 2016-2019 in the HGMGG and investigator interviews, according to the payback framework categorization model. Results: All studies analyzed had a positive impact based on outcomes in 5 "payback categories": (1) knowledge; (2) research targeting, capacity building, and absorption; (3) policy and product development; (4) health benefits; and (5) broader economic benefits. Conclusions: To date, it has not been possible to establish a set of indicators that show the results of the investigations carried out by medical specialists in training, who carry out the bulk of medical care in general hospitals and in the National Institutes of Health in Mexico. We identified, in the 5 categories of the payback framework model, different areas of opportunity to improve the benefits of the hospital's medical services through the development of scientific research projects.
Full-text available
Diabetes Action Canada Strategy for Patient-Oriented Research (SPOR) Network in Chronic Disease was formed in 2016 and is funded primarily through the Canadian Institutes of Health Research (CIHR). We propose a novel mixed-methods approach to a network evaluation integrating the State of Network Evaluation framework and the Canadian Academy of Health Sciences (CAHS) preferred framework and indicators. We measure key network themes of connectivity, health and results, and impact and return on investment associated with health research networks. Our methods consist of a longitudinal cross-sectional network survey of members and social network analysis to examine Network Connectivity and assess the frequency of interactions, the topics discussed during them, and how networking effectively facilitates interactions and collaboration among members. Network Health will be evaluated through semistructured interviews, a membership survey inquiring about satisfaction and experience with the Network, and a review of documentary sources related to funding and infrastructure to evaluate Network Sustainability. Finally, we will examine Network Results and Impact using the CAHS preferred framework and indicators to measure returns on investment in health research across the five domains of the CAHS framework, which include: advancing knowledge, capacity building, informing decision making, health impact, and economic and social impact. Indicators will be assessed with various methods, including bibliometric analyses, review of relevant documentary sources (annual reports), member activities informing health and research policy, and Patient Partner involvement. The Network Evaluation will provide members and stakeholders with information for planning, improvements, and funding future Network endeavors.
Full-text available
There is increasing pressure for research funders to demonstrate, and seek to maximize, the payback from the research they fund. This report, prepared for and funded by the Arthritis Research Campaign (ARC), presents the results of an evaluation of 16 research grants awarded by ARC in the early 1990s. The main objective was to develop a system for evaluating arthritis research, with a view to allowing arc to stimulate and manage the exploitation of research advances so that they translate into outcomes of practical benefit to people with arthritis. The report presents a framework that conceptualizes the relationship between research inputs, process, output and outcomes. Using this framework, we catalogue a diverse range of research output and outcomes arising from these 16 grants and make a series of quantitative and qualitative assessments comparing, for example, payback from project grants versus programme grants. In conclusion, we make six observations: --There is a diversity of research payback. --The researcher is the key driver of research translation. --Short, focused project grants seem to provide value for money. --Intended and unintended flexibility in funding is used advantageously. --Referees' contributions to the peer-review process are of variable benefit. --The payback framework could be operationalized and embedded by ARC. The companion Volume 2 is a collection of the case studies. These case studies all follow a similar format based on the conceptual model and provide a rich and detailed narrative on the payback of each research grant.
Full-text available
Background External and internal factors are increasingly encouraging research funding bodies to demonstrate the outcomes of their research. Traditional methods of assessing research are still important, but can be merged into broader multi-dimensional categorisations of research benefits. The onus has hitherto been on public sector funding bodies, but in the UK the role of medical charities in funding research is particularly important and the Arthritis Research Campaign, the leading medical charity in its field in the UK, commissioned a study to identify the outcomes from research that it funds. This article describes the methods to be used. Methods A case study approach will enable narratives to be told, illuminating how research funded in the early 1990s was (or was not) translated into practice. Each study will be organised using a common structure, which, with careful selection of cases, should enable cross-case analysis to illustrate the strengths of different modes and categories of research. Three main interdependent methods will be used: documentary and literature review; semi-structured interviews; and bibliometric analysis. The evaluative framework for organising the studies was previously used for assessing the benefits from health services research. Here, it has been specifically amended for a medical charity that funds a wide range of research and is concerned to develop the careers of researchers. It was further refined in three pilot studies. The framework has two main elements. First, a multi-dimensional categorisation of benefits going from the knowledge produced in peer reviewed journal articles through to the health and potential economic gain. The second element is a logic model, which, with various stages, should provide a way of organising the studies. The stock of knowledge is important: much research, especially basic, will feed into it and influence further research rather than directly lead to health gains. The cross-case analysis will look for factors associated with outcomes. Conclusions The pilots confirmed the applicability of the methods for a full study which should assist the Arthritis Research Campaign to demonstrate the outcomes from its funding, and provide it with evidence to inform its own policies.
Full-text available
Using a structured evaluation framework to systematically review and document the outputs and outcomes of research funded by the Arthritis Research Campaign in the early 1990s. To illustrate the strengths and weaknesses of different modes of research funding. The payback framework was applied to 16 case studies of research grants funded in the early 1990s. Case study methodology included bibliometric analysis, literature and archival document review and key informant interviews. A range of research paybacks was identified from the 16 research grants. The payback included 302 peer-reviewed papers, postgraduate training and career development, including 28 PhD/MDs, research informing recommendations in clinical guidelines, improved quality of life for people with RA and the reduction of the likelihood of recurrent miscarriage for women with antiphospholipid syndrome. The payback arising from project grants appeared to be similar to that arising from other modes of funding that were better resourced. There is a wide diversity of research payback. Short focused project grants seem to provide value for money.
Full-text available
To consider how the impact of the NHS Health Technology Assessment (HTA) Programme should be measured. To determine what models are available and their strengths and weaknesses. To assess the impact of the first 10 years of the NHS HTA programme from its inception in 1993 to June 2003 and to identify the factors associated with HTA research that are making an impact. Main electronic databases from 1990 to June 2005. The documentation of the National Coordinating Centre for Health Technology Assessment (NCCHTA). Questionnaires to eligible researchers. Interviews with lead investigators. Case study documentation. A literature review of research programmes was carried out, the work of the NCCHTA was reviewed, lead researchers were surveyed and 16 detailed case studies were undertaken. Each case study was written up using the payback framework. A cross-case analysis informed the analysis of factors associated with achieving payback. Each case study was scored for impact before and after the interview to assess the gain in information due to the interview. The draft write-up of each study was checked with each respondent for accuracy and changed if necessary. The literature review identified a highly diverse literature but confirmed that the 'payback' framework pioneered by Buxton and Hanney was the most widely used and most appropriate model available. The review also confirmed that impact on knowledge generation was more easily quantified than that on policy, behaviour or especially health gain. The review of the included studies indicated a higher level of impact on policy than is often assumed to occur. The survey showed that data pertinent to payback exist and can be collected. The completed questionnaires showed that the HTA Programme had considerable impact in terms of publications, dissemination, policy and behaviour. It also showed, as expected, that different parts of the Programme had different impacts. The Technology Assessment Reports (TARs) for the National Institute for Health and Clinical Excellence (NICE) had the clearest impact on policy in the form of NICE guidance. Mean publications per project were 2.93 (1.98 excluding the monographs), above the level reported for other programmes. The case studies revealed the large diversity in the levels and forms of impacts and the ways in which they arise. All the NICE TARs and more than half of the other case studies had some impact on policy making at the national level whether through NICE, the National Screening Committee, the National Service Frameworks, professional bodies or the Department of Health. This underlines the importance of having a customer or 'receptor' body. A few case studies had very considerable impact in terms of knowledge production and in informing national and international policies. In some of these the principal investigator had prior expertise and/or a research record in the topic. The case studies confirmed the questionnaire responses but also showed how some projects led to further research. This study concluded that the HTA Programme has had considerable impact in terms of knowledge generation and perceived impact on policy and to some extent on practice. This high impact may have resulted partly from the HTA Programme's objectives, in that topics tend to be of relevance to the NHS and have policy customers. The required use of scientific methods, notably systematic reviews and trials, coupled with strict peer reviewing, may have helped projects publish in high-quality peer-reviewed journals. Further research should cover more detailed, comprehensive case studies, as well as enhancement of the 'payback framework'. A project that collated health research impact studies in an ongoing manner and analysed them in a consistent fashion would also be valuable.
The UK Economic and Social Research Council funded exploratory evaluation studies to assess the wider impacts on society of various examples of its research. The Payback Framework is a conceptual approach previously used to evaluate impacts from health research. We tested its applicability to social sciences by using an adapted version to assess the impacts of the Future of Work (FoW) programme. We undertook key informant interviews, a programme-wide survey, user interviews and four case studies of selected projects. The FoW programme had significant impacts on knowledge, research and career development. While some principal investigators (PIs) could identify specific impacts of their research, PIs generally thought they had influenced policy in an incremental way and informed the policy debate. The study suggests progress can be made in applying an adapted version of the framework to the social sciences. However, some impacts may be inaccessible to evaluation, and some evaluations may occur too early or too late to capture the impact of research on a constantly changing policy environment.
This project for the University of Cambridge and the Arts and Humanities Research Council (AHRC) assesses the impacts of arts and humanities research at the University of Cambridge. Evidence from interviews, a survey of research staff and detailed case studies indicates that these disciplines already have a broad range of impacts. Many of these can be observed and described, while others are harder to define. This existing level of impact by the arts and humanities merits wider recognition, though continued efforts by the University and the AHRC remain worthwhile to ensure that it is maintained and, where possible, increased. The study used and adapted the "Payback Framework", which other universities can use to assess arts and humanities research impact. Benefits of research include: (1) Academic impacts: knowledge creation that increases understanding, challenges existing understanding or establishes new research trends; also the creation of resources for further research; (2) Policy impacts: research that informs new or revised policies (local, national or international), such as school curricula or professional guidelines, or that influences policy makers to make informed decisions; and (3) Impacts on practice: changes in professional behaviour such as shifts in legal interpretation and judgements, informed by research. Wider societal and economic impacts: public knowledge creation, preservation of heritage including objects, buildings and languages at risk, leisure and entertainment such as editions of literary works, theatrical productions; economic impacts such income from fees and grants, revenues from publications and exhibitions; and by training productive individuals whose activities are commercially competitive. Appended are: (1) List of interviewees; (2) Interview protocol for University of Cambridge interviewees; (3) Interview protocol for external interviews; and (4) Survey questions. (Contains 21 figures and 5 tables and 11 footnotes.) [The research described in this report was prepared for the University of Cambridge and the Arts and Humanities Research Council.]
Throughout the world there is a growing recognition that health care should be research-led. This strengthens the requirement for expenditure on health services research to be justified by demonstrating the benefits it produces. However, payback from health research and development is a complex concept and little used term. Five main categories of payback can be identified: Knowledge; research benefits; political and administrative benefits; health sector benefits; and broader economic benefits. Various models of research utilization together with previous assessments of payback from research helped in the development of a new conceptual model of how and where payback may occur. The model combines an input-output perspective with an examination of the permeable interfaces between research and its environment. The model characterizes research projects in terms of Inputs, Processes, and Primary Outputs. The last consist of knowledge and research benefits. There are two interfaces between the project and its environment. The first (Project Specification, Selection and Commissioning) is the link with Research Needs Assessment. The second (Dissemination) should lead to Secondary Outputs (which are policy or administrative decisions), and usually Applications (which take the form of behavioural changes), from which Impacts or Final Outcomes result. It is at this final stage that health and wider economic benefits can be measured. A series of case studies were used to assess the feasibility both of applying the model and the payback categorization. The paper draws various conclusions from the case studies and identifies a range of issues for further work.
The Returns from Arthritis Research
  • Wooding
  • Steve Steve
  • Martin Hanney
  • Jonathan Grant Buxton
Wooding, Steve, Steve Hanney, Martin Buxton and Jonathan Grant 2004. The Returns from Arthritis Research Volume 1: Approach, Analysis and Recommendations. Cambridge: RAND Europe. < RAND_MG251.pdf>, last accessed 15 July 2011.
Policy and Prac-tice Impacts of Research Funded by the Economic and Social Research Council: a Case Study of the Future of Work Pro-gramme, Approach and Analysis
  • Wooding
  • Edward Steven
  • Lisa Nason
  • Jennifer Rubin Klautzer
  • Stephen Hanney
Wooding, Steven, Edward Nason, Lisa Klautzer, Jennifer Rubin, Stephen Hanney and Jonathan Grant 2007. Policy and Prac-tice Impacts of Research Funded by the Economic and Social Research Council: a Case Study of the Future of Work Pro-gramme, Approach and Analysis. Santa Monica, CA: The RAND Corporation. < of_the_Future_of_Work_Programme_Volume_2_tcm8-4563. pdf>, last accessed 15 July 2011.