The use of economic evaluations in NHS decision-making: a review and empirical investigation.
ABSTRACT To determine the extent to which health economic information is used in health policy decision-making in the UK, and to consider factors associated with the utilisation of such research findings.
Major electronic databases were searched up to 2004.
A systematic review of existing reviews on the use of economic evaluations in policy decision-making, of health and non-health literature on the use of economic analyses in policy making and of studies identifying actual or perceived barriers to the use of economic evaluations was undertaken. Five UK case studies of committees from four local and one national organisation [the Technology Appraisal Committee of the National Institute for Health and Clinical Excellence (NICE)] were conducted. Local case studies were augmented by documentary analysis of new technology request forms and by workshop discussions with members of local decision-making committees.
The systematic review demonstrated few previous systematic reviews of evidence in the area. At the local level in the NHS, it was an exception for economic evaluation to inform technology coverage decisions. Local decision-making focused primarily on evidence of clinical benefit and cost implications. And whilst information on implementation was frequently requested, cost-effectiveness information was rarely accessed. A number of features of the decision-making environment appeared to militate against emphasis on cost-effectiveness analysis. Constraints on the capacity to generate, access and interpret information, led to a minor role for cost-effectiveness analysis in the local decision-making process. At the national policy level in the UK, economic analysis was found to be highly integrated into NICE's technology appraisal programme. Attitudes to economic evaluation varied between committee members with some significant disagreement and extraneous factors diluted the health economics analysis available to the committee. There was strong evidence of an ordinal approach to consideration of clinical effectiveness and cost-effectiveness information. Some interviewees considered the key role of a cost-effectiveness analysis to be the provision of a framework for decision-making. Interviewees indicated that NICE makes use of some form of cost-effectiveness threshold but expressed concern about its basis and its use in decision-making. Frustrations with the appraisal process were expressed in terms of the scope of the policy question being addressed. Committee members raised concerns about lack of understanding of the economic analysis but felt that a single measure of benefit, e.g. the quality-adjusted life-year, was useful in allowing comparison of disparate health interventions and in providing a benchmark for later decisions. The importance of ensuring that committee members understood the limitations of the analysis was highlighted for model-based analyses.
This study suggests that research is needed into structures, processes and mechanisms by which technology coverage decisions can and should be made in healthcare. Further development of 'resource centres' may be useful to provide independent published analyses in order to support local decision-makers. Improved methods of economic analyses and of their presentation, which take account of the concerns of their users, are needed. Finally, the findings point to the need for further assessment of the feasibility and value of a formal process of clarification of the objectives that we seek from investments in healthcare.
Article: Facts, fallacies, and politics of comparative effectiveness research: Part 2 - implications for interventional pain management.[show abstract] [hide abstract]
ABSTRACT: The United States leads the world in many measures of health care innovation. However, it has been criticized to lag behind many developed nations in important health outcomes including mortality rates and higher health care costs. The surveys have shown the United States to outspend all other Organisation for Economic Co-operation and Development (OECD) countries with spending on health goods and services per person of $7,290 - almost 2(1/2) times the average of all OECD countries in 2007. Rising health care costs in the United States have been estimated to increase to 19.1% of gross domestic product (GDP) or $4.4 trillion by 2018. CER is defined as the generation and synthesis of evidence that compares the benefits and harms of alternate methods to prevent, diagnose, treat, and monitor a clinical condition or to improve the delivery of care. The, comparative effectiveness research (CER) has been touted by supporters with high expectations to resolve most ill effects of health care in the United States providing high quality, less expensive, universal health care. The efforts of CER in the United States date back to the late 1970s and it was officially inaugurated with the enactment of the Medicare Modernization Act (MMA). It has been rejuvenated with the American Recovery and Reinvestment Act (ARRA) of 2009 with an allocation of $1.1 billion. CER has been the basis of decision for health care in many other countries. Of all the available agencies, the National Institute for Health and Clinical Excellence (NICE) of the United Kingdom is the most advanced, stable, and has provided significant evidence, though based on rigid and proscriptive economic and clinical formulas. While CER is taking a rapid surge in the United States, supporters and opponents are emerging expressing their views. Since interventional pain management is a new and evolving specialty, with ownership claimed by numerous organizations, at times it is felt as if it has many fathers and other times it becomes an orphan. Part 2 of this comprehensive review will provide facts, fallacies, and politics of CER along with discussion of potential outcomes, impact of CER on health care delivery, and implications for interventional pain management in the United States.Pain physician 13(1):E55-79. · 10.72 Impact Factor
Article: Combining multicriteria decision analysis, ethics and health technology assessment: applying the EVIDEM decision-making framework to growth hormone for Turner syndrome patients.[show abstract] [hide abstract]
ABSTRACT: To test and further develop a healthcare policy and clinical decision support framework using growth hormone (GH) for Turner syndrome (TS) as a complex case study. The EVIDEM framework was further developed to complement the multicriteria decision analysis (MCDA) Value Matrix, that includes 15 quantifiable components of decision clustered in four domains (quality of evidence, disease, intervention and economics), with a qualitative tool including six ethical and health system-related components of decision. An extensive review of the literature was performed to develop a health technology assessment report (HTA) tailored to each component of decision, and content was validated by experts. A panel of representative stakeholders then estimated the MCDA value of GH for TS in Canada by assigning weights and scores to each MCDA component of decision and then considered the impact of non-quantifiable components of decision. Applying the framework revealed significant data gaps and the importance of aligning research questions with data needs to truly inform decision. Panelists estimated the value of GH for TS at 41% of maximum value on the MCDA scale, with good agreement at the individual level (retest value 40%; ICC: 0.687) and large variation across panelists. Main contributors to this panel specific value were "Improvement of efficacy", "Disease severity" and "Quality of evidence". Ethical considerations on utility, efficiency and fairness as well as potential misuse of GH had mixed effects on the perceived value of the treatment. This framework is proposed as a pragmatic step beyond the current cost-effectiveness model, combining HTA, MCDA, values and ethics. It supports systematic consideration of all components of decision and available evidence for greater transparency. Further testing and validation is needed to build up MCDA approaches combined with pragmatic HTA in healthcare decision-making.Cost Effectiveness and Resource Allocation 04/2010; 8:4. · 0.87 Impact Factor
Article: Impact of methodology on the results of economic evaluations of varicella vaccination programs: is it important for decision-making?[show abstract] [hide abstract]
ABSTRACT: This study aims to review the literature on economic evaluation of childhood varicella vaccination programs and to discuss how heterogeneity in methodological aspects and estimation of parameters can affect the studies' results. After applying the inclusion criteria, 27 studies published from 1980 to 2008 were analyzed in relation to methodological differences. There was great heterogeneity in the perspective adopted, evaluation of indirect costs, type of model used, modeling of the effect on herpes zoster, and estimation of vaccine price and efficacy parameters. The factor with the greatest impact on results was the inclusion of indirect costs, followed by the perspective adopted and vaccine price. The choice of a particular methodological aspect or parameter affected the studies' results and conclusions. It is essential that authors present these choices transparently so that users of economic evaluations understand the implications of such choices and the direction in which the results of the analysis were conducted.Cadernos de saúde pública / Ministério da Saúde, Fundação Oswaldo Cruz, Escola Nacional de Saúde Pública 01/2009; 25 Suppl 3:S401-14. · 0.83 Impact Factor
THE USE OF ECONOMIC EVALUATIONS IN NHS
A REVIEW AND EMPIRICAL INVESTIGATION
Iestyn Williams 1,2
Stirling Bryan 1
Shirley McIver 2
(with David Moore and Carey Hendron)
1. Health Economics Facility, University of Birmingham
2. Health Services Management Centre, University of Birmingham
This research was funded by the UK Department of Health Research Methodology Programme
(Project Number: 99/57/08).
We would like to thank all of the research participants who agreed to be subjected to our
observation and questioning. In particular, we are grateful to the National Institute for Clinical
Excellence (NICE) for granting us access to the Technology Appraisals committee meetings and the
documentation associated with those meetings. Additional thanks to David Barnett for giving us
comments on a draft of the NICE chapter. We similarly thank those involved with the local case
study committees that were studied – we have guaranteed them anonymity and so we cannot thank
The research team has received much support and guidance from the internal and external advisory
groups. We particularly would like to acknowledge the contributions of Andrew Stevens, James
Raftery, Rod Tayor, Mike Drummond, Martin Buxton, Chris Hyde, Chris Hegginbotham and Alain
LiWanPo. On the review work specifically, we thank Anne Fry-Smith and Sue Bayliss. For
commenting on the draft final report, we thank Cam Donaldson. We would also like to thank the
two anonymous reviewers of the report for their helpful comments and suggestions.
Last, but not least, we have shared this work on numerous occasions with our colleagues in the
Health Economics Facility, and the wider department, the Health Services Management Centre, and
we thank them for their perseverance and helpful contributions.
Chapter 1: Introduction
1.1 Positive versus normative economics
1.2 Research utilisation
1.3 Providers and users of economic evaluation information in the UK
1.4 The problem to be addressed
1.5 Research questions
Chapter 2: A Systematic Review of the Literature on the use of
2.2 Review of existing reviews
2.3 Review of experimental and non-experimental studies in health care
Chapter 3: Methodology and Methods
3.1 Overview of research approach
3.2 Additional details of research methods for local decision-making
3.3 Research approach in national case study
Chapter 4: Local Decision Makers
4.2 Review of new technology request forms used by local
4.3 Case studies
4.4 Workshop discussions
4.5 Summary of chapter
Chapter 5: Case Study of the National Institute for Clinical Excellence
5. 1 Introduction
5. 2 The National Institute for Clinical Excellence: an introduction
5. 3 The seven technology appraisal topics
5.4 Data deriving from interviews with committee members
Chapter 6: Discussion
6.1 Levels of use of economic analyses
6.2 Issues of accessibility of economic analyses
6.3 Concerns relating to the acceptability of cost-effectiveness studies
6.4 Health system and organisational issues
Chapter 7: Conclusions
The central problem addressed by the discipline of economics is that of resource scarcity, and so the
purpose of economic analysis is, in a very broad sense, to help decision makers when addressing
problems arising due to the scarcity issue. Therefore, such evidence is generated with the direct
intention of influencing policy - but is that objective achieved? Over recent years there have been
repeated expressions of concern about the usefulness of health economic analyses, and responses
have tended to centre on questions of how research by health economists can be made more useful
and accessible to policy makers. This report is concerned with the use (or lack of use) of research
evidence relating to economic analyses in health care decision-making.
Aims and objectives
The research described in this report addresses two principal questions:
To what extent, and in what ways, is health economic information used in health policy
decision-making in the UK?
What factors are associated with the utilisation (or non-utilisation) of such research
It is important to emphasise that the focus for this project is on the ‘policy level’ use of economic
analyses, and not their use in decisions regarding treatment for individual patients.
The five specific objectives of this research were:
1. To identify and review previous theoretical and empirical work concerned with economic
evaluation and policy level decision-making in health care and in other sectors.
2. At a local level in the NHS, to explore the use of economic evaluation information in
resource allocation decisions concerning adoption of drugs and other therapies.
3. At a national level (using the National Institute for Clinical Excellence), to explore the use
of economic analyses and its influence on its technology appraisal decisions and
4. To explore with decision-makers how the impact of economic analyses might be increased,
particularly in relation to issues of accessibility and acceptability.
5. To make recommendations for improvements in the use of economic analyses by decision
makers in the NHS.
A systematic review of the literature was undertaken that considered existing reviews on the use of
economic evaluations in policy decision-making, health and non-health literature on the use
economic analyses in policy making, and studies that have identified actual or perceived barriers to
the use of economic evaluations.
Overall the review exposed the difficulties of attempting systematically to search for evidence when
considering topics such as this. Despite these difficulties the review established the following:
There are very few previous systematic reviews of the evidence in this area – we identified
only one. The vast majority of reviews have been conducted in a non-systematic manner or
are more accurately described as opinion pieces.
A number of previous studies in health care have looked at the use of economic evaluations
in decision-making. Whilst these undoubtedly contribute to our knowledge on this topic,
there are some concerns about the methodological approach adopted in these studies.
Studies using surveys, in particular, frequently raise more questions than they answer.
? There is a continuing need for research that addresses the range of policy decision-making
levels and which takes an in-depth, qualitative approach to addressing the research
Empirical Research Methods
The research team adopted a predominantly qualitative approach involving primarily the use of case
study methods. This included documentary analysis, meeting observation and semi-structured
interviewing. Five case studies were conducted in total, including committees from four local and
one national organisation. The national case study was the technology appraisal committee of the
National Institute for Clinical Excellence (NICE). The NICE case study had two components: first,
an in-depth consideration of seven technology appraisals (using a variety of research techniques),
and second, interviews with 30 committee members to explore more general issues relating to their
use of economic analyses.
Case studies were augmented with a documentary analysis of new technology request forms used
by local decision-making committees and workshop discussions with members of local decision-
making committees. Data analysis was carried out iteratively by members of the research team and
draft findings were distributed to research respondents for their comments and feedback.
Empirical Research Results: the ‘local’ level
The review of proformas used by local decision-making bodies, the case studies of local decision-
making committees and the workshops with local decision makers show that local decision-making
focuses primarily on evidence of clinical benefit and cost implications. A relatively small number
of committees routinely ask for evidence about cost-effectiveness and case study research suggests
that even those requesting this information are not necessarily receiving it with any regularity. The
main sources of written health economic information are the manufacturers of technologies under
consideration, and guidance produced by the National Institute for Clinical Excellence, with only
limited access to other sources of economic evaluation.
Barriers to better use of health economics include limited resources and capacity to generate or
locate evaluations in time to inform decisions, problems relating to the inability to realise savings
identified in analyses, concerns about biases due to the source of the analysis, the robustness of
analyses or appropriateness of the comparators used. These problems arose within a general context
of a lack of incentives to use economic analysis and a lack of skills and understanding.
Ways of overcoming the barriers suggested were the need for a clear, standardised and generally
accepted format for the presentation of economic analysis including greater clarification about the
assumptions that went into models. In addition, health economics analysis could be improved by
making it more sensitive to the questions that health providers need answers to. A national resource
centre and archive of health economics tools and models that could be used was proposed together
with information about the benefits of using these tools. Training for committee members, and the
co-opting onto the committee of health economics expertise in cases where this was required were
also suggested. The lack of incentives for use of economic evaluation is partly an issue of the remit
of committees. Without some clarity and standardisation as to specific functions – especially in
relation to finance and budgets – of decision-making committees, it is difficult to prescribe a
general strategy for improvement.
Empirical Research Results: the ‘national’ case study
A number of the appraisals were hampered by concerns regarding the scope of the decision. In
some cases the view was expressed that a highly appropriate comparator had been excluded from
consideration. In another case, it was felt that the analyses received had adopted too short a time-
frame in which to measure the full benefits of treatment options. Three appraisals saw significant