The use of economic evaluations in NHS decision-making: a review and empirical investigation.

Abstract

To determine the extent to which health economic information is used in health policy decision-making in the UK, and to consider factors associated with the utilisation of such research findings.
Major electronic databases were searched up to 2004.
A systematic review of existing reviews on the use of economic evaluations in policy decision-making, of health and non-health literature on the use of economic analyses in policy making and of studies identifying actual or perceived barriers to the use of economic evaluations was undertaken. Five UK case studies of committees from four local and one national organisation [the Technology Appraisal Committee of the National Institute for Health and Clinical Excellence (NICE)] were conducted. Local case studies were augmented by documentary analysis of new technology request forms and by workshop discussions with members of local decision-making committees.
The systematic review demonstrated few previous systematic reviews of evidence in the area. At the local level in the NHS, it was an exception for economic evaluation to inform technology coverage decisions. Local decision-making focused primarily on evidence of clinical benefit and cost implications. And whilst information on implementation was frequently requested, cost-effectiveness information was rarely accessed. A number of features of the decision-making environment appeared to militate against emphasis on cost-effectiveness analysis. Constraints on the capacity to generate, access and interpret information, led to a minor role for cost-effectiveness analysis in the local decision-making process. At the national policy level in the UK, economic analysis was found to be highly integrated into NICE's technology appraisal programme. Attitudes to economic evaluation varied between committee members with some significant disagreement and extraneous factors diluted the health economics analysis available to the committee. There was strong evidence of an ordinal approach to consideration of clinical effectiveness and cost-effectiveness information. Some interviewees considered the key role of a cost-effectiveness analysis to be the provision of a framework for decision-making. Interviewees indicated that NICE makes use of some form of cost-effectiveness threshold but expressed concern about its basis and its use in decision-making. Frustrations with the appraisal process were expressed in terms of the scope of the policy question being addressed. Committee members raised concerns about lack of understanding of the economic analysis but felt that a single measure of benefit, e.g. the quality-adjusted life-year, was useful in allowing comparison of disparate health interventions and in providing a benchmark for later decisions. The importance of ensuring that committee members understood the limitations of the analysis was highlighted for model-based analyses.
This study suggests that research is needed into structures, processes and mechanisms by which technology coverage decisions can and should be made in healthcare. Further development of 'resource centres' may be useful to provide independent published analyses in order to support local decision-makers. Improved methods of economic analyses and of their presentation, which take account of the concerns of their users, are needed. Finally, the findings point to the need for further assessment of the feasibility and value of a formal process of clarification of the objectives that we seek from investments in healthcare.

Full text

Page 1

FINAL REPORT





THE USE OF ECONOMIC EVALUATIONS IN NHS
DECISION-MAKING:

A REVIEW AND EMPIRICAL INVESTIGATION





Iestyn Williams 1,2
Stirling Bryan 1
Shirley McIver 2

(with David Moore and Carey Hendron)






August 2006



1. Health Economics Facility, University of Birmingham
2. Health Services Management Centre, University of Birmingham

Page 2

2
Acknowledgements
This research was funded by the UK Department of Health Research Methodology Programme
(Project Number: 99/57/08).

We would like to thank all of the research participants who agreed to be subjected to our
observation and questioning. In particular, we are grateful to the National Institute for Clinical
Excellence (NICE) for granting us access to the Technology Appraisals committee meetings and the
documentation associated with those meetings. Additional thanks to David Barnett for giving us
comments on a draft of the NICE chapter. We similarly thank those involved with the local case
study committees that were studied – we have guaranteed them anonymity and so we cannot thank
them individually.

The research team has received much support and guidance from the internal and external advisory
groups. We particularly would like to acknowledge the contributions of Andrew Stevens, James
Raftery, Rod Tayor, Mike Drummond, Martin Buxton, Chris Hyde, Chris Hegginbotham and Alain
LiWanPo. On the review work specifically, we thank Anne Fry-Smith and Sue Bayliss. For
commenting on the draft final report, we thank Cam Donaldson. We would also like to thank the
two anonymous reviewers of the report for their helpful comments and suggestions.

Last, but not least, we have shared this work on numerous occasions with our colleagues in the
Health Economics Facility, and the wider department, the Health Services Management Centre, and
we thank them for their perseverance and helpful contributions.

Page 3

3
Contents

Executive Summary

Chapter 1: Introduction
1.1 Positive versus normative economics
1.2 Research utilisation
1.3 Providers and users of economic evaluation information in the UK
1.4 The problem to be addressed
1.5 Research questions

Chapter 2: A Systematic Review of the Literature on the use of
Economic Evaluation
2.1 Introduction
2.2 Review of existing reviews
2.3 Review of experimental and non-experimental studies in health care
2.4 Discussion
2.5 Conclusions

Chapter 3: Methodology and Methods
3.1 Overview of research approach
3.2 Additional details of research methods for local decision-making
3.3 Research approach in national case study
3.4 Summary

Chapter 4: Local Decision Makers
4.1 Introduction
4.2 Review of new technology request forms used by local
decision-making bodies
4.3 Case studies
4.4 Workshop discussions
4.5 Summary of chapter

Chapter 5: Case Study of the National Institute for Clinical Excellence
5. 1 Introduction
5. 2 The National Institute for Clinical Excellence: an introduction
5. 3 The seven technology appraisal topics
5.4 Data deriving from interviews with committee members

Chapter 6: Discussion
6.1 Levels of use of economic analyses
6.2 Issues of accessibility of economic analyses
6.3 Concerns relating to the acceptability of cost-effectiveness studies
6.4 Health system and organisational issues

Chapter 7: Conclusions

References

Appendices
4












10
10
12
15
16
17


























19
19
20
26
29
30




















32
32
44
48
51
















52
52




















52
59
97
101
102
102
102
104
143
















180
180
181
184
186
190
192
197

Page 4

4
Executive Summary

Background
The central problem addressed by the discipline of economics is that of resource scarcity, and so the
purpose of economic analysis is, in a very broad sense, to help decision makers when addressing
problems arising due to the scarcity issue. Therefore, such evidence is generated with the direct
intention of influencing policy - but is that objective achieved? Over recent years there have been
repeated expressions of concern about the usefulness of health economic analyses, and responses
have tended to centre on questions of how research by health economists can be made more useful
and accessible to policy makers. This report is concerned with the use (or lack of use) of research
evidence relating to economic analyses in health care decision-making.

Aims and objectives
The research described in this report addresses two principal questions:

?
To what extent, and in what ways, is health economic information used in health policy
decision-making in the UK?
?
What factors are associated with the utilisation (or non-utilisation) of such research
findings?

It is important to emphasise that the focus for this project is on the ‘policy level’ use of economic
analyses, and not their use in decisions regarding treatment for individual patients.

The five specific objectives of this research were:

1. To identify and review previous theoretical and empirical work concerned with economic
evaluation and policy level decision-making in health care and in other sectors.
2. At a local level in the NHS, to explore the use of economic evaluation information in
resource allocation decisions concerning adoption of drugs and other therapies.
3. At a national level (using the National Institute for Clinical Excellence), to explore the use
of economic analyses and its influence on its technology appraisal decisions and
judgements.
4. To explore with decision-makers how the impact of economic analyses might be increased,
particularly in relation to issues of accessibility and acceptability.
5. To make recommendations for improvements in the use of economic analyses by decision
makers in the NHS.

Systematic Review
A systematic review of the literature was undertaken that considered existing reviews on the use of
economic evaluations in policy decision-making, health and non-health literature on the use
economic analyses in policy making, and studies that have identified actual or perceived barriers to
the use of economic evaluations.

Overall the review exposed the difficulties of attempting systematically to search for evidence when
considering topics such as this. Despite these difficulties the review established the following:

?
There are very few previous systematic reviews of the evidence in this area – we identified
only one. The vast majority of reviews have been conducted in a non-systematic manner or
are more accurately described as opinion pieces.
?
A number of previous studies in health care have looked at the use of economic evaluations
in decision-making. Whilst these undoubtedly contribute to our knowledge on this topic,
there are some concerns about the methodological approach adopted in these studies.
Studies using surveys, in particular, frequently raise more questions than they answer.

Page 5

5
? There is a continuing need for research that addresses the range of policy decision-making
levels and which takes an in-depth, qualitative approach to addressing the research
question.

Empirical Research Methods
The research team adopted a predominantly qualitative approach involving primarily the use of case
study methods. This included documentary analysis, meeting observation and semi-structured
interviewing. Five case studies were conducted in total, including committees from four local and
one national organisation. The national case study was the technology appraisal committee of the
National Institute for Clinical Excellence (NICE). The NICE case study had two components: first,
an in-depth consideration of seven technology appraisals (using a variety of research techniques),
and second, interviews with 30 committee members to explore more general issues relating to their
use of economic analyses.

Case studies were augmented with a documentary analysis of new technology request forms used
by local decision-making committees and workshop discussions with members of local decision-
making committees. Data analysis was carried out iteratively by members of the research team and
draft findings were distributed to research respondents for their comments and feedback.

Empirical Research Results: the ‘local’ level

The review of proformas used by local decision-making bodies, the case studies of local decision-
making committees and the workshops with local decision makers show that local decision-making
focuses primarily on evidence of clinical benefit and cost implications. A relatively small number
of committees routinely ask for evidence about cost-effectiveness and case study research suggests
that even those requesting this information are not necessarily receiving it with any regularity. The
main sources of written health economic information are the manufacturers of technologies under
consideration, and guidance produced by the National Institute for Clinical Excellence, with only
limited access to other sources of economic evaluation.

Barriers to better use of health economics include limited resources and capacity to generate or
locate evaluations in time to inform decisions, problems relating to the inability to realise savings
identified in analyses, concerns about biases due to the source of the analysis, the robustness of
analyses or appropriateness of the comparators used. These problems arose within a general context
of a lack of incentives to use economic analysis and a lack of skills and understanding.

Ways of overcoming the barriers suggested were the need for a clear, standardised and generally
accepted format for the presentation of economic analysis including greater clarification about the
assumptions that went into models. In addition, health economics analysis could be improved by
making it more sensitive to the questions that health providers need answers to. A national resource
centre and archive of health economics tools and models that could be used was proposed together
with information about the benefits of using these tools. Training for committee members, and the
co-opting onto the committee of health economics expertise in cases where this was required were
also suggested. The lack of incentives for use of economic evaluation is partly an issue of the remit
of committees. Without some clarity and standardisation as to specific functions – especially in
relation to finance and budgets – of decision-making committees, it is difficult to prescribe a
general strategy for improvement.

Empirical Research Results: the ‘national’ case study

A number of the appraisals were hampered by concerns regarding the scope of the decision. In
some cases the view was expressed that a highly appropriate comparator had been excluded from
consideration. In another case, it was felt that the analyses received had adopted too short a time-
frame in which to measure the full benefits of treatment options. Three appraisals saw significant