A systematic review of factors affecting the judgments produced by formal consensus development methods in health care (In press)

Health Services Research Unit, London School of Hygiene and Tropical Medicine, London, UK.
Journal of Health Services Research & Policy (Impact Factor: 1.73). 08/2006; 11(3):172-9. DOI: 10.1258/135581906777641659
Source: PubMed

ABSTRACT Formal consensus development methods are ways of obtaining and synthesising views of experts, opinion leaders and other stakeholders, and are increasingly being used to develop clinical practice guidelines. Our objective was to examine the impact that the characteristics of individual participants, groups and the consensus process have on the judgments produced by formal consensus development methods in health care.
Studies were identified from an earlier methodological review and a search of five bibliographic databases for the period January 1996 to December 2004. Studies were eligible if they involved formal consensus development methods and reported differences in judgments between groups or participants. For studies comparing two or more groups overall percentage agreement, the kappa coefficient and the odds ratio for differences in judgments were calculated.
There were 22 studies comparing the impact of the characteristics of individual participants within groups and 30 studies comparing the results produced by two or more groups. Practitioners who perform a procedure tend to emphasise the appropriateness of the procedure compared with non-performing practitioners, and individuals from groups that were subject to performance criteria are more critical of those criteria than individuals from other groups. There was no clear pattern for the differences in judgments produced by participants and groups from different countries.
Except for participant specialty there is little general evidence for how the characteristics of participants and groups influence the judgments produced in formal consensus development methods. Multi-specialty groups are preferable to single-specialty groups because of their potential for taking account of a wider range of opinions.

1 Follower
  • Source
    • "All members of this panel had been actively involved in NHS funded venous leg ulcer trials and had at least two years specialist leg ulcer nursing experience. Although this was a small group, research evidence suggests that the group was an adequate size (Hutchings and Raine, 2006). These nurses were asked to independently complete the online survey before the consensus meeting date. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background Nurses caring for the large numbers of people with leg ulceration play a key role in promoting quality in health via their diagnostic and treatment clinical judgements. In the UK, audit evidence suggests that the quality of these judgements is often sub optimal. Misdiagnosis and incorrect treatment choices are likely to affect healing rates, patients’ quality of life, patient safety and healthcare costs. Objectives To explore the diagnostic judgements and treatment choices of UK community nurses managing venous leg ulceration. Design A judgement analysis based on Brunswik's psychological Lens Model theory. Setting UK community and primary care nursing services. Participants 18 community generalist nurses working in district (home) nursing teams and general practitioner services and 18 community tissue viability specialist nurses. Methods During 2011 and 2012, 36 nurses made diagnostic judgements and treatment choices in response to 110 clinical scenarios. Scenarios were generated from real patient cases and presented online using text and wound photographs. The consensus judgements of a panel of nurses with advanced knowledge of leg ulceration judged the same scenarios and provided a standard against which to compare the participants. Correlations and logistic regression models were constructed to generate various indices of judgement and decision “performance”: accuracy (Ra), consistency (Rs) and information use (G) and uncertainty (Re). Results Taking uncertainty into account, nurses could theoretically have achieved a diagnostic level of accuracy of 0.63 but the nurses only achieved an accuracy of 0.48. For the treatment judgement (whether applying high compression was warranted) nurses could have achieved an accuracy of 0.88 but achieved only an accuracy of 0.49. This may have been due to the nurses giving insufficient weight to the diagnostic cues of medical history and appearance of the leg and ulcer and insufficient weight to the treatment cues of type of leg ulcer and pain. Conclusion Clinical judgements and decisions made by nurses managing leg ulceration are complex and uncertain and some of the variability in judgements and choices can be explained by the ways in which nurses process the information and handle the uncertainties, present in clinical encounters.
    International Journal of Nursing Studies 09/2014; 52(1). DOI:10.1016/j.ijnurstu.2014.09.004 · 2.25 Impact Factor
  • Source
    • "Advisory group members in this study spoke about the vested interests they and other members bring to the table. This is similar to previous work reporting how professional experiences and beliefs hold sway over evidence in influencing recommendations [12,13]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: There has been significant investment in developing guidelines to improve clinical and public health practice. Though much is known about the processes of evidence synthesis and evidence-based guidelines implementation, we know little about how evidence presented to advisory groups is interpreted and used to form practice recommendations or what happens where evidence is lacking. This study investigates how members of advisory groups of NICE (National Institute of Health and Clinical Excellence) conceptualize evidence and experience the process. Members of three advisory groups for acute physical, mental and public health were interviewed at the beginning and end of the life of the group. Seventeen were interviewed at both time points; five were interviewed just once at time one; and 17 were interviewed only once after guidance completion. Using thematic and content analysis, interview transcripts were analysed to identify the main themes. Three themes were identified:1.What is the task? Different members conceptualized the task differently; some emphasized the importance of evidence at the top of the quality hierarchy while others emphasized the importance of personal experience.2.Who gets heard? Managing the diversity of opinion and vested interests was a challenge for the groups; service users were valued and as was the importance of fostering good working relationships between group members.3.What is the process? Group members valued debate and recognized the need to marshal discussion; most members were satisfied with the process and output. Evidence doesn't form recommendations on its own, but requires human judgement. Diversity of opinion within advisory groups was seen as key to making well-informed judgments relevant to forming recommendations. However, that diversity can bring tensions in the evaluation of evidence and its translation into practice recommendations.
    Implementation Science 09/2013; 8(1):101. DOI:10.1186/1748-5908-8-101 · 3.47 Impact Factor
  • Source
    • "Even with our private scoring process, scores may be influenced to an unknown extent by prior knowledge and opinions, combined with psychological and social factors operating during the discussion. The influence of group composition on recommendations made by groups developing evidencebased clinical guidelines for medical practice is a subject of active research (Hutchings & Raine 2006; Gardner et al. 2009; Hopthrow et al. 2011). Similar research on the processes by which guidance or evaluations are developed from evidence to support decisions in environmental policy is greatly needed. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Political institutions are keen to use the best available scientific knowledge in decision-making. For environmental policy, relevant scientific evidence can be complex and extensive, so expert judgment is frequently relied upon, without clear links to the evidence itself. We propose a new transparent process for incorporating research evidence into policy decisions, involving independent synopsis of evidence relating to all possible policy options combined with ex-pert evaluation of what the evidence means for specific policy questions. We illustrate the process using reforms of the European Union's Common Agricul-tural Policy currently being negotiated. Under the reform proposals, 30% of direct payments to farmers will become conditional upon three "compulsory greening measures." Independently, we compiled and evaluated experimental evidence for the effects of 85 interventions to protect wildlife on northern Eu-ropean farmland, 12 of which correspond to aspects of the compulsory green-ing measures. Our evaluation clearly indicates evidence of consistent wildlife benefits for some, but not all, of the greening measures. The process of evi-dence synopsis with expert evaluation has three advantages over existing ef-forts to incorporate evidence into policy decisions: it provides a clear evidence audit trail, allows rapid response to new policy contexts, and clarifies sources of uncertainty.
    Conservation Letters 01/2013; DOI:10.1111/conl.12046 · 5.03 Impact Factor
Show more