A Systematic Review of Factors Affecting the Judgments Produced by Formal Consensus Development Methods in Health Care

Health Services Research Unit, London School of Hygiene and Tropical Medicine, London, UK.
Journal of Health Services Research & Policy (Impact Factor: 1.73). 08/2006; 11(3):172-9. DOI: 10.1258/135581906777641659
Source: PubMed


Formal consensus development methods are ways of obtaining and synthesising views of experts, opinion leaders and other stakeholders, and are increasingly being used to develop clinical practice guidelines. Our objective was to examine the impact that the characteristics of individual participants, groups and the consensus process have on the judgments produced by formal consensus development methods in health care.
Studies were identified from an earlier methodological review and a search of five bibliographic databases for the period January 1996 to December 2004. Studies were eligible if they involved formal consensus development methods and reported differences in judgments between groups or participants. For studies comparing two or more groups overall percentage agreement, the kappa coefficient and the odds ratio for differences in judgments were calculated.
There were 22 studies comparing the impact of the characteristics of individual participants within groups and 30 studies comparing the results produced by two or more groups. Practitioners who perform a procedure tend to emphasise the appropriateness of the procedure compared with non-performing practitioners, and individuals from groups that were subject to performance criteria are more critical of those criteria than individuals from other groups. There was no clear pattern for the differences in judgments produced by participants and groups from different countries.
Except for participant specialty there is little general evidence for how the characteristics of participants and groups influence the judgments produced in formal consensus development methods. Multi-specialty groups are preferable to single-specialty groups because of their potential for taking account of a wider range of opinions.

1 Follower
15 Reads
  • Source
    • "All members of this panel had been actively involved in NHS funded venous leg ulcer trials and had at least two years specialist leg ulcer nursing experience. Although this was a small group, research evidence suggests that the group was an adequate size (Hutchings and Raine, 2006). These nurses were asked to independently complete the online survey before the consensus meeting date. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background Nurses caring for the large numbers of people with leg ulceration play a key role in promoting quality in health via their diagnostic and treatment clinical judgements. In the UK, audit evidence suggests that the quality of these judgements is often sub optimal. Misdiagnosis and incorrect treatment choices are likely to affect healing rates, patients’ quality of life, patient safety and healthcare costs. Objectives To explore the diagnostic judgements and treatment choices of UK community nurses managing venous leg ulceration. Design A judgement analysis based on Brunswik's psychological Lens Model theory. Setting UK community and primary care nursing services. Participants 18 community generalist nurses working in district (home) nursing teams and general practitioner services and 18 community tissue viability specialist nurses. Methods During 2011 and 2012, 36 nurses made diagnostic judgements and treatment choices in response to 110 clinical scenarios. Scenarios were generated from real patient cases and presented online using text and wound photographs. The consensus judgements of a panel of nurses with advanced knowledge of leg ulceration judged the same scenarios and provided a standard against which to compare the participants. Correlations and logistic regression models were constructed to generate various indices of judgement and decision “performance”: accuracy (Ra), consistency (Rs) and information use (G) and uncertainty (Re). Results Taking uncertainty into account, nurses could theoretically have achieved a diagnostic level of accuracy of 0.63 but the nurses only achieved an accuracy of 0.48. For the treatment judgement (whether applying high compression was warranted) nurses could have achieved an accuracy of 0.88 but achieved only an accuracy of 0.49. This may have been due to the nurses giving insufficient weight to the diagnostic cues of medical history and appearance of the leg and ulcer and insufficient weight to the treatment cues of type of leg ulcer and pain. Conclusion Clinical judgements and decisions made by nurses managing leg ulceration are complex and uncertain and some of the variability in judgements and choices can be explained by the ways in which nurses process the information and handle the uncertainties, present in clinical encounters.
    International Journal of Nursing Studies 09/2014; 52(1). DOI:10.1016/j.ijnurstu.2014.09.004 · 2.90 Impact Factor
  • Source
    • "Advisory group members in this study spoke about the vested interests they and other members bring to the table. This is similar to previous work reporting how professional experiences and beliefs hold sway over evidence in influencing recommendations [12,13]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: There has been significant investment in developing guidelines to improve clinical and public health practice. Though much is known about the processes of evidence synthesis and evidence-based guidelines implementation, we know little about how evidence presented to advisory groups is interpreted and used to form practice recommendations or what happens where evidence is lacking. This study investigates how members of advisory groups of NICE (National Institute of Health and Clinical Excellence) conceptualize evidence and experience the process. Members of three advisory groups for acute physical, mental and public health were interviewed at the beginning and end of the life of the group. Seventeen were interviewed at both time points; five were interviewed just once at time one; and 17 were interviewed only once after guidance completion. Using thematic and content analysis, interview transcripts were analysed to identify the main themes. Three themes were identified:1.What is the task? Different members conceptualized the task differently; some emphasized the importance of evidence at the top of the quality hierarchy while others emphasized the importance of personal experience.2.Who gets heard? Managing the diversity of opinion and vested interests was a challenge for the groups; service users were valued and as was the importance of fostering good working relationships between group members.3.What is the process? Group members valued debate and recognized the need to marshal discussion; most members were satisfied with the process and output. Evidence doesn't form recommendations on its own, but requires human judgement. Diversity of opinion within advisory groups was seen as key to making well-informed judgments relevant to forming recommendations. However, that diversity can bring tensions in the evaluation of evidence and its translation into practice recommendations.
    Implementation Science 09/2013; 8(1):101. DOI:10.1186/1748-5908-8-101 · 4.12 Impact Factor
  • Source
    • "Many elements – absence of a reminder system, reimbursement, time, awareness or outcome expectancy – contribute to adherence barriers (Cabana et al., 1999; Carlsen and Bringedal, 2011; Lugtenberg et al., 2011; Yarnall et al., 2003). In addition, the failure to reach consensus within the whole body of existing recommendations is a major concern (Burgers et al., 2003; Grol, 2001; Hutchings and Raine, 2006; McMurray and Swedberg, 2006). Beyond the overcoming organizational barriers, a better consensus between national agencies could improve adherence to clinical practice guidelines in primary care settings. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Objective: To analyze the level of agreement between recommendations on preventive services developed by Canada, France and the USA. Methods: We gathered recommendations on primary and secondary preventive services to adults up to November 3rd, 2011 from Canadian and US Task Forces, and equivalent French agencies. We excluded recommendations on immunization, long-term diseases or pregnancy. Results: Among 250 recommendations, 84 (34%) issued by a single country could not be compared; 43 (26%) of the remaining 166 were in strong agreement (strictly identical grades between advising countries); 25 of 43 resulted in a proposal to be implemented in clinical practice, two others not to be implemented in clinical practice and 16 were indeterminate about implementation. Strong agreement was more frequent for recommendations concerning history-taking and physical examination than for those concerning interventions (odds ratio (OR)=11.3, 95%CI: 1.6-241.2; p=0.04), and for recommendations concerning a high-risk population than for those concerning the general population (OR=3.1, 95%CI: 1.4-7.0; p=0.006). Agreement did not differ either according to maximum time range between recommendations' publication or according to the advising country. Conclusion: Agreement between recommendations is low particularly on those concerning non-clinical preventive services or non-high-risk individuals.
    Preventive Medicine 04/2013; 57(1). DOI:10.1016/j.ypmed.2013.03.012 · 3.09 Impact Factor
Show more