ArticlePDF Available

Fink A, Kosecoff J, Chassin M & Brook RHConsensus methods: characteristics and guidelines for use. Am J Public Health 74: 979-983

Authors:
  • Moriah Partners, USA

Abstract

Consensus methods are being used increasingly to solve problems in medicine and health. Their main purpose is to define levels of agreement on controversial subjects. Advocates suggest that, when properly employed, consensus strategies can create structured environments in which experts are given the best available information, allowing their solutions to problems to be more justifiable and credible than otherwise. This paper surveys the characteristics of several major methods (Delphi, Nominal Group, and models developed by the National Institutes of Health and Glaser) and provides guidelines for those who want to use the techniques. Among the concerns these guidelines address are selecting problems, choosing members for consensus panels, specifying acceptable levels of agreement, properly using empirical data, obtaining professional and political support, and disseminating results.
A preview of the PDF is not available
... To ensure a wide array of possible needs and voices are included, members of the Abortion Service Quality Initiative (ASQ Initiative) have led a collaborative effort with key partners, donors, and Ministries of Health to achieve this goal. This effort is modeled after other consensus driven and stakeholder inclusive methods, including the over 80 articles which have used a Delphi process or modified Delphi process to select quality indicators (19)(20)(21)(22). Specifically for the selection of quality indicators, Campbell and colleagues note that different stakeholder groups have different foci of quality (21). ...
... Specifically for the selection of quality indicators, Campbell and colleagues note that different stakeholder groups have different foci of quality (21). They and others note that a process that combines evidence with consensus can facilitate the development, visibility and validity of decisions in areas of uncertainty or incomplete evidence (20,21). This paper describes the selection of indicators to be tested for a simplified and standard suite of measures for abortion service quality. ...
Article
Full-text available
Measurement of the quality of abortion services is essential to service improvement. Currently, its measurement is not standardized, and some of the tools which exist are very long, and may deter use. To address this issue, this study describes a process used to create a new, more concise measure of abortion care quality, which was done with the end users in mind. Using a collaborative approach and engaging numerous stakeholders, we developed an approach to defining and selecting a set of indicators, to be tested against abortion outcomes of interest. Indicators were solicited from 12 abortion service provision entities, cataloged, and grouped within a theoretical framework. A resource group of over 40 participants was engaged through surveys, webinars, and one in-person meeting to provide input in prioritizing the indicators. We began with a list of over 1,000 measures, and engaged stakeholders to reduce the list to 72 indicators for testing. These indicators were supplemented with an additional 39 indicators drawn from qualitative research with clients, in order to ensure the client perspective is well represented. The selected indicators can be applied in pharmacies, facilities, or with hotlines, and for clients of surgical or medical abortion services in all countries. To ensure that the final suggested measures are most impactful for service providers, indicators will be tested against outcomes from 2,000 abortion clients in three countries. Those indicators which are well correlated with outcomes will be prioritized.
Article
Full-text available
Background Hernia repairs are a commonly performed surgical procedure and the Shouldice Repair of inguinal hernias has been well described before in terms of its outcomes. However, the purpose of this paper was to understand what experts from Shouldice Hospital consider to be the essential steps to the Shouldice Method. Methods Utilizing a Delphi Methodology, surgeons at Shouldice Hospital who are considered content experts, provided their answers on what is essential to the Shouldice Method. The median, interquartile range, and percent agreement from the responses as well as the Delphi’s overall Cronbach’s Alpha were determined. All Delphi items were ranked on a five-point Likert scale and consensus was reached when Cronbach’s Alpha was ≥ 0.8. The items from the survey that ranked as 5-completely agree or 4-partially agree by ≥ 80% of participants on the five-point Likert scale were included in the final framework. Results The final consensus for the Shouldice Method included 39 items with 7 overarching steps: preoperative preparation of the patient, anesthetic component, incision and nerve identification, treatment of the cremasteric muscles, hernia identification and treatment, reconstruction of the posterior wall, and post-operative management of patients. Conclusion The results of this consensus provide a step-by-step approach to the Shouldice Method, as well as information that is timely and can be utilized by surgeons incorporating non-mesh hernia repairs into their practice.
Article
Introdução: No contexto da Web Semântica, a interoperabilidade entre ontologias heterogêneas é um desafio devido a diversos fatores entre os quais se destacam a ambiguidade e a redundância semântica. Para superar tais desafios, adota-se sistemas/algoritmos para alinhamento de ontologias. Neste estudo, entende-se que vocabulários controlados são uma forma particular de ontologias. Objetivo: obter um vocabulário resultante do alinhamento e fusão dos vocabulários Domínios Científicos e Áreas Científicas da FCT, EuroSciVoc e UNESCO nomenclature for fields of Science and Technology, no domínio Ciências da Computação, para ser usado no âmbito do projeto IViSSEM. Metodologia: revisão da literatura sobre sistemas/algoritmos para alinhamento de ontologias, utilizando a metodologia PRISMA; alinhamento dos três vocabulários; e validação do vocabulário por meio do estudo Delphi. Resultados: procedeu-se à análise dos 25 sistemas de alinhamento de ontologias e variantes que participaram de pelo menos uma track da competição entre 2018 e 2019. Destes sistemas foram selecionados Agreement Maker Light e LogMap para realizar o alinhamento dos três vocabulários, fazendo um recorte para a área da Ciência da Computação. Conclusão: O vocabulário obtido a partir do Agreement Maker Light apresentou uma melhor performance. Ao final foi obtido o vocabulário, com 98 termos, no domínio da Ciência da Computação a ser adotado pelo projeto IViSSEM. O alinhamento resultante entre o vocabulário utilizado a nível nacional pela FCT (Portugal) com um empregue pela União Europeia (EuroSciVoc) e outro do domínio da Ciência&Tecnologia (UNESCO), estabelecendo relações proveitosas para outras universidades e projetos, bem como para a própria FCT.
Article
Full-text available
Background: CME is a radical resection for colon cancer, but the procedure is technically demanding with significant variation in its practice. A standardised approach to the optimal technique and training is, therefore, desirable to minimise technical hazards and facilitate safe dissemination. The aim is to develop an expert consensus on the optimal technique for Complete Mesocolic Excision (CME) for right-sided and transverse colon cancer to guide safe implementation and training pathways. Methods: Guidance was developed following a modified Delphi process to draw consensus from 55 international experts in CME and surgical education representing 18 countries. Domain topics were formulated and subdivided into questions pertinent to different aspects of CME practice. A three-round Delphi voting on 25 statements based on the specific questions and 70% agreement was considered as consensus. Results: Twenty-three recommendations for CME procedure were agreed on, describing the technique and optimal training pathway. CME is recommended as the standard of care resection for locally advanced colon cancer. The essential components are central vascular ligation, exposure of the superior mesenteric vein and excision of an intact mesocolon. Key anatomical landmarks to perform a safe CME dissection include identification of the ileocolic pedicle, superior mesenteric vein and root of the mesocolon. A proficiency-based multimodal training curriculum for CME was proposed including a formal proctorship programme. Conclusions: Consensus on standardisation of technique and training framework for complete mesocolic excision was agreed upon by a panel of experts to guide current practice and provide a quality control framework for future studies.
Article
Full-text available
Background Although many quality indicator (QI) sets have been developed for acute cardiovascular diseases, a comprehensive summary is lacking. In this scoping review we aimed to summarize the available evidence on the QI sets for acute cardiovascular diseases, and assess the QI set development process. We followed the Joanna Briggs Institute framework and the PRISMA extension for scoping reviews. Methods We conducted a systematic search of MEDLINE, EMBASE, and major international guidelines on QIs for acute major cardiovascular diseases. The study included articles published after 2000. Results Among the 3112 articles screened, 18 were included in this scoping review. Among the 18 articles included, 12 were on acute coronary syndrome (ACS), five on acute heart failure (AHF), and two on acute aortic dissection (AAD); one article included QIs for both ACS and AHF. Only four of these studies conducted a systematic search with a search strategy. From the 18 articles, 268 QIs containing duplication between articles were identified (191 QIs were for ACS, 57 were for AHF, and 20 were for AAD) and QI measurements varied across articles. Conclusions This scoping review provides a comprehensive list of QIs for acute cardiovascular diseases. Our results may be helpful to clinicians and organizations seeking to develop QIs for acute cardiovascular care in the future.
Article
Objective: In recent years, there are several systematic reviews published on animal experiments of Traditional Chinese medicine (TCM). PRISMA (preferred reporting items for systematic reviews and meta-analysis) guidelines provide a guarantee for significantly improving the reporting quality of systematic reviews (SRs) and meta-analysis (MAs) to a certain extent; however, there are still certain defects found in the quality of SRs/MAs of animal experiments of TCM. It has been found that especially, the descriptions of the rationale and animal characteristics of TCM interventions are inadequate. As a result, we have developed a novel reporting guideline for SRs/MAs of animal experimental in the field of TCM (PRISMA-ATCM) to overcome these problems. Methods: PRISMA-ATCM reporting guidelines were formed by analyzing both the status and quality of published SRs/MAs of animal experiments and consulting experts in the related fields, and then by Delphi consultation, consensus meeting and revision. Results: Among the 27 items on the PRISMA checklist, Title (1), Structured summary (2), Rationale (3), Objectives (4), Protocol and registration (5), Eligibility criteria (6), Data items (11), Planned methods of analysis (14), Study characteristics (18), Summary of evidence (24), Limitations (25), and Funding (27) have been extensively revised and expanded, to specifically include the details about TCM intervention and animal characteristics. In addition, illustrative examples and explanations have been provided for each item. Conclusion: PRISMA-ATCM could markedly improve the quality SRs/MAs of animal experiments in the field of TCM.
Article
Full-text available
This article presents a dataset of service design skills which service design experts value as important requirements for design team members. Purposive sampling and a chain referral approach were used to recruit appropriate experts to conduct questionnaire-based research. Using the analytical hierarchy process (AHP), pairwise skills-rating questionnaires were designed to elicit the experts’ responses. The resulting dataset was processed using AHP algorithms programmed in R programming language. The transparent data and available codes of the research may be reused by design practitioners and researchers for replication and further analysis. This paper offers a reproduceable research process and associated dataset for conducting multiple-criteria decision analysis with expert purposive sampling. Measurement(s)perspective of experts on important service design skills for design team membersTechnology Type(s)pairwise skills-rating questionnairesFactor Type(s)service design skills Measurement(s) perspective of experts on important service design skills for design team members Technology Type(s) pairwise skills-rating questionnaires Factor Type(s) service design skills
Article
Full-text available
Background Structured, systematic methods to formulate consensus recommendations, such as the Delphi process or nominal group technique, among others, provide the opportunity to harness the knowledge of experts to support clinical decision making in areas of uncertainty. They are widely used in biomedical research, in particular where disease characteristics or resource limitations mean that high-quality evidence generation is difficult. However, poor reporting of methods used to reach a consensus – for example, not clearly explaining the definition of consensus, or not stating how consensus group panellists were selected – can potentially undermine confidence in this type of research and hinder reproducibility. Our objective is therefore to systematically develop a reporting guideline to help the biomedical research and clinical practice community describe the methods or techniques used to reach consensus in a complete, transparent, and consistent manner. Methods The ACCORD (ACcurate COnsensus Reporting Document) project will take place in five stages and follow the EQUATOR Network guidance for the development of reporting guidelines. In Stage 1, a multidisciplinary Steering Committee has been established to lead and coordinate the guideline development process. In Stage 2, a systematic literature review will identify evidence on the quality of the reporting of consensus methodology, to obtain potential items for a reporting checklist. In Stage 3, Delphi methodology will be used to reach consensus regarding the checklist items, first among the Steering Committee, and then among a broader Delphi panel comprising participants with a range of expertise, including patient representatives. In Stage 4, the reporting guideline will be finalised in a consensus meeting, along with the production of an Explanation and Elaboration (E&E) document. In Stage 5, we plan to publish the reporting guideline and E&E document in open-access journals, supported by presentations at appropriate events. Dissemination of the reporting guideline, including a website linked to social media channels, is crucial for the document to be implemented in practice. Discussion The ACCORD reporting guideline will provide a set of minimum items that should be reported about methods used to achieve consensus, including approaches ranging from simple unstructured opinion gatherings to highly structured processes.
Article
Construct: The construct being assessed is readiness-for-residency of graduating medical students, as measured through two assessment frameworks. Background: Readiness-for-residency of near-graduate medical students should be but is not consistently assessed. To address this, the Association of American Medical Colleges (AAMC), in 2014, identified and described 13 core Entrustable Professional Activities (EPAs), which are tasks that all residents should be able to perform unsupervised upon entering residency. However, the AAMC did not initially provide measurement guidelines or propose standardized assessments. We designed Night-onCall (NOC), an immersive simulation for our near-graduating medical students to assess and address their readiness-for-residency, framed around tasks suggested by the AAMC's core EPAs. In adopting this EPA assessment framework, we began by building upon an established program of competency-based clinical skills assessments, repurposing competency-based checklists to measure components of the EPAs where possible, and designing new checklists, when necessary. This resulted in a blended suite of 14 checklists, which theoretically provide substantive assessment of all 13 core EPAs. In this paper, we describe the consensus-based mapping process conducted to ensure we understood the relationship between competency and EPA-based assessment lenses and could therefore report meaningful feedback on both to transitioning students in the NOC exercise. Approach: Between January-November 2017, five clinician and two non-clinician health professions educators at NYU Grossman School of Medicine conducted a rigorous consensus-based mapping process, which included each rater mapping each of the 310 NOC competency-based checklist items to lists of entrustable behaviors expected of learners according to the AAMC 13 core EPAs. Findings: All EPAs were captured to varying degrees by the 14 NOC checklists (overall Intraclass Correlation Coefficient (ICC) = 0.77). Consensus meetings resolved discrepancies and improved ICC values for three (EPA-9, EPA-10, EPA-12) of the four EPAs that initially showed poor reliability. Conclusions: Findings suggest that with some limitations (e.g., EPA-7 "form clinical questions/retrieve evidence") established competency-based assessments can be repurposed to measure readiness-for-residency through an EPA lens and both can be reported to learners and faculty.
Article
Full-text available
Background: The goal of mucoactive therapies in cystic fibrosis (CF) is to enhance sputum clearance and to reduce a progressive decline in lung function over the patient's lifetime. We aimed to investigate the level of consensus among specialists from Italian CF Centers on appropriateness of therapeutic use of dornase alfa (rhDNase) for CF patients. Method: A consensus on appropriate prescribing in CF mucoactive agents was appraised by an online Delphi method, based on a panel of 27 pulmonologists, coordinated by a Scientific Committee of six experts in medical care of patients with CF. Results: Full or very high consensus was reached on several issues related to therapeutic use of dornase alfa for CF patients in clinical practice. Conclusions: The consensus reached on a number of topics regarding use of mucoactive agents in patients with CF can help guide clinicians in daily practice based on expert experience and define the most appropriate therapeutic strategy for the individual patient.
Article
Full-text available
This article sets forth a group process approach useful for practicing administrators charged with a program development task. More specifically, meeting formats are suggested for involving the following critical reference groups in successive phases of program development: (1) clients (consumers or users) and first-line staff, in problem exploration; (2) external resource people and internal specialists, in knowledge exploration; (3) key administrators and resource controllers, in priority development and local adaptation; (4) organizational staff, in program proposal building; and (5) all constituencies, in final approval and evaluation designs. The meeting formats were suggested by contemporary small-group theory; the sequencing of the involvement of reference groups came from studies of community planning.
Article
A retrospective review is reported of progress in academic pathology since 1974, when the Association of Pathology Chairmen undertook a Delphi study of pathology chairmen's expectations and desires for the future. In the authors' view, the Delphi study was useful in alerting academic pathologists to some important opportunities and in coalescing activities toward achievement of important goals. A second study to involve a broader community of pathologists is suggested.
Article
Proper care of persons infected with isoniazid-resistant tubercle bacilli is controversial because there are few data on the risks and benefits of the preventive treatment alternatives. Decision analysis was used to facilitate a comparison of outcomes using different strategies and the Delphi technique to obtain estimates of relevant probabilities. For all probabilities of isoniazid resistance, the observation and no-drug alternative is unsatisfactory because it would result in a twofold to sevenfold greater number of tuberculosis cases than any of the drug regimens. With a low probability of isoniazid resistance, isoniazid is the preventive treatment of least cost and proved efficacy. As the probability of isoniazid resistance increases, more cases are prevented by rifampin-containing regimens, but at added cost. These findings can be used to formulate appropriate preventive treatment recommendations. (JAMA 1980;244:2736-2740)
Article
The author describes an innovative, iterative review paradigm for synthesizing the knowledge base of a given subject, leading to a state-of-the-art consensus document regarding "best practice" in that field, namely, chronic obstructive pulmonary diseases (COPD). The method consisted of bringing together a project team of eminent physicians concerned with treatment and comprehensive care of COPD patients. The project director (a psychologist) served as process facilitator and project administrator. The state-of-the-art document went through five drafts that were carefully examined by the team members before the paper was deemed "fit" for review by peers. After eight more iterative critiques by 60 respected outside reviewers, the thirteenth draft was published in the Journal of the American Medical Association (1975). Over 8,500 reprint requests have been received; the paper has been endorsed by all of the medical groups in the U.S. that have a major concern with pulmonary disease; the article (and subsequent monograph) are being used for continuing medical education; and a 45-minute documentary of the treatment-in-action has been filmed by Warner-Chilcott Laboratories.
Article
This report describes the results of an extensive set of experiments, conducted at RAND, concerned with evaluating the effectiveness of the Delphi procedures for formulating group judgements. The study represents a small beginning in the field of research that could be called ‘opinion technology’, and is of direct relevance for the use of experts as advisers in decision making, especially in areas of broad or long-range policy formulation.
Article
Cost-effectiveness studies require one to estimate overall program effectiveness using multiple outcome measures. However, one must first establish the relative importance of the different outcomes. Using an iterative process of group discussion, voting, and feedback of results, subjects ranked nine outcome scales in priority and assigned importance weights. Improvement in client ability to be self-supporting was judged the most important outcome, with symptom reduction second. Involvement with friends and substance abuse were judged the least important outcomes. The group achieved a high degree of consensus, as measured by convergence towards increasingly precise and distinct importance weights.
Article
The research described in this paper originated as part of a master plan for redevelopment and expansion of the services and facilities of an urban hospital in the western United States. Hospital officials asserted, appropriately, that no major changes in the institution should be made without first obtaining as clear a picture as possible of the nature of health care organization 10 yr hence; only in this context should changes be contemplated. Efforts to obtain such a forecast relied primarily on the Delphi process, a technique designed to obtain consensus from a panel of experts on future events and conditions. Results from four realms of prediction are reported here, for their general interest and as an illustration and critique of a useful planning technique which has wide potential application. The four realms are: predominant organization patterns of health care delivery; methods of financing health care; modes of reimbursement; and the nature and role of hospitals.