Systems for grading the quality of evidence and the strength of recommendations II: Pilot study of a new system

Center for Practice and Technology Assessment, Agency for Healthcare Research and Quality, 540 Gaither Rd. Rokville, MD 20852, USA. <>
BMC Health Services Research (Impact Factor: 1.66). 04/2005; 5(1):25. DOI: 10.1186/1472-6963-5-25
Source: PubMed

ABSTRACT Systems that are used by different organisations to grade the quality of evidence and the strength of recommendations vary. They have different strengths and weaknesses. The GRADE Working Group has developed an approach that addresses key shortcomings in these systems. The aim of this study was to pilot test and further develop the GRADE approach to grading evidence and recommendations.
A GRADE evidence profile consists of two tables: a quality assessment and a summary of findings. Twelve evidence profiles were used in this pilot study. Each evidence profile was made based on information available in a systematic review. Seventeen people were given instructions and independently graded the level of evidence and strength of recommendation for each of the 12 evidence profiles. For each example judgements were collected, summarised and discussed in the group with the aim of improving the proposed grading system. Kappas were calculated as a measure of chance-corrected agreement for the quality of evidence for each outcome for each of the twelve evidence profiles. The seventeen judges were also asked about the ease of understanding and the sensibility of the approach. All of the judgements were recorded and disagreements discussed.
There was a varied amount of agreement on the quality of evidence for the outcomes relating to each of the twelve questions (kappa coefficients for agreement beyond chance ranged from 0 to 0.82). However, there was fair agreement about the relative importance of each outcome. There was poor agreement about the balance of benefits and harms and recommendations. Most of the disagreements were easily resolved through discussion. In general we found the GRADE approach to be clear, understandable and sensible. Some modifications were made in the approach and it was agreed that more information was needed in the evidence profiles.
Judgements about evidence and recommendations are complex. Some subjectivity, especially regarding recommendations, is unavoidable. We believe our system for guiding these complex judgements appropriately balances the need for simplicity with the need for full and transparent consideration of all important issues.

Download full-text


Available from: Signe Agnes Flottorp, Jul 03, 2015
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: There is evidence that group cognitive behavioural therapy for psychosis (CBTp) is an effective treatment, but much of this research has been conducted with outpatient populations. The aim of this review was to determine the utility of group CBTp for inpatients. We systematically searched Scopus, Web of Science and EBSCO electronic databases to identify relevant research. We reviewed the resulting articles and included those which had been conducted with inpatients, with symptoms of psychosis, using cognitive behaviour therapy, delivered in a group format. Fourteen articles relating to ten studies were identified. Two were randomized controlled trials; two were cohort studies and the rest were pre-/post-intervention studies. There was considerable heterogeneity between the studies and all had methodological limitations. The findings suggest positive trends towards the reduction of distress associated with psychotic symptoms, increased knowledge of symptoms, decreased affective symptoms and reduced readmissions over several years. However, there is currently not enough evidence to draw any strong conclusions regarding the utility of group CBTp for inpatients due to the small number of studies and limitations in quality and generalizability. Therefore, this review indicates the need for further research, particularly large, methodologically rigorous, randomized controlled trials.
    The Cognitive Behaviour Therapist 01/2015; 8. DOI:10.1017/S1754470X15000021
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Determination of the exact criteria for resectability in patients with cholangiocarcinoma and how they are most efficiently evaluated has many limitations. Among many factors taken into account in this decision-making process are: the condition of the patient, the biology of the disease, and the technical expertise of the surgeon and hospital. An attempt is made here to organize recommendations for the work-up of patients and the main criteria for resectability as best possible, keeping in mind that there will always be some limited room for exceptions, especially if the biology is favorable. Work-up and determination of resectability for patients with distal cholangiocarcinoma are more straightforward than at the other two sites of the disease (perihilar and peripheral). In general, these follow the same principles as those for other periampullary carcinomas (pancreas, ampullary, and duodenal). The work-up and determination of resectability for patients with peripheral cholangiocarcinoma can be relatively straightforward if the lesion is away from the hilus of the liver and does not involve a significant proportion of parenchyma, but can be problematic if it is more central or very large. Patients with perihilar cholangiocarcinomas are perhaps the most challenging, as factors such as patient condition, biology of the disease, local involvement of the major vessels and bile ducts at the hilum, and the future liver remnant all have a bearing in the decision-making process.
    HPB 02/2008; 10(2):122-5. DOI:10.1080/13651820801993540 · 2.05 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Evidence-based medicine, systematic reviews, and guidelines are part of modern interventional pain management. As in other specialties in the United States, evidence-based medicine appears to motivate the search for answers to numerous questions related to costs and quality of health care as well as access to care. Scientific, relevant evidence is essential in clinical care, policy-making, dispute resolution, and law. Consequently, evidence based practice brings together pertinent, trustworthy information by systematically acquiring, analyzing, and transferring research findings into clinical, management, and policy arenas. In the United States, researchers, clinicians, professional organizations, and government are looking for a sensible approach to health care with practical evidence-based medicine. All modes of evidence-based practice, either in the form of evidence-based medicine, systematic reviews, meta-analysis, or guidelines, evolve through a methodological, rational accumulation, analysis, and understanding of the evidentiary knowledge that can be applied in clinical settings. Historically, evidence-based medicine is traceable to the 1700s, even though it was not explicitly defined and advanced until the late 1970s and early 1980s. Evidence-based medicine was initially called "critical appraisal" to describe the application of basic rules of evidence as they evolve into application in daily practices. Evidence-based medicine is defined as a conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. Evidence-based practice is defined based on 4 basic and important contingencies, which include recognition of the patient's problem and construction of a structured clinical question, thorough search of medical literature to retrieve the best available evidence to answer the question, critical appraisal of all available evidence, and integration of the evidence with all aspects and contexts of the clinical circumstances. Systematic reviews provide the application of scientific strategies that limit bias by the systematic assembly, critical appraisal, and synthesis of all relevant studies on a specific topic. While systematic reviews are close to meta-analysis, they are vastly different from narrative reviews and health technology assessments. Clinical practice guidelines are systematically developed statements that aim to help physicians and patients reach the best health care decisions. Appropriately developed guidelines incorporate validity, reliability, reproducibility, clinical applicability and flexibility, clarity, development through a multidisciplinary process, scheduled reviews, and documentation. Thus, evidence-based clinical practice guidelines represent statements developed to improve the quality of care, patient access, treatment outcomes, appropriateness of care, efficiency and effectiveness and achieve cost containment by improving the cost benefit ratio. Part 1 of this series in evidence-based medicine, systematic reviews, and guidelines in interventional pain management provides an introduction and general considerations of these 3 aspects in interventional pain management.
    Pain physician 11(2):161-86. · 4.77 Impact Factor