Tools for Assessing Quality and Susceptibility to Bias in Observational Studies in Epidemiology: a Systematic Review and Annotated Bibliography

Primary Care Genetics, General Practice and Primary Care Research Unit, University of Cambridge and Public Health Genetics Unit, Strangeways Research Labs, Worts Causeway, Cambridge, UK.
International Journal of Epidemiology (Impact Factor: 9.18). 07/2007; 36(3):666-76. DOI: 10.1093/ije/dym018
Source: PubMed


Assessing quality and susceptibility to bias is essential when interpreting primary research and conducting systematic reviews and meta-analyses. Tools for assessing quality in clinical trials are well-described but much less attention has been given to similar tools for observational epidemiological studies.
Tools were identified from a search of three electronic databases, bibliographies and an Internet search using Google. Two reviewers extracted data using a pre-piloted extraction form and strict inclusion criteria. Tool content was evaluated for domains potentially related to bias and was informed by the STROBE guidelines for reporting observational epidemiological studies.
A total of 86 tools were reviewed, comprising 41 simple checklists, 12 checklists with additional summary judgements and 33 scales. The number of items ranged from 3 to 36 (mean 13.7). One-third of tools were designed for single use in a specific review and one-third for critical appraisal. Half of the tools provided development details, although most were proposed for future use in other contexts. Most tools included items for selection methods (92%), measurement of study variables (86%), design-specific sources of bias (86%), control of confounding (78%) and use of statistics (78%); only 4% addressed conflict of interest. The distribution and weighting of domains across tools was variable and inconsistent.
A number of useful assessment tools have been identified by this report. Tools should be rigorously developed, evidence-based, valid, reliable and easy to use. There is a need to agree on critical elements for assessing susceptibility to bias in observational epidemiology and to develop appropriate evaluation tools.

Download full-text


Available from: Iain Tatt, Jul 24, 2014
1 Follower
952 Reads
  • Source
    • "Three reviewers (A.B., A.C., and D.G.) independently evaluated the quality of the methodology used in studies included in the systematic review. The risk of bias of observational studies was assessed by a modified checklist of essential items stated in Strengthening the Reporting of Observational Studies in Epidemiology and in Fowkes and Sanderson [16] [17] [18] [19] [20] "
    [Show abstract] [Hide abstract]
    ABSTRACT: Objective: The objective of this study was to evaluate the burden of malaria in Latin America and the Caribbean countries through a systematic review and meta-analysis of published literature, gray literature, and information from countries' public health authorities for the period 1990 to 2009. Methods: The random-effects meta-analysis of the prospective studies, carried out in very highly endemic areas, showed an annual incidence rate of 409.0 malaria episodes/1000 person-years (95% confidence interval [CI] 263.1-554.9), considering all ages, which was 40-fold the one estimated from areas with passive surveillance only. Results: Overall, the most prevalent species was Plasmodium vivax (77.5%; 95% CI 75.6-79.4) followed by Plasmodium falciparum (20.8%; 95% CI 19.0-22.6) and Plasmodium malariae (0.08%; 95% CI 0.07-0.010). Data from regional ministries of health yielded an estimated pooled crude annual mortality rate of 6 deaths/100,000 people, mainly associated with P. falciparum. Conclusion: This study represents the first systematic review of the burden of malaria in Latin America and the Caribbean, with data from 21 countries. © 2015 International Society for Pharmacoeconomics and Outcomes Research (ISPOR).
  • Source
    • "A 134 measure of study quality can also be included in these forms to assess the quality of evidence 135 from each study. There are more than 80 tools available to assess the quality and risk of bias in 136 observational studies (Sanderson et al., 2007) reflecting the diversity of research approaches 137 between fields. These tools usually include an assessment of how dependent variables were 138 measured, appropriate selection of participants, and appropriate control for confounding factors. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Meta-analysis synthesizes a body of research investigating a common research question. Outcomes from meta-analyses provide a more objective and transparent summary of a research area than traditional narrative reviews. Moreover, they are often used to support research grant applications, guide clinical practice and direct health policy. The aim of this article is to provide a practical and nontechnical guide for psychological scientists that outlines the steps involved in planning and performing a meta-analysis of correlational datasets. I provide a supplementary R script to demonstrate each analytical step described in the paper, which is readily adaptable for researchers to use for their analyses. While the worked example is the analysis of a correlational dataset, the general meta-analytic process described in this paper is applicable for all types of effect sizes. I also emphasise the importance of meta-analysis protocols and pre-registration to improve transparency and help avoid unintended duplication. An improved understanding this tool will not only help scientists to conduct their own meta-analyses but also improve their evaluation of published meta-analyses.
    Frontiers in Psychology 09/2015; DOI:10.3389/fpsyg.2015.01549 · 2.80 Impact Factor
  • Source
    • "Greater awareness, consistency, and acceptance of specific quality assessment tools are needed across SR types. included items related to study variables (86%), designrelated bias (86%), and confounding (78%), although certain aspects such as conflict of interest were underrepresented (4%) [5]. Similar results have been identified within SRs concerning the epidemiology of chronic disease with only 55% of reviews referring to quality assessment of primary studies overall [6]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: To assess the use of quality assessment tools among a cross-section of systematic reviews (SRs) and to further evaluate whether quality was used as a parameter in the decision to include primary studies within subsequent meta-analysis. We searched PubMed for systematic reviews (interventional, observational and diagnostic) published in Core Clinical Journals between January 1st and March 31(st), 2014. 310 systematic reviews were identified. Quality assessment was undertaken in 223 (71.9%) with isolated use of the Cochrane risk of bias tool (26%, n= 58) and the Newcastle Ottawa Scale (15.3%, n= 34) most common. A threshold level of primary study quality for subsequent meta-analysis was used in 13.2% (41/310) of reviews. Overall, fifty-four combinations of quality assessment tools were identified with a similar preponderance of tools used among observational and interventional reviews. Multiple tools were used in 11.6% (n= 36) of SRs overall. We found that quality assessment tools were used in a majority of SRs; however, a threshold level of quality for meta-analysis was stipulated in just 13.2% (n= 41). This cross-sectional analysis provides further evidence of the need for more active or intuitive editorial processes to enhance the reporting of systematic reviews. Copyright © 2015 Elsevier Inc. All rights reserved.
    Journal of clinical epidemiology 07/2015; DOI:10.1016/j.jclinepi.2015.06.023 · 3.42 Impact Factor
Show more