Tools for Assessing Quality and Susceptibility to Bias in Observational Studies in Epidemiology: a Systematic Review and Annotated Bibliography

Primary Care Genetics, General Practice and Primary Care Research Unit, University of Cambridge and Public Health Genetics Unit, Strangeways Research Labs, Worts Causeway, Cambridge, UK.
International Journal of Epidemiology (Impact Factor: 9.18). 07/2007; 36(3):666-76. DOI: 10.1093/ije/dym018
Source: PubMed


Assessing quality and susceptibility to bias is essential when interpreting primary research and conducting systematic reviews and meta-analyses. Tools for assessing quality in clinical trials are well-described but much less attention has been given to similar tools for observational epidemiological studies.
Tools were identified from a search of three electronic databases, bibliographies and an Internet search using Google. Two reviewers extracted data using a pre-piloted extraction form and strict inclusion criteria. Tool content was evaluated for domains potentially related to bias and was informed by the STROBE guidelines for reporting observational epidemiological studies.
A total of 86 tools were reviewed, comprising 41 simple checklists, 12 checklists with additional summary judgements and 33 scales. The number of items ranged from 3 to 36 (mean 13.7). One-third of tools were designed for single use in a specific review and one-third for critical appraisal. Half of the tools provided development details, although most were proposed for future use in other contexts. Most tools included items for selection methods (92%), measurement of study variables (86%), design-specific sources of bias (86%), control of confounding (78%) and use of statistics (78%); only 4% addressed conflict of interest. The distribution and weighting of domains across tools was variable and inconsistent.
A number of useful assessment tools have been identified by this report. Tools should be rigorously developed, evidence-based, valid, reliable and easy to use. There is a need to agree on critical elements for assessing susceptibility to bias in observational epidemiology and to develop appropriate evaluation tools.

Download full-text


Available from: Iain Tatt, Jul 24, 2014
  • Source
    • "Three reviewers (A.B., A.C., and D.G.) independently evaluated the quality of the methodology used in studies included in the systematic review. The risk of bias of observational studies was assessed by a modified checklist of essential items stated in Strengthening the Reporting of Observational Studies in Epidemiology and in Fowkes and Sanderson [16] [17] [18] [19] [20] "
    [Show abstract] [Hide abstract]
    ABSTRACT: Objective: The objective of this study was to evaluate the burden of malaria in Latin America and the Caribbean countries through a systematic review and meta-analysis of published literature, gray literature, and information from countries' public health authorities for the period 1990 to 2009. Methods: The random-effects meta-analysis of the prospective studies, carried out in very highly endemic areas, showed an annual incidence rate of 409.0 malaria episodes/1000 person-years (95% confidence interval [CI] 263.1-554.9), considering all ages, which was 40-fold the one estimated from areas with passive surveillance only. Results: Overall, the most prevalent species was Plasmodium vivax (77.5%; 95% CI 75.6-79.4) followed by Plasmodium falciparum (20.8%; 95% CI 19.0-22.6) and Plasmodium malariae (0.08%; 95% CI 0.07-0.010). Data from regional ministries of health yielded an estimated pooled crude annual mortality rate of 6 deaths/100,000 people, mainly associated with P. falciparum. Conclusion: This study represents the first systematic review of the burden of malaria in Latin America and the Caribbean, with data from 21 countries. © 2015 International Society for Pharmacoeconomics and Outcomes Research (ISPOR).
  • Source
    • "This informed a decision of low, unclear, or high risk of bias in seven areas: population, predictor measurement/classification , outcome measurement/classification, attrition, analysis, reporting, and confounding. This centers on the Cochrane Collaboration's bias assessment tool (Higgins & Altman, 2008), which meant quality assessment was based on key areas of bias instead of a points system (Sanderson et al., 2007; Stroup et al., 2000). Systematic error within a study (defined as substantial error in methodology or analysis which undermines the study's findings) was also explored. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Protective factors against the victimization of children and young people within the school and community environments (extrafamilial victimization) have received less attention than risk factors. To date, there has been no systematic review on protective factors. This systematic review therefore aimed to synthesize the prospective longitudinal research findings on the protective factors against extrafamilial victimization. A systematic search of multiple sources led to the identification of 19,053 studies. Following application of a predefined inclusion and quality assessment criteria, 13 studies exploring protective factors against peer victimization and exposure to violence were included in this review. Across these studies, 19 protective factors were explored: 9 individual factors and 10 contextual factors. Four studies also explored the impact of mediating and moderating variables on the relationship between predictors and extrafamilial victimization. Findings highlight inconsistencies in the definition and measurement of victimization, along with bias in a number of areas. Nevertheless, a small number of variables (perceptions of social competence, physical strength, and aggression) were identified as potential protective factors against extrafamilial victimization. Additionally, mediating and moderating variables were identified, and the interaction between individual and contextual protective and risk factors were highlighted. These findings are explored under the theoretical framework of the ecological systems theory and their practical and research-based implications are discussed.
    Trauma Violence & Abuse 10/2015; DOI:10.1177/1524838015611675 · 3.27 Impact Factor
  • Source
    • "A measure of study quality can also be included in these forms to assess the quality of evidence from each study. There are more than 80 tools available to assess the quality and risk of bias in observational studies (Sanderson et al., 2007) reflecting the diversity of research approaches between fields. These tools usually include an assessment of how dependent variables were measured, appropriate selection of participants, and appropriate control for confounding factors. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Meta-analysis synthesizes a body of research investigating a common research question. Outcomes from meta-analyses provide a more objective and transparent summary of a research area than traditional narrative reviews. Moreover, they are often used to support research grant applications, guide clinical practice and direct health policy. The aim of this article is to provide a practical and nontechnical guide for psychological scientists that outlines the steps involved in planning and performing a meta-analysis of correlational datasets. I provide a supplementary R script to demonstrate each analytical step described in the paper, which is readily adaptable for researchers to use for their analyses. While the worked example is the analysis of a correlational dataset, the general meta-analytic process described in this paper is applicable for all types of effect sizes. I also emphasise the importance of meta-analysis protocols and pre-registration to improve transparency and help avoid unintended duplication. An improved understanding this tool will not only help scientists to conduct their own meta-analyses but also improve their evaluation of published meta-analyses.
    Frontiers in Psychology 09/2015; DOI:10.3389/fpsyg.2015.01549 · 2.80 Impact Factor
Show more