Observer bias in randomized clinical trials with binary outcomes: Systematic review of trials with both blinded and non-blinded outcome assessors

Nordic Cochrane Centre, Rigshospitalet Department 3343, Blegdamsvej 9, 2100 Copenhagen Ø, Denmark.
BMJ (online) (Impact Factor: 17.45). 02/2012; 344(3):e1119. DOI: 10.1093/ije/dyt270
Source: PubMed


We wanted to evaluate the impact of nonblinded outcome assessors on estimated treatment effects in time-to-event trials.

Systematic review of randomized clinical trials with both blinded and nonblinded assessors of the same time-to-event outcome. Two authors agreed on inclusion of trials and outcomes. We compared hazard ratios based on nonblinded and blinded assessments. A ratio of hazard ratios (RHR)<1 indicated that nonblinded assessors generated more optimistic effect estimates. We pooled RHRs with inverse variance random-effects meta-analysis.

We included 18 trials. Eleven trials (1969 patients) with subjective outcomes provided hazard ratios, RHR 0.88 (0.69 to 1.12), (I2=44%, P=0.06), but unconditional pooling was problematic because of qualitative heterogeneity. Four atypical cytomegalovirus retinitis trials compared experimental oral administration with control intravenous administration of the same drug, resulting in bias favouring the control intervention, RHR 1.33 (0.98 to 1.82). Seven trials of cytomegalovirus retinitis, tibial fracture and multiple sclerosis compared experimental interventions with standard control interventions, e.g. placebo, no-treatment or active control, resulting in bias favouring the experimental intervention, RHR 0.73 (0.57 to 0.93), indicating an average exaggeration of nonblinded hazard ratios by 27% (7% to 43%).

Lack of blinded outcome assessors in randomized trials with subjective time-to-event outcomes causes high risk of observer bias. Nonblinded outcome assessors typically favour the experimental intervention, exaggerating the hazard ratio by an average of approximately 27%; but in special situations, nonblinded outcome assessors favour control interventions, inducing a comparable degree of observer bias in the reversed direction.

Download full-text


Available from: Ann Sofia Skou Thomsen, May 19, 2015
  • Source
    • "When intraand inter-individual variability are low, and sample size relatively small (under 100 in the Gensburger example cited below), blinding may have little effect on study outcome (Gensburger et al., 2012). However, it has been shown that blinding can affect outcome, even with binary outcomes as determined from a meta-analysis of a large population sample (over 4000 in the example cited; Hróbjartsson et al., 2012). There is, additionally, the famous case of Benveniste who published a paper in Nature (which was never retracted) that identified an apparent activity of antibodies at a 'concentration' below that necessary to provide a single molecule in the solutions used in the experiment (Davenas et al., 1988). "
    [Show abstract] [Hide abstract]
    ABSTRACT: This Editorial is part of a series. To view the other Editorials in this series, visit:;; and © 2015 The British Pharmacological Society.
    British Journal of Pharmacology 07/2015; 172(14):3461-3471. DOI:10.1111/bph.12856 · 4.84 Impact Factor
  • Source
    • "We summarized the impact of nonblinded outcome assessment as the ratio of odds ratios (ROR 5 OR nonblind /OR blind ). The standard error of ROR was calculated as the square root of the sum of the variances of the two ORs [16] [23]. An ROR !1 indicates that nonblinded assessors exaggerate the benefit of the intervention. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Objectives: To examine the impact of not blinding outcome assessors on estimates of intervention effects in animal experiments modeling human clinical conditions. Study design and setting: We searched PubMed, Biosis, Google Scholar, and HighWire Press and included animal model experiments with both blinded and nonblinded outcome assessors. For each experiment, we calculated the ratio of odds ratios (ROR), that is, the odds ratio (OR) from nonblinded assessments relative to the corresponding OR from blinded assessments. We standardized the ORs according to the experimental hypothesis, such that an ROR <1 indicates that nonblinded assessor exaggerated intervention effect, that is, exaggerated benefit in experiments investigating possible benefit or exaggerated harm in experiments investigating possible harm. We pooled RORs with inverse variance random-effects meta-analysis. Results: We included 10 (2,450 animals) experiments in the main meta-analysis. Outcomes were subjective in most experiments. The pooled ROR was 0.41 (95% confidence interval [CI], 0.20, 0.82; I(2) = 75%; P < 0.001), indicating an average exaggeration of the nonblinded ORs by 59%. The heterogeneity was quantitative and caused by three pesticides experiments with very large observer bias, pooled ROR was 0.20 (95% CI, 0.07, 0.59) in contrast to the pooled ROR in the other seven experiments, 0.82 (95% CI, 0.57, 1.17). Conclusion: Lack of blinding of outcome assessors in animal model experiments with subjective outcomes implies a considerable risk of observer bias.
    Journal of Clinical Epidemiology 06/2014; 67(9). DOI:10.1016/j.jclinepi.2014.04.008 · 3.42 Impact Factor
  • Source
    • "First, although the trial was designed in order to minimize the risk of bias [58-60] and the risk of random errors [58], most outcomes are at risk of being assessed with some bias, as only the secondary outcome measure (EDE) is possible to blind [58-60]. Previous research has indicated that non-blinded assessors tended to be more optimistic about patient outcome when compared to blinded assessors [61]. Blinding is thus used whenever possible and data will be analyzed according to the intention-to-treat principle. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Continuous feedback on patient improvement and the therapeutic alliance may reduce the number of dropouts and increase patient outcome. There are, however, only three published randomized trials on the effect of feedback on the treatment of eating disorders, showing inconclusive results, and there are no randomized trials on the effect of feedback in group therapy. Accordingly the current randomized clinical trial, initiated in September 2012 at the outpatient clinic for eating disorders at Stolpegaard Psychotherapy Centre, aims to investigate the impact of continuous feedback on attendance and outcome in group psychotherapy.Methods/design: The hypothesis is that continuous feedback to both patient and therapist on treatment progress and alliance will increase attendance and treatment outcome. The trial is set up using a randomized design with a minimum of 128 patients allocated to either an experimental or control group at a ratio of 1:1. The experimental group will receive standard treatment (systemic and narrative group psychotherapy) with feedback intervention, whereas the control group will receive standard treatment only. The participants are diagnosed with bulimia nervosa binge eating disorder, or an eating disorder not otherwise specified, according to the DSM-IV. In the experimental group feedback to the participants, based on the Outcome Rating Scale (ORS) and the Group Session Rating Scale (GSRS), is actively added to standard treatment. The ORS assesses areas of life functioning known to change as a result of therapeutic intervention. The GSRS assesses key dimensions of effective therapeutic relationships. In the control group, the patients fill out the Outcome Rating Scale only, and feedback is not provided.The primary outcome is the rate of attendance to treatment sessions. The secondary outcome is the severity of eating disorder symptoms. Exploratory outcomes are the level of psychological and social functioning, and suicide or self-harm. This is measured with the ORS, Symptom Check List, WHO-Five Wellbeing Index, Sheehan Disability Scale and a modified version of the Self-Harm Inventory. If the results will confirm the hypothesis, this trial will support feedback as a way to improve group treatment attendance for outpatients with eating disorders.Trial registration: identifier: NCT01693237.
    Trials 04/2014; 15(1):138. DOI:10.1186/1745-6215-15-138 · 1.73 Impact Factor
Show more