Systematic reviews and original articles differ in relevance, novelty, and use in an evidence-based service for physicians: PLUS Project

Health Information Research Unit, Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada.
Journal of Clinical Epidemiology (Impact Factor: 3.42). 05/2008; 61(5):449-54. DOI: 10.1016/j.jclinepi.2007.10.016
Source: PubMed


To describe the ratings from physicians, and use by physicians, of high quality, clinically pertinent original articles and systematic reviews from over 110 clinical journals and the Cochrane Database of Systematic Reviews (CDSRs).
Prospective observational study. Data were collected via an online clinical rating system of relevance and newsworthiness for quality-filtered clinical articles and via an online delivery service for practicing physicians, during the course of the McMaster Premium LiteratUre Service Trial. Clinical ratings of articles in the MORE system by over 1,900 physicians were compared and the usage rates over 13 months of these articles by physicians, who were not raters, were examined.
Systematic reviews were rated significantly higher than original articles for relevance (P<0.001), but significantly lower for newsworthiness (P<0.001). Reviews published in the CDSR had significantly lower ratings for both relevance (P<0.001) and newsworthiness (P<0.001) than reviews published in other journals. Participants accessed reviews more often than original articles (P<0.001), and accessed reviews from journals more often than from CDSR (P<0.001).
Physician ratings and the use of high-quality original articles and systematic reviews differed, generally favoring systematic reviews over original articles. Reviews published in journals were rated higher and accessed more often than Cochrane reviews.

0 Reads
  • Source
    • "The basis of a KT intervention is synthesis of research in the form of systematic reviews, evidence summaries or online KT tools. Although health professionals generally prefer systematic reviews to original research articles [24], they still report that systematic reviews do no always answer their clinical questions [3,25]. There is an increasing call for customized, easy to read summaries. "
    [Show abstract] [Hide abstract]
    ABSTRACT: It is difficult to foster research utilization among allied health professionals (AHPs). Tailored, multifaceted knowledge translation (KT) strategies are now recommended but are resource intensive to implement. Employers need effective KT solutions but little is known about; the impact and viability of multifaceted KT strategies using an online KT tool, their effectiveness with AHPs and their effect on evidence-based practice (EBP) decision-making behavior. The study aim was to measure the effectiveness of a multifaceted KT intervention including a customized KT tool, to change EBP behavior, knowledge, and attitudes of AHPs. This is an evaluator-blinded, cluster randomized controlled trial conducted in an Australian community-based cerebral palsy service. 135 AHPs (physiotherapists, occupational therapists, speech pathologists, psychologists and social workers) from four regions were cluster randomized (n = 4), to either the KT intervention group (n = 73 AHPs) or the control group (n = 62 AHPs), using computer-generated random numbers, concealed in opaque envelopes, by an independent officer. The KT intervention included three-day skills training workshop and multifaceted workplace supports to redress barriers (paid EBP time, mentoring, system changes and access to an online research synthesis tool). Primary outcome (self- and peer-rated EBP behavior) was measured using the Goal Attainment Scale (individual level). Secondary outcomes (knowledge and attitudes) were measured using exams and the Evidence Based Practice Attitude Scale. The intervention group's primary outcome scores improved relative to the control group, however when clustering was taken into account, the findings were non-significant: self-rated EBP behavior [effect size 4.97 (95% CI -10.47, 20.41)(p = 0.52)]; peer-rated EBP behavior [effect size 5.86 (95% CI -17.77, 29.50)(p = 0.62)]. Statistically significant improvements in EBP knowledge were detected [effect size 2.97 (95% CI 1.97, 3.97(p < 0.0001)]. Change in EBP attitudes was not statistically significant. Improvement in EBP behavior was not statistically significant after adjusting for cluster effect, however similar improvements from peer-ratings suggest behaviorally meaningful gains. The large variability in behavior observed between clusters suggests barrier assessments and subsequent KT interventions may need to target subgroups within an organization.Trial registration: Registered on the Australian New Zealand Clinical Trials Registry (ACTRN12611000529943).
    Full-text · Article · Nov 2013 · Implementation Science
  • Source
    • "This was surprising, given that free or funded free access to the Cochrane Library is widely available in many countries (but not in North America), and Cochrane reviews have been shown to be of higher quality than other reviews [35,36]. A study by McKinlay and colleagues, however, showed that even when access was provided equally to Cochrane and journal reviews, the former were less popular [37]. Some criticisms that have been levelled at Cochrane reviews that may explain this fact are that there are too many empty reviews (reviews in which only one or no randomised controlled trial are found), they lack relevance to clinical practice because of the very narrow focus of the questions, their length and complexity make them difficult to read and extract the key clinical messages, and they often lack a clear answer as to which treatment was better [38-41]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Systematic reviews are important for informing clinical practice and health policy. The aim of this study was to examine the bibliometrics of systematic reviews and to determine the amount of variance in citations predicted by the journal impact factor (JIF) alone and combined with several other characteristics. We conducted a bibliometric analysis of 1,261 systematic reviews published in 2008 and the citations to them in the Scopus database from 2008 to June 2012. Potential predictors of the citation impact of the reviews were examined using descriptive, univariate and multiple regression analysis. The mean number of citations per review over four years was 26.5 (SD +/-29.9) or 6.6 citations per review per year. The mean JIF of the journals in which the reviews were published was 4.3 (SD +/-4.2). We found that 17% of the reviews accounted for 50% of the total citations and 1.6% of the reviews were not cited. The number of authors was correlated with the number of citations (r = 0.215, P < 0.001). Higher numbers of citations were associated with the following characteristics: first author from the United States (36.5 citations), an ICD-10 chapter heading of Neoplasms (31.8 citations), type of intervention classified as Investigation, Diagnostics or Screening (34.7 citations) and having an international collaboration (32.1 citations). The JIF alone explained more than half of the variation in citations (R2 = 0.59) in univariate analysis. Adjusting for both JIF and type of intervention increased the R2 value to 0.81. Fourteen percent of reviews published in the top quartile of JIFs (>=5.16) received citations in the bottom quartile (eight or fewer), whereas 9% of reviews published in the lowest JIF quartile (<=2.06) received citations in the top quartile (34 or more). Six percent of reviews in journals with no JIF were also in the first quartile of citations. The JIF predicted over half of the variation in citations to the systematic reviews. However, the distribution of citations was markedly skewed. Some reviews in journals with low JIFs were well-cited and others in higher JIF journals received relatively few citations; hence the JIF did not accurately represent the number of citations to individual systematic reviews.
    Full-text · Article · Sep 2013 · Systematic Reviews
  • Source
    • "Our subgroup analyses will also describe differences in types of evidence accessed. Our group previously determined that physicians prefer systematic reviews to primacy studies and that access was better through peer-reviewed journals than through Cochrane Collaboration [83]; we will now extend this type of analysis to look at evidence accessed across professions by comparing the types of primary studies (clinical, basic science; quantitative/qualitative), content (prognosis, diagnosis, clinical measurement, treatment effectiveness/outcomes, economic, etiology), and journals (disciplinary/interdisciplinary/other discipline). "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background Optimizing pain care requires ready access and use of best evidence within and across different disciplines and settings. The purpose of this randomized trial is to determine whether a technology-based “push” of new, high-quality pain research to physicians, nurses, and rehabilitation and psychology professionals results in better knowledge and clinical decision making around pain, when offered in addition to traditional “pull” evidence technology. A secondary objective is to identify disciplinary variations in response to evidence and differences in the patterns of accessing research evidence. Methods Physicians, nurses, occupational/physical therapists, and psychologists (n = 670) will be randomly allocated in a crossover design to receive a pain evidence resource in one of two different ways. Evidence is extracted from medical, nursing, psychology, and rehabilitation journals; appraised for quality/relevance; and sent out (PUSHed) to clinicians by email alerts or available for searches of the accumulated database (PULL). Participants are allocated to either PULL or PUSH + PULL in a randomized crossover design. The PULL intervention has a similar interface but does not send alerts; clinicians can only go to the site and enter search terms to retrieve evidence from the cumulative and continuously updated online database. Upon entry to the trial, there is three months of access to PULL, then random allocation. After six months, crossover takes place. The study ends with a final three months of access to PUSH + PULL. The primary outcomes are uptake and application of evidence. Uptake will be determined by embedded tracking of what research is accessed during use of the intervention. A random subset of 30 participants/ discipline will undergo chart-stimulated recall to assess the nature and depth of evidence utilization in actual case management at baseline and 9 months. A different random subset of 30 participants/ discipline will be tested for their skills in accessing evidence using a standardized simulation test (final 3 months). Secondary outcomes include usage and self-reported evidence-based practice attitudes and behaviors measured at baseline, 3, 9, 15 and 18 months. Discussion The trial will inform our understanding of information preferences and behaviors across disciplines/practice settings. If this intervention is effective, sustained support will be sought from professional/health system initiatives with an interest in optimizing pain management. Trial registration Registered as NCT01348802 on
    Full-text · Article · Nov 2012 · Implementation Science
Show more