Article

Citation networks of related trials are often disconnected: Implications for bidirectional citation searches

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... The studies typically evaluate the effectiveness of the tools used to undertake the search methods. Nine studies assessing the use of citation chasing were included [1][2][3][19][20][21][22][23][24]. ...
... Four studies claimed that an advantage of citation chasing is that it is not limited by keywords or indexing as is bibliographic database searching [2,3,20,21]. Accordingly, four studies claimed the following advantages: Robinson et al. claimed that a small initial number of studies can create a network [21]; Hinde et al. claimed that citation searching can help inform researchers of parallel topics that may be missed by the focus of bibliographic database searches [2]; Janssens and Gwinn claimed that citation searching may be valuable in topic areas where there is no consistent terminology, so searches focus on links between studies rather than keywords [20]; and Papaioannou et al. reported that citation searching facilitated 'serendipitous study identification' due to the unstructured nature of citations [3]. ...
... Four studies claimed that an advantage of citation chasing is that it is not limited by keywords or indexing as is bibliographic database searching [2,3,20,21]. Accordingly, four studies claimed the following advantages: Robinson et al. claimed that a small initial number of studies can create a network [21]; Hinde et al. claimed that citation searching can help inform researchers of parallel topics that may be missed by the focus of bibliographic database searches [2]; Janssens and Gwinn claimed that citation searching may be valuable in topic areas where there is no consistent terminology, so searches focus on links between studies rather than keywords [20]; and Papaioannou et al. reported that citation searching facilitated 'serendipitous study identification' due to the unstructured nature of citations [3]. ...
Article
Full-text available
Background The purpose and contribution of supplementary search methods in systematic reviews is increasingly acknowledged. Numerous studies have demonstrated their potential in identifying studies or study data that would have been missed by bibliographic database searching alone. What is less certain is how supplementary search methods actually work, how they are applied, and the consequent advantages, disadvantages and resource implications of each search method. The aim of this study is to compare current practice in using supplementary search methods with methodological guidance. Methods Four methodological handbooks in informing systematic review practice in the UK were read and audited to establish current methodological guidance. Studies evaluating the use of supplementary search methods were identified by searching five bibliographic databases. Studies were included if they (1) reported practical application of a supplementary search method (descriptive) or (2) examined the utility of a supplementary search method (analytical) or (3) identified/explored factors that impact on the utility of a supplementary method, when applied in practice. Results Thirty-five studies were included in this review in addition to the four methodological handbooks. Studies were published between 1989 and 2016, and dates of publication of the handbooks ranged from 1994 to 2014. Five supplementary search methods were reviewed: contacting study authors, citation chasing, handsearching, searching trial registers and web searching. Conclusions There is reasonable consistency between recommended best practice (handbooks) and current practice (methodological studies) as it relates to the application of supplementary search methods. The methodological studies provide useful information on the effectiveness of the supplementary search methods, often seeking to evaluate aspects of the method to improve effectiveness or efficiency. In this way, the studies advance the understanding of the supplementary search methods. Further research is required, however, so that a rational choice can be made about which supplementary search strategies should be used, and when.
... While intuitive, tracking citations is considered inefficient and inaccurate, even as a complement to keyword searching [4,5]. Tracking citations can only find articles that are connected in a single citation network [6]. A review of 259 meta-analyses, in which researchers aimed to retrieve all published articles on a specific topic, showed that this occurred in less than half (46%) of cases. ...
... A review of 259 meta-analyses, in which researchers aimed to retrieve all published articles on a specific topic, showed that this occurred in less than half (46%) of cases. In 39% of the meta-analyses, the articles were in two disconnected citation networks and in 15% of the meta-analyses in three or more networks [6]. ...
Article
Full-text available
Background: We recently developed CoCites, a citation-based search method that is designed to be more efficient than traditional keyword-based methods. The method begins with identification of one or more highly relevant publications (query articles) and consists of two searches: the co-citation search, which ranks publications on their co-citation frequency with the query articles, and the citation search, which ranks publications on frequency of all citations that cite or are cited by the query articles. Methods: We aimed to reproduce the literature searches of published systematic reviews and meta-analyses and assess whether CoCites retrieves all eligible articles while screening fewer titles. Results: A total of 250 reviews were included. CoCites retrieved a median of 75% of the articles that were included in the original reviews. The percentage of retrieved articles was higher (88%) when the query articles were cited more frequently and when they had more overlap in their citations. Applying CoCites to only the highest-cited article yielded similar results. The co-citation and citation searches combined were more efficient when the review authors had screened more than 500 titles, but not when they had screened less. Conclusions: CoCites is an efficient and accurate method for finding relevant related articles. The method uses the expert knowledge of authors to rank related articles, does not depend on keyword selection and requires no special expertise to build search queries. The method is transparent and reproducible.
... While intuitive, tracking citations is considered inefficient and inaccurate, even as a complement to keyword searching [4 5]. Tracking citations can only find articles that are connected in a single citation network [6]. A review of 259 meta-analyses, in which researchers aimed to retrieve all published articles on a specific topic, showed that this occurred in less than half (46%) of cases. ...
... A review of 259 meta-analyses, in which researchers aimed to retrieve all published articles on a specific topic, showed that this occurred in less than half (46%) of cases. In 39% of the meta-analyses, the articles were in two disconnected citation networks and in 15% of the meta-analyses in three or more networks [6]. ...
Preprint
Full-text available
Objective: We recently developed CoCites, a citation-based search method that is designed to be more efficient than traditional keyword-based methods. The method begins with identification of one or more highly relevant publications (query articles) and consists of two searches: the co-citation search, which ranks publications on their co-citation frequency with the query articles, and the citation search, which ranks publications on frequency of all citations that cite or are cited by the query articles. Materials and Methods : We aimed to reproduce the literature searches of published systematic reviews and meta-analyses (n=250) and assess whether CoCites retrieves all eligible articles while screening fewer titles. Results : CoCites retrieved a median of 75% of the articles that were included in the original reviews. The percentage of retrieved articles was higher (88%) when the query articles were cited more frequently and when they had more overlap in their citations. Applying CoCites to only the highest-cited article yielded similar results. The co-citation and citation searches combined were more efficient when the review authors had screened more than 500 titles, but not when they had screened less. Discussion: CoCites uses the expert knowledge of authors to rank related articles. The method does not depend on keyword selection and requires no special expertise to build search queries. The method is transparent and reproducible. Conclusion : CoCites is an efficient and accurate method for finding relevant related articles.
... CCIR could allow users to also use approaches developed in scientometric research, such as science maps (Chen, 2017), cluster labeling (Sjögårde et al., 2021), and visualization software (van Eck & Waltman, 2010). CCIR offers two potential benefits over other citation-based IR methods: 1) it is less hindered by documents that cite the relevant literature poorly (Robinson et al., 2014) and 2) it communicates the topic structure of a document corpus, including the relative size of different topics and the relations between topics (Pirolli et al., 1996). ...
Preprint
Full-text available
The field of scientometrics has shown the power of citation-based clusters for literature analysis, yet this technique has barely been used for information retrieval tasks. This work evaluates the performance of citation based-clusters for information retrieval tasks. We simulated a search process using these clusters with a tree hierarchy of clusters and a cluster selection algorithm. We evaluated the task of finding the relevant documents for 25 systematic reviews. Our evaluation considered several trade-offs between recall and precision for the cluster selection, and we also replicated the Boolean queries self-reported by the systematic review to serve as a reference. We found that citation-based clusters search performance is highly variable and unpredictable, that it works best for users that prefer recall over precision at a ratio between 2 and 8, and that when used along with query-based search they complement each other, including finding new relevant documents.
... For our author population, µ was set to 3 and d was set to 3. These parameters were chosen to ensure that authors had multiple-but few-connected components in their final estimates of the field [28]. ...
Preprint
Full-text available
In multiple academic disciplines, having a perceived gender of `woman' is associated with a lower than expected rate of citations. In some fields, that disparity is driven primarily by the citations of men and is increasing over time despite increasing diversification of the profession. It is likely that complex social interactions and individual ideologies shape these disparities. Computational models of select factors that reproduce empirical observations can help us understand some of the minimal driving forces behind these complex phenomena and therefore aid in their mitigation. Here, we present a simple agent-based model of citation practices within academia, in which academics generate citations based on three factors: their estimate of the collaborative network of the field, how they sample that estimate, and how open they are to learning about their field from other academics. We show that increasing homophily -- or the tendency of people to interact with others more like themselves -- in these three domains is sufficient to reproduce observed biases in citation practices. We find that homophily in sampling an estimate of the field influences total citation rates, and openness to learning from new and unfamiliar authors influences the change in those citations over time. We next model a real-world intervention -- the citation diversity statement -- which has the potential to influence both of these parameters. We determine a parameterization of our model that matches the citation practices of academics who use the citation diversity statement. This parameterization paired with an openness to learning from many new authors can result in citation practices that are equitable and stable over time. Ultimately, our work underscores the importance of homophily in shaping citation practices and provides evidence that specific actions may mitigate biased citation practices in academia.
... We may have missed some relevant studies by using citation analysis as the searching method. However, the accuracy rate of citation analysis has been found to be acceptable [35,36]. For instance, using this technique, Janssens and Gwinn identified 94% [75-100%] of all articles included in 10 systematic reviews that were originally used the conventional search strategy [35]. ...
Article
Full-text available
Background: Despite the established interest in evidence-based practice (EBP) as a core competence for clinicians, evidence for how best to teach and evaluate EBP remains weak. We sought to systematically assess coverage of the five EBP steps, review the outcome domains measured, and assess the properties of the instruments used in studies evaluating EBP educational interventions. Methods: We conducted a systematic review of controlled studies (i.e. studies with a separate control group) which had investigated the effect of EBP educational interventions. We used citation analysis technique and tracked the forward and backward citations of the index articles (i.e. the systematic reviews and primary studies included in an overview of the effect of EBP teaching) using Web of Science until May 2017. We extracted information on intervention content (grouped into the five EBP steps), and the outcome domains assessed. We also searched the literature for published reliability and validity data of the EBP instruments used. Results: Of 1831 records identified, 302 full-text articles were screened, and 85 included. Of these, 46 (54%) studies were randomised trials, 51 (60%) included postgraduate level participants, and 63 (75%) taught medical professionals. EBP Step 3 (critical appraisal) was the most frequently taught step (63 studies; 74%). Only 10 (12%) of the studies taught content which addressed all five EBP steps. Of the 85 studies, 52 (61%) evaluated EBP skills, 39 (46%) knowledge, 35 (41%) attitudes, 19 (22%) behaviours, 15 (18%) self-efficacy, and 7 (8%) measured reactions to EBP teaching delivery. Of the 24 instruments used in the included studies, 6 were high-quality (achieved ≥3 types of established validity evidence) and these were used in 14 (29%) of the 52 studies that measured EBP skills; 14 (41%) of the 39 studies that measured EBP knowledge; and 8 (26%) of the 35 studies that measured EBP attitude. Conclusions: Most EBP educational interventions which have been evaluated in controlled studies focus on teaching only some of the EBP steps (predominantly critically appraisal of evidence) and did not use high-quality instruments to measure outcomes. Educational packages and instruments which address all EBP steps are needed to improve EBP teaching.
... Citation analysis can efficiently elude the time-consuming and complex nature of traditional search strategies with an acceptable rate of accuracy. [22][23][24] Further, citation analysis does not depend on the use of specific keywords and search terms, which may be advantageous, particularly in disciplines in which there is inconsistent terminology. 25,26 However, this may also carry risk for the missing of a few relevant studies. ...
Article
Context: Complete reporting of intervention details in trials of evidence-based practice (EBP) educational interventions is essential to enable clinical educators to translate research evidence about interventions that have been shown to be effective into practice. In turn, this will improve the quality of EBP education. Objectives: This study was designed to examine the completeness of reporting of EBP educational interventions in published studies and to assess whether missing details of educational interventions could be retrieved by searching additional sources and contacting study authors. Methods: A systematic review of controlled trials that had evaluated EBP educational interventions was conducted using a citation analysis technique. Forward and backward citations of the index articles were tracked until March 2016. The TIDieR (template for intervention description and replication) checklist was used to assess the completeness of intervention reporting. Missing details were sought from: (i) the original publication; (ii) additional publicly available sources, and (iii) the study authors. Results: Eighty-three articles were included; 45 (54%) were randomised controlled trials (RCTs) and 38 (46%) were non-RCTs. The majority of trials (n = 62, 75%) involved medical professionals. None of the studies completely reported all of the main items of the educational intervention within the original publication or in additional sources. However, details became complete for 17 (20%) interventions after contact with the respective authors. The item most frequently missing was 'intervention materials', which was missing in 80 (96%) of the original publications, in additional sources for 77 (93%) interventions, and in 59 (71%) studies after contact with the authors. Authors of 69 studies were contacted; 33 provided the details requested. Conclusions: The reporting of EBP educational interventions is incomplete and remained so for the majority of studies, even after study authors had been contacted for missing information. Collaborative efforts involving authors and editors are required to improve the completeness of reporting of EBP educational interventions.
... Badampudi et al. [2] advise caution in selecting a good start set, since it is required to reach su cient recall. Medicine researchers have found that citation graphs of papers are o en disconnected [27], and also advise use of a good start set when snowballing. We recommend combining snowballing with our automated term based search, which automatically extends the start set. ...
Conference Paper
Background. Search and selection of primary studies in Systematic Literature Reviews (SLR) is labour intensive, and hard to replicate and update. Aims. We explore a machine learning approach to support semi-automated search and selection in SLRs to address these weaknesses. Method. We 1) train a classifier on an initial set of papers, 2) extend this set of papers by automated search and snowballing, 3) have the researcher validate the top paper, selected by the classifier, and 4) update the set of papers and iterate the process until a stopping criterion is met. Results. We demonstrate with a proof-of-concept tool that the proposed automated search and selection approach generates valid search strings and that the performance for subsets of primary studies can reduce the manual work by half. Conclusions. The approach is promising and the demonstrated advantages include cost savings and replicability. The next steps include further tool development and evaluate the approach on a complete SLR.
... Searching continued backward and forward until no further relevant studies are found. Citation analysis can evade the time-consuming and the complex nature of the standard search strategies, in areas where indexing is unlikely to retrieve relevant articles with an acceptable accuracy rate [21][22][23] . ...
Article
Objective: To systematically review current evidence regarding the minimum acceptable risk reduction of a cardiovascular event that patients feel would justify daily intake of a preventive medication. Methods: We used the Web of Science to track the forward and backward citations of a set of five key articles until 15 November 2016. Studies were eligible if they quantitatively assessed the minimum acceptable benefit-in absolute values-of a cardiovascular disease preventive medication among a sample of the general population and required participants to choose if they would consider taking the medication. Results: Of 341 studies screened, we included 22, involving a total of 17 751 participants: 6 studied prolongation of life (POL), 12 studied absolute risk reduction (ARR) and 14 studied number needed to treat (NNT) as measures of risk reduction communicated to the patients. In studies framed using POL, 39%-54% (average: 48%) of participants would consider taking a medication if it prolonged life by <8 months and 56%-73% (average: 64%) if it prolonged life by ≥8 months. In studies framed using ARR, 42%-72% (average: 54%) of participants would consider taking a medication that reduces their 5-year cardiovascular disease (CVD) risk by <3% and 50%-89% (average: 77%) would consider taking a medication that reduces their 5-year CVD risk by ≥3%. In studies framed using 5-year NNT, 31%-81% (average: 60%) of participants would consider taking a medication with an NNT of >30 and 46%-87% (average: 71%) with an NNT of ≤30. Conclusions: Many patients require a substantial risk reduction before they consider taking a daily medication worthwhile, even when the medication is described as being side effect free and costless.
... A study is more likely to be cited if it is in agreement with the more recent trial both in the case of trials and reviews (Helfer et al., 2015;Andrade, Flynn, & Bartanusz, 2013;Sawin & Robinson, 2015). The poor citation of prior studies is the main reason why the citation networks of related trials tend to be disconnected (Robinson et al., 2014). ...
Article
Citation frequencies are commonly interpreted as measures of quality or impact. Yet, the true nature of citations and their proper interpretation have been the center of a long, but still unresolved discussion in Bibliometrics. A comparison of 67,578 pairs of studies on the same healthcare topic, with the same publication age (1–15 years) reveals that when one of the studies is being selected for citation, it has on average received about three times as many citations as the other study. However, the average citation-gap between selected or deselected studies narrows slightly over time, which fits poorly with the name-dropping interpretation and better with the quality and impact-interpretation. The results demonstrate that authors in the field of Healthcare tend to cite highly cited documents when they have a choice. This is more likely caused by differences related to quality than differences related to status of the publications cited.
... Savoie et al. and Helmer et al. [61,62] found that 29.2% of all items retrieved for two SRs could be uncovered by extended systematic search methods; searching subject-specific or specialized databases (including trial registries), hand-searching, scanning reference lists, and communicating personally with experts. Yet Robinson et al. [63] recently showed that researchers do not cite all possible previous trials, and that less than half (38%) of RCTs could be identified by citation network searching. ...
Article
Full-text available
Background: When conducting systematic reviews, it is essential to perform a comprehensive literature search to identify all published studies relevant to the specific research question. The Cochrane Collaborations Methodological Expectations of Cochrane Intervention Reviews (MECIR) guidelines state that searching MEDLINE, EMBASE and CENTRAL should be considered mandatory. The aim of this study was to evaluate the MECIR recommendations to use MEDLINE, EMBASE and CENTRAL combined, and examine the yield of using these to find randomized controlled trials (RCTs) within the area of musculoskeletal disorders. Methods: Data sources were systematic reviews published by the Cochrane Musculoskeletal Review Group, including at least five RCTs, reporting a search history, searching MEDLINE, EMBASE, CENTRAL, and adding reference- and hand-searching. Additional databases were deemed eligible if they indexed RCTs, were in English and used in more than three of the systematic reviews. Relative recall was calculated as the number of studies identified by the literature search divided by the number of eligible studies i.e. included studies in the individual systematic reviews. Finally, cumulative median recall was calculated for MEDLINE, EMBASE and CENTRAL combined followed by the databases yielding additional studies. Results: Deemed eligible was twenty-three systematic reviews and the databases included other than MEDLINE, EMBASE and CENTRAL was AMED, CINAHL, HealthSTAR, MANTIS, OT-Seeker, PEDro, PsychINFO, SCOPUS, SportDISCUS and Web of Science. Cumulative median recall for combined searching in MEDLINE, EMBASE and CENTRAL was 88.9% and increased to 90.9% when adding 10 additional databases. Conclusion: Searching MEDLINE, EMBASE and CENTRAL was not sufficient for identifying all effect studies on musculoskeletal disorders, but additional ten databases did only increase the median recall by 2%. It is possible that searching databases is not sufficient to identify all relevant references, and that reviewers must rely upon additional sources in their literature search. However further research is needed.
... Search terms included (in various combinations) exercise testing, oxygen uptake, VO 2 , CRF, cardiovascular disease (CVD), CAD, CHF and physical activity. With publications having the prerequisite HR data extensive cross-referencing was undertaken to source other publications with eligible criteria [33]. ...
Article
Full-text available
Background Cardiorespiratory fitness measured by treadmill testing has prognostic significance in determining mortality with cardiovascular and other chronic disease states. The accuracy of a recently developed method for estimating maximal oxygen uptake (VO2peak), the heart rate index (HRI), is dependent only on heart rate (HR) and was tested against oxygen uptake (VO2), either measured or predicted from conventional treadmill parameters (speed, incline, protocol time). Methods The HRI equation, METs = 6 x HRI– 5, where HRI = maximal HR/resting HR, provides a surrogate measure of VO2peak. Forty large scale treadmill studies were identified through a systematic search using MEDLINE, Google Scholar and Web of Science in which VO2peak was either measured (TM-VO2meas; n = 20) or predicted (TM-VO2pred; n = 20) based on treadmill parameters. All studies were required to have reported group mean data of both resting and maximal HRs for determination of HR index-derived oxygen uptake (HRI-VO2). Results The 20 studies with measured VO2 (TM-VO2meas), involved 11,477 participants (median 337) with a total of 105,044 participants (median 3,736) in the 20 studies with predicted VO2 (TM-VO2pred). A difference of only 0.4% was seen between mean (±SD) VO2peak for TM- VO2meas and HRI-VO2 (6.51±2.25 METs and 6.54±2.28, respectively; p = 0.84). In contrast, there was a highly significant 21.1% difference between mean (±SD) TM-VO2pred and HRI-VO2 (8.12±1.85 METs and 6.71±1.92, respectively; p<0.001). Conclusion Although mean TM-VO2meas and HRI-VO2 were almost identical, mean TM-VO2pred was more than 20% greater than mean HRI-VO2.
... Searching these 'direct' citations could be an efficient strategy only if eligible studies consistently cited all relevant earlier work, thus creating a single citation network, but this is often not the case. For example, a review of 259 meta-analyses found that in fewer than half (46 %) were included articles connected in a single citation network; in the remainder, included articles were in either two (39 %) or three or more (15 %) disconnected citation networks [7]. Citation searching has thus gained only equivocal support, even as a complement to keyword searching [8,9]. ...
Article
Full-text available
Background Finding eligible studies for meta-analysis and systematic reviews relies on keyword-based searching as the gold standard, despite its inefficiency. Searching based on direct citations is not sufficiently comprehensive. We propose a novel strategy that ranks articles on their degree of co-citation with one or more “known” articles before reviewing their eligibility. Method In two independent studies, we aimed to reproduce the results of literature searches for sets of published meta-analyses (n = 10 and n = 42). For each meta-analysis, we extracted co-citations for the randomly selected ‘known’ articles from the Web of Science database, counted their frequencies and screened all articles with a score above a selection threshold. In the second study, we extended the method by retrieving direct citations for all selected articles. Results In the first study, we retrieved 82 % of the studies included in the meta-analyses while screening only 11 % as many articles as were screened for the original publications. Articles that we missed were published in non-English languages, published before 1975, published very recently, or available only as conference abstracts. In the second study, we retrieved 79 % of included studies while screening half the original number of articles. Conclusions Citation searching appears to be an efficient and reasonably accurate method for finding articles similar to one or more articles of interest for meta-analysis and reviews.
... Given the growing amount of research versus the urgency to provide the proper medical care, the research time has to be optimized. However the citation network of related clinical trials is disconnected, which reflects the possibility that the ''different camps'' of clinical researchers use different research tools and hence are unaware of the relevant literature from the other ''camps'' [16]. ...
Article
Full-text available
Systematic searching aims to find all possibly relevant research from multiple sources, the basis for an unbiased and comprehensive evidence base. Along with bibliographic databases, systematic reviewers use a variety of additional methods to minimise procedural bias. Citation chasing exploits connections between research articles to identify relevant records for a review by making use of explicit mentions of one article within another. Citation chasing is a popular supplementary search method because it helps to build on the work of primary research and review authors. It does so by identifying potentially relevant studies that might otherwise not be retrieved by other search methods; for example, because they did not use the review authors’ search terms in the specified combinations in their titles, abstracts, or keywords. Here, we briefly provide an overview of citation chasing as a method for systematic reviews. Furthermore, given the challenges and high resource requirements associated with citation chasing, the limited application of citation chasing in otherwise rigorous systematic reviews, and the potential benefit of identifying terminologically disconnected but semantically linked research studies, we have developed and describe a free and open source tool that allows for rapid forward and backward citation chasing. We introduce citationchaser, an R package and Shiny app for conducting forward and backward citation chasing from a starting set of articles. We describe the sources of data, the backend code functionality, and the user interface provided in the Shiny app. This article is protected by copyright. All rights reserved.
Chapter
Traditionally, literature identification for systematic reviews has relied on a two-step process: first, searching databases to identify potentially relevant citations, and then manually screening those citations. A number of tools have been developed to streamline and semi-automate this process, including tools to generate terms; to visualize and evaluate search queries; to trace citation linkages; to deduplicate, limit, or translate searches across databases; and to prioritize relevant abstracts for screening. Research is ongoing into tools that can unify searching and screening into a single step, and several protype tools have been developed. As this field grows, it is becoming increasingly important to develop and codify methods for evaluating the extent to which these tools fulfill their purpose.
Article
Automated literature reviews have the potential to accelerate knowledge synthesis and provide new insights. However, a lack of labeled ground-truth data has made it difficult to develop and evaluate these methods. We propose a framework that uses the reference lists from existing review papers as labeled data, which can then be used to train supervised classifiers, allowing for experimentation and testing of models and features at a large scale. We demonstrate our framework by training classifiers using different combinations of citation- and text-based features on 500 review papers. We use the R-Precision scores for the task of reconstructing the review papers’ reference lists as a way to evaluate and compare methods. We also extend our method, generating a novel set of articles relevant to the fields of misinformation studies and science communication. We find that our method can identify many of the most relevant papers for a literature review from a large set of candidate papers, and that our framework allows for development and testing of models and features to incrementally improve the results. The models we build are able to identify relevant papers even when starting with a very small set of seed papers. We also find that the methods can be adapted to identify previously undiscovered articles that may be relevant to a given topic.
Article
Background: Forwards citation searching is a valuable method for finding relevant studies in reviews where concepts are not clearly defined. Scopus and Web of Science can both be used for forwards citation searching but there is little evidence comparing the resources for this purpose. Method: 104 source records relevant to a scoping review of implementation and dissemination strategies in dementia care were checked for inclusion on Web of Science and Scopus. The number of citing references was recorded. Where citing references appeared unique to one of the resources, they were checked for inclusion on the other resource to assess the performance of citation links. Results: 1397 citing references were returned by Scopus and 1010 were returned by Web of Science. For the unique citing references returned by Web of Science (n = 52), 36 were subsequently found to be on Scopus but had failed to be picked up as citing a source record. Of the unique citing references returned by Scopus (n = 355), 83 were found to be on Web of Science but had failed to be picked up as citing a source record. 26 additional relevant records were identified for the review by forwards citation searching. All were found on Scopus; six would have been missed by searching Web of Science alone. Conclusion: Citation searching using Scopus alone would have found all additional relevant studies for the review. Both Scopus and Web of Science failed to return all citing references from the source records, even where they were present on the database, indicating poor links between citations. This article is protected by copyright. All rights reserved.
Article
Objective: Balanced citations are a necessary condition for a sound development of scientific knowledge, whereas selective citations may bias scientific consensus. In this study, we assess which determinants influenced the likelihood of being cited in the literature on trans fatty acids and cholesterol. Study design and setting: We conducted a citation network analysis of the literature concerning trans fats and low density cholesterol and high density cholesterol. Each publication was scored on various potential determinants of citation, such as study outcome, study design, sample size, journal impact factor, and funding source. We applied random effect logistic regression to identify determinants of citation. Results: A network of 108 publications was identified, containing 5,041 potential citation paths and 669 utilized citation paths. Reporting statistically significant results was found to be a strong predictor of citation, together with sample size, journal impact factor, and the authority of the authors. Conclusion: Within the literature on trans fat intake and cholesterol, selective citations are based on several grounds. Especially the effect of reporting significant results on citation requires special attention because disproportionate attention is paid to publications suggesting a harmful effect of trans fat on cholesterol.
Article
A growing number of researchers are exploring the use of citation relationships such as direct citation, bibliographic coupling, and co-citation for information retrieval in scientific databases and digital libraries. In this paper, I propose a method of ranking the relevance of citation-based search results to a set of key, or seed, papers by measuring the number of citation relationships they share with those key papers. I tested the method against 23 published systematic reviews and found that the method retrieved 87% of the studies included in these reviews. The relevance ranking approach identified a subset of the citation search results that comprised 27% of the total documents retrieved by the method, and 7% of the documents retrieved by these reviews, but that contained 75% of the studies included in these reviews. Additional testing suggested that the method may be less appropriate for reviews that combine literature in ways that are not reflected in the literature itself. These results suggest that this ranking method could be useful in a range of information retrieval contexts.
Conference Paper
Full-text available
Evidence-based medical guidelines contain a collection of recommendations which have been created using the best clinical research findings (a.k.a. evidences) of the highest value to aid in the delivery of optimum clinical care to patients. In evidence-based medical guidelines, the conclusions (a.k.a. recommendations) are marked with different evidence levels according to quality of the supporting evidences. Finding new relevant and higher quality evidences is an important issue for supporting the process of updating medical guidelines. In this paper, we propose a method to automatically identify all evidence classes. Furthermore, the proposed approach has been implemented by a rule-based approach, in which the identification knowledge is formalized as a set of rules in the declarative logic programming language Prolog, so that the knowledge can be easily maintained, updated, and re-used. Our experiments show that the proposed method for identifying the evidence quality has a recall of 0.35 and a precision of 0.42. For the identification of A-class evidences (the top evidence class), the performance of the proposed method improves to recall = 0.63 and precision = 0.74.
Article
Systematic reviews are essential for evaluating biomedical treatment options, but the growing size and complexity of the available biomedical literature combined with the rigor of the systematic review method mean that systematic reviews are extremely difficult and labor-intensive to perform. In this article, I propose a method of searching the literature by systematically mining the various types of citation relationships between articles. I then test the method by comparing its precision and recall to that of 14 published systematic reviews. The method successfully retrieved 74% of the studies included in these reviews and 90% of the studies it could reasonably be expected to retrieve. The method also retrieved fewer than half of the total number of publications retrieved by these reviews and can be performed in substantially less time. This suggests that the proposed method offers a promising complement to traditional text-based methods of literature identification and retrieval for systematic reviews.
Article
We assessed citation of prior research over time and the association of citation with the agreement of results between the trial being reported and the prior trial. Groups of pharmacological trials in cardiovascular disease were created using meta-analyses and we assessed citation within these groups. We calculated the proportion of prior trials cited, the proportion of study participants captured in citations, and agreement of results between citing and cited trials. Analysis included 86 meta-analyses with 580 trials published between 1982 and 2011. Reports of trials cited 25% (median; 95% CI, 23% to 27%) of prior trials, capturing 31% (95% CI, 25% to 36%) of trial participants. Neither measure differed by publication of the citing trial before versus after 2005. Prior trials with results that agreed with the reports of trials (supportive trials) were significantly more likely to be cited than nonsupportive trials (RR 1.45; 95% CI, 1.30 to 1.61, p<0.001). Selective under-citation of prior research continues; three-quarters of existing evidence is ignored. This source of waste may result in unnecessary, unethical, and unscientific studies. Copyright © 2015 Elsevier Inc. All rights reserved.
Article
Full-text available
Background: Snowballing involves recursively pursuing relevant references cited in the retrieved literature and adding them to the search results. Snowballing is an alternative approach to discover additional evidence that was not retrieved through conventional search. Snowballing's effectiveness makes it best practice in systematic reviews despite being time-consuming and tedious. Objective: Our goal was to evaluate an automatic method for citation snowballing's capacity to identify and retrieve the full text and/or abstracts of cited articles. Methods: Using 20 review articles that contained 949 citations to journal or conference articles, we manually searched Microsoft Academic Search (MAS) and identified 78.0% (740/949) of the cited articles that were present in the database. We compared the performance of the automatic citation snowballing method against the results of this manual search, measuring precision, recall, and F1 score. Results: The automatic method was able to correctly identify 633 (as proportion of included citations: recall=66.7%, F1 score=79.3%; as proportion of citations in MAS: recall=85.5%, F1 score=91.2%) of citations with high precision (97.7%), and retrieved the full text or abstract for 490 (recall=82.9%, precision=92.1%, F1 score=87.3%) of the 633 correctly retrieved citations. Conclusions: The proposed method for automatic citation snowballing is accurate and is capable of obtaining the full texts or abstracts for a substantial proportion of the scholarly citations in review articles. By automating the process of citation snowballing, it may be possible to reduce the time and effort of common evidence surveillance tasks such as keeping trial registries up to date and conducting systematic reviews.
Article
Literature reviews underpin the majority of research projects in the health sciences, and yet relatively little analysis has been published as to the most appropriate method to identify relevant literature, outside of specialist information journals. The method of applying keyword search queries to bibliographic databases using Boolean logic dominates literature reviews due to its easy application to the major online databases. However, it is recognised increasingly as being problematic where the research question cannot be clearly defined or requires an element of exploration, due to its reliance on author's use of titling and keywords and is unable to identify topics other than those defined in the search query. This paper discusses the relative merits of a systematic citation searching approach as both an alternative and a concurrent method to keyword searching. A method of citation searching, both forwards and backwards, which is iterated to form a closed loop solution, is discussed. An illustrative example is presented of both methods, applying them to the topic of the UK National Institute for Health and Care Excellence (NICE) cost-effectiveness threshold. The case study finds the citation searching approach dominates the traditional keyword searching approach, finding 76 papers of relevance, including all 15 found by the alternative approach. Conceptually, and in the example presented, it is demonstrated that the proposed method can represent a dominant strategy to the more traditional approach in some situations, highlighting that, wherever possible, it is preferential to employ multiple methods of searching. However, it is clear that a better understanding is required as to how we can most efficiently search the ever-growing sea of literature.
Article
Full-text available
Systematic reviews, a cornerstone of evidence-based medicine, are not produced quickly enough to support clinical practice. The cost of production, availability of the requisite expertise and timeliness are often quoted as major contributors for the delay. This detailed survey of the state of the art of information systems designed to support or automate individual tasks in the systematic review, and in particular systematic reviews of randomized controlled clinical trials, reveals trends that see the convergence of several parallel research projects. We surveyed literature describing informatics systems that support or automate the processes of systematic review or each of the tasks of the systematic review. Several projects focus on automating, simplifying and/or streamlining specific tasks of the systematic review. Some tasks are already fully automated while others are still largely manual. In this review, we describe each task and the effect that its automation would have on the entire systematic review process, summarize the existing information system support for each task, and highlight where further research is needed for realizing automation for the task. Integration of the systems that automate systematic review tasks may lead to a revised systematic review workflow. We envisage the optimized workflow will lead to system in which each systematic review is described as a computer program that automatically retrieves relevant trials, appraises them, extracts and synthesizes data, evaluates the risk of bias, performs meta-analysis calculations, and produces a report in real time.
Article
Full-text available
Would lead to best currently available evidence at the push of a button The Cochrane handbook stipulates that systematic reviews should be examined every two years and updated if needed,1 but time and resource constraints mean that this occurs for only a third of reviews.2 Indeed, it may take as much time to update a review as it did to produce the original review. If this effort were redirected at developing methods to automate reviews, then updating might one day become almost effortless, immediate, and universal. In his novel Player Piano , Kurt Vonnegut Jr described machines that record the hand motions of artisans and replay them to reproduce a perfect copy of the artefact, more quickly and more economically. Such automation is needed in the update and even creation of systematic reviews, because the capability of the human machinery for review increasingly lags behind our capacity to produce primary evidence.3 The current reality is that many reviews are missing or outdated,4 and it is hard to imagine a solution that does not involve some automation.5 Technology has advanced such that software can be used …
Article
Full-text available
Pulmonary metastasectomy for colorectal cancer is a commonly performed and well-established practice of ∼50 years standing. However, there have been no controlled studies, randomised or otherwise. We sought to investigate the evidence base that has been used in establishing its status as a standard of care. Among 51 papers used in a recent systematic review and quantitative synthesis, a citation network analysis was performed. A total of 344 publications (the 51 index papers and a further 293 cited in them) constitute the citation network. The pattern of citation is that of a citation cascade. Specific analyses show the frequent use of historical or landmark papers, which add authority. Papers expressing an opposing viewpoint are rarely cited. The citation network for this common and well-established practice provides an example of selective citation. This pattern of citation tends to escalate belief in a clinical practice even when it lacks a high-quality evidence base and may create an impression of more authority than is warranted.
Article
Full-text available
To understand belief in a specific scientific claim by studying the pattern of citations among papers stating it. A complete citation network was constructed from all PubMed indexed English literature papers addressing the belief that beta amyloid, a protein accumulated in the brain in Alzheimer's disease, is produced by and injures skeletal muscle of patients with inclusion body myositis. Social network theory and graph theory were used to analyse this network. Citation bias, amplification, and invention, and their effects on determining authority. The network contained 242 papers and 675 citations addressing the belief, with 220,553 citation paths supporting it. Unfounded authority was established by citation bias against papers that refuted or weakened the belief; amplification, the marked expansion of the belief system by papers presenting no data addressing it; and forms of invention such as the conversion of hypothesis into fact through citation alone. Extension of this network into text within grants funded by the National Institutes of Health and obtained through the Freedom of Information Act showed the same phenomena present and sometimes used to justify requests for funding. Citation is both an impartial scholarly method and a powerful form of social communication. Through distortions in its social use that include bias, amplification, and invention, citation can be used to generate information cascades resulting in unfounded authority of claims. Construction and analysis of a claim specific citation network may clarify the nature of a published belief system and expose distorted methods of social citation.
Article
Full-text available
To see if the claim that lowering cholesterol values prevents coronary heart disease is true or if it is based on citation of supportive trials only. Comparison of frequency of citation with outcome of all controlled cholesterol lowering trials using coronary heart disease or death, or both, as end point. 22 controlled cholesterol lowering trials. Trials considered by their directors as supportive of the contention were cited almost six times more often than others, according to Science Citation Index. Apart from trials discontinued because of alleged side effects of treatment, unsupportive trials were not cited after 1970, although their number almost equalled the number considered supportive. In three supportive reviews the outcome of the selected trials was more favourable than the outcome of the excluded and ignored trials. In the 22 controlled cholesterol lowering trials studied total and coronary heart disease mortality was not changed significantly either overall or in any subgroup. A statistically significant 0.32% reduction in non-fatal coronary heart disease seemed to be due to bias as event frequencies were unrelated to trial length and to mean net reduction in cholesterol value; individual changes in cholesterol values were unsystematically or not related to outcome; and after correction for a small but significant increase in non-medical deaths in the intervention groups total mortality remained unchanged (odds ratio 1.02). Lowering serum cholesterol concentrations does not reduce mortality and is unlikely to prevent coronary heart disease. Claims of the opposite are based on preferential citation of supportive trials.
Article
Full-text available
Cochrane-style systematic reviews increasingly require the participation of librarians. Guidelines on the appropriate search strategy to use for systematic reviews have been proposed. However, research evidence supporting these recommendations is limited. This study investigates the effectiveness of various systematic search methods used to uncover randomized controlled trials (RCTs) for systematic reviews. Effectiveness is defined as the proportion of relevant material uncovered for the systematic review using extended systematic review search methods. The following extended systematic search methods are evaluated: searching subject-specific or specialized databases (including trial registries), hand searching, scanning reference lists, and communicating personally. Two systematic review projects were prospectively monitored regarding the method used to identify items as well as the type of items retrieved. The proportion of RCTs identified by each systematic search method was calculated. The extended systematic search methods uncovered 29.2% of all items retrieved for the systematic reviews. The search of specialized databases was the most effective method, followed by scanning of reference lists, communicating personally, and hand searching. Although the number of items identified through hand searching was small, these unique items would otherwise have been missed. Extended systematic search methods are effective tools for uncovering material for the systematic review. The quality of the items uncovered has yet to be assessed and will be key in evaluating the value of the systematic search methods.
Article
Full-text available
Citation by other authors is important in the dissemination of published science, but factors predicting it are little studied. To identify characteristics of published research predicting citation in other journals, we searched the Science Citations Index database for a standardized 3.5 years for all citations of published articles originally submitted to a 1991 emergency medicine specialty meeting. Analysis was conducted by classification and regression trees, a nonparametric modeling technique of regression trees, to determine the impact of previously determined characteristics of the full articles on the outcome measures. We calculated the the number of times an article was cited each year and calculated the mean impact factor (citations per manuscript per year) in other citing journals. Of the 493 submitted manuscripts, 204 published articles met entry criteria. The mean citations per year was 2.04 (95% confidence interval, 1.6-2.4; range, 0-20.9) in 440 different journals. Nineteen articles (9.3%) were never cited. The ability to predict the citations per year was weak (pseudo R(2) = 0.14.). The strongest predictor of citations per year was the impact factor of the original publishing journal. The presence of a control group, the subjective newsworthiness score, and sample size predicted citation frequency (24.3%, 26.0%, and 26.5% as strongly, respectively). The ability to predict mean impact factor of the citing journals was even weaker (pseudo R(2) = 0.09). The impact factor of the publishing journal was the strongest predictor, followed by the newsworthiness score (89.9% as strongly) and a subjective quality score (61.5%). Positive outcome bias was not evident for either outcome measure. In this cohort of published research, commonly used measures of study methodology and design did not predict the frequency of citations or the importance of citing journals. Positive outcome bias was not evident. The impact factor of the original publishing journal was more important than any other variable, suggesting that the journal in which a study is published may be as important as traditional measures of study quality in ensuring dissemination.
Article
Full-text available
To quantify more reliably the benefits and risks of monoamine oxidase type B inhibitors (MAOBIs) in early Parkinson's disease. Searches of the Cochrane Library, Medline, Embase, PubMed, and Web of Science for years 1966-2003, plus major journals in the field, abstract books, and proceedings of meetings, for randomised trials comparing MAOBIs with placebo or levodopa. Available data on mortality, motor complications, side effects, treatment compliance, and clinician rated disability (for example, unified Parkinson's disease rating scale) were extracted from 17 trials and combined using standard meta-analytic methods. No significant difference in mortality existed between patients on MAOBIs and control patients (odds ratio 1.13, 95% confidence interval 0.94 to 1.34; P = 0.2). Patients randomised to MAOBIs had significantly better total scores, motor scores, and activities of daily living scores on the unified Parkinson's disease rating scale at three months compared with patients taking placebo; they were also less likely to need additional levodopa (0.57, 0.48 to 0.67; P < 0.00001) or to develop motor fluctuations (0.75, 0.59 to 0.95; P = 0.02). No difference existed between the two groups in the incidence of side effects or withdrawal of patients. MAOBIs reduce disability, the need for levodopa, and the incidence of motor fluctuations, without substantial side effects or increased mortality. However, because few trials have compared MAOBIs with other antiparkinsonian drugs, uncertainty remains about the relative benefits and risks of MAOBIs. Further large, long term comparative trials that include patient rated quality of life measures are needed.
Article
Full-text available
To describe where papers come from in a systematic review of complex evidence. Method Audit of how the 495 primary sources for the review were originally identified. Only 30% of sources were obtained from the protocol defined at the outset of the study (that is, from the database and hand searches). Fifty one per cent were identified by "snowballing" (such as pursuing references of references), and 24% by personal knowledge or personal contacts. Systematic reviews of complex evidence cannot rely solely on protocol-driven search strategies.
Article
Full-text available
Citation counts are often regarded as a measure of the utilization and contribution of published articles. The objective of this study is to assess whether statistical reporting and statistical errors in the analysis of the primary outcome are associated with the number of citations received. We evaluated all original research articles published in 1996 in four psychiatric journals. The statistical and reporting quality of each paper was assessed and the number of citations received up to 2005 was obtained from the Web of Science database. We then examined whether the number of citations was associated with the quality of the statistical analysis and reporting. A total of 448 research papers were included in the citation analysis. Unclear or inadequate reporting of the research question and primary outcome were not statistically significantly associated with the citation counts. After adjusting for journal, extended description of statistical procedures had a positive effect on the number of citations received. Inappropriate statistical analysis did not affect the number of citations received. Adequate reporting of the primary research question, statistical methods and primary findings were all associated with the journal visibility and prestige. In this cohort of published research, measures of reporting quality and appropriate statistical analysis were not associated with the number of citations. The journal in which a study is published appears to be as important as the statistical reporting quality in ensuring dissemination of published medical science.
Article
Full-text available
To compare beta lactam monotherapy with beta lactam-aminoglycoside combination therapy for severe infections. Medline, Embase, Lilacs, Cochrane Library, and conference proceedings, to 2003; references of included studies; contact with all authors. No restrictions, such as language, year of publication, or publication status. All randomised trials of beta lactam monotherapy compared with beta lactam-aminoglycoside combination therapy for patients without neutropenia who fulfilled criteria for sepsis. Two reviewers independently applied selection criteria, performed quality assessment, and extracted the data. The primary outcome assessed was all cause fatality by intention to treat. Relative risks were pooled with the random effect model (relative risk < 1 favours monotherapy). 64 trials with 7586 patients were included. There was no difference in all cause fatality (relative risk 0.90, 95% confidence interval 0.77 to 1.06). 12 studies compared the same beta lactam (1.02, 0.76 to 1.38), and 31 studies compared different beta lactams (0.85, 0.69 to 1.05). Clinical failure was more common with combination treatment overall (0.87, 0.78 to 0.97) and among studies comparing different beta lactams (0.76, 0.68 to 0.86). There was no advantage to combination therapy among patients with Gram negative infections (1835 patients) or Pseudomonas aeruginosa infections (426 patients). There was no difference in the rate of development of resistance. Nephrotoxicity was significantly more common with combination therapy (0.36, 0.28 to 0.47). Heterogeneity was not significant for these comparisons. In the treatment of sepsis the addition of an aminoglycoside to beta lactams should be discouraged. Fatality remains unchanged, while the risk for adverse events is increased.
Article
Full-text available
The citation rate for articles is viewed as a measure of their importance and impact; however, little is known about what features of articles are associated with higher citation rate. We conducted a cohort study of all original articles, regardless of study methodology, published in the Lancet, JAMA, and New England Journal of Medicine, from October 1, 1999 to March 31, 2000. We identified 328 articles. Two blinded, independent reviewers extracted, in duplicate, nine variables from each article, which were analyzed in both univariable and multivariable linear least-squares regression models for their association with the annual rate of citations received by the article since publication. A two-way interaction between industry funding and an industry-favoring result was tested and found to be significant (p = 0.02). In our adjusted analysis, the presence of industry funding and an industry-favoring result was associated with an increase in annual citation rate of 25.7 (95% confidence interval, 8.5 to 42.8) compared to the absence of both industry funding and industry-favoring results. Higher annual rates of citation were also associated with articles dealing with cardiovascular medicine (13.3 more; 95% confidence interval, 3.9 to 22.3) and oncology (12.6 more; 95% confidence interval, 1.2 to 24.0), articles with group authorship (11.1 more; 95% confidence interval, 2.7 to 19.5), larger sample size and journal of publication. Large trials, with group authorship, industry-funded, with industry-favoring results, in oncology or cardiology were associated with greater subsequent citations.
Article
Full-text available
To determine if citation counts at two years could be predicted for clinical articles that pass basic criteria for critical appraisal using data within three weeks of publication from external sources and an online article rating service. Retrospective cohort study. Online rating service, Canada. 1274 articles from 105 journals published from January to June 2005, randomly divided into a 60:40 split to provide derivation and validation datasets. 20 article and journal features, including ratings of clinical relevance and newsworthiness, routinely collected by the McMaster online rating of evidence system, compared with citation counts at two years. The derivation analysis showed that the regression equation accounted for 60% of the variation (R2=0.60, 95% confidence interval 0.538 to 0.629). This model applied to the validation dataset gave a similar prediction (R2=0.56, 0.476 to 0.596, shrinkage 0.04; shrinkage measures how well the derived equation matches data from the validation dataset). Cited articles in the top half and top third were predicted with 83% and 61% sensitivity and 72% and 82% specificity. Higher citations were predicted by indexing in numerous databases; number of authors; abstraction in synoptic journals; clinical relevance scores; number of cited references; and original, multicentred, and therapy articles from journals with a greater proportion of articles abstracted. Citation counts can be reliably predicted at two years using data within three weeks of publication.
Article
Objective: To evaluate the sensitivity and precision of various extended search methods in identifying randomized controlled trials (RCTs) for systematic reviews. Method: Prospective analysis of extended search methods (specialized databases or trial registries, reference lists, hand-searching, personal communication, and Internet) used in two systematic reviews of RCTs. The gold standard was the total number of RCTs identified by major databases (MEDLINE, EMBASE, etc.) and extended search strategies combined. Sensitivity was the proportion of all known RCTs identified by any extended search method. Precision reflected the proportion of all items uncovered by any extended search method that actually were RCTs. Results: The extended search identified 94 additional RCTs for the systematic reviews beyond those identified with the major databases. Specialized databases and trial registries had the highest sensitivity and precision for the lipid-lowering project (13.6% and 52.7%, respectively; p < .05) followed by scanning of reference lists (7.2% sensitivity and 41.9% precision; p < .05). Hand-searching was more effective than personal communication and Internet searching (1.7% sensitivity and 12.2% precision; p < .05). The acupuncture project had slightly different results, with the specialized databases and trial registries tied with the review of reference lists for highest sensitivity (14.2%). The precision followed the same trend as the lipid-lowering project (17.6% specialized databases; 8.3% reference lists; p < .05). A post-hoc analysis showed that 75 of the 94 RCTs were indexed in the major databases but missed by the major database search. Conclusions: Extended searching identified additional RCTs for the systematic reviews beyond those found in major databases. Specialized databases and trial registries were most effective. An important number of RCTs were missed by the major database search. Timing and accuracy of indexing may explain this finding. The definitive measure, whether there is an association between the method used to uncover RCTs, the quality of the items uncovered and their impact on systematic review results, is yet to be determined.
Article
Background: Checking reference lists to identify relevant studies for systematic reviews is frequently recommended by systematic review manuals and is often undertaken by review authors. To date, no systematic review has explicitly examined the effectiveness of checking reference lists as a method to supplement electronic searching. Objectives: To investigate the effectiveness of checking reference lists for the identification of additional, relevant studies for systematic reviews. Effectiveness is defined as the proportion of relevant studies identified by review authors solely by checking reference lists. Search strategy: We searched the databases of The Cochrane Library (Issue 3, 2008), Library and Information Science abstracts (LISA) (1969 to July 2008) and MEDLINE (1966 to July 2008). We contacted experts in systematic review methods and examined reference lists of articles. Selection criteria: Studies of any design which examined checking reference lists as a search method for systematic reviews in any area. The primary outcome was the additional yield of relevant studies (i.e. studies not found through any other search methodologies); other outcomes were publication types identified and data pertaining to the costs (e.g. cost-effectiveness, cost-efficiency) of checking reference lists. Data collection and analysis: We summarized data descriptively. Main results: We included 12 studies (in 13 publications) in this review, but interpretability and generalizability of these studies is difficult and the study designs used were at high risk of bias. The additional yield (calculated by dividing the additional 'unique' yield identified by checking reference lists by the total number of studies found to be eligible within the study) of relevant studies identified through checking reference lists ranged from 2.5% to 42.7%. Only two studies reported yield information by publication type (dissertations and systematic reviews). No cost data were reported although one study commented that it was impossible to isolate the time spent on reference tracking since this was done in parallel with the critical appraisal of each paper, and for that particular study costs were not specifically estimated. Authors' conclusions: There is some evidence to support the use of checking reference lists for locating studies in systematic reviews. However, this evidence is derived from weak study designs. In situations where the identification of all relevant studies through handsearching and database searching is difficult, it would seem prudent that authors of reviews check reference lists to supplement their searching. The challenge, therefore, is for review authors to recognize those situations.
Article
A randomized, controlled trial (RCT) should not be started or interpreted without accounting for evidence from preceding RCTs addressing the same question. Research has suggested that evidence from prior trials is often not accounted for in reports of subsequent RCTs. To assess the extent to which reports of RCTs cite prior trials studying the same interventions. Meta-analyses published in 2004 that combined 4 or more trials were identified; within each meta-analysis, the extent to which each trial report cited the trials that preceded it by more than 1 year was assessed. The proportion of prior trials that were cited (prior research citation index), the proportion of the total participants from prior trials that were in the cited trials (sample size citation index), and the absolute number of trials cited were calculated. 227 meta-analyses were identified, comprising 1523 trials published from 1963 to 2004. The median prior research citation index was 0.21 (95% CI, 0.18 to 0.24), meaning that less than one quarter of relevant reports were cited. The median sample size citation index (0.24 [CI, 0.21 to 0.27]) was similar, suggesting that larger trials were not selectively cited. Of the 1101 RCTs that had 5 or more prior trials to cite, 254 (23%) cited no prior RCTs and 257 (23%) cited only 1. The median number of prior cited trials was 2, which did not change as the number of citable trials increased. The mean number of preceding trials cited by trials published after 2000 was 2.4, compared with 1.5 for those published before 2000 (P < 0.001). The investigators could not ascertain why prior trials were not cited, and noncited trials may have been taken into account in the trial design and proposal stages. In reports of RCTs published over 4 decades, fewer than 25% of preceding trials were cited, comprising fewer than 25% of the participants enrolled in all relevant prior trials. A median of 2 trials was cited, regardless of the number of prior trials that had been conducted. Research is needed to explore the explanations for and consequences of this phenomenon. Potential implications include ethically unjustifiable trials, wasted resources, incorrect conclusions, and unnecessary risks for trial participants. None.
Article
Predicting the impact of any research article on its scientific discipline is often viewed as requiring the passage of time. A recent BMJ article, however, reported that an article's citation rate at 2 years could be predicted by data available 3 weeks following publication. The question remains as to whether establishing a citation trajectory at an early stage holds for psychiatric publications, given the low percentage of psychiatric articles in their analysis. The aim of the current article was to critically examine this area of the scientific literature. Data were collected from the Institute for Scientific Information on scientific papers published in January/February 2006, in the top 30 psychiatric journals. Analyses examined the comparative impact of early citation numbers and several predictors identified in the BMJ article. Only two BMJ variables (a larger number of references per article and larger number of authors) predicted higher citations at 2 years in the principal analysis. Citation counts at 1, 3, 6 and 12 months predicted citations at 2 years, with increasing success over time, and such citation counts were distinctly superior to the quantified variables in the previous study. It appears doubtful that data available at 3 weeks after publication for psychiatric articles are useful in predicting citation counts at 2 years. The trajectory of citation counts for a psychiatric article becomes more apparent with time.
Article
Researchers have no empirically based search stopping rule when looking for potentially relevant articles for inclusion in systematic reviews. We tested a stopping strategy based on capture-mark-recapture (CMR; i.e., the Horizon Estimate) statistical modeling to estimate the total number of articles in the domain of clinical decision support tools for osteoporosis disease management using four large bibliographic databases (Medline, EMBASE, CINAHL, and EBM reviews). Retrospective evaluation of the Horizon Estimate using a systematic review of randomized controlled trials (RCTs) at two levels of article screening: title and abstract (1,246 potentially relevant articles) and full text (42 potentially relevant articles). The CMR model suggests that the total number of potential articles was 1,838 for the first level of screening, and 49 for the full-text level. The four databases provided 68% of known articles for the first level of screening and 81% for full-text screening. The CMR technique can be used in systematic reviews to estimate the closeness to capturing the total body of literature on a given topic. More studies are needed to objectively determine the usefulness of Horizon Estimates as a stopping rule strategy for systematic review searching.
Article
It is hypothesized that health professionals in the United States and the United Kingdom are nationally biased in their citation practices. Articles published in the New England Journal of Medicine and Lancet were used to study citation practices of U.S. and U.K. authors. Percentages of cited references to material published in a specific country were calculated for both the New England Journal of Medicine and Lancet. Using a variation of a citation publication ratio based on Frame and Narin's original ratio, an attempt was made to quantify author bias. To calculate these ratios, values from SERLINE* and the British Library Lending Division were employed to find world journal counts. The results suggest that U.S. authors publishing in the New England Journal of Medicine and U.K. authors publishing in Lancet tend to cite material produced in their own countries more than would be warranted by the amount of material produced by these countries. In addition, these authors cited material produced in non-U.S. and non-U.K. countries far less than the amount of material produced by these countries would indicate.
Article
Articles published before 1985 describing double blind trials of two or more non-steroidal anti-inflammatory drugs in rheumatoid arthritis were examined to see whether there was any bias in the references they cited. Althogether 244 articles meeting the criteria were found through a Medline search and through examining the reference lists of the articles retrieved. The drugs compared in the studies were classified as new or as control drugs and the outcome of the trial as positive or not positive. The reference lists of all papers with references to other trials on the new drug were then examined for reference bias. Positive bias was judged to have occurred if the reference list contained a higher proportion of references with a positive outcome for that drug than among all the articles assumed to have been available to the authors (those published more than two years earlier than the index article). Altogether 133 of the 244 articles were excluded for various reasons--for example, 44 because of multiple publication and 19 because they had no references. Among the 111 articles analysed bias was not possible in the references of 35 (because all the references gave the same outcome); 10 had a neutral selection of references, 22 a negative selection, and 44 a positive selection--a significant positive bias. This bias was not caused by better scientific standing of the cited articles over the uncited ones. Thus retrieving literature by scanning reference lists may produce a biased sample of articles, and reference bias may also render the conclusions of an article less reliable.
Article
The objective of this study was to assess whether trials with a positive (i.e., statistically significant) outcome are cited more often than negative trials. We reviewed 530 randomized clinical trials on hepato-biliary diseases published in 11 English-language journals indexed in MEDLINE from 1985-1996. From each trial, we extracted the statistical significance of the primary study outcome (positive or negative), the disease area, and methodological quality (randomization and double blinding). The number of citations during two calendar years after publication was obtained from Science Citation Index. There was a significant positive association between a statistically significant study outcome and the citation frequency (beta, 0.55, 95% confidence interval, 0.39-0.72). The disease area and adequate generation of the allocation sequence were also significant predictors of the citation frequency. We concluded that positive trials are cited significantly more often than negative trials. The association was not explained by disease area or methodological quality.
Article
To evaluate the sensitivity and precision of various extended search methods in identifying randomized controlled trials (RCTs) for systematic reviews. Prospective analysis of extended search methods (specialized databases or trial registries, reference lists, hand-searching, personal communication, and Internet) used in two systematic reviews of RCTs. The gold standard was the total number of RCTs identified by major databases (MEDLINE, EMBASE, etc.) and extended search strategies combined. Sensitivity was the proportion of all known RCTs identified by any extended search method. Precision reflected the proportion of all items uncovered by any extended search method that actually were RCTs. The extended search identified 94 additional RCTs for the systematic reviews beyond those identified with the major databases. Specialized databases and trial registries had the highest sensitivity and precision for the lipid-lowering project (13.6% and 52.7%, respectively; p < .05) followed by scanning of reference lists (7.2% sensitivity and 41.9% precision; p <.05). Hand-searching was more effective than personal communication and Internet searching (1.7% sensitivity and 12.2% precision; p < .05). The acupuncture project had slightly different results, with the specialized databases and trial registries tied with the review of reference lists for highest sensitivity (14.2%). The precision followed the same trend as the lipid-lowering project (17.6% specialized databases; 8.3% reference lists; p < .05). A post-hoc analysis showed that 75 of the 94 RCTs were indexed in the major databases but missed by the major database search. Extended searching identified additional RCTs for the systematic reviews beyond those found in major databases. Specialized databases and trial registries were most effective. An important number of RCTs were missed by the major database search. Timing and accuracy of indexing may explain this finding. The definitive measure, whether there is an association between the method used to uncover RCTs, the quality of the items uncovered and their impact on systematic review results, is yet to be determined.
Article
Background. The UKPDS 34 showed that intensive treatment with metformin significantly reduces macrovascular end-points and mortality in individuals with newly diagnosed type 2 diabetes compared with intensive treatment with insulin or sulphonylurea derivatives, despite similar glycaemic control. How this should be explained is as yet unclear. We hypothesized that metformin may have a glucose-lowering independent effect on blood pressure and lipid profile. In order to test this hypothesis we systematically reviewed the literature and pooled the data obtained in a meta-analysis.
Article
The benefit of amantadine combination therapy, either with interferon (IFN) alone (double therapy) or with ribavirin and IFN (triple therapy) is unknown. We analyzed the effect of amantadine on the end-of-treatment virological response and the sustained response using meta-analysis of 31 randomized controlled trials. Overall analysis revealed a significant effect of amantadine. Triple therapy was the best regimen for improving the sustained response (mean difference: 8.4%, 95% CI: 2.4-13.8%, P=0.002). In subgroup analysis, amantadine did not have a significant effect upon naive patients or relapsers. In non-responders, combination therapy with amantadine was associated with a significant effect on the sustained response (mean difference: 8.3%, 95% CI: 1.9-14.6%, P=0.01). In sensitivity analysis, double therapy did not improve virological responses. Conversely, triple therapy tended to improve the end-of-treatment virological response and was associated with a significant effect upon the sustained response (mean difference: 12.7%, 95% CI: 3.8-21.6%, P=0.005). Combination therapy with amantadine is of no effect upon naive patients or relapsers. In non-responders, triple therapy with amantadine improved the sustained response. New randomized controlled trials are required to confirm this meta-analysis.
Article
Investigators aim to publish their work in top journals in an effort to achieve the greatest possible impact. One measure of impact is the number of times a paper is cited after its publication in a journal. We conducted a review of the highest impact clinical orthopedic journal (Journal of Bone and Joint Surgery, American volume [J Bone Joint Surg Am]) to determine factors associated with subsequent citations within 3 years of publication. We conducted citation counts for all original articles published in J Bone Joint Surg Am 2000 (12 issues). We used regression analysis to identify factors associated with citation counts. We identified 137 original articles in the J Bone Joint Surg Am. There were 749 subsequent citations within 3 years of publication of these articles. Study design was the only variable associated with subsequent citation rate. Meta-analyses, randomized trials and basic science papers received significantly more citations (mean 15.5, 9.3 and 7.6, respectively) than did observational studies (mean retrospective 5.3, prospective 4.2) and case reports (mean 1.5) (p = 0.01). These study designs were also significantly more likely to be cited in the general medical literature (p = 0.02). Our results suggest that basic science articles and clinical articles with greater methodological safeguards against bias (randomized controlled trials and meta-analyses) are cited more frequently than are clinical studies with less rigorous study designs (observational studies and case reports).
Citation bias of hepato-biliary randomized clinical trials:407e10. [8] Gotzsche PC. Reference bias in reports of drug trials
  • Ll Kjaergard
  • Gluud
Kjaergard LL, Gluud C. Citation bias of hepato-biliary randomized clinical trials. J Clin Epidemiol 2002;55:407e10. [8] Gotzsche PC. Reference bias in reports of drug trials. BMJ 1987;295: 654e6.