Editorial Peer Reviewers' Recommendations at a General Medical Journal: Are They Reliable and Do Editors Care?

Department of Medicine, University of California Davis, Sacramento, California, United States of America.
PLoS ONE (Impact Factor: 3.23). 04/2010; 5(4):e10072. DOI: 10.1371/journal.pone.0010072
Source: PubMed


Editorial peer review is universally used but little studied. We examined the relationship between external reviewers' recommendations and the editorial outcome of manuscripts undergoing external peer-review at the Journal of General Internal Medicine (JGIM).
We examined reviewer recommendations and editors' decisions at JGIM between 2004 and 2008. For manuscripts undergoing peer review, we calculated chance-corrected agreement among reviewers on recommendations to reject versus accept or revise. Using mixed effects logistic regression models, we estimated intra-class correlation coefficients (ICC) at the reviewer and manuscript level. Finally, we examined the probability of rejection in relation to reviewer agreement and disagreement. The 2264 manuscripts sent for external review during the study period received 5881 reviews provided by 2916 reviewers; 28% of reviews recommended rejection. Chance corrected agreement (kappa statistic) on rejection among reviewers was 0.11 (p<.01). In mixed effects models adjusting for study year and manuscript type, the reviewer-level ICC was 0.23 (95% confidence interval [CI], 0.19-0.29) and the manuscript-level ICC was 0.17 (95% CI, 0.12-0.22). The editors' overall rejection rate was 48%: 88% when all reviewers for a manuscript agreed on rejection (7% of manuscripts) and 20% when all reviewers agreed that the manuscript should not be rejected (48% of manuscripts) (p<0.01).
Reviewers at JGIM agreed on recommendations to reject vs. accept/revise at levels barely beyond chance, yet editors placed considerable weight on reviewers' recommendations. Efforts are needed to improve the reliability of the peer-review process while helping editors understand the limitations of reviewers' recommendations.

Download full-text


Available from: Richard L Kravitz, Oct 02, 2015
14 Reads
  • Source
    • "In face of these problems, many suggestions have been proposed to make the peer review and editorial process more efficient and equitable [5]. In particular, the role of editors in the process of selecting and managing reviewers has been increasingly discussed [8] [9] [10]. The main focus of these discussions are ethical issues and general, qualitative recommendations for both the editors and the reviewers [6] [7] [11] [12]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: We examine selected aspects of peer review and suggest possible improvements. To this end, we analyse a dataset containing information about 300 papers submitted to the Biochemistry and Biotechnology section of the Journal of the Serbian Chemical Society. After separating the peer review process into stages that each review has to go through, we use a weighted directed graph to describe it in a probabilistic manner and test the impact of some modifications of the editorial policy on the efficiency of the whole process.
  • Source
    • "Whether or not decisions of journal editors are influenced by recommendations of peer reviewers depends on the type of recommendation. If reviewers agree that a manuscript should be rejected, their recommendation is generally followed by the editors (Kravitz et al. 2010; Sposato et al. 2014). For other recommendations , the degree of concordance is modest. "
    [Show abstract] [Hide abstract]
    ABSTRACT: After an examination of research integrity in China, the journal Science concluded that there is a flourishing black market in publications (Hvistendahl 2013). For fees ranging from $1600 to $26,300 authorship in science citation index (SCI) journals is for sale. Shady companies are trading in SCI papers. Chinese regulatory agencies are concerned about global influence and the reputation of Chinese science. They have taken initiatives to cultivate research ethics through education and codes of conduct (Yang 2013). SCI papers are the basis of promotion in many universities; they also lead to privileges and financial rewards. In her study on bioethics governance in China, Zhang (2012) noticed that some Chinese scholars recall the impact factor of their publications but not the name of the journals in which they appeared.This situation is not exceptional. It occurs in many other countries. One of the underlying mechanisms is the blind faith in quantitative measures for scientific output. ...
    Medicine Health Care and Philosophy 02/2015; 18(2). DOI:10.1007/s11019-015-9626-9 · 0.91 Impact Factor
  • Source
    • "A number of large studies on IRR for peer reviewers have been published. One study examined 5,881 reviewer reports for the Journal of General Internal Medicine from 2004 to 2008 and found that IRR for the overall decision (reject vs. accept/ revise) was barely better than chance (Kravitz et al. 2010). Another study examined IRR for papers submitted to two neuroscience journals. "
    [Show abstract] [Hide abstract]
    ABSTRACT: A growing body of literature has identified potential problems that can compromise the quality, fairness, and integrity of journal peer review, including inadequate review, inconsistent reviewer reports, reviewer biases, and ethical transgressions by reviewers. We examine the evidence concerning these problems and discuss proposed reforms, including double-blind and open review. Regardless of the outcome of additional research or attempts at reforming the system, it is clear that editors are the linchpin of peer review, since they make decisions that have a significant impact on the process and its outcome. We consider some of the steps editors should take to promote quality, fairness and integrity in different stages of the peer review process and make some recommendations for editorial conduct and decision-making.
    Science and Engineering Ethics 01/2015; DOI:10.1007/s11948-015-9625-5 · 0.96 Impact Factor
Show more