Editorial Peer Reviewers' Recommendations at a General Medical Journal: Are They Reliable and Do Editors Care?

Department of Medicine, University of California Davis, Sacramento, California, United States of America.
PLoS ONE (Impact Factor: 3.53). 04/2010; 5(4):e10072. DOI: 10.1371/journal.pone.0010072
Source: PubMed

ABSTRACT Editorial peer review is universally used but little studied. We examined the relationship between external reviewers' recommendations and the editorial outcome of manuscripts undergoing external peer-review at the Journal of General Internal Medicine (JGIM).
We examined reviewer recommendations and editors' decisions at JGIM between 2004 and 2008. For manuscripts undergoing peer review, we calculated chance-corrected agreement among reviewers on recommendations to reject versus accept or revise. Using mixed effects logistic regression models, we estimated intra-class correlation coefficients (ICC) at the reviewer and manuscript level. Finally, we examined the probability of rejection in relation to reviewer agreement and disagreement. The 2264 manuscripts sent for external review during the study period received 5881 reviews provided by 2916 reviewers; 28% of reviews recommended rejection. Chance corrected agreement (kappa statistic) on rejection among reviewers was 0.11 (p<.01). In mixed effects models adjusting for study year and manuscript type, the reviewer-level ICC was 0.23 (95% confidence interval [CI], 0.19-0.29) and the manuscript-level ICC was 0.17 (95% CI, 0.12-0.22). The editors' overall rejection rate was 48%: 88% when all reviewers for a manuscript agreed on rejection (7% of manuscripts) and 20% when all reviewers agreed that the manuscript should not be rejected (48% of manuscripts) (p<0.01).
Reviewers at JGIM agreed on recommendations to reject vs. accept/revise at levels barely beyond chance, yet editors placed considerable weight on reviewers' recommendations. Efforts are needed to improve the reliability of the peer-review process while helping editors understand the limitations of reviewers' recommendations.

Download full-text


Available from: Richard L Kravitz, Jul 13, 2015
  • Source
    • "A number of large studies on IRR for peer reviewers have been published. One study examined 5,881 reviewer reports for the Journal of General Internal Medicine from 2004 to 2008 and found that IRR for the overall decision (reject vs. accept/ revise) was barely better than chance (Kravitz et al. 2010). Another study examined IRR for papers submitted to two neuroscience journals. "
    [Show abstract] [Hide abstract]
    ABSTRACT: A growing body of literature has identified potential problems that can compromise the quality, fairness, and integrity of journal peer review, including inadequate review, inconsistent reviewer reports, reviewer biases, and ethical transgressions by reviewers. We examine the evidence concerning these problems and discuss proposed reforms, including double-blind and open review. Regardless of the outcome of additional research or attempts at reforming the system, it is clear that editors are the linchpin of peer review, since they make decisions that have a significant impact on the process and its outcome. We consider some of the steps editors should take to promote quality, fairness and integrity in different stages of the peer review process and make some recommendations for editorial conduct and decision-making.
    Science and Engineering Ethics 01/2015; DOI:10.1007/s11948-015-9625-5
  • Source
    • "Editors rely heavily (sometimes almost exclusively) on the opinions of peer reviewers, even though they may not know the reviewers well enough to gauge their capability (Wilkes and Kravitz 1995). Agreement between reviewers has also been shown to be poor and not better than what can be expected by chance alone (Rothwell and Martyn 2000; Kravitz et al. 2010; Coronel and Opthof 1999). Agreements between reviewers and editors are also not high and are probably more for rejection than acceptance (Howard and Wilkinson 1999). "
    [Show abstract] [Hide abstract]
    ABSTRACT: With the ever expanding array of professional journals, pressures on the peer review process have increased considerably. Unless editors and publishers recognize the need for improving the efficiency of the process, the future of traditional peer review may be at risk. This is a review of the studies that have followed up the suggestions made by Ingelfinger in 1974 for improvement of manuscript peer review. Implementation of changes has been slow, despite the abundance of literature that suggests the necessary improvements. Conscientious self-regulation is expected of editors who, in the current publication scenario, possess enor- mous power without liability. Suitability of peer review to outsourcing should be assessed and if it is absolutely essential to outsource peer review (due to financial constraints on the publisher), care should be taken to ensure that it is implemented systematically and monitored regularly for quality. Finally, it is time for high earning publishers to consider compensation (financial or otherwise) for the efforts of the reviewers.
    Journal of Academic Ethics 10/2014; DOI:10.1007/s10805-014-9220-4
  • Source
    • "Previous studies have tried to measure the effect of peer review on the quality of publications and referees' reports as well as time and costs of different review processes (Jefferson et al. 2002). Other studies have examined the impact of peer review on authors' satisfaction (Weber et al. 2002), the motivation of referees (Squazzoni et al. 2013) and the editors' approach (Neff and Olden 2006; Kravitz et al. 2010; Newton 2010). A recent large-scale international online survey (Mulligan et al. 2012) called for initiatives to increase the quality of of the process. "
    [Show abstract] [Hide abstract]
    ABSTRACT: This paper presents an agent-based model of peer review that looks at the effect of different editorial policies of referee selection. We tested four author/referee matching scenarios as follows: random selection of referees, selection of referees with a similar status to submission authors, selection of higher-skilled and lower skilled referees. We tested these scenarios against three types of referee behaviour, i.e., fair, unreliable and strategic and measured their implications for the quality and efficiency of the process. Results show that in case of randomness of referee judgment, any editorial policy is detrimental for peer review. If referees behave strategically, certain matching policies, such as selecting referees of good quality, might counteract possible bias.
    ECMS European Conference on Modelling and Simulation 2014, University of Brescia; 05/2014
Show more