Content uploaded by Ali H. Al-Hoorie
Author content
All content in this area was uploaded by Ali H. Al-Hoorie on Nov 20, 2021
Content may be subject to copyright.
PEER REVIEWERS
EVALUATING THE EVALUATORS
Ali H. Al-Hoorie
Presented at
The 1st International Symposium on Educational Research
Prince Sultan University, Saudi Arabia
20-Nov-21
OUTLINE
•What is the purpose of peer review?
•Is peer review a valid tool?
•Why or why not? Empirical evidence?
•Are there alternatives?
PURPOSE OF PEER REVIEW
•Quality assurance (unintentional errors)
•Gatekeeping
•Epistemic filter
•Detecting fraud (deliberate falsification)
•Reviewer 2?
TWO MAIN FUNCTIONS
MEAN REVIEWERS
MEAN REVIEWERS
VALIDITY OF PEER REVIEW
EMPIRICAL EVIDENCE
See also
Ceci & Peters (1982)
•Selected 13 articles
•Published in top psychology journals (non-blind)
•Authored by researchers from prestigious institutions (e.g., Harvard, Stanford)
•Replaced the author names with fake names from fake low-status sounding institutions
•E.g., Tri-Valley Center for Human Potential
•Resubmitted these articles to the same journals that had accepted them
•If the peer review is valid, these articles should be accepted again
•Especially since this is the final (& best version) of each article
•10 articles were not detected (as resubmissions) and were sent out for review
DESIGN
•“Of the ten undetected manuscripts, nine were recommended for rejection resoundingly; that
is, both reviewers were in agreement on rejection. None of the twenty reviewers who
recommended rejection even hinted at the possibility that a manuscript might be acceptable
for publication pending revision or rewriting.”
•“When we removed the original affiliation (e.g., Harvard) and replaced it with the bogus Tri-
Valley Center for Human Development affiliation, the previously acclaimed works were
sharply criticized (almost always on statistical or design grounds).” (Ceci & Peters, 1982, p.
46)
•Their conclusion: “These findings were a convincing demonstration of the unreliability of
peer review.”
•(Ceci & Peters, 1982, p. 46)
RESULTS
REACTION
•“One editor in the study wrote a letter threatening a lawsuit for copyright violations. Actually, we
had obtained permission from the original authors and copyright holders (publishers) to use
their materials in our study.”
•“Quite unexpectedly, several editors who had not been directly involved with our study wrote
scathing letters calling into question our professional ethics because of our use of deception.”
•“when our chairman was personally contacted by an angry editor, he withdrew all of our
departmental resources until we finished the study. We were sent a memorandum informing us
that typists, photocopy, mail, etc. were ‘off limits’ to us as long as we continued using the
procedures we had adopted.”
•Their conclsion: “Obviously, the journal review system has become a ‘sacred cow’ to some.”
•(Ceci & Peters, 1982, pp. 46–48)
COUNTER-REACTION
•“These personal attacks took their toll. For a couple of years we doubted the wisdom of
our decision to do the research. Finally, after two unsuccessful attempts to publish our
findings, replete with personally insulting, ad hominen reviews, we found a publisher and
positive reviews.”
•“Soon press releases were telling a diverse audience of our findings.”
•“Letters of support (over one thousand) came pouring in. Every one of them was
complimentary.”
•(Ceci & Peters, 1982, p. 47)
INTER-REVIEWER RELIABILITY
INTER-REVIEWER RELIABILITY
META-ANALYSIS
META-ANALYSIS
“If researchers wrote up the peer review process as a measure and submitted it for a journal for
publication, it would get rejected because of its unreliability” Brian Nosek
INNOCENT QUESTIONS
•Do peer reviewers receive any training?
•Do they receive any payment?
•Do they have time available?
•Do editors follow a systematic way to choose reviewers?
•Do they use a valid instrument? Or subjective judgment?
NOT ONLY POOR, BUT ALSO EXPLOITATIVE
NOT ONLY POOR, BUT ALSO EXPLOITATIVE
ALTERNATIVES
SOLUTION 1
OUTSOURCE PEER REVIEW
PCI RR
PCI RR PROCESS
•Write a (detailed) proposal of your study
•Submit it to a preprint repository
•Peers review it, give feedback, & recommended
•Conferred “in-principle acceptance”
•Called Stage 1 Review
•Now you conduct your study
•Adhering to the original proposal
•Submit the full manuscript for review again
•Reviewers now check adherence to original proposal
•Called Stage 2 Review
•Now submit to a PCI RR-friendly journal
•Manuscript will be accepted WITHUOT further review by the journal
•Peer review is outsourced
PCI RR PROCESS
PCI RR BENEFITS
•Feedback before study is conducted
•Problems can be fixed
•Emphasize design, deemphasize outcome
•Minimize biases
•Lowers pressure on researchers
•Peer reviewers give feedback to help you
•Everything is public
•Increases transparency
•Reviewers can get credit (e.g., review content is not a secret)
SOLUTION 2
ABOLISH (PRE-PUBLICATION) PEER REVIEW
POST-PUBLICATION PEER REVIEW
POST-PUBLICATION PEER REVIEW PROCESS
•Authors submit their manuscript to a preprint repository
•This counts as publication
•Review occurs afterward
•Everything is public
•Authors may revise and update their manuscripts
•Benefits:
•Accelerates dissemination of information
•Separates publication from evaluation
•So the incentive is no longer to “pass” peer review & to get published
•Authors can focus on impact and on improving their work
•See also Al-Hoorie and Hiver (in press)
POST-PUBLICATION PEER REVIEW PROBLEMS
•Matthew effect
•The rich get richer, the poor get poorer
•Famous scholars will receive the most attention
•Guarantee for outsiders
•Journalists, general public
•Hessel and Bright (2020) propose solutions
FURTHER READING
Annual Review of Information Science and Technology (2011)
The British Journal for the Philosophy of Science (2020)
REFERENCES
•Aczel, B., Szaszi, B., & Holcombe, A.O. (2021). A billion-dollar donation: Estimating the cost of researchers’ time spent on peer review. Research Integrity and Peer Review,
6(14). https://doi.org/10.1186/s41073-021-00118-2
•Al-Hoorie, A. H., & Hiver, P. (in press). Open science in applied linguistics: An introduction to metascience. In Plonsky, L. (Ed.), Open science in applied linguistics. John
Benjamins.
•Bornmann, L. (2011). Scientific peer review. Annual Review of Information Science and Technology. 45,197–245. https://doi.org/10.1002/aris.2011.1440450112
•Bornmann, L., Mutz, R., Daniel, H.-D. (2010). A reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its
determinants. PLoS ONE, 5(12), e14331. https://doi.org/10.1371/journal.pone.0014331
•Ceci, S. J., & Peters, D. P. (1982) Peer review: A study of reliability. Change: The Magazine of Higher Learning, 14(6), 44–48,.
https://doi.org/10.1080/00091383.1982.10569910
•Frey, B. S. (2003). Publishing as prostitution? –Choosing between one’s own ideas and academic success. Public Choice, 116, 205–223.
https://doi.org/10.1023/A:1024208701874
•Heesen, R., & Bright, L. K. (2020). Is peer review a good idea? The British Journal for the Philosophy of Science, 72(3), 635–663. https://doi.org/10.1093/bjps/axz029
•Jiang, S. (2021). Understanding authors' psychological reactions to peer reviews: A text mining approach. Scientometrics, 126, 6085–6103.
https://doi.org/10.1007/s11192-021-04032-8
•Kravitz, R. L., Franks, P., Feldman, M.D., Gerrity, M., Byrne, C., Tierney, W.M. (2010). Editorial peer reviewers’ recommendations at a general medical journal: Are they
reliable and do editors care? PLoS ONE, 5(4): e10072. https://doi.org/10.1371/journal.pone.0010072
•Merriman, B. (2021). Peer review as an evolving response to organizational constraint: Evidence from sociology journals, 1952–2018. The American Sociologist, 52, 341–
366. https://doi.org/10.1007/s12108-020-09473-x
•Nosek, B. A. & Bar-Anan, Y. (2012) Scientific Utopia: I. Opening Scientific Communication. Psychological Inquiry, 23(3), 217–243.
https://doi.org/10.1080/1047840X.2012.692215
•Peters, D. P., & Ceci, S. J. (1982). Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences, 5(2), 187–
255. https://doi.org/10.1017/S0140525X00011183
•Smith, R. (2021). Richard Smith: Peer reviewers—Time for mass rebellion? The BMJ Opinion. https://blogs.bmj.com/bmj/2021/02/01/richard-smith-peer-reviewers-time-
for-mass-rebellion/
Thank You
@Ali_AlHoorie
hoorie_ali@hotmail.com
www.ali-alhoorie.com