ArticlePDF Available

Peer Review in Scientific Publications: Benefits, Critiques, & A Survival Guide

Authors:

Abstract and Figures

Peer review has been defined as a process of subjecting an author's scholarly work, research or ideas to the scrutiny of others who are experts in the same field. It functions to encourage authors to meet the accepted high standards of their discipline and to control the dissemination of research data to ensure that unwarranted claims, unacceptable interpretations or personal views are not published without prior expert review. Despite its wide-spread use by most journals, the peer review process has also been widely criticised due to the slowness of the process to publish new findings and due to perceived bias by the editors and/or reviewers. Within the scientific community, peer review has become an essential component of the academic writing process. It helps ensure that papers published in scientific journals answer meaningful research questions and draw accurate conclusions based on professionally executed experimentation. Submission of low quality manuscripts has become increasingly prevalent, and peer review acts as a filter to prevent this work from reaching the scientific community. The major advantage of a peer review process is that peer-reviewed articles provide a trusted form of scientific communication. Since scientific knowledge is cumulative and builds on itself, this trust is particularly important. Despite the positive impacts of peer review, critics argue that the peer review process stifles innovation in experimentation, and acts as a poor screen against plagiarism. Despite its downfalls, there has not yet been a foolproof system developed to take the place of peer review, however, researchers have been looking into electronic means of improving the peer review process. Unfortunately, the recent explosion in online only/electronic journals has led to mass publication of a large number of scientific articles with little or no peer review. This poses significant risk to advances in scientific knowledge and its future potential. The current article summarizes the peer review process, highlights the pros and cons associated with different types of peer review, and describes new methods for improving peer review.
No caption available
… 
Content may be subject to copyright.
In this issue: Focusing on Publicaon Ethics
Peer review in scientic publications:
benets, critiques, & a survival guide
Jacalyn Kelly1, Tara Sadeghieh1, Khosrow Adeli1,2,3
1 Clinical Biochemistry, Department of Pediatric Laboratory Medicine, The Hospital for Sick Children,
University of Toronto, Toronto, Ontario, Canada
2 Department of Laboratory Medicine and Pathobiology, University of Toronto, Toronto, Canada
3 Chair, Communicaons and Publicaons Division (CPD) , Internaonal Federaon for Sick Clinical
Chemistry (IFCC), Milan, Italy
ARTICLE INFO ABSTRACT
Corresponding author:
Khosrow Adeli
Clinical Biochemistry
The Hospital for Sick Children
University of Toronto
Toronto, Ontario
Canada, M5G 1X8
E-mail: khosrow.adeli@sickkids.ca
Disclosure
The authors declare no conicts of interest
regarding publicaon of this arcle.
Key words:
peer review, manuscript,
publication, journal, open access
Peer review has been dened as a process of subjecng an
author’s scholarly work, research or ideas to the scruny
of others who are experts in the same eld. It funcons to
encourage authors to meet the accepted high standards of
their discipline and to control the disseminaon of research
data to ensure that unwarranted claims, unacceptable inter-
pretaons or personal views are not published without pri-
or expert review. Despite its wide-spread use by most jour-
nals, the peer review process has also been widely cricised
due to the slowness of the process to publish new ndings
and due to perceived bias by the editors and/or reviewers.
Within the scienc community, peer review has become
an essenal component of the academic wring process.
It helps ensure that papers published in scienc journals
answer meaningful research quesons and draw accurate
conclusions based on professionally executed experimen-
taon. Submission of low quality manuscripts has become
increasingly prevalent, and peer review acts as a lter to
prevent this work from reaching the scienc community.
The major advantage of a peer review process is that peer-
reviewed arcles provide a trusted form of scienc com-
municaon. Since scienc knowledge is cumulave and
builds on itself, this trust is parcularly important. Despite
the posive impacts of peer review, crics argue that the
peer review process ses innovaon in experimentaon,
Peer review in scienc publicaons: benets, criques, & a survival guide
Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli
Page 227
eJIFCC2014Vol25No3pp227-243
Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli
Peer review in scienc publicaons: benets, criques, & a survival guide
and acts as a poor screen against plagiarism.
Despite its downfalls, there has not yet been a
foolproof system developed to take the place of
peer review, however, researchers have been
looking into electronic means of improving the
peer review process. Unfortunately, the recent
explosion in online only/electronic journals has
led to mass publicaon of a large number of sci-
enc arcles with lile or no peer review. This
poses signicant risk to advances in scienc
knowledge and its future potenal. The current
arcle summarizes the peer review process,
highlights the pros and cons associated with dif-
ferent types of peer review, and describes new
methods for improving peer review.
WHAT IS PEER REVIEW
AND WHAT I S ITS PURPOSE?
Peer Review is dened as “a process of sub-
jecng an author’s scholarly work, research or
ideas to the scruny of others who are experts
in the same eld” (1). Peer review is intended
to serve two primary purposes. Firstly, it acts as
a lter to ensure that only high quality research
is published, especially in reputable journals,
by determining the validity, signicance and
originality of the study. Secondly, peer review
is intended to improve the quality of manu-
scripts that are deemed suitable for publicaon.
Peer reviewers provide suggesons to authors
on how to improve the quality of their manu-
scripts, and also idenfy any errors that need
correcng before publicaon.
HISTORY OF PEER REVIEW
The concept of peer review was developed long
before the scholarly journal. In fact, the peer re-
view process is thought to have been used as
a method of evaluang wrien work since an-
cient Greece (2). The peer review process was
rst described by a physician named Ishaq bin
Ali al-Rahwi of Syria, who lived from 854-931
CE, in his book Ethics of the Physician (2). There,
he stated that physicians must take notes de-
scribing the state of their paents’ medical con-
dions upon each visit. Following treatment,
the notes were scrunized by a local medical
council to determine whether the physician had
met the required standards of medical care. If
the medical council deemed that the appropri-
ate standards were not met, the physician in
queson could receive a lawsuit from the mal-
treated paent (2).
The invenon of the prinng press in 1453 al-
lowed wrien documents to be distributed to
the general public (3). At this me, it became
more important to regulate the quality of the
wrien material that became publicly available,
and eding by peers increased in prevalence.
In 1620, Francis Bacon wrote the work Novum
Organum, where he described what eventually
became known as the rst universal method for
generang and assessing new science (3). His
work was instrumental in shaping the Scienc
Method (3). In 1665, the French Journal des sça-
vans and the English Philosophical Transacons
of the Royal Society were the rst scienc jour-
nals to systemacally publish research results
(4). Philosophical Transacons of the Royal Soci-
ety is thought to be the rst journal to formalize
the peer review process in 1665 (5), however,
it is important to note that peer review was ini-
ally introduced to help editors decide which
manuscripts to publish in their journals, and at
that me it did not serve to ensure the valid-
ity of the research (6). It did not take long for
the peer review process to evolve, and shortly
thereaer papers were distributed to reviewers
with the intent of authencang the integrity of
the research study before publicaon. The Roy-
al Society of Edinburgh adhered to the following
peer review process, published in their Medical
Essays and Observaons in 1731: “Memoirs
sent by correspondence are distributed accord-
ing to the subject maer to those members who
Page 228
eJIFCC2014Vol25No3pp227-243
Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli
Peer review in scienc publicaons: benets, criques, & a survival guide
are most versed in these maers. The report
of their identy is not known to the author.”
(7). The Royal Society of London adopted this
review procedure in 1752 and developed the
“Commiee on Papers” to review manuscripts
before they were published in Philosophical
Transacons (6).
Peer review in the systemazed and instuon-
alized form has developed immensely since the
Second World War, at least partly due to the
large increase in scienc research during this
period (7). It is now used not only to ensure that
a scienc manuscript is experimentally and
ethically sound, but also to determine which
papers suciently meet the journal’s standards
of quality and originality before publicaon.
Peer review is now standard pracce by most
credible scienc journals, and is an essenal
part of determining the credibility and quality
of work submied.
IMPACT OF THE PEER REVIEW PROCESS
Peer review has become the foundaon of the
scholarly publicaon system because it eecve-
ly subjects an author’s work to the scruny of
other experts in the eld. Thus, it encourages au-
thors to strive to produce high quality research
that will advance the eld. Peer review also sup-
ports and maintains integrity and authencity in
the advancement of science. A scienc hypoth-
esis or statement is generally not accepted by
the academic community unless it has been pub-
lished in a peer-reviewed journal (8). The Ins-
tute for Scienc Informaon (ISI) only considers
journals that are peer-reviewed as candidates
to receive Impact Factors. Peer review is a well-
established process which has been a formal part
of scienc communicaon for over 300 years.
OVERVIEW OF THE PEER REVIEW PROCESS
The peer review process begins when a scien-
st completes a research study and writes a
manuscript that describes the purpose, experi-
mental design, results, and conclusions of the
study. The scienst then submits this paper to
a suitable journal that specializes in a relevant
research eld, a step referred to as pre-submis-
sion. The editors of the journal will review the
paper to ensure that the subject maer is in line
with that of the journal, and that it ts with the
editorial plaorm. Very few papers pass this ini-
al evaluaon. If the journal editors feel the pa-
per suciently meets these requirements and
is wrien by a credible source, they will send
the paper to accomplished researchers in the
eld for a formal peer review. Peer reviewers
are also known as referees (this process is sum-
marized in Figure 1). The role of the editor is to
select the most appropriate manuscripts for the
journal, and to implement and monitor the peer
review process. Editors must ensure that peer
reviews are conducted fairly, and in an eecve
and mely manner. They must also ensure that
there are no conicts of interest involved in the
peer review process.
When a reviewer is provided with a paper, he or
she reads it carefully and scrunizes it to evalu-
ate the validity of the science, the quality of the
experimental design, and the appropriateness
of the methods used. The reviewer also assess-
es the signicance of the research, and judges
whether the work will contribute to advance-
ment in the eld by evaluang the importance
of the ndings, and determining the originality
of the research. Addionally, reviewers iden-
fy any scienc errors and references that are
missing or incorrect. Peer reviewers give rec-
ommendaons to the editor regarding whether
the paper should be accepted, rejected, or im-
proved before publicaon in the journal. The
editor will mediate author-referee discussion
in order to clarify the priority of certain referee
requests, suggest areas that can be strength-
ened, and overrule reviewer recommenda-
ons that are beyond the study’s scope (9). If
Page 229
eJIFCC2014Vol25No3pp227-243
Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli
Peer review in scienc publicaons: benets, criques, & a survival guide
Figure 1 Overview of the review process
the paper is accepted, as per suggeson by the
peer reviewer, the paper goes into the produc-
on stage, where it is tweaked and formaed
by the editors, and nally published in the sci-
enc journal. An overview of the review pro-
cess is presented in Figure 1.
WHO CONDUCTS REVIEWS?
Peer reviews are conducted by scienc experts
with specialized knowledge on the content of
the manuscript, as well as by sciensts with a
more general knowledge base. Peer review-
ers can be anyone who has competence and
Page 230
eJIFCC2014Vol25No3pp227-243
Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli
Peer review in scienc publicaons: benets, criques, & a survival guide
experse in the subject areas that the journal
covers. Reviewers can range from young and
up-and-coming researchers to old masters in
the eld. Oen, the young reviewers are the
most responsive and deliver the best quality
reviews, though this is not always the case. On
average, a reviewer will conduct approximately
eight reviews per year, according to a study on
peer review by the Publishing Research Consor-
um (PRC) (7). Journals will oen have a pool of
reviewers with diverse backgrounds to allow for
many dierent perspecves. They will also keep
a rather large reviewer bank, so that review-
ers do not get burnt out, overwhelmed or me
constrained from reviewing mulple arcles
simultaneously.
WHY DO REVIEWERS REVIEW?
Referees are typically not paid to conduct peer
reviews and the process takes considerable ef-
fort, so the queson is raised as to what incen-
ve referees have to review at all. Some feel an
academic duty to perform reviews, and are of
the mentality that if their peers are expected
to review their papers, then they should review
the work of their peers as well. Reviewers may
also have personal contacts with editors, and
may want to assist as much as possible. Oth-
ers review to keep up-to-date with the latest
developments in their eld, and reading new
scienc papers is an eecve way to do so.
Some sciensts use peer review as an opportu-
nity to advance their own research as it smu-
lates new ideas and allows them to read about
new experimental techniques. Other reviewers
are keen on building associaons with pres-
gious journals and editors and becoming part of
their community, as somemes reviewers who
show dedicaon to the journal are later hired
as editors. Some sciensts see peer review as a
chance to become aware of the latest research
before their peers, and thus be rst to develop
new insights from the material. Finally, in terms
of career development, peer reviewing can be
desirable as it is oen noted on one’s resume or
CV. Many instuons consider a researcher’s in-
volvement in peer review when assessing their
performance for promoons (11). Peer review-
ing can also be an eecve way for a scienst to
show their superiors that they are commied to
their scienc eld (5).
ARE REVIEWERS KEEN TO REVIEW?
A 2009 internaonal survey of 4000 peer re-
viewers conducted by the charity Sense About
Science at the Brish Science Fesval at the
University of Surrey, found that 90% of review-
ers were keen to peer review (12). One third of
respondents to the survey said they were happy
to review up to ve papers per year, and an ad-
dional one third of respondents were happy to
review up to ten.
HOW LONG DOES IT TAKE
TO REVIEW ONE PAPER?
On average, it takes approximately six hours
to review one paper (12), however, this num-
ber may vary greatly depending on the con-
tent of the paper and the nature of the peer
reviewer. One in every 100 parcipants in the
“Sense About Science” survey claims to have
taken more than 100 hours to review their last
paper (12).
HOW TO DETERMINE
IF A JOURNAL IS PEER REVIEWED
Ulrichsweb is a directory that provides informa-
on on over 300,000 periodicals, including in-
formaon regarding which journals are peer re-
viewed (13). Aer logging into the system using
an instuonal login (eg. from the University
of Toronto), search terms, journal tles or ISSN
numbers can be entered into the search bar.
The database provides the tle, publisher, and
country of origin of the journal, and indicates
Page 231
eJIFCC2014Vol25No3pp227-243
Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli
Peer review in scienc publicaons: benets, criques, & a survival guide
whether the journal is sll acvely publishing.
The black book symbol (labelled ‘refereed’) re-
veals that the journal is peer reviewed.
THE EVALUATION CRITERIA
FOR PEER REVIEW OF SCIENTIFIC PAPERS
As previously menoned, when a reviewer re-
ceives a scienc manuscript, he/she will rst
determine if the subject maer is well suited
for the content of the journal. The reviewer will
then consider whether the research queson is
important and original, a process which may be
aided by a literature scan of review arcles.
Scienc papers submied for peer review usu-
ally follow a specic structure that begins with
the tle, followed by the abstract, introducon,
methodology, results, discussion, conclusions,
and references. The tle must be descripve
and include the concept and organism inves-
gated, and potenally the variable manipu-
lated and the systems used in the study. The
peer reviewer evaluates if the tle is descripve
enough, and ensures that it is clear and concise.
A study by the Naonal Associaon of Realtors
(NAR) published by the Oxford University Press
in 2006 indicated that the tle of a manuscript
plays a signicant role in determining reader in-
terest, as 72% of respondents said they could
usually judge whether an arcle will be of inter-
est to them based on the tle and the author,
while 13% of respondents claimed to always be
able to do so (14).
The abstract is a summary of the paper, which
briey menons the background or purpose,
methods, key results, and major conclusions of
the study. The peer reviewer assesses whether
the abstract is suciently informave and if the
content of the abstract is consistent with the
rest of the paper. The NAR study indicated that
40% of respondents could determine whether
an arcle would be of interest to them based
on the abstract alone 60-80% of the me, while
32% could judge an arcle based on the ab-
stract 80-100% of the me (14). This demon-
strates that the abstract alone is oen used to
assess the value of an arcle.
The introducon of a scienc paper presents
the research queson in the context of what
is already known about the topic, in order to
idenfy why the queson being studied is of
interest to the scienc community, and what
gap in knowledge the study aims to ll (15). The
introducon idenes the studys purpose and
scope, briey describes the general methods of
invesgaon, and outlines the hypothesis and
predicons (15). The peer reviewer determines
whether the introducon provides sucient
background informaon on the research topic,
and ensures that the research queson and hy-
pothesis are clearly idenable.
The methods secon describes the experimen-
tal procedures, and explains why each experi-
ment was conducted. The methods secon also
includes the equipment and reagents used in
the invesgaon. The methods secon should
be detailed enough that it can be used it to re-
peat the experiment (15). Methods are wrien
in the past tense and in the acve voice. The
peer reviewer assesses whether the appropri-
ate methods were used to answer the research
queson, and if they were wrien with su-
cient detail. If informaon is missing from the
methods secon, it is the peer reviewer’s job to
idenfy what details need to be added.
The results secon is where the outcomes of
the experiment and trends in the data are ex-
plained without judgement, bias or interpre-
taon (15). This secon can include stascal
tests performed on the data, as well as gures
and tables in addion to the text. The peer re-
viewer ensures that the results are described
with sucient detail, and determines their
credibility. Reviewers also conrm that the text
is consistent with the informaon presented in
Page 232
eJIFCC2014Vol25No3pp227-243
Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli
Peer review in scienc publicaons: benets, criques, & a survival guide
tables and gures, and that all gures and ta-
bles included are important and relevant (15).
The peer reviewer will also make sure that table
and gure capons are appropriate both con-
textually and in length, and that tables and g-
ures present the data accurately.
The discussion secon is where the data is an-
alyzed. Here, the results are interpreted and
related to past studies (15). The discussion
describes the meaning and signicance of the
results in terms of the research queson and
hypothesis, and states whether the hypothesis
was supported or rejected. This secon may
also provide possible explanaons for unusual
results and suggesons for future research (15).
The discussion should end with a conclusions
secon that summarizes the major ndings of
the invesgaon. The peer reviewer determines
whether the discussion is clear and focused,
and whether the conclusions are an appropri-
ate interpretaon of the results. Reviewers also
ensure that the discussion addresses the limi-
taons of the study, any anomalies in the re-
sults, the relaonship of the study to previous
research, and the theorecal implicaons and
praccal applicaons of the study.
The references are found at the end of the pa-
per, and list all of the informaon sources cited
in the text to describe the background, meth-
ods, and/or interpret results. Depending on the
citaon method used, the references are listed
in alphabecal order according to author last
name, or numbered according to the order in
which they appear in the paper. The peer re-
viewer ensures that references are used appro-
priately, cited accurately, formaed correctly,
and that none are missing.
Finally, the peer reviewer determines whether
the paper is clearly wrien and if the content
seems logical. Aer thoroughly reading through
the enre manuscript, they determine whether
it meets the journal’s standards for publicaon,
and whether it falls within the top 25% of papers
in its eld (16) to determine priority for publica-
on. An overview of what a peer reviewer looks
for when evaluang a manuscript, in order of
importance, is presented in Figure 2.
To increase the chance of success in the peer
review process, the author must ensure that
the paper fully complies with the journal guide-
lines before submission. The author must also
be open to cricism and suggested revisions,
and learn from mistakes made in previous
submissions.
ADVANTAGES AND DISADVA NTAGES OF
THE DIFFERENT TYPES OF PEER REVIEW
The peer review process is generally conducted
in one of three ways: open review, single-blind
review, or double-blind review. In an open re-
view, both the author of the paper and the peer
reviewer know one another’s identy. Alter-
navely, in single-blind review, the reviewer’s
identy is kept private, but the author’s iden-
ty is revealed to the reviewer. In double-blind
review, the idenes of both the reviewer and
author are kept anonymous. Open peer review
is advantageous in that it prevents the reviewer
from leaving malicious comments, being care-
less, or procrasnang compleon of the re-
view (2). It encourages reviewers to be open
and honest without being disrespecul. Open
reviewing also discourages plagiarism amongst
authors (2). On the other hand, open peer re-
view can also prevent reviewers from being
honest for fear of developing bad rapport with
the author. The reviewer may withhold or tone
down their cricisms in order to be polite (2).
This is especially true when younger review-
ers are given a more esteemed author’s work,
in which case the reviewer may be hesitant to
provide cricism for fear that it will damper
their relaonship with a superior (2). Accord-
ing to the Sense About Science survey, editors
Page 233
eJIFCC2014Vol25No3pp227-243
Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli
Peer review in scienc publicaons: benets, criques, & a survival guide
nd that completely open reviewing decreases
the number of people willing to parcipate, and
leads to reviews of lile value (12). In the afore-
menoned study by the PRC, only 23% of au-
thors surveyed had experience with open peer
review (7).
Single-blind peer review is by far the most com-
mon. In the PRC study, 85% of authors surveyed
had experience with single-blind peer review (7).
This method is advantageous as the reviewer is
more likely to provide honest feedback when
their identy is concealed (2). This allows the
reviewer to make independent decisions with-
out the inuence of the author (2). The main
disadvantage of reviewer anonymity, howev-
er, is that reviewers who receive manuscripts
on subjects similar to their own research may
be tempted to delay compleng the review in
order to publish their own data rst (2).
Double-blind peer review is advantageous as
it prevents the reviewer from being biased
against the author based on their country of
origin or previous work (2). This allows the pa-
per to be judged based on the quality of the
content, rather than the reputaon of the au-
thor. The Sense About Science survey indicates
that 76% of researchers think double-blind
peer review is a good idea (12), and the PRC
survey indicates that 45% of authors have had
experience with double-blind peer review (7).
The disadvantage of double-blind peer review
is that, especially in niche areas of research,
it can somemes be easy for the reviewer to
determine the identy of the author based on
Figure 2 How a peer review evaluates a manuscript
Page 234
eJIFCC2014Vol25No3pp227-243
Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli
Peer review in scienc publicaons: benets, criques, & a survival guide
wring style, subject maer or self-citaon,
and thus, impart bias (2).
Masking the author’s identy from peer review-
ers, as is the case in double-blind review, is gen-
erally thought to minimize bias and maintain
review quality. A study by Jusce et al. in 1998
invesgated whether masking author identy
aected the quality of the review (17). One hun-
dred and eighteen manuscripts were random-
ized; 26 were peer reviewed as normal, and 92
were moved into the ‘intervenon’ arm, where
editor quality assessments were completed for
77 manuscripts and author quality assessments
were completed for 40 manuscripts (17). There
was no perceived dierence in quality between
the masked and unmasked reviews. Addion-
ally, the masking itself was oen unsuccessful,
especially with well-known authors (17). How-
ever, a previous study conducted by McNu et
al. had dierent results (18). In this case, blind-
ing was successful 73% of the me, and they
found that when author identy was masked,
the quality of review was slightly higher (18).
Although Jusce et al. argued that this dier-
ence was too small to be consequenal, their
study targeted only biomedical journals, and
the results cannot be generalized to journals
of a dierent subject maer (17). Addionally,
there were problems masking the idenes of
well-known authors, introducing a aw in the
methods. Regardless, Jusce et al. concluded
that masking author identy from reviewers
may not improve review quality (17).
In addion to open, single-blind and double-
blind peer review, there are two experimental
forms of peer review. In some cases, following
publicaon, papers may be subjected to post-
publicaon peer review. As many papers are
now published online, the scienc commu-
nity has the opportunity to comment on these
papers, engage in online discussions and post
a formal review. For example, online publish-
ers PLOS and BioMed Central have enabled
sciensts to post comments on published pa-
pers if they are registered users of the site (10).
Philica is another journal launched with this ex-
perimental form of peer review. Only 8% of au-
thors surveyed in the PRC study had experience
with post-publicaon review (7). Another ex-
perimental form of peer review called Dynamic
Peer Review has also emerged. Dynamic peer
review is conducted on websites such as Naboj,
which allow sciensts to conduct peer reviews
on arcles in the preprint media (19). The peer
review is conducted on repositories and is a
connuous process, which allows the public
to see both the arcle and the reviews as the
arcle is being developed (19). Dynamic peer
review helps prevent plagiarism as the scien-
c community will already be familiar with the
work before the peer reviewed version appears
in print (19). Dynamic review also reduces the
me lag between manuscript submission and
publishing. An example of a preprint server is
the ‘arXiv’ developed by Paul Ginsparg in 1991,
which is used primarily by physicists (19). These
alternave forms of peer review are sll un-
established and experimental. Tradional peer
review is me-tested and sll highly ulized. All
methods of peer review have their advantages
and deciencies, and all are prone to error.
PEER REVIEW OF OPEN ACCESS JOURNALS
Open access (OA) journals are becoming in-
creasingly popular as they allow the potenal
for widespread distribuon of publicaons in
a mely manner (20). Nevertheless, there can
be issues regarding the peer review process
of open access journals. In a study published
in Science in 2013, John Bohannon submied
304 slightly dierent versions of a conal sci-
enc paper (wrien by a fake author, working
out of a non-existent instuon) to a selected
group of OA journals. This study was performed
in order to determine whether papers sub-
mied to OA journals are properly reviewed
Page 235
eJIFCC2014Vol25No3pp227-243
Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli
Peer review in scienc publicaons: benets, criques, & a survival guide
before publicaon in comparison to subscrip-
on-based journals. The journals in this study
were selected from the Directory of Open Ac-
cess Journals (DOAJ) and Biall’s List, a list of
journals which are potenally predatory, and
all required a fee for publishing (21). Of the 304
journals, 157 accepted a fake paper, suggesng
that acceptance was based on nancial interest
rather than the quality of arcle itself, while 98
journals promptly rejected the fakes (21). Al-
though this study highlights useful informaon
on the problems associated with lower quality
publishers that do not have an eecve peer
review system in place, the arcle also general-
izes the study results to all OA journals, which
can be detrimental to the general percepon of
OA journals. There were two limitaons of the
study that made it impossible to accurately de-
termine the relaonship between peer review
and OA journals: 1) there was no control group
(subscripon-based journals), and 2) the fake
papers were sent to a non-randomized selec-
on of journals, resulng in bias.
JOURNAL ACCEPTANCE RATES
Based on a recent survey, the average accep-
tance rate for papers submied to scienc
journals is about 50% (7). Twenty percent of the
submied manuscripts that are not accepted
are rejected prior to review, and 30% are reject-
ed following review (7). Of the 50% accepted,
41% are accepted with the condion of revi-
sion, while only 9% are accepted without the
request for revision (7).
SATISFACTI ON WITH THE
PEER REVIEW SYSTEM
Based on a recent survey by the PRC, 64% of ac-
ademics are sased with the current system of
peer review, and only 12% claimed to be ‘dissat-
ised’ (7). The large majority, 85%, agreed with
the statement that ‘scienc communicaon is
greatly helped by peer review’ (7). There was a
similarly high level of support (83%) for the idea
that peer review ‘provides control in scienc
communicaon’ (7).
HOW TO PEER REVIEW EFFECTIVELY
The following are ten ps on how to be an eec-
ve peer reviewer as indicated by Brian Lucey,
an expert on the subject (22):
1) Be professional
Peer review is a mutual responsibility among
fellow sciensts, and sciensts are expected, as
part of the academic community, to take part in
peer review. If one is to expect others to review
their work, they should commit to reviewing
the work of others as well, and put eort into it.
2) Be pleasant
If the paper is of low quality, suggest that it be
rejected, but do not leave ad hominem com-
ments. There is no benet to being ruthless.
3) Read the invite
When emailing a scienst to ask them to con-
duct a peer review, the majority of journals will
provide a link to either accept or reject. Do not
respond to the email, respond to the link.
4) Be helpful
Suggest how the authors can overcome the
shortcomings in their paper. A review should
guide the author on what is good and what
needs work from the reviewer’s perspecve.
5) Be scienc
The peer reviewer plays the role of a scienc
peer, not an editor for proofreading or decision-
making. Don’t ll a review with comments on
editorial and typographic issues. Instead, focus
on adding value with scienc knowledge and
commenng on the credibility of the research
Page 236
eJIFCC2014Vol25No3pp227-243
Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli
Peer review in scienc publicaons: benets, criques, & a survival guide
conducted and conclusions drawn. If the paper
has a lot of typographical errors, suggest that
it be professionally proof edited as part of the
review.
6) Be mely
Sck to the meline given when conducng a
peer review. Editors track who is reviewing what
and when and will know if someone is late on
compleng a review. It is important to be mely
both out of respect for the journal and the au-
thor, as well as to not develop a reputaon of
being late for review deadlines.
7) Be realisc
The peer reviewer must be realisc about the
work presented, the changes they suggest and
their role. Peer reviewers may set the bar too
high for the paper they are eding by propos-
ing changes that are too ambious and editors
must override them.
8) Be empathec
Ensure that the review is scienc, helpful and
courteous. Be sensive and respecul with
word choice and tone in a review.
9) Be open
Remember that both specialists and generalists
can provide valuable insight when peer review-
ing. Editors will try to get both specialised and
general reviewers for any parcular paper to
allow for dierent perspecves. If someone is
asked to review, the editor has determined they
have a valid and useful role to play, even if the
paper is not in their area of experse.
10) Be organised
A review requires structure and logical ow.
A reviewer should proofread their review be-
fore subming it for structural, grammacal
and spelling errors as well as for clarity. Most
publishers provide short guides on structuring
a peer review on their website. Begin with an
overview of the proposed improvements; then
provide feedback on the paper structure, the
quality of data sources and methods of inves-
gaon used, the logical ow of argument, and
the validity of conclusions drawn. Then provide
feedback on style, voice and lexical concerns,
with suggesons on how to improve.
In addion, the American Physiology Society
(APS) recommends in its Peer Review 101 Hand-
out that peer reviewers should put themselves
in both the editor’s and author’s shoes to en-
sure that they provide what both the editor
and the author need and expect (11). To please
the editor, the reviewer should ensure that the
peer review is completed on me, and that it
provides clear explanaons to back up recom-
mendaons. To be helpful to the author, the re-
viewer must ensure that their feedback is con-
strucve. It is suggested that the reviewer take
me to think about the paper; they should read
it once, wait at least a day, and then re-read
it before wring the review (11). The APS also
suggests that Graduate students and research-
ers pay aenon to how peer reviewers edit
their work, as well as to what edits they nd
helpful, in order to learn how to peer review ef-
fecvely (11). Addionally, it is suggested that
Graduate students pracce reviewing by eding
their peers’ papers and asking a faculty member
for feedback on their eorts. It is recommend-
ed that young sciensts oer to peer review as
oen as possible in order to become skilled at
the process (11). The majority of students, fel-
lows and trainees do not get formal training in
peer review, but rather learn by observing their
mentors. According to the APS, one acquires ex-
perience through networking and referrals, and
should therefore try to strengthen relaonships
with journal editors by oering to review manu-
scripts (11). The APS also suggests that experi-
enced reviewers provide construcve feedback
Page 237
eJIFCC2014Vol25No3pp227-243
Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli
Peer review in scienc publicaons: benets, criques, & a survival guide
to students and junior colleagues on their peer
review eorts, and encourages them to peer
review to demonstrate the importance of this
process in improving science (11).
The peer reviewer should only comment on ar-
eas of the manuscript that they are knowledge-
able about (23). If there is any secon of the
manuscript they feel they are not qualied to
review, they should menon this in their com-
ments and not provide further feedback on
that secon. The peer reviewer is not permit-
ted to share any part of the manuscript with
a colleague (even if they may be more knowl-
edgeable in the subject maer) without rst
obtaining permission from the editor (23). If a
peer reviewer comes across something they are
unsure of in the paper, they can consult the lit-
erature to try and gain insight. It is important
for sciensts to remember that if a paper can
be improved by the experse of one of their
colleagues, the journal must be informed of
the colleague’s help, and approval must be ob-
tained for their colleague to read the protected
document. Addionally, the colleague must be
idened in the condenal comments to the
editor, in order to ensure that he/she is appro-
priately credited for any contribuons (23). It is
the job of the reviewer to make sure that the
colleague assisng is aware of the conden-
ality of the peer review process (23). Once the
review is complete, the manuscript must be de-
stroyed and cannot be saved electronically by
the reviewers (23).
COMMON ERRORS IN SCIENTIFIC PAPERS
When performing a peer review, there are
some common scienc errors to look out for.
Most of these errors are violaons of logic and
common sense: these may include contradict-
ing statements, unwarranted conclusions, sug-
geson of causaon when there is only support
for correlaon, inappropriate extrapolaon,
circular reasoning, or pursuit of a trivial ques-
on (24). It is also common for authors to sug-
gest that two variables are dierent because
the eects of one variable are stascally sig-
nicant while the eects of the other variable
are not, rather than directly comparing the two
variables (24). Authors somemes oversee a
confounding variable and do not control for it,
or forget to include important details on how
their experiments were controlled or the physi-
cal state of the organisms studied (24). Another
common fault is the author’s failure to dene
terms or use words with precision, as these
pracces can mislead readers (24). Jargon and/
or misused terms can be a serious problem in
papers. Inaccurate statements about specic
citaons are also a common occurrence (24).
Addionally, many studies produce knowledge
that can be applied to areas of science outside
the scope of the original study, therefore it is
beer for reviewers to look at the novelty of
the idea, conclusions, data, and methodology,
rather than scrunize whether or not the paper
answered the specic queson at hand (24). Al-
though it is important to recognize these points,
when performing a review it is generally beer
pracce for the peer reviewer to not focus on
a checklist of things that could be wrong, but
rather carefully idenfy the problems specic to
each paper and connuously ask themselves if
anything is missing (24). An extremely detailed
descripon of how to conduct peer review ef-
fecvely is presented in the paper How I Review
an Original Scienc Arcle wrien by Frederic
G. Hoppin, Jr. It can be accessed through the
American Physiological Society website under
the Peer Review Resources secon.
CRITICISM OF PEER REVIEW
A major cricism of peer review is that there is
lile evidence that the process actually works,
that it is actually an eecve screen for good
quality scienc work, and that it actually
Page 238
eJIFCC2014Vol25No3pp227-243
Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli
Peer review in scienc publicaons: benets, criques, & a survival guide
improves the quality of scienc literature. As
a 2002 study published in the Journal of the
American Medical Associaon concluded, ‘Edito-
rial peer review, although widely used, is largely
untested and its eects are uncertain’ (25). Crit-
ics also argue that peer review is not eecve
at detecng errors. Highlighng this point, an
experiment by Godlee et al. published in the
Brish Medical Journal (BMJ) inserted eight
deliberate errors into a paper that was nearly
ready for publicaon, and then sent the pa-
per to 420 potenal reviewers (7). Of the 420
reviewers that received the paper, 221 (53%)
responded, the average number of errors spot-
ted by reviewers was two, no reviewer spoed
more than ve errors, and 35 reviewers (16%)
did not spot any.
Another cricism of peer review is that the pro-
cess is not conducted thoroughly by scienc
conferences with the goal of obtaining large
numbers of submied papers. Such conferenc-
es oen accept any paper sent in, regardless of
its credibility or the prevalence of errors, be-
cause the more papers they accept, the more
money they can make from author registraon
fees (26). This misconduct was exposed in 2014
by three MIT graduate students by the names
of Jeremy Stribling, Dan Aguayo and Maxwell
Krohn, who developed a simple computer pro-
gram called SCIgen that generates nonsense
papers and presents them as scienc papers
(26). Subsequently, a nonsense SCIgen paper
submied to a conference was promptly ac-
cepted. Nature recently reported that French
researcher Cyril Labbé discovered that sixteen
SCIgen nonsense papers had been used by
the German academic publisher Springer (26).
Over 100 nonsense papers generated by SCIgen
were published by the US Instute of Electrical
and Electronic Engineers (IEEE) (26). Both or-
ganisaons have been working to remove the
papers. Labbé developed a program to detect
SCIgen papers and has made it freely available
to ensure publishers and conference organizers
do not accept nonsense work in the future. It
is available at this link: hp://scigendetecon.
imag.fr/main.php (26).
Addionally, peer review is oen cricized for
being unable to accurately detect plagiarism.
However, many believe that detecng plagia-
rism cannot praccally be included as a com-
ponent of peer review. As explained by Alice
Tu, development manager at Sense About
Science, The vast majority of authors and re-
viewers think peer review should detect plagia-
rism (81%) but only a minority (38%) think it is
capable. The academic me involved in detect-
ing plagiarism through peer review would cause
the system to grind to a halt(27). Publishing
house Elsevier began developing electronic pla-
giarism tools with the help of journal editors in
2009 to help improve this issue (27).
It has also been argued that peer review has
lowered research quality by liming creavity
amongst researchers. Proponents of this view
claim that peer review has repressed sciensts
from pursuing innovave research ideas and
bold research quesons that have the potenal
to make major advances and paradigm shis in
the eld, as they believe that this work will like-
ly be rejected by their peers upon review (28).
Indeed, in some cases peer review may result in
rejecon of innovave research, as some stud-
ies may not seem parcularly strong inially, yet
may be capable of yielding very interesng and
useful developments when examined under dif-
ferent circumstances, or in the light of new in-
formaon (28). Sciensts that do not believe in
peer review argue that the process ses the
development of ingenious ideas, and thus the
release of fresh knowledge and new develop-
ments into the scienc community.
Another issue that peer review is cricized for,
is that there are a limited number of people
that are competent to conduct peer review
Page 239
eJIFCC2014Vol25No3pp227-243
Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli
Peer review in scienc publicaons: benets, criques, & a survival guide
compared to the vast number of papers that
need reviewing. An enormous number of pa-
pers published (1.3 million papers in 23,750
journals in 2006), but the number of compe-
tent peer reviewers available could not have
reviewed them all (29). Thus, people who lack
the required experse to analyze the quality of
a research paper are conducng reviews, and
weak papers are being accepted as a result. It
is now possible to publish any paper in an ob-
scure journal that claims to be peer-reviewed,
though the paper or journal itself could be sub-
standard (29). On a similar note, the US Naon-
al Library of Medicine indexes 39 journals that
specialize in alternave medicine, and though
they all idenfy themselves as “peer-reviewed”,
they rarely publish any high quality research
(29). This highlights the fact that peer review of
more controversial or specialized work is typi-
cally performed by people who are interested
and hold similar views or opinions as the au-
thor, which can cause bias in their review. For
instance, a paper on homeopathy is likely to be
reviewed by fellow praccing homeopaths, and
thus is likely to be accepted as credible, though
other sciensts may nd the paper to be non-
sense (29). In some cases, papers are inially
published, but their credibility is challenged at
a later date and they are subsequently retract-
ed. Retracon Watch is a website dedicated to
revealing papers that have been retracted aer
publishing, potenally due to improper peer re-
view (30).
Addionally, despite its many posive out-
comes, peer review is also cricized for being
a delay to the disseminaon of new knowledge
into the scienc community, and as an unpaid-
acvity that takes sciensts’ me away from
acvies that they would otherwise priorize,
such as research and teaching, for which they
are paid (31). As described by Eva Amsen, Out-
reach Director for F1000Research, peer review
was originally developed as a means of helping
editors choose which papers to publish when
journals had to limit the number of papers they
could print in one issue (32). However, nowadays
most journals are available online, either ex-
clusively or in addion to print, and many jour-
nals have very limited prinng runs (32). Since
there are no longer page limits to journals, any
good work can and should be published. Con-
sequently, being selecve for the purpose of
saving space in a journal is no longer a valid
excuse that peer reviewers can use to reject
a paper (32). However, some reviewers have
used this excuse when they have personal ulte-
rior moves, such as geng their own research
published rst.
RECENT INITIATIVES TOWARDS
IMPROVING PEER REVIEW
F1000Research was launched in January 2013
by Faculty of 1000 as an open access journal
that immediately publishes papers (aer an
inial check to ensure that the paper is in fact
produced by a scienst and has not been pla-
giarised), and then conducts transparent post-
publicaon peer review (32). F1000Research
aims to prevent delays in new science reaching
the academic community that are caused by
prolonged publicaon mes (32). It also aims to
make peer reviewing more fair by eliminang
any anonymity, which prevents reviewers from
delaying the compleon of a review so they
can publish their own similar work rst (32).
F1000Research oers completely open peer re-
view, where everything is published, including
the name of the reviewers, their review reports,
and the editorial decision leers (32).
PeerJ was founded by Jason Hoyt and Peter Bin-
eld in June 2012 as an open access, peer re-
viewed scholarly journal for the Biological and
Medical Sciences (33). PeerJ selects arcles to
publish based only on scienc and methodolog-
ical soundness, not on subjecve determinants
Page 240
eJIFCC2014Vol25No3pp227-243
Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli
Peer review in scienc publicaons: benets, criques, & a survival guide
of ‘impact,‘novelty’ or ‘interest’ (34). It works
on a “lifeme publishing plan” model which
charges sciensts for publishing plans that give
them lifeme rights to publish with PeerJ, rather
than charging them per publicaon (34). PeerJ
also encourages open peer review, and authors
are given the opon to post the full peer review
history of their submission with their published
arcle (34). PeerJ also oers a pre-print review
service called PeerJ Pre-prints, in which paper
dras are reviewed before being sent to PeerJ
to publish (34).
Rubriq is an independent peer review service
designed by Shashi Mudunuri and Keith Collier
to improve the peer review system (35). Rubriq
is intended to decrease redundancy in the peer
review process so that the me lost in redundant
reviewing can be put back into research (35). Ac-
cording to Keith Collier, over 15 million hours are
lost each year to redundant peer review, as pa-
pers get rejected from one journal and are sub-
sequently submied to a less presgious journal
where they are reviewed again (35). Authors of-
ten have to submit their manuscript to mulple
journals, and are oen rejected mulple mes
before they nd the right match. This process
could take months or even years (35). Rubriq
makes peer review portable in order to help
authors choose the journal that is best suited
for their manuscript from the beginning, thus
reducing the me before their paper is pub-
lished (35). Rubriq operates under an author-
pay model, in which the author pays a fee and
their manuscript undergoes double-blind peer
review by three expert academic reviewers us-
ing a standardized scorecard (35). The major-
ity of the author’s fee goes towards a reviewer
honorarium (35). The papers are also screened
for plagiarism using iThencate (35). Once the
manuscript has been reviewed by the three ex-
perts, the most appropriate journal for submis-
sion is determined based on the topic and qual-
ity of the paper (35). The paper is returned to
the author in 1-2 weeks with the Rubriq Report
(35). The author can then submit their paper to
the suggested journal with the Rubriq Report
aached. The Rubriq Report will give the jour-
nal editors a much stronger incenve to con-
sider the paper as it shows that three experts
have recommended the paper to them (35).
Rubriq also has its benets for reviewers; the
Rubriq scorecard gives structure to the peer re-
view process, and thus makes it consistent and
ecient, which decreases me and stress for
the reviewer. Reviewers also receive feedback
on their reviews and most signicantly, they
are compensated for their me (35). Journals
also benet, as they receive pre-screened pa-
pers, reducing the number of papers sent to
their own reviewers, which oen end up re-
jected (35). This can reduce reviewer fague,
and allow only higher-quality arcles to be sent
to their peer reviewers (35).
According to Eva Amsen, peer review and sci-
enc publishing are moving in a new direc-
on, in which all papers will be posted online,
and a post-publicaon peer review will take
place that is independent of specic journal
criteria and solely focused on improving paper
quality (32). Journals will then choose papers
that they nd relevant based on the peer re-
views and publish those papers as a collecon
(32). In this process, peer review and individual
journals are uncoupled (32). In Keith Colliers
opinion, post-publicaon peer review is likely
to become more prevalent as a complement
to pre-publicaon peer review, but not as a re-
placement (35). Post-publicaon peer review
will not serve to idenfy errors and fraud but
will provide an addional measurement of im-
pact (35). Collier also believes that as journals
and publishers consolidate into larger systems,
there will be stronger potenal for “cascading”
and shared peer review (35).
Page 241
eJIFCC2014Vol25No3pp227-243
Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli
Peer review in scienc publicaons: benets, criques, & a survival guide
CONCLUDING REMARKS
Peer review has become fundamental in assist-
ing editors in selecng credible, high quality,
novel and interesng research papers to pub-
lish in scienc journals and to ensure the cor-
recon of any errors or issues present in sub-
mied papers. Though the peer review process
sll has some aws and deciencies, a more
suitable screening method for scienc papers
has not yet been proposed or developed. Re-
searchers have begun and must connue to
look for means of addressing the current issues
with peer review to ensure that it is a full-proof
system that ensures only quality research pa-
pers are released into the scienc community.
REFERENCES
1. “What Is Peer Review?” (2014). Int J Comput Appl.
Web. Retrieved July 02, 2014, from hp://www.ijcaon-
line.org/peer-review
2. “Peer Review”. (2014). Elsevier Publishing Guidelines.
Web. Retrieved June 24, 2014, from hp://www.elsevier.
com/about/publishing-guidelines/peer-review
3. Spier R. (2002). “The History of the Peer-review Pro-
cess.” Trends Biotechnol, 20(8): 357-58.
4. Liumbruno GM, Vela C, Pasquale P, Franchini M.
(2012). “How to Write a Scienc Manuscript for Publica-
on.” Blood Transfus, 11(2): 217-26.
5. “Peer Review: What It Is, Why It’s Done and How to
Do It. Elsevier. Web. Retrieved June 26, 2014, from
www.meatscience.org/WorkArea/DownloadAsset.
aspx?id=8503
6. Fitzpatrick K. (2009). “Planned Obsolecence”. Media-
Commons Press. Retrieved July 11, 2014 from hp://mc-
press.media-commons.org/plannedobsolescence/
7. Ware M. (2008). “Peer Review: Benets, Percepons
and Alternaves.PRC Summary Papers, 4:4-20.
8. Mulligan A. (2005). “Is Peer Review in Crisis?” Oral On-
col., 41(2): 135-41.
9. Simons-Morton B, Abraido-Lanza AF, Bernhardt
JM, Schoenthaler A, Schnitzer A, Allegrante JP. (2012).
“Demysfying Peer Review., 39(1): 3-7.
10. Swoger B. (2014). “Post Publicaon Peer-re-
view: Everything Changes, and Everything Stays the
Same”. Scienc Americanblogs. Web. Retrieved July
11, 2014 from hp://blogs.sciencamerican.com/
informaon-culture/2014/03/26/post-publicaon-peer-
review-everything-changes-and-everything-stays-the-
same/
11. “Peer Review 101.” (2013). The American Physiologi-
cal Society. Web. Retrieved July 02, 2014, from hp://
www.the-aps.org/mm/SciencePolicy/Agency-Policy/
Peer-Review/PeerReview101.pdf
12. Schley D. (2009).”Peer Reviewers Sased with
System.Times Higher Educaon. Web. Retrieved July
11, 2014 from hp://www.meshighereducaon.
co.uk/408108.arcle
13. Ulrichsweb Global Science Directory. Web. Retrieved
June 27, 2014 from hp://ulrichsweb.serialssoluons.com
14. Saxby C, Richardson M. (2006). “Assessing the Impact
of Open Access”. Oxford Journals Preliminary Report.
Web. Retrieved July 11th, 2014 from hp://www.oxford-
journals.org/news/oa_report.pdf
15. Steingraber S. (1985). “Guidelines For Wring Scienc
Papers”. Honors Organismal Biology Laboratory Manual.
Web. Retrieved July 11, 2014 from hp://www.bms.bc.ca/
resources/library/pdf/GuidelinesSciencPapers.pdf
16. “Reviewers Informaon Pack”. (2011). Internaonal
Conference on Mathemacal Modeling in Physical Sci-
ences.Web. Retrieved July 04, 2014, from hp://www.
icmsquare.net/FileStore/reviewerGuides.pdf
17. Jusce AC, Cho MK, Winker MA, Berlin JA, Rennie D.
(1998).”Does Masking Author Identy Improve Peer Re-
view Quality?” JAMA, 280(3):240-2.
18. McNu RA, Evans AT, Fletcher RH, Fletcher SW.
(1990). “The Eects of Blinding on the Quality of Peer Re-
view.JAMA, 263(10):1371-6.
19. Kumar M. (2009). “A Review of the Review Process:
Manuscript Peer-review in Biomedical Research.Biology
and Medicine, 1 (4): 1-16.
20. Falagas ME. (2007). “Peer Review in Open Access Sci-
enc Journals.Open Medicine, 1(1): 49-51.
21. Bohannon J. (2013). “Who’s Afraid of Peer Re-
view?” Science, 342(6154):60-65.
22. Lucey B. (2013). “Peer Review: How to Get It Right
10 Tips.” The Guardian. Web. Retrieved from hp://www.
theguardian.com/higher-educaon-network/blog/2013/
sep/27/peer-review-10-ps-research-paper
23. Nichols NL, Sasser JM. (2014). “The Other Side of the
Submit Buon: How to Become a Reviewer for Scienc
Journals.” The Physiologist, 57(2): 88-91.
24. Hoppin FG Jr. (2002). “How I Review an Original
Scienc Arcle.” Am J Respir Crit Care Med, 166(8):
1019-1023.
Page 242
eJIFCC2014Vol25No3pp227-243
Jacalyn Kelly, Tara Sadeghieh, Khosrow Adeli
Peer review in scienc publicaons: benets, criques, & a survival guide
25. Jeerson T, Alderson P, Wager E, Davido F. (2002).
“Eects of Editorial Peer Review: A Systemac Review.
JAMA, 287(21): 2784-2786.
26. Sample I. (2014). “How Computer-generated Fake
Papers Are Flooding Academia.” The Guardian. Web. Re-
trieved July 11, 2014 from hp://www.theguardian.com/
technology/shortcuts/2014/feb/26/how-computer-gen-
erated-fake-papers-ooding-academia
27. Saary L. (2009). “Peer Review under the Micro-
scope.” Royal Society of Chemistry.Web. Retrieved July
11, 2014 from hp://www.rsc.org/chemistryworld/
News/2009/September/09090901.asp
28. Corbyn Z. (2008). “Call to Scrap Peer Review in Hunt
for Brilliant Ideas.” Times Higher Educaon. Web. Re-
trieved July 11, 2014 from hp://www.meshigheredu-
caon.co.uk/404707.arcle
29. Colquhoun D. (2011). “Publish-or-perish: Peer Review
and the Corrupon of Science. The Guardian.Web. Re-
trieved from hp://www.theguardian.com/science/2011/
sep/05/publish-perish-peer-review-science
30. Retracon Watch. Web. Retrieved June 27, 2014,
from hp://retraconwatch.com/
31. Jennings CG. (2006). “Quality and Value: The True
Purpose of Peer Review.Nature blogs. Web. Retrieved
July 11, 2014 from hp://blogs.nature.com/peer-to-
peer/2006/06/quality_and_value_the_true_pur.html
32. Tippmann S. (2014). “New Avenues For Peer Review:
An (Audio) Interview With Eva Amsen.” Peer Review
Watch. Web. Retrieved July 07, 2014 from hp://peer-
reviewwatch.wordpress.com/2014/04/05/new-avenues-
for-peer-review-an-audio-interview-with-eva-amsen/
33. Wesolek A. (2013). “OA Now Interview with Pe-
ter Bineld of PeerJ. Open Access Now. Web. Re-
trieved July 07, 2014 from hp://oanow.org/2013/06/
oa-now-interview-with-peter-bineld-of-peerj/
34. “Two Publicaons”. (2012). PeerJ. Web. Re-
trieved July 08, 2014, from hps://peerj.com/about/
publicaons/#PeerJ
35. Meadows A. (2013). A New Approach to Peer Review
– an Interview with Keith Collier, Co-founder of Rubriq.”
Wiley Exchanges. Web. Retrieved July 07, 2014 from
http://exchanges.wiley.com/blog/2013/09/17/a-new-
approach-to-peer-review-an-interview-with-keith-collier-
co-founder-of-rubriq/
Page 243
eJIFCC2014Vol25No3pp227-243
... The issue of peer review is much more strident in medical, biomedical, and other natural sciences than in geological sciences. The practice of peer review, since it was first introduced by a physician named Shaq bin Ali al-Rahway of Syria (854-931 CE) (Kelly, 2014), has become a selfregulating mechanism for controlling quality of articles in journals by experts (peers) in a given domain. At present in 2022, journals adopt a doubleblind review process in which the identities of both the author and the reviewer are masked in maintaining objectivity. ...
... The history of peer review has been discussed by many scholars and publishing organizations (van Rooney et al., 1999;Biagioli, 2002;Spier, 2002;Kennefick, 2005;Benos et al., 2007;Kelly et al., 2014;Shema, 2014;Vyas and Hozain, 2014;Baldwin, 2015Baldwin, , 2019Belluz and Hoffman, 2015;Scissor, 2016;Dinerstein, 2017;Ronnie and Flanagan, 2018;Jana, 2019;Al-Mousawi, 2020;Elsevier, 2021;Roy, 2021;Wikipedia, 2021;Hoffman, 2022;among others). From these and other sources, I have selected some historical events dealing with scientific development and peer review. ...
... From these and other sources, I have selected some historical events dealing with scientific development and peer review. Although broad in scope, I have included the birth of some key journals in geological sciences worldwide: 1) 5 th Century BCE: Introduction of the concept of peer review as a method of evaluating written work in ancient Greece (Kelly et al., 2014;Roy, 2021). 2) 25-220 CE: Documentation of first papermaking process in China (Wikipedia, 2021). ...
... The issue of peer review is much more strident in medical, biomedical, and other natural sciences than in geological sciences. The practice of peer review, since it was first introduced by a physician named Shaq bin Ali al-Rahway of Syria (854-931 CE) (Kelly, 2014), has become a selfregulating mechanism for controlling quality of articles in journals by experts (peers) in a given domain. At present in 2022, journals adopt a double-blind review process in which the identities of both the author and the reviewer are masked in maintaining objectivity. ...
... The history of peer review has been discussed by many scholars and publishing organizations (van Rooney et al., 1999;Biagioli, 2002;Spier, 2002;Kennefick, 2005;Benos et al., 2007;Kelly et al., 2014;Shema, 2014;Vyas and Hozain, 2014;Baldwin, 2015Baldwin, , 2019Belluz and Hoffman, 2015;Scissor, 2016;Dinerstein, 2017;Ronnie and Flanagan, 2018;Jana, 2019;Al-Mousawi, 2020;Elsevier, 2021;Roy, 2021;Wikipedia, 2021;Hoffman, 2022;among others). From these and other sources, I have selected some historical events dealing with scientific development and peer review. ...
... From these and other sources, I have selected some historical events dealing with scientific development and peer review. Although broad in scope, I have included the birth of some key journals in geological sciences worldwide: 1) 5 th Century BCE: Introduction of the concept of peer review as a method of evaluating written work [n ancient Greece (Kelly et al., 2014;Roy, 2021). 2) 25-220 CE: Documentation of first paper-making process in China (Wikipedia, 2021). ...
... Manuscripts undergoing the peer review process prior to publication are usually subjected to either an open review, a single-blind review or a double-blind review. In an open review, the identity of both authors and the peer reviewers is known; in single-blind review, the identity of the reviewers is concealed while that of the authors is revealed to the reviewer; and in double-blind review, the identities of both the reviewers and author(s) are concealed [4]. Upon receiving comments from the reviewers, the editor will then make a decision on the manuscript, and this decision will be communicated to the authors. ...
... There is one quality that is of utmost importance in handling challenging reviewers' comments, and that is tactfulness. The aim of this article is to highlight various challenging reviewers' comments that were identified through a literature review [4][5][6][7][8][9][10][11][12][13][14][15][16][17] and based on the author's personal experience, as well as to provide tips to handle such comments. ...
Article
Full-text available
The goal of every author is to have their research work published. In the process of publishing a peer-reviewed article, authors are often required to revise their original manuscript based on the comments from the reviewers. Although some of these comments are straightforward and concise, others are conflicting and unclear and, as such, authors may find it challenging to plan and carry out the revision as well as compose the accompanying response letter. In this article I outline eight challenges in handling reviewers’ comments that may be useful for novice authors. In general, authors will always benefit from adopting a positive attitude towards reviewers’ comments and make the effort to improve their manuscript. Graphical abstract
... Besides, Nyagormey et al. (2020) opined that reliance on previous research works published in peer-reviewed journals in the Scopus database represents credible and significant knowledge resources for qualitative literature review studies. Kelly et al. (2014) also indicated that credible data resources guide the development of new knowledge areas because the information extracted is anchored on the expert knowledge of other researchers in the discipline. ...
Article
Extant studies on Self-Help Housing (SHH) have centered on the types, processes, value, and challenges, but rare on the criteria for assessing its affordability. This paper presents a comprehensive review on the criteria based on published literature in 32 peer- reviewed journals. 81 relevant peer-reviewed papers were used for the study. A total of 8 criteria set, including 31 associated determi- nants were identified to have influence on SHH affordability. The cri- teria were; financing, access to land, building quality, development controls and planning regulations, building cost, location, infrastruc- ture and services, and social issues. A framework method was adopted to select the papers. Authors from both developed and developing countries were noted to have made various pioneering contributions to SHH studies. The mean SCImago Journal Rank indi- cator obtained was above1.00, which showed that the journals used for the review were quality within the time frame of 1997 to 2020. Limitations were mainly seen in our dependence on Scopus, the few search engines, and the selected papers used. This study's outcome will broaden understanding of SHH affordability in developing coun- tries on; policy formulation, housing market conditions, and research This review provides the opportunity for further empirical studies on SHH affordability metric development.
... [8] The peer review is being criticized for many reasons including an inappropriate decision based on the reviewer's report. [9,10] In contrast, the some journals are bypassing the peer review process for faster process the articles. [11] Hence, importance of peer review in shaping science should be explored more transparently. ...
Article
Full-text available
Background: Peer review is one of the cornerstones of sound scientific publication. When a paper is submitted to a journal, after an initial assessment, the editor sends the article to reviewers with similar research experiences for comprehensive comments. Aims and Objectives: In this article, we aimed to analyze the pattern of peer review by the reviewers in the physiology domain from India as found in publons.com. Materials and methods: We searched https://publons.com and clicked on "Browse" to search the profile of "Researchers." The research field was selected as "Physiology" and the country as "India." A total of 158 profiles were found with the search criteria. The number of publications, citations, peer review, H-index, and the editorial record was obtained for further analysis. The number of the peer review was tested with the number of publications and citations for any existing correlation. Results: Among the 158 researchers in the physiology domain, an average verified review was 8.09 ± 32.12 and an average number of publications were 15.7 ± 31.1. Their number of citations was 359.75 ± 1296.24 with an average H-index of 4.15 ± 7.85. There was a significant positive correlation with peer review versus number of publication (0.39 [95% CI: 0.24 to 0.51], P < 0.0001) and peer review versus number of citation (0.26 [95% CI: 0.11 to 0.41)], P = 0.0008). Conclusion: Peer reviewers in the physiology domain from India have contributed by reviewing an average of 8 peer reviews. However, many of the profiles showed zero contribution. More publications and more citations would expose the researchers to a wider audience worldwide, including the editors. Hence, they would get peer review requests.
... Peer debriefing was chosen as the preferred method for enhancing the research's quality in this study. It encourages authors to adhere to their discipline's established high standards and to exercise control over the dissemination of research data in order to avoid the publication of unsupported claims, unfavourable interpretations, or personal viewpoints without prior expert review [18]. ...
Conference Paper
Full-text available
Effective school leadership behaviours and practices can help students learn and improve their achievement in rural secondary schools. This study examines the instrument's capability to gather data and familiarise the interviewer with the interview process. This interview also identifies a common code that might arise from the study. A principal, a teacher, and one ex-student are interviewed from one secondary school in rural secondary school for this pilot study. The data was obtained through interviews. The quantity of data in the form of codes is rich and high, indicating that the instrument is able to collect the intended data. The data was then analysed using NVIVO 12 Version 2020. The findings showed that Responsibility, School direction, and Student's Development codes were frequently collected during the pilot phase.
Article
Though not rare in the gatekeeping practices of journals, editorial bias in China has received scant scholarly attention. Drawing on in‐depth interviews with 15 academics and 10 editors in the field of education and collecting data on more than 1,522 scholarly articles published by the Chinese journal Educational Research, we assessed the causes and consequences of editorial bias in a subset of the education journals covered by the Chinese Social Sciences Citation Index (CSSCI). Our findings indicate that the editorial biases revealed in some top‐tier Chinese education journals were associated mainly with academic fame, rank, and affiliation as well as personal relationships described by the term guanxi. The causes of these biases include the desire to secure the status of CSSCI‐listed journals, short‐staffed editorial teams, and an inadequate peer‐review system. Further, we found these biases to be associated with three particular negative consequences: slowing the professional development of young academics, allowing the publication of mediocre articles and compromising the Chinese academic environment. We conclude that journal editors, reviewers and academics must work together to safeguard a rigorous and constructive scholarly publishing system in China.
Article
Full-text available
Background: Big data and data analysis methods and models are important tools in food security (FS) studies for gap analysis and preparation of appropriate analytical frameworks. These innovations necessitate the development of novel methods for collecting, storing, processing, and extracting data. Methodology: The primary goal of this study was to conduct a critical review of agricultural big data and methods and models used for FS studies published in peer-reviewed journals since 2010. Approximately 130 articles were selected for full content review after the pre-screening process. Results: There are different sources of data collection, including but not limited to online databases, the internet, omics, Internet of Things, social media, survey rounds, remote sensing, and the Food and Agriculture Organization Corporate Statistical Database. The collected data require analysis (i.e., mining, neural networks, Bayesian networks, and other ML algorithms) before data visualization using Python, R, Circos, Gephi, Tableau, or Cytoscape. Approximately 122 models, all of which were used in FS studies worldwide, were selected from 130 articles. However, most of these models addressed only one or two dimensions of FS (i.e., availability and access) and ignored the other dimensions (i.e., stability and utilization), creating a gap in the global context. Conclusions: There are certain FS gaps both worldwide and in the United Arab Emirates that need to be addressed by scientists and policymakers. Following the identification of the drivers, policies, and indicators, the findings of this review could be used to develop an appropriate analytical framework for FS and nutrition.
Article
Full-text available
The origins and development of the scientific and technical press can be traced back to 1665 when the first “modern” scientific papers appeared and were characterized by non standardised form and style1. Subsequently, nearly 300 years ago2, in an attempt to ensure that articles met the journal’s standards of quality and scientific validity, the peer-reviewed process for scientific manuscripts was born in England and France. Since then, there has been an enormous proliferation of scientific journals and manuscripts so that, at present, the numbers of biomedical papers published annually by over 20,000 journals, at a rate of 5,500 new papers per day, far exceeds 2,000,0001,2. Published scientific papers and professional meetings are really essential to disseminate relevant information and research findings. However, most of the abstracts of presentations given at scientific meetings are usually available only in conference proceedings although they have the potential to be subsequently published as articles in peer-reviewed journals. A recently published Cochrane review showed that only 44.5% of almost 30,000 scientific meeting abstracts were published as articles3. No association between full publication and authors’ country of origin was detected. Factors associated with full publication included acceptance vs rejection of abstracts for oral or poster presentations, acceptance for oral presentations rather than poster sessions, “positive” results, using the report authors’ definition of “positive”, randomised trial study design and basic rather than clinical research. Possible reasons for failed publication include lack of time, research still underway, problems with co-authors and negative results4. Undoubtedly, lack of the necessary skills and experience in the process of writing and publishing is another possible contributing factor also in the field of Transfusion Medicine although the specialists in this discipline are currently adopting the principles and research methodologies that support evidence-based medicine5, and high-level research is actually being carried out at the same rate as in all medical specialties. There are three broad groups of manuscripts: original scientific articles, reviews and case reports. Although case reports are part of the evidence hierarchy in evidence-based practice, albeit at a lower level, and case series are incorporated in a significant proportion of health technology assessments6, this article will address the multiple steps required in writing original articles and reviews with the aim of providing the reader with the necessary tools to prepare, submit and successfully publish a manuscript.
Article
Full-text available
This issue carries our annual homage to the many heretofore anonymous colleagues who during the last year have volunteered their valuable time and expertise in reviewing manuscripts for Health Education & Behavior (HEB in the scientific world to ensure that only the most important and meritorious research is funded; and in the publishing world to assess, improve, and control the quality of articles published in academic journals. Editorial peer review has been likened to democracy as a form of government and similarly described as “. . . the worst system . . . except for the alternatives” (DeMaria, 2002, p. 1018). Moreover, peer review in the assessment of scholarship often has been criticized for being slow, cumbersome, and sometimes ineffective. It has even been suggested that open review via social media technologies replace the current system of traditional peer review (Kolowich, 2011). Despite such criticism, peer review remains the most important and trusted means of distinguishing credible scientific
Article
Full-text available
Editorial peer review has remained the gold standard for evaluation of scientific research for nearly two centuries.Its importance has increased all the more due to the currently prevailing ‘publish or perish’ culture. This articlereviews the drawbacks in the current peer-review practices and the suggested methods to overcome suchdrawbacks. A universally acceptable method of peer-review is yet to be designed because of the practicaldifficulties in the objective analysis of a complex human behaviour such as the peer-review.
Article
Full-text available
All authors may not be equal in the eyes of reviewers. Specifically, well-known authors may receive less objective (poorer quality) reviews. One study at a single journal found a small improvement in review quality when reviewers were masked to author identity. To determine whether masking reviewers to author identity is generally associated with higher quality of review at biomedical journals, and to determine the success of routine masking techniques. A randomized controlled trial performed on external reviews of manuscripts submitted to Annals of Emergency Medicine, Annals of Internal Medicine, JAMA, Obstetrics & Gynecology, and Ophthalmology. Two peers reviewed each manuscript. In one study arm, both peer reviewers received the manuscript according to usual masking practice. In the other arm, one reviewer was randomized to receive a manuscript with author identity masked, and the other reviewer received an unmasked manuscript. Review quality on a 5-point Likert scale as judged by manuscript author and editor. A difference of 0.5 or greater was considered important. A total of 118 manuscripts were randomized, 26 to usual practice and 92 to intervention. In the intervention arm, editor quality assessment was complete for 77 (84%) of 92 manuscripts. Author quality assessment was complete on 40 (54%) of 74 manuscripts. Authors and editors perceived no significant difference in quality between masked (mean difference, 0.1; 95% confidence interval [CI], -0.2 to 0.4) and unmasked (mean difference, -0.1; 95% CI, -0.5 to 0.4) reviews. We also found no difference in the degree to which the review influenced the editorial decision (mean difference, -0.1; 95% CI,-0.3 to 0.3). Masking was often unsuccessful (overall, 68% successfully masked; 95% CI, 58%-77%), although 1 journal had significantly better masking success than others (90% successfully masked; 95% CI, 73%-98%). Manuscripts by generally known authors were less likely to be successfully masked (odds ratio, 0.3; 95% CI, 0.1-0.8). When analysis was restricted to manuscripts that were successfully masked, review quality as assessed by editors and authors still did not differ. Masking reviewers to author identity as commonly practiced does not improve quality of reviews. Since manuscripts of well-known authors are more difficult to mask, and those manuscripts may be more likely to benefit from masking, the inability to mask reviewers to the identity of well-known authors may have contributed to the lack of effect.
Article
Peer reviewers are blinded sometimes to authors' and institutions' names, but the effects of blinding on review quality are not known. We, therefore, conducted a randomized trial of blinded peer review. Each of 127 consecutive manuscripts of original research that were submitted to the Journal of General Internal Medicine were sent to two external reviewers, one of whom was randomly selected to receive a manuscript with the authors' and institutions' names removed. Reviewers were asked, but not required, to sign their reviews. Blinding was successful for 73% of reviewers. Quality of reviews was higher for the blinded manuscripts (3.5 vs 3.1 on a 5-point scale). Forty-three percent of reviewers signed their reviews, and blinding did not affect the proportion who signed. There was no association between signing and quality. Our study shows that, in our setting, blinding improves the quality of reviews and that research on the effects of peer review is possible.