Academic Psychiatry, 28:2, Summer 2004 81
How to Review a Manuscript: A “Down-to-Earth”
Laura Weiss Roberts, M.D., M.A.
John Coverdale, M.D., F.R.A.N.Z.C.P.
Kristin Edenharder, B.A.
Alan Louie, M.D.
Dr. Roberts is the Editor-in-Chief of Academic Psychiatry.
Drs. Coverdale and Louie are the Associate Editors for Academic
Psychiatry. Ms. Edenharder is the Assistant to the Editors.
Copyright 䉷 2004 Academic Psychiatry.
thoughtful review of a manuscript is a gift. It is
a gift of expertise, of time, and of careful consid-
eration. It is a gift for authors, editors, and the ﬁeld.
Performing a thoughtful peer evaluation of a manu-
script is an acquired skill, however, and one that may
not be formally taught in many academic depart-
ments or postgraduate training programs. The aim of
this short piece is to provide some “down-to-earth”
guidance for early career faculty, prospective authors,
and peer reviewers on how to write a manuscript re-
view. This repr esents a fundamental skill in academics,
and this paper is the ﬁrst in a series that will appear
in our journal providing “down-to-earth” guidance re-
lated to academic skills for our readers.
Peer Review and Editorial Decision-Making
The goals of peer review are to assist the editors in
forming a decision concerning publication of a manu-
script and to provide constructive feedback to au-
thors in order to enhance the quality of the ﬁnal writ-
Figure 1 depicts the editorial and publication pro-
cess common to most journals. Once a manuscript is
received, it is read and “triaged” by the Editors. Al-
though most journals handle this initial evaluation of
papers similarly (i.e., the papers are read by the edi-
tors and/or editorial assistants), different journals
have different thresholds at this stage of manuscript
consideration. Some journals are quick to reject pa-
pers at this stage, whereas others (like Academic Psy-
chiatry) seek to include all papers that have a chance
of success. Our journal sends out for peer review the
vast majority of manuscripts we receive because of
our commitment to helping authors. We believe that
the peer review process is inherently valuable, even
if it is decided that an individual manuscript will not
be published. For Academic Psychiatry, the main rea-
son for rejection at this early point would be an in-
appropriate topic, i.e., the goals of the manuscript
would be seen as falling outside of the mission of our
Once the decision is made to send a manuscript
out for review, up to ﬁve potential reviewers are con-
tacted via e-mail to inquire about their willingness to
review a manuscript within a 3-week period. This
time frame is necessary so that the review process and
editorial decision-making can occur in a timely fash-
ion for authors. An editorial assistant tracks the pro-
gress of each review and will send reviewers periodic
“reminder” e-mails once a review’s optimal deadline
has been reached. If a reviewer commits to complet-
ing a review, it is important that the reviewer follow
through with this commitment. Likewise, if a re-
viewer is unable to review a particular manuscript, it
is important to communicate this to the editorial staff.
We will then reassess the number of willing reviewers
for each manuscript and invite new reviewer candi-
dates as necessary. As the editors will often wait to
receive all invited reviews before making a decision,
a late review or an unanswered invitation to review
may cause substantial delays in the peer review pro-
Comments received back from peer reviewers
can be brief (1–2 paragraphs) or lengthy (up to 3
82 Academic Psychiatry, 28:2, Summer 2004
Figure 1. The Editorial and Publication Process
Submission of manuscript
and accompanying materials
and presubmission process
pages). Reviews usually identify a range of strengths
and weaknesses found in the manuscript, and the re-
viewers provide various ratings on publishability
and acceptability. The editors are responsible for us-
ing the reviews to inform their editorial decision. Or-
dinarily we will not render a decision unless we have
received a minimum of two reviews for a given
manuscript. This policy ensures that authors are
given a substantial amount of speciﬁc feedback from
which to draw in drafting a revision. Often reviews
are mixed (i.e., a single reviewer will have an ambig-
uous response to a paper) or in conﬂict (i.e., different
reviewers express different degrees of enthusiasm—
or not—for a paper).
Once a decision has been reached on a manu-
script, the authors receive a letter or e-mail from the
editor(s), accompanied by reviewers’ comments, in-
forming them of this decision. With the transition to
our new online submission system, more and more
communication from Academic Psychiatry’s main ed-
itorial ofﬁce is handled electronically. E-mail allows
for quick and efﬁcient contact, which is beneﬁcial to
all. Decisions on manuscripts range from “accepted”
(with or without a requirement for minor revisions)
to “rejection.” Most commonly, we defer making a
ﬁnal determination on a manuscript at this point—
again, this provides an opportunity for authors to
pursue a revision. A deferred decision should not be
perceived as a rejection. Should a decision on publi-
cation be deferred, authors may rework their paper
and resubmit for editorial re-review.
When a paper is revised and resubmitted, the ed-
itors read this next version and then there are several
possible outcomes, including acceptance, rejection, or
re-review. We typically ask for review of the revision
by the original reviewers. Further revision and iter-
ation of the preceding process may again be sug-
gested. For these reasons, providing a fair review and
editorial decision can be a protracted process. It is
also for these reasons that an expeditious contribu-
tion from the reviewers is appreciated by editors and
Elements of Peer Review
The elements of peer review include “comments to
the author(s)” and a rating form including “conﬁden-
tial comments to the editors.” The “comments to the
author(s)” are sent to the authors, with, in the case of
Academic Psychiatry, the reviewers’ identities masked.
The most helpful review is one that articulates the
strengths of a paper while also assiduously identify-
ing the limitations of the manuscript that can be ad-
dressed in a revision. Nevertheless, even if a paper is
well received overall, reviewer comments on manu-
ROBERTS ET AL.
Academic Psychiatry, 28:2, Summer 2004 83
scripts are commonly “negative” (1). Authors natu-
rally ﬁnd this discouraging, and we at Academic Psy-
chiatry make every effort to encourage reviewers to
identify positive aspects of the papers we receive. In
addition, while critical comments of a constructive
nature are solicited, our reviewers are actively dis-
couraged from using unduly harsh or pejorative lan-
The “comments to the editors” form used by Ac-
ademic Psychiatry incorporates a checklist with a di-
chotomous (yes/no) scale, a global rating of overall
publishability on a scale of 1 (unacceptable) to 10
(outstanding), and “conﬁdential comments to the ed-
itors.” The checklist presented on this form solicits
information on originality, clarity of presentation, use
of relevant references, length, clarity and adequacy of
design, clarity of results, need for a statistical consul-
tant, utility of tables and ﬁgures, and adequacy of
discussion and conclusions. The global rating of over-
all publishability combines a rating of quality plus
importance, with reviewers also asked to rate the
publishability of the manuscript should it be revised
in line with recommendations. Even though the edi-
tors decide the outcome of the peer review, the re-
viewers also check their recommendation, with pos-
sible responses including “accept as is,” “modify with
minor revisions,” “modify with major revisions,” or
“reject.” The comments to the editor generally pro-
vide a brief rationale for this recommendation. Based
on the quality and importance of the research, the
reviewer provides a succinct comment on the weight
of strengths compared to weaknesses of the manu-
script, and on the potential reversibility of signiﬁcant
weaknesses. The importance of a manuscript is
judged in part by the strength of the results but also
by whether it contains original ideas. For instance,
less substantive data may be acceptable if the work
is the ﬁrst of its kind.
A “Down-to-Earth” Approach to Writing a Review
When a reviewer is approached by our editorial staff,
he or she is given an abstract of the manuscript. There
are several things to be considered when deciding
whether or not to accept an invitation to review a
manuscript. The reviewer might have some conﬂict
of interest that prevents truly fair and objective re-
view. Further, the topic or methodology may fall out-
side of the reviewer’s ability to adequately appraise
it. Finally, it may not be possible for the reviewer,
given other time constraints or commitments, to com-
plete the review within the time frame requested. In
any of these cases, it is desirable for the reviewer to
promptly request recusal. If a reviewer is unable to
review a work, an editorial assistant may contact him
or her to ask for suggestions for alternate reviewers.
As experts in their ﬁeld, reviewers can often recom-
mend other qualiﬁed experts in that same area of
study. In all cases, manuscripts released to reviewers
are conﬁdential, and their contents should not be dis-
cussed with others until they are published. A re-
viewer may invite a colleague to assist him or her
with the review but must obtain prior approval from
the editor(s) or the editorial assistant.
In approaching a review, it can be helpful to read
the manuscript a ﬁrst time before ﬁnalizing a re-
sponse. Once time has passed and an initial impres-
sion is formed, the reviewer may return to the manu-
script to ﬁnish the review. Reading the manuscript
more than once, and coming back to it after a break
of a day or more, even when the bulk of the review
has been completed, enables a relatively fresh per-
spective. Depending on the type of article (e.g., re-
ports, commentaries, scholarly reviews, brief re-
ports), the review may require 1–3 hours (2) and 500
to 1,000 words.
At Academic Psychiatry, we advocate focusing on
the goals of a manuscript at an early stage. The goals
of the manuscript or the research hypothesis should
be clearly speciﬁed and should ﬁt with the mission
of our journal. For an empirical paper, for example,
it is important to appreciate that the research goals
determine the methodology; a careful reading of the
goals prepares one to review the Methods section,
one of the most important sections.
The Methods should be evaluated for adequacy
and clarity of the description of the methodological
processes including design, procedures, ethical safe-
guards, and quantitative or qualitative analyses. Lim-
itations in study design, like the absence of a control
group or confounding factors, may reduce the valid-
ity of a study. The sample and method for sampling
should be clearly described so that its representative-
ness to the population, to which the results will be
generalized, can be assessed. One common problem
in analytical research in education is the use of a small
sample size, resulting in a lack of statistical power,
such that even when true differences exist between
84 Academic Psychiatry, 28:2, Summer 2004
groups, these are not detected. Information on any
instruments used including validity (the extent to
which an instrument measures what it purports to
measure) and reliability (the extent to which an in-
strument provides consistent measurements) should
be made available when possible. In essence, the
Methods section should be sufﬁciently well described
to allow someone else to accurately replicate the
The Results should be clearly presented and
highlighted. Statistical tests should be appropriate to
the question. The most important ﬁndings should be
emphasized in a table, the less important in the text,
but none in both places. The real-life signiﬁcance of
results should be judged; sometimes signiﬁcant dif-
ferences between groups exist only on a statistical ba-
sis and do not translate into practical differences.
Some of the common weaknesses of manuscripts in
medical education are inappropriate or incomplete
statistics, omission of data, inconsistent or inaccurate
data, and defective or unclear tables or ﬁgures (1).
Reviewers who query their own ability to adequately
assess the statistics should inform the editor of their
concerns in case the opinion of a statistical consultant
The Discussion section should be judged in part
by the ability of the author(s) to frame and interpret
the main ﬁndings and to honestly assess strengths
and limitations of the research. The reviewer is asked
to judge whether the discussion and conclusions are
adequately supported by the manuscript’s ﬁndings
or arguments. Overinterpretation of the data is a
common weakness in medical education reports;
speculative, unsubstantiated, or unsupported com-
ments should be noted by the reviewer (1).
Paradoxically, the reviewer’s last task is to assess
the Introduction and References. The abstract should
be assessed for its utility as a summary and for ac-
curacy of content. The process of review is necessarily
iterative in that the abstract can be fully evaluated
only after the entire manuscript has been reviewed in
detail (3). Reviewers judge the adequacy of the liter-
ature review, especially searching for the possibility
that important references were omitted. Familiarity
with the relevant literature is expected of the re-
viewer. Similar or contradictory studies should be ex-
haustively referenced; here the author(s) should be
At the point of writing a review, it is helpful to
go over the checklist of criteria provided by Academic
Psychiatry on the “comments to editors” form in order
to ensure that the review will be complete. This
checklist is reviewed by the editors, but can also offer
an excellent outline from which reviewers can for-
mulate comments to the author(s). A common occur-
rence in reviews received by the editorial staff is that
reviewers write all of their comments regarding the
manuscript and potential improvements on the
“comments to the editors” form, which is strictly con-
ﬁdential, and then write little or nothing on the “com-
ments for author(s)” form. Many of these comments
are helpful and insightful and would be beneﬁcial to
the author(s). If a reviewer would like his or her com-
ments to be read by both editors and authors, it is
best to include these comments on both forms.
In formulating comments for the author(s), the
reviewer should ﬁrst describe the intent and potential
value of the manuscript, and note any strengths. Sub-
sequent comments usually enumerate concerns, start-
ing with the science (e.g., methodology, data analysis)
and how these may be remedied if possible. The re-
viewer next turns to a critique of the Discussion, In-
troduction, and References. Lastly and optionally,
some reviewers will include a few editing notes,
pointing out grammatical, formatting, or typograph-
Potential Problems in Review
One mistake often made by reviewers is to assume
that the manuscript will be rejected and to therefore
provide little feedback to the author(s). Regardless of
their recommendation on a manuscript, reviewers
should embrace the collegial collaborative and teach-
ing role entrusted to reviewers. We hope that review-
ers will strive to provide constructive feedback that
will aid authors in the future revision of their work.
One cautionary note: the decision concerning publi-
cation ultimately belongs to the editors who often re-
ceive ambiguous and conﬂicting reviews. Reviewers
should never predict the editorial decision in their
“comments to the author(s).” In addition, we hope
that reviewers will understand that their viewpoint
is always carefully considered when editors deliber-
ate over manuscript decisions. Even if a reviewer’s
recommendation about publishability does not line
up with the ultimate decision by the editors, the con-
ROBERTS ET AL.
Academic Psychiatry, 28:2, Summer 2004 85
tribution of the reviewer to the process is always sin-
cerely appreciated and valued.
One challenge for the reviewer is to achieve a
broad perspective on the potential contribution of a
manuscript to the ﬁeld. This is judged in part by the
extent to which the science or scholarship achieves
more than has been achieved before in our relatively
new and developing ﬁeld. The failure to appreciate
the relative merits of a manuscript in this light will
result in undue weight being assigned to perceived
Lastly, the reviewer must examine his or her po-
tential biases (4). This includes the overrating or un-
derrating of a study because the ﬁndings support or
do not support a preexisting view of the reviewer or
because it is published by a prominent or unrecog-
nized author (4). Although data concerning the value
of blinding reviewers to authors’ identities suggests
that the quality of review is not affected (5–7), Aca-
demic Psychiatry removes the authors’ names and
their afﬁliations from the reviewed manuscript in or-
der to protect against some of these potential biases.
Nevertheless, the responsibility of the reviewer is to
recognize any biasing inﬂuences and to put them
Peer Review and Academic Psychiatry
In an earlier column, one of us (LWR), wrote of the
centrality of peer review to the functions of Academic
Psychiatry and provided some resources for both writ-
ing and review (8). The Journal has an outstanding
tradition of support and mentorship of prospective
authors. We strongly encourage the efforts of those
who are new or relatively new to authoring papers
to get started and to contribute. We welcome queries
about how to proceed, about the suitability of ideas
for the Journal, and about relevant steps in the pub-
lication process. We are committed to professional de-
velopment and to helping new authors attain skills
of value to their academic achievement.
The tradition of Academic Psychiatry has empha-
sized an absolute respect, collegiality, and empathy
in all interactions. It is an honor and privilege to be
selected as reviewer and to have an opportunity to
work cooperatively and constructively as teacher or
mentor to the author. This is challenging, methodical,
conscientious work worth doing truly well. We ap-
plaud the genuine commitment of our reviewers in
this central role for the Journal, and we encourage
new reviewers to embrace the opportunity to become
involved in the peer review process. As editors, one
of our responsibilities is to ensure that we have op-
timized our processes for reviewers and authors alike
in advancing the missions of the Journal and the qual-
ity of publications. We welcome feedback from all
about how we can perform these functions better.
We consider a careful evaluation of a manuscript
by a knowledgeable and fair-minded peer as a criti-
cally important scholarly contribution. The “down-
to-earth” process of peer review is invaluable for the
continued success of authors and of journals. It as-
sures the rigor and fosters the advancement of our
ﬁeld. Though challenging, reviewing manuscripts
can be a gratifying duty in academics, a gift to our
colleagues and a gift that we receive in return.
Bordage G: Reasons reviewers reject and accept manuscripts:
the strengths and weaknesses in medical education reports.
Acad Med 2001; 76:889–896
Hoppin FG: How I review an original scientiﬁc article. Am J
Respir Crit Care Med 2002; 166:1019–1023
Bordage G, Caelleigh AS: How to read “review criteria for
research manuscripts.” Acad Med 2001; 76:908–909
Owen R: Reader bias. JAMA 1982; 247:2533–2534
Van Rooyen S, Godlee F, Evans S, Smith R, Black N: Effect of
blinding and unmasking on the quality of peer review: a ran-
domized trial. JAMA 1998; 280:234–237
Godlee F, Gale CR, Martyn CN: Effect on the quality of peer
review of blinding reviewers and asking them to sign their
reports.: a randomized controlled trial. JAMA 1998; 280:237–
AC, Cho MK, Winker MA, Berlin JA, Rennie D and the PEER
investigators: Does masking author identity improve peer re-
view quality? a randomized controlled trial. JAMA 1998; 280:
Roberts LW: On the centrality of peer review. Acad Psychi-
atry 2002; 26:221–222
86 Academic Psychiatry, 28:2, Summer 2004
CHECKLIST OF REVIEW CRITERIA*
Problem Statement, Conceptual Framework, and Research Question
• The introduction builds a logical case and context for the problem statement.
• The problem statement is clear and well articulated.
• The conceptual framework is explicit and justiﬁed.
• The research question (research hypothesis where applicable) is clear, concise, and complete.
• The variables being investigated are clearly identiﬁed and presented.
Reference to the Literature and Documentation
• The literature review is up-to-date.
• The number of references is appropriate and their selection is judicious.
• The review of the literature is well integrated.
• The references are mainly primary sources.
• The ideas are acknowledged appropriately (scholarly attribution) and accurately.
• The literature is analyzed and critically appraised.
• The study is relevant to the mission of the journal or its audience.
• The study addresses important problems or issues; the study is worth doing.
• The study adds to the literature already available on the subject.
• The study has generalizability because of the selection of subjects, setting, and educational intervention or materials.
• The research design is deﬁned and clearly described, and is sufﬁciently detailed to permit the study to replicated.
• The design is appropriate (optimal) for the research question.
• The design has internal validity, potential confounding variables or biases are addressed.
• The desgn has external validity, including subjects, settings, and conditions.
• The design allows for unexpected outcomes or events to occur.
• The design and conduct of the study are plausible.
Instrumentation, Data Collection, and Quality Control
• The development and content of the instrument are sufﬁciently described or referenced, and are sufﬁciently detailed to
permit the study to be replicated.
• The measurement of instrument is appropriate given the study’s variables; the scoring method is clearly deﬁned.
• The psychometric properties and procedures are clearly presented and appropriate.
• The data set is sufﬁciently described or referenced.
• Observers or raters were sufﬁciently trained.
• Data quality control is described and adequate.
Population and Sample
• The population is deﬁned clearly, both for subjects (participants) and stimulus (intervention), and is sufﬁciently detailed to
permit the study to be replicated.
• The sampling procedures are sufﬁciently described.
• Subject samples are appropriate to the research question.
• Stimulus samples are appropriate to the research questions.
• Selection bias is addressed.
Data Analysis and Statistics
• Data analysis procedures are sufﬁciently described, and are sufﬁciently detailed to permit the study to be replicated.
• Data analysis procedures conform to the research design; hypotheses, models, or theory drives the data analyses.
• The assumptions underlying the use of statistics are fulﬁlled by the data, such as measurement properties of the data and
normality of distributions.
• Statistical tests are appropriate (optimal).
• If statistical analysis involves multiple tests or comparisons, proper adjustment of signiﬁcance level for chance outcomes was
• Power issues are considered in statistical studies with small sample sizes.
• In qualitative research that relies on words instead of numbers, basic requirements of data reliability, validity,
trustworthiness, and absence of bias were fulﬁlled.
Reporting of Statistical Analyses
• The assumptions underlying the use of statistics are considered, given the data collected.
• The statistics are reported correctly and appropriately.
• The number of analyses is appropriate.
• Measures of functional signiﬁcance, such as effect size or proportion of variance accounted for, accompany hypothesis-testing
ROBERTS ET AL.
Academic Psychiatry, 28:2, Summer 2004 87
Presentation of Results
• Results are organized in a way that is easy to understand.
• Results are presented effectively; the results are contextualized.
• The results are complete.
• The amount of data presented is sufﬁcient and appropriate.
• Tables, graphs, or ﬁgures are used judiciously and agree with the text.
Discussion and Conclusion: Interpretation
• The conclusions are clearly stated; key points stand out.
• The conclusions follow from the design, methods, and results; justiﬁcation of conclusions is well articulated.
• Interpretations of the results are appropriate; the conclusions are accurate (not misleading).
• The study limitations are discussed.
• Alternative interpretations for the ﬁndings are considered.
• Statistical differences are distinguished from meaningful differences.
• Personal perspectives or values related to interpretations are discussed.
• Practical signiﬁcance or theoretical implications are discussed; guidance for future studies is offered.
Title, Authors and Abstract
• The title is clear and informative.
• The title is representative of the content and breadth of the study (not misleading).
• The title captures the importance of the study and the attention of the reader.
• The number of authors appears to be appropriate given the study.
• The abstract is complete (thorough); essential details are presented.
• The results in the abstract are presented in sufﬁcient and speciﬁc detail.
• The conclusions in the abstract are justiﬁed by the information in the abstract and the text.
• There are no inconsistencies in detail between the abstract and the text.
• All of the information in the abstract is present in the text.
• The abstract overall is congruent with the text; the abstract gives the same impression as the text.
Presentation and Documentation
• The text is well written and easy to follow.
• The vocabulary is appropriate.
• The content is complete and fully congruent.
• The manuscript is well organized.
• The data reported are accurate (e.g., numbers add up) and appropriate; tables and ﬁgures are used effectively and agree
with the text
• Reference citations are complete and accurate.
• There are no instances of plagiarism.
• Ideas and materials of others are correctly attributed.
• Prior publication by the author(s) of substantial portions of the data or study is appropriately acknowledged.
• There is no apparent conﬂict of interest.
• There is an explicit statement of approval by an institutional review board (IRB) for studies directly involving human
subjects or data about them.
*Reprinted with permission from Academic Medicine, journal of the Association of American Medical Colleges. This ‘‘Checklist of
Review Criteria’’ from the Task Force of Academic Medicine and the GEA-RIME Committee was originally published as Appendix
1 in Vol. 76, No. 9 (September 2001) Academic Medicine.