Content uploaded by Kesten Green
Author content
All content in this area was uploaded by Kesten Green on May 04, 2021
Content may be subject to copyright.
Paper title:
Reviewer:
Date:
Time spent (minutes):
Instructions for Raters
1. Skim the paper while you complete the checklist as a skeptical
reviewer.
2. Rate each lettered item, below, marking the relevant checkbox to
indicate
True if the research complies,
F/? (False/Unclear) if the research does not comply, or if you are
unsure.
IMPORTANT: If you are not convinced that the paper complied, rate
the item F/?
3. If you rate an item True, give reasons for your rating in your own
words.
4. Rate criteria 1-8 as True by marking the checkbox only if all lettered
items for the criterion are rated T.
First assess whether the paper complies with the lettered items
under each criterion below. Then assess whether it complies
with each of the eight criteria based on compliance with the
lettered items. Avoid speculation.
1. Problem is important for decision making, policy, or method
development
True
T F/?
a. Importance of the problem clear from the title, abstract,
result tables, or conclusions
b. Findings add to cumulative scientific knowledge
c. Uses of the findings are clear to you
d. The findings can be used to improve people’s lives without
resorting to duress or deceit
2. Prior knowledge was comprehensively reviewed and
summarized
True
T F/?
a. The paper describes objective and comprehensive
procedures used to search for prior useful scientific
knowledge
b. The paper describes how prior substantive findings were
used to develop hypotheses (e.g. direction and magnitude of
effects of causal variables) and research procedures
3. Disclosure is sufficiently comprehensive for understanding
and replication
True
T F/?
a. Methods are fully and clearly described so as to be
understood by all relevant stakeholders, including potential
users
b. Data are easily accessible using information provided in the
paper
c. Sources of funding are described, or absence of external
funding noted
4. Design is objective (unbiased by advocacy)
True
T F/?
a. Prior hypotheses are clearly described (e.g., regarding
directions and magnitudes of relationships, and effects of
conditions)
b. All reasonable hypotheses are included in the design,
including plausible naïve, no-meaningful-difference, and
current-practice hypotheses
c. Revisions to hypotheses are described, or absence of
revisions noted
5. Data are valid (true measures) and reliable
(repeatable measures)
True
T F/?
a. Data were shown to be relevant to the problem
b. All relevant data were used, including the longest
relevant time-series
c. Reliability of data was assessed
d. Other information needed for assessing the
validity of the data is provided, such as
adjustments, known shortcomings and potential
biases
6. Methods were validated (proven fit for purpose)
and simple
True
T F/?
a. Methods were explained clearly and shown
valid—unless well known to intended readers,
users, and reviewers, and validity is obvious
b. Methods were sufficiently simple for potential
users to understand
c. Multiple validated methods were used
d. Methods used cumulative scientific knowledge
explicitly
7. Experimental evidence was used to compare
alternative hypotheses
True
T F/?
a. Experimental evidence was used to compare
hypotheses under explicit conditions
b. Predictive validity of hypotheses was tested using
out-of-sample data
8. Conclusions follow logically from the evidence
presented
True
T F/?
a. Conclusions do not go beyond the evidence in the
paper
b. Conclusions are not the product of confirmation
bias
c. Conclusions do not reject a hypothesis by denying
the antecedent
d. Conclusions do not support a hypothesis by
affirming the consequent
Describe the most important scientific finding in your own
words.
Sum the criteria (1–8) rated True for compliance: [ ] of 8
Checklist 1: Compliance With Science Checklist
From Armstrong, J. S., & Green, K.
C. (2020) The Scientific Method.
Forthcoming.
Checklist 2: Identifying Important Problems
1. Work independently
2. Problem-storm using brainwriting
3. Develop solutions alone, ignoring others’ solutions at first
4. Get close to the problem to learn about current solutions
5. Seek help from others
6. Build on potential solutions while avoiding evaluation
From Armstrong, J. S., & Green, K.
C. (2020) The Scientific Method.
Forthcoming.
Checklist 3: Conducting a Useful Scientific Study
Selecting an important problem
1. Choose a problem for which findings are likely to provide
benefits without duress or deceit
2. Be skeptical about findings, theories, policies, methods, and
data when lacking experimental evidence
3. Consider conducting replications and extensions of papers
that address important problems
4. Ensure that you can address the problem impartially
5. If you need funding, ensure that you will nevertheless have
control over your study
Designing a study
6. Summarize existing scientific knowledge about the problem
7. Develop multiple reasonable hypotheses with specified
conditions prior to any analysis
8. Design a study that minimizes the risk of harm to subjects
9. Pretest experiments
10. Warn subjects if they might find the task unpleasant
11. Consider role-play as alternative to experimentation
12. Design experiments that estimate effect directions and sizes
so as to identify the best hypothesis
Collecting data
13. Obtain all valid data
14. Ensure that the data are reliable
Analyzing data
15. Use models that incorporate cumulative knowledge
16. Use simple methods
17. Use multiple validated methods
18. Draw conclusions only on practical importance of effects
From Armstrong, J. S., & Green, K.
C. (2020) The Scientific Method.
Forthcoming.
Checklist 4: Content of a Scientific Paper
1. Explanation of findings, and why they are credible and useful
2. Descriptions of prior hypotheses and any changes
3. Descriptions of data and methods allowing assessments of
validity, and replication
4. Verified citations of scientific papers with substantive
findings relevant to the evidence presented
5. List of those who provided author-solicited peer review
6. List of funders who expect to be acknowledged
7. An oath on responsibility and disclosure
From Armstrong, J. S., & Green, K.
C. (2020) The Scientific Method.
Forthcoming.
Checklist 5: Writing a Scientific Paper
1. Make the first word in the title descriptive, avoiding
adjectives, including unnecessary articles (“a”, “the”)
2. Use a short descriptive title describing your findings
3. Use past tense to report findings to avoid implying that
the issue is settled.
4. Provide a structured abstract (see Checklist 10)
5. Use an introduction to let readers know what to expect
6. Use descriptive headings to guide readers
7. Use numbers or letters for three or more items in a list
8. Use examples to illustrate findings, not as evidence
9. Organize tables and charts so conclusions are obvious
10. Avoid pie charts
11. Avoid colors, unless informative and necessary
12. Be specific using words with concrete meaning
13. Avoid negative words for ease of understanding
14. Use short sentences and avoid unnecessary words
15. Avoid uncommon words, unless explained
16. Break text into paragraphs that contain one idea each
17. Describe how each substantively cited work provides
evidence (i.e., avoid mysterious citations)
18. Cite a source for evidence only if it has been read by at
least one of the authors
19. Put citations near at the end of sentences
20. Use a common serif typeface with black-on-white text
21. Use a calm tone, avoiding exclamation marks and
uppercase in the text
22. Use footnotes sparingly
23. Rewrite the report until it is clear and interesting
24. Use editors to improve clarity
25. Proofread to eliminate errors
TOTAL
[ ]
From Armstrong, J. S., & Green, K.
C. (2020) The Scientific Method.
Forthcoming.
Checklist 6: Elements of a Structured Abstract
1. Purpose
What problem does the paper address?
2. Methods
How was the problem addressed?
3. Findings
What data were obtained, and what did
the analysis show?
4. Limitations
What are the caveats to the findings?
5. Implications
What are the practical implications and
why are they important?
From Armstrong, J. S., & Green, K.
C. (2020) The Scientific Method.
Forthcoming.
Checklist 7: Disseminating Useful Scientific Findings
1. Provide thorough responses to journal reviewers
2. If a paper with useful scientific findings is rejected, appeal
to the editor
3. Publish in PLOS-One or similar if you meet their criteria
4. Publish a working paper on ResearchGate or similar
5. Publish research findings in a book
6. Directly inform those who can use your findings
7. Cooperate with those who want to replicate your study
8. Publish corrections for mistakes found after publication
From Armstrong, J. S., & Green, K.
C. (2020) The Scientific Method.
Forthcoming.
Checklist 8: Preparing a Talk on Scientific Findings
Checklist items are based on evidence from Armstrong (2010) or are
logical or based on expert judgments.
Organization
1. Use a single theme to organize your scientific findings
2. Describe objectives of the talk, including actions-steps
3. Build the presentation around your scientific findings
4. Show evidence for your findings
5. Use two-sided arguments describing risks, limitations
6. Avoid jargon and uncommon words
7. Plan to take less time than is available to allow for
interruptions and problems
8. Have additional slides in reserve in case there are few
questions or you have more time than expected
9. Rehearse: If the talk is important, present your talk to one or
more people acting as your intended audience
10. Prepare a hard copy agenda for your audience
11. Provide a link to your paper or slides
12. Prepare hard copies of your slides in case of problems
Visuals
13. Use simple visual aids, especially for data
14. Synchronise oral and visual parts of the talk by using
animations that introduce one point at a time
15. Keep to 10 lines or fewer of text for most slides
16. Eliminate anything from visuals that contains no useful
information (e.g., wallpaper or color)
17. Use high contrast (e.g., black text on white background) to
enhance legibility
18. Use a sans serif font to enhance legibility
19. Provide informative titles for each exhibit
TOTAL
[ ]
From Armstrong, J. S., & Green, K.
C. (2020) The Scientific Method.
Forthcoming.
Checklist 9: Making an Oral Presentation
1. Use one speaker
2. Ask the audience to write suggestions for improvements
3. Answer only clarification questions during the talk
4. Acknowledge non-clarification questions, and undertake to address
them at the end of the talk
5. Pause for two seconds before key points to create interest
6. Pause after key points to allow people time to reflect
7. Pose questions, pause, then answer your own question in order to
gain attention
8. Make eye contact to raise interest and increase trust
9. Avoid humor so as not to distract from the talk’s content
10. Repeat key points by rephrasing them
11. Orient questions toward improving the paper
12. Ask for clarification if uncertain about a question and offer to
discuss after the talk if you don’t have an answer
13. Avoid answering questions that need your further consideration;
note the questions and respond later
TOTAL
[ ]
From Armstrong, J. S., & Green, K.
C. (2020) The Scientific Method.
Forthcoming.
Checklist 10: Self-Assessment of Self-Control
Indicate how much each of the following statements reflects how you typically are
(not at all to very much) by circling the appropriate number in one of the five columns
to the right. *
Frequency
rarely … often
I am good at resisting temptation
1
2
3
4
5
I have a hard time breaking bad habits
5
4
3
2
1
I am lazy
5
4
3
2
1
I say inappropriate things
5
4
3
2
1
I do certain things that are bad for me, if they are fun
5
4
3
2
1
I refuse things that are bad for me
1
2
3
4
5
I wish I had more self-discipline
5
4
3
2
1
People would say that I have iron self-discipline
1
2
3
4
5
Pleasure and fun sometimes keep me from getting work done
5
4
3
2
1
I have trouble concentrating
5
4
3
2
1
I am able to work effectively toward long-term goals
1
2
3
4
5
Sometimes I can’t stop myself from doing something, even if I know it is wrong
5
4
3
2
1
I often act without thinking through all the alternatives
5
4
3
2
1
Sum the circled figures to calculate a total score between 13 and 65
TOTAL SCORE
[ ]
*Adapted from Tangney, Baumeister, and Boone’s (2004) “Brief Self-Control Scale”.
From Armstrong, J. S., & Green, K.
C. (2020) The Scientific Method.
Forthcoming.