Improving the Peer-Review Process for Grant Applications: Reliability, Validity, Bias, and Generalizability

Department of Education, University of Oxford, United Kingdom.
American Psychologist (Impact Factor: 6.87). 05/2008; 63(3):160-8. DOI: 10.1037/0003-066X.63.3.160
Source: PubMed


Peer review is a gatekeeper, the final arbiter of what is valued in academia, but it has been criticized in relation to traditional psychological research criteria of reliability, validity, generalizability, and potential biases. Despite a considerable literature, there is surprisingly little sound peer-review research examining these criteria or strategies for improving the process. This article summarizes the authors' research program with the Australian Research Council, which receives thousands of grant proposals from the social science, humanities, and science disciplines and reviews by assessors from all over the world. Using multilevel cross-classified models, the authors critically evaluated peer reviews of grant applications and potential biases associated with applicants, assessors, and their interaction (e.g., age, gender, university, academic rank, research team composition, nationality, experience). Peer reviews lacked reliability, but the only major systematic bias found involved the inflated, unreliable, and invalid ratings of assessors nominated by the applicants themselves. The authors propose a new approach, the reader system, which they evaluated with psychology and education grant proposals and found to be substantially more reliable and strategically advantageous than traditional peer reviews of grant applications.

Download full-text


Available from: Upali Jayasinghe,
  • Source
    • "Moreover, the 'unreliability' of peerreview applies to the natural sciences, the humanities and social sciences. For Marsh et al. (2008), lack of acceptable agreement among independent assessors is the major weakness of peerreview . Some are not surprised. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Peer-review is neither reliable, fair, nor a valid basis for predicting 'impact': as quality control, peer-review is not fit for purpose. Endorsing the consensus, I offer a reframing: while a normative social process, peer-review also shapes the writing of a scientific paper. In so far as 'cognition' describes enabling conditions for flexible behavior, the practices of peer-review thus constrain knowledge-making. To pursue cognitive functions of peer-review, however, manuscripts must be seen as 'symbolizations', replicable patterns that use technologically enabled activity. On this bio-cognitive view, peer-review constrains knowledge-making by writers, editors, reviewers. Authors are prompted to recursively re-aggregate symbolizations to present what are deemed acceptable knowledge claims. How, then, can recursive re-embodiment be explored? In illustration, I sketch how the paper's own content came to be re-aggregated: agonistic review drove reformatting of argument structure, changes in rhetorical ploys and careful choice of wordings. For this reason, the paper's knowledge-claims can be traced to human activity that occurs in distributed cognitive systems. Peer-review is on the frontline in the knowledge sector in that it delimits what can count as knowing. Its systemic nature is therefore crucial to not only discipline-centered 'real' science but also its 'post-academic' counterparts.
    Frontiers in Psychology 11/2015; 6(1085). DOI:10.3389/fpsyg.2015.01706 · 2.80 Impact Factor
  • Source
    • "But peer review has also been criticized on the grounds that it imposes burden on research communities, that the selection of reviewers may introduce biases in the system, and that the reviewers' judgements may be subjective or arbitrary (Kassirer and Campion 1994; Hojat et al. 2003; Li and Agha 2015). Arbitrariness of peer review, which is the quality of accepting submitted items by chance or whim, and not by necessity or rationality, can be measured by the heterogeneity of evaluations among raters during the review process (Mutz et al. 2012; Marsh et al. 2008; Giraudeau et al. 2011). "
    [Show abstract] [Hide abstract]
    ABSTRACT: The principle of peer review is central to the evaluation of research, by ensuring that only high-quality items are funded or published. But peer review has also received criticism, as the selection of reviewers may introduce biases in the system. In 2014, the organizers of the ``Neural Information Processing Systems\rq\rq{} conference conducted an experiment in which $10\%$ of submitted manuscripts (166 items) went through the review process twice. Arbitrariness was measured as the conditional probability for an accepted submission to get rejected if examined by the second committee. This number was equal to $60\%$, for a total acceptance rate equal to $22.5\%$. Here we present a Bayesian analysis of those two numbers, by introducing a hidden parameter which measures the probability that a submission meets basic quality criteria. The standard quality criteria usually include novelty, clarity, reproducibility, correctness and no form of misconduct, and are met by a large proportions of submitted items. The Bayesian estimate for the hidden parameter was equal to $56\%$ ($95\%$CI: $ I = (0.34, 0.83)$), and had a clear interpretation. The result suggested the total acceptance rate should be increased in order to decrease arbitrariness estimates in future review processes.
  • Source
    • "Concerning grant peer reviewing, one of the most frequently cited studies on gender bias, that carried out by Wennerås and Wold (1997), demonstrated that female applicants for postdoctoral fellowships at the Swedish Medical Research Council had to be 2.5 times more productive than the average male applicant in order to obtain the same peer-review rating for scientific competence. Since then, an evergrowing body of academic research has found no conclusive evidence of sex discrimination in the awarding of specific project grants (Wellcome Trust 1997; Ward and Donnelly 1998; Bornmann et al. 2007; Marsh et al. 2008). In this regard, the meta-analyses conducted by Bornmann et al. (2007) and Marsh et al. (2009), and more recently the study by Mutz et al. (2012), have all concluded that there is negligible evidence of gender bias in grant awarding programs. "
    [Show abstract] [Hide abstract]
    ABSTRACT: The main objective of this paper is to study the development and growth of scientific literature on women in science and higher education. A total of 1415 articles and reviews published between 1991 and 2012 were extracted from the Thomson Reuters Web of Science database. Standard bibliometric indicators and laws (e.g. Price’s, Lotka’s, and Bradford’s laws) were applied to these data. In addition, the Gender Inequality Index (GII) was obtained for each country in order to rank them. The results suggest an upward trend not only in the number of papers but also in the number of authors per paper. However, this increase in the number of authors was not accompanied by greater international collaboration. The interest in gender differences in science extends too many authors (n = 3064), countries (n = 67), and research areas (n = 86). Data showed a high dispersion of the literature and a small set of core journals focused on the topic. Regarding the research areas, the area with the highest frequency of papers was Education and Educational Research. Finally, our results showed that countries with higher levels of inequality (higher GII values) tend to present higher relative values of scientific productivity in the field.
    Scientometrics 06/2015; 103(3). DOI:10.1007/s11192-015-1574-x · 2.18 Impact Factor
Show more