Article

Improving the peer-review process for grant applications - Reliability, validity, bias, and generalizability

Department of Education, University of Oxford, United Kingdom.
American Psychologist (Impact Factor: 6.87). 05/2008; 63(3):160-8. DOI: 10.1037/0003-066X.63.3.160
Source: PubMed

ABSTRACT Peer review is a gatekeeper, the final arbiter of what is valued in academia, but it has been criticized in relation to traditional psychological research criteria of reliability, validity, generalizability, and potential biases. Despite a considerable literature, there is surprisingly little sound peer-review research examining these criteria or strategies for improving the process. This article summarizes the authors' research program with the Australian Research Council, which receives thousands of grant proposals from the social science, humanities, and science disciplines and reviews by assessors from all over the world. Using multilevel cross-classified models, the authors critically evaluated peer reviews of grant applications and potential biases associated with applicants, assessors, and their interaction (e.g., age, gender, university, academic rank, research team composition, nationality, experience). Peer reviews lacked reliability, but the only major systematic bias found involved the inflated, unreliable, and invalid ratings of assessors nominated by the applicants themselves. The authors propose a new approach, the reader system, which they evaluated with psychology and education grant proposals and found to be substantially more reliable and strategically advantageous than traditional peer reviews of grant applications.

Download full-text

Full-text

Available from: Upali Jayasinghe, Jun 17, 2015
0 Followers
 · 
239 Views
  • [Show abstract] [Hide abstract]
    ABSTRACT: The selection of projects for funding can hide discriminatory decisions. We present a case study investigating gender discrimination in a dataset of scientific research proposals submitted to an Italian national call. The method for the analysis relies on a data mining classification strategy that is inspired by a legal methodology for proving evidence of social discrimination against protected-by-law groups.
    Data Mining Workshops (ICDMW), 2012 IEEE 12th International Conference on; 01/2012
  • [Show abstract] [Hide abstract]
    ABSTRACT: The main objective of this paper is to study the development and growth of scientific literature on women in science and higher education. A total of 1415 articles and reviews published between 1991 and 2012 were extracted from the Thomson Reuters Web of Science database. Standard bibliometric indicators and laws (e.g. Price’s, Lotka’s, and Bradford’s laws) were applied to these data. In addition, the Gender Inequality Index (GII) was obtained for each country in order to rank them. The results suggest an upward trend not only in the number of papers but also in the number of authors per paper. However, this increase in the number of authors was not accompanied by greater international collaboration. The interest in gender differences in science extends too many authors (n = 3064), countries (n = 67), and research areas (n = 86). Data showed a high dispersion of the literature and a small set of core journals focused on the topic. Regarding the research areas, the area with the highest frequency of papers was Education and Educational Research. Finally, our results showed that countries with higher levels of inequality (higher GII values) tend to present higher relative values of scientific productivity in the field.
    Scientometrics 06/2015; 103(3). DOI:10.1007/s11192-015-1574-x · 2.27 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Peer review represents the primary mechanism used by funding agencies to allocate financial support and by journals to select manuscripts for publication, yet recent Cochrane reviews determined literature on peer review best practice is sparse. Key to improving the process are reduction of inherent vulnerability to high degree of randomness and, from an economic perspective, limiting both the substantial indirect costs related to reviewer time invested and direct administrative costs to funding agencies, publishers and research institutions. Use of additional reviewers per application may increase reliability and decision consistency, but adds to overall cost and burden. The optimal number of reviewers per application, while not known, is thought to vary with accuracy of judges or evaluation methods. Here I use bootstrapping of replicated peer review data from a Post-doctoral Fellowships competition to show that five reviewers per application represents a practical optimum which avoids large random effects evident when fewer reviewers are used, a point where additional reviewers at increasing cost provides only diminishing incremental gains in chance-corrected consistency of decision outcomes. Random effects were most evident in the relative mid-range of competitiveness. Results support aggressive high- and low-end stratification or triaging of applications for subsequent stages of review, with the proportion and set of mid-range submissions to be retained for further consideration being dependent on overall success rate.
    PLoS ONE 04/2015; 10(4):e0120838. DOI:10.1371/journal.pone.0120838 · 3.53 Impact Factor