No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
(Trans-)National Criteria, Norms and Standards in Literary Studies: A Comparative Analysis of Criteria-Based ex ante Evaluation Forms of Funding Proposals in the Humanities
Abstract
In view of the demand for greater transparency and accountability regarding the expenditure of taxpayers' money, techniques and practices such as benchmarking and the establishment of seemingly objective criteria used in evaluation have been imported from the United States into European higher education where they are used in all sorts of contexts: for the evaluation of manuscripts, theses, applications, and funding proposals. Focusing on an analysis of criteria, the present study examines evaluation forms distributed among reviewers to assess funding proposals in the humanities. By using a comparative framework, the authors seek to make a meta-theoretical and empirical contribution to the lively debate about quality assessment and assurance in the humanities in general and literary studies more particularly. Drawing on data from ten funding organizations in eight European countries, this is the first empirical study of its kind to address the construction and dissemination of (trans-)national criteria, norms and standards and, in doing so, to make a modest attempt to get to grips with the complex meta-category called 'quality'.
... Prescriptive criteria of funding agencies, as summarized by Abdoul et al. (2012), Berning et al. (2015), Falk-Krzesinski and Tobin (2015), and Langfeldt and Scordato (2016), generally overlap with the criteria of peers identified in this article in terms of research quality, quality of description, and feasibility. They differ, however, in two important respects. ...
... First, only the criteria of peers include an assessment of the applicant in terms of the personal qualities motivation and traits. Second, the criteria of peers do not include criteria emphasized by funding agencies, such as strategic importance (Berning et al. 2015), promotion of the public understanding of science (Abdoul et al. 2012), environmental sustainability (Langfeldt and Scordato 2016), or return on investment (Falk-Krzesinski and Tobin 2015). This supports Guston's (2000) view of funding agencies as 'boundary organizations' that stabilize the boundary between the research and the policy domain against external forces and, at the same time, continue to negotiate this boundary internally. ...
Criteria are an essential component of any procedure for assessing merit. Yet, little is known about the criteria peers use to assess grant applications. In this systematic review we therefore identify and synthesize studies that examine grant peer review criteria in an empirical and inductive manner. To facilitate the synthesis, we introduce a framework that classifies what is generally referred to as ‘criterion’ into an evaluated entity (i.e., the object of evaluation) and an evaluation criterion (i.e., the dimension along which an entity is evaluated). In total, the synthesis includes 12 studies on grant peer review criteria. Two-thirds of these studies examine criteria in the medical and health sciences, while studies in other fields are scarce. Few studies compare criteria across different fields, and none focus on criteria for interdisciplinary research. We conducted a qualitative content analysis of the 12 studies and thereby identified 15 evaluation criteria and 30 evaluated entities, as well as the relations between them. Based on a network analysis, we determined the following main relations between the identified evaluation criteria and evaluated entities. The aims and outcomes of a proposed project are assessed in terms of the evaluation criteria originality, academic relevance, and extra-academic relevance. The proposed research process is evaluated both on the content level (quality, appropriateness, rigor, coherence/justification), as well as on the level of description (clarity, completeness). The resources needed to implement the research process are evaluated in terms of the evaluation criterion feasibility. Lastly, the person and personality of the applicant are assessed from a ‘psychological’ (motivation, traits) and a ‘sociological’ (diversity) perspective. Furthermore, we find that some of the criteria peers use to evaluate grant applications do not conform to the fairness doctrine and the ideal of impartiality. Grant peer review could therefore be considered unfair and biased. Our findings suggest that future studies on criteria in grant peer review should focus on the applicant, include data from non-Western countries, and examine fields other than the medical and health sciences.
... Prescriptive criteria of funding agencies, as summarized by Abdoul et al. (2012), Berning et al. (2015), Falk-Krzesinski and Tobin (2015), and Langfeldt and Scordato (2016), generally overlap with the criteria of peers identified in this article in terms of research quality, quality of description, and feasibility. They differ, however, in two important respects. ...
... First, only the criteria of peers include an assessment of the applicant in terms of the personal qualities motivation and traits. Second, the criteria of peers do not include criteria emphasized by funding agencies, such as strategic importance (Berning et al. 2015), promotion of the public understanding of science (Abdoul et al. 2012), environmental sustainability (Langfeldt and Scordato 2016), or return on investment (Falk-Krzesinski and Tobin 2015). This supports Guston's (2000) view of funding agencies as 'boundary organizations' that stabilize the boundary between the research and the policy domain against external forces and, at the same time, continue to negotiate this boundary internally. ...
Criteria are an essential component of any procedure for assessing merit. Yet, little is known about the criteria peers use to assess grant applications. In this systematic review we therefore identify and synthesize studies that examine grant peer review criteria in an empirical and inductive manner. To facilitate the synthesis, we introduce a framework that classifies what is generally referred to as ‘criterion’ into an evaluated entity (i.e., the object of evaluation) and an evaluation criterion (i.e., the dimension along which an entity is evaluated). In total, the synthesis includes 12 studies on grant peer review criteria. Two-thirds of these studies examine criteria in the medical and health sciences, while studies in other fields are scarce. Few studies compare criteria across different fields, and none focus on criteria for interdisciplinary research. We conducted a qualitative content analysis of the 12 studies and thereby identified 15 evaluation criteria and 30 evaluated entities, as well as the relations between them. Based on a network analysis, we determined the following main relations between the identified evaluation criteria and evaluated entities. The aims and outcomes of a proposed project are assessed in terms of the evaluation criteria originality, academic relevance, and extra-academic relevance. The proposed research process is evaluated both on the content level (quality, appropriateness, rigor, coherence/justification), as well as on the level of description (clarity, completeness). The resources needed to implement the research process are evaluated in terms of the evaluation criterion feasibility. Lastly, the person and personality of the applicant are assessed from a ‘psychological’ (motivation, traits) and a ‘sociological’ (diversity) perspective. Furthermore, we find that some of the criteria peers use to evaluate grant applications do not conform to the fairness doctrine and the ideal of impartiality. Grant peer review could therefore be considered unfair and biased. Our findings suggest that future studies on criteria in grant peer review should focus on the applicant, include data from non-Western countries, and examine fields other than the medical and health sciences.
... Compared with the evaluation of research output such as publications, the assessment of research proposals in the grant application process is more complex. Many scholars have examined grant review criteria from different perspectives, in which the originality, scientific value, social relevance, plausibility and feasibility are demonstrated as the key dimensions of research quality for peer reviewers (Hume et al. 2015;Aksnes et al. 2019;Hug and Aeschbach 2020), while the funding agencies place high value on criterion such as strategic importance, return on investment, environmental sustainability (Berning et al. 2015;Falk-Krzesinski and Tobin 2015). ...
Based on the theory of developmental evaluation, this paper develops a new development-oriented research evaluation model called "four abilities", with "basic development ability", "process development ability", "achievement ability" and "influence ability" as key dimensions, which can be applied in different situations such as recruitment, performance appraisal, promotion evaluation, tenure review, and selection of honorary academic awards. Based on the data from the Youth Fund of the Ministry of Management Science of National Natural Science Foundation of China (NSFC) from 2014 to 2018, this paper takes the developmental evaluation of early career researchers as an example, treating the existing research output as the development goal and the "development ability" of early career researchers as the explanatory factor. Then the propensity score matching method (PSM) method is adopted to control the sample self-selection bias in the way of reverse order evaluation, so as to explain whether the guiding indicator "development ability" is conducive to the development of researchers. The results indicate that strong “process development ability” of newly recruited researchers in the first 3 years of employment period can significantly promote their future research performance, which has important implications for the current evaluative culture overemphasizing short-term output. Through the application of the “four abilities” model integrating various dimensions composed of potential, capacity, output and impact, researchers at different career stages are guided to concentrate more on long-term academic mission and to achieve better career development based on their differentiated development needs.
... First, only the criteria of peers include an assessment of the applicant in terms of the personal qualities motivation and traits. Second, the criteria of peers do not include criteria emphasized by funding agencies, such as strategic importance (Berning et al., 2015), promotion of the public understanding of science (Abdoul et al., 2012), environmental sustainability (Langfeldt and Scordato, 2016), or return on investment (Falk-Krzesinski and Tobin, 2015). This supports Guston's (2000) view of funding agencies as 'boundary organizations' that stabilize the boundary between the research and the policy domain against external forces and, at the same time, continue to negotiate this boundary internally. ...
Criteria are an essential component of any procedure for assessing merit. Yet, little is known about the criteria peers use to assess grant applications. In this systematic review we therefore identify and synthesize studies that examine grant peer review criteria in an empirical and inductive manner. To facilitate the synthesis, we introduce a framework that classifies what is generally referred to as ‘criterion’ into an evaluated entity (i.e., the object of evaluation) and an evaluation criterion (i.e., the dimension along which an entity is evaluated). In total, the synthesis includes 12 studies on grant peer review criteria. Two-thirds of these studies examine criteria in the medical and health sciences, while studies in other fields are scarce. Few studies compare criteria across different fields, and none focus on criteria for interdisciplinary research. We conducted a qualitative content analysis of the 12 studies and thereby identified 15 evaluation criteria and 30 evaluated entities, as well as the relations between them. Based on a network analysis, we determined the following main relations between the identified evaluation criteria and evaluated entities. The aims and outcomes of a proposed project are assessed in terms of the evaluation criteria originality, academic relevance, and extra-academic relevance. The proposed research process is evaluated both on the content level (quality, appropriateness, rigor, coherence/justification), as well as on the level of description (clarity, completeness). The resources needed to implement the research process are evaluated in terms of the evaluation criterion feasibility. Lastly, the person and personality of the applicant are assessed from a ‘psychological’ (motivation, traits) and a ‘sociological’ (diversity) perspective. Furthermore, we find that some of the criteria peers use to evaluate grant applications do not conform to the fairness doctrine and the ideal of impartiality. Grant peer review could therefore be considered unfair and biased. Our findings suggest that future studies on criteria in grant peer review should focus on the applicant, include data from non-Western countries, and examine fields other than the medical and health sciences.
Although there is no single European way of doing research evaluation, it has been going on since the 1970s, importantly supported
by the European Community. Trends include: evaluating socio-economic impacts, commenting on the importance and appropriateness
of the evaluated activities, modified peer review, semi-professionalism, and tendering to groups of experts in different countries.
When distributing grants, research councils use peer expertise as a guarantee for supporting the best projects. However, there are no clear norms for assessments, and there may be a large variation in what criteria reviewers emphasize - and how they are emphasized. The determinants of peer review may therefore be accidental, in the sense that who reviews what research and how reviews are organized may determine outcomes.
This paper deals with how the review process affects the outcome of grant review. The case study considers the procedures of The Research Council of Norway, which practises several different grant-review models, and consequently is especially suited for explorations of the implications of different models. Data sources are direct observation of panel meetings, interviews with panel members and study of applications and review documents.
A central finding is that rating scales and budget restrictions are more important than review guidelines for the kind of criteria applied by the reviewers. The decision-making methods applied by the review panels when ranking proposals are found to have substantial effects on the outcome. Some ranking methods tend to support uncontroversial and safe projects, whereas other methods give better chances for scholarly pluralism and controversial research.
Many countries have introducedevaluations of university research, reflectingglobal demands for greater accountability. Thispaper compares methods of evaluation usedacross twelve countries in Europe and theAsia-Pacific region. On the basis of thiscomparison, and focusing in particular onBritain, we examine the advantages anddisadvantages of performance-based funding incomparison with other approaches to funding.Our analysis suggests that, while initialbenefits may outweigh the costs, over time sucha system seems to produce diminishing returns.This raises important questions about itscontinued use.
What are the institutional mechanisms that enable or hinder the development of new forms of knowledge production? This issue has been slightly neglected in the discussion of the “triple helix”. To redress this shortcoming, the authors suggest an institutionalist complement to the triple helix model. The article analyzes the institutional regulation of academic research, with a special emphasis on how norms in the academic system are constituted via research funding. It is argued that funding is a key mechanism of change in the norm system since its reward structure influences the performance and evaluation of research. The empirical analysis is based on the public financing of technical research in Sweden, with comparisons made with other countries. The structure of research funding has been reformed in all the countries studied. In addition to continuing recognition for scientific merit, the reforms have had the effect of emphasizing the commercial potential and the societal relevance of the research supported. The two dominant models of research funding, an intra-academic model and a top–down interventionist model, seem to be replaced partly with a catalytic one. However, there are counteracting tendencies. Some agencies still reproduce a model of reputational control and a collegial orientation among researchers. It is concluded, therefore, that the forces of change and continuity are engaged in a process of negotiation about the normative regulation of academic research.