No full-text available
To read the full-text of this research,
you can request a copy directly from the author.
Performance-Based University Research Funding Systems
Abstract
The university research environment has been undergoing profound change in recent decades and performance-based research funding systems (PRFSs) are one of the many novelties introduced. This paper seeks to find general lessons in the accumulated experience with PRFSs that can serve to enrich our understanding of how research policy and innovation systems are evolving. The paper also links the PRFS experience with the public management literature, particularly new public management, and understanding of public sector performance evaluation systems. PRFSs were found to be complex, dynamic systems, balancing peer review and metrics, accommodating differences between fields, and involving lengthy consultation with the academic community and transparency in data and results. Although the importance of PRFSs seems based on their distribution of universities’ research funding, this is something of an illusion, and the literature agrees that it is the competition for prestige created by a PRSF that creates powerful incentives within university systems. The literature suggests that under the right circumstances a PRFS will enhance control by professional elites. PRFSs since they aim for excellence, may compromise other important values such as equity or diversity. They will not serve the goal of enhancing the economic relevance of research.
... A large body of higher education examines policies that consistently highlight the significance and need for external funding. In today's academic landscape, the scientific reputation and prestige of universities are increasingly established through their participation in competitive external research funding schemes and research networks (Auranen & Nieminen, 2010;Brankovic, 2018;Frølich et al., 2017;Hicks, 2012;Muscio et al., 2013;Musselin, 2018;Pucciarelli & Kaplan, 2016). The availability of and access to external research funding has become strategically critical for universities (Hicks, 2012;Parker, 2013;Sharrock, 2012;Stachowiak-Kudla & Kudla, 2017;Young et al., 2017). ...
... In today's academic landscape, the scientific reputation and prestige of universities are increasingly established through their participation in competitive external research funding schemes and research networks (Auranen & Nieminen, 2010;Brankovic, 2018;Frølich et al., 2017;Hicks, 2012;Muscio et al., 2013;Musselin, 2018;Pucciarelli & Kaplan, 2016). The availability of and access to external research funding has become strategically critical for universities (Hicks, 2012;Parker, 2013;Sharrock, 2012;Stachowiak-Kudla & Kudla, 2017;Young et al., 2017). In these contexts, performing well using governmental core funding does not contribute to building an ideal reputation and prestige from which, for example, universities can communicate to potential new external research partners. ...
... In addition to various other actors, higher education institutions themselves are now taking proactive measures by initiating strategies and policies to increase external funding in order to enhance institutional competitiveness (Musselin, 2018;Parker, 2013;Pucciarelli & Kaplan, 2016;Stachowiak-Kudla & Kudla, 2017;Teixeira & Koryakina, 2013). Furthermore, universities are driven to compete for funding and prestige due to a combination of global pressures and policies, national policies, and funding incentives (Auranen & Nieminen, 2010;Franssen et al., 2018;Gunn & Mintrom, 2016;Hicks, 2012;Musselin, 2018). ...
Universities are facing increasing external pressures to compete for external funding and to develop mixed economies comprising both state budget funding and external sources. This study aims to provide empirical and theoretical insights by utilising the Resource Dependence Theory and the Ecology of Games Metaphor to examine how Finnish universities articulate their goals for external funding in their institutional strategies. The findings indicate that universities’ central strategic goals and interests in this specific domain of funding align with their aspirations for top-tier research university status and how to advance it. Rather than choosing between different funding streams, universities are adapting to the conditions of their funding environments. The strategies also reveal the presence of contradictory goals, where potential positive externalities serve as justifications for these contradictions. The strategies do not necessarily reveal unique or innovative choices made by the universities, but rather express their future institutional image as being more driven by external funding and research. These strategic goals can be seen as responses to critical dependence on academics and academic research, and as heuristic tools of the university management arena in navigating uncertain and dynamic university environments.
... Participating in competitive calls in order to obtain funding may seem quite natural in contemporary academia, but it has not always been the prevailing funding mechanism, nor has competition always been the process through which funds were allocated to academic institutions. Back in time, states distributed funds among academic institutions 'with no strings attached' (Geuna, 2001;Hallonsten, 2021), on the basis of the number of faculty and/or students (Hicks, 2012). Academics themselves were left to distribute these funds internally, on the assumption that they had the knowledge, experience and capacity to take the best decision regarding the appropriate use of that public funding (Hallonsten, 2021; see also De Oliveira, 2013). ...
... But these figures do not reflect the real influence and pervasiveness of CPBF. The rise of CPBF has been accompanied by the adoption of performance-based research funding systems (PRFSs, hereinafter) (Hicks, 2012;Wang et al., 2020). PRFSs basically involve governments conditioning part of the institutional funding transferred to universities on meeting predefined performance thresholds. ...
... These performance targets are multiple and may vary significantly between different cases, but always include teaching-and research-related items. The key point is that grant funding features prominently among the targets set in terms of research activities (Hicks, 2012;Wang et al., 2020). It means that, rather perversely, the system rewards with more institutional funding those universities that have raised more money through CPBF, which is usually public funding as well. ...
Academics today are expected not only to produce knowledge, but to communicate it to broad audiences as well. Science communication (SC) has become a central aspect of the scientific enterprise and an obligation that academics must comply with. Project-based SC is one of the forms through which that task is imposed on academics. Funding agencies around the world ask academics to carry out communication activities alongside their core scientific activities in those research projects funded by the former. By analyzing project-based SC in the light of the determinants of the capitalist relations of production captured by the Marxian category of ‘formal subsumption of labor under capital’, this chapter will demonstrate that this contemporary modality of SC can be regarded as an expression of the formal subsumption of academic labor under capital. This is because project-based SC is grounded on competitive project-based funding (CPBF), and CPBF is actually one of the most important mechanisms through which the separation of academic labor from its conditions of production is affected. As it explores the economic foundations of project-based SC and the nature of CPBF as a funding mechanism, this chapter adds to current Marxian scholarship focusing on the commodification of academic research and higher education, more broadly speaking.
... Back in time, in both Europe and in the United States 'block funding' was the main instrument for the allocation and distribution of public funds to universities and thus to academic research. Universities received funding on the basis of the number of faculty and/or students (Hicks, 2012), so there were no competitive criteria in place. Only in this sense alone it can be affirmed that block funding came with 'no strings attached' (Hallonsten, 2021). ...
... And, yet, the influence and pervasiveness of CPBF as a funding arrangement is not wholly reflected by these figures. For CPBF has developed alongside other transformations in institutional or block funding, changes that have led to the emergence of Performance-based Research Funding Systems (PRFS) (Hicks, 2012). PRFS means that part of the institutional funding is conditioned to universities meeting pre-defined thresholds of performance. ...
... Multiple targets are considered in each case, covering an ample range of academic activities, including teaching, research or even administration. The crux of the matter is that the volume of money raised through CPBF features prominently among those thresholds of performance that universities must meet (Hicks, 2012;Wang et al., 2020). In a rather perverse manner, PRFS funding schemes reward with extra institutional funding for those universities that have been more successful in raising third-party funding through CPBF, which is also public funding. ...
The current growth of competitive project-based funding (CPBF) as a funding instrument for academic science reveals that public funding plays a critical role in the spreading of the capitalist relations of production in academia. However, this issue has not been properly addressed in the extant literature. This paper examines CPBF in the light of the determinations of capitalist relations of production captured by the Marxist notion of ‘formal subsumption of labour under capital’. It will then show that CPBF mediates commodity-based productive relations between funding agencies and academic institutions, and that the latter are, in turn, premised on the separation of academic labour from the objective conditions of knowledge production. It will be also demonstrated how CPBF reproduces and deepens that split, leading from the partial to the complete formal subsumption of academic labour under capital. Our analysis challenges the assumption that increased public funding will put to a halt the commodification of academia and academic research.
... These divergent conceptions have given rise to distinct evaluation cultures, as noted by Hicks (2012), Lepori, Reale, and Spinello (2018), and Ochsner, Kulczycki, and Gedutis (2018). Over the centuries, evaluation practices have undergone transformations, resulting in hybrid and highly heterogeneous models. ...
... National agencies entrusted with the regulation and funding of higher education and research began to advocate for the establishment of permanent and standardized evaluation policies. According to these agencies, institutionalizing these processes would allow the assessment of the results resulting from increased public investments, while ensuring regularity, transparency, and consistency of criteria and procedures at the national level (Hicks, 2012;Ochsner, Kulczycki, and Gedutis, 2018). In addition to being an internal and exclusive concern of universities and researchers, evaluation gradually assumed a prominent role with-in national systems of higher education, science and technology (Amaral, 2009;Verhine and Freitas, 2012). ...
... Drawing from this study, we highlight certain aspects that can provide valuable insights for enhancing the higher education evaluation system. (Hicks, 2012;Drooge et al., 2013). ...
This paper presents the results of a study on the evaluation of research and graduate education in the Netherlands. The policies implemented in the country over the last four decades have consolidated a sui generis and stable evaluation system, contributing to strengthening the country's scientific leadership on the international scene. The Dutch experience has inspired several countries, and its lessons could also be relevant for the Brazilian reality. In this context, this study aimed to analyse the main characteristics of the Dutch system, understand the role of self-evaluation within that system and present a self-evaluation experience conducted by a noteworthy research centre from Leiden University. From the lessons learned, we conclude by presenting some subsidies for improving the evaluation of graduate education conducted in Brazil.
KEYWORDS
evaluation policies; self-evaluation; institutional autonomy; the Netherlands
... Scientific research is a driving force of innovation designed to expand the frontiers of human knowledge and improve economic and social progress. However, research policy and exploration of promising research directions are shaped in part by the decisions of funding bodies, such as governments and universities, as well as for-profit and nonprofit private entities, to fund these research studies [1,2]. Determining which proposed research projects are funded based on impact remains a dynamic process that involves a combination of peer review and quantitative research metrics [2,3]. ...
... However, research policy and exploration of promising research directions are shaped in part by the decisions of funding bodies, such as governments and universities, as well as for-profit and nonprofit private entities, to fund these research studies [1,2]. Determining which proposed research projects are funded based on impact remains a dynamic process that involves a combination of peer review and quantitative research metrics [2,3]. The funding decision process requires transparency in the way public research funds are allocated based on peer review and metrics and can ensure reproducibility through increased use of publicly available data for responsible decision-making [4,5]. ...
... The pursuit of scientific research is intricately tied to the progress of human society, and it is shaped by decisions made by various funding bodies, such as governmental organizations, universities, and both forprofit and nonprofit private entities, who provide financial support for these research studies [1,2]. The National Institutes of Health (NIH), for instance, allocated a budget of 33.34 billion dollars towards scientific research in 2022 [18], highlighting the significant investment made in this area. ...
Scientific research is propelled by allocation of funding to different research projects based in part on the predicted scientific impact of the work. Data-driven algorithms can inform decision-making of funding by identifying likely high-impact studies using bibliometrics. Compared to standardized citation-based metrics alone, we utilize a machine learning pipeline that analyzes high-dimensional relationships among a range of bibliometric features to improve the accuracy of predicting high-impact research. Random forest classification models were trained using 28 bibliometric features calculated from a dataset of 1,485,958 publications in medicine to retrospectively predict whether a publication would become high-impact. For each random forest model, the balanced accuracy score was above 0.95 and the area under the receiver operating characteristic curve was above 0.99. The high performance of high impact research prediction using our proposed models show that machine learning technologies are promising algorithms that can support funding decision-making for medical research.
... Systems of accreditation (Sanz-Menéndez and Cruz-Castro 2019) and of habilitation (Abramo and D'Angelo 2015) also involve evaluations at the individual level. In addition, evaluations could be applied to a whole institution such as performance-based research funding systems (Hicks 2012;Zacharewicz et al. 2019). ...
... Finally, in the New Zealand PBRF, the individual classification of researchers is carried out as a mechanism for the competitive allocation of budgets to universities. It can be better understood as a performance-based research funding system, closer to the European cases described by Hicks (2012) and others. ...
... The first three characteristics are essential for placing any system in a historical and institutional context and identifying policy transfer processes. The remaining six characteristics are related to the structure and governance of evaluation systems (Whitley and Glä ser 2007;Hicks 2012). The frequency of calls, the number of levels or categories, and size can help explain features related to the structure, while the benefits provided, disciplinary organization, and quotas enable us to analyze governance. ...
Evaluation procedures play a crucial role in science and technology systems, particularly within academic career structures. This article focuses on an approach to evaluation that has gained prominence in Latin America over the past four decades. This scheme assesses the individual performance of academics based on their academic activities and outputs and assigns them a ‘researcher category’, which carries prestige and, in many cases, additional monthly rewards. Initially implemented in higher education contexts with limited research focus, these systems aimed to bolster knowledge production by involving more academics in research. In this study, we define National Researcher Categorization Systems (NRCSs) and distinguish them from other evaluation systems. Subsequently, we present a comparative analysis of NRCSs in seven countries, identifying common trends. Additionally, we discuss categorization systems within the broader context of strategies employed to incentivize academic research, and we explore the potential structural effects that arise when NRCSs assume a central role in a science system. Through our research, we have identified a family of systems in five countries (Mexico, Argentina, Uruguay, Paraguay, and Panama) that share a common history and structure. Furthermore, we emphasize that NRCSs may reinforce a traditional model of the academic researcher, potentially impeding the development of professional profiles aligned with research directed toward social objectives. In summary, our study sheds light on NRCSs, providing insights into their nature, comparative analysis across countries, and implications within the broader academic research landscape.
... I sistemi PBRF per essere definiti tali devono avere le seguenti caratteristiche (Hicks, 2012;Zacharewicz et al., 2019): ...
... Da questo punto di vista, i rischi maggiori che sono stati riscontrati riguardano (Geuna, 2001;Laudel, 2006;Butler, 2010;Sandström et al., 2014;van den Besselaar et al., 2017): i) una selezione "opportunistica" da parte dei ricercatori dei temi su cui avviare progetti condizionata alla possibilità di successo nei finanziamenti di tipo competitivo; ii) una tendenza a perseguire linee di ricerca già consolidate e condivise dalla comunità accademica, piuttosto che quelle più incerte ed eterodosse; iii) la necessità da parte delle istituzioni pubbliche di approntare sistemi di valutazione sempre più complessi, intrusivi e costosi; iv) l'introduzione di comportamenti opportunistici tra ricercatori per ottenere risorse sempre più scarse; v) l'eccesso di competitività; vi) un potenziale calo in aspetti che non rientrano nelle formule, ad esempio quelli inerenti la didattica. Inoltre, c'è il pericolo di innescare fenomeni di cumulatività e auto-rafforzamento, in particolare l'"effetto Matteo" 46 , e conseguenti problemi di equità (Hicks, 2012), soprattutto a detrimento di istituzioni o territori più svantaggiati, e per la difficoltà di chi è escluso dal finanziamento di poter rientrare nella competizione. ...
... Questi effetti, reali o potenziali che siano, sono molto difficili da misurare e da mettere sotto controllo e sono in relazione con il meccanismo di finanziamento in uso. I più comuni (OCSE, 1990;Hicks, 2012;Cruz-Castro et al., 2011;Zacharewicz et al., 2019) sono: a) finanziamento istituzionale non competitivo o ordinario, vale a dire il finanziamento che il governo nazionale o locale assegna alle istituzioni per sostenere le attività, in particolare, le spese per le strutture e il costo del personale, soprattutto nei sistemi di ricerca in cui i ricercatori sono dipendenti pubblici; questo finanziamento viene generalmente attribuito senza passare per la selezione di progetti, lasciando una certa autonomia alle istituzioni sulle decisioni di stanziamento; si fonda generalmente su un criterio storico di allocazione, facendo riferimento ai livelli passati di risorse ottenute, le quali subiscono variazioni in relazione alle variazioni dei valori considerati per il calcolo (ad es. numero di docenti); b) finanziamento sulla base della contrattazione, basato su una serie di modi intermedi associati alla negoziazione tra il governo e le organizzazioni di ricerca, dove nella determinazione dell'ammontare del finanziamento possono avere un ruolo sia aspetti relativi a un'allocazione storica, sia elementi relativi alla performance dell'organizzazione; c) finanziamento istituzionale di tipo premiale, rivolto alle istituzioni, ma allocato sulla base di un sistema di calcolo, denominato formula, la cui composizione si basa su una serie di indicatori di input e di risultato (ad es. ...
The book traces the main stages of third mission and social impact assessment, in the context of the (recent) history of higher education and research assessment in Italy, starting with a review of the transformation of the relationship between the University and the society.
Several social drives are leading the university to a series of transformations, at the national, European and international level. The requirement to use transparency criteria and the associated returns of research spending (in accountability) has become a fundamental issue in informing government, business, citizens and society on the achieved results. The introduction of social impact assessment frameworks is also advancing research agendas towards socially relevant domains, to provide solutions in international competitiveness, social welfare, sustainability and other grand challenges.
A terminological and conceptual shift is occurring from a traditional concept of technology transfer and third mission towards a broader meaning of knowledge exchange and co-creation between universities and extra-academic actors and the impact generated. However, this shift towards a transdisciplinary, trans-epistemic and inclusive evaluation framework poses new challenges to capture this complexity, requiring the adoption of new methods and tools presented in this book. The international literature and debate on practices are vital combined with trial-and-error approaches, community involvement, and targeted pilot studies. Beyond controversies, resistances, and easily contrived enthusiasms, this is what this volume tries to investigate.
... Essas diferentes concepções deram origem a distintas culturas de avaliação (Hicks, 2012;Lepori, Reale e Spinello, 2018;Ochsner, Kulczycki e Gedutis, 2018). As práticas transformaram-se ao longo dos séculos, dando origem a modelos híbridos e bastante heterogêneos. ...
... As agências nacionais de regulação e de financiamento da educação superior e da pesquisa passaram a defender a necessidade de políticas permanentes e padronizadas de avaliação. Por meio da institucionalização desses processos seria possível, de acordo com as agências, avaliar os resultados dos crescentes investimentos públicos, conferindo regularidade, transparência e uniformidade de critérios e procedimentos em âmbito nacional (Hicks, 2012;Ochsner, Kulczycki e Gedutis, 2018). Além de ser uma dimensão interna e exclusiva das universidades e dos pesquisadores, a avaliação foi, gradativamente, se consolidando como uma das principais dimensões dos sistemas nacionais de educação superior, ciência e tecnologia (Amaral, 2009;Verhine e Freitas, 2012). ...
This paper presents the results of a study on the evaluation of research and graduate education in the Netherlands. The policies implemented in the country over the last four decades have consolidated a sui generis and stable evaluation system, contributing to strengthening the country's scientific leadership on the international scene. The Dutch experience has inspired several countries, and its lessons could also be relevant for the Brazilian reality. In this context, this study aimed to analyse the main characteristics of the Dutch system, understand the role of self-evaluation within that system and present a self-evaluation experience conducted by a noteworthy research centre from Leiden University. From the lessons learned, we conclude by presenting some subsidies for improving the evaluation of graduate education conducted in Brazil.
KEYWORDS
evaluation policies; self-evaluation; institutional autonomy; the Netherlands
... Sweden's PREA is at university level, with no formal nationallevel research evaluation, despite a national, indicators-based, performance-based research funding system (Hicks 2012). Without a national structure to evaluate research performance and internationally benchmark the universities, bespoke, organization-level self-evaluation arrangements are conducted by Swedish universities, involving an 'enabling' PREA that draws upon narratives and some indicators. ...
... These four PREAs broadly capture Steering I and II and Enabling I and II from our earlier ideal types (see Table 2). As we have noted, they capture a sufficient variety of PREA characteristics to represent known forms of varied research evaluation arrangements globally (see Hicks 2012;Kolarz et al. 2019;Zacharewicz et al. 2019). Covering these four contexts, representing all ideal types, enables us to assert that if qualityrelated selection effects occur for this field across these four contexts, the same pattern would hold true even if we selected to study other contexts for the same field, given they likely mirror the same range of ideal PREA types we are already covering. ...
This paper contributes to understanding the effects of research governance on global scientific fields. Using a highly selective comparative analysis of four national governance contexts, we explore how governance arrangements influence the dynamics of global research fields. Our study provides insights into second-level governance effects, moving beyond previous studies focusing primarily on effects on research organizations rooted in national contexts. Rather than study over 100 countries across which our selected CERN-based particle physics global research field operates, we explore conditions for changing the dynamics of global research fields and examine mechanisms through which change may occur. We predict then minimal effects on the epistemic choices and research practices of members of the four local knowledge networks despite variations in governance arrangements, and hence no second-level effects. We assert a research field’s independence from governance depends on its characteristics and the relative importance to researchers of research quality notions. This paper contributes methodologically and has practical implications for policymakers. It suggests governance arrangements affect the epistemic choices and research practices of the local knowledge networks only when certain conditions are met. Policymakers should consider the context and characteristics of a field when designing governance arrangements and policy.
... Research quality criteria are the foundation of university research evaluation systems. However, while those systems are intended to evaluate diverse research fields having different research patterns and quality standards, they tend to adopt universalistic criteria which follow the most dominant research conventions (Gulbrandsen 2000;Hicks 2012) but do not reflect the complex nature of research quality and thus exclude relevant aspects of research (Ochsner 2022). There are several efforts to expand the concept of research quality in evaluations (Andersen 2013;Franssen 2022;Hug, Ochsner, and Daniel 2013;Ochsner, Hug, and Daniel 2013), but they focus on traditional research fields (STEM & SSH) and do not include the field of Creative Arts (CA) which has traditionally been located beyond the context of university research. ...
... Specifically, we focus on the quality criteria used in the evaluation of the arts within performance-based research funding systems (PRFS) national systems used to evaluate research outputs and allocate research funding (see, e.g. Hicks 2012). While there is ample research on Art Education and their place and role in Higher Education Institutes (see, e.g. ...
This paper investigates research quality criteria in the Creative Arts (CA). The CA has been introduced into the higher education and research sector over the last three decades. It is thus a relatively new research field and there is little empirical knowledge on how outputs in this field should be evaluated. Our study applies a mixed-method approach to assess the relevance of quality criteria used in performance-based research funding systems (PRFSs) in 10 countries. The results of a qualitative analysis of interviews with artists-academics (N = 67) and Joint Correspondence Analysis show that when art is evaluated in the context of academic research, both the traditional indicators of artistic quality as well as the cognitive and research-related aspects of the arts are believed to be significant. The JCA analysis also showed that the majority of our respondents found both extrinsic quality criteria (related to reputation and prestige) and intrinsic criteria (related to cognition and development) relevant.
... The evaluation of the research performance of the universities and respective funding allocation based on their performance, so-called performance-based research funding systems (PBRFSs), have become widespread worldwide (Dougherty et al., 2016;Hicks, 2012;Pinar & Horne, 2022;Zacharewicz et al., 2019). It has been argued that the PBRFSs emerged based on the new public management reforms to increase public accountability (Hicks, 2012;Leišytė, 2016), and these funding systems aim to allocate resources to more efficient research institutes, provide accountability for public investment and establish reputational yardsticks (REF, 2022;Zacharewicz et al., 2019). ...
... The evaluation of the research performance of the universities and respective funding allocation based on their performance, so-called performance-based research funding systems (PBRFSs), have become widespread worldwide (Dougherty et al., 2016;Hicks, 2012;Pinar & Horne, 2022;Zacharewicz et al., 2019). It has been argued that the PBRFSs emerged based on the new public management reforms to increase public accountability (Hicks, 2012;Leišytė, 2016), and these funding systems aim to allocate resources to more efficient research institutes, provide accountability for public investment and establish reputational yardsticks (REF, 2022;Zacharewicz et al., 2019). ...
Performance-based research funding systems (PBRFSs) have been used in selectively distributing research funding, increasing public money accountability and efficiency. Two recent such evaluations in England were called the Research Excellence Framework (REF), which took place in 2014 and 2021, and the research environment, outputs and impact of the research were evaluated. Even though various aspects of the REF were examined, there has been limited research on how the performance of the universities and disciplines changed between the two evaluation periods. This paper assesses whether there has been convergence or divergence in research quality across universities and subject areas between 2014 and 2021 and found that there was an absolute convergence between universities in all three research elements evaluated, and universities that performed relatively worse in REF in 2014 experienced higher growth in their performance between 2014 and 2021. There was also an absolute convergence in the research environment and impact across different subject areas, but there is no significant convergence in the quality of research outputs across disciplines. Our findings also highlight that there has been an absolute convergence in research quality within the universities (between different disciplines in a given university) and within disciplines (between universities in a given subject).
... With this model, output-based research productivity has been used in the evaluation of scientists at HEIs. This has had profound consequences for both HEI and scientists, and many HEIs have switched to a performance-based research evaluation system [38]. It can be stated that this system is employed by almost all HEIs. ...
... The evaluation of research productivity has become a prominent focus for countries worldwide due to several factors, including international benchmarking systems, rankings, performance-based funding schemes, and the pressures of academic capitalism. Higher education scholars such as Altbach [75], Hicks [38], and Slaughter and Rhoades [76] have shed light on these pertinent issues. In this study, we focused on the effects of scientists' families and childhood periods on research productivity, which are partially neglected in studies on higher education. ...
In the past decades, the awareness about the concept of research productivity at higher education institutions has improved which led to an increase in the number of studies dealing with the subject. Such studies mostly deal with correlations between research productivity and organizational elements, gender, age, professional experience, and alma mater characteristics. To provide an innovative dimension to the existing studies this study focuses on the interaction between the research productivity of the scientists and their childhood period and childhood setting. In this context, the aim of our study is to examine the effects of cultural, economic, and social capitals on research productivity of both scientists' current status and their parents' during their childhood. The data were collected from 9499 faculty members through a survey questionnaire which included items on cultural, economic, and social capital. The data on research productivity of the participants were taken from the Web of Science. The major findings of the study are as follows: (a) Turkish scientists both have lower levels of parents' level of-during childhood-and their current level of cultural capital, and they mostly come from families with the lower-middle economic level; (b) they have medium level social capital; (c) cultural and social capitals together can account for 69% of research productivity, and the order of the related items are found to be childhood objectified cultural capital, current embodied cultural capital and parents' embodied cultural capital during childhood; (d) among social capital structures, relational social capital is the strongest predictor of research productivity and (e) economic capital is not a significant predictor of research productivity. We believe that our current findings contribute to the studies on higher education research by uncovering the new relationships between structures.
... The growth of the 'impact agenda' has taken at least three forms: (1) the introduction of impact as an implicit, and sometimes explicit, selection criterion for research funding (Bozeman and Boardman 2009;Bozeman and Youtie 2017;Chubb and Watermeyer 2017); (2) direct funding support for non-academic engagement and knowledge exchange activity (Ulrichsen 2015;Johnson 2022;Durrant and MacKillop 2022) and; (3) the introduction of impact as an assessment criterion for allocating public funding to a university (Smith, Ward and House 2011;Hicks 2012). The expansion of academic researchers' roles to include planning and delivery of impact affects multiple stages of the research process (Collini 2012;Watermeyer 2016;Power 2018). ...
... Martin (2011) argues that while social and economic impact of research can be assessed after the fact, the methodologies that produce robust results are often time-and labour-intensive and unsuited to operation at the scale that would facilitate the evaluation of an entire national research system. In countries with performance-based research funding systems (Hicks 2012), this introduces a substantial methodological dilemma. ...
Although ex post evaluation of impact is increasingly common, the extent to which research impacts emerge largely as anticipated by researchers, or as the result of serendipitous and unpredictable processes, is not well understood. In this article, we explore whether predictions of impact made at the funding stage align with realized impact, using data from the UK’s Research Excellence Framework (REF). We exploit REF impact cases traced back to research funding applications, as a dataset of 2,194 case–grant pairs, to compare impact topics with funder remits. For 209 of those pairs, we directly compare their descriptions of ex ante and ex post impact. We find that impact claims in these case–grant pairs are often congruent with each other, with 76% showing alignment between anticipated impact at funding stage and the eventual claimed impact in the REF. Co-production of research, often perceived as a model for impactful research, was a feature of just over half of our cases. Our results show that, contrary to other preliminary studies of the REF, impact appears to be broadly predictable, although unpredictability remains important. We suggest that co-production is a reasonably good mechanism for addressing the balance of predictable and unpredictable impact outcomes.
... Professors are expected, among others, to have a larger number of high-quality research works as compared to lower-rank academics. Nowadays most academic stakeholders tend to associate high-quality research with works published in journals indexed in Web of Science (WoS) or Scopus (Hicks, 2012). Indexed publications are those used to make world-university performance rankings. ...
... Other side-effects observed were discouraging research diversification, interdisciplinary and innovative research (Abramo et al., 2018;Hicks, 2012;Rafols et al., 2012;Wilsdon, 2015); and tilting time and energies from teaching to research activities (Enders et al., 2015;De Philippis, 2021). ...
This article aims to explore the effects of Ukrainian policy reform, introducing Scopus and WoS publication requirements for professorship, on the publication behaviour and research performance of professors. Our analysis reveals a better scientific profile, at the time of promotion, of those who obtained professorship after the reform as compared to those who obtained it before. Also, we observe a bandwagon effect since the research performance gap between the two observed cohorts decreased after the introduction of the publication requirements. The statistical difference in differences tests revealed that in general, the incentive to produce more indexed publications worked. Nevertheless, it did not always led to higher research performance. Evidently, in several cases, the increase in research output was obtained at the expense of research impact. The effects of the reform could be far greater if combined with initiatives aimed at assessing Ukrainian professor performance regularly and extending the requirements and assessment to the impact of research.
... Scientific research has traditionally been evaluated primarily based on scientific papers, which constitute science's most visible and measurable output (Geuna & Martin, 2003;Hicks, 2012). Academics and research institutions are evaluated and ranked based on a variety of publishing performance criteria (Hirsch, 2005;Narin & Hamilton, 1996), which involves the allocation of research funds as well as the assignment of academic roles (Geuna & Martin, 2003;Hicks, 2012). ...
... Scientific research has traditionally been evaluated primarily based on scientific papers, which constitute science's most visible and measurable output (Geuna & Martin, 2003;Hicks, 2012). Academics and research institutions are evaluated and ranked based on a variety of publishing performance criteria (Hirsch, 2005;Narin & Hamilton, 1996), which involves the allocation of research funds as well as the assignment of academic roles (Geuna & Martin, 2003;Hicks, 2012). A massive literature has focused on the accuracy of modern management and performance metrics, such as productivity, citation indexes, and peer review (Anninos, 2014;Basu, 2006;Werner, 2015). ...
This study explores how scientometric data and indicators are used to transform science systems in a selection of countries in the Middle East and North Africa region. I propose that scientometric-based rules inform such transformation. First, the research shows how research managers adopt scientometrics as 'global standards'. I also show how several scientometric data and indicators are adopted following a 'glocalization' process. Finally, I demonstrate how research managers use this data to inform decision-making and policymaking processes. This study contributes to a broader understanding of the usage of scientometric indicators in the context of assessing research institutions and researchers based on their publishing activities. Related to these assessments, I also discuss how such data transforms and adapts local science systems to meet so-called 'global standards'.
... A growing list of countries have strengthened competitive mechanisms in the public research sector, mainly through selective funding and merit-based access to public resources. The introduction of performance-based research funding systems (PBRFS) serves various aims, among which are the stimulation of continuous improvement in research effectiveness and efficiency (Hicks 2012;Lewis 2013;Woelert 2015;Geuna and Piolatto 2016). In EU28 (including the UK at the time of investigation), 16 countries have adopted some form of PBRFS (Zacharewicz et al. 2019). ...
The paper takes a scientometric approach to measure the change in research productivity of Italian academics before the COVID-19 pandemic outbreak. We propose a composite output/input bibliometric indicator and apply it at the field level, conducting a longitudinal analysis. Although the number of academics in the national academic system has decreased, we register very strong growth in both the number of publications and their scholarly impact. The growth in productivity, with only rare exceptions, crosses almost all fields. However, in areas that are traditionally very internationalized (Biology, Physics, and Chemistry), growth is less sustained than overall average, and also the variability of productivity across fields seems reduced. The main reason for this detail would be the smaller margins for improvement in the fields that had already reached high international standing. What emerges from the analysis goes counter to some alarms of declining scientific productivity at the global level. The Italian case is partly explained by the historic adoption of policies aimed at strengthening competitive mechanisms, in particular through the introduction of systems of performance-based research funding, and bibliometric accreditation for professorship.
... Indeed, one prevailing model for funding universities is the performance-based research funding system (PRFS), where national governments allocate research funding to universities according to the country's system of research output assessment. Hicks [70] suggests that PRFSs intend to incentive academic research excellence through a research performance-based competition among HEIs that can enhance their prestige and reputation. ...
This study analyzes the impact of national Artificial Intelligence (AI) strategies and policies on higher education institutions. The comparative analysis includes an exhaustive review of the AI regulatory framework of top leading economies, including the U.S., China, Japan, Germany, the U.K., France, South Korea, the U.A.E., Canada, Israel, Netherlands, and Sweden. The research also provides an extensive but summarized literature review of the most relevant AI applications in a long list of professional fields and academic majors. Finally, the manuscript discusses the implications of the extensive adoption of AI and the need for higher education institutions to adapt to the most compelling challenge of the Fourth Industrial Revolution: AI. The article also provides ample evidence of the urgent need for higher education institutions to include AI curriculum content as a critical factor for their long-term survival. The instruction of AI content in almost all academic majors will prepare future alumnae with the abilities and talent they will need to succeed in the job market.
... Academic research helps to produce scientific knowledge, and current research initiatives can foretell the future of any society (Ioana et al., 2015). University funding for research activities improves the innovative environment and adds to the country's economic growth (Hicks, 2012). Some scholars argue that the sheer presence of prominent research universities in the region does not guarantee economic growth in the region, despite being significant assets for such regional or urban economies (Wolfe, 2005). ...
The labour market is witnessing a significant increase in the levels of demand to learn new skills in areas such as data science, artificial intelligence and machine learning, which are among the most prominent features of the new industrial revolution. Universities have a vital function in the knowledge economy and innovation, as they generate and share knowledge through research, education, and creativity. The knowledge economy (KE) and innovation are key drivers of economic growth, as they promote innovation, productivity, and competitiveness. It creates new industries and jobs that require highly skilled workers. Weakness and inadequacy of the educational and research system have been considered as one of the main issues in Saudi Arabia. The aim of this study is to identify the role of universities in the knowledge economy and innovation in Saudi Arabia from the perspective of academics at the University of Hail. To achieve the objective of this research, a cross-sectional explanatory and descriptive research design with a quantitative approach was adopted by the researchers. A sample size of 83 academicians was chosen using a simple random sampling procedure. The analysis method employed was structural equation modelling (SEM) with partial least squares (PLS). The adopted instrument for collecting the data were a survey. The questionnaire measures were based on a 5-point Likert scale. To measure the reliability and validity of the instruments, Cronbach’s alpha, composite reliability, and Fornell-Lacker criterion tests were conducted. The findings of the study revealed that educational curriculum, educational policy, infrastructure, and scientific research have a significant role in (KE) and innovation, whereas the results of teaching methods showed no significant role in (KE) and innovation. To remain competitive in the ever-changing economic landscape, policymakers in Saudi Arabian universities must prioritize curriculum design, infrastructure, teaching methods, policies, and skilled manpower while paying more attention to science, technology, and innovation. In addition, universities should focus on developing the skills of their graduates to meet the demands of the job market. This can be achieved by offering internships and apprenticeships, as well as providing training in soft skills such as communication, teamwork, and problem-solving.
... 2. University employer. This includes funding reallocated from national competitive (e.g., performance-based research funding: Hicks, 2012) or non-competitive block research grants, from teaching income, investments and other sources that are allocated for research in general rather than equipment, time, or specific projects. 3. Other university (e.g., as a visiting researcher on a collaborative project). ...
Evaluating the effects of some or all academic research funding is difficult because of the many different and overlapping sources, types, and scopes. It is therefore important to identify the key aspects of research funding so that funders and others assessing its value do not overlook them. This article outlines 18 dimensions through which funding varies substantially, as well as three funding records facets. For each dimension, a list of common or possible variations is suggested. The main dimensions include the type of funder of time and equipment, any funding sharing, the proportion of costs funded, the nature of the funding, any collaborative contributions, and the amount and duration of the grant. In addition, funding can influence what is researched, how and by whom. The funding can also be recorded in different places and has different levels of connection to outputs. The many variations and the lack of a clear divide between “unfunded” and funded research, because internal funding can be implicit or unrecorded, greatly complicate assessing the value of funding quantitatively at scale. The dimensions listed here should nevertheless help funding evaluators to consider as many differences as possible and list the remainder as limitations. They also serve as suggested information to collect for those compiling funding datasets.
... share of highly cited papers) by amount of financial input would be obtained through these modalities rather than through traditional institutional block funding. Several justifications are advanced to explain this hypothesis (Geuna 2001;OECD 2002;Tapper and Salter 2003;Hicks 2012;Zacharewicz et al. 2019). ...
Over the last decades, most EU countries have profoundly reshaped their public research funding systems by shifting from traditional institutional block-funding towards more project-based mechanisms. The main rationale underlying this evolution builds on the assumption that project funding would foster research performance through the introduction of competitive allocation mechanisms. In contrast with the general increase of project funding, evidence is mixed regarding a positive effect of competitive funding mechanisms on research performance, as some studies find a positive impact, other a negative one or no impact. Differences also appear across studies regarding research actors, funding streams, and research outputs considered. This article integrates these different approaches through a multilevel design gathering funding inputs for 10 countries and 148 universities between 2011 and 2019 and assesses their impact on the quantity and quality of publications. Results highlight no impact of national and university-level competitive funding mechanisms on universities highly cited publications and no clear effect on the quantity of publications.
... A portfolio of quality publications in reputable journals is an asset that scholars are proud to showcase and build their careers on. Although definition of quality varies from one country and institution to another, quantitative quality criteria dominate other aspects of scholarship (Pontika et al., 2022) and impact factor, acceptance rate, and number of citations are frequently used as quality indicators to assess scientific productivity (Hicks, 2012;Ma & Ladisch, 2019). The economics schools in the US may require publications in the top five economics journals for tenure, while many UK business schools require publications in 3-and 4-star journals in the ABS rankings. ...
Purpose
This study takes advantage of newly released journal metrics to investigate whether local journals with more qualified boards have lower acceptance rates, based on data from 219 Turkish national journals and 2,367 editorial board members.
Design/methodology/approach
This study argues that journal editors can signal their scholarly quality by publishing in reputable journals. Conversely, editors publishing inside articles in affiliated national journals would send negative signals. The research predicts that high (low) quality editorial boards will conduct more (less) selective evaluation and their journals will have lower (higher) acceptance rates. Based on the publication strategy of editors, four measures of board quality are defined: Number of board inside publications per editor (INSIDER), number of board Social Sciences Citation Index publications per editor (SSCI), inside-to-SSCI article ratio (ISRA), and board citation per editor (CITATION). Predictions are tested by correlation and regression analysis.
Findings
Low-quality board proxies (INSIDER, ISRA) are positively, and high-quality board proxies (SSCI, CITATION) are negatively associated with acceptance rates. Further, we find that receiving a larger number of submissions, greater women representation on boards, and Web of Science and Scopus (WOSS) coverage are associated with lower acceptance rates. Acceptance rates for journals range from 12% to 91%, with an average of 54% and a median of 53%. Law journals have significantly higher average acceptance rate (68%) than other journals, while WOSS journals have the lowest (43%). Findings indicate some of the highest acceptance rates in Social Sciences literature, including competitive Business and Economics journals that traditionally have low acceptance rates.
Limitations
Research relies on local context to define publication strategy of editors. Findings may not be generalizable to mainstream journals and core science countries where emphasis on research quality is stronger and editorial selection is based on scientific merit.
Practical implications
Results offer useful insights into editorial management of national journals and allow us to make sense of local editorial practices. The importance of scientific merit for selection to national journal editorial boards is particularly highlighted for sound editorial evaluation of submitted manuscripts.
Originality/value
This is the first attempt to document a significant relation between acceptance rates and editorial board publication behavior.
... Research citation and academic evaluation are symbiotic (Hicks, 2012). It has become commonplace to compare research performance among institutions worldwide and against industry benchmarks to rank countries and institutions (Marginson, 2013). ...
Retracted articles by Indian scholars have received significant attention in recent times. However, no comprehensive study has been conducted to analyse the citations of retracted papers authored by Indian researchers. This study aimed to assess the citations to retracted works published between 2001 and 2020 pre‐ and post‐retraction. The study found that there was an increase in retractions over time, with empirical data suggesting that the number of retractions has increased significantly, from 72 papers between 2001 and 2010 to 365 papers between 2011 and 2020. Duplication ( n = 128) and plagiarism ( n = 119) were the primary reasons for retraction. Notably, 90% of the retracted articles continued to receive citations after retraction. Among the retracted papers, eight received more than 50 post‐retraction citations, 39 received 20 to 50 citations, 347 received one to 19 citations, and 43 were not cited at all post‐retraction. There was an overall 8% decrease in citations after retraction. Retractions were observed across journals of varying impact factor, with a higher number of retractions observed in journals with an impact factor of less than 5 ( n = 286; 65%). Furthermore, smaller research teams of two to five authors accounted for 72% of the total retractions.
... During the past decade, many nations around the world such as Australia, the Czech Republic, Finland, Norway, Poland, Turkey, the UK, and many others have chosen to implement performance-based research funding (Aagaard, 2015;Hicks, 2012;Kulczycki, 2017;Tonta, 2017) and incentive schemes (Franzoni et al., 2011;Quan et al., 2017). Such incentives are mostly related to the publication activities of researchers (Rochmyaningsih, 2019), which are traditionally analyzed by using multidisciplinary bibliographic data sources like the Web of Science, Scopus, Google Scholar, Dimensions and Crossref. ...
The Arabic Citation Index (ARCI) was launched in 2020. This article provides an overview of the scientific literature contained in this new database and explores its possible usage in research evaluation. As of May 2022, ARCI had indexed 138,283 scientific publications published between 2015 and 2020. ARCI’s coverage is characterised by using the metadata available in scientific publications. First, I investigate the distributions of the indexed literature at various levels (research domains, countries, languages, open access). Articles make up nearly all the documents indexed with a share of 99% of ARCI. The Arts & Humanities and Social Sciences fields have the highest concentration of publications. Most indexed journals are published in Egypt, Algeria, Iraq, Jordan, and Saudi Arabia. About 8% of publications in ARCI are published in languages other than Arabic. Second, I use an unsupervised machine learning model, LDA (Latent Dirichlet Allocation), and the text mining algorithm of VOSviewer to uncover the main topics in ARCI. These methods provide a better understanding of ARCI’s thematic structure. Next, I discuss how ARCI can complement global standards in the context of a more inclusive research evaluation. Finally, I suggest a few research opportunities after discussing the findings of this study.
Peer Review
https://www.webofscience.com/api/gateway/wos/peer-review/10.1162/qss_a_00261
... Researchers are primarily evaluated by their publication success, not only based on the number of publications and citations but also considering the rating of the respective scientific outlets or other forms of metrics related to the individual publishing performance (Narin and Hamilton 1996;Hirsch 2005). In some cases, the success in acquiring third-party funding may also play a role, but publication performance is the primary scientific quality indicator when applying for an academic position or for third-party funding (Geuna and Martin 2003;Hicks 2012). Where reputation in the scientific community is concerned, the importance of publication performance cannot be overstressed-as implied by the widely used term 'publish or perish' (Laband and Tollison 2003). ...
Scientific advisory boards are frequently established to provide scientific insights and advice to policymakers. Advisory board appointing bodies often state that research excellence and scientific seniority are the main grounds on which advisory board members are selected. Many authors have pointed out that there is more to giving good scientific advice than just being an expert for a specific research field. The aim of this study is to analyse if and how research excellence correlates with the probability of being appointed as a scientific advisory board member. We collected data for scientific advisory boards from both the USA and Germany. We use logit regression models to analyse how research excellence correlates with the probability of appointment to a scientific advisory board. Our results suggest that research excellence is insignificant or even correlates negatively with the probability of being appointed to a scientific advisory board.
... Governmental attempts to make universities and colleges more effective and efficient have been a recurrent effort in various reform initiatives during the last decades throughout the world (Christensen 2011;Enders, de Boer, and Weyer 2013). Although reforms tend to be country specific, there are also a number of commonalities among them including the ambitions of providing institutions with more autonomy, streamlining institutional governance, developing new incentive structures for universities, introducing accountability schemes and performance targets, and aligning organisational structures to strategic aims (Capano 2011;Frølich, Christensen, and Stensaker 2019;Hicks 2012;Thomas et al. 2020). ...
Many recent higher education reforms worldwide have been legitimated by their potential impact on the performance of universities and colleges. However, we know less about the actual impact of the changes implemented. This article examines the extent to which research performance can be associated with specific organizational characteristics at the department level. The analysis is based on Norwegian university departments, where high- and low-performing departments have been selected as cases for further investigations. The policy context is the organizational reform in Norway from 2016 onwards aiming at reorganizing the higher education landscape through institutional mergers. The key findings indicate that there are few distinct departmental characteristics associated with research performance, such as elected or appointed leadership, single or multi-campus organization, or departmental size. However, the study reveals that highly productive individuals do matter and suggests that cultural dimensions and working conditions may be interesting factors to pursue in further research.
... Research financing systems which are results-based do not usually differ entiate their assessment approach with regards to disciplines or research fields (Hicks, 2012) though there are significant differences between discip lines, and there is also the so-called non-academic impact (Bastow et al., 2014). Public agencies financing research and research organisations bear a great responsibility for a more comprehensive impact of the studies which they support financially. ...
This book provides analysis of current trends in research evaluation worldwide and compares the research assessment and innovation ecosystems in Austria, Bulgaria, the Czech Republic, Hungary, Lithuania, the Netherlands, Poland and Slovenia. It argues that in each country the research assessment system is interdependent with the national innovation system and the overall institutional governance/enforcement. The lead author, Albena Vutsova, has over 20 years of experience in research assessment both in Bulgaria and at European institutions (incl. JRC) and has been a member of the European Network for Research Evaluation in the Social Sciences and the Humanities. The three authors work at Sofia University, Bulgaria.
... Digital technologies have not only disrupted the centuries-long publishers-academia relationship but also significantly affected the process and practice of measurement of academic achievements. Following technological transformations, many countries have introduced performance-based research funding systems (Hicks, 2012;Kerr, 1975;Moed et al., 1995) or research assessment exercises (Elton, 2000). It is believed that such systems have fostered a "publish or perish" culture across the globe and increased the pressure on researchers to produce research outputs (Cronin and La Barre, 2004;De Rond, 2005;Ecklund et al., 2012;Seeber et al., 2019). ...
Books are an important output in many fields of research. However, they pose a significant challenge for research assessment systems, partly because of the limited availability of information to support the assessment of books. To inform book assessment practices, I present a systematic examination of the ISBN Manual and the Global Register of Publishers (GRP). I evaluate the extent to which these two sources can be used to determine the genre and publisher of a book as well as the country in which a book was issued. My analysis focuses on books submitted to the research assessment systems in Lithuania and the UK from 2008 to 2020. I show how the ISBN Manual captures the complex interactions between publishers, their imprints, and other organisations active in academic publishing, revealing the pitfalls of measuring books' quality by their publisher's status. The results also indicate that the ISBN standard provides no basis for the book genres mandated by research assessment systems in some countries. Finally, I demonstrate how the ISBN Manual and metadata accumulated in the GRP are convenient tools for designers of research assessment systems and are suitable for identifying ISBN registrants and performing bibliometric analysis.
... This can create tension between 'pure' and 'applied' science and mission or curiosity driven research. The problem is exacerbated by funders needing to demonstrate 'value for money' in short timeframes in recent years funders have required that funded science research should demonstrate 'impact' [18,19]. This is predominantly measured in terms of citations, financial outcomes such as spin out companies and new products that contribute to national gross domestic product, or the creation of a specific piece of policy [20][21][22] an increased focus on minimizing risk reduces opportunities for 'blue skies' exploratory research and innovation approaches [23] (Continued.) ...
The ‘early modern’ (Renaissance) workshop was predicated on the idea that informal, open-ended cooperation enables participants to experience difference and develop new insights, which can lead to new ways of thinking and doing. This paper presents the insights that emerged from a conversation event that brought wide-ranging voices together from different domains in science, and across the arts and industry, to consider science leadership as we look to the future in a time of interlocking crises. The core theme identified was a need to regain creativity in science; in the methods of scientific endeavours, in the way science is produced and communicated, and in how science is experienced in society. Three key challenges for re-establishing a culture of creativity in science emerged: (i) how scientists communicate what science is and what it is for, (ii) what scientists value, and (iii) how scientists create and co-create science with and for society. Furthermore, the value of open-ended and ongoing conversation between different perspectives as a means of achieving this culture was identified and demonstrated.
... Research management includes processes related to scientific (knowledge production) and social (introduction of scientific knowledge into practice) dynamics (Hicks, 2012;Saghafian, Austin, & Traub, 2015;Uwizeye, et.al., 2022). Thus, the boundaries between administrative and scientific fields are blurred. ...
Implementing Vocational Higher Education in The Army Polytechnic (Poltekad) has used the Triple Helix approach: universities, the private sector, and the government. In its development, Poltekad contributes to the direction of the development of the Army’s technological innovation field. Based on observations of the results of Poltekad research, it has yet to be utilized optimally in fulfilling Army equipment and weapons. In the last three years, 2020–2022, 163 final research assignments were recorded by Poltekad students, and 40 research by Poltekad lecturers. However, only eight lecturers’ research has become research models with the potential for defense and development in the Army. This study uses a qualitative research method, specifically a case study. The study results show that Poltekad has not been able to play a role as a research university, but is still limited to its function as a vocational education institution in the technology field. The allocation of research funds for Poltekad is still relatively small compared to the benefits of research to support the development of Army Armaments. The research themes of lecturers and students are not entirely following the needs of the development of Indonesian Army weaponry technology. The absence of a Poltekad external monitoring and evaluation team for research programs, limited educational laboratory facilities, and qualified lecturers as researchers in the Poltekad environment are several reasons for optimizing the utilization of Poltekad research results which cannot be carried out downstream in the implementation of production activities carried out by the Defense Industry in Indonesia as is stated in the Poltekad vision. Poltekad, as a Research University, needs to pay attention to elements, such as 1) budgeting, 2) Research Programs, 3) monitoring, and evaluation, 4) researcher competence, 5) facilitation facilities, 6) information and communication systems, and 7) scientific publications. Downstreaming the themes to utilize research results is carried out according to needs and developments.
... It characterizes the university's ability to generate knowledge at the cutting edge of science and to produce high-quality outputs. Research productivity has been linked to research funding opportunities (Hicks, 2012) and career progression in academia (Carr, et al., 2021) across the world. It is an important factor in determining "individual research performance and academic rank" (Abramo, et al., 2011, p. 915). ...
The purpose of this research study was to examine challenges experienced by academics at Stellenbosch University that hinder their research productivity during the COVID-19 pandemic, involving 248 academics who completed an online questionnaire. A qualitative analysis of open-ended responses revealed five themes that characterized the extent that the COVID-19 pandemic impacted these academics’ research productivity: Online Teaching, Increase in Research Productivity, No Difference to Research Productivity, Reduced Research Productivity, and No Research Productivity. A mixed methods analysis revealed that only 25% of academics were not adversely affected by online teaching in terms of research productivity.
Two thirds of the academics experienced either a reduction in productivity or reported no research productivity at all. Compared to academics who reported an increase in productivity, academics who reported undertaking no research productivity at all tended to be women, not to hold a professor position, not to have a doctorate degree, to have less experience as academics, to have access at home to a tablet, but not to have access at home to cellphone data.
... Later, Smith et al. [39] started a new vector of performance-based systems analysis with an exploration of the Research Excellence Framework aimed at changing the education funding system. Then Hicks [40] explained the impact of prestige on education investment. Following by Agyemang and Broadbent [41], who proved the negative impact of internal university processes on AFUAR reform's effectiveness. ...
... Both can be understood as enactments of a different accountability relationship between PROs and national governments, which are part of the more general trend of the rise of New Public Management in the public sector ( de Boer, Enders and Schimank 2007;Weingart and Maasen 2007;Bleiklie et al. 2011;Musselin 2021). While the nature and use of national research evaluation systems and performance-based funding systems varies widely between OECD countries (Whitley and Gläser 2007;Thomas et al. 2020), governments have generally moved away from providing unconditional core funding to PROs, and instead developed allocation models tied to performance and policy goals (Hicks 2012). At the same time, research funding has increasingly been distributed through competition for grants arranged by national and international funding bodies, decreasing the relative share of block funding (Whitley et al., 2018). ...
Public research organizations respond to external pressures from national research evaluation systems, performance-based funding systems and university rankings by translating them into internal goals, rules and regulations and by developing organizational identities, profiles and missions. Organizational responses have primarily been studied at the central organizational level, and research on the steering of research has primarily focused on the impacts of performance-based funding systems. However, research evaluation exercises may also have a formative impact, especially below the central organizational level. This paper uses a case study of a research unit of a biomedical research school in the Netherlands to explore the organizational response to a relatively critical external assessment report. It shows that the participation in the Dutch research evaluation cycle legitimated the formation of a new organizational identity for the research unit, which functions as a frame that suggests to staff members a new interpretation of the type of research that is at the core of what the research unit does. We identify three additional steering mechanisms that support the enactment of the organizational identity: steering by resource allocation, by suggesting and by re-organizing. We, furthermore, explore the epistemic effects – the direction and conduct of research – of the organizational response, through interview data in combination with a bibliometric analysis.
As interest increases in the educational quality of higher education institutions (HEIs), less is known about how the mergers and acquisitions (M&As) of these HEIs affect educational quality. In this study, we investigate the association of M&As on the educational quality of HEIs and the role of institutional incentives in this relation. We argue that M&As in HEIs can positively affect the educational quality of these institutions, although this relationship is substantially sensitive and moderated by institutional incentives. Ultimately, M&As can negatively influence educational quality if the incentives do not encourage HEIs to improve and maintain high standards in educational quality continuously. We empirically examine our theoretical framework using a longitudinal dataset on a fixed-effects model from 2007 to 2015 with M&As of Brazilian HEIs, and the results consistently support our arguments.
Whilst the extant literature on the publish-or-perish culture in the West is plentiful, there remains surprisingly little scholarship exploring the ways managerialist policies have become integral to local identities, work and life in Chinese universities. We address this gap by taking China's endeavour to become first in the global higher education field as an entry point and reflecting on our lived experiences as early career researchers (ECRs) in the form of a duoethnography. Our dialogues consider how, and with what effects, Chinese higher education privileges the notion of research excellence and works to construct professional identities. As grassroots ECRs epitomising a force to negotiate, challenge and resist the contemporary research order from below, we identify academic publishing as 'the best strategy' to get ahead in the academic game. Despite our compliance with the regime of new manage-rialism, our narratives also suggest that we are attempting to resist the drive for research productivity through envisaging a slower tempo in writing and aspiring for greater reflection. We provide readers with a range of individual-and collective-based strategies for being and progressing as grassroots ECRs, and implications for universities are also discussed at the end of the article. ARTICLE HISTORY
Modern üniversitelerin temel işlevi bilimsel araştırmaya, eğitim ve öğretime öncelik vermektir. Günümüzde üniversitelerin bu işlevleri yanında araştırmayı daha yüksek düzeyde yerine getirmeyi üstlenen araştırma üniversiteleri kavramı ön plana çıkmıştır. Bu araştırmanın amacı, Yükseköğretim Kurulu tarafından geliştirilen Araştırma Üniversitesi Projesinin, mevcut araştırma üniversitelerinin akademik performansı üzerindeki etkilerini incelemektir. Dolayısıyla araştırmada üniversitelerin Araştırma Üniversitesi Projesinden önceki akademik performansı ile sonraki akademik performansının farklı olup olmadığı analiz edilmektedir. Araştırmanın yöntemi olarak nicel analiz yöntemlerinden olan bağımlı gruplar için t testi kullanılmış ve araştırmaya ilişkin bulgulara sağlıklı ulaşmak için araştırmada tekrarlı ölçümler deseni tercih edilmiştir. Üniversitelerin; yayın sayılarına Web of Science veri tabanından, akademik performansa ilişkin verilerine URAP dünya sıralamasından, öğrenci sayılarına ise Yükseköğretim Kurulu tarafından paylaşılan bilgilerden elde edilmiştir. Araştırma, 2017 yılında araştırma üniversitesi olarak belirlenen; Ankara Üniversitesi, Boğaziçi Üniversitesi, Erciyes Üniversitesi, Gebze Teknik Üniversitesi, Hacettepe Üniversitesi, İstanbul Teknik Üniversitesi, İzmir Yüksek Teknoloji Enstitüsü ve Orta Doğu Teknik Üniversitesini kapsamaktadır. Araştırma üniversiteleri, ilk defa 26.09.2017 tarihinde ilan edilmiş olup, bu tarihte araştırma üniversiteleri için ilan edilen URAP dünya sıralaması puanları ile 2022 yılında ilan edilen URAP dünya sıralaması puanları karşılaştırılmıştır. Böylelikle araştırma üniversiteleri projesinden önceki dönem ile sonraki dönem performansları değerlendirilmiştir. Araştırmanın sonucunda ise, araştırma üniversiteleri projesinin araştırma üniversitelerinin akademik performansı üzerindeki olumlu etkisinin sınırlı olduğu bulgusuna ulaşılmıştır.
Greek research and innovation (R&I) performance has significantly improved over the last decades, moving up from the group of catching-up countries of the European Innovation Scoreboard (EIS) to the group of moderate innovators. This is based on a series of factors amongst which has been a coordinated effort to invest in the upgrade of the R&I ecosystem, research infrastructure, scientific equipment, and human capacities. During the multi-annual financial period 2014-2020, the Greek authorities, with the support of the European Structural and Investment Funds (ESIF), invested over a 100 million EUR in launching and developing 28 national research infrastructures (NRIs). In 2022, the Policy Support Facility (PSF) country review provided a first assessment of this policy and of the level of maturity of. the 28 NRIs. In November 2022, the Greek Deputy Ministry for Development requested a second phase of PSF support to assist the General Secretariat for Research and Innovation (GSRI) to prepare for the next round of support for the NRIs.
This report addresses three of the recommendations made by the PSF Country Report:
• Development of a framework for performance-based funding for NRIs.
• Development of a set of key performance indicators (KPIs) that reflect the role of the NRIs in the national R&I system
• Guidelines for open science (OS) and digitalisation to help NRIs further develop their data policies and connect with national and European research data platforms.
The report draws on a review of the literature and a series of interviews and consultations with Greek stakeholders and NRI coordinators that took place during two missions to Athens in spring 2023 by the expert panel.
In this study, we focus on the central question of how Turkey managed the expansion of its higher education system in terms of public policies and public interest over the past 20 years. We analyzed public policy documents, academic papers, and written records such as news websites and newspaper columns to evaluate how the recent expansion in Turkish higher education evolved and impacted the quality of Turkish universities. Our findings indicated that during the third expansion era starting in the early 2000s, the Turkish higher education system was successful in improving access to higher education immensely. The government’s openness to expansion and the pressure from the members of Parliament for different areas of Turkey were effective in this process. On the other hand, after the expansion period, the quality of instruction, student services, and research were impacted negatively in many newly founded universities. In conclusion, with a citizen-centered public policy making perspective, the relevant literature and the written records from different sources show that the expansion was a correct policy decision; however, all stakeholders of the higher education system had question marks about how this process was managed.KeywordsHigher education Public policy Policy analysis Expansion Universities Policy decision Stakeholders Citizen-centered Government Education system
The conditions of mainstream research funding constrain risky, novel research. However, alternative grants are emerging. We study grantees of a double-blinded funding scheme targeting risky, novel research: The Villum Experiment (VE). Without prompting, scientists juxtaposed the experience of performing research under these conditions with that of performing research funded by mainstream grants: fun and less fun. The conditions of VE felt less intrusive and appealed to their self-perceptions and idealized views of scientific work, which shaped how they conducted the funded research. This paper makes three contributions: (1) it reaffirms that how researchers experience grant conditions affects whether a scheme affords what it intends, (2) it highlights that the affordances of research funding are relative to other concurrent funding options, and (3) it shows that small, more broadly allocatable grants can afford scientists a protected space for autonomous research, usually associated with elusive tenure positions or European Research Council (ERC) grants.
What is the point of publishing in the humanities? This Element provides an answer to this question. It builds on a unique set of quantitative and qualitative data to understand why humanities scholars publish. It looks at both basic characteristics such as publication numbers, formats, and perceptions, and differences of national academic settings alongside the influences of the UK's Research Excellence Framework and the German Exzellenzinitiative. The data involve a survey of more than 1,000 humanities scholars and social scientists in the UK and Germany, allowing for a comprehensive comparative study, and a series of qualitative interviews. The resulting critique provides scholars and policy makers with an accessible and critical work about the particularities of authorship and publishing in the humanities. And it gives an account of the problems and struggles of humanities scholars in their pursuit of contributing to discourse, and to be recognised with their intellectual work.
Policy ideas are seldom original; they are often borrowed from other jurisdictions and sometimes adapted to the local context. This chapter reviews research on policy diffusion within the U.S. higher education context, with a focus on the diffusion of higher education policies across states. In addition to reviewing existing literature on U.S. higher education policy diffusion, we present findings from a study that examines newspaper language related to state-level performance-based funding policies for higher education. We focus on two states that considered performance-based funding but were among the few states that did not implement this higher education funding approach. By examining these two states, we are able to explore reasons for non-implementation and shed light on the phenomenon of resistance to policy diffusion.
The aim of this article is to clarify the nature of the management style most suited to the emergence of networked governance. The paradigms of traditional public administration and new public management sit uncomfortably with networked governance. In contrast, it is argued the public value management paradigm bases its practice in the systems of dialogue and exchange that characterize networked governance. Ultimately, the strength of public value management is seen to rest on its ability to point to a motivational force that does not solely rely on rules or incentives to drive public service practice and reform. People are, it suggests, motivated by their involvement in networks and partnerships, that is, their relationships with others formed in the context of mutual respect and shared learning. Building successful relationships is the key to networked governance and the core objective of the management needed to support it.
We use the principal-agent model as a focal theoretical frame for synthesizing what we know, both theoretically and empirically, about the design and dynamics of the implementation of performance management systems in the public sector. In this context, we review the growing body of evidence about how performance measurement and incentive systems function in practice and how individuals and organizations respond and adapt to them over time, drawing primarily on examples from performance measurement systems in public education and social welfare programs. We also describe a dynamic framework for performance measurement systems that takes into account strategic behavior of individuals over time, learning about production functions and individual responses, accountability pressures, and the use of information about the relationship of measured performance to value added. Implications are discussed and recommendations derived for improving public sector performance measurement systems. © 2010 by the Association for Public Policy Analysis and Management.
Conventional liberal frameworks – in which power is seen asthe property of states, and repressive in character, and market and stateexclude each other – are unable to comprehend the recent changes inliberal government, including the government of systems and institutions inhigher education. Neo-liberal government rests on self-managing institutionsand individuals, in which free agents are empowered to act on their ownbehalf but are steered from a distance by policy norms and rules of thegame. In the universities government-created markets and quasi-markets havebeen used to advance both devolution and central control, simultaneously,and national government and institutional management are increasinglyimplicated in each other. These issues are explored in relation to recenthigher education literature, and empirically, the latter by examining thechanges in the Australian higher education system in the last decade. TheAustralian system provides an example of a quasi-market in which thedevelopment of a stronger institutional management, the introduction ofgovernment-institution negotiations over educational profiles, and the newsystems of competitive bidding, performance management and qualityassessment have all been used to steer academic work and to install aprocess of continuous self-transformation along modern neo-liberal lines.Following a change of government in 1996 there has been some movement from aquasi-market to a more fully developed economic market, but no relaxation ofgovernment control.
Many countries have introducedevaluations of university research, reflectingglobal demands for greater accountability. Thispaper compares methods of evaluation usedacross twelve countries in Europe and theAsia-Pacific region. On the basis of thiscomparison, and focusing in particular onBritain, we examine the advantages anddisadvantages of performance-based funding incomparison with other approaches to funding.Our analysis suggests that, while initialbenefits may outweigh the costs, over time sucha system seems to produce diminishing returns.This raises important questions about itscontinued use.
This paper describes the development of research evaluation in Spain. It assumes that research evaluation, R&D policy and programme evaluation are embedded in the development of and R&D system and are characterised by general Spanish policy-making. Research evaluation in a context of delegation and as a self-organising system for research actors guaranteed by the state, has been strongly developed in the last few years: R&D policy and programmme evaluation in less institutionalised. The explanation is linked to the sequence of reforms of the R&D system and to the set up of the first Spanish science and technology policy. The support of the European Commission is acknowledged (HCM contract CHRX-CT93-0240) and the Spanish National R&D Plan (projects SEC 93-0688 and SEC 94-0796).
The establishment of national systems of retrospective research evaluations is one of the most significant of recent changes in the governance of science. In many countries, state attempts to manage public science systems and improve their quality have triggered the institutionalisation of such systems, which vary greatly in their methods of assessing research performance, and consequences for universities. The contributions to this volume discuss, inter alia, the birth and development of research evaluation systems as well as the reasons for their absence in the United States, the responses by universities and academics to these new governance regimes, and their consequences for the production of scientific knowledge. By integrating new theoretical approaches with country studies and studies of general phenomena such as university rankings and bibliometric evaluations, the book shows how these novel state steering mechanisms are changing the organisation of scientific knowledge production and universities in different countries.
In combining latest research and an overview of trends in the changing governance of research, the book is essential not only for scholars engaged in higher education research, science policy studies, and the sociology of science but also for policy makers and analysts from science policy and higher education policy as well as university managers and senior scientists.
This paper reports the results of empiricalresearch designed to explore the impact of researchselectivity on the work and employment of academiceconomists in U.K. universities.Research selectivityis seen as part of the general trend towardmanagerialism in higher education in both the U.K. andabroad. Managerialism based on performance indicatorsand hierarchical control has been contrasted withcollegiate control-based or informal peer review. However,analysis of the academic labor process has idealizedcollegiate relations at the expense of professionalhierarchies and intellectual authority relations. We argue that in the U.K., there has evolved amainstream economics which is located within awell-defined neoclassical core. We find that theexistence of lists of core mainstream journals which arebelieved to count most in the periodic ranking exerciseposes a serious threat to academic freedom and diversitywithin the profession, institutionalizing the controlwhich representatives of the mainstream exercise over both the academic labor process and jobmarket. In this way, managerialism combines with peerreview to outflank resistance to new forms ofcontrolling academic labor at the same time asreinforcing disciplinary boundaries through centralizedsystems of bureaucratic standardization andcontrol.
Technical and political difficulties involved in allocating block infrastructure research grants to institutions in large, post-binary higher education systems are identified through consideration of the operation of the British Research Assessment Exercise (RAE) and the Australian Composite Index used for allocation of the Research Quantum. These two mechanisms use distinctively different assessment approaches but each provides the basis for annual allocations of substantial public funds. Both have generated considerable controversy, involved their institutions and their staffs in considerable additional work and effort, and led to various unintended consequences, especially affecting institutional and researcher behaviour. While the RAE and the Research Quantum have limited objectives, comparison with other national systems of research assessment raises the possibility of the design of assessment systems that could perform a variety of purposes, including allocation of block research grants, quality assurance and public accountability, informing national research policy, and international benchmarking.
Given that the current Research Assessment Exercise (RAE 2001) has been completed, it is an appropriate time to explore the impact of the RAEs upon the character of British higher education. This timeliness is reinforced by the earlier publication of HEFCE's own ‘Review of Research’ (September 2000), the report from the House of Commons’ Select Committee on Science and Technology Committee (April 2000), with a report due in April 2003 from the Joint Funding Bodies (under the auspices of Gareth Roberts). We are therefore in a period of review and consultation, which may culminate in a new assessment regime or, as its severest critics would hope, even its demise. While our analysis genuflects to these contemporary developments, it is constructed within a framework that interprets the RAE process as constituting a continuous struggle for the control of the production of high-status knowledge.
A system of research assessment was developed and implemented inHong Kong during the period from 1991 to 1994 as an input to theassessment of the public recurrent funding allocations of theterritory's higher education institutions and as an extension ofthe University Grants Committee's other quality assuranceactivities. Refinements were subsequently introduced for the nexttwo assessment exercises in 1996 and 1999. This paper describesthe evolution of the process, identifies some significantdifferences from that in the UK on which it was modelled, andevaluates the 1999 research assessment exercise, in particularthe application in that context of the Carnegie Foundation'sdefinitions of research and research-related scholarlyactivities, viz. the scholarships of ``discovery, integration,application and teaching''.
Year-on-year trends in research outputs show increases in research activity as the date of the research assessment exercise—in
New Zealand the Performance-Based Research Fund (PBRF)—looms. Moreover, changes with time in the number and types of conference
presentation indicate that the vehicle of publication is also being influenced by the PBRF. Within New Zealand business schools,
relating the published journal articles to the Australian Business Deans Council rankings list shows a trend towards more
publications of lower rank, raising doubts about whether the rhetoric about the PBRF raising the quality of research is really
justified. This ‘drive’ towards increasing numbers of research outputs is also fostered by an increasing trend towards co-authorship
in publishing across all disciplines.
KeywordsResearch outputs-Research publications-Research quality-Research collaboration-Author collaboration-Performance-based research fund (PBRF)
Many recent discussions of state science and technology policies in OECD economies since the end of the Second World War have
distinguished a number of distant phases or ‘paradigms’ (Ruivo 1994) in state-science relations, which reflected both different
perceptions of the role of scientific research in industrialised societies and changes in the size and complexity of the public
science systems (see e.g. Brooks 1990; Freeman and Soete 1997: 374-395; Martin 2003).
The article draws on two research projects to explore the implications of policy change in the UK for academic identities within a predominantly communitarian theoretical perspective. It focuses on biological scientists and science policies. It examines the impacts of changes upon the dynamic between individuals, disciplines and universities within which academic identities are formed and sustained and upon individual and collective values central to academic identity, namely the primacy of the discipline in academic working lives and academic autonomy. Challenges to these have been strong but they have retained much of their normative power, even if the meaning of academic autonomy has changed. Communitarian theories of academic identity may need to be modified in the contemporary environment but they do not need to be abandoned.
Since 1980, national university departmental ranking exercises have developed in several countries. This paper reviews exercises
in the U.S., U.K. and Australia to assess the state-of-the-art and to identify common themes and trends. The findings are
that the exercises are becoming more elaborate, even unwieldy, and that there is some retreat from complexity. There seems
to be a movement towards bibliometric measures. The exercises also seem to be effective in enhancing university focus on research
strategy.
Australia’s share of publications in the Science Citation Index (SCI) has increased by 25% in the last decade. The worrying aspect associated with this trend is the significant decline in citation impact Australia is achieving relative to other countries. It has dropped from sixth position in a ranking of 11 OECD countries in 1988, to 10th position by 1993, and the distance from ninth place continues to widen.The increased publication activity came at a time when publication output was expected to decline due to pressures facing the higher education sector, which accounts for over two-thirds of Australian publications. This paper examines possible methodological and contextual explanations of the trends in Australia’s presence in the SCI, and undertakes a detailed comparison of two universities that introduced diverse research management strategies in the late 1980s. The conclusion reached is that the driving force behind the Australian trends appears to lie with the increased culture of evaluation faced by the sector. Significant funds are distributed to universities, and within universities, on the basis of aggregate publication counts, with little attention paid to the impact or quality of that output. In consequence, journal publication productivity has increased significantly in the last decade, but its impact has declined.
Nobel Prizes are an important indicator of research excel- lence for a country. Spain has not had a science Nobel Prize winner since 1906, although its gross domestic product (GDP) is high, research and development (R&D) investments, in monetary terms, are high, and conven- tional bibliometric parameters are fairly good. Spanish research produces many sound papers that are reason- ably cited but does not produce top-cited publications. This absence of top-cited publications suggests that important achievements are scarce and, consequently, explains the absence of Nobel Prize awards. I argue that this negative research trend in Spain is caused by the extensive use of formal research evaluations based on the number of publications, impact factors, and journal rankings. These formal evaluations were introduced to establish a national salary bonus that mitigated the lack of research incentives in universities. When the process was started, the results were excellent but, currently, it has been kept too long and should be replaced by methods to determine the actual interest of the research. However, this replacement requires greater involvement of universities in stimulating research.
A bstract
We analyze Italy's recent research evaluation exercise (VTR) as a salient example in discussing some internationally relevant issues emerging from the evaluation of research in economics. We claim that evaluation and its criteria, together with its linkage to research institutions' financing, are likely to affect the direction of research in a problematic way. As the Italian case documents, it is specifically economists who adopt unorthodox paradigms or pursue less diffused topics of research that should be concerned about research evaluation and its criteria. After outlining the recent practice of economic research in Italy and highlighting the relevant scope for pluralism that traditionally characterizes it, we analyze the publications submitted for evaluation to the VTR. By comparing these publications to all the entries in the EconLit database authored by economists located in Italy, we find a risk that the adopted ranking criteria may lead to disregarding historical methods in favor of quantitative and econometric methods, and heterodox schools in favor of mainstream approaches. Finally, by summarizing the current debate in Italy, we claim that evaluation should not be refused by heterodox economists, but rather that a reflection on the criteria of evaluation should be put forward at an international level in order to establish fair competition among research paradigms, thus, preserving pluralism in the discipline.
In this paper, we describe the development of a methodology and an instrument to support a major research funding allocation decision by the Flemish government. Over the last decade, and in parallel with the decentralization and the devolution of the Belgian federal policy authority towards the various regions and communities in the country, science and technology policy have become a major component of regional policy making. In the Flemish region, there has been an increasing focus on basing the funding allocation decisions that originate from this policy decentralization on "objective, quantifiable and repeatable" decision parameters. One of the data sources and indicator bases that have received ample attention in this evolution is the use of bibliometric data and indicators. This has now led to the creation of a dedicated research and policy support staff, called "Steunpunt O&O Statistieken," and the first time application of bibliometric data and methods to support a major inter-university funding allocation decision. In this paper, we analyze this evolution. We show how bibliometric data have for the first time been used to allocate 93 million Euro of public research money between 6 Flemish universities for the fiscal year 2003, based on Web-of-Science SCI data provided to "Steunpunt O&O Statistieken" via a license agreement with Tbomson-ISI. We also discuss the limitations of the current approach that was based on inter-university publication and citation counts. We provide insights into future adaptations that might make it more representative of the total research activity at the universities involved (e.g., by including data for the humanities) and of its visibility (e.g., by including impact measures). Finally, based on our current experience and interactions with the universities involved, we speculate on the future of the specific bibliometric approach that has now been adopted. More specifically, we hypothesize that the a
In December 2003, seventeen years after the first UK research assessment exercise, Italy started up its first-ever national research evaluation, with the aim to evaluate, using the peer review method, the excellence of the national research production. The evaluation involved 20 disciplinary areas, 102 research structures, 18,500 research products and 6,661 peer reviewers (1,465 from abroad); it had a direct cost of 3.55 millions Euros and a time length spanning over 18 months. The introduction of ratings based on ex post quality of output and not on ex ante respect for parameters and compliance is an important leap forward of the national research evaluation system toward meritocracy. From the bibliometric perspective, the national assessment offered the unprecedented opportunity to perform a large-scale comparison of peer review and bibliometric indicators for an important share of the Italian research production. The present investigation takes full advantage of this opportunity to test whether peer review judgements and (article and journal) bibliometric indicators are independent variables and, in the negative case, to measure the sign and strength of the association. Outcomes allow us to advocate the use of bibliometric evaluation, suitably integrated with expert review, for the forthcoming national assessment exercises, with the goal of shifting from the assessment of research excellence to the evaluation of average research performance without significant increase of expenses.
The Polish Parliament this month began voting on legislation creating a new national agency charged with distributing competitive
grants for frontier research. The proposed National Center for Science (NCN), to be located in Krakow, is meant to be free
from political pressures and would use an international peer-review system modeled on those of the European Research Council
and the U.S. National Science Foundation. NCN would also earmark at least 20% of its budget to grants for scientists under
age 35. Michal Kleiber, president of the Polish Academy of Sciences, sees in NCN the type of reform the country's scientific
community needs.
Nature - the world's best science and medicine on your desktop
This article outlines the evolution of international scientific production in Spain over the last 25 years, a period characterised by steady growth in research production. The following stages in this process are identified in accordance with some of the factors that predominated at different times. From 1974 to 1982 production increased due to causes endogenous to the scientific system itself, as scientists brought their work into line with the patterns which characterised research in other industrialised countries. From 1982 to 1991 the prioritisation of R&D by government administrative bodies represented a constant stimulus, implemented through a set of legal measures, investments and the creation of posts for new researchers. From 1989 to the present the creation of the Comisión Nacional de Evaluación de la Actividad Investigadora (National Commission for the Evaluation of Research Activity, CNEAI) and the research incentive system have provided a further stimulus, which has led to the maintenance of, and an increase in, the rate of research production in spite of the net decrease in the monetary value of research grants awarded during the last period analysed. Other special characteristics of Spanish research, such as its dependence on the public sector and its essentially academic nature, are discussed.
In previous research on the impact of the Research Assessment Exercise on heterodox economics and heterodox economists in the UK, the author concluded that reliance on Diamond List journals to rank departments would drive economic departments to discriminate positively in terms of their hiring, promotion and research strategies in favour of mainstream economists and their research, in order to maintain or improve their ranking. As a consequence, the author predicted there would be no or only a token presence of heterodox economists in an increasing number of departments. Whether the conclusions still hold and the predictions materialise is the subject of the paper. Copyright 2007, Oxford University Press.
Research Evaluation: Methods, Practice, and Experi-ence. Danish Agency for Science, Technology and Innovation (downloaded on
- Fosse Hansen
Fosse Hansen, H., 2009. Research Evaluation: Methods, Practice, and Experi-ence. Danish Agency for Science, Technology and Innovation (downloaded on June 9, 2011 from http://en.fi.dk/research/rearch-evaluation-methods-practice-and-experience/Research%20Evaluation%20Methods-%20Practice-%20and%20Experience.pdf/).
Using Metrics to Allocate Research Funds: A short evaluation of alternatives to the Research Assessment Exercise. Higher Educa-tion Policy Institute A bibliometric funding model based on a national research information system. Presented at the ISSI
- T Sastry
- B Bekhradnia
Sastry, T., Bekhradnia, B., 2006. Using Metrics to Allocate Research Funds: A short evaluation of alternatives to the Research Assessment Exercise. Higher Educa-tion Policy Institute, Oxford. Sivertsen, G., 2009. A bibliometric funding model based on a national research information system. Presented at the ISSI 2009, Rio de Janeiro, Brazil (downloaded on June 9, 2011 from www.issi2009.org/ agendas/issiprogram/activity.php?lang=en&id=108).
How jour-nal rankings can suppress interdisciplinarity. The case of innovation studies in business and management Sound research, unimportant discoveries: research, universities, and formal evaluation of research in Spain
- I Rafols
- L Leydesdorff
- O Hare
- A Nightingale
- P Stirling
Rafols, I., Leydesdorff, L., O'Hare, A., Nightingale, P., Stirling, A., 2011. How jour-nal rankings can suppress interdisciplinarity. The case of innovation studies in business and management, May. Rodríguez-Navarro, A., 2009. Sound research, unimportant discoveries: research, universities, and formal evaluation of research in Spain. Journal of the American Society for Information Science and Technology 60, 1845–1858.
The ratings game Australian Research Council (ARC), 2009. ERA 2010 Submission Guidelines (down-loaded on
- Anonymous
Anonymous, 2010. The ratings game. Nature 464 (March (7285)), 7. Australian Research Council (ARC), 2009. ERA 2010 Submission Guidelines (down-loaded on June 9, 2011 from www.arc.gov.au/era/key docs10.htm).
Feasibility Study: the Evaluation and Benchmarking of Humanities Research in Europe. HERA D4.2.1. Arts and Humanities Research Council, United Kingdom. European Commission, 2010. Assessing Europe's University-Based Research
- C Dolan
Dolan, C., 2007. Feasibility Study: the Evaluation and Benchmarking of Humanities Research in Europe. HERA D4.2.1. Arts and Humanities Research Council, United Kingdom. European Commission, 2010. Assessing Europe's University-Based Research. EUR 24187 EN, Science in Society 2008 Capacities, 1.
RAE Selection Gets Brutal. Times Higher Education Supplement 1779
- A Lipsett
Lipsett, A., 2007. RAE Selection Gets Brutal. Times Higher Education Supplement 1779, p. 1.
Tertiary Education Commission, 2010. Performance-based Research Fund-Tertiary Education Commission, www.tec.govt.nz/Funding
- New Zealand
New Zealand, Tertiary Education Commission, 2010. Performance-based Research Fund-Tertiary Education Commission, www.tec.govt.nz/Funding/Fund-finder/Performance-Based-Research-Fund-PBRF-/ (accessed 09.06.11).
Towards a Bibliometric Database for the Social Sciences and Humanities—A European Scoping Project. Final Report on Project for the European Science Foundation. Higher Education Funding Council for England (HEFCE), 1997. The Impact of the
- D Hicks
- J Wang
Hicks, D., Wang, J., 2009. Towards a Bibliometric Database for the Social Sciences and Humanities—A European Scoping Project. Final Report on Project for the European Science Foundation. Higher Education Funding Council for England (HEFCE), 1997. The Impact of the 1992 Research Assessment Exercise on Higher Education Institutions in England, No. M6/97. Higher Education Funding Council for England, Bristol (downloaded on June 9, 2011 from www.hefce.ac.uk/pubs/hefce/1997/m6 97.htm).
The UK Research Assessment Exercise: a case of regulatory capture? Reconfiguring Knowledge Production: Changing Authority Relationships in the Sciences and their Consequences for Intellectual Innovation
- B R Martin
- R Whitley
Martin, B.R., Whitley, R., 2010. The UK Research Assessment Exercise: a case of regulatory capture? In: Whitley, R., Gläser, J., Engwall, L. (Eds.), Reconfiguring Knowledge Production: Changing Authority Relationships in the Sciences and their Consequences for Intellectual Innovation. Oxford University Press, Oxford, UK, pp. 51–80 (Chapter 2).
Improvements to Excellence in Research for Australia Press release, Senator the Hon Kim Carr, Minister for Innovation
- K Carr
Carr, K., 2011. Improvements to Excellence in Research for Australia. Press release, Senator the Hon Kim Carr, Minister for Innovation, Industry, Science and Research, 30 May 2011. http://minister.innovation.gov.au/Carr/MediaReleases/ Pages/IMPROVEMENTSTOEXCELLENCEINRESEARCHFORAUSTRALIA.aspx (accessed 09.06.11).
Impacts of performance-based research funding systems: a review of the concerns and the evidence. In: Chapter 4 in OECD, Performance-based Funding for Public Research in Tertiary Education Institutions: Workshop Pro-ceedings
- L Butler
Butler, L.,2010. Impacts of performance-based research funding systems: a review of the concerns and the evidence. In: Chapter 4 in OECD, Performance-based Funding for Public Research in Tertiary Education Institutions: Workshop Pro-ceedings. OECD Publishing.
General Principles of Parametric Evaluation of Scientific Institutions
- Poland
Poland, Ministry of Science and Higher Education, 2010. General Principles of Parametric Evaluation of Scientific Institutions, http://translate.google. com/translate?hl=en&sl=pl&u=http://www.nauka.gov.pl/finansowanie/ finansowanie-nauki/dzialalnosc-statutowa/&ei=lDG7S-KLEJGu9gSdlZCFCA &sa=X&oi=translate&ct=result&resnum=7&ved=0CCQQ7gEwBg&prev=/ search%3Fq%3Dfinansowania%2Bbada%25C5%2584%2Bstatutowych %26hl%3Den%26sa%3DG%26rlz%3D1G1GGLQ ENUS375 (accessed 09.06.11).
Performance-based R&D institutional funding in Flemish Universi-ties
- M Luwel
Luwel, M., 2010. Performance-based R&D institutional funding in Flemish Universi-ties. In: Presented at the OECD Workshop, 21 June, Paris.