No full-text available
To read the full-text of this research,
you can request a copy directly from the author.
Commonly Used Indexes for Assessment of Research Production
Abstract
In this chapter, selected indicators and indexes (constructed on the basis of research publications and/or on the basis of a set of citations of these publications) are discussed. These indexes are frequently used for assessment of production of individual researchers. The chapter begins with several general remarks about indicators and indexes used in scientometrics. Then the famous h-index of Hirsch, its variants, and indexes complementary to the h-index are discussed. Next the g-index of Egghe as well as the -indexes are described. The h-index, g-index, and -indexes may provide a minimum of information for the quantitative part of assessment of the production of a researcher. Numerous indexes are described further in the text such as the m-index, p-index, -index, A-index, R-index. The discussion of indexes continues with a discussion of indexes for the success of a researcher. In addition, a short list of indexes for quantitative characterization of research networks and their dynamics is presented.
... The sets of quantities that are used in scientometrics are mentioned, and in addition, the importance of understanding the inequality of scientific achievements and the usefulness of knowledge landscapes for understanding and evaluating research performance is stressed. In [55] selected indicators and indexes are discussed (about 45 indexes and indicators constructed on the basis of research publications and/or on the basis of a set of citations of these publications). Among them are the h-index of Hirsch, its variants, and indexes complementary to the h-index; the g-index of Egghe; the i n -indexes; the m-index; p-index, A-index, R-index, etc. ...
... In addition, a short list of indexes for quantitative characterization of research networks and their dynamics is given. In addition to indexes from [55], other indexes useful for assessment of production of groups of researchers may be used. About ninety such indexes are discussed in [56]. ...
Machine learning is connected to the scientific study of algorithms and statistical models used by computer systems to perform a specific task without using explicit instructions. In this chapter we describe results of our studies on the methods connected to machine learning and the practical application of these methods to various problems by our team in the last two decades. We discuss in more detail the research on concepts of the nonlinear time series analysis and extreme events theory and their applications to natural, economic and social systems, statistical analysis of flows in channels of networks and the methodology for solving nonlinear differential equations.
... Importance of the non-Gaussianity of many statistical characteristics of social processes is stressed, because non-Gaussianity is connected to important requirements for study of these processes such as the need for multifactor analysis or probabilistic modeling. In [55] selected indicators and indexes are discussed. Among them are the h-index of Hirsch, its variants, and indexes complementary to the h-index; the g-index of Egghe; the i nindexes; the m-index; p-index, A-index, R-index, etc.. ...
... In addition, a short list of indexes for quantitative characterization of research networks and their dynamics is given. In addition to indexes from [55], other indexes useful for assessment of production of groups of researchers may be used. About ninety such indexes are discussed in [56]. ...
We discuss several results of our studies on the methods connected to study of nonlinear and practical application of these methods to various problems by our team in last two decades. We discuss in more detail the research on concepts of nonlinear time series analysis and extreme events theory and their applications to natural, economic and social systems, statistical analysis of flows in channels of networks and to methodology for solving nonlinear differential equations.
... The h-index, for example, is a widely acknowledged tool to evaluate the influence of articles and authors and is a better alternative to traditional citation metrics. However, it fails to account for the impact of multiple citation count and therefore leaves a significant quantity of citation unaccounted for (Vitanov, 2016;Bihari et al., 2021). The g-index, created in 2006 is an alternative to the h-index and gives more weightage to highly cited publications. ...
Abstract
Purpose
The purpose of this study is to gain insight into how citations are distributed and concentrated in the introduction, methods, discussion, results and other sections of journal articles to determine which section has received the most citations and whether the citation concentration score affects how articles rank.
Design/methodology/approach
The present study uses scite.ai and the Dimensions database to emphasize the significance of including multiple in-text citations in evaluating the impact and quality of journal publications. The study has two approaches: paper-based and author-based.
Findings
The study provides empirical insights into how variations in ranking are observed when citation concentration is considered in the evaluation process. It also suggests that in-text citations be used as an evaluation criterion or aspect for assessing the impact and quality of journals, publications and authors.
Originality/value
This study underscores the importance of considering citation concentration when evaluating journal articles. To assess highly cited articles, it suggests using the CC-index method, which is based on scite.ai.
... There are various centrality measures that have been defined to rank researchers based on the analysis of the citation network, such as h-index [208], g-index, and so on. Vitanov studied some of these centrality metrics like hindex [208], variations of h-index, g-index, i n -index, on citation network for the assessment of researchers [264]. He further discusses m-index, p-index, IQ p -index, A-index, and R-index with respect to the success of a researcher. ...
In complex networks, each node has some unique characteristics that define the importance of the node based on the given application-specific context. These characteristics can be identified using various centrality metrics defined in the literature. Some of these centrality measures can be computed using local information of the node, such as degree centrality and semi-local centrality measure. Others use global information of the network like closeness centrality, betweenness centrality, eigenvector centrality, Katz centrality, PageRank, and so on. In this survey, we discuss these centrality measures and the state of the art literature that includes the extension of centrality measures to different types of networks, methods to update centrality values in dynamic networks, methods to identify top-k nodes, approximation algorithms, open research problems related to the domain, and so on. The paper is concluded with a discussion on application specific centrality measures that will help to choose a centrality measure based on the network type and application requirements.
Recently, the abundance of digital data is enabling the implementation of graph-based ranking algorithms that provide system level analysis for ranking publications and authors. Here, we take advantage of the entire Physical Review publication archive (1893-2006) to construct authors' networks where weighted edges, as measured from opportunely normalized citation counts, define a proxy for the mechanism of scientific credit transfer. On this network, we define a ranking method based on a diffusion algorithm that mimics the spreading of scientific credits on the network. We compare the results obtained with our algorithm with those obtained by local measures such as the citation count and provide a statistical analysis of the assignment of major career awards in the area of physics. A website where the algorithm is made available to perform customized rank analysis can be found at the address http://www.physauthorsrank.org.
Currently the ranking of scientists is based on the h-index, which is
widely perceived as an imprecise and simplistic though still useful metric. We
find that the h-index actually favours modestly performing researchers and
propose a simple criterion for proper ranking.
Scientific collaboration networks have been studied systematically since 1960 by scholars belonging to various disciplinary backgrounds. As a result, the complex phenomenon of scientific collaboration networks has been investigated within different approaches. Although the term “scientific collaboration network” has different connotations in the literature, we use the term more narrowly to focus on scientific collaboration resulting in co-authored public documents. We broaden this beyond journal articles to include many types of scientific productions in addition to journal articles and books. We insist that these productions are public items available in each field. In this chapter, we focus on the main quantitative approaches dealing with the structure and dynamics of scientific collaboration networks through co-authorized publications. We provide a brief history of social network analysis that serves as a foundation. We further review earlier conceptual classifications of co-authorship networks and distinguish cross-disciplinarily, cross-sectoral and cross-national levels. We couple the newer ideas of “small world” models and “preferential attachment” to older sociological conceptions of scientific collaboration. This is followed by descriptions of deterministic and stochastic models that have been used to study dynamic scientific collaboration networks. We stress the importance of delineating the topology of collaboration networks, understanding micro-level processes and then coupling them. We conclude by outlining the strengths and limitations of various modeling strategies.
We study the lobby index (ll-index for short) as a local node centrality measure for complex networks. The ll-index is compared with degree (a local measure), betweenness and Eigenvector centralities (two global measures) in the case of a biological network (Yeast interaction protein–protein network) and a linguistic network (Moby Thesaurus II). In both networks, the ll-index has a poor correlation with betweenness but correlates with degree and Eigenvector centralities. Although being local, the ll-index carries more information about its neighbors than degree centrality. Also, it requires much less time to compute when compared with Eigenvector centrality. Results show that the ll-index produces better results than degree and Eigenvector centrality for ranking purposes.
The compartmental models used to study epidemic spreading often assume the same susceptibility for all individuals, and are therefore, agnostic about the effects that differences in susceptibility can have on epidemic spreading. Here we show that-for the SIS model-differential susceptibility can make networks more vulnerable to the spread of diseases when the correlation between a node's degree and susceptibility are positive, and less vulnerable when this correlation is negative. Moreover, we show that networks become more likely to contain a pocket of infection when individuals are more likely to connect with others that have similar susceptibility (the network is segregated). These results show that the failure to include differential susceptibility to epidemic models can lead to a systematic over/under estimation of fundamental epidemic parameters when the structure of the networks is not independent from the susceptibility of the nodes or when there are correlations between the susceptibility of connected individuals.
Despite its increasing role in communication, the world wide web remains the least controlled medium: any individual or institution can create websites with unrestricted number of documents and links. While great efforts are made to map and characterize the Internet's infrastructure, little is known about the topology of the web. Here we take a first step to fill this gap: we use local connectivity measurements to construct a topological model of the world wide web, allowing us to explore and characterize its large scale properties. Comment: 5 pages, 1 figure, updated with most recent results on the size of the www
Recent studies have shown that counting citations from books can help scholarly impact assessment and that Google Books (GB) is a useful source of such citation counts, despite its lack of a public citation index. Searching GB for citations produces approximate matches, however, and so its raw results need time-consuming human filtering. In response, this article introduces a method to automatically remove false and irrelevant matches from GB citation searches in addition to introducing refinements to a previous GB manual citation extraction method. The method was evaluated by manual checking of sampled GB results and comparing citations to about 14,500 monographs in the Thomson Reuters Book Citation Index (BKCI) against automatically extracted citations from GB across 24 subject areas. GB citations were 103% to 137% as numerous as BKCI citations in the humanities, except for tourism (72%) and linguistics (91%), 46% to 85% in social sciences, but only 8% to 53% in the sciences. In all cases, however, GB found substantially more citing books than did BKCI, with BKCI's results coming predominantly from journal articles. Moderate correlations between the GB and BKCI citation counts in social sciences and humanities, with most BKCI results coming from journal articles rather than books, suggests that they could measure the different aspects of impact, however.
Based on the foundation laid by the h-index we introduce and study the R- and AR-indices. These new indices eliminate some of the disadvantages of the h-index, especially when they are used in combination with the h-index. The R-index measures the h-core’s citation intensity, while AR goes one step further and takes the age of publications into account. This allows for an index that can actually increase and decrease over time. We propose the pair (h, AR) as a meaningful indicator for research evaluation. We further prove a relation characterizing the h-index in the power law model.
With reference to Vanclay (Scientometrics in press, 2012) the paper argues for a pragmatic approach to the Thomson-Reuter’s journal impact factor. The paper proposes and discusses to replace the current synchronous Thomson-Reuter journal impact factor by an up-to-date diachronic version (DJIF), consisting of a three-year citation window over a one year publication window. The DJIF online data collection and calculation is exemplified and compared to the present synchronous journal impact factor. The paper discusses briefly the dimensions of currency, robustness, understandability and comparability to other impact factors used in research evaluation.
The partnership ability index (φ) combines the number of co-authors and the times each of them acted as co-authors with a given author exactly the same way as Hirsch’s h-index combines the number of publications and their citation rate. The index φ was tested on the sample of the Hevesy medal awardees. It was found that φ is consistent with Glänzel’s model of h-index, and that higher φ values—at least until a certain limit—may be accompanied with higher citation visibility (h-index). Some further possibilities of application both within and outside the area of scientometrics are suggested.
Describes a new method of evaluation of scientific output by laboratories engaged in diverse fields of research. This method helps to evaluate those outputs which are quite recent and not amenable to citation analysis. For the purpose of analysis, impact factor of journals in which papers are published are considered. A method for normalisation of impact factor of journals has been described and, normalised impact factors have also been used for the purpose of analysis. It is found that in such analysis normalised impact factor tends to show better results compared to simple impact factor. The analysis helps us to generate numerous performance indicators such as average impact factor and normalised impact factor for each laboratory and the research complex such as CSIR as a whole; average impact factor and normalised impact factor for each scientist of a laboratory and the research complex; spectral distribution of papers falling within various ranges of impact factors and normalised impact factors. By comparing the performances over several years the trend of research activity of each laboratory can also be obtained.
An individual’s h-index corresponds to the number h of his/her papers that each has at least h citations. When the citation count of an article exceeds h, however, as is the case for the hundreds or even thousands of citations that accompany the most highly cited papers, no
additional credit is given (these citations falling outside the so-called “Durfee square”). We propose a new bibliometric
index, the “tapered h-index” (h
T), that positively enumerates all citations, yet scoring them on an equitable basis with h.
The career progression of h
T and h are compared for six eminent scientists in contrasting fields. Calculated h
T for year 2006 ranged between 44.32 and 72.03, with a corresponding range in h of 26 to 44. We argue that the h
T-index is superior to h, both theoretically (it scores all citations), and because it shows smooth increases from year to year as compared with the
irregular jumps seen in h. Conversely, the original h-index has the benefit of being conceptually easy to visualise. Qualitatively, the two indices show remarkable similarity
(they are closely correlated), such that either can be applied with confidence.
Bibliometrie ist ein Forschungsgebiet, das sich mit der statistischen Analyse bibliographischer Informationen beschäftigt.
Die meisten bibliometrischen Untersuchungen sind auch als scientometrisch einzustufen, weil sie sich auf den Publikationsoutput
der Wissenschaft beziehen, vor allem auf Zeitschriftenaufsätze. Informetrie erfasst auch Informationsströme jenseits der Welt
der Bücher und Zeitschriften, einschließlich der Kommunikation über das Web (Webometrie) und das Internet.
The objective of the present study is twofold: (1) to show the aims and means of quantitative interpretation of bibliographic features in bibliometrics and their re-interpretation in research policy, and (2) to summarise the state-of-art in self-citation research. The authors describe three approaches to the role of author self-citations and possible conflicts arising from the different perspectives.L’objectif de cette étude est de caractériser les effets de quantification comme appliqués dans la bibliométrie et leur interprétation en politique de recherche. Nous présentons un sommaire de la recherche en ce qui concerne le phénomène des auto-citations et nous décrivons trois possibilités d’expliquer les auto-citations aussi bien que les conflits qui résultent de ces points de vues.
I find that a firm's innovation output increases with the number of collaborative linkages maintained by it, the number of structural holes it spans, and the number of partners of its partners. However, innovation is negatively related to the interaction between spanning many structural holes and having partners with many partners.
In our communication the expression for the simple matching function should read:| ATI Y\ + \A-X\JY\ A-X/JY/>for subsets ?, ? ?? a finite set A.
The science of graphs and networks has become by now a well-established tool for modelling and analyzing a variety of systems with a large number of interacting components. Starting from the physical sciences, applications have spread rapidly to the natural and social sciences, as well as to economics, and are now further extended, in this volume, to the concept of innovations, viewed broadly.
In an abstract, systems-theoretical approach, innovation can be understood as a critical event which destabilizes the current state of the system, and results in a new process of self-organization leading to a new stable state.
The contributions to this anthology address different aspects of the relationship between innovation and networks. The various chapters incorporate approaches in evolutionary economics, agent-based modeling, social network analysis and econophysics and explore the epistemic tension between insights into economics and society-related processes, and the insights into new forms of complex dynamics.
Aimed at academics, academic managers and administrators, professionals in scientometrics, information scientists and science policy makers at all levels. This book reviews the principles, methods and indicators of scientometric evaluation of information processes in science and assessment of the publication activity of individuals, teams, institutes and countries. It provides scientists, science officers, librarians and students with basic and advanced knowledge on evaluative scientometrics. Especially great stress is laid on the methods applicable in practice and on the clarification of quantitative aspects of impact of scientific publications measured by citation indicators.
Until recently, little attention was paid in Belgium to the efficiency of the research system. The University of Ghent was the first Belgian university to decide to perform a systematic evaluation of its research performance using bibliometric indicators derived from the Science. Citation Index. The results were compared to the world average and to the average impact of the set of journals, in which the Ghent scientists published. Subsequently, these results were discussed with scientists of several evaluated departments to obtain tentative explanations for the bibliometric patterns. The university policy makers can use these results for research planning and the scientists for improving their publication strategy.
This two-part article reviews the current literature on journal peer review. Research on this subject has grown during the 1980s and 1990s and has increased our awareness of both the myths and facts about peer review. Part 1 summarizes research findings on the participants in the system (the appointment mechanisms of editors and referees, and reviewer tasks and qualifications) and systemic problems of reliability, accuracy, and bias. Part 2 describes current research on how fraud, favoritism, and self-interest may affect the review system and on such policy issues as interference of particularistic criteria; connections among editors, authors, and referees; and double-blind review. Although the literature indicates that peer review has many problems, the author concludes that it is difficult to imagine how science could advance without such a key quality control mechanism.
The concept of h-index has been proposed to easily assess a researcher’s performance with a single number. However, by using only this number, we lose significant information about the distribution of citations per article in an author’s publication list. In this article, we study an author’s citation curve and we define two new areas related to this curve. We call these “penalty areas”, since the greater they are, the more an author’s performance is penalized. We exploit these areas to establish new indices, namely Perfectionism Index and eXtreme Perfectionism Index (XPI), aiming at categorizing researchers in two distinct categories: “influentials” and “mass producers”; the former category produces articles which are (almost all) with high impact, and the latter category produces a lot of articles with moderate or no impact at all. Using data from Microsoft Academic Service, we evaluate the merits mainly of PI as a useful tool for scientometric studies. We establish its effectiveness into separating the scientists into influentials and mass producers; we demonstrate its robustness against self-citations, and its uncorrelation to traditional indices. Finally, we apply PI to rank prominent scientists in the areas of databases, networks and multimedia, exhibiting the strength of the index in fulfilling its design goal.
In a bibliometric study of nine research departments in the field of biotechnology and molecular biology, indicators of research capacity, output and productivity were calculated, taking into account the researchers' participation in scientific collaboration as expressed in co-publications. In a quantitative approach, rankings of departments based on a number of different research performance indicators were compared with one another. The results were discussed with members from all nine departments involved. Two publication strategies were identified, denoted as a quantity of publication and a quality of publication strategy, and two strategies with respect to scientific collaboration were outlined, one focusing on multi-lateral and a second on bi-lateral collaborations. Our findings suggest that rankings of departments may be influenced by specific publication and management strategies, which in turn may depend upon the phase of development of the departments or their personnel structure. As a consequence, differences in rankings cannot be interpreted merely in terms of quality or significance of research. It is suggested that the problem of assigning papers resulting from multi-lateral collaboration to the contributing research groups has not yet been solved properly, and that more research is needed into the influence of a department's state of development and personnel structure upon the values of bibliometric indicators. A possible implication at the science policy level is that different requirements should hold for departments of different age or personnel structure.
AbstractA metrics‐based assessment can predict reasonably well the overall outcome of the Research Assessment Exercise 2008 for social work and social policy and administration in terms of research environment, but not in terms of research outputs. It is not possible to replicate peer review of the research outputs using existing data. It is sometimes argued that citation counts provide an alternative approach that might help research assessment, but it is one fraught with difficulties. Academics did not, in fact, routinely chose to submit their most cited work. At least in this subject, metrics are more suited as handmaiden to peer review than its replacement.
Nearly a decade ago, the science community was introduced to the h-index, a
proposed statistical measure of the collective impact of the publications of
any individual researcher. It is of course undeniable that any method of
reducing a complex data set to a single number will necessarily have certain
limitations and introduce certain biases. However, in this paper we point out
that the definition of the h-index actually suffers from something far
deeper: a hidden mathematical incompleteness intrinsic to its definition. In
particular, we point out that one critical step within the definition of h
has been missed until now, resulting in an index which only achieves its stated
objectives under certain rather limited circumstances. For example, this
incompleteness explains why the h-index ultimately has more utility in
certain scientific subfields than others. In this paper, we expose the origin
of this incompleteness and then also propose a method of completing the
definition of h in a way which remains close to its original guiding
principle. As a result, this "completed" h not only reduces to the usual h
in cases where the h-index already achieves its objectives, but also extends
the validity of the h-index into situations where it currently does not.
We show mathematically that the success-index can be any of the following impact indices, dependent on the value of the threshold used in the definition of the success-index: Hirsch-index (h-index), g-index, generalized Wu- and Kosmulski-indices, the average.
This paper examines the relationship between metrics, markets and affect in the contemporary UK academy. It argues that the emergence of a particular structure of feeling amongst academics in the last few years has been closely associated with the growth and development of ‘quantified control’. It examines the functioning of a range of metrics: citations; workload models; transparent costing data; research assessments; teaching quality assessments; and commercial university league tables. It argues that these metrics, and others, although still embedded within an audit culture, increasingly function autonomously as a data assemblage able not just to mimic markets but, increasingly, to enact them. It concludes by posing some questions about the possible implications of this for the future of academic practice.
Recent reductions in research budgets have led to the need for greater selectivity in resource allocation. Measures of past performance are still among the most promising means of deciding between competing interests. Bibliometry, the mea surement of scientific publications and of their impact on the scientific community, assessed by the citations they attract, provides a portfolio of indicators that can be combined to give a useful picture of recent research activity. In this state-of-the- art review the various methodologies that have been developed are outlined in terms of their strengths, weaknesses and par ticular applications. The present limitations of science indica tors in research evaluation are considered and some future directions for developments in techniques are suggested.
This research examines publication and citation data to assess the contribution of Australian educational research to major international journals. Additionally a set of highly cited articles is reviewed to assess the relevance of educational research to teaching practice and policy making. Australia makes a significant contribution to international educational research, although its share of publications has been increasing over time, whereas its share of citations has remained relatively constant. Substantial turnover in leading research institutions is found over time. It might be possible to explain this turnover by the productivity of a small number of academic staff members. Finally it is found that Australian educational research is extremely relevant to educational practice.
Using the dataset based on Thomson Reuters Scientific “Web of Science” the distributions of some well-known indicators, such as h-index and g-index, were investigated, and different citation behaviors across different scientific fields resulting from their field dependences were found. To develop a field-independent index, two scaling methods, based on average citation of subject category and journal, were used to normalize the citation received by each paper of a certain author. The distributions of the generalized h-indices in different fields were found to follow a lognormal function with mean and standard deviation of approximately −0.8 and 0.8, respectively. A field-independent index fi-index was then proposed, and its distribution was found to satisfy a universal power-law function with scaling exponent α approaching 3.0. Both the power-law and the lognormal universality of the distributions verified the field independence of these indicators. However, deciding which of the scaling methods is the better one is necessary for the validation of the field-independent index.
In an age of intensifying scientific collaboration, the counting of papers by multiple authors has become an important methodological issue in scientometric based research evaluation. Especially, how counting methods influence institutional level research evaluation has not been studied in existing literatures. In this study, we selected the top 300 universities in physics in the 2011 HEEACT Ranking as our study subjects. We compared the university rankings generated from four different counting methods (i.e. whole counting, straight counting using first author, straight counting using corresponding author, and fractional counting) to show how paper counts and citation counts and the subsequent university ranks were affected by counting method selection. The counting was based on the 1988–2008 physics papers records indexed in ISI WoS. We also observed how paper and citation counts were inflated by whole counting. The results show that counting methods affected the universities in the middle range more than those in the upper or lower ranges. Citation counts were also more affected than paper counts. The correlation between the rankings generated from whole counting and those from the other methods were low or negative in the middle ranges. Based on the findings, this study concluded that straight counting and fractional counting were better choices for paper count and citation count in the institutional level research evaluation.
Hypes occur in every domain of human behavior, including scientific research. We show in this contribution that journals and authors who studied the h-index benefited in terms of short-term citations. As, moreover, the introduction of the h-index is more a ‘clever find’ than a first rate intellectual achievement, its rise can be compared to a stock market bubble.
Based on an idea by Kosmulski, Franceschini et al. (2012, Scientometrics 92(3), 621–641) propose to classify a publication as “successful” when it receives more citations than a specific comparison term (CT). In the intention of the authors CT should be a suitable estimate of the number of citations that a publication – in a certain scientific context and period of time – should potentially achieve. According to this definition, the success-index is defined as the number of successful papers, among a group of publications examined, such as those associated to a scientist or a journal. In the first part of the paper, the success-index is recalled, discussing its properties and limitations. Next, relying on the theory of Information Production Processes (IPPs), an informetric model of the index is formulated, for a better comprehension of the index and its properties. Particular emphasis is given to a theoretical sensitivity analysis of the index.
Citation curves for researchers with the same h index can vary greatly in the heaviness of their top (excess citations to core papers) or the heaviness of their tail (citations to non-core papers), revealing quantitative differences across researchers. Also, promotion to the next higher h depends only on citations received by a small subset of papers, so that researchers with a given h may have citation curves whose top and tail reveal a weaker impact than that of researchers with a lower h. To overcome these problems, we propose a two-sided h index, an extension that computes additional h indices progressively up the top and out the tail of the citation curve. This extension represents a citation curve descriptor one of whose elements is the scalar h. The advantages of the two-sided h index are illustrated through analysis of citation curves for 88 researchers with h indices ranging from 8 to 20. Several schemes are also discussed that use the two-sided h index to define criteria for ranking researchers within and across scalar h indices, according to whether the top of the citation curve, its tail, or both are deemed relevant under the circumstances in which research accomplishments are assessed.
In the present work we introduce a modification of the h-index for multi-authored papers with contribution based author name ranking. The modified h-index is denoted by hmc-index. It employs the framework of the hm-index, which in turn is a straightforward modification of the Hirsch index, proposed by Schreiber. To retain the merit of requiring no additional rearrangement of papers in the hm-index and in order to overcome its shortage of benefiting secondary authors at the expense of primary authors, hmc-index uses combined credit allocation (CCA) to replace fractionalized counting in the hm-index. The hm-index is a special form of hmc-index and fits for papers with equally important authors or alphabetically ordered authorship. There is a possibility of an author of lower contribution to the whole scientific community obtaining a higher hmc-index. Rational hmc-index, denoted by hmcr-index, can avoid it. A fictitious example as a model case and two empirical cases are analyzed. The correlations of the hmcr-index with the h-index and its several variants considering multiple co-authorship are inspected with 30 researchers’ citation data. The results show that the hmcr-index is more reasonable for authors with different contributions. A researcher playing more important roles in significant work will obtain higher hmcr-index.
In this paper we focus on the analysis of peer reviews and reviewers behaviour in a number of different review processes. More specifically, we report on the development, definition and rationale of a theoretical model for peer review processes to support the identification of appropriate metrics to assess the processes main characteristics in order to render peer review more transparent and understandable. Together with known metrics and techniques we introduce new ones to assess the overall quality (i.e. ,reliability, fairness, validity) and efficiency of peer review processes e.g. the robustness of the process, the degree of agreement/disagreement among reviewers, or positive/negative bias in the reviewers’ decision making process. We also check the ability of peer review to assess the impact of papers in subsequent years. We apply the proposed model and analysis framework to a large reviews data set from ten different conferences in computer science for a total of ca. 9,000 reviews on ca. 2,800 submitted contributions. We discuss the implications of the results and their potential use toward improving the analysed peer review processes. A number of interesting results were found, in particular: (1) a low correlation between peer review outcome and impact in time of the accepted contributions; (2) the influence of the assessment scale on the way how reviewers gave marks; (3) the effect and impact of rating bias, i.e. reviewers who constantly give lower/higher marks w.r.t. all other reviewers; (4) the effectiveness of statistical approaches to optimize some process parameters (e.g. ,number of papers per reviewer) to improve the process overall quality while maintaining the overall effort under control. Based on the lessons learned, we suggest ways to improve the overall quality of peer-review through procedures that can be easily implemented in current editorial management systems.
J.K. Vanclay’s article is a bold attempt to review recent works on the journal impact factor (JIF) and to call for alternative certifications of journals. The too broad scope did not allow the author to fulfill all his purposes. Attempting after many others to organize the various forms of criticism, with targets often broader than the JIF, we shall try to comment on a few points. This will hopefully enable us to infer in which cases the JIF is an angel, a devil, or a scapegoat. We shall also expand on a crucial question that Vanclay could not really develop in the reduced article format: the field-normalization. After a short recall on classical cited-side or ex post normalization and of the powerful influence measures, we will devote some attention to the novel way of citing-side or ex ante normalization, not only for its own interest, but because it directly proceeds from the disassembling of the JIF clockwork.
The impact factor is a highly polemic metric. It was designed to help scientists in searching for bibliographic references for their own works, enabling communication among researchers and helping librarians in deciding which journal they should purchase. Nevertheless, it has soon become the most important measure of scientific performance applied to journals, articles, scientists, universities, etc. Since then, some researchers argue that it is a useless and flawed measure, while others defend its utility. The current study is the first survey on the opinion on the topic of a broad sample of scientists from all over the world. The questionnaire was answered by 1,704 researchers from 86 different countries, all the continents and all the UNESCO major fields of knowledge. The results show that the opinion is slightly above the median which could be understood as “neither positive nor negative”. Surprisingly, there is a negative correlation between the number of articles published by the respondents and their opinion on the impact factor.
Except the alphabetic ordering authorship papers, the citations of multi-authored papers are allocated to the authors based on their contributions to the paper. For papers without clarification of contribution proportion, a function of author number and rank is presented to rightly determine the credit allocated proportion and allocated citations of each author. Our citation allocation scheme is between the equally fractional counting and the one using the inverse of author rank. It has a parameter to adjust the credit distribution among the different authors. The allocated citations can either be used alone to indicate one’s performance in a paper, or can be applied in the modification of h-index and g-index to represent the achievement of a scientist on the whole. The modified h-index and g-index of an author makes use of more papers in which he or she played important roles. Our method is suitable for the papers with wide range of author numbers.
This paper argues that evaluations of basic research are best carried out using a range of indicators. After setting out the reasons why assessments of government-funded basic research are increasingly needed, we examine the multi-dimensional nature of basic research. This is followed by a conceptual analysis of what the different indicators of basic research actually measure. Having discussed the limitations of various indicators, we describe the method of converging partial indicators used in several SPRU evaluations. Yet although most of those who now use science indicators would agree that a combination of indicators is desirable, analysis of a sample ofScientometrics articles suggests that in practice many continue to use just one or two indicators. The paper also reports the results of a survey of academic researchers. They, too, are strongly in favour of research evaluations being based on multiple indicators combined with peer review. The paper ends with a discussion as to why multiple indicators are not used more frequently.
This two-part article reviews the current literature on journal peer review. Research on this subject has grown during the 1980s and 1990s, and has increased our awareness of both the myths and facts about peer review. Part 1 summarizes research findings on the participants in the system (the appointment mechanisms of editors and referees, and reviewer tasks and qualifications), and systemic problems of reliability, accuracy, and bias. Part 2 describes current research on how fraud, favoritism, and self-interest may affect the review system and on such policy issues as interference of particularistic criteria; connections among editors, authors, and referees; and double-blind reviewing. Although the literature indicates that peer review has many problems, the author concludes that it is difficult to imagine how science could advance without such a key quality control mechanism.
Purpose
This review aims to show, broadly, how the h ‐index has become a subject of widespread debate, how it has spawned many variants and diverse applications since first introduced in 2005 and some of the issues in its use.
Design/methodology/approach
The review drew on a range of material published in 1990 or so sources published since 2005. From these sources, a number of themes were identified and discussed ranging from the h ‐index's advantages to which citation database might be selected for its calculation.
Findings
The analysis shows how the h ‐index has quickly established itself as a major subject of interest in the field of bibliometrics. Study of the index ranges from its mathematical underpinning to a range of variants perceived to address the indexes' shortcomings. The review illustrates how widely the index has been applied but also how care must be taken in its application.
Originality/value
The use of bibliometric indicators to measure research performance continues, with the h ‐index as its latest addition. The use of the h ‐index, its variants and many applications to which it has been put are still at the exploratory stage. The review shows the breadth and diversity of this research and the need to verify the veracity of the h ‐index by more studies.