Content uploaded by Cengiz Acarturk
Author content
All content in this area was uploaded by Cengiz Acarturk on Jul 21, 2018
Content may be subject to copyright.
Multi-authoring and university rankings
1
Multi-authoring and its impact on university rankings: A case study of CERN effect on
Turkish universities
M. P. Çakır, C. Acartürk*, S. Alkan and U. Akbulut
URAP University Ranking by Academic Performance Research Laboratory,
Informatics Institute, Orta Dogu Teknik Universitesi 06800 Ankara, Turkey
Abstract. Multi-authoring has been gaining popularity since the foundation of academic
publishers. Today we observe a global trend in favor of multi-authoring. On the other hand, in
certain domains, such as CERN collaborations in physics, multi-authoring has a different
connotation than other domains. The number of co-authors may be above 5000, an all-time
record in the history of academic publishing. CERN collaborations have been accomplishing
one of the most significant achievements in physics, as well; therefore, the collaborations
produce highly cited articles. Highly cited, multi-authored articles have influenced university
rankings in the past decade, due to the distribution of the citations per affiliated institution. In
the present study, we analyze the influence of multi-authored articles on ranking
methodologies. Our findings reveal that the presence of multi-authored CERN articles have a
significant impact on all bibliometric ranking indicators, and consequently on the ranking
positions of the institutions worldwide.
Keywords. Multi-authored articles; academic performance; university rankings; ranking
indicators, academic publishing
* Corresponding author. Tel.: +90 312 210 7704; fax: +90 312 2103745
e-mail: acarturk@metu.edu.tr
Multi-authoring and university rankings
2
1. Introduction
Multi-authoring is as old as the history of scientific publications. Scientific journal publication
started soon after the foundation of science academies, such as Leopoldina Nationale Akademie
der Wissenschaften in Germany
1
(founded in 1652), Royal Society in United Kingdom
2
(founded in 1660), and Academie des Sciences in France
3
(founded in 1666). The first science
journals in history were Journal des Scavans in France (1665), Transactions of the Royal
Society in Great Britain (1665) and Miscellanea Curiosa in Germany (1670). Beaver and Rosen
(1978) report that the earliest multi-authored article was published by R. Hooke, H. Oldenburg,
J. D. Casini and R. Boyle in 1665. They also report that 6 multi-authored
4
articles were
published in in the 17th Century, and 41 were published in the 18th Century, thus an increase in
number.
The increase in the number of multi-authored articles, however, was not a global trend in the
19th Century. Multi-authoring has been subject to variance among countries. Between 1800 and
1819, the majority (81%) of multi-authored articles were published in France, 6% in Germany
and 2% in Great Britain. Between 1884 and 1900, the majority (39%) of multi-authored articles
were published in Germany, 30% in Great Britain and the US, and 18% in France. Today we
observe a global trend in multi-authoring, which has gone beyond national boundaries (Knobel,
et al., 2013).
Multi-authoring has gained stronger popularity in the 20th Century. According to Price (1963),
“Data from Chemical Abstracts showed that in 1900 more than 80 percent of all papers had a
single author, and almost all the rest were pairs, the greater number being those signed by a
1
http://www.leopoldina.org/, retrieved on August 29, 2017.
2
http://royalsociety.org/, retrieved on August 29, 2017.
3
http://www.academie-sciences.fr/, retrieved on August 29, 2017.
4
aka. collaborative, collaborated, coauthored Price (1963) coined the term “multi-authored papers”.
Multi-authoring and university rankings
3
professor and his graduate student” (p. 90). Beaver and Rosen (1979a) report that the share of
multi-authored articles in chemistry was around 20% in 1910, and it went up to 65% in 1960.
Price (1963) had gone further, by estimating that single-authored articles would be extinct by
the 1980s, based on the sharp increase in the number of multi-authored articles after 1900s.
Today we see that Price’s prediction was mostly correct. We observe a continuous increase in
multi-authoring (e.g., Crase & Rosato, 1992; Woods, Youn, & Johanson, 2010), leading to
approximately 95% share of multi-authored articles among all articles worldwide today.
The major factor that drives the popularity of multi-authoring is the stronger visibility of multi-
authored articles compared to single-authored articles. Beaver and Rosen (1979b) report that
the multi-authored articles led by J. B. Biot in France were considerably more visible than the
articles produced by the rest of the scientists between 1800 and 1869: “[…] the Biot group’s
work stood three times the chance of being reprinted and consequently had a greater chance of
being seen” (p. 146). Diamond (1985) “[…] presents evidence for the surprising conclusion
that a citation to a multiple-authored article is worth more to its author than a citation to a single-
authored article” (p.315). Rousseau (2001) state that “multi-authored articles have usually
higher citation frequencies than single-authored ones, but this relation does not hold in all cases.
On the other hand, it seems favorable (in the sense of receiving more citations) for a small
university in a small country, to collaborate with scientists from abroad” (p. 173). Today, the
stronger visibility of multi-authored articles and other advantages of team work continue
leading scientists towards multi-authored publications.
Multi-authored article publication exhibits differences among domains of research. For
instance, single-authored articles constituted the norm in social sciences and mathematics for a
longer period in contrast to other fields, whereas publications with two or three authors have
Multi-authoring and university rankings
4
been prevalent in chemistry, physics and medicine. In medicine, the percentage of multi-
authored articles among all scientific publications, as well as the number of authors in articles
increased rapidly. An article with 972 authors was published on acute myocardial infarction in
The New England Journal of Medicine, in 1993 (GUSTO Investigators, 1993). The
Management of Elevated Cholesterol in the Primary Prevention Group of Adult Japanese
(MEGA) analyzed 8009 patients in 924 hospitals in Japan and published the results in an article,
which consisted of 2459 authors (Nakamura et al., 2004). In physics, the scientists and
engineers affiliated with the Collider Detector at Fermilab (CDF) have been added to the
standard author list of all the articles published by CDF, in alphabetical order, since 1998. The
criteria for being involved in the list of standard authorship is to work full time for at least one
year as a member of CDF. The previous members also have the right for authorship until one
year after their leave (Biagioli, 2003; Hyland, 2015). The CDF collaboration published an
article in 2003 with 817 authors (Acosta, et al., 2003). Another collaboration in physics, namely
the BaBar collaboration, was established at Stanford University SLAC National Accelerator
Laboratory. The BaBar collaboration, being specialized on high energy physics, carried out the
BaBar experiment. An article published by the collaboration involved as many as 823 authors
(Aubert, et al., 2002).
More recently, a group of high energy physicists in CERN (Conseil Européen pour la Recherche
Nucléaire, European Council for Nuclear Research) published an article on the final
electroweak measurements performed with data taken at the Z resonance based on the
experiments conducted at the electron-positron colliders SLC and LEP. The article had 2517
authors from ALEPH, DELPHI, L3, OPAL and SLD Collaborations (Schael, et al., 2006).
Multi-authored articles were published by CERN scientists in the following years: Bayatian, et
al. (2007) had 2011 authors, Chatrchyan, et al. (2008) had 3101 authors, Aad, et al. (2010) had
Multi-authoring and university rankings
5
3221 authors, and Aad, et al. (2011) had 3179 authors. In 2012, CERN researchers announced
that they found experimental evidence for the presence of the Higgs boson, both in ATLAS and
CMS detectors. The ATLAS collaboration and the CMS collaboration published their findings
separately. The ATLAS article (Aad, et al., 2012) had 2992 authors, whereas the CMS article
(Attebury, et al., 2012) had 2891 authors. ATLAS and CMS collaborations later published
together a collaborative article about Higgs boson (Aad, et al., 2015), which consisted of 5154
authors.
CERN collaborations, which is also the topic of the present study, have been significant players
not only in contemporary physics but also in the history of science. Their findings pave the way
to answer questions that have been awaiting solution, based on theoretical predictions that were
made in the past century, such as the presence of the Higgs boson. CERN has a major influence
in terms of bibliometric analyses, as well. More generally, a review of the history and the
current state of multi-authored articles shows that the current inflated multi-authorship trends
in physics, medicine and genetics, may expand into other research fields, evolving into a
logistics issue for scientific journals, as well as a methodological problem for university ranking
systems.
In the present study, we focus on the impact of multi-authored CERN articles on ranking
systems. Section 2 presents a brief history of ranking systems for Higher Education Institutions
(HEI). In Section 3, we present a survey and an analysis of multi-authored articles published
by CERN, and the impact of the CERN articles on HEIs worldwide, their impact on HEIs in
Turkey, and their impact on ranking indicators. Section 4 concludes the article.
Multi-authoring and university rankings
6
2. University Rankings: A Brief History
The earliest university ranking attempt was made by Galton (1875), who examined the
backgrounds of Royal Society fellows and investigated whether their success was predestined
by their parents’ educational status and occupation or by the environment. After Galton, the
British universities were the first HEIs to be ranked based on the number of successful graduates
(Maclean, 1900). Similarly, Ellis (1904) ranked British universities according to the number of
genius graduates. The US university ranking by Cattell (1906a,b) was based on the number of
top scientists employed by each university (viz. American Men of Science). The US universities
were ranked by Kunkel and Prentice, according to the percentage of their living alumni in the
1928 edition of Who’s Who in America (Wegner, 1967). The first ranking based on reputation
was conducted in the US by Hughes (1925), who based his ranking on the surveys which he
sent to hundreds of academicians in the US. The ranking involved 38 universities in 20 graduate
disciplines among a total of 65 US universities in 1924.
The first annual, national university ranking is published by the US News and World Report
magazine. The US News ranked higher education institutions of the US first in 1983 and then
annually since 1987. The first world university ranking was announced in 2003 by ARWU
(Academic Ranking of World-class Universities). Many world university rankings appeared
since the past three decades, such as the Leiden Ranking, Times Higher Education (THE)
Ranking, QS World Ranking, National Taiwanese University Ranking (formerly known as the
HEEACT Ranking), SCImago, URAP World Ranking and Webometrics. National university
Multi-authoring and university rankings
7
rankings have also been increasing in number
5
(see Bowden, 2000; Li et al., 2012; Liu & Liu
2005; Usher & Savino, 2007; Stolz et al., 2010; Cheng, 2011, for reviews).
University rankings serve various purposes in the society. They influence the choices of
prospective HEI students and their families, who comprise the most interested stakeholder of
university rankings (Bowman & Bastedo, 2009; Griffith & Rask, 2007). University
administrations are interested in rankings for developing policies and strategies for institutional
development (Hazelkorn, 2008). At a broader level, HEI policy makers employ rankings to
observe trends at both national and international levels (Hazelkorn, 2007). Finally, news media
employ university rankings for informing the society about the status of national HEIs among
world universities or among national ones (Alasehir et al., 2014).
University ranking systems are characterized by the ranking indicators that they employ. Each
indicator emphasizes a certain aspect of institutional activity. For instance, the number of
published articles and the number of citations accrued by the published articles are frequently
used as indicators of institutional research output. The number of doctoral students, as a ranking
indicator, aims at taking into account the size of the graduate-level studies within HEIs.
Ranking indicators may be categorized under various dimensions. A major dimension divides
the indicators into two families: size-dependent indicators vs. size-independent indicators.
Size-dependent ranking indicators, such as the number of articles, citations and doctoral
students, are based on absolute numbers. On the other hand, size-independent indicators, such
5
The list includes Folha’s University Ranking of Brazil, ARWU’s Greater China Ranking, ARWU’s
Macedonian HEIs Ranking, EI Mercurio in Chile, Netbig in China, CHE University Ranking in Germany,
India Today-Nielsen in India, Vision in Italy, IQAA Ranking in Kazakhstan, KCUE Ranking in Korea,
Veidas in Lithuania, Setera in Malaysia, PBRF Ranking in New Zealand, HEC Ranking in Pakistan,
Perspectywy in Poland, Ad-Astra Ranking in Romania, ARRA in Slovakia, and Top-200 Ranking in Ukraine.
Multi-authoring and university rankings
8
as the number of articles per faculty, normalize size-dependent measures in various ways. The
number of citations is divided by the number of faculty members in a HEI to derive the number
of citations per faculty. It can also be divided by the number of publications, yielding the so-
called Citations per Publication (CPP) indicator (Li & Ho, 2008). CPP was used by the THE-
QS ranking and the Leiden ranking.
In the early years of ranking systems, CPP seemed to be a suitable indicator for HEI ranking
since it is a size-independent indicator. However, after the increase in the number of highly
cited multi-authored articles, the use of the CPP as a reliable indicator became questionable.
This is because CPP is adversely influenced by the number of citations of the HEIs that produce
small numbers of articles, as we discuss in the following section. A similar situation applies to
the indicators, such as P(top 1%) and P(top 10%), where P specifies the largest number of
publications with a certain property in the Leiden Ranking (Waltman, et al., 2012). To sum up,
the continuous increase in the number of multi-authored articles resulted in adverse effects in
ranking systems, which use indicators that are influenced by them, such as CPP, P(top 1%) and
P(top 10%), the number of highly cited articles or the number of highly cited scientists.
In the present study, we focus on analyzing the influence of multi-authored articles of various
collaborations in CERN on world university rankings. In the following section, we present our
methodology and results of the analyses.
Multi-authoring and university rankings
9
3. Methodology and Results
We developed bibliometric measures for the CERN articles, which were published in between
2011 and 2015. The data were obtained from Clarivate Analytics (formerly Thomson Reuters
Scientific) Web of ScienceTM and InCitesTM.
6
We employed the filters for institutional
affiliations provided by the publishers.
7
We focused on the last five years since it is a typical
time window employed by several international ranking systems. In the sections below, we
present the description of the sample data set, the influence of CERN articles on the HEIs, and
a close investigation of this impact in Turkish universities, as well as ranking systems.
3.1. CERN Articles and Citations
Our search identified 5661 CERN-affiliated publications (henceforth, CERN articles) published
between 2011-2015 and a total of 97,082 citations accrued by those articles. As noted above, a
salient feature of CERN publications is that they are mostly multi-authored publications. This
applies to our sample dataset, as well, where the mean number of authors were 459, with a
median value of 7. Moreover, 869 of 5661 articles had 1000 or more co-authors. One single
article had 5154 authors. This was followed by an article with 3178 authors. Figure 1 shows the
distribution of the number of authors for the CERN articles in our sample.
6
Data set was accessed on November 15, 2016, which reflects the last InCites dataset update on November
11, 2016, including the Web of Science content indexed through September 30, 2016. Since the publisher
databases are updated frequently, data for the previous years are more stable than the data for more recent
years.
7
Multi-authorship has been subject to threshold designs by data publishers, as well. For instance, Thomson-
Reuters (2014) state that “it was decided to eliminate from consideration any paper with more than 30
institutional addresses” as part of the methodology of calculating Highly Cited Researchers in 2014.
Multi-authoring and university rankings
10
Figure 1. The distribution of the number of authors for CERN articles (2011-2015) into
clusters identified by the k-means algorithm. The vertical axis shows the number of authors;
the horizontal axis shows the cluster.
A closer look at the distribution of the number of authors reveals clusters that are separated by
gaps. Visual inspection of the dot histogram suggested gaps among 5 clusters of articles. A k-
means algorithm iteratively devised the clusters shown in Figure 1. In particular, 4401 articles
are clustered in the first group with number of authors in the range [1-352]. The second, third
and fourth clusters include 394 (author range: [400-1062]), 410 (author range: [1649-2540] and
455 (author range: [2810-3178]) articles respectively. The fifth cluster included a single article
with 5154 authors, which remains to be the article with the highest number of co-authors in the
Web of Science database. These findings suggest a likely relationship between the cluster
structure of the number of authors and specific research groups in CERN.
Citations and Institutions. Among 5661 CERN articles in our sample, there are 118 articles
that accrued 100 or more citations in the past five years (until the end of data collection in Nov.,
2016, see fn. 6). One hundred or more citations in a five-year period exceed usual citation
averages in most subject areas indexed in Web of Science. For instance, the average number of
Multi-authoring and university rankings
11
five-year citations is 3.76 in biology, whereas it is 1.96 in mathematics for 2011-2015 period.
Those highly-cited CERN articles, in particular the ones with multi-authorship may have a
potential to result in a multiplied citation effect (viz. inflated impact). A closer look at the
relationship between the number of authors and the number of citations reveal that it is indeed
the case, as we report below.
The majority of the 118 CERN articles (78 of 118), which accrued 100 or more citations in the
past five years, has more than 20 co-authors. More generally, the CERN articles that generate
the highest citation impact have an average of 1331 co-authors per multi-authored paper. The
top five mostly cited articles, which exceeded 2000 citations, are co-authored by 192, 2932,
2891, 264 and 209 authors respectively. Figure 2 shows a scatterplot of the number of authors
and the citations accrued by 5661 CERN articles in the Web of Science database.
Figure 2. A scatterplot of the number of citations versus the number of authors contributed to
each CERN article. Clusters are identified through the k-means algorithm.
Multi-authoring and university rankings
12
Figure 2 shows that there are large concentrations of publications with approximately 1000,
2250 and 3000 co-authors, which accrued citations as high as 500 or more. There are two more
groups of authors, which form clusters around 300 and 750. There are also clusters with smaller
numbers of authors, below 300. Since a citation is counted once per each affiliated institute,
this picture results in an inflated citation impact of CERN articles on affiliated institutions of
the authors worldwide.
8
The inflated citation impact of CERN is not only limited to indicators typically used by
international and national university ranking systems (e.g., URAP, THE, etc.), but it also
influences widely used rank-scoring indices that are based on citation data, such as the H-index
(Hirsch, 2005). This issue is of importance, since ranking methodologies may be used for
purposes such as academic promotion of individual researchers in HEI. Below, we report an
analysis of CERN-affiliated institutions worldwide, institutions in Turkey, and how CERN-
affiliated articles influence the rankings of Turkish institutions.
CERN-Affiliated HEIs. In our sample, 5661 CERN articles are affiliated with 1046
institutions worldwide. The majority of them (852 institutions) are academic institutions. We
limited our analysis on those 852 academic institutions, since they are the primary HEIs
evaluated by international university rankings. Figure 3 shows the top 30 countries that have
the largest number of CERN-affiliated institutions, according to our sample dataset. There are
142 CERN-affiliated institutions in the US, which is followed by Italy (51), France (51),
Germany (46), Japan (42) and Turkey (41).
8
We emphasize that what we call in this article the “inflated CERN impact” does not refer to CERN’s
scientific impact, at all. Our use of the term refers to a scientometrics analysis, in particular, its cumulative
impact on ranking methodologies.
Multi-authoring and university rankings
13
Figure 3. Top 30 countries with the highest number of CERN affiliated institutions.
The number of CERN publications and the number of citations accrued by those publications
reveal a similar but not fully aligned picture compared to the number of CERN-affiliated
institutions in the countries, due to differences in research output productivity of the institutions.
Figure 4 presents two boxplots of CERN articles and their citations for the top 20 countries,
whose institutions have the highest number of CERN articles and citations. Italy, France, Spain,
Switzerland and Russia hold the top five positions in terms of the maximum number of CERN-
affiliated articles and citations respectively. However, there is variability among institutions of
all countries due to the presence of institutions with rather small number of CERN articles.
020 40 60 80 100 120 140 160
USA
ITALY
FRANCE
GERMANY
JAPAN
TURKEY
ENGLAND
SOUTH KOREA
SPAIN
INDIA
BRAZIL
CANADA
CHINA MAINLAND
POLAND
RUSSIA
ROMANIA
PORTUGAL
SWITZERLAND
AUSTRALIA
FINLAND
SOUTH AFRICA
BELGIUM
SWEDEN
NETHERLANDS
GREECE
AUSTRIA
MEXICO
CHILE
CZECH REPUBLIC
TAIWAN
Multi-authoring and university rankings
14
Figure 4. The number of CERN-affiliated articles (top) and citations (bottom) per country for
the top 20 countries, in decreasing order in number.
The numbers, in particular the number of CERN-affiliated citations comprises a significant
portion of research output worldwide. The picture becomes more salient by a closer look at the
Multi-authoring and university rankings
15
level of institutions. Figure 5 shows the top 33 institutions that have the largest number of
CERN affiliated articles (for the 2011-2015 period), sorted by the number of CERN-excluded
articles. The number column on the right of the figure shows CERN articles as a percentage of
the total number of articles published by the institution.
0500 1000 1500 2000 2500 3000 3500
Piri)Reis)Uni.
Inst.)of)Atomic)Physics)9Romania
KTO)Karatay)Uni.
Chicago)State)Uni.
Uni.)Autonoma)de)Si naloa
Cag)Uni.
Dumlupinar)Uni.
Mim ar)Si nan)Guzel)Sanat lar)Uni.
Utah)Valley)Uni.
Ozyegin )Uni .
Izmir)Yuksek)Teknoloji)Enstitusu
Mersi n)Uni.
Erzi ncan )Uni.
Helwan)Uni.
Fairfield)Uni.
Uni.)Federal)de)Sao)Joao)del9Rei
Adi yaman)Uni.
Kafkas)Uni.
Brandenburg)Uni .)of)Tech.)Cottbus
National)Centre)for)Physics)9Pakistan
California)State)Uni.)Fresno
Universite)Mohammed)Premier
Moh ammed)V)Uni.
West)Uni.)of)Timisoara
Gaziantep)Uni.
TOBB)Ekonom i)ve)Tekno loji )Uni.
Dogus)Uni.
Uni.)Autonoma)de)San)Luis)P otosi
Benemerita)Uni.)Autonoma)de)Puebla
Ivane)Javakhishvili)Tbilisi)State)Uni.
Bogazici)Uni.
H.)Hulubei)Nat.)Inst.)of)Phys.)Nucl.)Eng.
Yerevan )Ph ysi c s)In st.
CERN)Articles Remaining)Articles
!"#$%
&$#'%
()#)%
'(#*%
($#'%
(&#"%
'(#+%
&(#'%
")#)%
$&#+%
($#'%
'+#+%
$!#(%
&+#'%
()#&%
$*#"%
$!#"%
()#&%
&)#&%
($#*%
&$#!%
(+#*%
()#$%
&'#!%
*"#+%
*)#)%
",#'%
!*#*%
"!#*%
$)#)%
!$#$%
&&#"%
$(#!%
-./012%34%5267819%
:;<-%
=267819%
<1/5>?>?@%
=267819% A%34%:;<-%=267819%>?%B3B58%
Multi-authoring and university rankings
16
Figure 5. Top 33 institutions sorted by the total number of CERN-affiliated articles. The
numbers in the column show the percentage of CERN articles in the total number of articles
published by the institution.
According to Figure 5, CERN articles constitute between 17.6% and 76.6% of the total number
of articles published by the top institutions that have the largest number of CERN affiliated
articles in the 2011-2015 period. For instance, 76.7% of the total articles published by Cag
University (a young, private Turkish university, founded in 1997) are CERN-affiliated. Turkish
institutions occupy almost half of the list. An investigation of the number of citations that were
accrued by the CERN articles reveals striking results, as well. Figure 6 shows citation data for
the same institutions that have the largest number of CERN affiliated articles.
Multi-authoring and university rankings
17
Figure 6. Top 33 institutions sorted by the total number of citations from CERN affiliated
articles. The numbers in the column show the percentage of CERN citations in the total
number of citations accrued by institutions’ publications.
According to Figure 6, the citations accrued by the CERN-affiliated articles constitute between
70.4% and 97.2% of the total citations of the top institutions that have the largest number of
CERN-affiliated articles in the sample data set. For instance, 70.4% of the total citations, which
were accrued by Bogazici University’s (a Turkish state university, founded in 1971)
publications, are the citations accrued by CERN-affiliated articles.
010000 20000 30000 40000
Piri)Reis)Uni.
KTO)Karatay)Uni.
Inst.)of)Atomic)Physics)=Romania
Chicago)State)Uni.
Cag)Uni.
Uni.)Autonoma)de)Si naloa
Mim ar)Si nan)Guzel)Sanat lar)Uni.
Ozyegin )Uni .
Dumlupinar)Uni.
Utah)Valley)Uni.
Izmir)Yuksek)Teknoloji)Enstitusu
Uni.)Federal)de)Sao)Joao)del=Rei
Erzi ncan )Uni.
California)State)Uni.)Fresno
Mersi n)Uni.
Moh ammed)V)Uni.
Universite)Mohammed)Premier
West)Uni.)of)Timisoara
Helwan)Uni.
Gaziantep)Uni.
TOBB)Ekonom i)ve)Tekno loji )Uni.
Dogus)Uni.
Kafkas)Uni.
Adi yaman)Uni.
Brandenburg)Uni .)of)Tech.)Cottbus
Fairfield)Uni.
National)Centre)for)Physics)=Pakistan
Benemerita)Uni.)Autonoma)de)Puebla
Uni.)Autonoma)de)San)Luis)P otosi
Ivane)Javakhishvili)Tbilisi)State)Uni.
H.)Hulubei)Nat.)Inst.)of)Phys.)Nucl.)Eng.
Bogazici)Uni.
Yerevan )Ph ysi c s)In st.
CERN)Articles Remaining)Articles
!"#$%
&$#'%
&!#(%
!(#)%
&&#"%
&$#'%
*+#,%
!(#$%
&(#*%
**#,%
!(#,%
*)#"%
*+#(%
&"#,%
&'#$%
*(#"%
*&#)%
&$#'%
&'#,%
&&#$%
*&#!%
&)#"%
&,#"%
!+#(%
&(#,%
*+#'%
!&#$%
&$#(%
**#)%
&*#!%
*,#+%
!)#"%
&"#,%
-./012%34%567893:;%
<=>-%
<67893:;%
>1/86:6:?%
<67893:;% @%34%<=>-%<67893:;%6:%7378A%
Multi-authoring and university rankings
18
The presence of the high number of Turkish universities in the list of institutions that were most
influenced by CERN articles, as well as their citations, led us to conduct a more detailed
analysis of the Turkish universities, as a case study. The following section presents the results
of the analysis.
3.2 The Impact of CERN Articles on Turkish Universities
As of the end of 2015, there were 179 universities in Turkey, 117 of them being state
universities and the rest being foundation (aka. private) universities. In our data set there are 41
Turkish universities affiliated with CERN. We selected top 20 of them according to decreasing
number of CERN articles published between 2011 and 2015. Figure 7 shows the number of
CERN articles published by those 20 Turkish universities. The number column on the graph
shows the percentage of CERN articles in total publications for each university.
Multi-authoring and university rankings
19
Figure 7. Top 20 Turkish universities sorted by the total number of CERN affiliated articles.
The numbers in the column show the percentage of CERN articles in the total number of
articles published by the institution.
As suggested by Figure 7, the impact of the CERN articles on the number of publications
becomes more pronounced in smaller universities with a low number of publications compared
to large ones with a high number of publications. For instance, Mimar Sinan Guzel Sanatlar
University (founded in 1882) is a relatively small state university in Turkey in terms of the
number of publications, despite its high reputation as an institute of fine arts in Turkey.
Interestingly, the percentage of CERN articles in the total number of publications is 69.9%. On
the other hand, in Ege University (founded in 1955), which is a relatively large state university
in terms of the number of publications, the percentage of the CERN articles in the total number
!"!!! #!!! $!!! %!!! &!!! '!!! (!!!
)*+,*-./0123/.4
5.+0126./0/ 27*89,26423/.4
6*,9:+0/2)9+.19,23/.4
;<923/.4
=8:9<./23/.4
>8+.12?*@A9@2B9@/C,CD.2>/AE4
591A. / 23/ . 4
;18. /F0/23/.4
708.23/.4
708.CA+0/-0A023/.4
GH. :0+0/23/. 4
I0J@0A23/.4
K*@*1CL023/.4
708.0/E9-23/.4
B=MM2;B323/.4
G/@01023/.4
)C<*A23/.4
>AE0/N*,2B9@/.@23/.4
=1E02)C<*2B9@/.@23/.4
MC<08.F.23/.4
K;OP2G1E.F,9A O9+0././<2G1E.F,9A
!"#"$
%!#!$
%%#&$
'!#($
)#*$
+!#'$
%"#"$
%&#,$
&,#%$
&)#%$
!!#'$
,#!$
+&#)$
!(#,$
!"#&$
+'#)$
+#'$
"#'$
,"#"$
%*#'$
-./012$34$5267819$
:;<-$
=267819$
<1/5>?>?@$
=267819$
A$34$:;<-$=267819$>?$B3B58$
Multi-authoring and university rankings
20
of publications is 4.5%. An investigation of the institutional citations, and the percentages of
the citations accrued by CERN-affiliated articles reveals a striking picture, as shown in Figure
8.
Figure 8. Top 20 Turkish universities sorted by the total number of CERN affiliated articles.
The numbers show the percentage of CERN article citations in total number of citations of the
institution.
Figure 8 shows a considerable impact of CERN-article citations among the total number of
citations. The percentage becomes extreme for Mimar Sinan Guzel Sanatlar University, where
97.2% of its total citations are the citations accrued by CERN-affiliated articles.
010000 20000 30000 40000 50000 60000
Mimar-Sinan -Guzel-S.-Uni.
Ozyegin-Uni.
Gaziosmanpasa-Uni.
Dumlupinar-Uni.
Izmir-Yuksek-Teknoloji-Inst.
Suleyman-Demirel-Uni.
Ege-Uni.
Erzi ncan-Uni.
Gazi-Uni.
Mersi n -Un i .
Gaziantep-Uni.
TOBB-ETU-Uni.
Dogus-Uni.
Ankara-Uni.
Kafkas-Uni.
Adi yaman-Uni .
Cukurova-Uni.
Istanbul-Teknik-Uni.
Orta-Dogu-Teknik-Uni.
Bogazici-Uni.
CERN-Articles Remaining-Articles
!"#$%
&"#'%
(!#!%
&'#&%
))#(%
'*#(%
$)#'%
)+#,%
)&#*%
!,#(%
!$#(%
$)#,%
)!#'%
$"#!%
&!#*%
!(#,%
!*#(%
+!#*%
)&#$%
'!#"%
-./012%34%567893:;%
<=>-%
<67893:;%
>1/86:6:?%
<67893:;%
@%34%<=>-%<67893:;%6:%7378A%
Multi-authoring and university rankings
21
In summary, a bibliometric analysis of CERN articles reveals its impact both in terms of the
number of articles and in terms of the number of citations. The influence in terms of the
percentage of CERN articles and citations in the total number of articles citations is salient. The
CERN impact, and more generally the impact of multi-authored articles play an unintended role
in university rankings due to citation credit per affiliated institution. In the following section,
we demonstrate the impact of CERN articles in worldwide ranking systems.
3.3 The Impact of CERN articles on University Ranking Indicators
University rankings have been employing various measures that underlie their ranking
methodology (see Cakir, et al., 2015 for a review of global and national university ranking
systems). Some ranking systems employ mostly size-dependent measures (e.g., the number of
articles and citations), whereas others employ size-independent measures (e.g., the number of
articles and citations per faculty), in addition to size-dependent measures. Further indicators,
such as citation per publication (CPP), were proposed for assessing the impact of article output
per year (Li & Ho, 2008). Similarly, CWTS Leiden Ranking (Waltman, et al., 2012) has been
employing impact indicators, which consist of P(top 1%) and P(top 10%), where P specifies
the highest number of publications with a certain property. For instance, P(top 1%) is an
indicator that aims at measuring a university’s publications that belong to the top 1% most
frequently cited publications in the same field and in the same year.
9
An analysis of the impact of the CERN articles on measures such as CPP and P-indicators
revealed that in our sample dataset, 264 of 5661 CERN publications were classified as “highly
9
CWTS Leiden Ranking Indicators http://www.leidenranking.com/information/indicators, retrieved on
August 29, 2017.
Multi-authoring and university rankings
22
cited” by the Web of Science database. These publications accrued a CPP value of 17.15, a
value much higher compared to the CPP values in other fields of research. The impact of CERN
articles is evident in the impact indicators, as well. Figure 9 shows the percentages of cited
documents, highly cited papers, papers in Top 10% and papers in Top 1% in CERN publications
on a yearly basis.
Figure 9. The impact indicators for the CERN publications, based on the percentage of
CERN articles in the sample data set. The vertical axis shows the ratio of the CERN
documents as a percentage of total publications in the data set.
As shown in Figure 9, more than 80% of the CERN articles were cited at least once since the
past five years, except for the 2015. Roughly 30% of all CERN articles are in Top 10% list and
10% of them are in Top 1% list. These findings suggest that CERN articles have the potential
to have significant impact on indicators like CPP, Top 1%, Top 10% and other impact
0
10
20
30
40
50
60
70
80
90
100
2011 2012 2013 2014 2015
% Documents in Top 1% % Documents in Top 10%
% Highly Cited Papers % Docs Cited
Multi-authoring and university rankings
23
indicators. Table 1 shows top 33 institutions worldwide that CERN affiliated articles play a
major role on citation data (cf. Figure 6).
Table 1: The CPP values for the top 33 institutions worldwide that CERN affiliated articles
play a major role on citation data.
CERN Affiliated Organization
CPP
CERN
articles
CPP
Other
articles
Total
CPP
Inst. of Atomic Physics - Romania
52.57
7.63
27.46
Yerevan Physics Inst.
33.00
8.88
26.08
Utah Valley Uni.
40.24
3.26
25.81
Mimar Sinan Guzel Sanatlar Uni.
34.74
2.29
24.98
Fairfield Uni.
36.45
3.45
19.74
KTO Karatay Uni.
24.78
2.83
18.91
Ivane Javakhishvili Tbilisi State Uni.
32.56
3.32
18.69
Dogus Uni.
30.17
5.30
18.23
Ozyegin Uni.
33.27
4.83
17.83
Universite Mohammed Premier
31.04
4.40
17.72
Erzincan Uni.
35.17
3.76
17.48
Chicago State Uni.
33.62
5.99
17.02
Cag Uni.
19.65
8.25
16.99
National Centre for Physics - Pakistan
35.44
4.14
16.82
H. Hulubei Nat. Inst. of Phys. Nucl. Eng.
29.30
5.95
16.11
TOBB Ekonomi ve Teknoloji Uni.
30.39
3.93
15.17
Adiyaman Uni.
35.57
2.75
14.91
California State Uni. Fresno
30.59
5.35
14.74
Brandenburg Uni. of Tech. Cottbus
35.11
5.74
14.37
Kafkas Uni.
35.54
1.88
14.05
Bogazici Uni.
32.63
5.33
13.50
Izmir Yuksek Teknoloji Enstitusu
34.02
4.96
13.47
West Uni. of Timisoara
30.77
3.71
12.90
Uni. Autonoma de San Luis Potosi
40.15
3.81
12.55
Benemerita Uni. Autonoma de Puebla
36.45
4.29
11.84
Helwan Uni.
36.16
4.08
11.64
Uni. Federal de Sao Joao del-Rei
30.07
3.94
11.62
Dumlupinar Uni.
44.78
4.07
11.58
Mersin Uni.
36.88
3.48
10.36
Mohammed V Uni.
30.95
3.66
10.08
Uni. Autonoma de Sinaloa
40.25
3.33
9.81
Gaziantep Uni.
30.57
3.19
8.65
Piri Reis Uni.
15.11
3.08
7.02
Multi-authoring and university rankings
24
Table 1 reveals the significant difference between the CPP values of the CERN articles
published by the institutions and the CPP values of the other articles published by the same
institutions. Since Shapiro-Wilk tests indicated CPP distributions for CERN (S-W(33)=.92,
p<.05) and other articles (S-W(33)=.90, p<.01) were significantly non-normal, a non-
parametric Wilcoxon signed ranks test was employed to test if this difference reached statistical
significance. The results suggest that the CPP of CERN articles (Median=33.62) is significantly
higher than the CPP of other articles (Median=3.94), z=5.01, p<.001.
Table 2 shows a similar picture for the 20 Turkish universities that were influenced by CERN-
affiliated articles in terms of citation data (cf. Figure 8). Since Shapiro-Wilk tests indicated that
CERN articles (S-W(20)=.93, p>.05) and other articles (S-W(20)=.95, p>.05) have
approximately normal distributions, a dependent t-test was conducted to compare the two
groups of articles in terms of their CPPs. The test indicated that the CPP values of CERN articles
(M=35.31, SD=5.0) are significantly higher than the CPP of other articles (M=3.95, SD=.97)
published by Turkish universities, t(19)=27.37, p<.001.
Table 2: CPP values of the 20 Turkish universities that were influenced by CERN-affiliated
articles in terms of citation data.
CERN Affiliated Organization
CPP
CERN
articles
CPP
Other
articles
Total
CPP
Mimar Sinan Guzel S. Uni.
34.74
2.29
24.98
Dogus Uni.
30.17
5.30
18.23
Ozyegin Uni.
33.27
4.83
17.83
Erzincan Uni.
35.17
3.76
17.48
TOBB ETU Uni.
30.39
3.93
15.17
Adiyaman Uni.
35.57
2.75
14.91
Kafkas Uni.
35.54
1.88
14.05
Bogazici Uni.
32.63
5.33
13.50
Izmir Yuksek Teknoloji Inst.
34.02
4.96
13.47
Dumlupinar Uni.
44.78
4.07
11.58
Mersin Uni.
36.88
3.48
10.36
Istanbul Teknik Uni.
38.01
5.30
8.99
Orta Dogu Teknik Uni.
38.82
4.78
8.93
Multi-authoring and university rankings
25
Gaziosmanpasa Uni.
26.21
3.75
8.81
Gaziantep Uni.
30.57
3.19
8.65
Cukurova Uni.
35.48
3.79
8.09
Suleyman Demirel Uni.
44.49
3.49
7.37
Ankara Uni.
30.54
4.08
6.15
Ege Uni.
44.63
4.29
6.09
Gazi Uni.
34.36
3.72
5.63
The influence of multi-authored articles has become more salient in university rankings that
employ indicators such as CPP, leading to an unstable picture of the rankings in the past several
years. A closer look into the ranking positions of Turkish universities exemplify the situation.
For this, we selected Times Higher Education (THE) ranking system, which employs CPP, and
four Turkish universities (Orta Dogu Teknik Uni., Bogazici Uni., Istanbul Teknik Uni., and
Bilkent Uni.) that have exhibited a continuous appearance in the THE ranking system each year
since 2012, without interruption. Orta Dogu Teknik Uni. and Bogazici Uni. are also among the
top 20 Turkish universities that were subject to high CERN citation impact (cf. Figure 8 and
Table 2).
Table 3: Times Higher Education (THE) ranking position of six Turkish universities between
2012-2017.
Orta Dogu
Teknik Uni.
Bogazici
Uni.
Istanbul
Teknik Uni.
Bilkent Uni.
2012
276-300
301-350
276-300
201-225
2013
201-225
276-300
276-300
226-250
2014
201-225
199
201-225
226-250
2015
85
139
165
201-225
2016
501-600
501-600
501-600
351-400
2017
601-800
401-500
501-600
351-400
The breaking point is 2015-2016, when a major change in the ranking position took place for
the three universities (Orta Dogu Teknik Uni., Bogazici Uni., and Istanbul Teknik Uni.,) that
experienced the largest influenced by CERN. Bilkent University, which was not among the top
20 institutions influenced by CERN impact, exhibited a smaller change in the rankings. Such
Multi-authoring and university rankings
26
large leaps from one year to another are not expected from a ranking system, since it is unusual
to observe sudden changes in the institutional measures that specify ranking indicator values.
Therefore, it is likely that unexpected changes in the rankings are due to changes in ranking
methodology or due to the methodological changes in data providers.
4. Conclusion
In the 20th Century, the research conducted in CERN has been a breakthrough in science.
Groundbreaking experiments of CERN researchers have been giving answers to major
scientific questions of our time, such as the CMS/ATLAS experiments (ATLAS Collaboration,
2012; CMS Collaboration, 2012) that provided evidence for the existence of the Higgs boson,
a problem that were awaiting a solution since 1962. The worldwide reputation of CERN has
resulted in positive impact on the reputation of its researchers, making them valuable candidates
for recruitment as faculty members in the universities worldwide. Independent of its impact at
a bibliometric level, we emphasize that CERN and its affiliated researchers have been one of
the most significant players not only in the science of the 20th Century, but also in the history
of science.
In the present study, we conceive CERN as a case that has been implementing a novel
publication policy. This novel approach to publication favors multi-authored publications in an
unusual way, by supporting publications that involve hundreds and thousands of authors. The
scientific success of CERN may not be overshadowed by the publication policy by any means.
We propose that the impact of multi-authored articles has been going beyond their scientific
domain of research due to its large influence on institutional-level bibliometrics. This influence
has a significant impact on ranking systems and their indicators.
Multi-authoring and university rankings
27
Ranking systems have societal influence. They suffer from any major change in the positions
of institutions in the ranking from a year to the other, given that institutional measures rarely
exhibit sudden changes. To overcome this problem, several ranking systems, such as THE, QS
and URAP, have been taking into account the multi-authored publication policy in their ranking
methodology by employing various thresholds since the past few years. For instance, some
exclude the articles that exceeds a threshold for the number of authors (or the number of
institutions) from the analyses. This approach has been adopted by data providers, as well. For
instance, the Highly Cited Researchers ranking of Clarivate AnalyticsTM employs a threshold
of 30 institutions in the Physics category, and eliminate the papers with a higher number of
institutions from the ranking data, which is even more restrictive than eliminating authors. Their
justification for applying the threshold is to “remove the problem of overweighting to high-
energy physics”
10
. In other words, a significant part of CERN’s academic output has been
eliminated, since the past few years, despite their high potential for contribution into scientific
research. Accordingly, the “inflated” multi-authored approach has put its researchers into a
disadvantageous situation in ranking systems. These ranking systems do not only include
university ranking institutions but also in-house academic performance evaluation systems,
such as the evaluation of faculty promotions.
Our findings indicate that the size of the CERN impact will become larger as more HEIs are
involved in the rankings, since highly cited multi-authored articles will lead to instabilities in
the rankings. The impact of CERN articles upon the ranking of small universities, which publish
about few hundred articles per year, is striking compared to its impact upon the ranking of
universities that publish several thousand articles per year. We note that the impact of multi-
authored articles is not bound to ranking systems. The inflated multi-authorship is a first-time
10
Clarivate Analytics Highly Cited Researchers Methodology
http://hcr.stateofinnovation.thomsonreuters.com/page/methodology, retrieved on December 18, 2016.
Multi-authoring and university rankings
28
event in the history of science. All the stakeholders, including both the ranking systems and
HEIs should address this topic to design their future strategies for further development and
growth.
Multi-authoring and university rankings
29
References
Aad, G., Abajyan, T., Abbott, B., Abdallah, J., Khalek, S. A., Abdelalim, A. A., et al. (2012).
Observation of a new particle in the search for the Standard Model Higgs boson with the
ATLAS detector at the LHC. Physics Letters B, 716(1), 1-29.
Aad, G., Abat, E., Abbott, B., Abdallah, J., Abdelalim, A. A., Abdesselam, A., et al. (2010).
Charged-particle multiplicities in pp interactions at root s=900 GeV measured with the
ATLAS detector at the LHC ATLAS Collaboration, Physics Letters B, 688(1), 21-42.
Aad, G.; Abbott, B.; Abdallah, J.; Abdelalim, A. A., et al. (2011). Search for quark contact
interactions in dijet angular distributions in pp collisions at root s=7 TeV measured with the
ATLAS detector. Physics Letters. Section B: Nuclear, Elementary Particle and High-Energy
Physics, 694(4-5), 327 - 345.
Aad, G., Abbott, B., Abdallah, J., Abdinov, O., Aben, R., Abolins, M., et al. (2015). Combined
Measurement of the Higgs Boson Mass in p p Collisions at s= 7 and 8 TeV with the ATLAS
and CMS Experiments. Physical Review Letters, 114(19), 191803.
Acosta, D., et al. (CDF II Collaboration) (2003). Measurement of the mass difference
M(D(s)(+))-m(D(+)) at CDF II, Physical Review D, 68(7), No 072004.
Alasehir, O., Cakir, M. P., Acarturk, C., Baykal, N., & Akbulut, U. (2014). URAP-TR: a
national ranking for Turkish universities based on academic performance. Scientometrics,
101(1), 159–178. doi:10.1007/s11192-11014-11333-11194.
Attebury, G., Avdeeva, E., Bloom, K. A., Bockelman, B., et al. (2012). Observation of a new
boson at a mass of 125 GeV with the CMS experiment at the LHC, Physics Letters B 716(1),
30-61.
Aubert, B., et al. (2002). The BABAR Detector, Nuclear Instruments and Methods in Physics
Research Section A, 479(1), 1-116.
Multi-authoring and university rankings
30
Bayatian, G.L., et al. (2007). CMS physic technical design report, volume II: Physics
performance, Journal of Physics G: Nuclear and Particle Physics, 6, 995-1579.
Beaver, D. deB. & Rosen, R. (1978). Studies in scientific collaboration Part I. The professional
origins of scientific coauthorship, Scientometrics, 1(1), 65-84.
Beaver, D. deB. & Rosen, R. (1979a). Studies in scientific collaboration Part III.
Professionalization and the natural history of scientific coauthorship, Scientometrics, 1(3),
231-245.
Beaver, D. deB. & Rosen, R. (1979b). Studies in scientific collaboration Part II. Scientific
coauthorship, research productivity and visibility in the French scientific elite, 1799-1830,
Scientometrics, 1(2), 133-149.
Biagioli, M. (2003). Rights or rewards? Changing frameworks of scientific authorship. In M.
Biagioli & P. Galison (Eds.) Scientific authorship: Credit and intellectual property in
science (pp. 253-280), Routledge.
Bowden, R. (2000). Fantasy higher education: University and college league tables. Quality in
Higher Education, 6(1), 41–60.
Bowman, N. A., & Bastedo, M. N. (2009). Getting on the front page: Organizational reputation,
status signals, and the impact of US News and World Report on student decisions. Research
in Higher Education, 50(5), 415–436.
Cakir, M. P., Acarturk, C., Alasehir, O., & Cilingir, C. (2015). A comparative analysis of global
and national university ranking systems. Scientometrics, 103, 813-848, doi:
10.1007/s11192-015-1586-6.
Cattell, J. M. (1906a). American men of science. Science Press.
Cattell, J. M. (1906b). A statistical study of American men of science III. Science, 24(623), 732–
742.
Multi-authoring and university rankings
31
Chatrchyan, S., et al. (2008). The CMS experiment at the CERN LHC, Journal of
Instrumentation, 3, No. S08004.
Cheng, S. K. (2011). World university rankings: Take with a large pinch of salt. European
Journal of Higher Education, 1(4), 369-381.
Crase, D., & Rosato, F. D. (1992). Single versus multiple authorship in professional journals.
Journal of Physical Education, Recreation & Dance, 63(7), 28-32.
Diamond, A. M. (1985). The money value of citations to single-authored and multiple-authored
articles. Scientometrics 8(5-6), 315-320.
Ellis, H. (1904). A study of British genius. London: Hurst and Blackett.
Galton, F. (1875). English men of science: Their nature and nurture. D. Appleton.
Griffith, A., & Rask, K. (2007). The influence of the US News and World Report collegiate
rankings on the matriculation decision of high-ability students: 1995–2004. Economics of
Education Review, 26(2), 244–255.
GUSTO Investigators (1993). An international randomized trial comparing four thrombolytic
strategies for acute myocardial infarction, The New England Journal of Medicine, 329(10),
673-682.
Hazelkorn, E. (2007). The impact of league tables and ranking system on higher education
decision making. Higher Education Management and Policy, 19, 1–24.
Hazelkorn, E. (2008). Learning to live with league tables and ranking: The experience of
institutional leaders. Higher Education Policy, 21, 193–215.
Hirsch, J.E. (2005). An index to quantify an individual’s scientific research output. Proceedings
of the National Academy of Sciences, USA, 102(46), 16569–16572.
Hughes, R. M. (1925). A study of the graduate schools of America. Miami University.
Hyland, K. (2015). Academic Publishing: Issues and Challenges in the Construction of
Knowledge. Oxford University Press.
Multi-authoring and university rankings
32
Knobel, M., Simões, T. P., & Cruz, H. B. C. (2013). International collaborations between
research universities: experiences and best practices. Studies in Higher Education, 38(3),
405-424.
Li, F., Yi, Y., Guo, X., & Qi, W. (2012). Performance evaluation of research universities in
Mainland China, Hong Kong and Taiwan: Based on a two-dimensional approach.
Scientometrics, 90(2), 531–542, doi:10.1007/s11192-011-0544-1.
Li, Z. & Ho, Y.S. (2008). Use of citation per publication as an indicator to evaluate contingent
valuation research, Scientometrics, 75(1), 97-110.
Liu, N. C., & Liu, L. (2005). University rankings in China. Higher Education in Europe, 30(2),
217–227.
Maclean, A. H. H. (1900). Where we get our best men: Some statistics showing their
nationalities, counties, towns, schools, universities, and other antecedents, 1837-1897.
Simpking, Marshall, Hamilton.
Nakamura, H., et al. (2004). Design and baseline characteristics of a study of primary
prevention of coronary events with pravastatin among Japanese with mildly elevated
cholesterol levels, Circulation Journal, 68(9), 860-867.
Price, D. J. de Solla (1963). Little science, big-science. Columbia University Press.
Rousseau, R. (2001). Are multi-authored articles cited more than single-authored ones? Are
collaborations with authors from other countries more cited than collaborations within the
country? A case study. In Havemann, F. & Wagner-Döbler, R. & Kretschmer, H. (Ed.)
Proceedings of the second Berlin workshop on scientometrics and informetrics. (pp. 173-
176). Berlin: Gesellschaft für Wissenschaftsforschung.
Schael, S., et al. (2006). Precision electroweak measurements on the Z resonance, Physics
Reports, 427(5-6), 257-454.
Multi-authoring and university rankings
33
Stolz, I., Hendel, D. D., & Horn, A. S. (2010). Ranking of rankings: Benchmarking twenty-five
higher education ranking systems in Europe. Higher Education, 60(5), 507–528.
Thompson-Reuters (2014). Details of method used to select Highly Cited Researchers in 21
fields of the Sciences and Social Sciences (2002-12), retrieved on August 29, 2017, URL
http://community.thomsonreuters.com/t5/InCites-Customer-
Forum/Preliminarypublication-of-new-lists-of-Highly-Cited-Researchers/td-
p/36685/page/3/
Usher, A., & Savino, M. (2007). A global survey of university ranking and league tables.
Higher Education in Europe, 32(1), 5–15.
Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E.C.M., Tijssen, R.J.W., Van Eck, N.J.,
Van Leeuwen, T.N., Van Raan, A.F.J., Visser, M.S., & Wouters, P. (2012). The Leiden
Ranking 2011/2012: Data collection, indicators, and interpretation. Journal of the American
Society for Information Science and Technology, 63(12), 2419–2432.
Wegner, E. L. (1967). The relationship of college characteristics to graduation. Madison:
University of Wisconsin–Madison.
Woods, R. H., Youn, H., & Johanson, M. M. (2010). Single vs. Co-Authored and Multi-
Authored Research Articles: Evaluating the views and opinions of ICHRIE Scholars.
International CHRIE Conference-Refereed Track, 2010 ICHRIE Conference.
