ArticlePDF Available

Abstract and Figures

In November 2012 the Google Scholar Metrics (GSM) journal rankings were updated, making it possible to compare bibliometric indicators in the 10 languages indexed and their stability with the April 2012 version. The h-index and h 5 median of 1000 journals were analysed, comparing their averages, maximum and minimum values and the correlation coefficient within rankings. The bibliometric figures grew significantly. In just seven and a half months the h index of the journals increased by 15% and the median h-index by 17%. This growth was observed for all the bibliometric indicators analysed and for practically every journal. However, we found significant differences in growth rates depending on the language in which the journal is published. Moreover, the journal rankings seem to be stable between April and November, reinforcing the credibility of the data held by Google Scholar and the reliability of the GSM journal rankings, despite the uncontrolled growth of Google Scholar. Based on the findings of this study we suggest, firstly, that Google should upgrade its rankings at least semiannually and, secondly, that the results should be displayed in each ranking proportionally to the number of journals indexed by language
Content may be subject to copyright.
Paper accepted for publication in the Scientometrics
Google Scholar Metrics evolution: an analysis according to languages
Enrique Orduña-Malea
and Emilio Delgado López-Cózar
EC3 Research Group, Universidad Politécnica de Valencia. Camino de Vera s/n, Valencia 46022, Spain.
EC3 Research Group, Universidad de Granada, 18071 Granada, Spain
Abstract In November 2012 the Google Scholar Metrics (GSM) journal rankings were updated, making
it possible to compare bibliometric indicators in the 10 languages indexed and their stability with the
April 2012 version. The h-index and h-5 median of 1,000 journals were analysed, comparing their
averages, maximum and minimum values and the correlation coefficient within rankings. The
bibliometric figures grew significantly. In just seven and a half months the h-index of the journals
increased by 15% and the median h-index by 17%. This growth was observed for all the bibliometric
indicators analysed and for practically every journal. However, we found significant differences in growth
rates depending on the language in which the journal is published. Moreover, the journal rankings seem to
be stable between April and November, reinforcing the credibility of the data held by Google Scholar and
the reliability of the GSM journal rankings, despite the uncontrolled growth of Google Scholar. Based on
the findings of this study we suggest, firstly, that Google should upgrade its rankings at least semi-
annually and, secondly, that the results should be displayed in each ranking proportionally to the number
of journals indexed by language.
Keywords Google Scholar Metrics, Google Scholar, Scientific Journals, h-index, Journal Rankings,
Bibliometric Databases
1. Introduction
Google Scholar Metrics (GSM)
is a free product launched in April 2012 by Google that
provides bibliometric information on a wide range of scholarly journals,
as well as
other published material, such as conference articles and repositories.
The selection of journals is based on publications that are indexed in Google Scholar
(excluding dissertations, books and patents), have published at least 100 articles over a
period of five years, and have received at least one citation in that time.
These criteria
represent an effort (albeit occasionally automatic and crude) to filter raw data from the
journals indexed in Google Scholar (GS) and which, together with Google Scholar
Citations (Googles product for the creation of researcher profiles) makes up Googles
array of academic information tools today.
The features of this new product, its services, limitations and potential uses as a tool for
the evaluation of academic activity, have been analysed in previous studies (Delgado-
López-Cózar and Cabezas-Clavijo 2013), in which certain strengths (ease of use,
coverage, free-of-charge, etc.) are highlighted, which are more an asset of the features
of Google technology than of the product itself, which favour its use as a
complementary source. Furthermore, the draw of the Google brand and the convenience
of not having to install additional software point to a potential large-scale use of this
product by various different stakeholders from the academic world (Jacsó 2012).
These previous analyses conclude that the GSM tool is still a product in its infancy with
multiple limitations and errors, so that its use in evaluation processes is discouraged at
present (Delgado-López-Cózar and Cabezas-Clavijo 2012). The recent appearance of
this product means that there are many aspects to be investigated, in particular, those
related to the reliability and validity of journal rankings based on the h-index. One of
Google Scholar Metrics evolution: an analysis according to languages
the aspects that is yet to be tested, given the rapid growth of GS, is the stability and
variation of indicators over time.
In November 2012, GSM updated journal rankings published in the first version of
April 2012 (fig. 1), extending the citation window up to 15 November of that year (fig.
2). Although there is no information on the product’s website about any policy of
regular content updates, this second version in the same year made it possible to take a
quick and easy look at changes in the various journal rankings and, especially, at the
growth and stability of the rankings. It therefore admitted, for the first time, longitudinal
Figure. 1. Google Scholar Metrics (April 2012 version)
Figure. 2. Google Scholar Metrics (November 2012 version)
Since GSM is fed by GS data, a study of the former must be contextualised against the
of the latter. In fact, any study on the evolution of GS (or any other web data source)
must also be aligned with studies related to the dynamism and evolution of search
engines and, in general terms, of the Web (Brewington 2000), based on research into the
durability and stability of online resources over time as well as the control variables.
In this regard, web resources can remain stable, change, disappear and reappear
(intermittence effect), modified or not (which explains how the number of records in GS
can fall over time). Studies worth mentioning in this area are those by Koehler (2002;
2004), Cho and García-Molina (2003) and Fetterly et al. (2003). In academic study
environments, mention should also be made of the research by Ortega et al. (2006) and
Payne and Thelwall (2007), among others.
If there are few longitudinal studies on the evolution of GS over time, those focused on
measuring changes in various bibliometric values (h-index, total number of citations,
etc.), as calculated from this database, are virtually nonexistent; as are comparisons with
the rankings offered by traditional bibliometric journal evaluation systems (WoS and
Although studies focusing on the coverage of search engines have demonstrated high
variability and irregularity in their academic content updates (Orduña-Malea et al.
2010), GS is more stable and it is estimated that its size increases every two weeks
(Aguillo 2012). Even so, the growth of GS is subject to greater instability than that of
WoS or Scopus, since it is more vulnerable to certain changes in the Google search
engine. Orduña-Malea et al. (2009) illustrated this in their study of the monthly
evolution in the size of Spanish public university websites in GS, identifying stable and
slightly positive trends during the months studied, with the exception of occasional
measurements in which Google coverage changes significantly. In any case, knowledge
and understanding of the growth of GS is essential to properly interpret bibliometric
indicators produced on consultation of its records.
Another interesting line of research is the analysis of the growth of the number of
citations collected by GS over time. Kousha and Thelwall (2007) collected citations
received by a sample of 880 articles (39 open access journals listed in WoS) from GS in
two different time periods (October 2005 and January 2006), observing significant
citation growth patterns, although this was dependent on the area of knowledge.
Google Scholar Metrics evolution: an analysis according to languages
Chen (2010) compared citations obtained from GS for a set of articles from different
databases with citations previously obtained by Neuhaus et al. (2006) for the same
sample, revealing a significant increase in the coverage of GS.
Finally, Winter et al. (2013) also analysed the citations retrieved on GS, in this case for
a chosen set of 56 articles in various fields of knowledge, and compared them with
citations retrieved on WoS. The novelty of this research lies in the calculation, not only
of current growth, but also of retroactive growth, i.e. it studied the citations retrieved for
an item at two different moments in time (x and y), but considered, for both
samples, only the citations received up to date x. Thus, it is possible to ascertain the
growth of GS in literature looked at retrospectively. The results show that retroactive
growth for GS is substantially higher than for WoS (GS median of 170% versus 2% for
WoS); current growth is also higher, although to a lesser extent (GS median 54% versus
41% for WoS).
Recently, Harzing (2013) performed another GS-based longitudinal study, in order to
test the stability of the coverage of this database over time and its direct effect on the
calculation of bibliometric indicators. In this case, the analysis focused on researchers,
but on a fairly small sample (20 Nobel science laureates), and limited to only four
disciplines (physics, chemistry, medicine, and mathematics). For his study, he collected
the total citations received by these authors at three moments in time (April 2011,
September 2011 and January 2012), and calculated the h-index for each author at each
moment. The main outcome of this study is that the citations increase by a ratio of
approximately 3% per month, although growth is unequal, depending on the area of
knowledge. However, the growth of the h-index is moderate (except in chemistry,
where the coverage grows more sharply), and furthermore, the average h-index in
Scholar for each researcher in the areas of physics and medicine is very similar to
calculations for WoS.
Harzings study is extremely important because it indicates, in summary, that GS is
growing rapidly but that the h-index continues to be more stable, which is precisely one
of the advantages of using this indicator (Costas and Bordons, 2007). Moreover, it
obtains similar values to classical bibliographic databases (Scopus and WoS).
The GSM update is therefore an opportunity to expand on and confirm Harzings
findings in a larger sample, and for journals. In addition, GSM has a special feature
whereby it provides a ranking of journals based on the 10 most representative languages
in the world, a resource unusual in bibliometrics (Delgado López-Cózar and Cabezas-
Clavijo, 2013). Therefore, the study of its evolution between the April and November
versions may be contextualised in relation to language, a novel aspect in this type of
1.1. Objectives
Given the recent apparition of GSM, no other longitudinal study of this nature has been
published to date. This paper therefore aims to answer the following questions:
a) Have there been changes in the values of the bibliometric indicators adopted by
Google Scholar Metrics between April and November? Can a journals h-index
Google Scholar Metrics evolution: an analysis according to languages
change in just seven months and a half? And if so, what is the volume and size of the
b) Are there significant differences in the h-index for rankings by different
c) Do these changes affect the positions of the journals in the rankings? In other
words, is there stability in the rankings?
Thus, the specific objectives of this study are:
1. Measure the differences in the bibliometric indicators of the 10 journal rankings
provided by Google Scholar Metrics, according to language, between April and
November 2012.
2. Compare the stability of the rankings at these two moments in time.
3. Contextualise the results according to coverage and overall growth of GS as the
GSM data source.
2. Methodology
In order to meet the objectives and to respond to the questions raised above, we
designed a prospective, longitudinal, descriptive analysis of GSM.
For this purpose, we took a sample of the 1,000 journals listed in the rankings provided
by GSM in April and November 2012 by language, which were as follows: English,
Chinese, Portuguese, German, Spanish, French, Korean, Japanese, Dutch and Italian.
For each of the journals, and in each of the languages, the h-index and the median h-
index (h-median) were obtained for both April and November. The data were then
transferred to a spreadsheet where h-index and h-median averages were calculated, as
well as maximum and minimum values, both overall and per language. Finally, the
Spearman correlation coefficient of the journal rankings was calculated by language for
the two versions (April and November). This process took place during the month of
December 2012.
The bibliometric indicators were retrieved just as they were provided by GSM, in this
case with a citation window of 5 years (h5-index and h5-median):
- The h-index of a publication is the largest number h such that at least h articles in
that publication were cited at least h times each.
- The h-median of a publication is the median of the citation counts in its h-core (the
articles that the h-index is based on).
- Finally, the h5-index and h5-median of a publication are, respectively, the h-index
and h-median of only those of its articles that were published in the last five
complete calendar years.
It should be noted that GSM does not keep a file of journal positions and data from the
previous version. In other words, the November 2012 update deleted the information
from the first April 2012 version (Delgado López-Cózar and Cabezas-Clavijo, 2013).
For the purposes of this study, data from the first version were captured prior to their
deletion, giving added value to the analysis and comparison with November data.
Google Scholar Metrics evolution: an analysis according to languages
Since the growth of GSM relies on GS, we then conducted an additional experiment in
which we collected data on GS coverage (measured in number of records), in order to
contextualise the evolution of GSM. To match this with the GSM rankings by language,
data on overall size were collected from the geographic domains of the main countries
whose official language is one of the 10 considered by GSM.
This methodology for calculating the size of GS through cybermetric indicators has
already been employed by Aguillo (2012). To compare the growth of GS with the other
bibliometric databases, the number of records for the same countries was retrieved from
WoS and Scopus. The data for the three databases were collected on a weekly basis
during the month of March 2013 (four samples) and in November 2012 (with the
release of the second version of GSM). Both total records and records from 2003
onwards were retrieved, to ascertain the rate of growth for contemporary literature.
The countries considered and queries performed for each of the three databases are
shown in Table 1. In the case of the United States, GS measurement was performed by
adding the results to the domains .edu, .gov, .mil, and .us. Despite the limitations of this
procedure (overlapping records and use of the domain .edu in other countries), this is
the only process for retrieving geographic web data for the US.
Table 1. Countries, sources and queries performed
site:edu +site:gov
+site:mil +site:us
AFFILCOUNTRY(united states)
AD= (united states)
AFFILCOUNTRY(united kingdom)
AD= (uk) OR AD= (england)
OR AD= (united kingdom)
AD= (italy)
AD= (france)
AD= (germany)
AD= (netherlands)
AD= (spain)
AD= (brazil)
AD= (japan)
AFFILCOUNTRY(south korea)
AD= (south korea)
AD= (china)
This procedure is not intended to consider the GS / WoS-Scopus isomorphism, since
cybermetric indicators applied to GS retrieve the records deposited in the domain of
each country, which will not necessarily be the output of that country (although the
majority undoubtedly is). In addition, there are countries that have not been considered
for the most important languages, like English (Australia), Spanish (all of South
America), German (Austria) or French (Canada, Africa, etc.). However, this simple
method is effective for determining, in a simple exploratory manner, the growth of GS
as backdrop for GSM, and how it compares with WoS and Scopus.
3. Results
First, results are provided for the growth of the two main bibliometric indicators (h-
index and h-median) between April and November 2012. Then detailed information is
Google Scholar Metrics evolution: an analysis according to languages
given on growth data according to the GSM language ranking and ranking correlations
retrieved from both version for each of the languages analysed.
Finally, GS, Scopus and WoS growth data are given (total and 2003 onwards) for the
selected countries in order to compare and contextualise journal rankings, according to
language, that have been the object of previous studies.
3.1. Growth of indicators: h-index and h-median
From the analysis of the 1,000 scholarly journals displayed in the 10 GSM rankings,
significant growth is observed in the bibliometric values adopted by Google to measure
academic impact. Table 2 shows the results for the h-index and h-median (average,
maximum and minimum values in both cases) for April and November .
Table 2. Bibliometric indicators of journal rankings published by GSM
April 2012
Growth rate (%)
h-index (average)
h-index (maximum)
h-index (mimimum)
h-median (average)
h-median (maximum)
h-median (minimum)
As seen in Table 2, the h-index average for the 1,000 journals has grown from 20 to 23
(15.4%) in less than eight months, while the h-median has a slightly higher growth rate
(17%). This increase in the indicators also occurs in the other parameters considered
(minimum and maximum values of both the h-index and the h-median). There is,
however, a notable increase in the minimum values, 24.9% in the case of the h-median,
a figure considered very significant given the limited time period in which it is
3.2. Ranking according to language: differences and growth
After analysing overall indicator values, Table 3 presents the results obtained for the
journal sets published in each language. Here it is apparent how the bibliometric
indicators for journals in English have significantly higher values than those for the
other languages, five times the average h-index values obtained for the language ranked
second (Chinese), while Portuguese and Spanish occupy third and fourth place in both
samples, a certain distance behind Chinese.
Table 3. Bibliometric indicators of journal rankings published by GSM according to language
Growth rate (%)
Growth rate (%)
Google Scholar Metrics evolution: an analysis according to languages
The fact that Chinese journals prominently occupy second place in the h-index average
journal ranking is not surprising, given the large amount of current scholarly production
in this language (as discussed later). Conversely, the high position of journals in
Portuguese is striking, higher than for those published in Spanish (the differences
remain constant in November), considering that the size of the Spanish-speaking
scientific community is greater than the Portuguese. Also of note are the low values for
French, which ranks sixth.
As for the differences between April and November, there is an increase in h-index
across nearly all the rankings by language. Growth rates are highest for journals in
Italian (33.3%) and Portuguese (21.4%), while the lowest are for Japanese and Dutch,
which are the only languages with a constant h-index (5 and 2 respectively).
In any case, these growth rates should be approached with some caution, due to the
nonlinearity of the growth in the h-index. The h-index for Italian-language journals
increases from 3 to 4 (representing a 33.3% growth). The value of the h-index for other
languages has also increased by one point, for example Korean (from 5 to 6) and French
(from 7 to 8), yet their growth rates are lower since the index is higher (20% and 14%
respectively). This is especially significant as it is more difficult for an h-index to
increase from 7 to 8 than it is to go from 3 to 4.
If we focus on this issue, Table 3 then shows how the journals in languages that already
had high values in April are those that most increase their h-index values in November
(in whole integers): English increases 7 points, Chinese 4, Portuguese 3, and Spanish
and German 2.
3.3. Correlation analysis: stability of the rankings
From the point of view of the bibliometric evaluation of journals, one of the key aspects
in determining the reliability and validity of the GSM rankings is their stability over
time. In this respect, both the coverage and growth of GS, on the one hand, and the
robustness and progressiveness of the h-index, on the other, are established as key
variables of analysis, as noted in the introduction.
Table 4 shows the correlation coefficients (Spearman) between GSM journal rankings
in April and November for the following languages:
Table 4. Correlation coefficients (Spearman) between GSM editions (April and November)
Google Scholar Metrics evolution: an analysis according to languages
As the data in Table 4 show, there is a very high correlation between the journal
rankings of April and November. They also show how this correlation coefficient is
slightly dependent on the language of the journals and their h-index, i.e. the correlation
decreases practically to the same extent that the values of the h-index are of a lesser
scale. This is a logical trend because, as discussed above, the h-index is more sensitive
to errors the lower its value, becoming more robust the higher the value. For this reason
the values achieved for English (.99), Portuguese (.97) and Spanish (.96) are so high.
This fact clearly indicates that there is little variation in the positions of the journals.
3.4. Database growth: Scholar, WoS and Scopus
Finally, this section looks at the growth data for GS, as the GSM data source, and
compares these data with those of Scopus and WoS.
Table 5 shows the statistical range obtained from weekly measurement of records in the
three databases. The complete raw data for each country and database are available in
the supplementary material for consultation.
The data correspond to both the total number of records and those obtained only from
2003 onwards. Moreover, each of the languages included in GSM is represented by a
country, except in the case of English, represented both the UK and the United States.
In the case of GS, there are positive ranges (i.e. growth during the four weeks of
measurement) for China (8,100,000 more records) and Italy (55,000). In the case of the
United States, despite the fact that it also obtained a very high positive range (106,300
more records), the data must be treated with some caution, as they are the sum of four
domains (.edu, .mil, .us, .gov), which means that there may be a high degree of
overlapping. In addition, the .edu domain is also used in other countries, so the GS
figures relating to web indicators for the United States should be taken only as a rough
Table 5. Statistical range (total and recent) of records retrieved in GS, Scopus and WoS
R (tot)
R (recent)
R (tot)
R (recent)
R (tot)
R (recent)
For recent literature (from 2003 onwards), Netherlands (6,000 fewer records), Germany
(41,000), South Korea (92,000), France (160,000) and, in particular, the UK (518,000)
have a negative range. Finally, Brazil has inconsistent results (while recent literature
remains the same, the total figure decreases significantly). In fact, recent data show
certain inconsistencies in relation to total data for many countries, such as Italy (lower
Google Scholar Metrics evolution: an analysis according to languages
total growth than recent), Germany (greater recent decrease than total), Spain (total
decrease and recent growth) and South Korea (total growth and recent decrease).
These results are consistent with the instability and limitations of GS in retrieving web
data due to changes in Google coverage. For example, in the third sample (see
supplementary materials) a fall in UK records may be observed (from 3,830,000 to
1,190,000, which helps to explain the final figures obtained).
Table 6 expands the GS results, comparing the data from the last sample (March 2013)
with data collected just at the time of the appearance of the second version of the
rankings (November 2012). It also adds the data calculated by Aguillo (2013), collected
in August 2010.
Table 6. Google Scholar evolution according to geographic web domain
The results show, on one hand, the high growth rate since the first data were reported
(when the United States still surpassed China) up to 2013, also highlighting the high
rate of growth in Japan. In fact, just as the figures for China and the United States bear
relation to the data in Table 3, the low h-index values obtained for Japanese journals
contrast with the Japanese web domain size, an aspect that should be studied in greater
depth in the future.
Taking the last week of data collection as a reference, the countries with the highest
total number of records are China (30,700,000 records), followed by the United States
(16,019,000), Japan (10,400,000), France (4,210,000) and Spain (2,990,000). The
weekly evolution of the number of records for these countries is shown in Figure 3,
where we can see how, despite the observed data, growth is relatively stable, except for
the weeks when there are updates; this is consistent with the results obtained previously
by Aguillo (2012).
Figure 3. Evolution of page count according to geographic domains in Google
The positions of the countries correspond approximately to the rankings provided in
Table 3, with the exception of Portuguese and German. In any case, caution should be
exercised with these results, as they evidently do not correspond to the actual growth
data by country since every citizen or institution is free to choose the geographical or
generic name when registering a domain name, although this process is indeed useful
for determining geographic coverage in general.
Google Scholar Metrics evolution: an analysis according to languages
As for the records retrieved from the other databases, the results are as expected: the
number of records is lower, their growth rates are more stable and there are hardly any
inconsistencies between total and recent results, with the exception of Germany, for
which high figures were obtained for the number of recent records in WoS between the
first (802,889) and the second sample (1,175,266).
Except for these last data which are assumed to be the result of an error after the
United States, China is the country (among those analysed) with the greatest number of
both recent and total records in Scopus and WoS. Chinas elevated results correspond to
those displayed in Table 3, ranking according to language, and are especially significant
given the high percentage of recent records in relation to the total (86.02% in Scopus;
82.33% in WoS), reflecting high productivity at present. In fact, South Korea, Japan and
Brazil are the countries with the highest percentages of recent records.
Figure 4, meanwhile, shows the weekly evolution of total records for China in the three
databases analysed, in which differences in coverage and the staggered updates of GS
may be observed. Moreover, the higher results in Scopus as compared to WOS are
associated with the greater orientation of the former towards the Chinese publishing
system (Leydesdorff 2012), a phenomenon which can also be seen clearly in Table 5.
Figure. 4. Number of records for China (Scholar, Scopus & WoS)
Finally, Table 7 shows a summary of the number of records (averaged weekly) and the
monthly growth rate (from the first to the last weekly sample) by language and
database, as well as total values (obtained from the sum of the number of records in all
languages analysed).
Table 7. Weekly average size and monthly growth rate per source (Scholar, Scopus, WoS)
Size average
Growth rate
Size average
Growth rate
Size average
Growth rate
Although the variability of GS data is greater and changes are staggered, we can see
how the average weekly number of records in GS is higher than in the other databases,
except for UK, Italy, Germany and Netherlands, where the weekly average values are
higher in both Scopus and WoS (due to the negative trends identified in GS for these
countries). As for the total values, the prevalence of GS (due to the United States and,
especially, to Japan and China) may clearly be observed, but the data must be
interpreted with caution because of the methodological limitations already discussed
regarding records extracted by geographic web domains.
Google Scholar Metrics evolution: an analysis according to languages
With regard to growth rates, Table 7 shows, in summary, mismatches in GS data for
some countries, both positive (China: 42.13%) and negative (UK: -68.67%). In any
case, the values obtained for the entire set of languages clearly show a higher overall
growth rate for GS in comparison with the other databases (GS: 11.5%; Scopus: 0.41%;
WoS: 0.37%). Growth rates for individual countries must also be contextualised in
relation to size in terms of the number of records (considerably higher in GS), to
account for the effects of scale.
4. Discussion and conclusions
The data collected for this first longitudinal study of a representative sample of journals
(1,000) in GSM, of very different origin (because they are published in 10 languages),
show a significant increase in the h-index (15%) in a very short period of time (seven
and a half months). These figures are higher than those reported by Harzing (2013) in
his sample of the h-index of 20 Nobel laureates in physics, chemistry, medicine and
economics (comprising 400 publications), in which the growth rate of the h-index was
7.3% (and 13.1% of the total citations). These lower values are justified by the fact that
the period analysed in Harzings study (April to September) is shorter than that applied
in this study (April to November).
This remarkable increase in the GSM bibliometric indicators is evidently in keeping
with, on the one hand, the spectacular growth of open access scholarly literature on the
Internet and, secondly, the capacity of the GS search engine to index this academic
literature immediately, which partly explains the staggered growth detected in the
previous point.
The robot used by GS to automatically crawl the web immediately incorporates and
processes all this information. In previous studies (Delgado López-Cózar and Cabezas-
Clavijo, 2013), it was found that the indexing of a document and its citations does not
take more than three days in the case of documents deposited in institutional or thematic
academic repositories, and between a week and a month for the personal and
institutional websites of scientists, research groups, departments, institutes and
academic centres. Harzing (2013) showed a monthly citation growth rate of 3% in his
study, double the increase in WoS. It is clear that GS is growing more and much faster
than traditional databases. This growth in bibliometric values occurs for virtually every
journal, regardless of the language in which it is published. However, significant
differences in growth rates were detected, depending on the language in which the
journal is published.
Another aspect revealed by this study is the marked differences in the size of the h-
index for journals according to the publication language. The extremely high h-index of
the English-language journals (ten times the values achieved by the journals in other
languages, with the exception of Chinese where it is five times larger) is undoubtedly a
faithful reflection of both the scale and size of the English-speaking academic
community, the predominance of English as the lingua franca of academic
communication, and the volume of scholarly production circulating on the Web.
An analysis of the growth of GS based on the web size of the main geographical top
level domains (gTLD) shows, in particular, the dramatic increase in the number of items
retrieved in the space of two years. Comparing the data collected by Aguillo in August
Google Scholar Metrics evolution: an analysis according to languages
2010 with data collected in March 2013, the differences are illustrative enough to
understand this phenomenon of English-language predominance. However, a
particularly striking aspect is still the disproportionate position of Portuguese in relation
to Spanish in terms of volume of data in the national domains and the h-index levels of
the respective journals in GSM, as well as the results obtained for Japanese magazines.
The other important finding of this study is that, despite all the technical and
methodological problems posed by GS as a source of information for academic
evaluation (misidentification of documents and citations, lack of transparency in the
selection of sources, deficiencies in the control and standardisation of records, etc.), the
precise nature and true size of which we cannot determine (something that is, moreover,
almost impossible given the universal nature of Google), the fact that its evolution is
unpredictable, vulnerable to coverage of the commercial search engine, and different
from the other bibliometric databases (i.e. WoS and Scopus), the stability of journal
rankings between April and November 2012 is very high. This reinforces the credibility
of the data handled by GS in the creation of GSM journal rankings, despite the fact that
their size, coverage, updating and nature are, in principle, different from those provided
by WoS and Scopus. Ultimately, journal rankings produced by GSM are sound and
Regardless of the data that this study has contributed to determining the size, nature and
stability of bibliometric indicators produced by GS, conclusions may be drawn from this
research which should lead to changes in the GSM rankings. These are:
1. The growth of bibliometric data demonstrates the need to update the rankings at
least twice a year. This would be contrary to the usual practice of the bibliometric
journal assessment industry, which updates its products annually. We would even
go so far as to advise Google to turn its product into a fully dynamic information
system, so that the rankings are updated instantly in the same way as the
bibliometric author profiles provided by Google Scholar Citations.
2. Google Scholar Metrics is shown to have adopted a wrong policy in displaying
only 100 journals in each ranking and, even worse, using the same threshold for
rankings in the different languages. Such marked differences in the volume of
indicators would require the results displayed in each ranking to be proportional to
the number of journals indexed in every language. Our advice would be to use
thresholds of 1%, 5% or 10% of the journals with the highest h-index by
5. Notes
Google Scholar Metrics (accessed 1 September 2013).
2 (accessed 1
September 2013).
3 (accessed 1 September 2013).
In July 2013 the third version of GSM appeared, in which the citation window varied (2008-2012). For
this reason this version is not taken into account in this longitudinal study. More information: (accessed 1 September 2013).
5 (accessed 1 September 2013).
The raw data for the two GSM editions are available in Annex I of the supplementary material.
The raw data for the weekly analysis of coverage in GS, Scopus and WoS are available in Annex II of
the supplementary material.
Google Scholar Metrics evolution: an analysis according to languages
6. References
Aguillo, Isidro F. (2012). Is Google Scholar useful for bibliometrics? A webometric analysis.
Scientometrics, 91(2), 343-351.
Brewington, B. E. & Cybenko, G. (2000). How dynamic is the Web?. Computer Networks, 33(1-6), 257-
Cabezas-Clavijo, A. & Delgado López-Cózar, E. (2013). Google Scholar e índice h en biomedicina: la
popularización de la evaluación bibliométrica. Medicina intensiva, 37(5), 343-354.
Cho, Y. & Garcia-Molina, H. (2000). The evolution of the web and implications for an incremental
crawler. Proceedings of the 26th International Conference on Very Large Data Bases, 200-209.
Costas, R. & Bordons, M. (2007). The h-index: Advantages, limitations and its relation with other
bibliometric indicators at the micro level. Journal of Informetrics, 1(3), 193-203.
Delgado López-Cózar, E. & Cabezas-Clavijo, A. (2012). Google Scholar Metrics: an unreliable tool for
assessing scientific journals. El profesional de la información, 21(4), 419-427.
Delgado López-Cózar, E. & Cabezas-Clavijo, A. (2013). Ranking journals: could Google Scholar Metrics
be an alternative to Journal Citation Reports and Scimago Journal Ranks. Learned publishing, 26(2), 101-
Fetterly, D., Manasse, M., Najork, M. & Wiener, J. (2003). A large scale study of the evolution of web
pages. Proceedings of the Twelfth International Conference on World Wide Web, 669-678.
Harzing, A-W. (2013). A preliminary test of Google Scholar as a source for citation data: a longitudinal
study of Nobel prize winners. Scientometrics, 94(3), 1057-1075.
Jacsó, P. (2012). Google Scholar Metrics for Publications The software and content feature of a new
open access bibliometric service. Online Information Review, 36(4), 604-619.
Koehler, W. (2002). Web page change and persistence -4- year longitudinal web study. Journal of the
American Society for Information Science and Technology, 53(2), 162-171.
Koehler, W. (2004). A longitudinal study of Web pages continued a consideration of document
persistence. Information Research, 9(2). Available at
(accessed 1 September 2013).
Kousha, K. & Thelwall, M. (2007). Google Scholar Citations and Google Web/URL citations: a
multidiscipline exploratory analysis. Journal of the American Society for Information Science and
Technology, 58(7), 1055-1065.
Leydesdorff, L. (2012). World shares of publications of the USA, EU-27, and China compared and
predicted using the new Web of Science interface versus Scopus. El profesional de la información, 21(1),
Orduña-Malea, E., Serrano-Cobos, J. & Lloret-Romero, N. (2009). Las universidades públicas españolas
en Google Scholar: presencia y evolución de su publicación académica web. El profesional de la
información, 18(5), 493-500.
Orduña-Malea, E., Serrano-Cobos, J., Ontalba-Ruipérez, J-A. & Lloret-Romero, N. (2010). Revista
española de documentación científica, 33(2), 246-278.
Ortega, J. L., Aguillo, I. F. & Prieto, J. A. (2006). Longitudinal study of contents and elements in the
scientific Web environment. Journal of Information Science, 32(4), 344-351.
Google Scholar Metrics evolution: an analysis according to languages
Payne, N. & Thelwall, M. (2007). A longitudinal study of academic webs: growth and stabilization.
Scientometrics, 71(3), 523-539.
Winter, Joost C.F. de, Zadpoor, Amir A. & Dodou, D. (2013). The expansion of Google Scholar versus
Web of Science: a longitudinal study. Scientometrics [Online first]. DOI 10.1007/s11192-013-1089-2.
... 5 It now publishes a number of metrics for scholars including total citations, h-index, and i-10 index in the profiles. Owing to its popularity, it has been studied extensively in different studies, both as a source of scientific information in comparison to other bibliographic databases (such as Falagas et al., 2008;Aguillo, 2012;Harzing & Alakangas, 2016;Halevi et al., 2017;Martin-Martin et al., Martín-Martín, Orduña-Malea, et al., 2018, etc.) and as a provider of different metrics (such as Jacso, 2012;Lopez-Cozar & Cabezas-Clavijo, 2013;Orduña-Malea & López-Cózar, 2014;Martin-Martin et al., 2017, etc.). However, at the same time, it is sometimes criticized on account of its accuracy and its policy towards including certain predatory journals in its index (Beall, 2014;Kolata, 2017). ...
... While some studies evaluated it as a source of scientific information and compared its coverage with other bibliographic databases (Jacso, 2005;Mayr & Walter, 2007;Walters, 2007;Falagas et al., 2008;Aguillo, 2012;De Winter et al., 2014;Harzing & Alakangas, 2016;Halevi et al., 2017;Martin-Martin et al., Martín-Martín, Orduña-Malea, et al., 2018;Gusenbauer, 2019); several others highlighted the role of Google Scholar in providing evidence of open access of scientific publications (Orduña-Malea & López-Cózar, 2015;Norris et al., 2008;Jamali & Nabavi, 2015;Martin-Martin et al., Martín-Martín, Costas, et al., 2018). Some studies have also focused their attention on different metrics computed by Google Scholar and their usefulness (Jacso, 2008;Jacso, 2012;Lopez-Cozar & Cabezas-Clavijo, 2013;Bohannon, 2014;Orduña-Malea & López-Cózar, 2014;Martin-Martin et al., 2017). ...
Full-text available
ResearchGate has emerged as a popular professional network for scientists and researchers in a very short span. Similar to Google Scholar, the ResearchGate indexing uses an automatic crawling algorithm that extracts bibliographic data, citations, and other information about scholarly articles from various sources. However, it has been observed that the two platforms often show different publication and citation data for the same institutions, journals, and authors. While several previous studies analysed different aspects of ResearchGate and Google Scholar, the quantum of differences in publications, citations, and metrics between the two and the probable reasons for the same are not explored much. This article, therefore, attempts to bridge this research gap by analysing and measuring the differences in publications, citations, and different metrics of the two platforms for a large data set of highly cited authors. The results indicate that there are significantly high differences in publications and citations for the same authors captured by the two platforms, with Google Scholar having higher counts for a vast majority of the cases. The different metrics computed by the two platforms also differ in their values, showing different degrees of correlation. The coverage policy, indexing errors, author attribution mechanism, and strategy to deal with predatory publishing are found to be the main probable reasons for the differences in the two platforms.
... However, this assessment method is not entirely suitable for Chinese scholars, since a considerable number of them writes in their native language, and there are many excellent works in Chinese papers [According to (Orduña-Malea and López-Cózar, 2014), Chinese journals prominently occupy second place in the h-index average journal ranking in April and November 2012]. Therefore, it is necessary to pay more attention to Chinese papers in academic assessments. ...
... Therefore, it is necessary to pay more attention to Chinese papers in academic assessments. Although some databases such as Google Scholar include a considerable number of Chinese literature (Orduña-Malea and López-Cózar, 2014), related citation analyses are based on citations between both Chinese and English literature. In this case, we aim to implement an academic assessment based on Chinese literature through citation analysis, limiting citations to those between selected Chinese journals, so as to see something that has always been neglected and make our own voice heard. ...
Full-text available
Citation analysis is one of the most commonly used methods in academic assessments. Up to now, most of academic assessments are based on English literature, ignoring the fact that the role of Chinese papers in academic assessments has become increasingly indispensable. Therefore, to give full play to the role of Chinese literature in academic assessments is an urgent task of current academic circle. Based on Chinese academic data from ScholarSpace, i.e., 82826 Chinese computer science journal papers, we conduct a comprehensive assessment of academic influence from the perspectives of fields, journals and institutions, in order to achieve a better understanding of the development of Chinese computer literature in the past 60 years. We find that Chinese scholars tend to cite papers in English, discover evolution trend of fields, journals and institutions, and call on journals, institutions, and scholars to strengthen their cooperation.
... The poor representation of non-English literature from the two subscription-based databases/platforms found by this study aligns with previous findings (Harzing and Alakangas, 2016;Orduña-Malea and Delgado López-Cózar, 2014); however, it is disappointing, given that both claim to 'support' non-English researchers by providing alternative interface languages (see Supplementary Text S.4). Rather than a lack of non-English climate change literature on Africa, this demonstrates a lack of indexing by these databases. ...
Disregarding research not published in English may pose a risk to finding solutions for urgent global concerns, such as biodiversity loss, or climate change. To assess the extent of this ‘missing voice’, we compared the representation of 22 languages in scientific publications on climate change in Africa, indexed by widely used databases. Between 87 % and 95 % of publications were in English, with a small, but noteworthy, number in languages of the former European colonisers of Africa. We then assessed undergraduate monographs, master’s dissertations, doctoral theses, and peer-reviewed papers derived from the doctoral theses, that are about Lusophone Africa and written in Portuguese, and found this research largely not accessible in English on online databases. If the goal of researchers, practitioners and policy makers is to obtain climate change information on, or present solutions for, individual developing countries, cultures, or localised issues, then searching in English may exclude local, context specific knowledge. This may prevent global assessments from being truly global, or locally down-scalable, by biasing science towards a single world view that marginalises key local stakeholders.
... Its coverage of some areas of psychology has been characterized as uneven, containing fewer journals in the areas of organizational and clinical psychology than PsycINFO, while having more extensive coverage of neuroscience, methodology, and statistics journals ( García-Pérez, 2010). Approximately 90% of all documents in WoS originates from English-speaking countries (Orduña-Malea & López-Cózar, 2014). While this may reflect a geographic or linguistic bias, this is not particularly troubling for our purposes, as we focus on developments within American psychology. ...
The history of twentieth-century American psychology is often depicted as a history of the rise and fall of behaviorism. Although historians disagree about the theoretical and social factors that have contributed to the development of experimental psychology, there is widespread consensus about the growing and (later) declining influence of behaviorism between approximately 1920 and 1970. Since such wide-scope claims about the development of American psychology are typically based on small and unrepresentative samples of historical data, however, the question rises to what extent the received view is justified. This paper aims to answer this question in two ways. First, we use advanced scientometric tools (e.g. bibliometric mapping, co-citation analysis, and term co-occurrence analysis) to quantitatively analyze the metadata of 119.278 papers published in American journals between 1920 and 1970. We reconstruct the development and structure of American psychology using co-citation and co-occurrence networks and argue that the standard story needs reappraising. Second, we argue that the question whether behaviorism was the ‘dominant’ school of American psychology is historically misleading to begin with. Using the results of our bibliometric analyses, we argue that questions about the development of American psychology deserve more fine-grained answers.
... In addition, there are other factors that are not fully explored but might be useful. Specifically, features such as occurrence frequency (Zhu et al. 2015), time interval, the average length of citing sentences, the average density of citation occurrences (Wan and Liu 2014), the publication language (Ordunamalea and Lopezcozar 2014), the keywords similarity (Fujita et al. 2014), and social network impact (Priem and Hemminger 2010;Costas et al. 2015) might be valuable factors. ...
Full-text available
Scholarly impact assessment has always been a hot issue. It has played an important role in evaluating researchers, scientific papers, scientific teams, and institutions within science of science. Scholarly impact assessment is also used to address fundamental issues, such as reward evaluation, funding allocation, promotion and recruitment decision. Scholars generally agree that it is more reasonable to use weighted citations to assess the scholarly impact. Although a great number of researchers use weighted citations to access the scholarly impact, there is a lack of a systematic summary of citation weighting methods. To fill the gap, this paper summarizes the existing classical indicators and weighting methods used in measuring scholarly impact from the perspectives of articles, authors and journals. We also summarize the focus of the indicators involved in this paper and the weighting factors that involved in the weighting methods. Finally, we discuss the open issues to try to discover the hidden trends of citation weighting. Through this paper, we can not only have a clearer understanding of the weighting methods in the scholarly impact assessment, but also think more deeply about the weighting factors to be explored.
Full-text available
En este artículo se propone un análisis del impacto de la investigación en comunicación a través del estudio comparado de la citación en los principales formatos de publicación -libros y artículos- en las áreas de periodismo y de comunicación audiovisual y publicidad. Para ello, se han tomado como referencia las tres publicaciones de mayor impacto de 281 investigadores de universidades públicas españolas, contando así con 843 trabajos que han arrojado un total de 72.993 citas según Google Scholar. La investigación concluye la convivencia armónica de formatos en este campo, si bien los libros muestran un mayor recorrido temporal mientras que las revistas son claramente elegidas por los investigadores de menor edad. Dichas conclusiones son finalmente contextualizadas en el marco de las actuales políticas de la investigación en España.
Purpose This paper aims to answer the question of how the Polish representatives of social communication and media sciences communicate the most recent scientific findings in the media space, i.e. what types of publications are shared, what activities do they exemplify (sharing information about their own publications, leading discussions, formulating opinions), what is the form of the scientific communication created by them (publication of reference lists' descriptions, full papers, preprints and post prints) and what is the audience reception (number of downloads, displays, comments). Design/methodology/approach The authors present the results of analysis conducted on the presence of the most recent (2017–2019) publications by the Polish representatives of the widely understood social communication and media sciences in three selected social networking services for scientists: ResearchGate, Google Scholar and The analyses covered 100 selected representatives of the scientific environment (selected in interval sampling), assigned, according to the OECD classification “Field of Science”, in the “Ludzie nauki” (Men of Science) database to the “media and communication” discipline. Findings The conducted analyses prove a low usage level of the potential of three analysed services for scientists by the Polish representatives of social communication and media sciences. Although 60% of them feature profiles in at least one of the services, the rest are not present there at all. From the total of 113 identified scientists' profiles, as little as 65 feature publications from 2017 to 2019. Small number of alternative metrics established in them, implies, in turn, that if these metrics were to play an important role in evaluation of the value and influence of scientific publications, then this evaluation for the researched Polish representatives of social communication and media sciences would be unfavourable. Originality/value The small presence of the Polish representatives of the communication and media sciences in three analysed services shows that these services may be – for the time being – only support the processes of managing own scientific output. Maybe this quite a pessimistic image of scientists' activities in the analysed services is conditioned by a simple lack of the need to be present in electronic channels of scientific communication or the lack of trust to the analysed services, which, in turn, should be linked to their shortcomings and flaws. However, unequivocal confirmation of these hypotheses might be brought by explorations covering a larger group of scientists, and complemented with survey studies. Thus, this research may constitute merely a starting point for further explorations, including elaboration of good practices with respect to usage of social media by scientists.
This study determined how useful Google Scholar (GS) is for the evaluation of non‐English journals based on a sample of 150 Chinese journals listed in the Report on Chinese Academic Journals Evaluation of Research Center for Chinese Science Evaluation (2013–2014). This study investigated two disciplines: Library, Information & Documentation Science and Metallurgical Engineering & Technology. We collected data from GS and the Chongqing VIP database to evaluate GS as a citation database for Chinese journals on its resource coverage, journal ranking, and citation data. We found that GS covered 100% of the sample journals but indexed 22% more article records than the number of articles published. The ranking of Chinese journals by GS Metrics was not suitable to present a dependable ranking of Chinese journals. GS appeared suitable to provide an alternative source of Chinese citation data, even though there existed coverage problems, including article duplication and citation omission and potential duplication. The GS Metric average citation provided results highly correlated to traditional citation results, showing that it would be suitable for evaluating Chinese journals.
Full-text available
Information on the size of academic search engines and bibliographic databases (ASEBDs) is often outdated or entirely unavailable. Hence, it is difficult to assess the scope of specific databases, such as Google Scholar. While scientometric studies have estimated ASEBD sizes before, the methods employed were able to compare only a few databases. Consequently, there is no up-to-date comparative information on the sizes of popular ASEBDs. This study aims to fill this blind spot by providing a comparative picture of 12 of the most commonly used ASEBDs. In doing so, we build on and refine previous scientometric research by counting query hit data as an indicator of the number of accessible records. Iterative query optimization makes it possible to identify a maximum number of hits for most ASEBDs. The results were validated in terms of their capacity to assess database size by comparing them with official information on database sizes or previous scientometric studies. The queries used here are replicable, so size information can be updated quickly. The findings provide first-time size estimates of ProQuest and EbscoHost and indicate that Google Scholar’s size might have been underestimated so far by more than 50%. By our estimation Google Scholar, with 389 million records, is currently the most comprehensive academic search engine.
Full-text available
Information on the size of academic search engines and bibliographic databases (ASEBDs) is often outdated or entirely unavailable. Hence, it is difficult to assess the scope of specific databases, such as Google Scholar. While scientometric studies have estimated ASEBD sizes before, the methods employed were able to compare only a few databases. Consequently, there is no up-to-date comparative information on the sizes of popular ASEBDs. This study aims to fill this blind spot by providing a comparative picture of 12 of the most commonly used ASEBDs. In doing so, we build on and refine previous scientometric research by counting query hit data as an indicator of the number of accessible records. Iterative query optimization makes it possible to identify a maximum number of hits for most ASEBDs. The results were validated in terms of their capacity to assess database size by comparing them with official information on database sizes or previous scientometric studies. The queries used here are replicable, so size information can be updated quickly. The findings provide first-time size estimates of ProQuest and EbscoHost and indicate that Google Scholar's size might have been underestimated so far by more than 50%. By our estimation Google Scholar, with 389 million records, is currently the most comprehensive academic search engine.
Full-text available
The launch of Google Scholar Metrics as a tool for assessing scientific journals may be serious competition for Thomson Reuters Journal Citation Reports, and for Scopus powered Scimago Journal Rank. A review of these bibliometric journal evaluation products is performed. We compare their main characteristics from different approaches: coverage, indexing policies, search and visualization, bibliometric indicators, results analysis options, economic cost and differences in their ranking of journals. Despite its shortcomings, Google Scholar Metrics is a helpful tool for authors and editors in identifying core journals. As an increasingly useful tool for ranking scientific journals, it may also challenge established journals products
Full-text available
Most governmental research assessment exercises do not use citation data for the Social Sciences and Humanities as Web of Science or Scopus coverage in these disciplines is considered to be insufficient. We therefore assess to what extent Google Scholar can be used as an alternative source of citation data. In order to provide a credible alternative, Google Scholar needs to be stable over time, display comprehensive coverage, and provide non-biased comparisons across disciplines. This article assesses these conditions through a longitudinal study of 20 Nobel Prize winners in Chemistry, Economics, Medicine and Physics. Our results indicate that Google Scholar displays considerable stability over time. However, coverage for disciplines that have traditionally been poorly represented in Google Scholar (Chemistry and Physics) is increasing rapidly. Google Scholar’s coverage is also comprehensive; all of the 800 most cited publications by our Nobelists can be located in Google Scholar, although in four cases there are some problems with the results. Finally, we argue that Google Scholar might provide a less biased comparison across disciplines than the Web of Science. The use of Google Scholar might therefore redress the traditionally disadvantaged position of the Social Sciences in citation analysis.
This article reports a 2010 empirical study using a 2005 study as a base to compare Google Scholar's coverage of scholarly journals with commercial services. Through random samples of eight databases, the author finds that, as of 2010, Google Scholar covers 98 to 100 percent of scholarly journals from both publicly accessible Web contents and from subscription-based databases that Google Scholar partners with. In 2005 the coverage of the same databases ranged from 30 to 88 percent. The author explores de-duplication of search results by Google Scholar and discusses its impacts on searches and library resources. With the dramatic improvement of Google Scholar, the uniqueness and effectiveness of subscription-based abstracts and indexes have dramatically changed.
How fast does the web change? Does most of the content remain unchanged once it has been authored, or are the documents continuously updated? Do pages change a little or a lot? Is the extent of change correlated to any other property of the page? All of these questions are of interest to those who mine the web, including all the popular search engines, but few studies have been performed to date to answer them.One notable exception is a study by Cho and Garcia-Molina, who crawled a set of 720,000 pages on a daily basis over four months, and counted pages as having changed if their MD5 checksum changed. They found that 40% of all web pages in their set changed within a week, and 23% of those pages that fell into the .com domain changed daily.This paper expands on Cho and Garcia-Molina's study, both in terms of coverage and in terms of sensitivity to change. We crawled a set of 150,836,209 HTML pages once every week, over a span of 11 weeks. For each page, we recorded a checksum of the page, and a feature vector of the words on the page, plus various other data such as the page length, the HTTP status code, etc. Moreover, we pseudo-randomly selected 0.1% of all of our URLs, and saved the full text of each download of the corresponding pages.After completion of the crawl, we analyzed the degree of change of each page, and investigated which factors are correlated with change intensity. We found that the average degree of change varies widely across top-level domains, and that larger pages change more often and more severely than smaller ones.This paper describes the crawl and the data transformations we performed on the logs, and presents some statistical observations on the degree of change of different classes of pages.
Purpose – The purpose of this paper is to review the software and content features of the Google Scholar Metrics (GSM) service launched in April 2012. Design/methodology/approach – The paper reviews GSM, examining the software, browsing, searching and sorting functions, citation matching and content. Findings – The paper reveals that the service can offer a better alternative than the traditional Google Scholar service to discover and judge the standing of journals through the prism of their citedness. GSM could become a potentially useful complementary resource primarily by virtue of its brand recognition, and the convenience of not requiring the installation of additional software, but currently its bibliometric indicators are often inappropriate for decision making in matters of tenure, promotion, grants and accreditation. Originality/value – The paper provides a good understanding of the GSM service.
The aim of this study is to review the features, benefits and limitations of the new scientific evaluation products derived from Google Scholar, such as Google Scholar Metrics and Google Scholar Citations, as well as the h-index, which is the standard bibliometric indicator adopted by these services. The study also outlines the potential of this new database as a source for studies in Biomedicine, and compares the h-index obtained by the most relevant journals and researchers in the field of intensive care medicine, based on data extracted from the Web of Science, Scopus and Google Scholar. Results show that although the average h-index values in Google Scholar are almost 30% higher than those obtained in Web of Science, and about 15% higher than those collected by Scopus, there are no substantial changes in the rankings generated from one data source or the other. Despite some technical problems, it is concluded that Google Scholar is a valid tool for researchers in Health Sciences, both for purposes of information retrieval and for the computation of bibliometric indicators.
Web of Science (WoS) and Google Scholar (GS) are prominent citation services with distinct indexing mechanisms. Comprehensive knowledge about the growth patterns of these two citation services is lacking. We analyzed the development of citation counts in WoS and GS for two classic articles and 56 articles from diverse research fields, making a distinction between retroactive growth (i.e., the relative difference between citation counts up to mid-2005 measured in mid-2005 and citation counts up to mid-2005 measured in April 2013) and actual growth (i.e., the relative difference between citation counts up to mid-2005 measured in April 2013 and citation counts up to April 2013 measured in April 2013). One of the classic articles was used for a citation-by-citation analysis. Results showed that GS has substantially grown in a retroactive manner (median of 170 % across articles), especially for articles that initially had low citations counts in GS as compared to WoS. Retroactive growth of WoS was small, with a median of 2 % across articles. Actual growth percentages were moderately higher for GS than for WoS (medians of 54 vs. 41 %). The citation-by-citation analysis showed that the percentage of citations being unique in WoS was lower for more recent citations (6.8 % for citations from 1995 and later vs. 41 % for citations from before 1995), whereas the opposite was noted for GS (57 vs. 33 %). It is concluded that, since its inception, GS has shown substantial expansion, and that the majority of recent works indexed in WoS are now also retrievable via GS. A discussion is provided on quantity versus quality of citations, threats for WoS, weaknesses of GS, and implications for literature research and research evaluation.