The relation between Eigenfactor, audience factor, and influence weight

Full text

Page 1

1
The relation between Eigenfactor, audience factor,
and influence weight

Ludo Waltman and Nees Jan van Eck

Centre for Science and Technology Studies, Leiden University, The Netherlands
{waltmanlr, ecknjpvan}@cwts.leidenuniv.nl





We present a theoretical and empirical analysis of a number of bibliometric indicators of journal
performance. We focus on three indicators in particular, namely the Eigenfactor indicator, the audience
factor, and the influence weight indicator. Our main finding is that the last two indicators can be
regarded as a kind of special cases of the first indicator. We also find that the three indicators can be
nicely characterized in terms of two properties. We refer to these properties as the property of
insensitivity to field differences and the property of insensitivity to insignificant journals. The empirical
results that we present illustrate our theoretical findings. We also show empirically that the differences
between various indicators of journal performance are quite substantial.
Introduction
The impact factor (Garfield, 1972, 2006) is without doubt the most commonly
used bibliometric indicator of the performance of scientific journals. Various
alternatives to the impact factor have been proposed in the literature. These
alternatives include indicators based on cited-side normalization (e.g., Van Leeuwen
& Moed, 2002), indicators based on citing-side normalization (Moed, in press; Zitt &
Small, 2008), indicators based on the h-index (e.g., Braun, Glänzel, & Schubert,
2006), and indicators based on recursive citation weighting. Indicators based on
recursive citation weighting were first proposed by Pinski and Narin (1976; see also
Geller, 1978), and they have been popular in the field of economics (Kalaitzidakis,
Mamuneas, & Stengos, 2003; Kodrzycki & Yu, 2006; Laband & Piette, 1994;
Liebowitz & Palmer, 1984; Palacios-Huerta & Volij, 2004). The successful PageRank
algorithm of the Google search engine (Brin & Page, 1998; Page, Brin, Motwani, &
Winograd, 1998; see also Langville & Meyer, 2006) has caused a renewed interest in
recursive indicators of journal performance. Three PageRank-inspired indicators that
have been recently introduced are the weighted PageRank indicator (Bollen,
Rodriguez, & Van de Sompel, 2006; Dellavalle, Schilling, Rodriguez, Van de
Sompel, & Bollen, 2007), the Eigenfactor indicator (Bergstrom, 2007; West,
Bergstrom, & Bergstrom, in press), and the SCImago Journal Rank indicator
(González-Pereira, Guerrero-Bote, & Moya-Anegón, 2009).
In this paper, we point out the relation between three indicators of journal
performance, namely the audience factor (Zitt & Small, 2008), the influence weight
indicator (Pinski & Narin, 1976), and the Eigenfactor indicator. The audience factor is
based on citing-side normalization, while the other two indicators are based on
recursive citation weighting. Unlike the audience factor and the influence weight
indicator, the Eigenfactor indicator is a parameterized indicator. Hence, the behavior
of the Eigenfactor indicator depends on the choice of a parameter. Our main finding is
that the audience factor and the influence weight indicator can be regarded as a kind
of special cases of the Eigenfactor indicator. Related to this, we show how the three

Page 2

2
indicators can be characterized in terms of two properties that we introduce. We refer
to these properties as the property of insensitivity to field differences and the property
of insensitivity to insignificant journals. Interestingly, it turns out that the parameter
of the Eigenfactor indicator can be used to make a trade-off between the two
properties. In addition to a theoretical analysis of the audience factor, the influence
weight indicator, and the Eigenfactor indicator, we also report some results of an
empirical analysis of these indicators.
This paper is organized as follows. First, we discuss our indicators of interest and
we point out how these indicators are mathematically related to each other. Next, we
study the indicators empirically. Finally, we briefly discuss some other related
indicators and we summarize our conclusions. Some technical details are elaborated
in an appendix.
Indicators
In this section, we discuss the indicators that we study in this paper. We use the
following mathematical notation. Let there be n journals, denoted by 1, …, n. Let T1
and T2 denote two time periods, where period T1 precedes period T2. (The two periods
may overlap or coincide.) We are interested in measuring the performance of journals
1, …, n based on citations from articles published in period T2 to articles published in
period T1. Let ai1 and ai2 denote the number of articles published in journal i in,
respectively, periods T1 and T2, and let cij denote the number of citations from articles
published in journal i in period T2 to articles published in journal j in period T1. We
define si as



i
s

j
ij
c
. (1)

Hence, si denotes the total number of citations from articles published in journal i in
period T2 to articles published in journals 1, …, n in period T1.
Using the above mathematical notation, we now discuss our indicators of interest.
We focus on the essential characteristics of the indicators. We ignore practical issues
such as the document types (e.g., articles, letters, and reviews) that are taken into
account, the length of the time window within which citations are counted, and the
way in which self citations are handled. For our present purposes, issues such as these
are not important.
Impact factor
Although the impact factor is not our main interest in this paper, we include it for
completeness. The impact factor is defined as the average number of citations that a
journal has received per article (Garfield, 1972, 2006). Hence, the impact factor of
journal i can be written as




1
IF
i
j
ji
i
a
c
. (2)

The impact factor is a very simple indicator. It is well known that in some fields
articles are on average cited much more frequently than in other fields. The impact
factor does not correct for such differences among fields. Because of this, impact
factors of journals in different fields should not be directly compared with each other.

Page 3

3
Audience factor
The audience factor is a recent proposal of Zitt and Small (2008). The audience
factor is similar to the impact factor except that citations are weighted based on the
journal from which they originate. The larger a journal’s average number of
references per article, the lower the weight of a citation originating from the journal.
The audience factor of journal i is defined as


i
a
1

where mj and mS are given by



j
ji
j
i
c
m
m
S
1
AF
, (3)

2 j
j
j
a
s
m 
, (4)





j
j
j
j
a
s
m
2
S
, (5)

that is, mj denotes journal j’s average number of references per article and mS denotes
the average number of references per article for all journals taken together. Notice that
in the definitions of mj and mS only references to articles published in journals 1, …, n
in period T1 are taken into account. These references are called active references by
Zitt and Small. All non-active references are ignored.
By assigning weights to citations, the audience factor aims to correct for
differences among fields. Unlike indicators based on cited-side normalization (e.g.,
Van Leeuwen & Moed, 2002), which also aim to correct for field differences, the
audience factor has the advantage that it does not rely on an externally imposed field
classification. In Appendix A, we introduce the property of insensitivity to field
differences. This property provides a formal definition of the idea of correcting for
field differences. Informally, the property of insensitivity to field differences has the
following interpretation. Suppose that we have two equally-sized fields and that each
journal gives away only a small amount of citations to journals that are not in its own
field. We then say that an indicator is insensitive to field differences if the average
value of the indicator for one field deviates from the average value of the indicator for
the other field only by a small amount. In the case of two fields without any between-
fields citation traffic, the property of insensitivity to field differences requires that the
average value of an indicator is the same for both fields. We show in appendix A that
under a relatively mild assumption the audience factor has the property of
insensitivity to field differences.
Influence weight
The influence weight indicator was proposed by Pinski and Narin (1976). The
influence weights of journals 1, …, n, denoted by IW1, …, IWn, are obtained by
solving the following system of linear equations:1

1 This system of linear equations has a unique solution if the journal citation matrix C = [cij] is
irreducible. In other words, the system of linear equations has a unique solution if in the journal citation
graph there exists for any two journals i and j a path from i to j and a path from j to i. We note that for

Page 4

4


ni
s
c
i
j
jij
i
, , 1for
IW
IW


(6)


1
IW



i
i
i
ii
s
s
. (7)

Unlike the impact factor and the audience factor, the influence weight indicator is a
measure of a journal’s average performance per reference rather than of its average
performance per article. Based on the influence weight of journal i, a measure of
journal i’s average performance per article can be obtained by

IW
IPP
i


1i
ii
a
s
. (8)

Following Pinski and Narin (1976), we refer to the indicator in Equation 8 as the
influence per publication indicator. Theoretical studies of the influence weight
indicator and the influence per publication indicator can be found in papers by Geller
(1978), Palacios-Huerta and Volij (2004), and Serrano (2004). In the last two papers,
the influence per publication indicator is referred to as the invariant method.
In Appendix A, we show that the influence per publication indicator does not have
the property of insensitivity to field differences. However, the influence per
publication indicator does have another interesting property, referred to as the
property of insensitivity to insignificant journals. To see this, consider the following
example. There are n = 8 journals. Each journal publishes 100 articles in each time
period. Hence, ai1 = ai2 = 100 for i = 1, …, n. The journal citation matrix C = [cij] is
shown in Table 1. Based on this matrix, two fields can be distinguished. One field
consists of journals 1, 2, 3, and 4. The other field consists of journals 5, 6, 7, and 8. A
distinction can also be made between frequently cited journals and infrequently cited
journals. Journals 1, 2, 5, and 6 are frequently cited, while journals 3, 4, 7, and 8 are
infrequently cited. In practice, it is almost impossible to have publication and citation
data for all infrequently cited journals in a field. This is because the coverage of
infrequently cited journals in bibliographic databases such as Web of Science and
Scopus is far from complete. Some infrequently cited journals are covered by these
databases, but many others are not. To examine the consequences of incomplete
coverage of infrequently cited journals, we look at two scenarios, scenario 1 and
scenario 2. In scenario 1, journals 1, …, 8 are all covered by the bibliographic
database that we use. In scenario 2, journals 1, …, 7 are covered while journal 8 is
not. For both scenarios, influence per publication scores calculated using Equations 6,
7, and 8 are reported in Table 2. As can be seen in the table, the influence per
publication scores of journals 1, …, 7 are very similar in the two scenarios. This
demonstrates that the influence per publication indicator is rather insensitive to
incomplete coverage of infrequently cited journals. We therefore say that the
influence per publication indicator has the property of insensitivity to insignificant
journals.

computational reasons it is convenient if the system of linear equations is not only irreducible but also
aperiodic.

Page 5

5

TABLE 1. Journal citation matrix. Rows correspond with citing journals. Columns
correspond with cited journals.

Journal 1 2 3
1 1000 1000 10
2 1000 1000 10
3 1000 1000 10
4 1000 1000 10
5 100 100 1
6 100 100 1
7 100 100 1
8 100 100 1


TABLE 2. Journals’ influence per publication scores and audience factors.

Journal 1 2 3
IPP (scenario 1) 5.500 5.500 0.055
IPP (scenario 2) 5.513 5.513 0.055
AF (scenario 1) 44.000 44.000 0.440
AF (scenario 2) 42.938 42.938 0.429

What is the relevance of the property of insensitivity to insignificant journals?
This can be seen as follows. Suppose that instead of the influence per publication
indicator the audience factor is used in the above example. For both scenario 1 and
scenario 2, audience factors calculated using Equations 3, 4, and 5 are reported in
Table 2. Comparing the two scenarios, it is clear that the audience factor does not
have the property of insensitivity to insignificant journals. Due to the non-coverage of
journal 8 in scenario 2, journals 5, 6, and 7 have substantially lower audience factors
in this scenario than in scenario 1. Journals 1, 2, 3, and 4 have only marginally lower
audience factors. Hence, the non-coverage of journal 8 in scenario 2 causes a
substantial decrease of the audience factors of journals 5, 6, and 7 relative to the
audience factors of journals 1, 2, 3, and 4. The results reported in Table 2 demonstrate
that, when using an indicator that does not have the property of insensitivity to
insignificant journals, the score of a journal in a certain field may strongly depend on
the number of infrequently cited journals in the same field that are covered by the
bibliographic database that one uses. This sensitivity to infrequently cited journals
may be problematic when comparing scores of journals in different fields. If the
bibliographic database that one uses covers relatively more infrequently cited journals
in one field than in another, journals in the former field have an advantage over
journals in the latter field.
We have now introduced two properties that bibliometric indicators of journal
performance may or may not have, namely the property of insensitivity to field
differences and the property of insensitivity to insignificant journals. It is important to
note that these two properties rule out each other, that is, an indicator cannot have
both properties. The following example shows this. Suppose that an infrequently cited
journal is added to the bibliographic database that one uses. The property of
insensitivity to insignificant journals then requires that, because the newly added
journal is infrequently cited, the scores of all other journals remain more or less
unchanged. The property of insensitivity to field differences, on the other hand,
requires that the average score of the journals in a field remains unchanged. Hence, in
4 5
100
100
100
100
1000
1000
1000
1000
6
100
100
100
100
1000
1000
1000
1000
7 8
10
10
10
10
1
1
1
1
1
1
1
1
1
1
1
1
10
10
10
10
10
10
10
10
4 5 6 7 8
0.055
0.055
0.440
0.429
5.500
5.490
44.000
34.063
5.500
5.490
44.000
34.063
0.055
0.055
0.440
0.341
0.055

0.440