Ranking forestry journals using the h-index

Ranking forestry journals using the h-index

Jerome K Vanclay
Southern Cross University
PO Box 157, Lismore NSW 2480, Australia
JVanclay@scu.edu.au

Abstract
An expert ranking of forestry journals was compared with journal impact factors and h-indices
computed from ISI and internet-based data. Citations reported by Google Scholar appear to offer
the most efficient way to rank all journals objectively, in a manner consistent with other
indicators. A ranking of 180 forestry journals is presented, on the basis of an h-index computed
from data supplied by Google Scholar.
Keywords: Hirsch index, Research quality framework, Journal impact factor, forestry

Introduction

The Thomson ISI Journal Impact Factor (JIF, Garfield 1955) has been the dominant measure of
journal impact, and is often used to rank journals and gauge relative importance, despite several
recognised limitations (Hecht et al 1998, Moed et al 1999, van Leeuwen et al 1999, Saha et al
2003, Dong et al 2005, Moed 2005, Dellavalle et al 2007). Hirsch’s h-index (Hirsch 2005, van
Raan 2006, Bornmann and Daniel 2007) has been suggested as an alternative that may be more
reliable and robust (Miller 2006, Rousseau 2007, Vanclay 2007, 2008). This study presents an
analysis of the JIF, h-index, and other indicators of journal utility, with a view to ranking forestry
literature.

In preparation for the Australian government’s Research Quality Framework (RQF, Gale et al
2005, DEST 2007), professional bodies in Australia were asked to identify and rank relevant
journals within their discipline into 4 prestige bands, based on journal quality. Participants were
asked to allocate journals to one of four classes, representing the top 5 percentile (A1), the 80-95
percentile (A), the 50-80 percentile (B), and the residue (C). The classification offered by the
Institute of Foresters of Australia (pers comm., 21 November 2007) implied a ranking
substantially different to the JIF, even though the 2005 JIF data were available to members to
assist them in their classification. The wide range of JIFs within a class was noteworthy, as was
the disagreement regarding the top journal. This study attempts to shed some light on this
controversy.

Method
The study draws on journal rankings proposed by four individuals (not including the author),
nominated by and senior members of the Institute of Foresters of Australia, which was asked by
the Australian Academy of Technological Sciences and Engineering (ATSE 2007) to assist in
ranking forestry journals. The Institute of Foresters of Australia publishes one of the journals
under consideration, Australian Forestry. Three of the four experts ranked Australian Forestry as
A, one as B, and this study suggests that it is near the 76 percentile, in category B. The four
experts all had a PhD, and represented current or past heads of a university department, a national
research agency, a development assistance agency, and a consultancy firm.

This study also draws on Thomson ISI Journal Impact Factors from the 2006 Journal Citation
Reports, and on h-indices computed from two sources, the ISI Web of Science (version 4, WoS)
and Harzing’s (2007) Publish or Perish (PoP), a software package that harvests data from Google
Scholar (GS), a specialised internet search engine (Noruzi 2005, Pauly and Stergiou 2005, Meho
and Yang 2007). Hirsch’s h-indices were computed for several intervals, but the 8-year interval
2000-2007 seemed insightful for forestry journals. The h-indices computed from WoS and GS

data are similar (r = 0.93, n = 43 for 2000-2007 data), but the former are available only for ISI-
listed journals, whereas the latter can be computed for any journal or citation visible to Google
Scholar.

Results
Figure 1 illustrates the correspondence between classification allocated by experts and the JIF, for
each of the four contributors and the 28 journals recognised by both ATSE (2007) and ISI. There
was a considerable discrepancy between the assigned classification and the ISI ranking of core
forestry journals. The spread of points and the weak trend illustrate the magnitude of the
differences between experts and the ranking implied by the JIF. The shape of the trend (Figure 1)
is unsurprising, because the ISI data are censored to represent the top few journals (about 15%).
Although the h-index (2000-2007) is well correlated with the JIF (Figure 2; r=0.88 for PoP data,
r=0.91 for ISI data), it exhibits closer agreement with the expert assessment (r=0.62) than does
the JIF (r=0.56), suggesting that the h-index may be useful for ranking journals objectively. A
further advantage of the h-index is that it may be computed for the many journals not
acknowledged by the ISI.
0
0.5
1
1.5
2
2.5
3
0% 20% 40% 60% 80% 100%
Expert classification
IS
I J
o
u
rn
a
l I
m
pa
ct

Fa
ct
o
r

Figure 1. Journal impact factors contrasted with an expert classification of forestry journals by
four individuals into four classes (n=28, a small subset of the journals ranked).

010
20
30
40
0 1 2 3
ISI Journal Impact Factor (2006)
h-
in
de
x

(20
00
-
20
07
)

Figure 2. The relationship between the JIF and the PoP h-index (based on all citations accruing to
journal publications during 2000-2007). The filled point near the top of the figure is Forest
Ecology and Management. Journals not recognised by ISI are assumed to have a zero JIF.

Expert ranking of two journals, Agricultural and Forest Meteorology (AFM) and Forest Ecology
and Management (FEM), differed greatly to that implied by the JIF. The former has a higher JIF,
but experts ranked the latter as more influential, as did the h-index (Figure 2). Table 1 lists some
key differences between these journals: AFM is relatively low-volume and tends to be cited soon
after publication, whereas FEM has a higher volume and is slower to accrue citations. Overall, the
h-indices of the journals are comparable, but there is a tendency for WoS to report higher
statistics for AFM, and for PoP to report higher statistics for FEM. Superficial examination of
Table 1 may lead to the suggestion that AFM publishes relatively few papers all of which are
high-quality, reflecting a high editorial standard, and in turn, credit to any author who has a paper
accepted for publication (which is what the RQF seeks to achieve). However, this interpretation is
a simplistic, and warrants closer examination.

Table 1. Statistics for two forestry journals.
Indicator Agricultural and Forest
Meteorology
Forest Ecology and
Management
Panel assessment
Year established
A1/A2 (95 percentile)
1964
A1 (95-100 percentile)
1977
JIF (2006)
Immediacy
Cited half-life
Total articles
2.903
0.669
6.7
130
1.839
0.356
5.8
601
Lifetime h-index (ISI)
h-index 2005-6
h-index 2000-7
Total cites 2000-7
60
12
43
9 113
58
12
36
21 470
Lifetime h-index (PoP)
h-index 2005-6
Mean cites/paper 2005-6
h-index 2000-7
Total cites 2000-7
67
9
2.09
41
8 544
69
12
1.67
43
25 913

The RQF seeks a proxy for research quality, and assumes that acceptance and publication by a
journal indicates attainment of the standard indicated by the journal’s ranking. The JIF is not a
good indicator for this purpose, because it reflects the average number of citations, and may
conceal many ‘free-riders’ (Walter et al 2003). Table 2 examines this issue, year-by-year for the
last decade, and tabulates the proportion of papers in each journal that remain uncited (Weale et
al 2004), or fail to accrue at least one citation per year since publication. Despite its lower JIF,
FEM has a lower proportion of papers that remain uncited, or that remain infrequently cited, for
almost every year during the past decade, suggesting that by these yardsticks, FEM may be the
journal that reflects better on contributors. This conclusion from Table 2 is reflected in the h-
index, but not in the JIF (Table 1). Table 2 also illustrates that the h-index appears to plateau after
eight years (i.e., in 2000), at least for these two forestry journals.

Table 2. Annualised data for two forestry journals.
Agricultural and Forest Meteorology Forest Ecology and Management
Year h-index
(WoS)
h-index
(PoP)
% uncited
(annualized)
Not cited
>1/year
h-index
(WoS)
h-index
(PoP)
% uncited
(annualized)
Not cited
>1/year
2007 3 2
2006 6 4 49% 49% 7 6 62% 62%
2005 12 9 45% 39% 12 13 46% 38%
2004 18 15 54% 35% 16 18 47% 34%
2003 19 17 65% 48% 21 24 53% 33%
2002 20 17 71% 45% 26 29 55% 33%
2001 22 20 67% 42% 24 30 64% 38%
2000 24 27 71% 39% 30 35 64% 38%
1999 24 25 65% 34% 28 34 71% 40%
1998 21 23 70% 35% 31 34 73% 44%
Mean 62% 41% 59% 40%

0
20
40
60
80
100
120
140
0.001 0.01 0.1 1
Article rank (cumulative distribution)
Ci
ta
tio
n
s
ac
cr
u
ed

Figure 3. Citation accrual to two journals, Agricultural and Forest Meteorology (dotted) and
Forest Ecology and Management (solid), showing their h-index (•), using data from PoP.

Figure 3 illustrates the trends in citations to individual papers published in these two journals
during the year 2000. The publication year 2000 was chosen because it reflects the half-life of
these journals, and allows citation patterns to be fully expressed (Table 2; also Vanclay 2008).
Figure 3 reveals the number of citations for each paper in rank order, scaled to reflect the
cumulative distribution function because of a three-fold difference in the number of papers
published. A logarithm scale is used because the great majority of papers accrue few citations,
and exhibit a log-linear trend in their citation rate.

Figure 3 shows that the two journals have a very similar pattern of citation accrual to the majority
of contributions, and that it is only in the most-frequently-cited 10% of papers that differences in
citations appear. This equivalence is reflected in the h-indices (27 for AFM, 35 for FEM, PoP
data), but not in the JIFs of the two journals (Table 1), which create the impression that AFM is a
substantially better journal than FEM.

The log-linear trend in citation accrual (Figure 3) appears generic, applies to many journals, and
is neatly summarised by the h-index, since it reflects the gradient of this relationship. Fewer than
h papers (where h is the h-index) depart from this trend, and appear to reflect the fortunate
juxtaposition of ready accessibility and a topical issue, rather than research quality per se. The
pattern revealed in Figure 3 leads to the suggestion that a classification of journals based on the h-
index provides a better indicator for the RQF than the JIF. Figure 3 implies that the median
journal contribution will be cited about h/3 times, an estimate that (unlike the JIF) is unaffected
by the few papers that are frequently cited. A further advantage is that it can be calculated quickly
and easily (e.g., with the PoP software, Harzing 2007) for all journals, including those not
recognised by Thomson-ISI. Figure 2 indicates that there are many frequently-cited journals that
are not recognised by the ISI.

Table 3. Papers contributing to the PoP h-index, but excluded from the ISI h-index (2000-2007)
for Forest Ecology and Management.
Cites Authors Title Year
114
97
72
61
58
56
54
52
51
51
48
48
48
47
46
46
45
43
43
de Vries et al
Guariguata, Ostertag
Marcot et al
Swank et al
Schoenholtz et al
Ripple, Beschta
Gardiner, Quine
Tiedemann et al
Vesterdal et al
Griffis et al
Liski et al
Knoepp et al
Bowman et al
Fule et al
Ketterings et al
Emborg et al
Pretzsch et al
Kavvadias et al
Yanai et al
Intensive monitoring of forest ecosystems in Europe …
Neotropical secondary forest succession …
Using Bayesian belief networks to evaluate fish and wildlife …
Long-term hydrologic and water quality responses …
A review of chemical and physical properties as indicators of forest soil …
Wolf reintroduction, predation risk, and cottonwood recovery …
Management of forests to reduce the risk of abiotic damage …
Solution of forest health problems with prescribed fire …
Change in soil organic carbon following afforestation …
Understory response to management treatments in northern Arizona …
Increasing carbon stocks in the forest soils of western Europe.
Biological indices of soil quality: an ecosystem case study of their use.
The association of small mammals with coarse woody debris …
Comparing ecological restoration alternatives …
Reducing uncertainty in the use of allometric biomass equations …
The structural dynamics of Suserup Skov …
The single tree-based stand simulator SILVA …
Litterfall, litter accumulation and litter decomposition rates …
Challenges of measuring forest floor organic matter dynamics …
2003
2001
2001
2001
2000
2003
2000
2000
2002
2001
2002
2000
2000
2002
2001
2000
2002
2001
2000


Tables 1 and 2, and Figure 3 suggest that AFM and FEM are similar in many regards, but Figure
2 highlights the large discrepancy between the JIF and the h-index for these two journals. The
total number of citations reported in Table 1 may shed some light on this difference. AFM
appears to service to a specialised audience that is more visible to ISI than to Google Scholar. In
contrast, FEM is cited in a substantial number of non-academic publications visible to Google
Scholar, which reports 20% more citations than ISI (Table 1). An analysis of the differences in
citation patterns for these 20,000 citations is a formidable task, but an insight may be gained by
examining the differences in the few papers that contribute to the h-index estimated from ISI and
PoP records. The FEM papers contributing to the ISI h-index (2000-2007) of 36 are not a
complete subset of those contributing to the PoP h-index of 43, so there are 19 papers

contributing to the PoP h-index but not the ISI h-index (Table 3). These 19 papers were cited a
total of 1022 times, half of which (according to GS) accrued from ISI-listed journals, and the
remainder from various sources including academic and government publications (Table 4). In
the case of these 19 papers, there are at least as many citations from non-ISI sources as there are
from ISI-listed journals. In this particular example, all these citations appear to bona fide and
draw upon, rather than criticise the cited works. The citation of these FEM papers in academic
theses and government reports (Table 4) suggests that FEM reaches practitioners as well as
researchers. Although unproven, the difference in ratio of PoP:ISI h-indices (0.94 for AFM and
1.2 for FEM) seems to suggest that AFM is cited mainly by (and hence likely to be used mainly
by) researchers, while the higher ratio for FEM may indicate greater uptake by practitioners.

Table 4. Sources of citations contributing to the PoP h-index but not to the ISI h-index (2000-
2007) for Forest Ecology and Management.
Source of citation Cites
ISI-listed journals (including FEM self-citations 9%) 49%
Academic publications (including theses 10%) 15%
Journals not listed by ISI (mostly refereed) 12%
Government publications 12%
Books 6%
Conferences proceedings and presentations 3%
Publications by NGOs and associations 3%
Consultants reports and other commercial documents 1%
Total 100%

Discussion
There is no doubt that an h-index based on Google Scholar is imperfect, because it can be
manipulated with bogus documents on personal websites, and may be inflated by provocative
contributions (such as A.D. Sokal’s satirical 1996 contribution to Social Text, for which ISI
records 18 citations, compared to 339 citations recorded by Google Scholar). However, the ISI

JIF is also imperfect, because it is available only for journals selected by ISI, and because of
limitations in the calculation of the JIF (Jacso 2001, Dong et al 2005, Vanclay 2008).

The appendix offers a list of 180 forestry journals that have been cited at least once since 2000,
and appear to be central to forestry research and practice. This list has been compiled from the
Thomson ISI list, the Forest Science Database, Ulrich’s Periodicals Directory, JournalSeek and
Metla’s Virtual Forestry library, and was further supplemented with Google Scholar searches for
journals with a high frequency of forestry terms. The list was then culled to remove non-core
forestry material, by removing titles that infrequently mentioned core forestry terms (such as
forestry, silviculture, wood and timber). Google Scholar makes it easy to identify such journals
efficiently, and to judge objectively whether or not a journal is central to a discipline. The list was
ranked using h-indices computed by PoP (and for Tree-Ring Research, manually from GS data).
RQF classifications (A1, A, B, C) were assigned to the 180 journals cited more than once during
2000-2007.

Conclusion
The ranked list of journals provided in the appendix has several implications. Thomson-ISI may
wish to recognise more of the high-ranked journals (such as Dendrochronologia with h-index 11),
editors of some journals may wish to work with Google to make their contents more visible to
search engines (e.g., Ann. Forest Sci. which is not recognised by Google as Annals of Forest
Science, and Tree-Ring Research which does not provide the date of publication in Google-
readable format), and editors of journals not published in English (which are disadvantaged in
internet searches) may wish to add English abstracts and keywords to raise their profile.

Because of its broader coverage, Hirsch’s h-index based on Google Scholar data may be more
useful than the ISI Journal Impact Factor, as a measure of journal quality, and in providing a basis
to rank journals.

References
ATSE, 2007. RQF Journal Ranking Project. Australian Academy of Technological Sciences and
Engineering. http://www.atse.org.au/index.php?sectionid=1096 [7 December 2007].
Bornmann, L. and H.-D. Daniel, 2007. What Do We Know About the h Index? Journal of the
American Society for Information Science and Technology 58:1381–1385.
Dellavalle, R.P., L.M. Schilling, M.A. Rodriguez, H. Van de Sompel and J. Bollen, 2007.
Refining dermatology journal impact factors using PageRank. Journal of the American
Academy of Dermatology 57: 116-119.
DEST, 2007. Research Quality Framework. Department of Education, Employment and
Workplace Relations, Australian Government,
http://www.dest.gov.au/sectors/research_sector/policies_issues_reviews/key_issues/resea
rch_quality_framework/ [2 December 2007].
Dong, P., M. Loh and A. Mondry, 2005. The "impact factor" revisited. Biomedical Digital
Libraries 2:7 doi:10.1186/1742-5581-2-7
Garfield, E., 1955. Citation indexes for science. Science 122:108-111.
Gale, T., R. Gilbert, T. Seddon and J. Wright, 2003. The RQF and educational research in
Australia: Implications and responses across the research community. Paper presented to
the Annual Conference of the Australian Association for Research in Education, Sydney,
November 2005. http://www.aare.edu.au/05pap/gil05744.pdf
Harzing, A.W., 2007. Publish or Perish. http://www.harzing.com/pop.htm [7 December 2007].
Hecht, F., B.K. Hecht and A.A. Sandberg, 1998. The journal impact factor: A misnamed,
misleading, misused measure. Cancer Genetics and Cytogenetics 104:77-81.
Hirsch, J.E., 2005. An index to quantify an individual’s scientific research output. Proceedings of
the National Academy of Sciences 102:16569–16572.
Jacso, P., 2001. A deficiency in the algorithm for calculating the impact factor of scholarly
journals: The journal impact factor. Cortex 37:590-594.

Meho, L.I. and K. Yang, 2007. Impact of Data Sources on Citation Counts and Rankings of LIS
Faculty: Web of Science Versus Scopus and Google Scholar. Journal of the American
Society for Information Science and Technology 58(13):2105–2125.
Miller, C.W., 2006. Superiority of the h-index over the Impact Factor for Physics.
arXiv:physics/0608183v1
Moed, H.F., 2005. Citation analysis of scientific journals and journal impact measures. Current
Science 89:1990-1996.
Moed, H.F., T.N. van Leeuwen and J. Reedijk, 1999. Towards appropriate indicators of journal
impact. Scientometrics 46:575-589.
Noruzi, A., 2005. Google Scholar: The new generation of citation indexes. Libri 55:170–180.
Pauly, D. and K.I. Stergiou, 2005. Equivalence of results from two citation analyses: Thomson
ISI’s Citation Index and Google’s Scholar service. Ethics in Science and Environmental
Politics 2005:33–35
Rousseau, R., 2007. The influence of missing publications on the Hirsch index. Journal of
Informetrics 1:2–7.
Saha, S., S. Saint and D.A. Christakis, 2003. Impact factor: a valid measure of journal quality?
Journal of the Medical Library Association 91:42-46.
Vanclay, J.K., 2007. On the robustness of the h-index. Journal of the American Society for
Information Science and Technology 58(10):1547-1550.
Vanclay, J.K., 2008. Bias in the Journal Impact Factor. Scientometrics in press (accepted 15 Dec
2006).
van Leeuwen, T.N., H.F. Moed and J. Reedijk, 1999. Critical comments on Institute for Scientific
Information impact factors: a sample of inorganic molecular chemistry journals. Journal
of Information Science 25(6):489-498.
van Raan, A.F.J., 2006. Comparison of the Hirsch-index with standard bibliometric indicators and
with peer judgment for 147 chemistry research groups. Scientometrics 67:491-502.

Walter, G., S. Bloch, G. Hunt and K. Fisher, 2003. Counting on citations: a flawed way to
measure quality. The Medical Journal of Australia 178: 280-281.
Weale, A., M. Bailey and P. Lear, 2004. The level of non-citation of articles within a journal as a
measure of quality: a comparison to the impact factor. BMC Medical Research
Methodology 4:14.

Appendix. Ranked list of 180 selected forestry journals.
Full Title JIF h-index
2000-7
Class
Forest Ecology and Management 1.839 43 A1
Agricultural and Forest Meteorology 2.903 41 A1
Journal of Vegetation Science 2.382 29 A1
Tree Physiology 2.297 28 A1
Plant Ecology (Vegetatio) 1.383 27 A1
Canadian Journal of Forest Research 1.549 23 A1
Forest Science 1.457 23 A1
Journal of Forestry 1.188 23 A1
Trees Structure and Function 1.461 22 A1
International Journal of Wildland Fire 1.679 21 A1
Annals of Forest Science 1.290 19 A
Agroforestry Systems 0.921 19 A
Agricultural and Forest Entomology 1.473 18 A
Scandinavian Journal of Forest Research 0.868 18 A
Holzforschung 1.014 17 A
Forest Policy and Economics 0.907 17 A
Forestry 0.847 16 A
Holz als Roh- und Werkstoff 0.514 15 A
Applied Vegetation Science 1.214 14 A
Silva Fennica 0.878 14 A
Forest Products Journal 0.387 14 A
Forestry Chronicle 0.831 13 A
Wood Science and Technology 0.740 13 A
International Forestry Review 0.618 12 A
Journal of Wood Science 0.574 12 A
Forest Pathology 0.729 11 A
New Forests 0.681 11 A
Dendrochronologia
11 A
Unasylva
11 A
Wood and Fiber Science 0.540 10 A
Revista Arvore
10 A
Journal of Wood Chemistry and Technology 1.000 9 B
Southern Journal of Applied Forestry 0.704 9 B
Tree-Ring Research 0.625 9 B
Silvae Genetica 0.311 9 B
European Journal of Forest Pathology
9 B

Journal of Forest Economics
9 B
IAWA Journal 0.667 8 B
Western Journal of Applied Forestry 0.515 8 B
Forests, Trees and Livelihoods
8 B
Forstwissenschaftliches Centralblatt (German Journal of Forest Science)
8 B
Urban Forestry & Urban Greening
8 B
Nordic Pulp & Paper Research Journal 0.562 7 B
Appita Journal 0.301 7 B
Journal of Tropical Forest Science 0.160 7 B
Australian Forestry
7 B
Forest Genetics
7 B
Journal of Sustainable Forestry
7 B
Linye Kexue (Scientia Silvae Sinicae)
7 B
Small-Scale Forestry
7 B
Tasforests
7 B
Northern Journal of Applied Forestry 0.779 6 B
European Journal of Forest Research 0.776 6 B
Allgemeine Forst- und Jagdzeitung 0.315 6 B
Ciencia Florestal
6 B
Forst und Holz
6 B
International Journal of Forest Engineering
6 B
Investigacion Agraria. Sistemas y Recursos Forestales
6 B
Journal of Forest and Livelihood
6 B
Journal of Forest Research
6 B
Scientia Forestalis
6 B
Mokuzai Gakkaishi (Journal of the Japan Wood Research Society) 0.168 5 B
Bois et Forets des Tropiques
5 B
Cerne
5 B
Dendrobiology
5 B
Floresta e Ambiente
5 B
Forest Snow and Landscape Research
5 B
Journal of Beijing Forestry University
5 B
Journal of Forest Science
5 B
Journal of the Japanese Forestry Society
5 B
L'italia Forestale e Montana
5 B
Revue Forestiere Francaise
5 B
American Forests
4 C
Baltic Forestry
4 C
Floresta
4 C