Page 1

BioMed Central

Page 1 of 10

(page number not for citation purposes)

BMC Medical Research

Methodology

Open Access

Correspondence

Assessing the impact of biomedical research in academic

institutions of disparate sizes

Vana Sypsa and Angelos Hatzakis*

Address: Department of Hygiene and Epidemiology, Athens University Medical School, Athens, Greece

Email: Vana Sypsa - vsipsa@cc.uoa.gr; Angelos Hatzakis* - ahatzak@med.uoa.gr

* Corresponding author

Abstract

Background: The evaluation of academic research performance is nowadays a priority issue. Bibliometric

indicators such as the number of publications, total citation counts and h-index are an indispensable tool

in this task but their inherent association with the size of the research output may result in rewarding high

production when evaluating institutions of disparate sizes. The aim of this study is to propose an indicator

that may facilitate the comparison of institutions of disparate sizes.

Methods: The Modified Impact Index (MII) was defined as the ratio of the observed h-index (h) of an

institution over the h-index anticipated for that institution on average, given the number of publications

(α and β denote the intercept and the slope, respectively, of the line

MII =

N

(N) it produces i.e.

describing the dependence of the h-index on the number of publications in log10 scale). MII values higher

than 1 indicate that an institution performs better than the average, in terms of its h-index. Data on

scientific papers published during 2002–2006 and within 36 medical fields for 219 Academic Medical

Institutions from 16 European countries were used to estimate α and β and to calculate the MII of their

total and field-specific production.

Results: From our biomedical research data, the slope β governing the dependence of h-index on the

number of publications in biomedical research was found to be similar to that estimated in other disciplines

(≈0.4). The MII was positively associated with the average number of citations/publication (r = 0.653, p <

0.001), the h-index (r = 0.213, p = 0.002), the number of publications with ≥ 100 citations (r = 0.211, p =

0.004) but not with the number of publications (r = -0.020, p = 0.765). It was the most highly associated

indicator with the share of country-specific government budget appropriations or outlays for research and

development as % of GDP in 2004 (r = 0.229) followed by the average number of citations/publication (r

= 0.153) whereas the corresponding correlation coefficient for the h-index was close to 0 (r = 0.029). MII

was calculated for first 10 top-ranked European universities in life sciences and biomedicine, as provided

by Times Higher Education ranking system, and their total and field-specific performance was compared.

Conclusion: The MII should complement the use of h-index when comparing the research output of

institutions of disparate sizes. It has a conceptual interpretation and, with the data provided here, can be

computed for the total research output as well as for field-specific publication sets of institutions in

biomedicine.

Published: 29 May 2009

BMC Medical Research Methodology 2009, 9:33 doi:10.1186/1471-2288-9-33

Received: 12 September 2008

Accepted: 29 May 2009

This article is available from: http://www.biomedcentral.com/1471-2288/9/33

© 2009 Sypsa and Hatzakis; licensee BioMed Central Ltd.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0),

which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

h

10αβ

Page 2

BMC Medical Research Methodology 2009, 9:33 http://www.biomedcentral.com/1471-2288/9/33

Page 2 of 10

(page number not for citation purposes)

Background

Bibliometric indices are an indispensable tool in evaluat-

ing the research output of individuals and institutions.

Recently, novel indicators have been proposed with the

aim to overcome deficiencies of the "traditional" biblio-

metric indices (e.g. number of publications, total citation

count, average number of citations per publication) and

to combine more efficiently information on both the

quantity and the quality of the research output [1-4]. H-

index is the most known example of such an indicator [1]

and is now routinely provided by Thomson Scientific Web

of Science and other bibliometric databases. This indica-

tor is defined as the number h of papers of an individual

or an institution with number of citations higher or equal

to h. As a result, it combines information on both the

number of papers and the number of citations. However,

due to its inherent association with the size of the research

output it may result in rewarding institutions with high

production [2]. Thus, when comparing institutions, a

proper calibration of the h-index for the size of the output

may provide additional information.

Recenlty, it has been shown that when evaluating sets of

publications ranging from several hundreds to 105 papers,

the dependence of the h-index on the size of the set is

characterised by a "universal" growth rate [2]. This was

shown for interdisciplinary, mechanics and materials sci-

ence data [2] as well as for nonbiomedical research data

[5]. Thus, the h-index can be decomposed into the prod-

uct of a factor depending on the population size and of an

impact index. This impact index can be used to compare

the research output of institutions of disparate number of

publications. However, as most bibliometric indicators,

the impact index of an institution is not informative on its

own, unless it is compared to the corresponding indices of

other institutions. Furthermore, Molinari and Molinari

[2] have provided parameter estimates to calculate this

index only for a large number of papers and therefore, it

cannot be extended to assess the impact in e.g. specific

fields where the sets of publications range on a much

lower scale.

In the present study we aim to extend the interpretation of

the h-index by proposing a size-corrected, h-index based

indicator (Modified Impact Index – MII). The concept of

this index is to assess whether the h-index of an institution

deviates from the average h-index, as estimated for a par-

ticular number of publications. MII shares all the merits of

the impact index. Additionally, we will show that it has a

more informative numerical interpretation and, with the

data that we will provide in the following sections, it may

be used also in the case of smaller publication sets. We

will illustrate the use of this index in biomedical research

and explore its application within specific biomedical dis-

ciplines.

Methods

The Academic Medical Institutions located in 16 Euro-

pean countries (Austria, Belgium, Denmark, Finland,

France, Germany, Greece, Ireland, Italy, Netherlands,

Norway, Portugal, Spain, Sweden, Switzerland, United

Kingdom) were identified from the database of medical

schools provided by the Institute for International Medi-

cal Education [6]. Once the final list of 219 institutions

was compiled, all publications affiliated to the corre-

sponding universities (excluding meeting abstracts) and

classified into any of the 36 pre-specified medical subjects

(Table 1) were identified using Thomson Scientific Web of

Science (WoS). The number of papers published during

2002–2006 and the corresponding h-index have been

recorded for each institution. Two databases have been

constructed; one with data on all publications within the

36 medical fields and a second with data on publications

from each medical field separately. The intercept α and

slope β of the line describing the dependence of h-index

on the number of publications (log10 scale) were obtained

through least-squares estimation.

The impact index of each institution was calculated as

h

N

where h: h-index and N: number of publica-

tions. As Molinari and Molinari have shown in their paper

[2], the slope β of 0.4 estimated when accumulating data

on h-index over time is similar to the slope of the regres-

sion line obtained from cross-sectional data (e.g. in their

paper: h-index per country as calculated in 2006 vs. the

corresponding number of publications). Thus, we used

the latter approach and estimated the impact index of

papers published within 2002–2006 using the slope β

obtained from our data on 219 institutions.

To illustrate our findings, we used the rankings provided

by Times Higher Education to select top-ranked European

universities in life sciences and biomedicine [7].

Results

Modified Impact Index (MII) in biomedical research

When the h-index of each institution was plotted against

the corresponding number of papers from 36 medical

fields on a log-log plot, the resulting points were fitted by

a regression line (Figure 1):

where hi and Ni the h-index and the number of publica-

tions of the ith institution, respectively, α and β the inter-

cept and the slope of the regression line and εi the ith

residual. The estimated α and β were 0.207 and 0.445,

respectively. The parameter β = 0.445 governing the

dependence of h-index on the number of publications in

hm

=

β

loglog

10 10

hN

iii

=++αβε

(1)

Page 3

BMC Medical Research Methodology 2009, 9:33http://www.biomedcentral.com/1471-2288/9/33

Page 3 of 10

(page number not for citation purposes)

Table 1: List of 36 medical subjects included in the evaluation along with the estimated α s and β s (as obtained from data on

publications of 219 European Academic Medical Institutions within 2002–2006) for the calculation of the modified impact index

MII =

N

( where h: h-index, N: number of publications)

Subject Intercept α

Slope β

1 Allergy-0.0330.668

2Anatomy & Morphology -0.0580.623

3 Anesthesiology-0.0160.554

4 Cardiac & Cardiovascular Systems-0.0040.600

5 Chemistry, Medicinal0.0670.563

6 Clinical Neurology0.0270.545

7 Critical Care Medicine0.053 0.594

8 Dermatology-0.031 0.560

9 Emergency Medicine -0.016 0.498

10Endocrinology & Metabolism 0.0980.560

11 Gastroenterology & Hepatology0.0280.592

12 Geriatrics & Gerontology0.049 0.558

13 Health Care Sciences & Services-0.0220.538

14 Hematology0.0460.614

15Immunology0.162 0.528

16 Infectious Diseases0.075 0.566

17Medicine, General & Internal -0.1240.644

18 Medicine, Research & Experimental-0.0080.621

19Obstetrics & Gynecology 0.040 0.521

20Oncology 0.2050.500

21 Ophthalmology-0.0550.581

22 Orthopedics-0.0530.555

23 Otorhinolaryncology 0.0040.488

24 Pathology -0.0420.621

25 Pediatrics -0.0270.528

26 Peripheral Vascular Disease0.022 0.616

h

10αβ

Page 4

BMC Medical Research Methodology 2009, 9:33http://www.biomedcentral.com/1471-2288/9/33

Page 4 of 10

(page number not for citation purposes)

biomedical research was found to be similar to that esti-

mated in other disciplines (≈0.4). The number of publica-

tions ranged from 102 to 104 papers, with the exception of

one institution with very low number of publications. The

exclusion of this institution did not alter the estimated

slope. Our estimate for β in biomedical sciences was con-

sistent among different countries (Figure 2).

The fitted regression line of equation (1) provides the

average h-index for a particular number of publications.

Thus, points above the regression line correspond to insti-

tutions with h-index higher than the average. Similarly,

points below the regression line correspond to institu-

tions with h-index lower than the average. The difference

log10 hi - (α + β log10 Ni) between the observed log10 hi

(denoted as circles in the Figure 1 and 2) and the corre-

sponding fitted value α + β log10 Ni (superimposed regres-

sion line) expresses the deviation εi of the observed h-

index of the ith institution from the average estimate for

the number of publications it produces. In the original

scale, this difference is transformed into the ratio .

This ratio expresses how many times the observed h-index

is higher than that estimated by the regression model

based on the number of publications. Thus, a value higher

than 1 indicates that the particular institution performs

better in terms of h-index than it would be expected for

the number of publications it produces. Similarly, a value

lower than 1 indicates that the particular institution per-

forms worse in terms of h-index than it would be expected

for the number of publications it produces. The ratio

h

N

was found to be equivalent to the impact index

proposed by Molinari and Molinari [2] multi-

plied by the constant and was therefore named Mod-

hi

Ni10α

β

10αβ

hm

h

N

=

β

1

10α

27Physiology0.0620.565

28 Psychiatry-0.012 0.566

29Public, Environmental & Occupational Health 0.0200.535

30Radiology, Nuclear Medicine & Medical Imaging0.0010.560

31Respiratory System -0.0250.607

32 Rheumatology-0.0060.638

33 Surgery 0.0700.490

34Transplantation 0.0060.572

35Tropical Medicine 0.003 0.595

36Urology & Nephrology -0.0120.594

Table 1: List of 36 medical subjects included in the evaluation along with the estimated α s and β s (as obtained from data on

publications of 219 European Academic Medical Institutions within 2002–2006) for the calculation of the modified impact index

MII =

N

( where h: h-index, N: number of publications) (Continued)

h

10αβ

Log-log plot of h-index versus the total number of results found in 219 Medical schools from 16 European countries

Figure 1

Log-log plot of h-index versus the total number of

results found in 219 Medical schools from 16 Euro-

pean countries. The solid line indicates the fitted regres-

sion line and β indicates its slope.

Number of publications 2002-2006 (log10scale)

10

100 1000 10000

5

20

40

60

80

100

h-index ((log10scale)

All 219 Academic

Medical Institutions

?=0.445

Page 5

BMC Medical Research Methodology 2009, 9:33http://www.biomedcentral.com/1471-2288/9/33

Page 5 of 10

(page number not for citation purposes)

ified Impact Index (MII). The variance of the MII can be

computed as follows. In log10 scale:

In the original scale

lognormal

and thus it follows the

From standard distribution. theory,

. Based on the data col-

lected from 219 European Medical Institutions, Var(MII)

was estimated to be equal to 0.013475.

We explored the validity of the MII by examining its asso-

ciation with other indices. The MII was positively associ-

ated with the average number of citations/publication

(Spearman's r = 0.653, p < 0.001), the h-index (r = 0.213,

p = 0.002), the number of publications with ≥ 100 cita-

tions (r = 0.211, p = 0.004) but not with the number of

publications (r = -0.020, p = 0.765). We further examined

log log(log)~ ( ,0)

10 1010

2

MII =−+=

hNN

iii

αβεσ

MII =10εi

Varee

()()

(ln) (ln)

MII =−

10 10

22

1

σσ

Log-log plot of h-index versus the total number of results by country (including countries with more than 10 Academic Medical Institutions)

Figure 2

100100010000

10

20

40

60

80

100

120

Number of publications 2002-2006 (log10scale)

h-index ((log10scale)

France

?=0.477

100 100010000

10

20

40

60

80

100

120

Number of publications 2002-2006 (log10scale)

h-index ((log10scale)

Germany

?=0.439

100100010000

10

20

40

60

80

100

120

Number of publications 2002-2006 (log10scale)

h-index ((log10scale)

Italy

?=0.431

Number of publications 2002-2006 (log10scale)

100100010000

10

20

40

60

80

100

120

Spain

?=0.434

h-index ((log10scale)

100100010000

10

20

40

60

80

100

120

Number of publications 2002-2006 (log10scale)

h-index ((log10scale)

UK

?=0.433