Assessing the impact of biomedical research in academic institutions of disparate sizes

BMC Medical Research Methodology (Impact Factor: 2.27). 05/2009; 9(1):33. DOI: 10.1186/1471-2288-9-33
Source: PubMed


The evaluation of academic research performance is nowadays a priority issue. Bibliometric indicators such as the number of publications, total citation counts and h-index are an indispensable tool in this task but their inherent association with the size of the research output may result in rewarding high production when evaluating institutions of disparate sizes. The aim of this study is to propose an indicator that may facilitate the comparison of institutions of disparate sizes.
The Modified Impact Index (MII) was defined as the ratio of the observed h-index (h) of an institution over the h-index anticipated for that institution on average, given the number of publications (N) it produces i.e. MII = h/10alphaNbeta (alpha and beta denote the intercept and the slope, respectively, of the line describing the dependence of the h-index on the number of publications in log10 scale). MII values higher than 1 indicate that an institution performs better than the average, in terms of its h-index. Data on scientific papers published during 2002-2006 and within 36 medical fields for 219 Academic Medical Institutions from 16 European countries were used to estimate alpha and beta and to calculate the MII of their total and field-specific production.
From our biomedical research data, the slope beta governing the dependence of h-index on the number of publications in biomedical research was found to be similar to that estimated in other disciplines ( approximately 0.4). The MII was positively associated with the average number of citations/publication (r = 0.653, p < 0.001), the h-index (r = 0.213, p = 0.002), the number of publications with > or = 100 citations (r = 0.211, p = 0.004) but not with the number of publications (r = -0.020, p = 0.765). It was the most highly associated indicator with the share of country-specific government budget appropriations or outlays for research and development as % of GDP in 2004 (r = 0.229) followed by the average number of citations/publication (r = 0.153) whereas the corresponding correlation coefficient for the h-index was close to 0 (r = 0.029). MII was calculated for first 10 top-ranked European universities in life sciences and biomedicine, as provided by Times Higher Education ranking system, and their total and field-specific performance was compared.
The MII should complement the use of h-index when comparing the research output of institutions of disparate sizes. It has a conceptual interpretation and, with the data provided here, can be computed for the total research output as well as for field-specific publication sets of institutions in biomedicine.

9 Reads
  • Source
    • "Articles were restricted to those published between January 1, 1998, and December 31, 2012, to help ensure relevancy to the contemporary understanding of translational research. We excluded articles that assessed research impact solely on the basis of bibliographic considerations, for example, the impact factor of published research articles (e.g., Rosas, Kagan, Schouten, Slack, & Trochim, 2011; Sypsa & Hatzakis, 2009). Nearly 100 articles matched one or more of these criteria. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Increasing emphasis is being placed on measuring return on research investment and determining the true impacts of biomedical research for medical practice and population health. This article describes initial progress on development of a new standardized tool for identifying and measuring impacts across research sites. The Translational Research Impact Scale (TRIS) is intended to provide a systematic approach to assessing impact levels using a set of 72 impact indicators organized into three broad research impact domains and nine subdomains. A validation process was conducted with input from a panel of 31 experts in translational research, who met to define and standardize the measurement of research impacts using the TRIS. Testing was performed to estimate the reliability of the experts' ratings. The reliability was found to be high (ranging from .75 to .94) in all of the domains and most of the subdomains. A weighting process was performed assigning item weights to the individual indicators, so that composite scores can be derived.
    Evaluation &amp the Health Professions 09/2013; 37(1). DOI:10.1177/0163278713506112 · 1.91 Impact Factor
  • Source
    • "They are worried about how the current system rewards raw production rather than 'scholarship that addresses the questions that matter most to society'. Due to the importance of studying each subject field separately; as most rankings focus on top universities using elitist indicators, and because of the concerns expressed by some authors in regard with the inadequacy of most bibliometric indicators when applied to 772 D. Torres-Salinas et al. smaller size institutions (Sypsa and Hatzakis 2009); a new methodology is needed. Said methodology should take into account these factors in order to create rankings that allow the comparison between universities that cannot reach tops 100, 250 or 500 worldwide, and do so in a given subject field. "
    [Show abstract] [Hide abstract]
    ABSTRACT: The problem of comparing academic institutions in terms of their research production is nowadays a priority issue. This paper proposes a relative bidimensional index that takes into account both the net production and the quality of it, as an attempt to provide a comprehensive and objective way to compare the research output of different institutions in a specific field, using journal contributions and citations. The proposed index is then applied, as a case study, to rank the top Spanish universities in the fields of Chemistry and Computer Science in the period ranging from 2000 until 2009. A comparison with the top 50 universities in the ARWU rankings is also made, showing the proposed ranking is better suited to distinguish among non-elite universities.
    Scientometrics 09/2011; 88(3). DOI:10.1007/s11192-011-0418-6 · 2.18 Impact Factor
  • Source
    • "The search for publications identified 199 studies published between 2005 and 2010 as article (n = 172), grey literature (n = 15), conference proceeding (n = 11), or book section (n = 1). Thirty-five out of the 199 publications reported all information required for the meta-analysis presented here: (1) at least one correlation coefficient between the h index and an h index variant, and (2) the sample size of the study (Antonakis & Lalive, 2008; Arencibia-Jorge & Rousseau, 2009; Bornmann, Marx, & Schier, 2009; Bornmann, Mutz, & Daniel, 2008; Bornmann, Mutz, Daniel, Wallon, & Ledin, 2009; Cabrerizo, Alonso, Herrera- Viedma, & Herrera, 2010; Costas & Bordons, 2008; de Visscher, 2010; Franceschet, 2009; García-Pérez, 2009; Harzing & van der Wal, 2008; Haslam & Laham, 2010; Hu et al., 2010; Hua, Wan, & Wu, 2010; Jin, Liang, Rousseau, & Egghe, 2007; Kosmulski, 2006; Lee, Kraus, & Couldwell, 2009; Liu & Rousseau, 2007, 2009; Lovegrove & Johnson, 2008; Mingers, 2009; Moussa & Touzani, 2010; Opthof & Wilde, 2009; Rousseau et al., 2010; Ruane & Tol, 2007; Sanderson, 2008; Schreiber, 2008a, 2009a, 2009b; Schubert, Korn, & Telcs, 2009; Sypsa & Hatzakis, 2009; Tol, 2009; Vinkler, 2009; Wohlin, 2009; Wu, 2010). Since three papers by Schreiber (2008a, 2009a, 2009b) and two papers by Liu and Rousseau (2007, 2009) refer to one and the same dataset, the number of studies that could be included in the meta-analysis decreased from 35 to 32. "
    [Show abstract] [Hide abstract]
    ABSTRACT: This paper presents the first meta-analysis of studies that computed correlations between the h index and variants of the h index (such as the g index; in total 37 different variants) that have been proposed and discussed in the literature. A high correlation between the h index and its variants would indicate that the h index variants hardly provide added information to the h index. This meta-analysis included 135 correlation coefficients from 32 studies. The studies were based on a total sample size of N=9005; on average, each study had a sample size of n=257. The results of a three-level cross-classified mixed-effects meta-analysis show a high correlation between the h index and its variants: Depending on the model, the mean correlation coefficient varies between .8 and .9. This means that there is redundancy between most of the h index variants and the h index. There is a statistically significant study-to-study variation of the correlation coefficients in the information they yield. The lowest correlation coefficients with the h index are found for the h index variants MII and m index. Hence, these h index variants make a non-redundant contribution to the h index.
    Journal of Informetrics 07/2011; 5(3):346-359. DOI:10.1016/j.joi.2011.01.006 · 4.23 Impact Factor
Show more

Preview (2 Sources)

9 Reads
Available from