Content uploaded by Anna Guillaumet
Author content
All content in this area was uploaded by Anna Guillaumet on Feb 12, 2020
Content may be subject to copyright.
New metrics for the research evaluation. January 2020
New metrics for the research evaluation within the
OpenScience scenario.
Anna Guillaumet
UOC 2020
Anna.guillaumet@gmail.com
Abstract
Today, it’s clear that with the advance of the OpenScience and the digital publication scenario,
the entire research system is undergoing very significant changes.
One of the most controversial issues is the evaluation of science that should experiment
significant changes.
Thus there are many critical voices about the metrics for science evaluation that have
maintained their hegemony for many years.
It seems clear that the metrics from commercial publishers are not aligned with the objectives
of OpenScience and are not enough for the current research evaluation needs.
In this article we will see the main drawbacks of traditional metrics and alternative systems for
research evaluation such the research inquiry, among others, that aims to evaluate more the
qualitative research than the quantitative research.
Key Words
Metrics, Bibliometrics, openScience, science, evaluation, researcher profile, citations, impact, H-
index, evaluative inquiry.
Introduction
The knowledge generation plays a key role in the mankind evolution. It’s so relevant that one
aspect that has more controversial and studies are its evaluation. Lots of studies and analysis
arise from the research evaluation and how it should be done to assure the quality of the
research results, especially with the latest advances of the digital environments and the open
Science. But research evaluation it’s not new. Statistical analysis to measure science would
appear in 1917 with Cole and Eales and it was not until 1955 when Eugene Garfield (Wouters,
2017) proposed the concept of Science Citation Index that allows quantitative analyses of
research outputs and citations to evaluate funding programs, research groups, individuals and
nations. The impact factor had been computed for selected journals in the SCI from early’s
1960’s. And it was not until 1969 that Alan Pritchard used the bibliometric term for the first
time. It is also relevant that in 1978 Scientometrics magazine was founded, a publication in the
New metrics for the research evaluation. January 2020 1
field of metrics that deals with the quantitative aspects of science, communication and scientific
policies on an international scale. (Chaviano, 2004).
The appearance of the Institute for Scientific Information (ISI) of the United States in 1954 with
the leadership of Eugene Garfield, was fundamental for the development of what we have
called until now, modern science. The elements incorporated by Garfield and his collaborators
revolutionized the quantitative studies of the information and still today maintain their validity
and strength. His specialists also incorporated concepts such as visibility and impact, so
important for the design of research strategies.
Since then, not very relevant innovations has been done in the bibliometric field, where
citations, impact factor plays a key role in the research assessment, and were considered the
best options for many years, but recently and with the advance of a new research scenario
(digital and open), many relevant voices are talking of the necessity of new methods for
research assessment. So the hypothesis of this research is demonstrate that these changes must
be made.
The objective of this research is to answer why and how we need to change the evaluation of
the science. In this way, we will see the main drawbacks of traditional metrics and alternative
systems for research evaluation such the research inquiry, among others, that aims to evaluate
more the qualitative research than the quantitative research.
Methods
To demonstrate the need of rethink the research metrics, we first find and study relevant papers
about the evolution of research assessment, until the question what are research metrics? has
been answered.
After that, the search and analysis of current articles related to the hypothesis raised: the need
to change the methods for research evaluation, has been made, answering the question why we
need new research assessment system?
Finally, to propose a new general method, that is the objective, it was necessary to rethink the
information obtained, to answer the question which new metrics would arise?
All of this comes together converges on a series of discussions and conclusions to take into
account.
What are research metrics?
As (Andersson & Lau Börjesson, 2018), explains in their Guide to research metrics: “Research
metrics are quantitative measures that can help you in assessing research outputs at author,
article and journal level”.
New metrics for the research evaluation. January 2020 2
We can use research metrics to do research evaluation that is called Bibliometrics.
“Bibliometrics is the quantitative analysis of publication data using article-, author-, and journal-
level data to determine and to demonstrate research productivity, quality, and impact. Citation
analysis is the most commonly used bibliometric method. It includes analyzing the number of
articles published by an author, the number of citation from those articles, and the ranking of the
journals in which those articles were published (Wortman, n.d.).” says (Mering, 2017).
In short, bibliometrics measures research productivity and impact. To measure the impact, the
impact factor is used, but what is impact factor? According to Garfield's own paper, "The History
and Meaning of the Journal Impact Factor" (Garfield, 2006) "I first mentioned the idea of an
impact factor in Science in 1955. With the support of the National Institutes of Health, the Index
of citation of experimental genetics was published, and this led to the publication in 1961 of the
Science Citation Index.2 Irving H. Sher and I created the impact factor of the magazine to help to
select additional magazines from the source. To do this, we simply reordered the author's
citation index in the magazine's citation index. From this simple exercise, we learned that
initially a large group of large and highly cited magazines had to be covered in the new science
citation index (LIC). Consider that in 2004, the Journal of Biological Chemistry published 6,500
articles, while the articles of the Proceedings of the National Academy of Sciences were cited
more than 300,000 times that year. It is possible that smaller journals will not be selected if we
rely solely on the count of publications, so we have created the magazine's impact factor (JIF).”
Undoubtedly, one of the fundamental aspects is the impact factor, an indicator of documentary
use currently generated by Thomson and Reuters' Institute for Scientific Information (ISI).
The other relevant indicator is the citation that counts the number of times a publication is cited
by other authors.
There are other indicators but the great majority is based on impact and citations.
New metrics for the research evaluation. January 2020 3
In Image 1, we can see examples of metrics. There are 3 levels of metrics: Researcher, article
and journal:
Image 1: Example of metrics (Belter, Chris; Michalek, Andrea; Cole, 2017)
Where we can find metrics? According to Elsevier, we can find metrics in several places:
Imatge 2 (Guillaumet, 2019)
Also, as M. Mering explains (Mering, 2017), the metrics we use actually are basically from these
3 sources:
- Web of Science database (WOS) of Clarivate Analytics, that provides the number of
publications by a given author, they have metrics such as: Science Citation Index
New metrics for the research evaluation. January 2020 4
Expanded, Social Science Citation Index, Arts & humanities index and Book citation
index. The strong coverage of WOS is STEM disciplines and tends to favor English-
language publications .
- SCOPUS, the Elsevier database product, have shorter track than WOS, have more
coverage of types of publications and is more internationally used.
- Google Scholar covers and indexes a lot of internet publications including, OA journals &
preprints, dissertations & theses, books, abstracts & articles from academic publishers.
It’s widely used by researchers due its wide coverage.
According to (Chaviano, 2004), there are 4 categories of indicators:
- Scientific quality indicators
- Indicators of scientific significance
- Indicators of scientific impact (FI, Appointments, autocites ...)
- Indicators of science and technology
Why we need new research assessment system?
Bibliometrics are widely used, but we have seen that only evaluates productivity and science
impact, but, for example, not evaluates the quality of the contents. Bibliometrics has been a
good tool before the popularization of the Open Science and the digital publication that has
provided a radically different scenario for the research evaluation where the researchers not
only publish in a journal, but have impact through the social media and the interaction with the
Internet, so, it’s clear that, build on the research evaluation only thought a postprint published
in a Journal, seems that is not enough.
We can learn more about the research evaluation and it’s changes, from the Leiden Manifesto
and the DORA declaration explained by Codina (Codina, 2019), i.e. this paragraph provides from
DORA: “The Journal Impact Factor, as calculated by Thomson Reuters, was originally created as
a tool to help librarians identify journals to purchase, not as a measure of the scientific quality of
research in an article. With that in mind, it is critical to understand that the Journal Impact
Factor has a number of well-documented deficiencies as a tool for research assessment.”
So, evaluate the research is more complex and wide than using only bibliometrics. In the article
"Guide to research metrics" (Taylor&Francis, 2018), its explained well: “First of all, you should
ask what aspect of the research you want to evaluate and what you need to understand, if it can
be measured, how can it be done? Then look for the correct metrics that will answer your
question.
Use quantitative - metrics- with qualitative –opinions-. Research metrics are a useful tool but you
can improve by picking up expert opinions: ask to colleagues.
New metrics for the research evaluation. January 2020 5
Finally, look for a more rounded image. Each metric obtains its data from different sources and
uses a different calculation. Use at least a couple of metrics to reduce the bias and get a more
rounded vision”.
Therefore, look at the impact factor of a magazine, but also the details of the Altmetric, for
example, for the most read articles. In this way we can use metrics (index-h, appointments, SJR,
JIF ...), Altmetrics (social media ...), author profiles, opinions, etc. (Guillaumet, 2019)
There are weight arguments for "wary" of the actual value of the impact metrics, citations and
rankings that are most commonly used, after several readings, the most weighted arguments
are:
1) The current evaluation system is homogeneous at an international level. It does not take
into account geographic or demographic components.
2) It is inspired by the so-called experimental disciplines or natural sciences with a clear
detriment of the social sciences.
This metrics has been key for long time; in addition, many have been used by the main
international commercial publishers that until now have made a profit from science, with
restricted access and high payments to access research and their metrics. It’s for all these
reasons that increasingly, there are relevant voices from research community that "criticize" the
hegemony, for example, of the Impact Factor and its excessive use.
All this is argued by (Aguillo, 2019), which makes a fierce criticism of indicators that have shown
hegemony for years, thus speaks of the 'Infamous impact factor', saying that it contains
expected citations (not real or current), with a short window citation for some disciplines, with a
Pareto distribution, not comparable between disciplines and with a clear lack of precision (3
decimals). Cannot operate with the FI (medium or sums ...), it is not comparable on a yearly
basis, and has a very skewed coverage in terms of types of activities and journals. He also
criticizes the quartiles that show a false normalization and that mediocre journals are observed
in the first quartile. And it is only usually analyzed by a source, for example, WOS, which clearly
has deficiencies in terms of coverage and diversity.
The list of drawbacks associated with the impact factor begins with its own name. A more
descriptive title - less sensationalist and more in agreement with the scientific spirit - would be
more appropriate to call it the restricted rate of citation. The rate is calculated by dividing the
number of citations received by the journal, in the last two years, and the number of
computable articles published by the journal in the same period; It is restricted in several ways,
such as the set of journals considered to perform calculations -correlating itself with regions,
disciplines and languages- or the meaning that is given to a citation. (Manzano-Arrondo, 2017).
Determining the real impact of a job can take months; citation behaviors vary markedly
according to areas and disciplines; and the quotes an article receives are not always a reflection
of the quality of the article. There are articles of high scientific level that have barely been cited
New metrics for the research evaluation. January 2020 6
for being published in low profile or commercial journals and, on the contrary, there are articles
that have been highly cited as a result of a refutation process rather than for their quality. (De
Volder, 2016).
Results
Which new metrics would arise?
Once we have seen the main backdraws of the current metrics system, it’s clear the necessity of
use new kind of metrics. A clear example is the popularization of the called alternative metrics,
represented by Altmetrics (Altmetric web, 2018) (De Volder, 2016).
Altmetrics are new indicators proposed to analyze the impact and visibility of scientific activity
on the social web: number of tweets, mentions in a blog, amount of "like" on Facebook,
presence in bibliographic managers, etc. The difference fundamental with traditional metrics is
that while the former measure in function of the prestige of the journals or their impact factor,
the latter measure at the article level.
But Altmetrics and Bibliometrics, seems that are not enough for that as (Holtrop, 2018) assert
proposing a new method called: evaluative inquiry. This method proposes the realizations of
interviews to researchers or groups or whatever to try to answer the how, what, who and why
of the evaluative inquiry.
Another method was explained in the last euroCRIS membership meeting in Münster, Germany
in a presentation called: “Research Information and the Selection of Germany‘s “Universities of
Excellence“ (Spang-Grau, 2019). In Germany they elaborate a series of reports to select their
Excellence universities every 7 years:
The universities must provide information in their proposals and their activity and excellence is
evaluated. They NOT use impact or citations nor rankings or any of the traditional metrics:
(Spang-Grau, 2019), as we can see in the next images:
New metrics for the research evaluation. January 2020 7
Image 3: Evaluation criteria for excellence universities.
Image 4: Type of metrics for the evaluation
New metrics for the research evaluation. January 2020 8
Discussion and conclusions
Compiling all this information and returning to the classification of (Chaviano, 2004) and the
(Spang-Grau, 2019) system for excellent universities and finally for the evaluative inquiry
system (Holtrop, 2018), we can do research evaluation taking into account the following
dimensions:
1) Geographical area. Every country or region has its own properties that must take into
account, for example, the language.
2) Scientific field. Every field has different characteristics, it’s not the same evaluate health
than humanities. Or must take into account if the research is from some scientific field,
for example, Technology&humanities for AI research.
3) Research from one Individual or from large group
After this first classification we need to define the method:
1) Personal interviews (to a researcher or research group)
2) Surveys
3) Indicators of scientific impact
4) Scientific quality indicators
With all the collected information, we should do a report that should be evaluated by a
committee depending on the evaluation goals.
Along the paper it has been assess that, too many times, research evaluation focuses too much
on the quantitative indicators (bibliometrics). Bibliometrics are a good evaluation system and
has been good for several years, but with the digital publication, the research scenario has
changed so radically, that this only system is no longer valid or it shouldn’t be.
It’s necessary to do more rounded research evaluations taking into account some dimensions
and views to generate reports that assures a more correct research assessment.
Even it has been seen that this research assessment should be different depending the research
area that should have different evaluation patterns.
So, in the near future it could happen that every area of research will have their own
assessment system, and the only common aspect should be that needs to take into account
more aspects of the research and the researcher qualitative research.
New metrics for the research evaluation. January 2020 9
Acknowledgements
I would like to acknowledge to Isidro F. Aguillo (Aguillo, 2010) that I follow in twitter
(@isidroaguillo) from many time and it’s a fierce advocate for the evolution and adaptation of
the research evaluation and that has inspired me to do this research and write this paper.
References
Aguillo, I. F. (2010). Medir y evaluar la investigación [presentación].
Aguillo, I. F. (2019). Medir y evaluar la investigación [presentación].
Altmetric web. (2018). Discover the attention surrounding your research – Altmetric. Altmetrics.
Retrieved from https://www.altmetric.com/
Andersson, A., & Lau Börjesson, J. (2018, October 2). A new contribution to research metrics.
Upsala Journal of Medical Sciences, Vol. 123, pp. 191–193.
https://doi.org/10.1080/03009734.2018.1542644
Belter, Chris; Michalek, Andrea; Cole, E. (2017). Researcher profiles and metrics that matter.
Retrieved from https://www.slideshare.net/Library_Connect/researcher-profiles-and-
metrics-that-matter
Chaviano, O. G. (2004). Algunas consideraciones teórico-conceptuales sobre las disciplinas
métricas. ACIMED, 12(5). Retrieved from
http://scielo.sld.cu/scielo.php?script=sci_arttext&pid=S1024-94352004000500007
Codina, L. (2019). Manifiesto {Leiden} y {Declaración} de {San} {Francisco} sobre métricas de
investigación. Lluís Codina. Retrieved from https://www.lluiscodina.com/leiden-
manifiesto-dora-metricas/amp/?__twitter_impression=true
De Volder, C. (2016). MÉTRICAS ALTERNATIVAS: ¿UNA NUEVA FORMA DE MEDIR EL IMPACTO
CIENTÍFICO? https://doi.org/http://hdl.handle.net/10760/38818
Garfield, E. (2006, April 1). The history and meaning of the journal impact factor. Journal of the
American Medical Association, Vol. 295, pp. 90–93. https://doi.org/10.1001/jama.295.1.90
Guillaumet, A. (2019). Not only h-index… #research #metrics #openScience
« investigarlainvestigacion. Retrieved December 4, 2019, from Wordpress website:
https://investigarlainvestigacion.wordpress.com/2019/02/22/not-only-h-index-research-
metrics-openscience/
Holtrop, T. (2018). The evaluative inquiry : a new approach to academic evaluation. Retrieved
November 10, 2019, from CWTS Blog website:
https://blogs.lse.ac.uk/impactofsocialsciences/2018/11/29/the-evaluative-inquiry-a-new-
approach-to-research-evaluation/
Manzano-Arrondo, V. (2017). Hacia un cambio paradigmático para la evaluación de la actividad
científica en la Educación Superior. Revista de La Educación Superior, 46(183), 1–35.
https://doi.org/10.1016/j.resu.2017.08.003
Mering, M. (2017). Bibliometrics: Understanding Author-, Article- and Journal-Level Metrics.
Serials Review, 43(1), 41–45. https://doi.org/10.1080/00987913.2017.1282288
Spang-Grau, I. (2019). Research information and the selection of Germany’s universities of
excellence (euroCRIS, Ed.). Münster: DRIS.
Taylor&Francis. (2018). A guide to research philosopy. Retrieved from
https://think.taylorandfrancis.com/a-guide-to-research-metrics/
Wouters, P. (2017). Eugene Garfield (1925-2017). Nature, 543(7646), 492.
New metrics for the research evaluation. January 2020 10
https://doi.org/10.1038/543492a
