Paul Wouters

Leiden University | LEI · Center for Science and Technology Studies

This study investigates the extent to which scholarly tweets of scientific papers are engaged with by Twitter users through four types of user engagement behaviors, i.e., liking, retweeting, quoting, and replying. Based on a sample consisting of 7 million scholarly tweets of Web of Science papers, our results show that likes is the most prevalent e...

To provide some context for the potential engagement behavior of Twitter users around science, this article investigates how Bitly short links to scientific publications embedded in scholarly Twitter mentions are clicked on Twitter. Based on the click metrics of over 1.1 million Bitly short links referring to Web of Science (WoS) publications, our...

Sufficient data presence is one of the key preconditions for applying metrics in practice. Based on both Altmetric.com data and Mendeley data collected up to 2019, this paper presents a state-of-the-art analysis of the presence of 12 kinds of altmetric events for nearly 12.3 million Web of Science publications published between 2012 and 2018. Resul...

We need a broader, more-transparent suite of metrics to improve science publishing, say Paul Wouters, colleagues and co-signatories. We need a broader, more-transparent suite of metrics to improve science publishing, say Paul Wouters, colleagues and co-signatories.

‘Neuromarketing’ designates both a developing industry and an academic research field. This study documents the emergence of neuromarketing through the first mention of the term in traditional and new media until the stabilization of the field. Our main interest is to establish whether neuromarketing developed separately as an academic field and as...

The cognitive and social structures, and publication practices, of the humanities have been studied bibliometrically for the past 50 years. This article explores the conceptual frameworks, methods, and data sources used in bibliometrics to study the nature of the humanities, and its differences and similarities in comparison with other scientific d...

This chapter approaches, both from a theoretical and practical perspective, the most important principles and conceptual frameworks that can be considered in the application of social media metrics for scientific evaluation. We propose conceptually valid uses for social media metrics in research evaluation. The chapter discusses frameworks and uses...

This article1 describes the possibilities to analyze open access (OA) publishing in the Netherlands in an international comparative way. OA publishing is now actively stimulated by Dutch science policy, similar to the United Kingdom. We conducted a bibliometric baseline measurement to assess the current situation, to be able to measure developments...

Global university rankings have become increasingly important ‘calculative devices’ for assessing the ‘quality’ of higher education and research. Their ability to make characteristics of universities ‘calculable’ is here exemplified by the first proper university ranking ever, produced as early as 1910 by the American psychologist James McKeen Catt...

In the current discussions concerning the pressure for publication and to obtain grants, the questions about what publication and grant pressure actually involve and how they are linked to the academic job market, are often neglected. In this study, we show that publication and grand pressure are not just external forces but internal ones as scient...

In this article, we present the results of an analysis that describes the research centered on Journal Impact Factors (JIFs). The purpose of the analysis is to make a start of studying part of the field of quantitative science studies that relates to the most famous and classic bibliometric indicator around and to see what characteristics apply to...

Inventor of the Science Citation Index.

This study presents a large scale analysis of the distribution and presence of Mendeley readership scores over time and across disciplines. We study whether Mendeley readership scores (RS) can identify highly cited publications more effectively than journal citation scores (JCS). Web of Science (WoS) publications with DOIs published during the peri...

This is the final report of the European Commission's Expert Group on Altmetrics, which undertook its work over the course of 2016. The report outlines a framework for next-generation metrics in the context of the EC's Open Science agenda and includes a series of recommendations for how responsible metrics can be built into the design and evaluatio...

p>Paul Wouters’ essay is concerned with bridging the gap between what we value in our academic work and how we are assessed in formal evaluation exercises. He reflects on the recent evaluation of his own center, and reminds us that it is productive to see evaluations not as the (obviously impossible) attempt to produce a true representation of past...

This panel pays tribute to Dr. Eugene Garfield, one of the “fathers” of bibliometrics, former president of ASIS (1999–2000) and the founder of the ISI citation databases. Dr. Garfield passed away on February 25, 2017. In this panel, we will highlight his contributions to information science. The panelists are all well-known researchers who have kno...

Bibliometric indicators such as journal impact factors, h-indices, and total citation counts are algorithmic artifacts that can be used in research evaluation and management. These artifacts have no meaning by themselves, but receive their meaning from attributions in institutional practices. We distinguish four main stakeholders in these practices...

Citation metrics are increasingly used to appraise published research. One challenge is whether and how to normalize these metrics to account for differences across scientific fields, age (year of publication), type of document, database coverage, and other factors. We discuss the pros and cons for normalizations using different approaches. Additio...

Scientific research revolves around the production, analysis, storage, management, and re-use of data. Data sharing offers important benefits for scientific progress and advancement of knowledge. However, several limitations and barriers in the general adoption of data sharing are still in place. Probably the most important challenge is that data s...

Brief comment about digital humanities published in the Isis Viewpoint section on the History Manifesto.

In a critical and provocative paper, Abramo and D'Angelo claim that commonly used scientometric indicators such as the mean normalized citation score (MNCS) are completely inappropriate as indicators of scientific performance. Abramo and D'Angelo argue that scientific performance should be quantified using indicators that take into account the prod...

In this paper, we report on the application of Google Scholar (GS)-based metrics in the formal assessment of research programs. Involved were programs in the fields of Education, Pedagogical Sciences, and Anthropology in The Netherlands. Also, a comparative analysis has been conducted of the results based on GS and Web of Science (WoS). Studies cri...

This review of the international literature on evaluation systems, evaluation practices, and metrics (mis)uses was written as part of a larger review commissioned by the Higher Education Funding Council for England (HEFCE) to inform their independent assessment of the role of metrics in research evaluation (2014–5). The literature on evaluation sys...

In this study we combine the registered output of a whole university in the Netherlands with data retrieved from the Web of Science. The initial research question was: is it possible to show the impact of the university in its’ full broadness, taking into account the variety of disciplines covered in the research profile of the university? In order...

Nutzen Sie diese zehn Grundsätze um Forschung zu bewerten, drängen Diana Hicks, Paul Wouters und Kollegen. Daten werden immer stärker zur Steuerung von Forschung genutzt. Forschungsevaluationen, die einst von Fachkollegen bezeugt und durchgeführt wurden, sind heute alltäglich und abhängig von Metrikeni. Das Problem ist, dass diese Evaluationen heu...

This report presents the findings and recommendations of the Independent Review of the Role of Metrics in Research Assessment and Management. The review was chaired by Professor James Wilsdon, supported by an independent and multidisciplinary group of experts in scientometrics, research funding, research policy, publishing, university management an...

In this study, the academic status of users of scientific publications in Mendeley is explored in order to analyse the usage pattern of Mendeley users in terms of subject fields, citation and readership impact. The main focus of this study is on studying the filtering capacity of Mendeley readership counts compared to journal citation scores in det...

The Independent Review of the Role of Metrics in Research Assessment and Management was set up in April 2014 to investigate the current and potential future roles that quantitative indicators can play in the assessment and management of research. Its report, ‘The Metric Tide’, was published in July 2015 and is available below. The review was chair...

Purpose – The purpose of this paper is to analyze the disciplinary orientation of scientific publications that were mentioned on different social media platforms, focussing on their differences and similarities with citation counts. Design/methodology/approach – Social media metrics and readership counts, associated with 500,216 publications and t...

Use these ten principles to guide research evaluation, urge Diana Hicks, Paul Wouters and colleagues. were introduced, such as InCites (using the Web of Science) and SciVal (using Scopus), as well as software to analyse individual citation profiles using Google Scholar (Publish or Perish, released in 2007). In 2005, Jorge Hirsch, a physicist at the...

This talk will focus on the forces that are currently leading to a need for modifying or in some cases completely rebuilding systems for measurements of the achievements of science and scientists. This will be an entry point as well to describe the importance of such measurements and where these measurements might be used or mis-used, irrespective...

An extensive analysis of the presence of different altmetric indicators provided by Altmetric.com across scientific fields is presented, particularly focusing on their relationship with citations. Our results confirm that the presence and density of social media altmetric counts are still very low and not very frequent among scientific publications...

The main focus of this paper is to investigate the impact of publications read (saved) by the different users in Mendeley in order to explore the extent to which their readership counts correlate with their citation indicators. The potential of filtering highly cited papers by Mendeley readerships and its different users have been also explored....

- The journal impact factor (JIF) and the Hirsch index, are two widely used parameters for evaluating scientific achievement.- The JIF is a parameter which shows the citation score of a journal over the previous two years.- The Hirsch index is a simple index to measure the citation performance of individual scientists.- These achievement indicators...

The online reference manager tool Mendeley (http://www.mendeley.com/) is one of the most promising tools for altmetrics research (Li, Thelwall and Giustini, 2011;Wouters & Costas, 2012) and it has been already used in other previous studies, for example in Library and Information Science Journals (Bar-Ilan et al., 2012b), for Nature and Science jou...

Is more always better? We address this question in the context of bibliometric indices that aim to assess the scientific impact of individual researchers by counting their number of highly cited publications. We propose a simple model in which the number of citations of a publication correlates with the scientific impact of the publication but also...

In this paper we present the results of an analysis that describes the research centred on journal impact factors. The purpose of the analysis is to make a start of studying part of the field of quantitative science studies that relates to the most famous bibliometric indicator around, and see what characteristics apply to the research on journal i...

In this paper an analysis of the presence and possibilities of altmetrics for bibliometric and performance analysis is carried out. Using the web based tool Impact Story, we have collected metrics for 20,000 random publications from the Web of Science. We studied the presence and frequency of altmetrics in the set of publications, across fields, do...

The Leiden Ranking 2011/2012 is a ranking of universities based on bibliometric indicators of publication output, citation impact, and scientific collaboration. The ranking includes 500 major universities from 41 different countries. This paper provides an extensive discussion of the Leiden Ranking 2011/2012. The ranking is compared with other glob...

The social and neural sciences share a common interest in understanding the mechanisms that underlie human behaviour. However, interactions between neuroscience and social science disciplines remain strikingly narrow and tenuous. We illustrate the scope and challenges for such interactions using the paradigmatic example of neuroeconomics. Using qua...

What is my impact? This question has become ever more pressing for all actors in the scientific and scholarly system. The shift to web based publishing has enabled new impact measurement tools that may address the current limitations of peer evaluation and citation analysis. This paper presents the first comprehensive assessment of the limitations...

The recent turmoil in the financial markets has demonstrated the growing need for automated information monitoring tools that can help to identify the issues and patterns that matter and that can track and predict emerging events in business and economic processes. One of the techniques that can address this need is sentiment mining. Existing appro...

This paper studies the digitization of a well-established print bibliography for Dutch literary studies and its implications for the discipline. On the basis of qualitative interviews with users, we first compare three particular research practices in the field, and how they are affected by the digitization. Secondly we analyze the controversy that...

This article focuses on the process of scientific and scholarly communication. Data on open access publications on the Internet not only provides a supplement to the traditional citation indexes but also enables analysis of the microprocesses and daily practices that constitute scientific communication. This article focuses on a stage in the life c...

New mass publishing genres, such as blogs and personal home pages provide a rich source of social data that is yet to be fully exploited by the social sciences and humanities. Information-centered research (ICR) not only provides a genuinely new and useful information science research model for this type of data, but can also contribute to the emer...

Open access is a key issue in the development of the information society. It may also shape the extent to which the generation of new scientific and scholarly research itself can be tuned to the future needs of developed and developing countries. Much of the promise of e-science is based on an implicit notion that open access will accelerate scient...

This article addresses the need to problematize "cases" in science and technology studies (STS) work, as a middle-range theory issue. The focus is not on any one case study per se, but on why case studies exist and endure in STS. Case studies are part of a specific problematization in the field. We therefore explore relations between motivation for...

Currently, there exists little evidence concerning how various characteristics of research cultures are associated with patterns of use of electronic library resources. The present study addresses this gap by exploring how research-group membership, across-fields scattering of literature, and degree of establishment of research area are related to...

E-science initiatives are technology-enabled interventions in current research practices. These interventions are justified by the hope that e-science infrastructures and tools will foster new venues for researchers and scholars. This triggers a complex interaction between hope, hype, and accountability. In this article, we discuss a new initiative...

Internet search engines function in a present which changes continuously. The search engines update their indices regularly, overwriting webpages with newer ones, adding new pages to the index and losing older ones. Some search engines can be used to search for information on the internet for specific periods of time. However, these ‘date stamps’ a...

The article analyzes the emergence of the Science Citation Index (SCI) and argues that the concept of citation indexing was not a "natural" outgrowth of the scientific field. It originated in the area of US legal publishing and information services, and was translated into a scientific reference service by an information entrepreneur, Eugene Garfie...

On the Web, information that is not presented by the search engine in response to a specific query is in fact inaccessible. This problem has been defined as “the invisible web” or the “deep web” problem. Specific characteristics of the format of information may make it inaccessible to the crawlers that create the databases of search engines. We exp...

This chapter problematizes the relation between the varied modes of knowledge production in the sciences and humanities, and the assumptions underlying the design of current e-science initiatives. Using the notion of "epistemic culture" to analyze various areas of scientific research practices, we show that current conceptions of e-science are firm...

The article analyzes the emergence of the Science Citation Index (SCI) and argues that the concept of citation indexing was not a “natural” outgrowth of the scientific field. It originated in the area of US legal publish-ing and information services, and was translated into a scientific reference service by an information entrepreneur, Eugene Garfi...

In this paper we discuss the need and possibilities for the development of web indicators for the characterization of emergent features of ICT-mediated scholarly practices based on the European WISER project 1. We summarize these developments as a turn to e-science. We place the research of web indicators in a virtual landscape of measuring and exp...

This chapter problematizes the relation between the varied modes of knowledge production in the sciences and humanities, and the assumptions underlying the design of current e-science initiatives. Using the notion of “epistemic culture” to analyze various areas of scientific research practices, we show that current conceptions of e-science are firm...

Since many nations have provided substantial funding for new e-social science and humanities investigations, there is now an opportunity for information scientists to adopt an enabling role for this new kind of research. Logically, a more information-centred environment should be more conducive to information science and to information scientists t...

This paper presents the goals and theoretical underpinning of a new national programme in e-social science in the Netherlands. Recent transformations in communication and information exchange have created new opportunities for researchers in the humanities and social sciences. It is not self-evident, however, in what ways scholars can best use thes...

How do authors refer to Web-based information sources in their formal scientific publications? It is not yet well known how scientists and scholars actually include new types of information sources, available through the new media, in their published work. This article reports on a comparative study of the lists of references in 38 scientific journ...

Research in genomics is an example of changes induced by information and communication technologies (ICT). The emergence of interconnected ICT support for scientific work and the handling of information have changed the challenges in genomics as well as other scientific fields. The promises are significant but a large degree of uncertainty remains....

Search engines are unreliable tools for data collection for research that aims to reconstruct the historical record. This unreliability is not caused by sudden instabilities of search engines. On the contrary, their operational stability in systematically updating the Internet is the cause. We show how both Google and Altavista systematically reloc...

The emergence of an global cyberinfrastructure is rapidly increasing the ability of scientists to produce, manage, and use data, leading to new understanding and modes of scientific inquiry that depend on broader data access. As research becomes increasingly global, data intensive, and multifaceted, it is imperative to address national and internat...

Access to and sharing of data are essential for the conduct and advancement of science. This article argues that publicly funded research data should be openly available to the maximum extent possible. To seize upon advancements of cyberinfrastructure and the explosion of data in a range of scientific disciplines, this access to and sharing of publ...

This study explores the use of freely available search engines to locate mediated interaction on the World Wide Web. We use the concepts of mediated interaction and quasi-interaction as developed by Thompson (Thompson, 1995) and Slevin (Slevin, 2000). We conclude that the publicly available search engines lack stability of results, that their behav...

Scientific literature is expected to contain a body of knowledge that can be indexed and retrieved using references and citations. References are subtexts which refer to a supertext, that is, the body of scientific literature. The Science Citation Index has provided an electronic representation of science at the supertextual level by aggregating th...