Mike Thelwall

• PhD
• Professor of Data Science at The University of Sheffield

Citation indictors are increasingly used in some subject areas to support peer review in the evaluation of researchers and departments. Nevertheless, traditional journal-based citation indexes may be inadequate for the citation impact assessment of book-based disciplines. This article examines whether online citations from Google Books and Google S...

Sentiment analysis is concerned with the automatic extraction of sentiment-related information from text. Although most sentiment analysis addresses commercial tasks, such as extracting opinions from product reviews, there is increasing interest in the affective dimension of the social web, and Twitter in particular. Most sentiment analysis algorit...

The microblogging site Twitter generates a constant stream of communication, some of which concerns events of general interest. An analysis of Twitter may, therefore, give insights into why particular events resonate with the population. This article reports a study of a month of English Twitter posts, assessing whether popular events are typically...

This article compares (1) citation analysis with OpenAlex and Scopus, testing their citation counts, document type/coverage and subject classifications and (2) three citation-based indicators: raw counts, (field and year) Normalised Citation Scores (NCS) and Normalised Log-transformed Citation Scores (NLCS). Methods (1&2): The indicators calculated...

Purpose Journal Impact Factors and other citation-based indicators are widely used and abused to help select journals to publish in or to estimate the value of a published article. Nevertheless, citation rates primarily reflect scholarly impact rather than other quality dimensions, including societal impact, originality, and rigour. In response to...

Google Gemini 1.5 Flash scores were compared with ChatGPT 4o-mini on evaluations of (a) 51 of the author’s journal articles and (b) up to 200 articles in each of 34 field-based Units of Assessment (UoAs) from the UK Research Excellence Framework (REF) 2021. From (a), the results suggest that Gemini 1.5 Flash, unlike ChatGPT 4o-mini, may work better...

Purpose Evaluating the quality of academic journal articles is a time consuming but critical task for national research evaluation exercises, appointments and promotion. It is therefore important to investigate whether Large Language Models (LLMs) can play a role in this process. Design/methodology/approach This article assesses which ChatGPT inpu...

Quickly detecting problematic research articles is crucial to safeguarding the integrity of scientific research. This study explores whether Twitter mentions of retracted articles can signal potential problems with the articles prior to their retraction, potentially serving as an early warning system for scholars. To investigate this, we analysed a...

Purpose: Journal Impact Factors and other citation-based indicators are widely used and abused to help select journals to publish in or to estimate the value of a published article. Nevertheless, citation rates primarily reflect scholarly impact rather than other quality dimensions, including societal impact, originality, and rigour. In contrast, J...

Some research now suggests that ChatGPT can estimate the quality of journal articles from their titles and abstracts. This has created the possibility to use ChatGPT quality scores, perhaps alongside citation-based formulae, to support peer review for research evaluation. Nevertheless, ChatGPT's internal processes are effectively opaque, despite it...

While previous studies have demonstrated that Large Language Models (LLMs) can predict peer review outcomes to some extent, this paper builds on that by introducing two new contexts and employing a more robust method - averaging multiple ChatGPT scores. The findings that averaging 30 ChatGPT predictions, based on reviewer guidelines and using only...

ChatGPT and other large language models (LLMs) have been successful at natural and computer language processing tasks with varying degrees of complexity. This brief communication summarizes the lessons learned from a series of investigations into its use for the complex text analysis task of research quality evaluation. In summary, ChatGPT is very...

Evaluating the quality of published research is time-consuming but important for departmental evaluations, appointments, and promotions. Previous research has shown that ChatGPT can score articles for research quality, with the results correlating positively with an indicator of quality in all fields except Clinical Medicine. This article investiga...

Although the cultural and heritage roles of museums and art galleries are well recognised, they can also be vehicles to help scholars generate societal impact. This study systematically investigates this role for the first time, using evidence from UK Research Excellence Framework (REF) 2021 impact case studies (ICSs). We identified mentions of ove...

Academics and departments are sometimes judged by how their research has benefitted society. For example, the UK Research Excellence Framework (REF) assesses Impact Case Studies (ICS), which are five-page evidence-based claims of societal impacts. This study investigates whether ChatGPT can evaluate societal impact claims and therefore potentially...

Time spent by academics on research quality assessment might be reduced if automated approaches can help. Whilst citation-based indicators have been extensively developed and evaluated for this, they have substantial limitations and Large Language Models (LLMs) like ChatGPT provide an alternative approach. This article assesses whether ChatGPT 4o-m...

Evaluating the quality of academic journal articles is a time consuming but critical task for national research evaluation exercises, appointments and promotion. It is therefore important to investigate whether Large Language Models (LLMs) can play a role in this process. This article assesses which ChatGPT inputs (full text without tables, figures...

This book critically analyses the value of citation data, altmetrics, and artificial intelligence to support the research evaluation of articles, scholars, departments, universities, countries, and funders. It introduces and discusses indicators that can support research evaluation and analyses their strengths and weaknesses as well as the generic...

Purpose Assess whether ChatGPT 4.0 is accurate enough to perform research evaluations on journal articles to automate this time-consuming task. Design/methodology/approach Test the extent to which ChatGPT-4 can assess the quality of journal articles using a case study of the published scoring guidelines of the UK Research Excellence Framework (REF...

International collaboration is sometimes encouraged in the belief that it generates higher quality research or is more capable of addressing societal problems. Nevertheless, while there is evidence that the journal articles of international teams tend to be more cited than average, perhaps from increased international audiences, there is no science...

Journal field classifications in Scopus are used for citation-based indicators and by authors choosing appropriate journals to submit to. Whilst prior research has found that Scopus categories are occasionally misleading, it is not known how this varies for different journal types. In response, we assessed whether specialist, cross-field and genera...

Billions of short messages are posted daily to the public social web. This gives opportunities for researchers to gain insights into the issues discussed but extracting useful information can be challenging. On one hand, the simplifying quantitative approaches for large scale analysis risk misinterpreting the patterns found because of the many diff...

Purpose Technology is sometimes used to support assessments of academic research in the form of automatically generated bibliometrics for reviewers to consult during their evaluations or by replacing some or all human judgements. With artificial intelligence (AI), there is increasing scope to use technology to assist research assessment processes i...

Evaluating the effects of some or all academic research funding is difficult because of the many different and overlapping sources, types, and scopes. It is therefore important to identify the key aspects of research funding so that funders and others assessing its value do not overlook them. This article outlines 18 dimensions through which fundin...

Technology is being developed to support the peer review processes of journals, conferences, funders, universities, and national research evaluations. This literature and software summary discusses the partial or complete automation of several publishing‐related tasks: suggesting appropriate journals for an article, providing quality control for su...

Journal field classifications in Scopus are used for citation-based indicators and by authors choosing appropriate journals to submit to. Whilst prior research has found that Scopus categories are occasionally misleading, it is not known how this varies for different journal types. In response, we assessed whether specialist, cross-field and genera...

Peer review is a key gatekeeper for academic journals, attempting to block inadequate submissions or correcting them to a publishable standard, as well as improving those that are already satisfactory. The three key aspects of research quality are rigour, significance, and originality but no prior study has assessed whether journal reviewers are ev...

Identifying factors that associate with more cited or higher quality research may be useful to improve science or to support research evaluation. This article reviews evidence for the existence of such factors in article text and metadata. It also reviews studies attempting to estimate article quality or predict long‐term citation counts using stat...

Purpose Diaspora researchers work in one country but have ancestral origins in another, either through moves during a research career (mobile diaspora researchers) or by starting research in the target country (embedded diaspora researchers). Whilst mobile researchers might be tracked through affiliation changes in bibliometric databases, embedded...

The Journal Impact Factor and other indicators that assess the average citation rate of articles in a journal are consulted by many academics and research evaluators, despite initiatives against overreliance on them. Undermining both practices, there is limited evidence about the extent to which journal impact indicators in any field relate to huma...

Citation counts are widely used as indicators of research quality to support or replace human peer review and for lists of top cited papers, researchers, and institutions. Nevertheless, the relationship between citations and research quality is poorly evidenced. We report the first large‐scale science‐wide academic evaluation of the relationship be...

Purpose To assess whether interdisciplinary research evaluation scores vary between fields. Design/methodology/approach The authors investigate whether published refereed journal articles were scored differently by expert assessors (two per output, agreeing a score and norm referencing) from multiple subject-based Units of Assessment (UoAs) in the...

National research evaluation initiatives and incentive schemes choose between simplistic quantitative indicators and time-consuming peer/expert review, sometimes supported by bibliometrics. Here we assess whether machine learning could provide a third alternative, estimating article quality using more multiple bibliometric and metadata inputs. We i...

Whilst funding is essential for some types of research and beneficial for others, it may constrain academic choice and creativity. Thus, it is important to check whether it ever seems unnecessary. Here we investigate whether funded UK research tends to be higher quality in all fields and for all major research funders. Based on peer review quality...

Collaboration is encouraged because it is believed to improve academic research, supported by indirect evidence in the form of more coauthored articles being more cited. Nevertheless, this might not reflect quality but increased self‐citations or the “audience effect”: citations from increased awareness through multiple author networks. We address...

Collaborative research causes problems for research assessments because of the difficulty in fairly crediting its authors. Whilst splitting the rewards for an article amongst its authors has the greatest surface-level fairness, many important evaluations assign full credit to each author, irrespective of team size. The underlying rationales for thi...

Academic research often involves teams of experts, and it seems reasonable to believe that successful main authors or co-authors would tend to help produce better research. This article investigates an aspect of this across science with an indirect method: the extent to which the publishing record of an article’s authors associates with the citatio...

Purpose Scholars often aim to conduct high quality research and their success is judged primarily by peer reviewers. Research quality is difficult for either group to identify, however and misunderstandings can reduce the efficiency of the scientific enterprise. In response, we use a novel term association strategy to seek quantitative evidence of...

Purpose This study investigates differences and commonalities in data production, sharing and reuse across the widest range of disciplines yet and identifies types of improvements needed to promote data sharing and reuse. Design/methodology/approach The first authors of randomly selected publications from 2018 to 2019 in 20 Scopus disciplines were...

Citation counts are widely used as indicators of research quality to support or replace human peer review and for lists of top cited papers, researchers, and institutions. Nevertheless, the extent to which citation counts reflect research quality is not well understood. We report the largest-scale evaluation of the relationship between research qua...

The Journal Impact Factor and other indicators that assess the average citation rate of articles in a journal are consulted by many academics and research evaluators, despite initiatives against overreliance on them. Despite this, there is limited evidence about the extent to which journal impact indicators in any field relates to human judgements...

International collaboration is sometimes encouraged in the belief that it generates higher quality research or is more capable of addressing societal problems. In support of this, there is evidence that the journal articles of international teams tend to be more cited than average. Reasons other than the benefits of international collaboration coul...

The search for and management of external funding now occupies much valuable researcher time. Whilst funding is essential for some types of research and beneficial for others, it may also constrain academic choice and creativity. Thus, it is important to assess whether it is ever detrimental or unnecessary. Here we investigate whether funded resear...

Purpose: Scholars often aim to conduct high quality research and their success is judged primarily by peer reviewers. Research quality is difficult for either group to identify, however, and misunderstandings can reduce the efficiency of the scientific enterprise. In response, we use a novel term association strategy to seek quantitative evidence o...

National research evaluation initiatives and incentive schemes have previously chosen between simplistic quantitative indicators and time-consuming peer review, sometimes supported by bibliometrics. Here we assess whether artificial intelligence (AI) could provide a third alternative, estimating article quality using more multiple bibliometric and...

This literature review identifies indicators that associate with higher impact or higher quality research from article text (e.g., titles, abstracts, lengths, cited references and readability) or metadata (e.g., the number of authors, international or domestic collaborations, journal impact factors and authors' h-index). This includes studies that...

Co-authored articles tend to be more cited in many academic fields, but is this because they tend to be higher quality or is it an audience effect: increased awareness through multiple author networks? We address this unknown with the largest investigation yet into whether author numbers associate with research quality, using expert peer quality ju...

Collaborative research causes problems for research assessments because of the difficulty in fairly crediting its authors. Whilst splitting the rewards for an article amongst its authors has the greatest surface-level fairness, many important evaluations assign full credit to each author, irrespective of team size. The underlying rationales for thi...

Altmetrics are web-based quantitative impact or attention indicators for academic articles that have been proposed to supplement citation counts. This article reports the first assessment of the extent to which mature altmetrics from Altmetric.com and Mendeley associate with journal article quality. It exploits expert norm-referenced peer review sc...

The paper investigates changing levels of online concern about the Kashmiri Pandit migration of the 1990s on Twitter. Although decades old, this movement of people is an ongoing issue in India, with no current resolution. Analysing changing reactions to it on social media may shed light on trends in public attitudes to the event. Tweets were downl...

The papers in this special section focus on bringing multidisciplinary knowledge into sentiment analysis. The last two decades have witnessed an enormous amount of research works on sentiment analysis and significant progress has been made. In terms of depth, finer-grained semantic schemas are defined, such as aspect, target, and category. In terms...

El objetivo de este artículo es identificar características relacionadas con el contenido de los mensajes más retuiteados creados por editoriales de libros españolas y extranjeras en Twitter. Se ha realizado un análisis de contenido para identificar el tema de los tuits y si incluyen hashtag para el título del libro, imágenes e hipervínculos, y en...

New academic knowledge in journal articles is partly built on peer reviewed research already published in journals or books. Academics can also draw from non-academic sources, such as the websites of organisations that publish credible information. This article investigates trends in the academic citing of this type of grey literature for 17 health...

New academic knowledge in journal articles is partly built on peer reviewed research already published in journals or books. Academics can also draw from non-academic sources, such as the websites of organisations that publish credible information. This article investigates trends in the academic citing of this type of grey literature for 17 health...

LGBTQ+ labels and terminology in society embed ideological assumptions and affect who gains community support and protection. In academia, terminology is also needed to help define study objects, methods, and goals. Academics therefore need to choose their words to be both precise and appropriate, adjusting to changes in societal language. This art...

Purpose Toward achieving a better guest experience, the current study aims to use the word frequency comparison technique to evaluate the types of attributes and services that are used most frequently in guests’ five- and one-star reviews on TripAdvisor. The working-paper also aims to investigate the differences between reviews written by men and w...

Research co-authorship is useful to combine different skillsets, especially for applied problems. Whilst it has increased over the last century, it is unclear whether this increase is universal across academic fields and which fields co-author the most and least. In response, this article assesses changes in the rate of journal article co-authorshi...

Purpose Performers may generate loyalty partly through eliciting illusory personal connections with their audience, parasocial relationships (PSRs), and individual illusory exchanges, parasocial interactions (PSIs). On social media, semi-PSIs are real but imbalanced exchanges with audiences, including through comments on influencers’ videos, and st...

Understanding more about variations in peer review is essential to ensure that editors and reviewers harness it effectively in existing and new formats, including for mega-journals and when published online. This article analyzes open reviews from the MDPI suite of journals to identify commonalities and differences from a simplistic quantitative pe...

Formal assessments of the quality of the research produced by departments and universities are now conducted by many countries to monitor achievements and allocate performance-related funding. These evaluations are hugely time consuming if conducted by post-publication peer review and are simplistic if based on citations or journal impact factors....

Patents are key documents to support the commercial exploitation of inventions. Patent documents must claim inventiveness, industrial application, and novelty to be granted and may use citations and URLs to support these claims as well as to explain their ideas. Although there is much research into the citations used to support inventions, almost n...

Female under-representation continues in senior roles within academic medicine, potentially influenced by a perception that female research has less citation impact. This article provides systematic evidence of (a) female participation rates from the perspective of published journal articles in 46 Scopus medical subject categories 1996–2018 and (b)...

Scientometric research often relies on large-scale bibliometric databases of academic journal articles. Long term and longitudinal research can be affected if the composition of a database varies over time, and text processing research can be affected if the percentage of articles with abstracts changes. This article therefore assesses changes in t...

Two partly conflicting academic pressures from the seriousness of the Covid-19 pandemic are the need for faster peer review of Covid-19 health-related research and greater scrutiny of its findings. This paper investigates whether decreases in peer review durations for Covid-19 articles were universal across 97 major medical journals, Nature, Scienc...

This article assesses the balance of research concerning women and men over the past quarter century using the crude heuristic of counting Scopus-indexed journal articles relating to women or men, as suggested by their titles or abstracts. A manual checking procedure together with a word-based heuristic was used to identify whether an article relat...

Abstract Objectives Examine the online interactions, social networks, and perspectives of nursing actors on COVID-19 from conversations on Twitter to understand how the profession responded to this global pandemic. Design Mixed methods. Sample Ten-thousand five-hundred and seventy-four tweets by 2790 individuals and organizations. Measurements N...

New Year’s resolutions are acts of valuation where people express ideas about what is important and worthwhile in life. Although resolutions have a long history, the twenty-first century has transformed the practice into a social media ritual with greater visibility, interactivity, and reach. Using this unique event to explore the globalization of...

Encyclopedias are sometimes cited by scholarly publications, despite concerns about their credibility as sources for academic information. This study investigates trends from 2002 to 2020 in citing two crowdsourced and two expert-based encyclopedias to investigate whether they fit differently into the research landscape: Wikipedia, Britannica, Baid...

Questionnaires are a device to elicit human perspectives, self‐reports or knowledge. This article investigates which broad academic fields use questionnaires, whether this use is increasing, and whether it generates average citation impact. This is investigated through a nonprobability sample: articles mentioning questionnaires in their titles, abs...

Although social media plays an increasingly important role in communication around the world, social media research has primarily focused on Western users. Thus, little is known about how cultural values shape social media behavior. To examine how cultural affective values might influence social media use, we developed a new sentiment analysis tool...

Some complementary and alternative medicines (CAM) are frequently criticised for being based on faith rather than scientific evidence. Despite this, researchers, academic departments, and institutes teach and investigate them. This article assesses whether the scholarship produced by four CAMs is valued by the academic community in terms of citatio...

Purpose The purpose of this study is to explore current practices, challenges and technological needs of different data repositories. Design/methodology/approach An online survey was designed for data repository managers, and contact information from the re3data, a data repository registry, was collected to disseminate the survey. Findings In tot...

Purpose Methods to tackle Covid-19 have been developed by a wave of biomedical research but the pandemic has also influenced many aspects of society, generating a need for research into its consequences, and potentially changing the way existing topics are investigated. This article investigates the nature of this influence on the wider academic re...

Quantile regression presents a complete picture of the effects on the location, scale, and shape of the dependent variable at all points, not just the mean. We focus on two challenges for citation count analysis by quantile regression: discontinuity and substantial mass points at lower counts. A Bayesian hurdle quantile regression model for count d...

Identifying neologisms is important for natural language processing of social web text when informal language is standard and youth slang is common. For example, failing to identify neologisms can reduce the accuracy of the lexical sentiment analysis if opinions are frequently expressed in words that are too new to be in the sentiment dictionary. T...

Whilst funders increasingly request evidence of the societal benefits of research, all academics in the UK must periodically provide this information to gain part of their block funding within the Research Excellence Framework (REF). The impact case studies produced in the UK are public and can therefore be used to gain insights into the types of s...

As the Covid-19 pandemic is a global threat to health that few can fully escape, it has provided a unique opportunity to study international reactions to a common problem. Such reactions can be partly obtained from public posts to Twitter, allowing investigations of changes in interest over time. This study analysed English-language Covid-19 tweets...

Vaccination programs may help the world to reduce or eliminate Covid-19. Information about them may help countries to design theirs more effectively, with important benefits for public health. This article investigates whether it is possible to get insights into national vaccination programmes from a quick international comparison of public comment...

As one of the world's most visited websites, YouTube is potentially influential for learning gendered attitudes. Nevertheless, despite evidence of gender influences within the site for some topics, the extent to which YouTube reflects or promotes male/female or other gender divides is unknown. This article analyses 10,211 YouTube videos published i...

Purpose Despite lifestyle information needs being an important part of our daily lives, little is known about the role of common sources. Whilst magazines and television are traditional providers of lifestyle content, including for fashion, makeup, fitness and cookery, they have been partly replaced by content-creating online influencers. Design/m...

Victims of bullying are often reluctant to seek formal support, and instead, internalise their emotions, hindering recovery. Some will subsequently encounter discussions of bullying online, giving them an unexpected opportunity to share their feelings or experience vicarious support. In this article, we investigate reactions to discussions of offli...

Information about the relative strengths of scholars is needed for the efficient running of knowledge systems. Because academic research requires many skills, more experienced researchers might produce better research and attract more citations. This article assesses career citation impact changes 2001–2016 for domestic researchers (definition: fir...

Researchers, editors, educators and publishers need to understand the mix of research methods used in their field to guide decision making, with a current concern being that qualitative research is threatened by big data. Although there have been many studies of the prevalence of different methods within individual narrow fields, there have been no...

Since the Covid-19 pandemic is a global threat to health that few can fully escape, it has given a unique opportunity to study international reactions to a common problem. Such reactions can be partly obtained from public posts to Twitter, allowing investigations of changes in interest over time. This study analysed English-language Covid-19 tweets...

The h-index is an indicator of the scientific impact of an academic publishing career. Its hybrid publishing/citation nature and inherent bias against younger researchers, women, people in low resourced countries, and those not prioritizing publishing arguably give it little value for most formal and informal research evaluations. Nevertheless, it...

With qualitative research apparently threatened by big data, researchers, editors, educators, librarians, and publishers need to understand the mix of research methods used in their field to guide decision making. In response, this study assesses the prevalence and citation impact of academic research between 1996 and 2019 that reports one of four...

Vaccination programs may help the world to reduce or eliminate Covid-19. Information about them may help countries to design theirs more effectively, with important benefits for public health. This article investigates whether it is possible to get insights into national vaccination programmes from a quick international comparison of public comment...

Covid-19 vaccine hesitancy seems likely to increase mortality rates and delay the easing of social distancing restrictions. Online platforms with large audiences may influence vaccine hesitancy by spreading fear and misinformation that is avoided by the mainstream media. Understanding what types of vaccine hesitancy information is shared on the pop...

Purpose Although gender identities influence how people present themselves on social media, previous studies have tested pre-specified dimensions of difference, potentially overlooking other differences and ignoring nonbinary users. Design/methodology/approach Word association thematic analysis was used to systematically check for fine-grained sta...

Despite growing evidence of open biodiversity data reuse by scientists, information about how data is reused and cited is rarely openly accessible from research data repositories. This study explores data citation and reuse practices in biodiversity by using openly available metadata for 43,802 datasets indexed in the Global Biodiversity Informatio...

Quantile regression is a technique to analyse the effects of a set of independent variables on the entire distribution of a continuous response variable. Quantile regression presents a complete picture of the effects on the location, scale, and shape of the dependent variable at all points, not just at the mean. This research focuses on two challen...