Available via license: CC BY 4.0
Content may be subject to copyright.
A systematic review of Hyperpartisan News
detection: A Comprehensive Framework for
Denition, Detection, and Evaluation
Michele Joshua Maggini ( michelejoshua.maggini@usc.es )
University of Santiago de Compostela
Davide Bassi
University of Santiago de Compostela
Pablo Gamallo Otero
University of Santiago de Compostela
Paloma Piot
University of A Coruña
Gaël Dias
Université de Caen Normandie
Research Article
Keywords: hyperpartisan detection, disinformation, machine learning, deep learning, language models,
natural language processing
Posted Date: January 29th, 2024
DOI: https://doi.org/10.21203/rs.3.rs-3893574/v1
License: This work is licensed under a Creative Commons Attribution 4.0 International License.
Read Full License
Additional Declarations: No competing interests reported.
A systematic review of Hyperpartisan News
detection: A Comprehensive Framework for
Detection, Definition, and Evaluation
Michele Joshua Maggini 1, Davide Bassi 1, Paloma Piot 2,
Ga¨el Dias 3, Pablo Gamallo Otero 1
1Centro Singular de Investigaci´on en Tecnolox´ıas Intelixentes da USC,
Universidade de Santiago de Compostela, Santiago de Compostela,Spain.
2IRLab, CITIC Research Centre, Universidade da Coru˜na, A Coru˜na,
Spain.
3Normandie Univ, UNICAEN, ENSICAEN, CNRS, GREYC, France.
Contributing authors: michelejoshua.maggini@usc.es;
davide.bassi@usc.es;paloma.piot@udc.es;gaeldias@unicaen.fr;
pablo.gamallo@usc.es;
Abstract
Hyperpartisan news detection employs advanced technological methods to iden-
tify news articles characterized by extreme bias, reporting information in a highly
polarized manner. The spread of such news could lead to detrimental effects in
terms of social cohesion and facts-perception.
This systematic review delves into the uncharted territory of hyperpartisan news
detection, where a conspicuous absence of comprehensive studies has left a void in
scholarly discourse. Following the PRISMA methodology, we selected 80 pertinent
articles.
The paper commences by addressing the inherent challenges arising from dis-
parate and ambiguous definitions of hyperpartisan found in existing literature.
Through a meticulous analysis, we propose a refined and comprehensive defini-
tion of hyperpartisan news, serving as a foundational framework for subsequent
exploration.
The review systematically evaluates Non-Deep Learning, Deep Learning (i.e. non-
Transformer and Transformer-based methods), and mixed approaches employed
in the identification of hyperpartisan content across various studies. By scruti-
nizing methodologies, algorithms, and key features, we categorize and compare
their performances, shedding light on their efficacy and limitations.
1
Furthermore, we conducted an examination of 36 datasets utilized in diverse stud-
ies, offering insights into the commonalities and variations in data sources. This
inclusive dataset analysis contributes to a nuanced understanding of the chal-
lenges faced in the development and validation of hyperpartisan news detection
models.
This paper not only pioneers the systematic review of hyperpartisan news detec-
tion, but also establishes a robust foundation for future research endeavors in this
critical domain. We emphasize the necessity of standardized definitions, rigorous
evaluation metrics, and shared datasets to advance the field.
Keywords: hyperpartisan detection, disinformation, machine learning, deep learning,
language models, natural language processing
1 Introduction
Recently, there has been increasing attention given to hyperpartisan detection. Hyper-
partisan news reflects extreme one-sided political leaning. Hyperpartisan detection is
considered a classification task. Specifically, from a textual point of view, given a text
and an algorithm, the machine should predict the affiliation of that piece of text to
a political party or if the content is hyperpartisan. This issue is closely linked to the
spread of disinformation and political outcomes. Parties tend to conduct campaigns
to gather as many voters as possible relying both on physical sources like newspa-
pers but also on social media and online news publishers. Thus, governments are the
results of citizen’s voting process and parties shape the public policies (Falkenbach
et al,2019). Some factors genuinely contribute to democracy like transparent informa-
tion and political participation. However, populism and polarization imply an internal
division of the social fabric, reducing the trust in governmental entities and in main-
stream news (Guess et al,2021). Polarization is an intrinsic and founding trait of
democracy, as democracy is based on the relationship between the government and its
oppositions, but their gap has been increasing in the last years (Dalton,2015).
When polarization declines in its extremist form, i.e. hyperpartisanship, it is per-
ceived to endanger the proper functioning of democracy (Lorenz-Spreen et al,2022).
Highly polarized party systems drive greater voter participation by offering clearer
choices and emphasizing the impact of individual votes. These systems connect voting
decisions more strongly to ideologies and policies. Yet, this polarization can affect how
well governments represent their people and may highlight divisive tensions within
a society, making governance tougher and potentially alienating citizens if opposing
sides gain power (Dalton,2021). In the digital sphere, hyperpartisanship entails the
conscious partial coverage of the truthfulness of information, if not even the complete
falsification, the distortion of events through a mystifying, excessively sentimental
and violent language (Weeks,2015). The success rate of such political language is
demonstrated by the users forming closed clusters, like echo chambers or extremist
movements, where the exposition to outer opinions only enforces the group’s beliefs
2
and belonging, radicalizing its users. These factors, namely the consumption of harm-
ful contents and the formation of radicalized entities, negatively impact the social
capital (Hawdon et al,2020).
Furthermore, considering the decreasing rate of text comprehension affecting new
generations (Kuhfeld et al,2023), the failure to present information as objectively
and transparently as possible means that the pool of new voters will increasingly
struggle to understand government communication, seeing their democratic participa-
tion threatened. To date, hyperpolarization, which as we have seen comprises various
types of biases manifested in exaggeration, is exploited by media companies, not only
politically to favor certain candidates but also to generate revenue through polarizing
headlines and even clickbait. Political influence manifests itself in the digital realm
(Hawdon et al,2020) and influences the infosphere (Bawden and Robinson,2018).
In 2018, the EU Commission gathered a group of experts to publish a report
(Commission,2018) on related topics like disinformation, notably defamation, hate
speech and incitement to violence, which are subject to regulatory remedies under
EU or national laws, nor other forms of deliberate but not misleading distortions of
facts such a satire and parody. In the following year, SemEval-2019 task 4 (Kiesel
et al,2019) took place and was the first testimony of a rising interest in hyperpartisan
detection. It involved 42 teams in the detection task. Lastly, in 2022, the European
Parliament adopted a package of rules about digital services, the Digital Services Act
(DSA) (European Parliament and Council,2020), whose one of the main objectives
is to provide “a secure, predictable and trustworthy online environment” (Article
1. 1). With this aim, news aggregators like Google News, Politifact were created.
Additionally, other websites like Allsides collect the diverse points of view on the same
topic from news publishers with diverse political leanings.
The literature contains a variety of survey papers (Hamborg et al,2019;Medeiros
and Braga,2020;Kapantai et al,2021;Rodrigo-Gin´es et al,2024) on the detection of
fake news and related tasks. Nonetheless, any kind of survey has never been written
about hyperpartisan detection. Given the increasing discussion of the topic in both
academic and non-academic spheres, its social impact and considering the unsatisfied
need for a systematic approach regarding hyperpartisan detection, we have decided to
propose an extensive and comprehensive investigation of the state-of-the-art through
this paper. Researchers in both social and computer science will find this systematic
review valuable.
The major focus of the study on different hyperpartisan detection models is given
here:
•Compile a comprehensive in-depth survey on hyperpartisan detection through the
collection of the diverse approaches and algorithms used in the literature.
•Show how researchers contributed to this field by presenting the main findings, the
engineering innovations and research designs.
•Report performance metrics, features and embeddings considered in the stud-
ies, employed datasets, and delineating prevailing research gaps and challenges in
hyperpartisan detection models.
3
The remainder of this paper is given here: in section 2 we discuss the methodol-
ogy adopted for this systematic review, describing the beginning of the process with
preliminary research questions, search strategy, criteria selection, and selection proce-
dure; section 3 focuses on the definition of hyperpartisanship, highlighting that is a
multi-task with a cross-disciplinary nature. Then, we propose our definition of hyper-
partisanship. Afterward, we present the various places and frames where hyperpartisan
traits could be detected and the spectrum of methodologies used in different compu-
tational sub-fields. Lastly, we discuss the diverse strategies and scales used to label
hyperpartisanship; section 4 is a global categorization of the most performative model
found in the papers screened and selected. We distinguished between the typology
of the model, the results, the features and the approaches employed. section 5 is a
descriptive overview of the datasets on which models were trained. Finally, section 6
concludes the article by presenting the main findings of our literature review.
2 Methodology
A systematic review is an investigation designed to identify and assess a research topic,
going beyond surface-level knowledge to reveal gaps and highlight the most significant
contributions in a specific research area (Kitchenham and Charters,2007). The sys-
tematic review undergoes stages of quality evaluation, wherein studies are scrutinized
in-depth to confirm their consistency or identify contradictions (Kitchenham et al,
2011). According to (Kitchenham and Charters,2007), a systematic review encom-
passes three phases: Planning, Execution, and Reporting. Moreover, we encapsulated
this procedure in the schema presented by (Biolchini et al,2005), consisting of these
steps: Question Formulization, Search Approach, Studies Selection and Evaluation,
Information Extraction and Results Summarization.
For the paper selection, we adopted the Preferred Reporting Items for Systematic
Reviews and Meta-Analysis (PRISMA) (Moher et al,2011) guidelines, consisting of
a checklist and a flow diagram to illustrate in a simplified and clear way the steps
made. The planning and execution phases of this study are detailed in the following
subsections, while the results phase is expounded upon in section 4 and section 5.
2.1 Research Questions
The Research Questions (RQ) that motivated the need for this systematic review are
the following:
[RQ1] How have publications on hyperpartisan detection been distributed over the
years? [RQ2] Does there exist a categorization for hyperpartisan detection methods?
[RQ3] How can the current state of research on hyperpartisan detection be character-
ized in diverse languages and countries? [RQ4] Is hyperpartisan detection a stand-alone
or over-lapping task? [RQ5] What are the proposed solutions? [RQ6] What are the
results of the models developed? [RQ7] Does the task keep up with the new NLP tech-
nologies like autoregressive models? [RQ8] What are the datasets used for this task,
how they are structured and are they limited to this domain?
4
Queries
Database Query Docs.
ACM Digital
Library
[[[Title: hyperpartisan] OR [[Title: political polarization] AND [[Title:
news] OR [Title: article]]]] AND [[Abstract: hyperpartisan] OR
[Abstract: media bias] OR [Abstract: or(((news] OR [All: article]]]
OR [All: ))) and] OR [[[Keywords: hyperpartisan] OR [[Keywords:
media bias] AND [[Keywords: news] OR [Keywords: article]]]] AND
[[Title: hyperpartisan] OR [Title: hyper-partisan] OR [Title: partisan]
OR [Title: political polarization] OR [[Title: media bias] AND [[Title:
news] OR [Title: article]]]]] AND [E-Publication Date: (01/01/2015
TO 12/31/2023)]
723
Google Scholar (hyperpartisan OR hyper-partisan OR hyperpartisanship OR polar-
ization) AND (news OR bias OR articles) AND (detection OR
classification)
1800
IEEExplorer (”All Metadata”:hyperpartisan OR ”All Metadata”:”hyper-partisan”
OR ”All Metadata”:hyperpartisanship OR ”All Metadata”:hyper-
partisanship) AND (”All Metadata”:detection)
1
ProQuest ”hyperpartisan news” + Filters 159
ScienceDirect hyperpartisan OR ”hyper-partisan” OR ”political polarization” OR
”media bias” AND NLP
2
Scopus (hyperpartisan OR ”hyper-partisan” OR partisanship OR hyperpar-
tisanship OR ”political polarization” ) AND ( news OR articles OR
bias ) AND ( classification OR detection))
97
Table 1 This table describes the keywords wrote for each dataset and the number of papers
corresponding to the query. ”Database” refers to the online platforms used to retrieve documents;
under ”Query” we report the final keywords combination used in each database; ”Number of
documents” indicates the amount of documents retrieved with that query.
2.2 Search Strategy
To perform a systematic investigation and categorization of literature concern-
ing hyperpartisan detection, we followed the PRISMA methodology for both
the structuring of the paper and information workflow. We used PRISMA 2020
Checklist1and PRISMA 2020 flow-diagram (see Figure 1). To collect infor-
mation about the topic, several primary academic databases were used to
overcome their respective limitations (Gusenbauer and Haddaway,2020): ACM
Digital Library, Google Scholar, Scopus, ProQuest, IEEExplore. Our query
archetype was: ((hyperpartisan OR "political bias" OR "hyper-partisan"
OR partisanship OR hyperpartisanship OR "political polarization") AND
(news OR bias OR articles) AND (detection OR classification)). The first
set contains the different ways of writing hyperpartisan. We also searched in all fields,
to capture as many semantically similar papers as possible.
For the purpose of obtaining pertinent papers related to our questions, the queries
in subsection 2.2 were a refined result of a structured process based on different steps
introduced by Rodrigo-Gin´es et al (2024). We extended these steps by adding another
1http://www.prisma-statement.org/documents/PRISMA 2020 checklist.pdf
5
one: ”Network visualization and exploration”, concerning the usage of the research
software ResearchRabbit2.
•Keywords domain extrapolation: Initial reviews on similar topics helped us identify
the keywords used in this domain. We noticed a lack of scientific agreement on how
to write ”hyperpartisan”. To cover all these morphologically diverse forms (hyper-
partisan, hyperpartisan, hyper partisan), we included them in our queries, treating
them as synonyms;
•Iterative searches: In each iteration, previously obtained keywords were used in
various combinations. This process allowed us to select the most appropriate terms
by comparing the results retrieved with their titles and abstracts;
•Verifying against established literature: To ensure the efficiency of our search terms,
we compared the results to a list of papers in the domain of hyperpartisan detection;
•Network visualization and exploration: To further validate our verification, we used
ResearchRabbit, a tool to visualize the citation-links between papers in the same
collection. It suggested similar papers written by the same or different authors, high-
lighting stored publications in the user’s folder. This helped us gauge the coherence
of our results.
The period interested in our analysis ranges from 2015 up to 2023. Queries within
each database were structured to match titles, abstracts, and keywords.
2.3 Criteria for selection
Before describing the screening process, we illustrate the criteria employed for the
paper selection.
•Inclusion criteria
– Papers primarily focused on hyperpartisan automatic detection;
– Publications from 2015 to 2023;
– Presence of news article-based datasets suitable for hyperpartisan detection.
•Exclusion criteria
– Exclusion of sources that either address the hyperpartisan detection problem from
a theoretical perspective, namely theory papers, or manual detection;
– Studies exclusively discussing only related and distinct topics, such as fake news
detection, stance detection, or political bias;
– Findings that do not use news domain datasets as the main source for hyper-
partisan detection, i.e. social network analysis, comments analysis, and tweets
detection-based approaches;
– Literature reviews, books and thesis;
– Out-of-context papers.
2https://researchrabbitapp.com
6
Records identified from*:
ACM Digital Library (n
= 723)
Google Scholar (n =
571)
IEEE Xplore(n = 1)
ProQuest (n=159)
ScienceDirect (n = 2)
Scopus (n = 97)
Records removed before
screening:
Duplicate records
removed (n = 112)
Records screened
(n = 1441)
Records excluded
(n = 1331)
Reports sought for retrieval
(n = 110)
Reports not retrieved
(n = 0)
Reports assessed for
eligibility
(n = 110)
Reports excluded (n=42):
Unrelated papers (n
=17)
Not suitable dataset (n
= 5)
Unrelated Media Bias (n
= 20)
Records identified from:
Citation searching (n =
13)
Reports assessed for
eligibility
(n = 12)
Reports excluded (n = 0)
Studies included in review
(n=80)
Identification of studies via databases and registers
Identification of studies via other methods
Identification
Screening
Include
Reports sought for retrieval
(n = 13)
Reports not retrieved
(n = 1)
Fig. 1 PRISMA flow diagram.
2.4 Screening and selection process
To manage the screening and selection processes, we opted for Rayyan3because of
its AI-powered capabilities, allowing reviewers a blind papers selection, to prevent
influencing each other during the selection process. The initial dataset consisted of 723
papers from ACM Digital Library4, 571 from Google Scholar5, 1 from ScienceDirect6,
97 from Scopus7, 159 from ProQuest8, and 1 from IEEE Xplorer9. Notably, Google
Scholar initially retrieved 1800 results, but we noted that, after the threshold of 500
results, there were some inconsistencies that led us to manually collect only the first
571 papers.
The entire selection process, as described in Figure 1, was conducted using Rayyan,
which automatically detected 112 duplicates. After manual checks, they were removed.
With a total of 1441 studies, screening titles and abstracts was the initial step. Fol-
lowing thorough evaluations, 68 papers were retained from a curated pool of 110,
eliminating 42 papers that did not meet specific focus or dataset criteria.
3https://www.rayyan.ai/
4https://dl.acm.org/
5https://scholar.google.com/
6https://www.sciencedirect.com/
7https://www.scopus.com/home.uri
8https://www.proquest.com/index
9https://ieeexplore.ieee.org/Xplore/home.jsp
7
Additionally, to examine the cohesion and coherence of our references, we used
ResearchRabbit to visualize the citation network, identifying two prominent clusters:
(Potthast et al,2018) and (Kiesel et al,2019). Thirteen additional papers were included
after exploring similar works and citations offered by the application.
Despite various reviews on related topics, our data collection reveals that, to date
and to our knowledge, no systematic review focusing on automatic hyperpartisan
detection has been conducted.
2.5 Transparency and replicability
Adhering to rigorous academic standards, emphasis was placed on transparency and
replicability. A GitHub repository has been created to house the results and the
precise queries employed, enabling fellow researchers to replicate the methodology
and verify the findings. The repository is accessible at: https://github.com/mik28j/
Hyperpartisan-detection-Systematic-review.
3 Hyperpartisanship
In this section, we first analyze the characteristics of hyperpartisanship comparing
it with a spectrum of related tasks and definitions. After that, as a result of careful
consideration, we propose a tailored definition with regard to the applied task in the
Computer Science and Computational Linguistic fields, without entering the merits of
the Political Sciences. Then, we present the various biases from which hyperpartisan-
ship forms or to which it is related. We further focus on the different sources concerning
hyperpartisan detection like sources, sentences, headlines, etc. and its domains of
application: articles, social networks, and recommendation systems. Lastly, we give an
overview of the different strategies to label an entity as hyperpartisan.
3.1 Definitions and characteristics
The term Hyperpartisanship10 is not certificated in any dictionary. It denotes an
extreme political allegiance to a party, leading to intense disagreement with the oppos-
ing faction. In the online sphere, hyperpartisanship proliferates through various media
channels, ranging from social networks to publishers’ websites. The dissemination of
hyperpartisan news, characterized by highly polarized political and ideological con-
tent, capitalizes on the virality facilitated by platform algorithms (Tucker et al,2018).
Although this term first emerged during the 2016 U.S. election (Anthonio,2019),
there is no evidence suggesting that this specific event triggered a systemic hyper-
polarization event (Bartels,2018). From a linguistic point of view, hyperpartisan
articles exhibit a high adjective and adverbs count (P´erez-Almendros et al,2019;
Dumitru and Rebedea,2019), massive use of pronouns and words of disgust (Knauth,
2019), tending to write longer paragraphs (Hanawa et al,2019) with a sensationalist
style full of emotional language and rare terms (Sengupta and Pedersen,2019). It has
been proven that right-media often tend to write hyperpartisan headlines (Lyu et al,
10https://claremontreviewofbooks.com/hyperpartisanship/
8
2023), and that news especially shows hyperpartisan traits in the titles (Amason et al,
2019).
Additionally, alternative media outlets sharing polarized content have proliferated
in recent years (Kristoffer Holt and Frischlich,2019). This phenomenon is viewed as a
threat to democracies (McCoy and Somer,2019). We selected the 2015-2023 timeframe
to analyze trends before the term was coined, considering a period in which studies on
this topic grew, and increasingly powerful models were employed. In 2019, SemEval-
2019 task 4: Hyperpartisan news detection (Kiesel et al,2019) took place, attracting
participation from 42 teams. In this context, automatic hyperpartisan news detection
consisted in a binary classification with hyperpartisan and mainstream news labels,
not considering the political and/or ideological leaning features.
This approach is specifically related to the news domain and can focus on linguistic,
semantic, and meta-data features, as detailed in section 4. For this systematic review,
we only considered automated text-based strategies applied to articles because hyper-
partisan features in the sources are intrinsically expressed within language. We are
making this distinction since even manual detection of news has been proposed and
it focused mainly on discourse analysis (Sousa-Silva,2022;Dykstra,2019;Xu et al,
2020). Although they prove to be efficient, this approach does not scale with the volu-
minous amount of daily news. To address this issue, alternative automated methods,
such as Social Network Analysis, Recommender Systems, Bots, or cross-methodology
approaches like Pescetelli et al (2022) can be considered. Hyperpartisan detection may
be categorized into three macro-approaches: content, source, and user-based (Pitoura
et al,2017). Both of them could involve data like comments (Wu and Resnick,2021),
posts, articles, user profiles and interactions, and sources. Depending on which one
is stressed, distinct datasets are built. We have outlined the primary task-tailored
datasets used for detection in section 5.
3.1.1 The problematics of the definition
The definition provided above for hyperpartisanship is minimalist and is shared by the
majority of researchers engaged in automatic hyperpartisan detection. Hyperpartisan-
ship coexists within the broader category of junk news and shares characteristics with
overlapping tasks such as political, ideological, and fake news detection (Zannettou
et al,2019). Furthermore, experts struggle to precisely agree on what hyperpartisan-
ship is. Indeed, due to the vagueness of the definition hyperpartisan headlines are
hardly clustered in the misinformation set (Altay et al,2023). While humans could
assess the degree of hyperpartisanship in a given text due to their awareness of cul-
tural and linguistic backgrounds, machines lack the same awareness. We should claim
that hyperpartisan is a subject-shifting definition, too. In other words, given a sub-
ject, the understanding of a news as hyperpartisan wether not will depend on her/his
epistemic bubble (Ross Arguedas et al,2022). Thus, classifying hyperpartisan news
presumes working as objectively as possible. Furthermore, it has also been found that
both left and right extremisms do not show significant stylistic differences (Potthast
et al,2018). Given its complexity and the limitations of datasets, when researchers
apply algorithms, they cannot simultaneously address the various components that
constitute the definition of hyperpartisanship.
9
Hyperpartisan definitions
Reference Claim Chars.
(Barnidge
and Pea-
cock,
2019)
Social media also have facilitated the rise of hyperpartisan
news. Hyperpartisan news: (1) has an obviously one-sided
political agenda, which makes no effort to balance oppos-
ing views; (2) pushes anti-system messages that are critical
of both mainstream media and establishment politics, often
relying on misinformation to do so; and (3) relies heavily on
social media as a platform for dissemination. Thus, hyper-
partisan news can be situated squarely at the intersection
of partisan and alternative news, and considerable overlap
exists between hyperpartisan news and “fake” news.
1, 6, 7
(Kiesel
et al,2019)
Hyperpartisan articles mimic the form of regular news arti-
cles, but are one-sided in the sense that opposing views are
either ignored or fiercely attacked.
1, 2,
3, 4,
5, 7, 8
(Potthast
et al,2018)
Prone to misunderstanding and misue, the term “fake news”
arose from the observation that, in social media, a certain
kind of ‘news’ spreads much more successfully than oth-
ers, and this kind of ‘news’ is typically extremely one-sided
(hyperpartisan), inflammatory, emotional, and often riddled
with untruths.
1, 3,
4, 5,
7, 8
(Lyu et al,
2023)
We think that a better understanding of hyperpartisan-
ship can be achieved by considering not only (1) the news
that contains one-sided opinions but also (2) the news that
describes conflicts and the underlying politically polarized
climate because both of them could lead to an increase in
the public’s perceived polarization (Yang et al. 2016; Fiorina,
Abrams, and Pope 2005; Levendusky and Malhotra 2016).
Additionally, coverage quantity itself can be considered as
a particular form of bias (Lin, Bagrow, and Lazer 2011). In
particular, we seek to extend previous studies’ definitions
of hyperpartisan news to include news that covers partisan
conflicts and confrontations.
1, 3,
4, 7
Continued on next page
10
Continued from previous page
Reference Claim Chars.
(Gangula
et al,2019)
Media bias can be observed and defined through various fac-
tors. In political domain, it ranges from selectively publishing
articles to specifically choosing to highlight some events, par-
ties and leaders. We also come across articles where bias can
be detected by observing the unclear assumptions, loaded
language, or lack of proper context.
1, 6,
7, 8
(Pierri
et al,2020)
Researchers use different terms to indicate the same issue,
namely disinformation, misinformation, propaganda, junk
news and click-bait. In this work we use the word disinfor-
mation, rather than the more popular ”fake news”, to refer
to a variety of low credibility content which comprises false
news intended to harm, misleading and non-factual report-
ing, hyper-partisan news and propaganda, and unverified
rumors
1, 4, 7
(Huang
and Lee,
2019)
Hyperpartisan news is news riddled with untruth and twisted
statements of information. This type of news spread more
successfully than others. Hyperpartisan news not only can
mislead readers but also can cause polarisation within a
community or society.
1, 4, 7
(University
Politehnica
of
Bucharest
et al,2021)
Although there are more subcategories related to fake news
such as satire, parody or clickbait, a general definition of
the term could be the following: fake news represent a way
to spread false information in order to mislead the public,
damage the reputation of an entity or have a political or
financial gain. The idea of misleading and influencing the
public is also linked to the notion of hyper-partisan news
that have the role of presenting extremist or conspiratorial
opinions with intentional misconceptions.
2, 3,
4, 5, 7
Continued on next page
11
Continued from previous page
Reference Claim Chars.
(Garg and
Sharma,
2022)
The aim is to classify news as real or fake. Fake news is news
that is intentionally generated to misguide people. It may
exist in various forms, including misleading content, biased
news, satire news, rumours, hyper-partisan news, deceptive
news, disinformation, clickbait, and hoax.
1, 4, 7
Table 2 This table reports diverse definitions of hyperpartisanship found in the selected papers.
All the internal quotes in the ”Claim” column belong to the paper cited and are not reported in our
bibliography. ”Reference” indicates the authors; ”Claim” refers to the textual definition given in that
paper; ’Chars’ is an abbreviation for characteristics and there we report numerical values indicating
the various types of biases that make up the definition of hyperpartisan given by the authors.
3.1.2 Definition proposal
By reviewing the various definitions collected in Table 2, several key observations
emerge:
•it is commonly acknowledged that hyperpartisan news exhibits one-sided political
bias, incorporating specific statements aligning with the ideology of a particular
political party;
•hyperpartisan detection occasionally experiences overlap or confusion (University
Politehnica of Bucharest et al,2021;Garg and Sharma,2022) with other tasks within
the disinformation realm (Pierri et al,2020;Sousa-Silva,2022;Potthast et al,2018),
particularly like Fake News (Garg and Sharma,2022;Ross et al,2021;Mour˜ao
and Robertson,2019;Sousa-Silva,2022;Nguyen et al,2019;Agerri,2019), click-
bait and stance detection (Bourgonje et al,2017). Specifically, Hyperpartisanism
might potentially be conveyed through certain elements of fake news in the article,
often aimed at propagating a specific agenda and manipulating readers to adopt a
particular position on a given topic (Agerri,2019).
•define the concept of hyperpartisanship as an intersected field sharing features with
these typologies of media biases discussed in subsection 3.2:
1. spin bias;
2. ad hominem bias;
3. opinion statement bias;
4. ideology bias;
5. framing bias;
6. coverage bias;
7. political bias;
8. slant bias;
•claim that some researches (Kim and Johnson,2022;Patankar et al,2019) imply
the definition while approaching this task.
Given this analysis, we can define the shared and individual traits between the
various biases and hyperpartisanship. There are several taxonomies proposal for junk
news like (Zannettou et al,2019;Kapantai et al,2021). We will use the categories
12
collected by Oxford11 and Rodrigo-Gin´es et al (2024). In conclusion, these biases
form the definition of hyperpartisanship, which therefore varies depending on the
task, and it is not possible to define it unambiguously without generalizing it in the
following formula:
Hyperpartisan news detection refers to the identification of extremely
one-sided articles. In this statement, the use of the adjective "hyper-" is
justified by the exaggerated adoption of at least one of the aforementioned
types of biases. Thus, the ideological belonging is realized through the
linguistic exaggeration expressed in the type of bias used in at least one
part of the text considered.
In this context, it is essential to avoid conflating the reification of the social phe-
nomenon involving linguistic indicators with the entirety of the specified biases. Not
all categories of biases mentioned earlier can be classified as hyperpartisan when they
manifest. The linguistic element of exaggeration, in isolation, does not automatically
denote hyperpartisanship; rather, it necessitates contextual positioning, such as align-
ing with a particular party or ideology. Simply adopting a stance is insufficient for
categorization as hyperpartisan; it is the degree of exaggeration in that stance that
holds significance, as illustrated in Table 3.
While in the Political Sciences, some formulas such as the CSES Polarization Index
(PI) are used to track the degree of partisanship variation across parties (Dalton,2021),
regrettably, the textual instances of exaggeration that validate the term ”hyper” in
the provided examples lack mathematical representation. It is important to note the
absence of mathematical translations for linguistic phenomena embodying the ”hyper”
in ”hyperpartisanship”. This gap significantly influences how datasets are constructed
and which analogous tasks they are conflated with when utilized for training models.
In light of these considerations, hyperpartisan detection must necessarily consider
both variables simultaneously: positioning and exaggeration. Does the current state of
the art in detection methodologies do this? As mentioned earlier, a detection method
that simultaneously considers the different types of biases and these two variables
has not been conducted. Various research works tend to focus individually on specific
subsets of linguistic and content-based features, as outlined in the following chapter.
3.2 Analogue biases
Hyperpartisan detection is a task in which certain textual features indicate that the
writer is expressing an extremist, one-sided opinion. This can be achieved through
various typologies of bias that contribute to the creation of hyperpartisan news.
Spin bias, or rhetoric bias (Rodrigo-Gin´es et al,2024), strictly concerns the lin-
guistic structure of the article, from the moment that rhetoric is strictly linked with
persuasion. The deliberate or inadvertent misrepresentation of research outcomes,
leading to unjustified indications of positive or negative results, potentially could result
in misleading conclusions. Written language is the product of the conscious application
of both certain strategic discursive patterns and persuasion to persuade the readers.
11https://catalogofbias.org/biases/spin- bias/
13
Examples of hyperpartisan bias
Bias Biased example Hyperpartisan biased example
Spin bias ”How can we trust their solutions when
they fail to understand the basic prin-
ciples of economics?”
Their economic policies are a disaster,
proving yet again their ignorance and
incompetence.
Ad hominem The proposal is flawed because it
comes from someone who has no expe-
rience in the field.
We can’t expect anything good from
someone who has never done anything
worthwhile!
Opinion state-
ment bias
It’s clear that this is the best approach
for our society.
This is unequivocally the only right
path forward for our nation; anyone
who disagrees is simply blind to the
truth.
Ideology bias Socialist policies always lead to ineffi-
ciency and economic downfall.
The leftist agenda destroys economies
and personal freedoms every time it’s
implemented.
Framing bias The data supports that this policy will
decrease crime rates.
The indisputable facts confirm that
only this policy can save us from spi-
raling into a crime-infested nightmare.
Political bias This party’s proposal will bankrupt
the nation.
The opposition’s plan is a surefire way
to plunge our country into insurmount-
able debt and chaos.
Slant bias The research somewhat suggests that
this course of action might be benefi-
cial.
The overwhelming evidence proves
that this action is the only solution we
urgently need!
Table 3 This table reports diverse examples of ”Biased example”: statements for specific biases; and
”Hyperpartisan biased example”: hyperpartisan statements for that bias.
In its specter, the words used contribute to giving a particular meaning to the entire
text, especially if they leverage an emotional lexicon with superlatives.
The personal attack or ad hominem bias refers to a tendency or inclination to focus
on the personal characteristics or traits of an individual rather than considering the
substance of their argument or position. It involves attacking the person making an
argument rather than addressing the argument itself, often by discrediting or belittling
the individual instead of engaging with the ideas they present (Walton,1998).
Opinion statement or presence bias involves the inclusion of subjective opinions
within news articles, influencing readers’ perceptions. It occurs when factual report-
ing is mingled with subjective viewpoints or opinions (Tran,2020). In other words,
it reflects the degree of agreement and statement sharing of an entity, i.e. users or
publishers (Anand et al,2007).
Ideological bias occurs when news reporting or content is influenced by a particu-
lar ideological stance or viewpoint, impacting the presentation and selection of news
topics. Ideological detection is different from political bias because some ideologies can
be shared even by opposite parties. Often ideologies contrast each other, but to be
classified they need this comparison (Sharma et al,2020).
Framing bias involves the presentation of information in a way that shapes or
influences people’s perceptions of an issue or event by emphasizing certain aspects
14
while downplaying others (Baumer et al,2015;Roy and Goldwasser,2020). In this
case, the specific use of both linguistics and rhetorical figures helps the author partially
present the selected information. Therefore, framing is a feature of publishers leaning
towards a certain ideology. It is performed in both moral content and style used (Xu
et al,2020).
Coverage bias, that is not present in Table 3 since it is not a textual bias. It
refers to the disproportionate attention or neglect of certain topics or events in news
reporting, leading to an imbalance in coverage across different subjects (Leeson and
Coyne,2011).
Political bias could be easily confused with ideological bias. Since a party is a
combination of both an ideology and a political leaning, this bias is related to the
inclination of news media or information sources or people to favor one political party,
ideology, or perspective over others, impacting the content and presentation of news
(Honeycutt and Jussim,2023).
Slant bias involves the subtle or overt leaning of news coverage towards a particular
viewpoint or interpretation, impacting the overall narrative presented to the audience.
Differently from the framing, it influences the overall tone, emphasis, or interpretation
of a story. Frames are tools that emphasize specific information while potentially
favoring one aspect over another, with or without being slanted (Kong et al,2018).
3.3 Where can Hyperpartisanship be detected? Perspectives
on the sources
In the following paragraphs, we are going to introduce a broader view of the paths
that lead to partial solutions of hyperpartisan detection. There, we also discuss papers
not included in the selection process since we aim to give a much more exhaustive idea
as possible. We will cover the papers selected in the section 4.
In light of the prevalence of hyperpartisan news dissemination online, the method-
ologies discovered are implemented specifically on online publishers and social
networks. Hyperpartisan characteristics permeate various online spheres and can be
disseminated by multiple entities. Initially, when considering the domain of publish-
ers, a linguistic approach can be applied to news analysis to detect hyperpartisanship.
This involves studying textual information within articles using style-based or topic-
based models (S´anchez-Junquera et al,2021;Potthast et al,2018;Lyu et al,2023;
Sm˘adu et al,2023). Detection methods may begin with specific sections, such as the
title (Lyu et al,2023;Amason et al,2019), sentences (jeong Lim et al,2018), quotes in
the body (P´erez-Almendros et al,2019), or encompass both (Naredla and Adedoyin,
2022;Gangula et al,2019;Papadopoulou et al,2019;Lyu et al,2023;Nguyen et al,
2019). On the other hand, the entities involved in the writing and publishing process
were taken into account. Indeed, the polarization detection of the article starting from
the journalist’s leaning was implemented by M. Alzhrani (2022). Considering publish-
ers as entities often interconnected through economic and political bonds (Hermann
and Chomsky,1994) they form a polarized network, which can be analyzed using
metadata like external links (Hrckova et al,2021;Kulkarni et al,2018;Alabdulka-
rim and Alhindi,2019;Joo and Hwang,2019). While determining bias based on the
source is feasible (M. Alzhrani,2022;Alzhrani,2020), it is important to note that
15
an article from a biased media outlet may not always be hyperpartisan (Tran,2020;
Jiang et al,2019). This issue was underscored by Jiang et al (2019), which highlighted
the inadequacy of the information source in determining an article’s hyperpartisan-
ship. Additionally, research (Patankar et al,2019) proposes a hybrid methodology
focusing on both source, topic modeling, and real-world knowledge through graph
(Ko et al,2023). This method generates a system capable of indicating bias scores in
news and suggesting similar topics from different sources to encourage readership of
diverse perspectives or to avoid extremely biased news. Lastly, working with textual
data allows for sentiment feature extraction (Potthast et al,2018;Chen et al,2019;
Baly et al,2019;Pali´c et al,2019;Anthonio and Kloppenburg,2019;Srivastava et al,
2019). Finally, there is also research that used textual and image features to detect
hyperpartisanship (Spezzano et al,2021).
In (Kim et al,2023a), the objective is to identify political bias in news arti-
cles by making use of prompt-based learning. More precisely, the authors designed a
strategy involving two sequential stages of prompt tuning (soft prompts) for political
perspective detection: First, the strategy tunes the domain-specific prompts with a
frozen pre-trained language model and then, after having tuned the domain-specific
prompts, the second stage prepends task-specific prompts to the model for use in
downstream tasks. Their experimental results confirm that this methodology signifi-
cantly outperforms the strong baseline methods: both fine-tuning and the use of hard
prompts.
Polarization deeply affects both analog and digital domains, particularly social
media (Yarchi et al,2020;Hawdon et al,2020). Another significant field contributing
to the detection of polarized communities and the analysis of spreading information
is Social Network Analysis. It aims to reconstruct and analyze the interaction graph
between subjects, i.e. users, news publishers, etc. Within the user domain, distinctions
are made between filter bubbles and echo chambers. Filter bubbles result from con-
tent personalization by search engine ranking algorithms, while echo chambers emerge
from the self-aggregation of partisan individuals sharing common beliefs and resisting
ideological change, fostering the homophily within the group (Ross Arguedas et al,
2022). Both phenomena exacerbate societal polarization. Consequently, the study of
filter bubbles involves analyzing recommender systems (Liu et al,2021), while investi-
gating the creation and evolution of echo chambers among hyperpartisan Reddit users
including modeling their interaction graph (Morini et al,2021).
3.4 How hyperpartisanship is labeled?
Understanding the measurement of hyperpartisanship involves considering the diverse
scales utilized across various selected papers. In the realm of Social Sciences, a range
of indexes and scales is employed for this purpose, leveraging distinct features from
those used in automatic detection methodologies. For instance, polarization could be
measured with an 11-point Left-Right scale or with the CSES Polarization Index.
The Common CSES Polarization Index (PI) is a tool used to assess the distribution
of political parties across the Left/Right ideological spectrum. It not only gauges
ideological positioning but also accounts for party sizes or vote shares, offering a
comprehensive view of ideological stance and political influence (Dalton,2021). In
16
contrast, automatic hyperpartisan detection relies on linguistic features. Some studies
employ binary classification methods, utilizing labels such: hyperpartisan/mainstream
(i.e. non hyperpartisan) (Kiesel et al,2019), Left/Right (Kim and Johnson,2022;
M. Alzhrani,2022). However, such distinctions often overlook nuanced differences
within diverse political leanings (Potthast et al,2018). Few studies have extended their
scope to include a broader polarization range (Sridharan and S.,2022). For example,
Aksenov et al (2021) approached hyperpartisan detection as a multi-class classification
problem, employing both 7- and 5-point scales to define affiliations: 1-2.5 – far-left,
2.5-3.5 – center-left, 3.5-4.5 – center, 4.5-5.5 – center-right, 5.5-7 – far-right. Similarly,
Baly et al (2019) used a scale and Azizov et al (2023) sought to manage granularity
by distinguishing between right, center, and left affiliations.
4 Approaches for Automatic Hyperpartisan news
detection
The detection of hyperpartisan content encompasses a range of methodologies, varying
from traditional non-deep learning approaches to cutting-edge deep learning tech-
niques, as well as mixed learning algorithms. Non-deep learning methodologies often
rely on traditional machine learning algorithms, leveraging handcrafted features and
rule-based systems to identify linguistic patterns, stylistic markers, and network struc-
tures within textual and metadata sources. These approaches commonly include
stylometric analysis and topic modeling methods to discern biased content. In contrast,
deep learning methodologies harness the power of neural networks to automatically
extract intricate features from raw data, enabling the identification of complex pat-
terns and relationships in unstructured text or network data. These techniques, such
as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and
Transformers, excel in learning representations directly from the data.
4.1 Model Perspective
In the following subchapters, we will differentiate between Non-deep learning, Deep
learning, and Other methodologies adopted in the papers selected for the systematic
review. The Deep Learning section is divided into Non-transformers Deep Learning
models and the Transformers family. In the following tables: Table 4,Table 5,Table 6,
Table 7 we categorize the best model and report its performance for each paper,
even though authors compared diverse methodologies with several baselines. In some
circumstances, we report more models when researchers adopted them for the same
task but on different datasets, like the case of Kim and Johnson (2022).
4.1.1 Non-Deep Learning methods
In Table 4, we categorized papers using the traditional machine learning approaches.
The majority of the methodologies involved algorithms like Support Vector Machines,
Random Forest, and Logistic Regression, followed by Linear regression, Naive Bayes,
Linear SVC, KNN, XGBoost, and Maxent.
17
Non Deep-Learning Methods
Algorithm Reference
Support Vector
Machine (SVM)
(Hearst et al,1998)
P´erez-Almendros et al (2019); Nguyen et al (2019); Knauth (2019);
Alabdulkarim and Alhindi (2019); Anthonio and Kloppenburg (2019);
Yeh et al (2019); Dumitru and Rebedea (2019)
XGBoost (Chen
and Guestrin,
2016)
Gupta et al (2019)
Maxent (Merow
et al,2013)
Agerri (2019)
Random Forest
(Breiman,2001)
Chakravartula et al (2019); Cruz et al (2019); Stevanoski and Gievska
(2019); Naredla and Adedoyin (2022); Aksenov et al (2021)
Naive Bayes Chen et al (2019); Amason et al (2019)
Ordinal Regression Baly et al (2019)
Logistic Regression Sengupta and Pedersen (2019); Garg and Sharma (2022); Saleh et al
(2019); Bestgen (2019); Srivastava et al (2019)
Linear Support
Vector Classifier
(SVC)
Pali´c et al (2019)
Linear Classifier Hanawa et al (2019)
Table 4 This table describes the traditional Machine Learning algorithms used in the
selected literature. ”Algorithm” specifies the method; ”Reference” indicates the authors who
used that method.
There are effective strategies adopted with the SVM model. For instance, Alab-
dulkarim and Alhindi (2019) used an algorithm to analyze the emotional content in
the article: the National Council Canada’s Emotion Lexicon (NRC Emotion Lexi-
con). Moreover, they extended the linguistic approach with the vocabulary Linguistic
Inquiry and Word Count (LIWC). Moreover, both the articles’ structure and their
meta-data were considered as features. Another team at the competition, Nguyen
et al (2019), applied n-grams, i.e. bi- and tri-grams, and dependency sub-trees that
impacted on the performance. On the other hand, Yeh et al (2019) experimented sev-
eral embeddings: Doc2Vec (Le and Mikolov,2014), Glove (Pennington et al,2014),
ELMo (Peters et al,2018) and many others. The substantial difference with the previ-
ously mentioned team was that they found out that the inclusion of basic lexical and
sentiment features adversely impacts the overall performance. Dumitru and Rebedea
(2019) studied the linguistics divergencies between fake and hyperpartisan news using
SVM along with other models. In this case, it emerged that hyperpartisan articles
exhibit more sentences and a higher adjective count compared to normal news. When
comparing the characteristics of extremely polarized against fake news they noted that
the former contains high usage of question/exclamation marks and adjectives. These
sentence-related features delineate distinct linguistic patterns. The robust potentiali-
ties of the Logistic Regression are confirmed by the second place at the SemEval-2019
obtained by Srivastava et al (2019). The team built representations with Universal
Sentence Encoder (USE) (Cer et al,2018) and combined semantic and handcrafted
18
features, paying attention to not only the grade of the adjectives and subjectivity, but
also to two distinct levels of polarity: sentence and article level. Garg and Sharma
(2022) used the Reuter Dataset for the training and the test, combining the ELMo
embedding with a logistic regression classifier as already done by (Jiang et al,2019)
and (Srivastava et al,2019), confirming the effectiveness of this method. Srivastava
et al (2019) discovered that the most relevant features are the ones concerning bias
lexicon and polarity. Hanawa et al (2019) placed third at the SemEval-2019 finding
that article length was a distinctive trait of biased articles. Working at the phrase level
they created a set of phrases to discern the different types of articles, paying atten-
tion to remove n-grams containing publishers’ style biases. Lyu et al (2023) focused on
news titles with a topic-based approach. They also built a dataset considering two dis-
tinct typologies of news titles, augmenting the granularity of the detection. The first
category pertains to descriptions of confrontations or conflicts between opposing par-
ties, suggesting a deeply polarized political climate, while the second involves opinions
that express a biased, inflammatory, and aggressive stance against a policy, a politi-
cal party, or a politician. Baly et al (2019) thought that there is an interdependence
between factuality and political ideology bias. Thus, they introduced a multi-task
learning setup with the Copula Ordinal Regression (COR) (Walecki et al,2016). They
used the entire news outlets, as opposed to working at the article level. Lastly, they
considered diverse scales for measuring factuality (3-point scale) and political bias
(7-point scale). Another approach was introduced by Agerri (2019). Referred to as
Maximum Entropy Modeling (MaxEnt), it predicts species occurrences by identify-
ing distributions that display maximal dispersion or closely resemble uniformity. This
method takes into account environmental constraints observed in known locations. In
fact, by-passing linguistics features in order to build a model capable of generalizing
as much as possible, Agerri (2019) devised a document classification system that com-
bines clustering features with simple local features. They showcased the effectiveness
of employing distributional features from large in-domain unlabeled data. Gupta et al
(2019) approached the task using n-gram embeddings with article and title polarity,
implementing the XGBoost model with all of these scalar features, but it performed
poorly. They derived their methodology of applying stylometric analysis from Potthast
et al (2018), who utilized n-grams along with readability scores and Part-of-Speech
(PoS) followed by binary classification. Thanks to unmasking information, they were
able to compare simultaneously documents with opposite political leaning. In doing
so, Potthast et al (2018) investigated the style variations depending on the political
orientation and confront it with topic-based bag of words models. This methodology
highlighted the limited usefulness of integrating corpus characteristics when perform-
ing a granular distinction amongst left, right and mainstream styles. Indeed, both the
political extremes show similarities and can produce confounding effects in the model.
Hence, concerning the style analysis for the hyperpartisan detection, the categories
should be limited to mainstream and hyperpartisan without considering the specific
leaning.
Furthermore, for a complete understanding of the approaches used in the litera-
ture, we summarized them in the Table 5. In this case, although ELMo, BERT, and
19
Word2Vec embeddings are deep learning techniques, they were used just as features
of Non-Deep Learning algorithms.
20
Features used in Non Deep-Learning Models
Reference Morpho-
syntactic
Lexicon Semantic Sentiment Metadata style-
based
topic-
based
(P´erez-
Almendros
et al,2019)
x x GloVe x
(Nguyen et al,
2019)
xn-grams x
(Dumitru
and Rebedea,
2019)
TF-IDF x x x
(Hanawa
et al,2019)
BERT
embed-
dings
x
(Knauth,
2019)
x x x x
(Alabdulkarim
and Alhindi,
2019)
x x TF-IDF
uni-
grams
x x x
(Anthonio
and Klop-
penburg,
2019)
x x x
(Yeh et al,
2019)
BoW
(Sengupta
and Pedersen,
2019)
unigrams
(Gupta et al,
2019)
xn-gram x x
(Agerri,2019) x Word2Vec x
(Chakravartula
et al,2019)
BoW
(Garg and
Sharma,
2022)
ELMo
embed-
ding
x
Continued on next page
21
Continued from previous page
Reference Morpho-
syntactic
Lexicon Semantic Sentiment Metadata style-
based
topic-
based
(Cruz et al,
2019)
x BoW x
(Stevanoski
and Gievska,
2019)
x x Word2Vec x x
(Naredla and
Adedoyin,
2022)
ELMo
embed-
ding
x
(Chen et al,
2019)
x x BoW x
(Amason
et al,2019)
x x BoW x x
(Baly et al,
2019)
x x Word2Vec x x
(Saleh et al,
2019)
x x BoW x
(Bestgen,
2019)
x BoW
(Srivastava
et al,2019)
x x Universal
Sentence
Encoder
(Cer
et al,
2018)
x x
(Pali´c et al,
2019)
x x Word2Vec x x
Table 5 This table describes only the features used with the best models proposed in Non-Deep
Learning approaches in Table 4. We distinguish between features (Morpho-syntactic, Lexicon,
Semantic, Sentiment and Metadata) and approaches (style-based and topic-based). We indicate the
latter only when explicitly stated in the article. The features are the following: Morpho-syntactic,
Lexicon, Semantic and Sentiment. While the approaches are: Style-based and Topic-based. The
style-based approach consists of using tools and methods like vocabularies, reading scores, count of
pronouns and other parts of speech, n-grams of part of speech, punctuation, lemma n-gram
similarity score, and article length features to analyze the writing style. Topic-based refers to
methods for the usage of topic clusters and the creation of ad-hoc vocabularies to compare topics.
Meta-data includes internal and external links, dates, and journalists’ bios. ”Morpho-syntactic”
involves using Part-of-Speech tagging, dependencies trees, and quantities like the count of words,
and adjectives. ”Lexicon” refers to the utilization of external vocabularies like LIWC, or
handcrafted. ”Semantic” regards the adoption of embeddings and frequency-based features: Tf-IDF,
FastText, GloVe, and BERT embeddings. ”Sentiment” indicates that sentiment analysis algorithms
(VADER, NRC Emotion Lexicon, etc.) were applied.
22
4.1.2 Deep Learning methods
In the following paragraphs, we analyzed the Deep Learning methods adopted by
diverse authors to solve the hyperpartisan detection task. In Table 6, we first grouped
them considering the utilized model. Lastly, at the end of the chapter, a comprehensive
table, Table 7, illustrates the results scored by the authors.
23
Features used in Non Deep-Learning Models
Algorithm Description Reference
Recurrent
Neural
Network-
based
(RNN)
(Rumel-
hart et al,
1986)
Recurrent Neural Networks are a class
of artificial neural networks designed
to process sequential data by maintain-
ing an internal memory. They possess
loops that allow information persis-
tence, enabling them to consider past
inputs for present computations, mak-
ing them adept at handling time series
or sequential data in various applica-
tions.
(Sharma et al,2020)
Mesh
Neural
Network
(Sridha-
ran and
S.,2022)
It is a neural network aiming at max-
imazing the weight of past examples.
Its structure is composed by recurrent
nodes that boost the inductive abilities
of the machine.
(Sridharan and S.,2022)
Catboost
(Doro-
gush
et al,
2017)
Catboost stands out as a specialized
library crafted for boosting gradients. It
employs a unique algorithm to prevent
target leakage, ensuring high efficiency
and precision.
(Azizov et al,2023)
Convolutional
Neural
Network
Convolutional Neural Networks were
designed for images processing, but
were applied to Natural Language Pro-
cessing, where treat the input text as
a one-dimensional vector of tokens, or
like a matrix of embeddings. This model
can learn both local and global depen-
dencies as well as hierarchical features
in the data.
(Papadopoulou et al,
2019;Gerald Ki Wei and
Jun Choi,2021;P´erez-
Almendros et al,2019;Jiang
et al,2019;F¨arber et al,
2019;Joo and Hwang,2019;
Zehe et al,2019;Ruan et al,
2021)
Hierarchical
Attention
Network
(HAN)
The HAN is a model capable of balanc-
ing the information in a current state,
deciding whether updating it and how
much the past information contributes
to its new state.
(Cruz et al,2020;Moreno
et al,2019;M. Alzhrani,
2022;Gangula et al,2019;
Ko et al,2023)
Continued on next page
24
Continued from previous page
Algorithm Description Reference
Long
Short
Term
Memory-
based
models
(Hochre-
iter and
Schmid-
huber,
1997)
Unidirectional LSTM retains informa-
tion solely from preceding inputs as it
has only processed data from the past.
Employing a bidirectional approach
involves processing inputs in two direc-
tions: from past to future and from
future to past. The distinctive aspect
of this method, compared to unidirec-
tional LSTM, lies in the LSTM run-
ning in reverse, which captures informa-
tion from the future. By amalgamating
the two hidden states, this technique
enables the retention of information
from both past and future at any given
time. In NLP, the main difference lies
on the context captured. Bi-LSTM can
discover hidden reversed semantic and
syntactic structures.
(Isbister and Johansson,
2019;Li and Goldwasser,
2021;Cramerus and Schef-
fler,2019;Zhang et al,2019;
Roy and Goldwasser,2020)
Continued on next page
25
Continued from previous page
Algorithm Description Reference
RoBERTa
(Liu et al,
2019)
RoBERTa, an extension of BERT,
refines language understanding by opti-
mizing training techniques, removing
sentence order prediction, and lever-
aging larger-scale data. It enhances
pre-training for improved performance
across various language tasks, achiev-
ing state-of-the-art results in natural
language processing.
(Kim et al,2023b;Liu et al,
2022;Kim and Johnson,
2022;Sm˘adu et al,2023)
BERT
(Devlin
et al,
2018)
BERT (Bidirectional Encoder Repre-
sentations from Transformers) is a pre-
trained language model by Google,
adept at understanding context in
both directions of a sentence. It uti-
lizes Transformer architecture, enabling
versatile language understanding for
diverse tasks like question answering
and natural language understanding.
(Ruan et al,2021;Lee et al,
2019;Kim and Johnson,
2022;S´anchez-Junquera
et al,2021;Da Silva and
Paraboni,2023;Baly et al,
2020;Shaprin et al,2019;
Ahmed et al,2023b;Shaprin
et al,2019;Drissi et al,
2019;Lyu et al,2023;Tran,
2020;Huang and Lee,2019;
Ahmed et al,2023a;Mutlu
et al,2019;Ning et al,2019;
S´anchez-Junquera,2021)
Table 6 This table describes the Deep Learning algorithms and Tranformers used in the literature.
Deep Learning - non Transformers methods
P´erez-Almendros et al (2019) employed a fusion of CNN and LSTM, utilizing quan-
titative linguistic features extracted through GloVe. This amalgamation, combined
with document embeddings, highlighted the crucial role of incorporating linguistic
features alongside representations based on word vectors. Additionally, they built a
meta-classifier to filter noisy data to apply to the by-publisher dataset. The winner of
the SemEval-2019 Task 4 competition, Jiang et al (2019), combined rich morphologi-
cal and contextual representations by averaging the three vectors per word into ELMo
embeddings. Their model was used for further studies by Ruan et al (2021) for pseudo-
labeling frameworks: Overlap-checking and Meta-learning. The first, by the addition
of more data, helps the model to be trained, while the second allows the model to be
continually trained on a clean dataset and a pseudo dataset. This work inspired Cruz
et al (2020). In their article, they used a HAN combined with ELMo embeddings.
The HAN is a model capable of balancing the information in a current state, decid-
ing whether to update it and how much the past information contributes to its new
state. In this case, the information stems from sentence level, confirming that richer
article representations yield better performances. They reached the state-of-the-art in
26
2020 on the SemEval-2019 Task 4 dataset, by encapsulating the structure of the arti-
cle, connectors and paying attention to stylistic markers, handcrafted stylistic features
and emotion lexicons. Ko et al (2023) proposed an improvement for the HAN stan-
dard model by introducing knowledge encoding (KE) components. The HAN segment
functions to grasp word and sentence relationships within a news article, employing a
structured hierarchy across three levels—word, sentence, and title. Meanwhile, the KE
component integrates common and political knowledge associated with real-world enti-
ties into the prediction process for determining the political stance of the news article.
Since the model is not language-based, it could work with diverse languages beyond
English. Li and Goldwasser (2021) developed a pre-training framework with the aim
of encoding knowledge about entity mentions, namely masked tokens as frame indi-
cators, and modeling the propagation between users with a social information graph.
They observed that models pre-trained on general sources and tasks lack the capacity
to concentrate on biased text segments. Cramerus and Scheffler (2019) introduced a
voting system of LSTMs to build a controlled dataset to train another LSTM. It was
an example to demonstrate the importance of having a balanced and clean dataset to
run experiments. While Roy and Goldwasser (2020) built a Hierarchical-LSTM applied
to subframes (n-grams) to tackle the framing bias. In this paper, they introduced a
pioneering framework aimed at pretraining text models utilizing signals derived from
the abundant social and linguistic context available, encompassing elements such as
entity mentions, news dissemination, and frame indicators.
Transformers
Regarding the Transformers architectures, in the studied literature we observed a
massive utilization of BERTbase and BERTlarge. BERTbase is a pre-trained BERT
model trained on a smaller dataset than BERTlarge. BERTbase differentiates itself
in cased and uncased, depending on whether distinguishes between cases and uncased
words. Baly et al (2020) wanted to remove the bias when modeling the medium. They
observed that a combination of bias mitigation with triplet loss, Twitter bios and
media-level representations increased the model efficacy. Kim and Johnson (2022) by
adopting a contrastive learning method proposed a multi-task BERT-based model
to tackle framing bias in news articles. Da Silva and Paraboni (2023) with BERT
and combinations of both syntactic bigram counts and psycholinguistic features inves-
tigated the inference of political information and hyperpartisanship on author and
text level starting from linguistic data. Shaprin et al (2019) considered hyperpartisan
classification as related to hyperpartisan detection, showing that fine-tuning entails
better results. Ahmed et al (2023b) introduced a semi-supervised framework trained
using federating learning, namely algorithms are trained independently across diverse
datasets. Furthermore, textual data are tagged in order to extrapolate wh-questions
replies and temporal lexicon information with ease. The same author replicated this
approach in (Ahmed et al,2023a). In the quest for precise detection and data denois-
ing, there employed an attention-based strategy to learn text representation. This
method aims to identify target expressions accurately while also extracting pertinent
contextual information. Utilizing lexicon expansion, content segmentation and tempo-
ral event analysis, it generates a BERT attention embedding query. Ultimately, this
27
approach enhances the understanding of consecutive news articles within a temporal
framework. Drissi et al (2019) experimented using BERTbase and BERTlarge feeding
them with embeddings of different lengths. They were interested in analyzing the parts
of the articles, looking for a consistent level of hyperpartisanship that demonstrated to
exist. Huang and Lee (2019) from the confrontation between BERT and ELMo mod-
els, confirmed that the inputs and embeddings dimensions contributed to affecting
positively the performance. Mutlu et al (2019) performed domain adaptation, showing
its efficacy. Kim et al (2023b) operated in a low-resource scenario with prompt-based
learning and employed masked political phrase prediction and a frozen pre-trained lan-
guage model that relies on transformer architecture, specifically utilizing the robustly
optimized BERT approach known as RoBERTa as a backbone for their own model,
MP-tuning. Liu et al (2022) focuses on political ideology and stance detection, compar-
ing triplets of documents on the same history to detect dissimilarities amongst them.
They trained RoBERTa through continual learning. Whereas, concerning Sm˘adu et al
(2023), they improved their model’s performance with domain adaptation, particu-
larly with cross-domain contrastive learning and this work is noticeable that they used
GPT-2 for augmenting hyperpartisan textual data.
4.1.3 Other methods
Within the vast landscape of computational frameworks, certain algorithms defy clas-
sification within the traditional realms of deep learning or non deep learning. This
chapter delves into the exploration of these unique frameworks—sophisticated amal-
gamations of diverse models, labeling techniques and graph approaches—that operate
beyond the conventional boundaries of established categorizations.
Tran (2020) applies a framework for presentation bias, studying hyperpartisanship
with a graph-based method. This three-step framework is so structured: collecting
related-articles clusters on the same topic; applying Aspect-based Sentiment Analy-
sis (ABSA) with BERTbase to rate and classify fine-grained opinions in the pairs of
sentences; the variation in bias between news sources within similar categories is fig-
ured out by contrasting the scores of matching pairs of articles. This comparison is
done for every combination of news sources within these categories, and the differ-
ences in bias are averaged across all article groups. This averaging process leads to the
development of a bias matrix. Kulkarni et al (2018) proposed a multi-view document
attention model (MVDAM) capable of modeling at the same time title, structure and
metadata like links in order to estimate the political ideology of a news article. This
framework based on the Bayesian approach, utilizes different models for creating the
3-D representation: a convolutional neural network for learning the title, Node2Vec
for the network and HAN for the content. Afsarmanesh et al (2019) worked mostly
on manual features like metatopic, namely polarizing topics, using an end-to-end tool:
The Gavagai Explorer, which performed poorly. The IMPED model (noa,2021), uti-
lizes language and time-based structures found in social media posts to predict the
probability of content removal online. It achieves this by employing a parametric sur-
vival model. This model operates under the assumption that content of poor quality
displays distinctive traits enabling its classification on a large scale. Patankar et al
(2019) delved into three pivotal areas to assess a bias-ranking system’s efficacy within
28
news articles. Researchers initially analyzed the correlation between user-generated
and algorithm-generated bias ranks to gauge the system’s alignment with user per-
ceptions. Subsequently, subjective user reviews indicated widespread potential for the
news recommendation system, albeit with concerns about personal political aware-
ness influencing biased judgment. Lastly, bias scores across diverse news article types
and sources unveiled significant variations, spanning Wikipedia featured articles, non-
featured articles, blogs, and current news pieces. These findings substantially deepen
comprehension of user perceptions, system usability, and its effectiveness in delivering
balanced news. They provide crucial insights for refining bias-aware news recommen-
dation systems, aiming to offer more equitable information to users. Potthast et al
(2018) performed both an orientation prediction and hyperpartisan classification task
using an unmasking technique with binary classifiers. For the first task, they found
out that left-wing news tends to be easily misclassified. Whereas when comparing
hyperpartisanship they used a binary classification. This study noticed that indi-
vidual political orientation is struggling to predict and that a style-based approach
overcomes the content-based one. Moreover, they discovered that there are subtle
differences in style between hyperpartisan news belonging to different political lean-
ings. S´anchez-Junquera et al (2021) using masking and transformer-based models
proved that topic-based approaches lead to better results than style-based. Instead,
S´anchez-Junquera (2021) made a comparative examination of BERT-based models
and masking-based models, enriching comprehension regarding the strengths and con-
straints of varied approaches in bias detection, offering crucial insights for upcoming
research and advancements in this domain. In essence, these models’ contribution lies
in their capacity to augment the precision, clarity, and comprehensibility of bias detec-
tion within political and social discussions. Consequently, they propel advancements
in this pivotal research domain.
29
Reference Dataset Accuracy F-1
Potthast et al (2018) The BuzzFeed-Webis
Fake News Corpus 2016
0.75 0.78
Sridharan and S. (2022) Own 0.4467
Ruan et al (2021) Semeval-2019 0.870 0.815
Papadopoulou et al (2019) Semeval-2019 0.608 0.712
P´erez-Almendros et al (2019) Semeval-2019 0.742 0.710
Gupta et al (2019) Semeval-2019 0.548 0.283
Kim and Johnson (2022) Framing Triplet Dataset 0.841
Lyu et al (2023) Own 0.84 0.78
Naredla and Adedoyin (2022) Semeval-2019 0.88
Isbister and Johansson (2019) Semeval-2019 0.803 0.806
Agerri (2019) Semeval-2019 0.737 0.728
Sengupta and Pedersen (2019) Semeval-2019 0.704 0.679
Dumitru and Rebedea (2019) Own 0.928 0.810
Chakravartula et al (2019) Semeval-2019 0.591 0.695
Aksenov et al (2021) GERMAN Dataset 0.79
Sm˘adu et al (2023) Semeval-2019 0.644 0.694
Chen et al (2019) Semeval-2019 Task 4 by-
article
0.739 0.745
Drissi et al (2019) Semeval-2019 Task 4 by-
article
0.771 0.747
Amason et al (2019) Semeval-2019 Task 4 by-
article
0.653 0.730
Tran (2020) PoliNews - -
Huang and Lee (2019) Semeval-2019 0.684
Gerald Ki Wei and Jun Choi (2021) The BuzzFeed-Webis
Fake News Corpus 2016
0.734
Sharma et al (2020) Own 0.846 0.867
Ko et al (2023) Semeval-2019 0.9521
S´anchez-Junquera et al (2021) The BuzzFeed-Webis
Fake News Corpus 2016
(cleaned)
0.89 0.86
Continued on next page
30
Continued from previous page
Reference Dataset Accuracy F-1
Kim et al (2023b) Semeval-2019 0.914
Kulkarni et al (2018) Own 0.801 0.796
Nguyen et al (2019) Semeval-2019 0.747 0.743
Azizov et al (2023) Task 3A 0.694 0.690
Cruz et al (2020) Semeval-2019 0.825 0.815
S´anchez-Junquera (2021) Stereoimmigrants 0.86 0.83
Knauth (2019) Semeval-2019 0.672 0.690
M. Alzhrani (2022) Presidential 0.91 0.90
Da Silva and Paraboni (2023) Semeval-2019 0.78 0.77
Liu et al (2022) Semeval-2019 0.713 0.77
Garg and Sharma (2022) Reuter 0.93
Moreno et al (2019) Semeval-2019 0.725 0.691
Ahmed et al (2023a) 0.91
Alabdulkarim and Alhindi (2019) Semeval-2019 0.742 0.709
Joo and Hwang (2019) Semeval-2019 0.745 0.699
Pali´c et al (2019) Semeval-2019 0.791 0.763
Jiang et al (2019) Semeval-2019 0.822 0.809
Cruz et al (2019) Semeval-2019 0.717 0.668
Afsarmanesh et al (2019) Semeval-2019 0.565 0.686
Mutlu et al (2019) Semeval-2019 0.783 0.765
Shaprin et al (2019) Semeval-2019 0.646 0.646
Anthonio and Kloppenburg (2019) Semeval-2019 0.621 0.694
Cramerus and Scheffler (2019) Semeval-2019 0.578 0.683
Stevanoski and Gievska (2019) Semeval-2019 0.775 0.744
F¨arber et al (2019) Semeval-2019 0.602 0.706
Ning et al (2019) Semeval-2019 0.503 0.608
Saleh et al (2019) Semeval-2019 0.729 0.733
Zehe et al (2019) Semeval-2019 0.675 0.738
Lee et al (2019) Semeval-2019 0.621 0.694
Gangula et al (2019) Telugu 0.895
(Ahmed et al,2023b) 0.83
Hanawa et al (2019) Semeval-2019 0.809 0.805
Bestgen (2019) Semeval-2019 0.706 0.683
Yeh et al (2019) Semeval-2019 0.806 0.790
Continued on next page
31
Continued from previous page
Reference Dataset Accuracy F-1
Zhang et al (2019) Semeval-2019 0.683 0.638
Li and Goldwasser (2021) Semeval-2019 0.862 0.843
Srivastava et al (2019) Semeval-2019 0.820 0.821
Baly et al (2020) Own 0.72
Roy and Goldwasser (2020) Own 0.797
Table 7 This table report the descriptive results for the papers examined. The results cited are
reported in the respective papers. ”Dataset” refers to the dataset used for testing model. We report
only the measures mostly used: ”Accuracy” and ”F-1”, even though some papers, (Baly et al,
2019), used Mean Absolute Error (MAE).
32
Fig. 2 An example of Allsides.com homepage.
5 Datasets
In the previous section, we provided an overview of methodologies employed in address-
ing hyperpartisan detection. Effective models depend on top-notch data quality to function
optimally. However, constructing a high-quality, well-balanced dataset can be both time-
consuming and resource-intensive. This challenge is compounded by shifts in data policies
across social networks since the Cambridge Analytical scandal, leading to potential difficul-
ties or cost changes in obtaining data. Additionally, a trend has emerged within news sources
where access to data is restricted due to its previous utilization in training models like GPT
12. Consequently, news sources have implemented paywalls and crawler restrictions13, making
it exceedingly challenging to gather suitable information for this and similar tasks.
5.1 Datasets presentation
To support upcoming studies on identifying hyperpartisan news and related tasks, we have
created the extensive Table 8, which outlines key attributes of datasets relevant to hyperpar-
tisan news detection. This table includes datasets referenced in different papers, even if they
were originally used in related fields rather than specifically for hyperpartisan or fake news
detection. It is important to note that when subsets or improved iterations of earlier datasets
exist, we consider them as separate entities denoted by *. Additionally, datasets marked with
** signify merged collections. The column labeled ”Data” indicates the quantity of articles
gathered by the researchers.
To provide comprehensive insights into the table, we will delve into the datasets marked
with the symbols * and **. Framing Triplet Dataset is a combination of the following
datasets: SemEval-2019 task 4 along with Roy and Goldwasser (2020)’s data. Furthermore,
Roy and Goldwasser (2020) expands the SemEval-2019 task 4 dataset by incorporating
articles collected from polarized sources and then labeled through mediabiasfactcheck.com.
Regarding BIGNEWS, collected by Liu et al (2022), it has two subsets, respectively:
BIGNEWSBLN is a downsampled corpus maintaining an equal distribution of ideologies,
and BIGNEWSALIGN, which clusters news stories from opposing sources but on the same
12https://www.washingtonpost.com/technology/interactive/2023/ai-chatbot- learning/
13https://ilmanifesto.it/termini- e-condizioni
33
topic. In their research, Tran (2020) utilized a subset of All-the-news 14 . Furthermore, Pot-
thast et al (2018) worked with a subset of articles crawled from the URLs contained in The
BuzzFeed-Webis Fake News Corpus collected by Silverman et al (2016). From the cleaning of
Potthast et al (2018)’s dataset, stemmed S´anchez-Junquera et al (2021)’s dataset. The same
researchers created StereoImmigrants, namely a collection of Spanish news about immigrants,
for S´anchez-Junquera (2021).
Despite the heterogeneity of Non-Deep Learning and Deep Learning approaches, these
datasets are tailored for text-based methodologies. The majority of them are labeled and
retrieved using specialized platforms like Allsides15, Factcheck16, Politifact17 , as ground truth
for establishing the bias of an article and as source where to collect data. The homepage
of Allsides.com shown in Figure 2 hosts the partisan scores given to each article and the
presence of same-topic news headlines collected from politically opposed sources.
Since Baly et al (2020) noted that training models with big datasets reduce the perfor-
mance due to their noise, researchers started to prefer the quality rather than the dimension.
Indeed, Lyu et al (2023), after a deeper analysis of the SemEval 2019 dataset, revealed several
issues with this ground truth dataset widely used: class imbalance, task-label unalignment,
and distribution shift.
14https://www.kaggle.com/datasets/snapcrack/all-the- news
15https://allsides.com/
16https://mediabiasfactcheck.com/
17https://www.politifact.com/
34
Dataset Ref. Year Size Bias Label Lang. Avail.
Task 3A 18 Azizov
et al
(2023)
2023 55,000 AllSides English Yes
Task 3B 19 Azizov
et al
(2023)
2023 8,000 AllSides English Yes
Allsides-L20 Ko et al
(2023)
2023 719,256 Allsides English Yes
No Name Lyu et al
(2023)
2023 1,824,824 AllSides,
Media Bias
Factcheck
English No
Framing
Triplet
Dataset 21
Kim and
Johnson
(2022)
2022 25,627 Media Bias
Factcheck
English Yes
No Name Sridharan
and S.
(2022)
2022 ¿10k Allsides English No
TVP Info Szwoch
et al
(2022)
2022 81,694 NONE Polish Upon
request
TVN 24 Szwoch
et al
(2022)
2022 128,527 NONE Polish Upon
request
BIGNEWS Liu et al
(2022)
2022 3,689,229 Allsides,
adfontesme-
dia
English Upon
request
*BIGNEWS
BLN
Liu et al
(2022)
2022 2,331,552 Allsides,
adfontesme-
dia
English Upon
request
*BIGNEWS
ALIGN
Liu et al
(2022)
2022 1,060,512 Allsides,
adfontesme-
dia
English Upon
request
GERMAN
dataset 22 Aksenov
et al
(2021)
2021 47,362 None German Yes
Continued on next page
35
Continued from previous page
Dataset Ref. Year Size Bias Label Lang. Avail.
*No Name S´anchez-
Junquera
et al
(2021)
2021 1,555 BuzzFeed English No
Stereo
Immigrants23 S´anchez-
Junquera
(2021)
2021 3,704 Manual Spanish Yes
*No Name 24 Roy and
Gold-
wasser
(2020)
2020 21,645 Media Bias
Factcheck,
English Yes
The Anno-
tated Data
Dataset
Lim et al
(2020)
2020 46 Media Bias
Factcheck
English No
No Name Pierri
et al
(2020)
2020 ca
37000
None Italian No
*PoliNews Tran
(2020)
2020 ca
83,000
None English No
Presidential Alzhrani
(2020)
2020 178,572 Allsides,
Media Bias
Factcheck
English No
POLUSA 25 Gebhard
and Ham-
borg
(2020)
2020 ca
0.9M
None English Yes
*Politifact University
Politehnica
of
Bucharest
et al
(2021)
2020 18,027 Politifact
.com
English No
Continued on next page
36
Continued from previous page
Dataset Ref. Year Size Bias Label Lang. Avail.
No Name Sharma
et al
(2020)
2020 4,627 Manual English No
No Name 26 Baly et al
(2020)
2020 34,737 Allsides English Yes
All-Sides Li and
Gold-
wasser
(2019)
2019 10,385 None English No
Telugu27 Gangula
et al
(2019)
2019 1,327 Manual Telugu Yes
NELA-2018 28 Norregaard
et al
(2019)
2019 713,534 8-cross
combina-
tion
English Yes
by-article 29 Kiesel
et al
(2019)
2019 1273 manually
labeled
English Yes
by-publisher
30 Kiesel
et al
(2019)
2019 754,000 BuzzFeed
news,
Media Bias
Factcheck
English Yes
BASIL31 Fan et al
(2019)
2019 300 Manual English
The BuzzFeed-
Webis Fake
News Corpus
2016 32
Potthast
et al
(2018)
2018 1,627 BuzzFeed English Yes
Reuter33 - - 18,519 English Yes
No Name jeong
Lim et al
(2018)
2018 88 Crowd-
sourcing
English No
MBCF 34 Baly et al
(2018)
2018 1,066 Media Bias
Factcheck
English Yes
NELA-201735 Horne
et al
(2018)
2018 136K Unlabeled English Yes
Continued on next page
37
Continued from previous page
Dataset Ref. Year Size Bias Label Lang. Avail.
BuzzFeed
201636 buzz feed
news
.com
2016 2,282 BuzzFeed English Yes
No Name Baumer
et al
(2015)
2015 74 crowd-
sourcing
English No
Table 8 This table includes all the Datasets found in the selected papers. ”Dataset”: Name of the
dataset (only if provided by the authors); ”Ref.”: authors who introduced the paper presenting the
dataset in the literature; ”Year”: when it is introduced; ”Size”: number of the articles contained in
the dataset; ”Bias Label”: indicates the website used as benchmark for annotating data; ”Lang.”:
we report the representative language of the dataset; ”Avail.”: indicates if the dataset was released.
In the affirmative case the link is provided in the footnotes.
38
Fig. 3 Datasets’ language distribution. The figure was made with Excel.
5.2 Potential limitations
The studies conducted until now hold significant value, yet several inherent limitations
in the datasets collected could influence the comprehensiveness and applicability of future
findings. Firstly, the absence of a distinct dataset designed to differentiate between hyperpar-
tisan and partisan news poses a fundamental challenge, potentially impacting the accuracy
of your classification. Secondly, Figure 3 shows that the predominant focus is on English
news articles within the dataset. This fact raises concerns about minority languages and
their respective democratic contexts, possibly skewing the representation and applicability
of Anglo-American papers’ conclusions to their different socio-cultural environment. This
discrepancy might lead to situations where certain democracies lack the necessary tools
and datasets in their native language, hindering their ability to develop similarly effective
analytical tools as over-represented democracies. Additionally, the phenomenon of hyper-
partisanship varies significantly between countries due to the variety of party systems and
different cultural backgrounds (Dalton,2015). Consequently, the development of models
trained on linguistically non-representative data may compromise their ability to efficiently
detect hyperpartisanship in under-represented democracies, thereby impacting their success
rates. Furthermore, issues pertaining to dataset maintenance, such as broken URLs, may
impede replicability and accessibility for future research endeavors (Potthast et al,2018).
Furthermore, temporal lexicon constraints might hinder capturing shifts in textual patterns,
tones, and context, affecting the accuracy of temporal analysis (Lyu et al,2023). We high-
light that cross-lingual comparison of hyperpartisan traits has never been studied from a
computational approach. Thus, it is not possible to define if the online environment flattens
cultural-linguistic traits pertinent to hyperpartisanship independently from the country and
its political system. Another consideration regards the limited availability of data over time
due to paywalls and copyright restrictions poses a significant barrier, potentially restrict-
ing the depth and breadth of future analysis within certain timeframes. Lastly, despite the
popularity and the good results that researchers achieved, as far as we know, autoregressive
models were not used.
39
6 Conclusions and future works
In synthesizing insights from 80 studies, our systematic review illuminates the profound value
of existing research in understanding hyperpartisan news. However, critical limitations intrin-
sic to the amassed datasets cast shadows over the breadth and relevance of forthcoming
findings. Foremost among these constraints is the absence of a dedicated dataset specifically
designed to delineate between hyperpartisan and partisan news, posing a foundational chal-
lenge that could compromise classification accuracy. Moreover, the predominant emphasis
on English news articles within these datasets raises concerns regarding the representation
of minority languages and their unique democratic contexts, potentially skewing conclusions
drawn from Anglo-American sources when applied to diverse socio-cultural environments.
This discrepancy risks leaving certain democracies bereft of the necessary linguistic tools and
datasets, hindering the development of effective analytical models. One limitation of our def-
inition of hyperpartisan could be that hyperpartisan is a subject-shifting definition. Indeed,
considering subjects within an echo chamber, from their point of view, hyperpartisan news
will be one that presents content with moral values distant from theirs and where the con-
nections between subject-action active/passive-object express aversions or attacks against
their own ideology, in order to feel like a victim of the great Other (Zizek,1999), or, in more
technical terms, the Lacan’s virtual symbolic order, belonging to the opposite group. Addi-
tionally, the multifaceted nature of hyperpartisanship across countries, influenced by distinct
party systems and cultural backgrounds, underscores the limitations of models trained on
linguistically biased datasets. Consequently, the efficacy of these models in detecting hyper-
partisanship in underrepresented democracies might be compromised, affecting their success
rates. Issues pertaining to dataset maintenance, such as broken URLs, could impede replica-
bility and accessibility for future research endeavors. Moreover, temporal lexicon constraints
might hinder capturing shifts in textual patterns, tones, and context, thereby impacting the
accuracy of temporal analysis within this field. Importantly, our review sheds light on the
unexplored realm of cross-lingual comparison of hyperpartisan traits from a computational
perspective. This dearth of exploration hampers our ability to ascertain if the online envi-
ronment homogenizes cultural-linguistic traits relevant to hyperpartisanship across diverse
countries and political systems. Limited datasets in languages other than English pose a crit-
ical hurdle, particularly impacting minorities and underrepresented cultures. This scarcity
inhibits the development of robust tools like hyperpartisan news detection for diverse linguis-
tic realms. The absence of varied data impedes fair representation and hampers the accuracy
of AI models, exacerbating challenges for these communities to combat misinformation and
cultural bias. Thus, there is the urgent need for more inclusive and diverse datasets. More-
over, Aksenov et al (2021) demonstrated the efficiency of using a more fine-grained label
set. Unfortunately, labels are often binary and do not consider neither the fact that also the
Center leaning could be hyperpartisan, nor the complexity of the phenomenon that involves
different biases, textual and cultural data. On this trail, another limitation concerns the
non-existence of any mathematical formulas to describe the amount of hyperpartisanship
that occurs in a text. Furthermore, the restricted availability of data over time due to pay-
walls and copyright restrictions poses a significant barrier, potentially limiting the depth and
breadth of future analyses within specific timeframes. Lastly, despite the successes achieved
by researchers, the lack of utilization of autoregressive models, to the best of our knowledge,
remains a notable absence in this domain. In conclusion, while extant research holds substan-
tial value in unraveling the complexities of hyperpartisan news, these identified limitations
underscore the necessity for more robust, diverse, and linguistically comprehensive datasets,
along with a broader approach that encompasses diverse cultural and linguistic landscapes.
40
Additionally, exploring novel computational approaches and models, such as autoregressive
models, presents promising avenues for future investigation within this dynamic field.
7 Declarations
7.1 Funding and Competing Interests
This project has received funding from the European Union´s Horizon Europe research and
innovation programme under the Marie Sk lodowska-Curie Grant Agreement No. 101073351.
Views and opinions expressed are however those of the author(s) only and do not necessarily
reflect those of the European Union or European Research Executive Agency (REA). Neither
the European Union nor the granting authority can be held responsible for them. The authors
have no relevant financial or non-financial interests to disclose.
7.2 Author contributions
Pablo Gamallo Otero and Ga¨el Dias as scientific reviewer were assessed the appropriateness
and scientific relevance of the methodological approach used. Data collection and analysis
were performed by Michele Joshua Maggini and Davide Bassi. Michele Joshua Maggini wrote
the first draft of the article and compiled the tables. Paloma Piot edited the text file, tables
and images. Pablo Gamallo Otero reviewed and evaluated the article. All authors read and
approved the final manuscript.
7.3 Data availability
Research data are available online at: https://github.com/mik28j/
Hyperpartisan-detection-Systematic- review. corresponding author.
=======================================================
References
(2021) The IMPED model: Detecting low-quality information in social media. https:
//doi.org/https://doi.org/10.1177/0002764221989776
Afsarmanesh N, Karlgren J, Sumbler P, et al (2019) Team harry friberg at SemEval-
2019 task 4: Identifying hyperpartisan news through editorially defined metatopics.
In: Proceedings of the 13th International Workshop on Semantic Evaluation. Asso-
ciation for Computational Linguistics, pp 1004–1006, https://doi.org/https://doi.
org/10.18653/v1/S19-2174
Agerri R (2019) Doris martin at SemEval-2019 task 4: Hyperpartisan news detection
with generic semi-supervised features. In: Proceedings of the 13th International
Workshop on Semantic Evaluation. Association for Computational Linguistics, pp
944–948, https://doi.org/https://doi.org/10.18653/v1/S19-2161
Ahmed U, Lin JCW, Srivastava G (2023a) Semisupervised federated learning for
temporal news hyperpatism detection. IEEE Transactions on Computational
Social Systems 10(4):1758–1769. https://doi.org/https://doi.org/10.1109/TCSS.
2023.3247602
41
Ahmed U, Lin JCW, Srivastava G (2023b) Temporal positional lexicon expan-
sion for federated learning based on hyperpatism detection. Expert Sys-
tems 40(5):e13183. https://doi.org/https://doi.org/10.1111/exsy.13183, eprint:
https://onlinelibrary.wiley.com/doi/pdf/10.1111/exsy.13183
Aksenov D, Bourgonje P, Zaczynska K, et al (2021) Fine-grained classification of politi-
cal bias in german news: A data set and initial experiments. In: Mostafazadeh Davani
A, Kiela D, Lambert M, et al (eds) Proceedings of the 5th Workshop on Online
Abuse and Harms (WOAH 2021). Association for Computational Linguistics, pp
121–131, https://doi.org/https://doi.org/10.18653/v1/2021.woah-1.13
Alabdulkarim A, Alhindi T (2019) Spider-jerusalem at SemEval-2019 task 4: Hyper-
partisan news detection. In: Proceedings of the 13th International Workshop on
Semantic Evaluation. Association for Computational Linguistics, pp 985–989, https:
//doi.org/https://doi.org/10.18653/v1/S19-2170
Altay S, Berriche M, Heuer H, et al (2023) A survey of expert views on misinformation:
Definitions, determinants, solutions, and future of the field. Harvard Kennedy School
Misinformation Review 4(4):1–34
Alzhrani K (2020) Ideology detection of personalized political news coverage: A new
dataset. In: Proceedings of the 2020 the 4th International Conference on Com-
pute and Data Analysis. ACM, pp 10–15, https://doi.org/https://doi.org/10.1145/
3388142.3388149
Amason E, Palanker J, Shen MC, et al (2019) Harvey mudd college at SemEval-2019
task 4: The d.x. beaumont hyperpartisan news detector. In: Proceedings of the 13th
International Workshop on Semantic Evaluation. Association for Computational
Linguistics, pp 967–970, https://doi.org/https://doi.org/10.18653/v1/S19-2166
Anand B, Di Tella R, Galetovic A (2007) Information or opinion? media
bias as product differentiation. Journal of Economics & Management Strategy
16(3):635–682. https://doi.org/https://doi.org/10.1111/j.1530-9134.2007.00153.x,
https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1530-9134.2007.00153.x
Anthonio T (2019) Robust document representations for hyperpartisan and fake news
detection
Anthonio T, Kloppenburg L (2019) Team kermit-the-frog at SemEval-2019 task 4:
Bias detection through sentiment analysis and simple linguistic features. In: Pro-
ceedings of the 13th International Workshop on Semantic Evaluation. Association
for Computational Linguistics, pp 1016–1020, https://doi.org/https://doi.org/10.
18653/v1/S19-2177
Azizov D, Nakov P, Liang S (2023) Frank at checkthat! 2023: Detecting the political
bias of news articles and news media. In: Conference and Labs of the Evaluation
Forum
42
Baly R, Karadzhov G, Alexandrov D, et al (2018) Predicting factuality of reporting
and bias of news media sources. In: Riloff E, Chiang D, Hockenmaier J, et al (eds)
Proceedings of the 2018 Conference on Empirical Methods in Natural Language
Processing. Association for Computational Linguistics, Brussels, Belgium, pp 3528–
3539, https://doi.org/https://doi.org/10.18653/v1/D18-1389
Baly R, Karadzhov G, Saleh A, et al (2019) Multi-task ordinal regression for jointly
predicting the trustworthiness and the leading political ideology of news media. In:
Burstein J, Doran C, Solorio T (eds) Proceedings of the 2019 Conference of the
North American Chapter of the Association for Computational Linguistics: Human
Language Technologies, Volume 1 (Long and Short Papers). Association for Com-
putational Linguistics, pp 2109–2116, https://doi.org/https://doi.org/10.18653/v1/
N19-1216
Baly R, Da San Martino G, Glass J, et al (2020) We can detect your bias: Predicting
the political ideology of news articles. In: Webber B, Cohn T, He Y, et al (eds)
Proceedings of the 2020 Conference on Empirical Methods in Natural Language
Processing (EMNLP). Association for Computational Linguistics, pp 4982–4991,
https://doi.org/https://doi.org/10.18653/v1/2020.emnlp-main.404
Barnidge M, Peacock C (2019) A third wave of selective exposure research? the chal-
lenges posed by hyperpartisan news on social media. Media and Communication
7(3):4–7. https://doi.org/https://doi.org/10.17645/mac.v7i3.2257
Bartels LM (2018) Partisanship in the trump era. The Journal of Politics 80:1483 –
1494. https://doi.org/https://doi.org/10.1086/727602
Baumer E, Elovic E, Qin Y, et al (2015) Testing and comparing computational
approaches for identifying the language of framing in political news. In: Mihal-
cea R, Chai J, Sarkar A (eds) Proceedings of the 2015 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Lan-
guage Technologies. Association for Computational Linguistics, Denver, Colorado,
pp 1472–1482, https://doi.org/10.3115/v1/N15-1171
Bawden D, Robinson L (2018) Curating the infosphere: Luciano floridi’s philosophy of
information as the foundation for library and information science. J Documentation
74:2–17. https://doi.org/https://doi.org/10.1108/JD-07-2017-0096
Bestgen Y (2019) Tintin at SemEval-2019 task 4: Detecting hyperpartisan news arti-
cle with only simple tokens. In: Proceedings of the 13th International Workshop
on Semantic Evaluation. Association for Computational Linguistics, pp 1062–1066,
https://doi.org/https://doi.org/10.18653/v1/S19-2186
Biolchini J, Mian PG, Natali ACC, et al (2005) Systematic review in software
engineering. Tech. rep., COPPE/UFRO
43
Bourgonje P, Moreno Schneider J, Rehm G (2017) From clickbait to fake news detec-
tion: An approach based on detecting the stance of headlines to articles. In: Popescu
O, Strapparava C (eds) Proceedings of the 2017 EMNLP Workshop: Natural Lan-
guage Processing meets Journalism. Association for Computational Linguistics, pp
84–89, https://doi.org/https://doi.org/10.18653/v1/W17-4215
Breiman L (2001) Random forests. Machine Learning 45(1):5–32. https://doi.org/http:
//dx.doi.org/10.1023/A:1010933404324
Cer D, Yang Y, Kong S, et al (2018) Universal sentence encoder. CoRR
abs/1803.11175. https://doi.org/https://doi.org/10.48550/arXiv.1803.11175,
1803.11175
Chakravartula N, Indurthi V, Syed B (2019) Fermi at SemEval-2019 task 4: The sarah-
jane-smith hyperpartisan news detector. In: Proceedings of the 13th International
Workshop on Semantic Evaluation. Association for Computational Linguistics, pp
954–956, URL https://aclanthology.org/S19-2163
Chen C, Park C, Dwyer J, et al (2019) Harvey mudd college at SemEval-2019
task 4: The carl kolchak hyperpartisan news detector. In: Proceedings of the 13th
International Workshop on Semantic Evaluation. Association for Computational
Linguistics, pp 957–961, https://doi.org/https://doi.org/10.18653/v1/S19-2164
Chen T, Guestrin C (2016) Xgboost: A scalable tree boosting system. pp 785–794,
https://doi.org/https://doi.org/10.1145/2939672.2939785
Commission E (2018) A multi-dimensional approach to disinformation – Report of the
independent High level Group on fake news and online disinformation. Publications
Office, https://doi.org/doi/10.2759/739290
Cramerus R, Scheffler T (2019) Team kit kittredge at SemEval-2019 task 4: LSTM
voting system. In: Proceedings of the 13th International Workshop on Semantic
Evaluation. Association for Computational Linguistics, pp 1021–1025, https://doi.
org/https://doi.org/10.18653/v1/S19-2178
Cruz A, Rocha G, Sousa-Silva R, et al (2019) Team fernando-pessa at SemEval-2019
task 4: Back to basics in hyperpartisan news detection. In: Proceedings of the 13th
International Workshop on Semantic Evaluation. Association for Computational
Linguistics, pp 999–1003, https://doi.org/https://doi.org/10.18653/v1/S19-2173
Cruz AF, Rocha G, Cardoso HL (2020) On document representations for detection
of biased news articles. In: Proceedings of the 35th Annual ACM Symposium on
Applied Computing. ACM, pp 892–899, https://doi.org/10.1145/3341105.3374025
Da Silva SC, Paraboni I (2023) Politically-oriented information inference from text.
JUCS - Journal of Universal Computer Science 29(6):569–594. https://doi.org/
https://doi.org/10.3897/jucs.96652
44
Dalton RJ (2015) Party identification and nonpartisanship. In: International Ency-
clopedia of the Social & Behavioral Sciences. Elsevier, p 562–566, https://doi.org/
https://doi.org/10.1016/B978-0-08-097086-8.93088-5
Dalton RJ (2021) Modeling ideological polarization in democratic party systems. Elec-
toral Studies 72:102346. https://doi.org/https://doi.org/10.1016/j.electstud.2021.
102346
Devlin J, Chang M, Lee K, et al (2018) BERT: pre-training of deep bidirectional
transformers for language understanding. CoRR abs/1810.04805. https://doi.org/
https://doi.org/10.48550/arXiv.1810.04805,1810.04805
Dorogush AV, Gulin A, Gusev G, et al (2017) Fighting biases with dynamic boost-
ing. CoRR abs/1706.09516. https://doi.org/https://doi.org/10.48550/arXiv.1706.
09516,1706.09516
Drissi M, Sandoval Segura P, Ojha V, et al (2019) Harvey mudd college at SemEval-
2019 task 4: The clint buchanan hyperpartisan news detector. In: Proceedings of
the 13th International Workshop on Semantic Evaluation. Association for Com-
putational Linguistics, pp 962–966, https://doi.org/https://doi.org/10.18653/v1/
S19-2165
Dumitru VC, Rebedea T (2019) Fake and hyper-partisan news identification. In:
Moldoveanu A, Dix AJ (eds) 16th International Conference on Human-Computer
Interaction, RoCHI 2019, Bucharest, Romania, October 17-18, 2019. Matrix Rom,
pp 60–67
Dykstra A (2019) Critical reading of online news commentary headlines: Stylistic
and pragmatic aspects. Topics in Linguistics 20(2):90–105. https://doi.org/https:
//doi.org/10.2478/topling-2019-0011
European Parliament, Council (2020) Proposal for a regulation of the european par-
liament and of the council on a single market for digital services (digital services
act) and amending directive 2000/31/ec
Falkenbach M, Bekker M, Greer S (2019) Do parties make a difference? a review of
partisan effects on health and the welfare state. European journal of public health
30. https://doi.org/https://doi.org/10.1093/eurpub/ckz133
Fan L, White M, Sharma E, et al (2019) In plain sight: Media bias through the lens
of factual reporting. In: Inui K, Jiang J, Ng V, et al (eds) Proceedings of the 2019
Conference on Empirical Methods in Natural Language Processing and the 9th
International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Association for Computational Linguistics, Hong Kong, China, pp 6343–6349, https:
//doi.org/https://doi.org/10.18653/v1/D19-1664
45
F¨arber M, Qurdina A, Ahmedi L (2019) Team peter brinkmann at SemEval-2019
task 4: Detecting biased news articles using convolutional neural networks. In: Pro-
ceedings of the 13th International Workshop on Semantic Evaluation. Association
for Computational Linguistics, pp 1032–1036, https://doi.org/https://doi.org/10.
18653/v1/S19-2180
Gangula RRR, Duggenpudi SR, Mamidi R (2019) Detecting political bias in news
articles using headline attention. In: Proceedings of the 2019 ACL Workshop Black-
boxNLP: Analyzing and Interpreting Neural Networks for NLP. Association for
Computational Linguistics, pp 77–84, https://doi.org/https://doi.org/10.18653/v1/
W19-4809
Garg S, Sharma DK (2022) Role of ELMo embedding in detecting fake news on social
media. In: 2022 11th International Conference on System Modeling & Advancement
in Research Trends (SMART). IEEE, pp 57–60, https://doi.org/https://doi.org/10.
1109/SMART55829.2022.10046789
Gebhard L, Hamborg F (2020) The POLUSA dataset: 0.9m political news articles
balanced by time and outlet popularity. Proceedings of the ACM/IEEE Joint Con-
ference on Digital Libraries in 2020 pp 467–468. https://doi.org/https://doi.org/10.
1145/3383583.3398567, conference Name: JCDL ’20: The ACM/IEEE Joint Confer-
ence on Digital Libraries in 2020 ISBN: 9781450375856 Place: Virtual Event China
Publisher: ACM
Gerald Ki Wei H, Jun Choi L (2021) Hyperpartisan news classification with ELMo
and bias feature. Journal of Information Science and Engineering 37(5). https://
doi.org/https://doi.org/10.6688/JISE.202109 37(5).0013
Guess AM, Barber´a P, Munzert S, et al (2021) The consequences of
online partisan media. Proceedings of the National Academy of Sciences
118(14):e2013464118. https://doi.org/https://doi.org/10.1073/pnas.2013464118,
https://www.pnas.org/doi/pdf/10.1073/pnas.2013464118
Gupta V, Kaur Jolly BL, Kaur R, et al (2019) Clark kent at SemEval-2019 task 4:
Stylometric insights into hyperpartisan news detection. In: Proceedings of the 13th
International Workshop on Semantic Evaluation. Association for Computational
Linguistics, pp 934–938, https://doi.org/https://doi.org/10.18653/v1/S19-2159
Gusenbauer M, Haddaway NR (2020) Which academic search systems are suitable
for systematic reviews or meta-analyses? evaluating retrieval qualities of google
scholar, pubmed, and 26 other resources. Research Synthesis Methods 11:181 – 217.
https://doi.org/10.1002/jrsm.1378
Hamborg F, Donnay K, Gipp B (2019) Automated identification of media
bias in news articles : an interdisciplinary literature review. Inter-
national Journal on Digital Libraries 20(4):391–415. https://doi.org/
https://doi.org/10.1007/s00799-018-0261-y, MAG ID: 2900571251S2ID:
46
ee409021960573baf01e82d0b663ac6df6948030
Hanawa K, Sasaki S, Ouchi H, et al (2019) The sally smedley hyperpartisan news
detector at SemEval-2019 task 4. In: Proceedings of the 13th International Workshop
on Semantic Evaluation. Association for Computational Linguistics, pp 1057–1061,
https://doi.org/https://doi.org/10.18653/v1/S19-2185
Hawdon J, Ranganathan S, Leman S, et al (2020) Social media use, political polar-
ization, and social capital: Is social media tearing the u.s. apart? In: Meiselwitz
G (ed) Social Computing and Social Media. Design, Ethics, User Behavior, and
Social Network Analysis. Springer International Publishing, pp 243–260, https:
//doi.org/https://doi.org/10.1007/978-3-030-49570-1 17
Hearst M, Dumais S, Osuna E, et al (1998) Support vector machines. IEEE Intelli-
gent Systems and their Applications 13(4):18–28. https://doi.org/https://doi.org/
10.1109/5254.708428
Hermann ES, Chomsky N (1994) Manufacturing consent. The political economy of
the mass media. Vintage, London
Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural computation
9:1735–80. https://doi.org/https://doi.org/10.1162/neco.1997.9.8.1735
Honeycutt N, Jussim L (2023) Political bias in the social sciences: A critical,
theoretical, and empirical review. Ideological and Political Bias in Psychology:
Nature, Scope, and Solutions pp 97–146. https://doi.org/https://doi.org/10.1007/
978-3-031-29148-7 5
Horne BD, Dron W, Khedr S, et al (2018) Sampling the news producers:
A large news and feature data set for the study of the complex media
landscape. CoRR abs/1803.10124. https://doi.org/https://doi.org/10.48550/arXiv.
1803.10124,1803.10124
Hrckova A, Moro R, Srba I, et al (2021) Quantitative and qualitative analysis
of linking patterns of mainstream and partisan online news media in central
europe. Online Information Review 46(5):954–973. https://doi.org/https://doi.org/
10.1108/OIR-10-2020-0441, publisher: Emerald Publishing Limited
Huang GKW, Lee JC (2019) Hyperpartisan news and articles detection using
BERT and ELMo. In: 2019 International Conference on Computer and Drone
Applications (IConDA). IEEE, pp 29–32, https://doi.org/https://doi.org/10.1109/
IConDA47345.2019.9034917
Isbister T, Johansson F (2019) Dick-preston and morbo at SemEval-2019 task 4:
Transfer learning for hyperpartisan news detection. In: Proceedings of the 13th
International Workshop on Semantic Evaluation. Association for Computational
Linguistics, pp 939–943, https://doi.org/https://doi.org/10.18653/v1/S19-2160
47
Jiang Y, Petrak J, Song X, et al (2019) Team bertha von suttner at SemEval-2019 task
4: Hyperpartisan news detection using ELMo sentence representation convolutional
network. In: Proceedings of the 13th International Workshop on Semantic Evalua-
tion. Association for Computational Linguistics, pp 840–844, https://doi.org/https:
//doi.org/10.18653/v1/S19-2146
Joo Y, Hwang I (2019) Steve martin at SemEval-2019 task 4: Ensemble learning
model for detecting hyperpartisan news. In: Proceedings of the 13th International
Workshop on Semantic Evaluation. Association for Computational Linguistics, pp
990–994, https://doi.org/https://doi.org/10.18653/v1/S19-2171
Kapantai E, Christopoulou A, Berberidis C, et al (2021) A systematic literature review
on disinformation: Toward a unified taxonomical framework. New Media & Soci-
ety 23(5):1301–1326. https://doi.org/https://doi.org/10.1177/1461444820959296,
https://doi.org/10.1177/1461444820959296
Kiesel J, Mestre M, Shukla R, et al (2019) SemEval-2019 task 4: Hyperpartisan news
detection. In: Proceedings of the 13th International Workshop on Semantic Evalua-
tion. Association for Computational Linguistics, pp 829–839, https://doi.org/https:
//doi.org/10.18653/v1/S19-2145
Kim KM, Lee M, Won HS, et al (2023a) Multi-stage prompt tuning for political
perspective detection in low-resource settings. Applied Sciences 13(10). https://doi.
org/https://doi.org/10.3390/app13106252
Kim KM, Lee M, Won HS, et al (2023b) Multi-stage prompt tuning for political
perspective detection in low-resource settings. Applied Sciences 13(10):6252. https:
//doi.org/https://doi.org/10.3390/app13106252
Kim MY, Johnson KM (2022) CLoSE: Contrastive learning of subframe embeddings
for political bias classification of news media. In: Calzolari N, Huang CR, Kim
H, et al (eds) Proceedings of the 29th International Conference on Computational
Linguistics. International Committee on Computational Linguistics, pp 2780–2793
Kitchenham B, Charters S (2007) Guidelines for performing systematic literature
reviews in software engineering. Tech. rep.
Kitchenham BA, Budgen D, Pearl Brereton O (2011) Using mapping studies as
the basis for further research – a participant-observer case study. Information
and Software Technology 53(6):638–651. https://doi.org/https://doi.org/10.1016/j.
infsof.2010.12.011, special Section: Best papers from the APSEC
Knauth J (2019) Orwellian-times at SemEval-2019 task 4: A stylistic and content-
based classifier. In: Proceedings of the 13th International Workshop on Semantic
Evaluation. Association for Computational Linguistics, pp 976–980, https://doi.
org/https://doi.org/10.18653/v1/S19-2168
48
Ko Y, Ryu S, Han S, et al (2023) KHAN: Knowledge-aware hierarchical attention
networks for accurate political stance prediction. Proceedings of the ACM Web
Conference 2023 pp 1572–1583. https://doi.org/https://doi.org/10.1145/3543507.
3583300, conference Name: WWW ’23: The ACM Web Conference 2023 ISBN:
9781450394161 Place: Austin TX USA Publisher: ACM
Kong HK, Liu Z, Karahalios K (2018) Frames and slants in titles of visualizations on
controversial topics. In: Proceedings of the 2018 CHI Conference on Human Factors
in Computing Systems. Association for Computing Machinery, New York, NY, USA,
CHI ’18, p 1–12, https://doi.org/10.1145/3173574.3174012
Kristoffer Holt TUF, Frischlich L (2019) Key dimensions of alternative news
media. Digital Journalism 7(7):860–869. https://doi.org/https://doi.org/10.1080/
21670811.2019.1625715,https://doi.org/10.1080/21670811.2019.1625715
Kuhfeld M, Lewis K, Peltier T (2023) Reading achievement declines during
the COVID-19 pandemic: evidence from 5 million u.s. students in grades
3–8. Reading and Writing 36(2):245–261. https://doi.org/https://doi.org/10.1007/
s11145-022-10345-8
Kulkarni V, Ye J, Skiena S, et al (2018) Multi-view models for political ideology detec-
tion of news articles. In: Proceedings of the 2018 Conference on Empirical Methods
in Natural Language Processing. Association for Computational Linguistics, pp
3518–3527, https://doi.org/https://doi.org/10.18653/v1/D18-1388
Le QV, Mikolov T (2014) Distributed representations of sentences and documents.
CoRR abs/1405.4053. https://doi.org/https://doi.org/10.48550/arXiv.1405.4053,
1405.4053
Lee N, Liu Z, Fung P (2019) Team yeon-zi at SemEval-2019 task 4: Hyperparti-
san news detection by de-noising weakly-labeled data. In: Proceedings of the 13th
International Workshop on Semantic Evaluation. Association for Computational
Linguistics, pp 1052–1056, https://doi.org/https://doi.org/10.18653/v1/S19-2184
Leeson PT, Coyne CJ (2011) Manipulating the media. SSRN
Li C, Goldwasser D (2019) Encoding social information with graph convolutional
networks forPolitical perspective detection in news media. In: Korhonen A, Traum
D, M`arquez L (eds) Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics. Association for Computational Linguistics, Florence,
Italy, pp 2594–2604, https://doi.org/https://doi.org/10.18653/v1/P19-1247
Li C, Goldwasser D (2021) Using social and linguistic information to adapt pretrained
representations for political perspective identification. In: Zong C, Xia F, Li W,
et al (eds) Findings of the Association for Computational Linguistics: ACL-IJCNLP
2021. Association for Computational Linguistics, pp 4569–4579, https://doi.org/
https://doi.org/10.18653/v1/2021.findings-acl.401
49
jeong Lim S, Jatowt A, Yoshikawa M (2018) Understanding characteristics of biased
sentences in news articles. In: CIKM Workshops
Lim S, Jatowt A, Yoshikawa M (2020) Creating a dataset for fine-grained bias detection
in news articles
Liu P, Shivaram K, Culotta A, et al (2021) The interaction between political typol-
ogy and filter bubbles in news recommendation algorithms. In: Proceedings of the
Web Conference 2021. Association for Computing Machinery, New York, NY, USA,
WWW ’21, p 3791–3801, https://doi.org/https://doi.org/10.1145/3442381.3450113
Liu Y, Ott M, Goyal N, et al (2019) Roberta: A robustly optimized BERT pretraining
approach. CoRR abs/1907.11692. https://doi.org/https://doi.org/10.48550/arXiv.
1907.11692,1907.11692
Liu Y, Zhang XF, Wegsman D, et al (2022) POLITICS: Pretraining with same-
story article comparison for ideology prediction and stance detection. In: Carpuat
M, de Marneffe MC, Meza Ruiz IV (eds) Findings of the Association for Compu-
tational Linguistics: NAACL 2022. Association for Computational Linguistics, pp
1354–1374, https://doi.org/https://doi.org/10.18653/v1/2022.findings-naacl.101
Lorenz-Spreen P, Oswald L, Lewandowsky S, et al (2022) A systematic review
of worldwide causal and correlational evidence on digital media and democracy.
Nature Human Behaviour 7(1):74–101. https://doi.org/https://doi.org/10.1038/
s41562-022-01460-1
Lyu H, Pan J, Wang Z, et al (2023) Computational assessment of hyperparti-
sanship in news titles. https://doi.org/https://doi.org/10.48550/arXiv.2301.06270,
2301.06270[cs]
M. Alzhrani K (2022) Political ideology detection of news articles using deep neural
networks. Intelligent Automation & Soft Computing 33(1):483–500. https://doi.org/
https://doi.org/10.32604/iasc.2022.023914
McCoy J, Somer M (2019) Toward a theory of pernicious polarization
and how it harms democracies: Comparative evidence and possible reme-
dies. The ANNALS of the American Academy of Political and Social Sci-
ence 681(1):234–271. https://doi.org/https://doi.org/10.1177/0002716218818782,
https://doi.org/10.1177/0002716218818782
Medeiros FD, Braga RB (2020) Fake news detection in social media: A systematic
review. In: XVI Brazilian Symposium on Information Systems. ACM, pp 1–8, https:
//doi.org/https://doi.org/10.1145/3411564.3411648
Merow C, Smith M, Silander J (2013) A practical guide to maxent for modeling species’
distributions: What it does, and why inputs and settings matter. Ecography 36.
https://doi.org/https://doi.org/10.1111/j.1600-0587.2013.07872.x
50
Moher D, Altman DG, Liberati A, et al (2011) Prisma statement. Epidemiology
22(1):128. https://doi.org/10.1097/EDE.0b013e3181fe7825, cited by: 466; All Open
Access, Bronze Open Access
Moreno JG, Pitarch Y, Pinel-Sauvagnat K, et al (2019) Rouletabille at SemEval-2019
task 4: Neural network baseline for identification of hyperpartisan publishers. In:
Proceedings of the 13th International Workshop on Semantic Evaluation. Associ-
ation for Computational Linguistics, pp 981–984, https://doi.org/https://doi.org/
10.18653/v1/S19-2169
Morini V, Pollacci L, Rossetti G (2021) Toward a standard approach for echo chamber
detection: Reddit case study. Applied Sciences https://doi.org/https://doi.org/10.
3390/app11125390
Mour˜ao RR, Robertson CT (2019) Fake news as discursive integration: An anal-
ysis of sites that publish false, misleading, hyperpartisan and sensational infor-
mation. Journalism Studies 20(14):2077–2095. https://doi.org/https://doi.org/10.
1080/1461670X.2019.1566871,https://doi.org/10.1080/1461670X.2019.1566871
Mutlu O, Can OA, Dayanik E (2019) Team howard beale at SemEval-2019 task 4:
Hyperpartisan news detection with BERT. In: Proceedings of the 13th International
Workshop on Semantic Evaluation. Association for Computational Linguistics, pp
1007–1011, https://doi.org/https://doi.org/10.18653/v1/S19-2175
Naredla NR, Adedoyin FF (2022) Detection of hyperpartisan news articles using nat-
ural language processing technique. International Journal of Information Manage-
ment Data Insights 2(1):100064. https://doi.org/https://doi.org/10.1016/j.jjimei.
2022.100064
Nguyen DV, Dang T, Nguyen N (2019) NLP@UIT at SemEval-2019 task 4: The
paparazzo hyperpartisan news detector. In: Proceedings of the 13th International
Workshop on Semantic Evaluation. Association for Computational Linguistics, pp
971–975, https://doi.org/https://doi.org/10.18653/v1/S19-2167
Ning Z, Lin Y, Zhong R (2019) Team peter-parker at SemEval-2019 task 4: BERT-
based method in hyperpartisan news detection. In: Proceedings of the 13th
International Workshop on Semantic Evaluation. Association for Computational
Linguistics, pp 1037–1040, https://doi.org/https://doi.org/10.18653/v1/S19-2181
Norregaard J, Horne BD, Adali S (2019) NELA-GT-2018: A large multi-labelled news
dataset for the study of misinformation in news articles. CoRR abs/1904.01546.
https://doi.org/https://doi.org/10.48550/arXiv.1904.01546,1904.01546
Pali´c N, Vladika J, ˇ
Cubeli´c D, et al (2019) TakeLab at SemEval-2019 task 4:
Hyperpartisan news detection. In: Proceedings of the 13th International Workshop
on Semantic Evaluation. Association for Computational Linguistics, pp 995–998,
https://doi.org/10.18653/v1/S19-2172
51
Papadopoulou O, Kordopatis-Zilos G, Zampoglou M, et al (2019) Brenda starr at
SemEval-2019 task 4: Hyperpartisan news detection. In: Proceedings of the 13th
International Workshop on Semantic Evaluation. Association for Computational
Linguistics, pp 924–928, https://doi.org/https://doi.org/10.18653/v1/S19-2157
Patankar A, Bose J, Khanna H (2019) A bias aware news recommendation system. In:
2019 IEEE 13th International Conference on Semantic Computing (ICSC). IEEE,
pp 232–238, https://doi.org/https://doi.org/10.1109/ICOSC.2019.8665610
Pennington J, Socher R, Manning C (2014) GloVe: Global vectors for word repre-
sentation. In: Moschitti A, Pang B, Daelemans W (eds) Proceedings of the 2014
Conference on Empirical Methods in Natural Language Processing (EMNLP). Asso-
ciation for Computational Linguistics, Doha, Qatar, pp 1532–1543, https://doi.org/
10.3115/v1/D14-1162
Pescetelli N, Barkoczi D, Cebrian M (2022) Bots influence opinion dynamics
without direct human-bot interaction: the mediating role of recommender sys-
tems. Applied Network Science 7(1):46. https://doi.org/https://doi.org/10.1007/
s41109-022-00488-6
Peters ME, Neumann M, Iyyer M, et al (2018) Deep contextualized word representa-
tions. CoRR abs/1802.05365. https://doi.org/https://doi.org/10.48550/arXiv.1802.
05365, URL http://arxiv.org/abs/1802.05365,1802.05365
Pierri F, Artoni A, Ceri S (2020) HoaxItaly: a collection of italian disinformation and
fact-checking stories shared on twitter in 2019. https://doi.org/https://doi.org/10.
48550/arXiv.2001.10926,2001.10926[cs]
Pitoura E, Tsaparas P, Flouris G, et al (2017) On measuring bias in online information.
https://doi.org/https://doi.org/10.48550/arXiv.1704.05730,1704.05730[cs]
Potthast M, Kiesel J, Reinartz K, et al (2018) A stylometric inquiry into hyperpartisan
and fake news. In: Gurevych I, Miyao Y (eds) Proceedings of the 56th Annual
Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Association for Computational Linguistics, pp 231–240, https://doi.org/https://doi.
org/10.18653/v1/P18-1022
P´erez-Almendros C, Espinosa-Anke L, Schockaert S (2019) Cardiff university at
SemEval-2019 task 4: Linguistic features for hyperpartisan news detection. In:
Proceedings of the 13th International Workshop on Semantic Evaluation. Associ-
ation for Computational Linguistics, pp 929–933, https://doi.org/https://doi.org/
10.18653/v1/S19-2158
Rodrigo-Gin´es FJ, Carrillo-de Albornoz J, Plaza L (2024) A systematic review on
media bias detection: What is media bias, how it is expressed, and how to detect
it. Expert Systems with Applications 237:121641. https://doi.org/https://doi.org/
10.1016/j.eswa.2023.121641
52
Ross RM, Rand DG, Pennycook G (2021) Beyond “fake news”: Analytic think-
ing and the detection of false and hyperpartisan news headlines. Judgment
and Decision Making 16(2):484–504. https://doi.org/http://dx.doi.org/10.1017/
S1930297500008640, publisher: Cambridge University Press
Ross Arguedas A, Robertson C, Fletcher R, et al (2022) Echo chambers, filter bubbles,
and polarisation: a literature review. Tech. rep.
Roy S, Goldwasser D (2020) Weakly supervised learning of nuanced frames for ana-
lyzing polarization in news media. In: Webber B, Cohn T, He Y, et al (eds)
Proceedings of the 2020 Conference on Empirical Methods in Natural Language
Processing (EMNLP). Association for Computational Linguistics, pp 7698–7716,
https://doi.org/https://doi.org/10.18653/v1/2020.emnlp-main.620
Ruan Q, Mac Namee B, Dong R (2021) Bias bubbles: Using semi-supervised learning
to measure how many biased news articles are around us. In: AICS, pp 153–164
Rumelhart DE, Hinton GE, Williams RJ (1986) Learning internal representations by
error propagation. URL https://api.semanticscholar.org/CorpusID:62245742
Saleh A, Baly R, Barr´on-Cede˜no A, et al (2019) Team QCRI-MIT at SemEval-2019
task 4: Propaganda analysis meets hyperpartisan news detection. In: Proceedings
of the 13th International Workshop on Semantic Evaluation. Association for Com-
putational Linguistics, pp 1041–1046, https://doi.org/https://doi.org/10.18653/v1/
S19-2182
Sengupta S, Pedersen T (2019) Duluth at SemEval-2019 task 4: The pioquinto
manterola hyperpartisan news detector. In: Proceedings of the 13th International
Workshop on Semantic Evaluation. Association for Computational Linguistics, pp
949–953, https://doi.org/https://doi.org/10.18653/v1/S19-2162
Shaprin D, Da San Martino G, Barr´on-Cede˜no A, et al (2019) Team jack ryder
at SemEval-2019 task 4: Using BERT representations for detecting hyperpartisan
news. In: Proceedings of the 13th International Workshop on Semantic Evaluation.
Association for Computational Linguistics, pp 1012–1015, https://doi.org/https:
//doi.org/10.18653/v1/S19-2176
Sharma A, Kaur N, Sen A, et al (2020) Ideology detection in the indian mass
media. In: 2020 IEEE/ACM International Conference on Advances in Social Net-
works Analysis and Mining (ASONAM). IEEE, pp 627–634, https://doi.org/https:
//doi.org/10.1109/ASONAM49781.2020.9381344
Silverman C, Strapagiel L, Shaban H, et al (2016) Hyperpartisan facebook pages are
publishing false and misleading information at an alarming rate. BuzzFeed News
Sm˘adu RA, Echim SV, Cercel DC, et al (2023) From fake to hyperpartisan
news detection using domain adaptation. https://doi.org/https://doi.org/10.48550/
53
arXiv.2308.02185,2308.02185[cs]
Sousa-Silva R (2022) Fighting the fake: A forensic linguistic analysis to fake
news detection. International Journal for the Semiotics of Law - Revue inter-
nationale de S´emiotique juridique 35. https://doi.org/https://doi.org/10.1007/
s11196-022-09901-w
Spezzano F, Shrestha A, Fails JA, et al (2021) That’s fake news! reliability of news
when provided title, image, source bias & full article. Proceedings of the ACM
on Human-Computer Interaction 5:1–19. https://doi.org/https://doi.org/10.1145/
3449183
Sridharan A, S. N (2022) An automated news bias classifier using caenorhabditis
elegans inspired recursive feedback network architecture. CoRR abs/2207.12724.
https://doi.org/https://doi.org/10.48550/ARXIV.2207.12724,2207.12724
Srivastava V, Gupta A, Prakash D, et al (2019) Vernon-fenwick at SemEval-2019 task
4: Hyperpartisan news detection using lexical and semantic features. In: Proceedings
of the 13th International Workshop on Semantic Evaluation. Association for Com-
putational Linguistics, pp 1078–1082, https://doi.org/https://doi.org/10.18653/v1/
S19-2189
Stevanoski B, Gievska S (2019) Team ned leeds at SemEval-2019 task 4: Explor-
ing language indicators of hyperpartisan reporting. In: Proceedings of the 13th
International Workshop on Semantic Evaluation. Association for Computational
Linguistics, pp 1026–1031, https://doi.org/https://doi.org/10.18653/v1/S19-2179
Szwoch J, Staszkow M, Rzepka R, et al (2022) Creation of polish online news corpus
for political polarization studies. In: Afli H, Alam M, Bouamor H, et al (eds) Pro-
ceedings of the LREC 2022 workshop on Natural Language Processing for Political
Sciences. European Language Resources Association, pp 86–90
S´anchez-Junquera J (2021) On the detection of political and social bias
S´anchez-Junquera J, Rosso P, Montes M, et al (2021) Masking and transformer-based
models for hyperpartisanship detection in news. pp 1244–1251, https://doi.org/10.
26615/978-954-452-072-4 140
Tran M (2020) How biased are american media outlets? a framework for presentation
bias regression. In: 2020 IEEE International Conference on Big Data (Big Data).
IEEE, pp 4359–4364, https://doi.org/https://doi.org/10.1109/BigData50022.2020.
9377987
Tucker J, Guess A, Barbera P, et al (2018) Social media, political polarization,
and political disinformation: A review of the scientific literature. SSRN Electronic
Journal https://doi.org/https://doi.org/10.2139/ssrn.3144139
54
University Politehnica of Bucharest, Dumitru VC, Rebedea T, et al (2021) Topic-based
models with fact checking for fake news identification. In: RoCHI - International
Conference on Human-Computer Interaction. MATRIX ROM, pp 182–190, https:
//doi.org/https://doi.org/10.37789/rochi.2021.1.1.28
Walecki R, Rudovic O, Pavlovic V, et al (2016) Copula ordinal regression for joint
estimation of facial action unit intensity. In: Proceedings of the IEEE Conference
on computer vision and pattern recognition, pp 4902–4910, https://doi.org/https:
//doi.org/10.1109/CVPR.2016.530
Walton D (1998) Ad Hominem Arguments. University Alabama Press
Weeks B (2015) Emotions, partisanship, and misperceptions: How anger and anxiety
moderate the effect of partisan bias on susceptibility to political misinformation.
Journal of Communication 65. https://doi.org/https://doi.org/10.1111/jcom.12164
Wu S, Resnick P (2021) Cross-partisan discussions on youtube: Conservatives talk
to liberals but liberals don’t talk to conservatives. CoRR abs/2104.05365. https:
//doi.org/https://doi.org/10.48550/arXiv.2104.05365,2104.05365
Xu WW, Sang Y, Kim C (2020) What drives hyper-partisan news sharing: Exploring
the role of source, style, and content. Digital Journalism 8(4):486–505. https://doi.
org/https://doi.org/10.1080/21670811.2020.1761264
Yarchi M, Baden C, Kligler-Vilenchik N (2020) Political polarization on the digi-
tal sphere: A cross-platform, over-time analysis of interactional, positional, and
affective polarization on social media. Political Communication 38:1–42. https:
//doi.org/https://doi.org/10.1080/10584609.2020.1785067
Yeh CL, Loni B, Schuth A (2019) Tom jumbo-grumbo at SemEval-2019 task 4: Hyper-
partisan news detection with GloVe vectors and SVM. In: Proceedings of the 13th
International Workshop on Semantic Evaluation. Association for Computational
Linguistics, pp 1067–1071, https://doi.org/https://doi.org/10.18653/v1/S19-2187
Zannettou S, Sirivianos M, Blackburn J, et al (2019) The web of false information:
Rumors, fake news, hoaxes, clickbait, and various other shenanigans. J Data and
Information Quality 11(3). https://doi.org/https://doi.org/10.1145/3309699
Zehe A, Hettinger L, Ernst S, et al (2019) Team xenophilius lovegood at SemEval-
2019 task 4: Hyperpartisanship classification using convolutional neural networks.
In: Proceedings of the 13th International Workshop on Semantic Evaluation. Asso-
ciation for Computational Linguistics, pp 1047–1051, https://doi.org/https://doi.
org/10.18653/v1/S19-2183
Zhang C, Rajendran A, Abdul-Mageed M (2019) UBC-NLP at SemEval-2019 task
4: Hyperpartisan news detection with attention-based bi-LSTMs. In: Proceedings
55
