ArticlePDF Available

The spread of COVID-19 conspiracy theories on social media and the effect of content moderation

Authors:

Abstract and Figures

We investigate the diffusion of conspiracy theories related to the origin of COVID-19 on social media. By analyzing third-party content on four social media platforms, we show that: (a) In contrast to conventional wisdom, mainstream sources contribute overall more to conspiracy theories diffusion than alternative and other sources; and (b) platforms’ content moderation practices are able to mitigate the spread of conspiracy theories. Nevertheless, we locate issues regarding the timeliness and magnitude of content moderation, as well as that platforms filter significantly fewer conspiracy theories coming from mainstream sources. Given this, we discuss policy steps that can contribute to the containment of conspiracy theories by media sources, platform owners, and users.
Content may be subject to copyright.
The Harvard Kennedy School Misinformation Review1
August 2020, Volume 1, Special Issue on COVID-19 and Misinformation
Attribution 4.0 International (CC BY 4.0)
Reprints and permissions: misinforeview@hks.harvard.edu
DOI: https://doi.org/10.37016/mr-2020-034
Website: misinforeview.hks.harvard.edu
Research Article
!
The spread of COVID-19 conspiracy theories on social media
and the effect of content moderation
We investigate the diffusion of conspiracy theories related to the origin of COVID-19 on social media. By
analyzing third-party content on four social media platforms, we show that: (a) In contrast to conventional
wisdom, mainstream sources contribute overall more to conspiracy theories diffusion than alternative and
other sources; and (b) Platforms’ content moderation practices are able to mitigate the spread of conspir-
acy theories. Nevertheless, we locate issues regarding the timeliness and magnitude of content modera-
tion, as well as that platforms filter significantly fewer conspiracy theories coming from mainstream
sources. Given this, we discuss policy steps that can contribute to the containment of conspiracy theories
by media sources, platform owners, and users.
Authors: Orestis Papakyriakopoulos (1), Juan Carlos Medina Serrano (1), Simon Hegelich (1)
Affiliations: (1) Political Data Science, Technical University of Munich, Germany
How to cite: Papakyriakopoulos, O.; Medina Serrano, J. C.; Hegelich, S. (2020). The spread of COVID-19 conspiracy theories on
social media and the effect of content moderation. The Harvard Kennedy School (HKS) Misinformation Review, Volume 1, Spe-
cial Issue on COVID-19 and Misinformation. https://doi.org/10.37016/mr-2020-033
Received: June 6th, 2020. Accepted: July 25th, 2020. Published: August 17th, 2020.
Research questions
What are the dynamics of conspiracy theories related to the origin of COVID-19 on social media?
What is the role of mainstream and alternative sources in the spread of conspiracy theories?
What is the impact of social media platforms’ content moderation policies on the diffusion of conspir-
acy theories?
Essay summary
We identified 11,023 unique URLs referring to the origin of COVID-19 appearing in 267,084 Facebook,
Twitter, Reddit, and 4chan posts between January and March 2020. We classified them based on their
source (mainstream, alternative, other) and their content (supporting conspiracy theories, used as
evidence for conspiracy theories, neither). We considered URLs in the first two content categories as
stories reinforcing conspiracy theories. We investigated whether posts containing these stories were
removed or labeled as such by the platforms. Then, we employed appropriate statistical techniques
to quantify conspiracy theory diffusion between social media platforms and measured the impact of
content moderation.
1
A publication of the Shorenstein Center for Media, Politics, and Public Policy, at Harvard University, John F. Ken-
nedy School of Government.
The spread of COVID-19 conspiracy theories on social media and the effect of content moderation 2
We found that alternative sources generated more stories reinforcing conspiracy theories than main-
stream sources. However, similar stories coming from mainstream sources reached significantly more
users. We further quantified conspiracy theory dynamics in the social media ecosystem. We found
that stories reinforcing conspiracy theories had a higher virality than neutral or debunking stories.
We measured the amount of moderated content on Reddit, Twitter, and Facebook. We concluded
that content moderation on each platform had a significant mitigating effect on the diffusion of con-
spiracy theories. Nevertheless, we found that a large number of conspiracy theories remained un-
moderated. We also detected a moderation bias towards stories coming from alternative and other
sources (with other sources comprising personal blogs and social media submissions, e.g. tweets, Fa-
cebook posts, Reddit comments, etc.).
Results suggest that policymakers and platform owners should reflect on further ways that can con-
tain COVID-19-related conspiracy theories. Content moderation is an effective strategy but can be
further improved by overcoming issues of timeliness and magnitude. There should also be additional
transparency on how and why content moderation takes place, as well as targeted design interven-
tions, which can inform and sensitize users regarding conspiracy theories.
Argument & Implications
The COVID-19 health crisis resulted in the burst of an unprecedented misinfodemic on social media: A
vast amount of pandemic related misinformation appeared, which in turn influenced society's response
to the virus (Gyenes et al., 2018). Given the absence of exact social and scientific knowledge about the
origin, nature, and impact of the coronavirus, many conspiracy theories quickly emerged, seeking to pro-
vide explanations. To confront the overwhelming amount of misinformation, social media platforms and
fact-checking agencies increased attempts to moderate such content by removing or flagging it, often
relying on algorithmic decision making (ADM) systems (Brennen et al., 2020; Newton, 2020).
In this study, we aim to understand how conspiracy theories spread at the beginning of the COVID-19
health crisis, and based on this, uncover possibilities and issues of fact-checking in the social media eco-
system (Marwick & Lewis, 2017). To achieve this, we measured the appearance of stories reinforcing con-
spiracy theories on four platforms: Facebook, Twitter, Reddit, and the subsection of 4chan called “politi-
cally incorrect” or “/pol/”, which is a prominent forum of conspiracy theorists. In contrast to other cases
(Cosentino, 2020), 4chan was not the only source of conspiracy theories in the ecosystem (finding 1). We
found that stories reinforcing conspiracy theories became more viral than stories either debunking them
or having a neutral stance (finding 1). This complies with previous findings on misinformation (Vosoughi
et al., 2018; Vicario et al., 2016).
Most of the stories reinforcing conspiracy theories originated from alternative sources, personal
blogs, and social media posts (83%). However, such content coming from mainstream sources (17%) re-
sulted in higher numbers of Facebook and Twitter shares (60% and 55% of the total respectively). Main-
stream sources included high-credibility news outlets, such as the New York Post or Fox News, scientific
websites such as biorxiv.org, and other widely credible sites, such as Wikipedia. Alternative sources in-
cluded untrustworthy and low-credibility outlets, such as Infowars and Breitbart. Although alternative and
other sources were the main carriers of conspiracy theories, mainstream sources had a higher impact on
the spread of conspiracy theories (finding 2).
We investigated the platforms’ moderation practices, which varied to a certain degree. Twitter and
YouTube removed stories that supported conspiracy theories (Gadde & Derella, 2020; Binder, 2020), while
Reddit and Facebook either removed or flagged them (Reddit content policy, 2020; Jin, 2020). On Reddit,
removing or flagging depended on the rules of each sub-community, whereas on Facebook on whether
the company reviewed the stories themselves (removed) or relied on third-party fact-checkers (flagged).
Papakyriakopoulos; Medina Serrano; Hegelich 3
We concluded that the platforms’ moderation practices strongly reduced the probability of stories
reappearing in the total ecosystem. Hence, theoretically, instantly removing or filtering conspiracy theo-
ries would contain their spread. However, content moderation is a complex and time-consuming process,
with human workers and ADM systems facing obstacles in accuracy and efficiency (Roberts, 2019; Graves,
2018; Gillespie, 2018; Serrano et al., 2020). This lies both in the large amount of content to be fact-
checked, but also in the nature of the content, which is often difficult to categorize as conspiracy theory
or not (Krafft et al. 2020; Uscinski et al. 2013; Byfold, 2011; Dentith, 2014; Krause et al., 2020). In our
study, we found that the platforms managed to fact-check only between 15% to 50% of posts containing
stories reinforcing conspiracy theories, with moderation in many cases taking place weeks after they be-
came viral (finding 3,4).
We observed that each platform faced different obstacles in content moderation. For example, con-
tent moderation on Twitter was less effective than on the other platforms (finding 4). We can probably
explain this effect by the timeliness of content removal, as misinformation on Twitter spreads significantly
in the first hours after its first appearance (Vosoughi et al., 2018). YouTube also faced issues of timeliness.
For instance, a video that stated that the pandemic is a planned conspiracy gathered up to 5 million views
in a period of only two days (Wong, 2020), with copies of the video continuously being re-uploaded after
its removal. Facebook filtered the least amount of stories reinforcing conspiracy theories, while Reddit
appeared to not moderate older content. These results illustrate the challenges that platforms and poli-
cymakers should overcome. Besides issues of timeliness and moderation magnitude, platforms should
investigate if removing or flagging content is an optimal practice, not only for containing misinformation
but also for maintaining a politically inclusive environment. Since Facebook and Reddit have mixed mod-
eration policies, it would be important to quantify different effects between misinformation control and
user engagement.
A further implication of the study is related to the existence of a moderation bias on all platforms,
with stories reinforcing conspiracy theories and coming from mainstream sources being filtered signifi-
cantly less. This is an important finding, given that mainstream sources prevailed as a key factor for con-
spiracy spread in our study, and that many ADM systems for classifying contents take a source’s credibility
level as input (Atananosova et al., 2019). Therefore, platform owners should pay more attention to what
they moderate and why, and clearly explain their decisions to the users. Studies show that additional
transparency and deliberation in content removal make users more aware of the type of information they
are consuming, change the way they interact with it, and build trust between them and the services (Fazio,
2020; Ruzenberg, 2019; Suzor et al., 2018; Ruzenberg, 2019; Krause et al. 2020). Finally, mainstream
sources should be aware that the information they produce in the process of reportage could be exploited
for the support and general reinforcement of conspiracy theories.
We hope that these recommendations can guide platforms and policymakers towards solutions that
can accompany traditional content moderation, which we found to be an effective technique for contain-
ing the spread of conspiracy theories.
Findings
This study investigates content moderation practices about conspiracy theories related to the origin of
COVID-19. It includes four important findings regarding conspiracy theory dynamics on social media, as
well as the possibilities and issues of fact-checking for mitigating the spread of conspiracy theories.
Finding 1: URLs reinforcing conspiracy theories went more viral than URLs being neutral or debunking con-
spiracy theories. In both cases, URL dissemination followed complex paths in the social media ecosystem.
The spread of COVID-19 conspiracy theories on social media and the effect of content moderation 4
For the three months under investigation, we quantified the spread of conspiracy theory related URLs on
social media (RQ1). Results suggest that paths and intensity varied depending on the type of URL (Figure
1). Neutral or debunking stories primarily spread in the ecosystem after being presented on Twitter, while
a significant amount of URLs disseminated on other platforms through 4chan. On the other hand, stories
reinforcing conspiracy theories followed different routes. URLs present on 4chan spread on Twitter, and
stories on Twitter were further distributed on Facebook. Reddit had an impact on stories reinforcing con-
spiracy theories both on Facebook and 4chan, while Facebook was feeding 4chan with both URL types.
Overall, conspiracy theory diffusion models showed that stories reinforcing conspiracy theories became
more viral within the ecosystem than the rest. This complies with previous research studies stating that
misinformation and provocative content is disseminated more than factual content on social networks
(Vosoughi et al., 2018; Vicario et al., 2016). Furthermore, these findings show that information paths be-
tween social media are complex and content dependent, and reject the statement that fringe social media
are the only contributors to conspiracy theory dissemination (Cosentino, 2020). In contrast, we found
that all platforms contributed to the spread of stories reinforcing conspiracy theories.
Figure 1. Heatmap depicting the virality of conspiracy theory reinforcing and neutral/debunking stories about the origin of
COVID-19. The darker the color, the higher the virality of URLs, as calculated by virality parameter α. The heatmap describes
how each platform in a column affected other platforms. For example, the first column shows how stories on Facebook affected
the virality of similar stories on itself, Twitter, Reddit, and 4chan.
Finding 2: Mainstream sources played a bigger role in conspiracy theory dissemination than alternative
and other sources.
We classified our sample of stories reinforcing conspiracy theories based on their source and quantified
their popularity using Twitter and Facebook shares (RQ2). The 83% of the conspiracy theory reinforcing
URLs originated from alternative or other sources, and only 17% came from mainstream sources. How-
ever, stories coming from mainstream sources were on average and overall more popular (Figure 2). On
average, mainstream URLs supporting conspiracy theories were shared four times more on Facebook and
Twitter in comparison to URLs coming from alternative sources. Similarly, mainstream URLs used as evi-
dence for the truthfulness of conspiracy theories were shared two times more. Overall, 17% of stories
reinforcing conspiracy theories coming from mainstream sources resulted in 60% and 55% of the total
Facebook and Twitter shares, respectively. These results are explainable since users usually read and share
sources they trust (Brennen et al., 2020; Epstein et al., 2020), and mainstream sources have a higher reach
Papakyriakopoulos; Medina Serrano; Hegelich 5
and acceptance in society. In Table 1, we provide an exemplary set of URLs, their content type, and the
number of Facebook and Twitter shares they evoked. For a more detailed analysis, refer to the appendix.
Figure 2. Bar plots illustrating median Facebook and Twitter shares for URLs supporting conspiracy theories or used as evi-
dence by source type. Mainstream sources included scientific articles, patent repositories, Wikipedia, government websites,
high credibility and widely acceptable media outlets. Alternative sources included media outlets defined as low credibility. Other
sources included social media submissions from Facebook, YouTube, Twitter, and Reddit, or personal websites and blogs.
Table 1. Exemplary URLs from the dataset. The table includes the title of the URL, its source, the assigned source
label, content label, and the number of Facebook and Twitter shares.
Title
Source
Source type
Content type
Twitter
shares
Don’t buy China’s story: The coronavirus
may have leaked from a lab
New York Post
mainstream
supporting conspiracy
theory
25,000
Uncanny similarity of unique inserts in
the 2019-nCoV spike protein to HIV-1
gp120 and Gag
bioRxiv
mainstream
supporting conspiracy
theory
41,000
Coronavirus Contains "HIV Insertions",
Stoking Fears Over Artificially Created
Bioweapon
Zerohedge
alternative
supporting conspiracy
theory
15,000
Coronavirus isolated from humans
Google patents
mainstream
evidence for conspiracy
theory
0
Finding 3: Moderating content either by removing or flagging it significantly reduced the spread of con-
spiracy theories in the ecosystem.
Information diffusion models quantified the impact of content moderation on the virality of conspiracy
theories (RQ3). The models yielded for each case a value α <= 1, which denotes the probability that a
submission containing a URL will lead to the creation of another submission containing the same URL.
Table 2 illustrates the mean difference of that probability when comparing models trained on URLs that
were either moderated or not. Results suggest that content moderation significantly decreased the prob-
ability that a story reinforcing conspiracy theories will reappear on the same platform, but also that it will
diffuse on another platform. For Facebook and Reddit, this probability reduction exceeded 90%. By
The spread of COVID-19 conspiracy theories on social media and the effect of content moderation 6
contrast, moderating content on Twitter had a smaller in-platform effect, which did not exceed 10%. A
potential explanation for this is the nature of retweeting on the platform, with users spreading copies of
a message in short periods after its initial submission. Thus, information can get viral before moderation
mechanisms can trace it and remove it. Nonetheless, models provided evidence that content moderation
practices indeed can reduce the spread of conspiracy theories in the social media ecosystem.
Table 2. Change in the probability that a submission containing a URL will lead to the creation of another submis-
sion containing the same URL between unmoderated and moderated content. The change is given for submissions
on the same platform, between platforms, and in the total ecosystem.
Virality reduction (%) after content moderation on:
Same platform
Other plat-
forms
Overall ecosystem
Facebook
-0.96
-0.99
-0.96
Twitter
-0.10
-0.93
-0.61
Reddit
-0.97
-0.98
-0.94
Finding 4: Content moderation results revealed issues regarding the extent and nature of content removal.
Despite the finding that content moderation practices reduced the spread of conspiracy theories, our
study also detected open issues when investigating RQ3. First, the biggest part of stories reinforcing con-
spiracy theories on the platforms remained unmoderated (between 50-85% depending on the platform)
as shown on Figure 4. Especially for Reddit and Facebook, we found that if stories were not removed close
to their initial submissions, the probability of them being removed later was very low. In contrast, YouTube
and Twitter kept on filtering content later in time, although many of the stories had already reached peak
virality. Second, we calculated the ratios of removed stories for each source type and located a source
bias. On all three platforms, submissions with URLs coming from mainstream sources were removed or
flagged significantly less by content moderators. This bias in content removal was translated into a relative
percentage of 10 to 30 percent, depending on the platform.
Papakyriakopoulos; Medina Serrano; Hegelich 7
Figure 4. Left: Proportion of moderated stories reinforcing conspiracy theories as investigated in April and May 2020. Right:
Relative percentage of moderated stories reinforcing conspiracy theories by source (mainstream, alternative, other).
Methods
We collected social media submissions from Reddit, Facebook, Twitter, and 4chan related to COVID-19
between January 1 and April 1, 2020. We extracted 9.5 million Reddit submissions and comments, 4.2
million Facebook posts, and 83 million tweets matching the query “COVID-19 OR coronavirus.” For this,
we used the Pushshift Reddit API (Baumgartner, 2018), Crowdtangle’s historical data (Silverman, 2019),
and the COVID-19 Twitter dataset developed by Chen et al. (2020). For 4chan, we crawled the total “Co-
rona” thread and its sub-threads in 4chan’s “politically incorrect” board and collected 1.5 million posts.
From the complete dataset, we selected only the submissions that referred to the origin of COVID-19 by
using the query “biowarfare OR biological weapon OR bioweapon OR umbrella corp OR man-made OR
human origin OR man-made OR biosafety.” We selected this query after reading multiple submissions and
locating conspiracy theories that were reoccurring. As a final preprocessing step, we obtained all URLs
from these submissions and created a list of 11,023 unique URLs.
We visited each of the 11,023 URLs and manually coded the stories depending on their relation to
conspiracy theories. To develop a coding scheme, we adopted a definition of conspiracies and conspiracy
theories based on prior theoretical work. According to Keely (1999), a conspiracy is a secret plot by two
or more powerful actors. Conspiracy theories are efforts to explain the ultimate causes of significant so-
ciopolitical events, such as the origin of COVID-19, by claiming the existence of a secret plot, by challenging
institutionalized explanations (Byford, J., 2011) and many times by denying science (Douglas et al., 2019).
As Byford (2011) states, conspiracy theories follow a three-point explanatory logic:
(a) There is a conspiracy as the main narrative of a story. For COVID-19, stories argued that a set of
powerful individuals or groups, be that governments, institutions, or wealthy actors developed the virus
for their specific interests.
(b) Conspiracy theories generally ground their validity either on indirect evidence or on the absence of
evidence. For COVID-19, many stories claimed that there is a conspiracy because there exist patents on
engineering coronaviruses and even a book mentioning a virus originating from Wuhan. Similarly, some
stories argued that the virus should be man-made because specific research publications could not con-
clude on the exact animal that carried the virus.
(c) Conspiracy theories are structured in a way that stories become irrefutable, and hence hard to chal-
lenge (Pelkmans et al., 2011; Sunstein et al., 2009). For example, the statement “A book talked about a
virus originating from Wuhan 40 years ago. Therefore, COVID-19 is man-made” is causally oversimplified
and thus impossible to provide counterevidence to reject it.
By using this framework, we defined three labels for classifying URLs:
[1] Supporting conspiracy theories. In this case, URLs supported a conspiracy theory. The authors believed
that some actors conspired to create COVID-19 and justified their thesis in the existence or absence of
specific evidence.
[2] Evidence used to support a conspiracy theory. This class included URLs that did not directly link to a
conspiracy theory, but social media users cited them as evidence for the conspiracy theories. For example,
The spread of COVID-19 conspiracy theories on social media and the effect of content moderation 8
users linked to older articles about bioweapons to prove that specific countries created COVID-19. We
considered this category also as reinforcing conspiracy theories because social media submissions con-
taining these URLs were moderated by social media platforms. Furthermore, users grounded conspiracy
theories on them in the way mentioned in (b).
[3] Neither. URLs with stories that did not refer to any type of conspiracy, that debunked conspiracy the-
ories, mentioned conspiracy theories without believing them, or cited third parties that did believe in
them.
We further labeled the URLs according to their source type. We defined three classes:
[i] Mainstream sources. These included scientific articles, patent repositories, Wikipedia, government
websites, high credibility and widely acceptable media outlets. We used the list generated by Shao et al.
(2016) and fact-checking websites (e.g. adfontesmedia.com, newsguardtech.com, allsides.com) to iden-
tify credible media outlets.
[ii] Alternative sources. These included media outlets defined as low credibility by Shao et al. (2016), or
ranked as untrustworthy by previously mentioned fact-checking websites.
[iii] Other sources. These included social media submissions from Facebook, YouTube, Twitter, and Red-
dit, or personal websites and blogs.
To validate coding, two additional reviewers labeled a subsample of 300 URLs. The Krippendorf alpha was
0.92, while the pairwise Cohen’s kappa were in all cases equal or greater than 0.9. These values suggest
that there was high intercoder reliability in the labeled dataset. The subsample of 300 URLs and their
corresponding reviewers’ labels are available at the data repository of the study (see data availability).
For further examples of our coding scheme, please refer to Table 5 in the appendix. After labeling, 4,724
URLs were supporting conspiracy theories (1) or were used as evidence of conspiracy theories (2). We
searched for these URLs in the original dataset and identified 267,084 submissions that contained them.
We modeled URL cross-platform diffusion by using a mathematical technique known as Hawkes pro-
cess. Hawkes process is a model that quantifies how specific events influence each other over time in an
ecosystem containing multiple components. In our case, the components are the social media platforms,
and an event is the appearance of a post containing a specific URL on any platform. The Hawkes process
can quantify how likely it is that the posting of a URL on a platform will cause the same URL to be posted
again on any platform in the information ecosystem (Zannetou et al., 2017). This is given by a parameter
αij, which gives the expected number of times a URL will appear on platform j if it was only posted on
platform i, and functions as a proxy for a URL’s virality (Rizoiu et al., 2017). We calculated parameters αij
to study the flow of conspiracy related content across the four social media platforms, as illustrated in
finding 1. For more information refer to the appendix.
To investigate the role of mainstream, alternative, and other sources in conspiracy theory dissemina-
tion, we used Buzzsumo to obtain the total number of shares each URL evoked on Facebook and Twitter.
Buzzsumo provided metrics for 1,850 URLs in our sample (finding 2). We then studied whether submis-
sions containing conspiracy theory reinforcing URLs have been removed or flagged by the platforms. We
crawled Facebook, Twitter, YouTube, and Reddit once at the beginning of April 2020 and once at the be-
ginning of May 2020 to understand content moderation (finding 4). With this new information, we ran
Hawkes processes separately on moderated and unmoderated content. Finally, we compared virality pa-
rameters α for each case to quantify how much content moderation influenced conspiracy diffusion in the
ecosystem (finding 3).
Papakyriakopoulos; Medina Serrano; Hegelich 9
Bibliography
Atanasova, P., Nakov, P., Màrquez, L., Barrón-Cedeño, A., Karadzhov, G., Mihaylova, T., ... & Glass, J.
(2019). Automatic fact-checking using context and discourse information. Journal of Data and
Information Quality (JDIQ), 11(3), 1-27. https://doi.org/10.1145/3297722
Baumgartner, J. M. (2018). Pushshift API.
Binder, M. (2020). YouTube tells creators to expect more video removals during coronavirus pandemic.
Mashable. https://mashable.com/article/YouTube-coronavirus-content-moderation
Brennen, J. S., Simon, F. M., Howard, P. N., & Nielsen, R. K. (2020). Types, sources, and claims of COVID-
19 misinformation. Reuters Institute. https://reutersinstitute.politics.ox.ac.uk/types-sources-
and-claims-covid-19-misinformation
Byford, J. (2011). Conspiracy theories: A critical introduction. Springer.
https://doi.org/10.1057/9780230349216
Cosentino, G. (2020). From Pizzagate to the Great Replacement: The Globalization of Conspiracy Theo-
ries. In Social Media and the Post-Truth World Order (pp. 59-86). Palgrave Pivot, Cham.
https://doi.org/10.1007/978-3-030-43005-4_3
Chen, E., Lerman, K., & Ferrara, E. (2020). COVID-19: The first public coronavirus Twitter dataset. arXiv
preprint arXiv:2003.07372. https://doi.org/10.2196/19273
Del Vicario, M., Bessi, A., Zollo, F., Petroni, F., Scala, A., Caldarelli, G., Stanley, H., & Quattrociocchi, W.
(2016). The spreading of misinformation online. Proceedings of the National Academy of Sciences,
113(3), 554-559. https://doi.org/10.1073/pnas.1517441113
Dentith, M. (2014). The philosophy of conspiracy theories. Springer.
https://doi.org/10.1057/9781137363169
Douglas, K. M., Uscinski, J. E., Sutton, R. M., Cichocka, A., Nefes, T., Ang, C. S., & Deravi, F. (2019). Under-
standing conspiracy theories. Political Psychology, 40, 3-35. https://doi.org/10.1111/pops.12568
Epstein, Z., Pennycook, G., & Rand, D. (2020, April). Will the crowd game the algorithm? Using layperson
judgments to combat misinformation on social media by downranking distrusted sources. In
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1-11.
https://doi.org/10.1145/3313831.3376232
Fazio, L. (2020). Pausing to consider why a headline is true or false can help reduce the sharing of false
news. Harvard Kennedy School Misinformation Review, 1(2). https://doi.org/10.37016/mr-2020-
009
Gadde, V., & Derella, M. (2020). An update on our continuity strategy during COVID-19. blog. Twit-
ter.com. https://blog.Twitter.com/en_us/topics/company/2020/An-update-on-our-continuity-
strategy-during-COVID-19.html
Gillespie, T. (2018). Custodians of the Internet: Platforms, content moderation, and the hidden decisions
that shape social media. Yale University Press. https://doi.org/10.12987/9780300235029
Graves, D. (2018). Understanding the promise and limits of automated fact-checking. https://reuter-
sinstitute.politics.ox.ac.uk/sites/default/files/2018-02/graves_factsheet_180226%20FINAL.pdf
Gyenes, N., & Mina, A. X. (2018). How misinfodemics spread disease. The Atlantic. https://www.theat-
lantic.com/technology/archive/2018/08/how-misinfodemics-spread-disease/568921/
Jin, K. X. (2020). Keeping people safe and informed about the coronavirus. About Facebook.
https://about.fb.com/news/2020/07/coronavirus/
Keeley, B. L. (1999). Of conspiracy theories. The Journal of Philosophy, 96(3), 109-126.
https://doi.org/10.2307/2564659
The spread of COVID-19 conspiracy theories on social media and the effect of content moderation 10
Krafft, P. M., & Donovan, J. (2020). Disinformation by Design: The Use of Evidence Collages and Platform
Filtering in a Media Manipulation Campaign. Political Communication, 37(2), 194-214.
https://doi.org/10.1080/10584609.2019.1686094
Krause, N. M., Freiling, I., Beets, B., & Brossard, D. (2020). Fact-checking as risk communication: the
multi-layered risk of misinformation in times of COVID-19. Journal of Risk Research, 1-8.
https://doi.org/10.1080/13669877.2020.1756385
Marwick, A., & Lewis, R. (2017). Media manipulation and disinformation online. New York: Data & Soci-
ety Research Institute.
Myers West, S. (2018). Censored, suspended, shadowbanned: User interpretations of content modera-
tion on social media platforms. New Media & Society, 20(11), 4366-4383.
https://doi.org/10.1177/1461444818773059
Newton, C. (2020, March 18). The coronavirus is forcing tech giants to make a risky bet on AI. The Verge.
Retrieved from https://www.theverge.com/interface/2020/3/18/21183549/coronavirus-con-
tent-moderators-facebook-google-Twitter
Pelkmans, M., & Machold, R. (2011). Conspiracy theories and their truth trajectories. Focaal, 2011(59),
66-80. http://dx.doi.org/10.3167/fcl.2011.590105
Reddit Content policy. (2020). Reddit. https://www.redditinc.com/policies/content-policy
Rizoiu, M. A., Lee, Y., Mishra, S., & Xie, L. (2017). A tutorial on hawkes processes for events in social me-
dia. arXiv preprint arXiv:1708.06401.
Roberts, S. T. (2019). Behind the screen: Content moderation in the shadows of social media. Yale Univer-
sity Press. https://doi.org/10.2307/j.ctvhrcz0v
Roozenbeek, J., & Van Der Linden, S. (2019). The fake news game: actively inoculating against the risk of
misinformation. Journal of Risk Research, 22(5), 570-580.
https://doi.org/10.1080/13669877.2018.1443491
Serrano, J. C. M., Papakyriakopoulos, O., & Hegelich, S. (2020). NLP-based Feature Extraction for the De-
tection of COVID-19 Misinformation Videos on YouTube. In Proceedings of the ACL 2020 Work-
shop on Natural Language Processing for COVID-19 (NLP-COVID).
Silverman, B. (2019). CrowdTangle for academics and researchers. https://www.facebook.com/face-
bookmedia/blog/crowdtangle-for-academics-and-researchers
Sunstein, C. R., & Vermeule, A. (2009). Conspiracy theories: Causes and cures. Journal of Political Philos-
ophy, 17(2), 202-227. https://doi.org/10.1111/j.1467-9760.2008.00325.x
Suzor, N. P., West, S. M., Quodling, A., & York, J. (2019). What do we mean when we talk about transpar-
ency? Toward meaningful transparency in commercial content moderation. International Jour-
nal of Communication, 13, 18.
Shao, C., Ciampaglia, G. L., Flammini, A., & Menczer, F. (2016). Hoaxy: A platform for tracking online mis-
information. In Proceedings of the 25th International Conference Companion on World Wide
Web, 745-750. https://doi.org/10.1145/2872518.2890098
Uscinski, J. E., & Butler, R. W. (2013). The epistemology of fact checking. Critical Review, 25(2), 162-180.
https://doi.org/10.1080/08913811.2013.843872
Vosoughi, S., Roy, D., & Aral, S. (2018). The spread of true and false news online. Science, 359(6380),
1146-1151. https://doi.org/10.1126/science.aap9559
Wong, Q. (2020). Facebook, YouTube and Twitter struggle with viral Plandemic conspiracy video. Cnet.
Retrieved from https://www.cnet.com/news/facebook-YouTube-Twitter-viral-plandemic-con-
spiracy-video/
Zannettou, S., Caulfield, T., De Cristofaro, E., Kourtellis, N., Leontiadis, I., Sirivianos, M., Stringhini, G. &
Blackburn, J. (2017). The web centipede: understanding how web communities influence each
other through the lens of mainstream and alternative news sources. In Proceedings of the 2017
Internet Measurement Conference, 405-417.
Papakyriakopoulos; Medina Serrano; Hegelich 11
Funding
The project was not funded by a specific organization.
Competing interests
There are no competing interests of the authors.
Ethics
The data collection and processing complied with the EU General Data Protection Regulation (GDPR).
Copyright
This is an open access article distributed under the terms of the Creative Commons Attribution License,
which permits unrestricted use, distribution, and reproduction in any medium, provided that the original
author and source are properly credited.
Data availability
Based on the possibilities and limitations set by the GDPR and the guidelines of the Technical University
of Munich, necessary materials and code to replicate this study are available online here:
https://doi.org/10.7910/DVN/JVOUFN.
The spread of COVID-19 conspiracy theories on social media and the effect of content moderation 12
Appendix
Conspiracy theories: definition & coding
To develop a coding scheme, we adopted a definition of conspiracies and conspiracy theories based on
prior theoretical work. According to Keely (1999), a conspiracy is a secret plot by two or more powerful
actors. Conspiracy theories are efforts to explain the ultimate causes of significant sociopolitical events,
such as the origin of COVID-19, by claiming the existence of a secret plot, by challenging institutionalized
explanations (Byford, J., 2011) and many times by denying science (Douglas et al., 2019). As Byford (2011)
states, conspiracy theories follow a three-point explanatory logic:
(a) There is a conspiracy as the main narrative of a story. For COVID-19, stories argued that a set of
powerful individuals or groups, be that governments, institutions, or wealthy actors developed the virus
for their specific interests.
(b) Conspiracy theories generally ground their validity either on indirect evidence or on the absence of
evidence. For COVID-19, many stories claimed that there is a conspiracy because there exist patents on
engineering coronaviruses and even a book mentioning a virus originating from Wuhan. Similarly, some
stories argued that the virus should be man-made because specific research publications could not con-
clude on the exact animal that carried the virus. Clarke (2002) explained this type of argumentation in
conspiracy theories by the fundamental attribution error bias: the tendency of humans to overstate or
understate the relation between events and individuals to confirm personal dispositions.
(c) Conspiracy theories are structured in a way that stories become irrefutable, and hence hard to chal-
lenge (Pelkmans et al., 2011; Sunstein et al., 2009). This feature is a result of the nature of evidence used
to support the conspiracy theories as mentioned in (b). For example, the statement “A book talked about
a virus originating from Wuhan 40 years ago. Therefore, COVID-19 is man-made” is causally oversimplified
and thus impossible to provide counterevidence to reject it.
By using this framework, we defined three labels for classifying URLs (Table 3):
[1] Supporting conspiracy theories. In this case, URLs supported a conspiracy theory. The authors believed
that some actors conspired to create COVID-19 and justified their thesis in the existence or absence of
specific evidence.
[2] Evidence used to support a conspiracy theory. This class included URLs that did not directly link to a
conspiracy theory, but social media users cited them as evidence for the conspiracy theories. For example,
users linked to older articles about bioweapons to prove that specific countries created COVID-19. They
also cited a Wikipedia article about the Wuhan Biosafety lab as proof that the virus leaked from there. We
considered this category as reinforcing conspiracy theories because social media submissions containing
these URLs were moderated by social media platforms. Furthermore, users grounded conspiracy theories
on them in the way mentioned in (b).
[3] Neither. URLs with stories that did not refer to any type of conspiracy, that debunked conspiracy the-
ories, mentioned conspiracy theories without believing them, or cited third parties that did believe in
them.
Papakyriakopoulos; Medina Serrano; Hegelich 13
Table 3. Labels used to classify the relation of a URL to conspiracy theories related to the origin of
COVID-19.
URL Label
Meaning
1
Supporting conspiracy theory
2
Evidence used to support conspir-
acy theory
3
Neither
We further labeled the URLs according to their source type. We defined three classes (Table 4):
[i] Mainstream sources. These included scientific articles, patent repositories, Wikipedia, government
websites, high credibility and widely acceptable media outlets. We used the list generated by Shao et al.
(2016) and fact-checking websites (e.g. adfontesmedia.com, newsguardtech.com, allsides.com) to iden-
tify credible media outlets.
[ii] Alternative sources. These included media outlets defined as low credibility by Shao et al. (2016), or
ranked as untrustworthy by previously mentioned fact-checking websites.
[iii] Other sources. These included social media submissions from Facebook, YouTube, Twitter, and Red-
dit, or personal websites and blogs.
Table 4. Labels used to classify the source type of the collected URLs.
Source La-
bel
Meaning
i
Mainstream sources
ii
Alternative sources
iii
Other sources
Table 5 presents exemplary URLs classified under each label, the reason for this, as well as an exemplary
social media submission that contained them. It gives examples of URLs coming from a mainstream
source, from an alternative source, as well as other sources. It also gives examples of URLs classified as
supporting conspiracy theories, as evidence used to support a conspiracy theory, or neither.
The spread of COVID-19 conspiracy theories on social media and the effect of content moderation 14
Table 5. Exemplary URLs from our dataset. The table includes the assigned source label, URL label, and
the reason behind that decision. It also provides an exemplary social media submission that the URL ap-
peared in.
URL
Submission
example
Source Label
URL Label
Reason
https://www.biorxiv.org/con-
tent/10.1101/2020.01.30.927871v1
When sections of
this coronavirus di-
rectly match HIV,
hard not to con-
clude it's an engi-
neered bug and
part of the failing
NWO's plan to
wrest power from
populist move-
ments the world
over.
i
1
Research article
preprint that
was rejected by
the scientific
community sup-
porting conspir-
acy theory
https://nypost.com/2020/02/22/dont-buy-chinas-story-
the-coronavirus-may-have-leaked-from-a-lab/
The Post reported
that the virus got
out of a bio lab in
Wuhan. If so, they
may have found a
way for it to avoid
pre-pubescents.
Crazy, scary stuff.
i
1
Media article di-
rectly supports
conspiracy the-
ory
https://video.foxnews.com/v/6133941690001#sp=show-
clips
my opinion is their
is enough evidence
for there being a
cover up for the
Wuhan bio-lab sci-
entist releasing the
man-made virus
created on a joint
venture between
aus-USA-China after
a vile was smashed
during the scientists
arrest in the mar-
kets, this virus 2
was designed to
stop the protesting
in Hong kong etc.
i
1
Media bundit
supports con-
spiracy theory
https://en.m.wikipedia.org/wiki/Wuhan_Institute_of_Vi-
rology
Letting the WHO
genocide you with
their escaped lab
project would be
retarded. They've
been involved with
the BSL4 facility
from the start.
They're playing
dumb/savior and ly-
ing.
i
2
Wikipedia is
cited as evi-
dence of a con-
spiracy theory
Papakyriakopoulos; Medina Serrano; Hegelich 15
https://www.sciencedirect.com/science/arti-
cle/pii/S0166354220300528
More evidence the
virus was created in
a lab to attack Wu-
han. This would
then be a Biowar-
fare DARPA
PREEMPTIVE attack.
The US is still the
prime suspect.
i
2
Research cited
as evidence of a
conspiracy the-
ory
https://patents.google.com/patent/US7220852B1/en
If this is true, then it
could be that the
UNITED STATES
committed an act of
BIOLOGICAL WAR-
FARE against
china!!!!
i
2
Patent cited as
evidence of a
conspiracy the-
ory
https://www.wired.com/1998/11/israels-ethnic-
weapon/
Goyim, Israel has a
COVID-19 vaccine,
only 666 shekels,
but today, just for
Jew, only 660 shek-
els! Such a deal!"----
-------------------Israeli
Scientists Claim It’s
‘Pure Luck’ They
Were Already
Working On A
COVID-19 Vaccine
Prior To The Out-
break
i
2
Older media ar-
ticle used as evi-
dence of con-
spiracy theory
https://www.zerohedge.com/economics/real-umbrella-
corp-wuhan-ultra-biohazard-lab-was-studying-worlds-
most-dangerous-pathogens
Alternative source
for the conspiracy
theorist...I do find it
difficult to believe
any official narra-
tive from China!
ii
1
Media article di-
rectly supports
conspiracy the-
ory
https://www.infowars.com/coronavirus-chinese-espio-
nage-behind-wuhan-bioweapon/
Chinese Espionage
Behind #Wuhan
Bioweapon
Help get the truth
out by sharing this
censored link!
ii
1
Media article di-
rectly supports
conspiracy the-
ory
The spread of COVID-19 conspiracy theories on social media and the effect of content moderation 16
https://www.newstarget.com/2020-02-20-full-tran-
script-smoking-gun-interview-prof-frances-boyle-corona-
virus-bioweapons.html
National institute of
Health caught Red
Handed in develop-
ment and sale of
Coronavirus to
China!! https
ii
1
Media interview
supporting the
conspiracy the-
ory
https://www.opindia.com/2020/03/korean-series-kan-
nada-magazine-predictions-dean-koontz-sylvia-browne-
chinese-coronavirus-warnings/
Someone tweaked
it to increase the
mortality rate to 90
per cent
ii
2
Media article
providing evi-
dence for a con-
spiracy theory
https://leozagami.com/2020/02/10/a-1981-book-by-
dean-koontz-predicts-a-deadly-bacteriological-weapon-
called-wuhan-400/
I did NOT write
it...just seems inter-
esting. What ya'll
think? OPEN YOUR
GOD DAMN EYES
PEEPS 👀 DON'T BE
A SHEEP
iii
2
Personal blog
providing evi-
dence for a con-
spiracy theory
https://www.reddit.com/r/aznidentity/com-
ments/evxe96/novel_coronavirus_could_it_be_a_ra-
cial_bioweapon/
-
iii
1
Social media
submission sup-
porting conspir-
acy theory
https://www.businessinsider.com/coronavirus-white-su-
premacists-discussed-using-covid-19-as-bioweapon-
2020-3?r=DE&IR=T
'Absolutely sicken-
ing.
i
3
Media discuss
about conspir-
acy theories
Papakyriakopoulos; Medina Serrano; Hegelich 17
https://www.foxbusiness.com/politics/coronavirus-
covid-outbreak-tom-cotton
President Xi Jinping
big lying about
"CORONAVIRUS"
outbreak...
i
3
Media presents
Senators opinion
about conspir-
acy theories
https://www.reuters.com/article/us-health-coronavirus-
disinformation/russia-deploying-coronavirus-disinfor-
mation-to-sow-panic-in-west-eu-document-says-
idUSKBN21518F
Russians are not
friends. They’re en-
emies. Remember.
#COVID19 #russia
#TrumpPandemic
i
3
Media talks
about interna-
tional security
and conspiracy
theories
https://www.dailywire.com/news/chinese-ambassador-
does-not-deny-coronavirus-came-from-biological-war-
fare-program
Coronavirus up-
date: China sup-
pressed infor-
mation. Ambassa-
dor of China doesn’t
deny that the virus
could have come
from a Chinese bio-
weapon lab, but in-
stead insinuates it
may have come
from USA weapons
program.
ii
3
Media presents
Chinese ambas-
sadors opinion
related to con-
spiracy theory
https://edition.cnn.com/2020/03/13/us/dean-koontz-
novel-coronavirus-debunk-trnd/index.html
No, Dean Koontz
did not predict the
coronavirus in a
1981 novel\n#coro-
navirusalgerie
i
3
Media article re-
futes conspiracy
theory
https://www.theepochtimes.com/involvement-of-wu-
han-p4-lab-questioned_3230182.html
I think this story is
more realistic about
the nCoV.
ii
3
Media articles
reports on peo-
ple speculating
on conspiracy
theories
The spread of COVID-19 conspiracy theories on social media and the effect of content moderation 18
https://www.face-
book.com/watch/?v=630625861056708
-
iii
3
Social media
submission re-
futing conspir-
acy theory
Hawkes processes
We modeled the diffusion of conspiratorial and normal URLs in the social media ecosystem in order to
understand cross-platform dynamics and the effect of content moderation practices. We assumed that in
a cross-platform setting, users share contents on a platform, and other users consume it and sometimes
reshare it on the same or on another platform in the ecosystem. The total life-span of a specific content
in the ecosystem can be described by a point-process, i.e., a set of points in time, where each point de-
notes the appearance of the content. This point-process is self-exciting, meaning that the occurrence of
previous points makes the occurrence of future points more probable. For example, the appearance of a
tweet will trigger the appearance of a set of retweets in the future, which will not have happened without
the occurrence of the initial event.
In our case, the appearance of an event (a submission containing a specific URL on a specific social media
platform) can trigger the appearance of a new event on any platform in the ecosystem. A mathematical
model that can describe such a multi-dimensional self-exciting point-process is the Hawkes process. A
Hawkes process is a
𝐷
-dimensional counting process
𝑁
(
𝑡
)=(
𝑁
1(
𝑡
)
𝑁𝐷
(
𝑡
)), where each component is a
counting process:
, with D the number of social media platforms under consideration, and
𝑡𝑖
,1 ,
𝑡𝑖
,2,… being the timestamps
that an event (the appearance of a specific URL) will be observed on platform
𝑖
. The intensity N of such a
process is given by the function:
for
𝑖
=1,…,
𝐷
. Such an intensity function describes cross-platform effects induced by events created on
platform i on events on platform
𝑗
. The nature of the effects are encoded by the kernel function
𝜙𝑖𝑗
, tjk
are the timestamps for all events on platform j, and
𝜇𝑖
is a baseline intensity that gives the magnitude of
influence (how much are specific contents spread in the ecosystem in general). In our study, we use a
kernel function of exponential decay, because it is able to describe content virality (how rapidly and widely
is specific content diffused in the network given its previous appearance), and memory over time (for how
long it remains prevalent in the ecosystem). This function is described by:
Papakyriakopoulos; Medina Serrano; Hegelich 19
, where αij is the virality parameter that gives how viral content became on platform j given the appear-
ance of content on platform i, and βij is the parameter that describes for how long contents appeared on
platform j given their appearance on platform i. To calculate parameters αijij we performed maximum
likelihood estimation after splitting our data on train and test set. We fitted various Hawkes processes in
order to understand differences in virality between normal and conspiratorial URLs, as well as between
contents that were moderated or not for each platform.
Bibliography (Appendix)
Byford, J. (2011). Conspiracy theories: A critical introduction. Springer.
https://doi.org/10.1057/9780230349216
Clarke, S. (2002). Conspiracy theories and conspiracy theorizing. Philosophy of the Social Sciences, 32(2),
131-150. https://doi.org/10.1177/004931032002001
Douglas, K. M., Uscinski, J. E., Sutton, R. M., Cichocka, A., Nefes, T., Ang, C. S., & Deravi, F. (2019). Under-
standing conspiracy theories. Political Psychology, 40, 3-35. https://doi.org/10.1111/pops.12568
Keeley, B. L. (1999). Of conspiracy theories. The Journal of Philosophy, 96(3), 109-126.
https://doi.org/10.2307/2564659
Pelkmans, M., & Machold, R. (2011). Conspiracy theories and their truth trajectories. Focaal, 2011(59),
66-80. http://dx.doi.org/10.3167/fcl.2011.590105
Sunstein, C. R., & Vermeule, A. (2009). Conspiracy theories: Causes and cures. Journal of Political Philos-
ophy, 17(2), 202-227. https://doi.org/10.1111/j.1467-9760.2008.00325.x
... to COVID-19 on social media platforms further impeded efforts to mitigate its spread [3]. An investigation into the mechanisms underlying the spread of contagions across these diverse contexts such as the spread of infectious diseases and misinformation, has received broad attention under the frameworks of network-based epidemiological models, spreading processes and cascades [4]- [7]. ...
... We 2 Note that we focus on the spread of contagions where a single infectious contact can lead to infection or awareness of a piece of information similar to the independent cascade models [43], but in contrast to models such as linear threshold models [44] for influence maximization. 3 We say that a network is not clustered if the global clustering coefficient is vanishingly small in the limit of large network size. 4 The single-layer model typically refers to a single network-layer generated using the configuration model. ...
... The proof of Theorem 3.2 is linked to the stability of the fixed point solutions of (2,3). We note that for i = 1, 2, the set of equations (2, 3) admits a trivial fixed point h 1 (1) = h 2 (1) = g 1 (1) = g 2 (1) = 1. ...
Preprint
In an increasingly interconnected world, a key scientific challenge is to examine mechanisms that lead to the widespread propagation of contagions, such as misinformation and pathogens, and identify risk factors that can trigger large-scale outbreaks. Underlying both the spread of disease and misinformation epidemics is the evolution of the contagion as it propagates, leading to the emergence of different strains, e.g., through genetic mutations in pathogens and alterations in the information content. Recent studies have revealed that models that do not account for heterogeneity in transmission risks associated with different strains of the circulating contagion can lead to inaccurate predictions. However, existing results on multi-strain spreading assume that the network has a vanishingly small clustering coefficient, whereas clustering is widely known to be a fundamental property of real-world social networks. In this work, we investigate spreading processes that entail evolutionary adaptations on random graphs with tunable clustering and arbitrary degree distributions. We derive a mathematical framework to quantify the epidemic characteristics of a contagion that evolves as it spreads, with the structure of the underlying network as given via arbitrary {\em joint} degree distributions of single-edges and triangles. To the best of our knowledge, our work is the first to jointly analyze the impact of clustering and evolution on the emergence of epidemic outbreaks. We supplement our theoretical finding with numerical simulations and case studies, shedding light on the impact of clustering on contagion spread.
... Owing to diverse social media platforms, information could easily spread among them, leading to an emergent research problem [29,47,67]. Understanding how information propagates across different platforms could assist analysts in analyzing complex user behaviors [43], designing effective marketing strategies [6,19], and preventing harmful content [20,25,46,57]. Despite the existence of numerous studies [7,12,56,63,74] that visualize information propagation on social media, few have considered cross-platform propagation due to its complex and implicit mechanisms. ...
... Social media platforms offer a huge space for information diffusion. Extensive research has been undertaken on three primary objectives: modeling and explaining this process [14,20,22,25,26,28,38,41,43,46,55,56,60,61,63], predicting and simulating the diffusion process [9, 15, 17, 24, 27, 30, 32-34, 37, 44, 49, 51, 53, 68, 71-73], and detecting anomalous diffusion patterns [54,74]. Gravity model [55] and influence model [63] have been utilized to approximate the propagation process. ...
... Lu et al. [38] introduced a semi-automated method for the quantitative analysis of pandemic-related information flow from Twitter to Weibo. Additionally, other research mainly relies on the detection of common URLs or identification of cross-platform user relationships [14,20,25,43,46,61,64], to unveil specific cross-platform dissemination scenarios. However, due to the absence of a systematic summary of cross-platform diffusion patterns, existing efforts are not enough to achieve comprehensive analyses despite the precise identification of cases under some specific patterns. ...
Article
Propagation analysis refers to studying how information spreads on social media, a pivotal endeavor for understanding social sentiment and public opinions. Numerous studies contribute to visualizing information spread, but few have considered the implicit and complex diffusion patterns among multiple platforms. To bridge the gap, we summarize cross-platform diffusion patterns with experts and identify significant factors that dissect the mechanisms of cross-platform information spread. Based on that, we propose an information diffusion model that estimates the likelihood of a topic/post spreading among different social media platforms. Moreover, we propose a novel visual metaphor that encapsulates cross-platform propagation in a manner analogous to the spread of seeds across gardens. Specifically, we visualize platforms, posts, implicit cross-platform routes, and salient instances as elements of a virtual ecosystem — gardens, flowers, winds, and seeds, respectively. We further develop a visual analytic system, namely BloomWind, that enables users to quickly identify the cross-platform diffusion patterns and investigate the relevant social media posts. Ultimately, we demonstrate the usage of BloomWind through two case studies and validate its effectiveness using expert interviews.
... As social media use has grown over the past two decades, it has become a popular medium for conducting disinformation campaigns and a fertile environment for misinformation diffusion. Social media users often use multiple platforms and, in doing so, can spread misinformation across their various social networks on different platforms (Gottfried 2024;Papakyriakopoulos et al. 2020). Additionally, the actors behind disinformation campaigns have intentionally leveraged multiple platforms to conduct their operations, often capitalizing on the unique characteristics of each platform (Starbird and Wilson 2020;Lukito 2020). ...
... The spread of misinformation across multiple social media platforms has been studied in topics ranging from natural disasters (Hunt et al. 2020) to global pandemics (Papakyriakopoulos et al. 2020). Similarly, the role that multiple platforms can play in the execution of disinformation campaigns has been examined in various contexts. ...
... It showed how cross-platform connections make it harder for platforms to get rid of fake or harmful information and can obscure the actual level of such content on the platforms (Velásquez et al. 2021). However, a different study of COVID-19 conspiracy theories concluded that removing and flagging content reduced the spread of conspiracies across multiple paltforms (Papakyriakopoulos et al. 2020). Another study found that 2020 U.S. election-related tweets that had been posted by President Trump and were subsequently blocked from user engagement were posted more often and received more attention on Facebook, Instagram, and Reddit than those that were labeled or received no intervention from Twitter (Sanderson et al. 2021). ...
Article
Full-text available
Social media platforms are highly interconnected because many users maintain a presence across multiple platforms. Consequently, efforts to limit the spread of misinformation taken by individual platforms can have complex consequences on misinformation diffusion across the social media ecosystem. This is further complicated by the diverse social structures, platform standards, and moderation mechanisms provided on each platform. We study this issue by extending our previous model of Reddit interactions and community-specific moderation measures. By adding a followership-based model of Twitter interactions and facilitating cross-platform user participation, we simulate information diffusion across heterogeneous social media platforms. While incorporating platform-specific moderation mechanisms, we simulate interactions at the user level and specify user-specific attributes. This allows practitioners to conduct experiments with various types of actors and different combinations of moderation. We show how the model can simulate the impacts of such features on discussions facilitated by Reddit and Twitter and the cross-platform spread of misinformation. To validate this model, we use a combination of empirical datasets from three U.S. political events and prior findings from user surveys and studies.
... These interventions introduce obstacles (ie, friction) that make it more difficult for harmful misinformation to spread, including content moderation and platform regulation. 89,90 For example, by removing or flagging conspiratorial content on platforms such as X (formerly known as Twitter), Reddit, and Facebook, they become less likely to reappear in the information ecosystem. 90 WHO has collaborated with digital and social media companies to review and enhance their misinformation moderation policies. ...
... 89,90 For example, by removing or flagging conspiratorial content on platforms such as X (formerly known as Twitter), Reddit, and Facebook, they become less likely to reappear in the information ecosystem. 90 WHO has collaborated with digital and social media companies to review and enhance their misinformation moderation policies. 91 Furthermore, behavioural interventions such as simple reminders or nudges that encourage people to consider the accuracy of the information they encounter could help stop the spread of false information on social media. ...
Article
Full-text available
The COVID-19 pandemic has highlighted how infodemics (defined as an overabundance of information, including misinformation and disinformation) pose a threat to public health and could hinder individuals from making informed health decisions. Although public health authorities and other stakeholders have implemented measures for managing infodemics, existing frameworks for infodemic management have been primarily focused on responding to acute health emergencies rather than integrated in routine service delivery. We review the evidence and propose a framework for infodemic management that encompasses upstream strategies and provides guidance on identifying different interventions, informed by the four levels of prevention in public health: primary, secondary, tertiary, and primordial prevention. On the basis of a narrative review of 54 documents (peer-reviewed and grey literature published from 1961 to 2023), we present examples of interventions that belong to each level of prevention. Adopting this framework requires proactive prevention and response through managing information ecosystems, beyond reacting to misinformation or disinformation.
... Finding ways to address misinformation is, therefore, crucial at both the individual and societal levels. inoculation measures [5], promoting media literacy [6], and employing fact-checking [7] and content moderation practices (e.g., removing posts with misinformation, using flags, and warning labels) [8]. However, these solutions require a significant amount of time, resources, and long-term investments, such as educational programs and trained personnel. ...
Article
Full-text available
This article overviews the state of the art, research challenges, and future directions in our vision: integrating social explanation into explainable artificial intelligence (XAI) to combat misinformation. In our context, “social explanation” is an explanatory approach that reveals the social aspect of misinformation by analyzing sociocontextual cues, such as user attributes, user engagement metrics, diffusion patterns, and user comments. Our vision is motivated by the research gap in the existing XAI that tends to overlook the broader social context in which misinformation spreads. In this article, we first define social explanation, demonstrating it through examples, enabling technologies, and real-world applications. We then outline the unique benefits social explanation brings to the fight against misinformation and discuss the challenges that make our vision complex. The significance of this article lies in introducing the “social explanation” concept in XAI, which has been underexplored in the previous literature. Also, we demonstrate how social explanations can be effectively employed to tackle misinformation and promote collaboration across diverse fields by drawing upon interdisciplinary techniques spanning from computer science, social computing, human–computer interaction, to psychology. We hope that this article will advance progress in the field of XAI and contribute to the ongoing efforts to counter misinformation.
... Despite the absence of empirical evidence from the scientific community, these claims rapidly gained traction on popular social media platforms such as Facebook and YouTube, ultimately leading to the removal of 5G towers in some countries. There has been an observable increase recently in the popularity of urban legends that assert a causal relationship between vaccinations and conditions such as autism and other disorders (Papakyriakopoulos et al., 2020). The onset of the Covid-19 pandemic was accompanied by an infodemic, especially the extensive dissemination of misinformation (Bin Naeem et al., 2020). ...
Article
Purpose This study evaluates how publicly available archival documents shaped online discussions about allegations that thousands of children were kidnapped during the 1950s in Israel, known as the Yemenite children’s affair. It examines if access to historical records leads to more informed and rational public discourse, especially on social media. Design/methodology/approach Using content analysis, this study examines Facebook posts from media outlets, politicians, NGOs and public groups between 2016 and 2021 to understand how the Israeli State Archives’ release of over 300,000 documents affected support of the kidnapping. Findings Despite extensive archival information debunking the kidnapping theory, public opinion and discourse largely continued to support it. This suggests a complex interaction between information availability, preexisting beliefs, echo chambers and group allegiances, suggesting that access to factual data alone may not effectively challenge established beliefs in online public settings. Research limitations/implications Since data were collected only from Facebook, our conclusions cannot be generalized to other platforms. The study relies only on publicly accessible data and does not establish causality between exposure to information and shifts in opinion. Our findings show that disclosing archival information does not significantly benefit public political discourse on contentious topics but also point to the advantages of mediating information by politicians, NGOs and journalists. Originality/value As a unique case study, this research contributes to understanding the role of historical archives in digital-age public discourse. It highlights their potential and limitations in facilitating informed debate and deliberation, emphasizing the complexity of influencing established beliefs with factual data.
... First, platforms' content moderation strategies against misinformation and CT vary greatly from service to service. While we cannot extensively review past work here, some prior work indicates that Facebook moderated such content less than Twitter (Papakyriakopoulos et al., 2020). Moreover, by the time YouTube reported to be cracking down on pandemic-related misinformation (Duffy, 2021), many still criticized the slow pace with which Facebook was responding to the flood of misinformation during the pandemic (Jackson et al., 2021;O'Sullivan et al., 2021). ...
Article
Full-text available
Using an original survey covering 17 countries, this paper documents the prevalence of beliefs in conspiracy theories related to the COVID-19 pandemic and characterizes the informational, demographic, and trust profiles of individuals who believe them. There is considerable variation across countries in the level of conspiracy beliefs, with people in a set of countries like Romania, Poland, Greece, and Hungary being relatively more susceptible than respondents in Northern Europe. We find several factors are correlated with conspiracy beliefs across countries. Relative to respondents who do not read news on social media, social media users tend to endorse more conspiracies, and this is the case for Facebook, Instagram, and YouTube users in particular. We also observe a link between distrust in medical experts or government and endorsement of conspiracy theories in most countries. In a subset of countries, we also find individuals with medium level of education and those who are younger to believe in a higher number of conspiracy theories.
... For example, the Chinese government was quick to hush and rebuke "rumormongers" as the initial outbreak in Wuhan, China sparked public health concerns [84]. Outlandish COVID-19 conspiracies quickly developed at the start of the pandemic despite any factual evidence to support [85][86][87]. People concocted COVID-19 was a biological weapon to reduce the global population [88]. ...
Article
Full-text available
COVID-19 vaccinations have demonstrated effectiveness in reducing severe infections. However, vaccine hesitancy posed a major public health hurdle to combat the COVID-19 pandemic. Online spread of vaccine conspiracy beliefs generated unwarranted mistrust and resistance to vaccines. While numerous studies have explored the factors influencing vaccine hesitancy, there remains a lack of comprehensive understanding regarding the interplay between perceived disease vulnerability, COVID-19 fear, and vaccine hesitancy. Protection motivation theory posits citizens will evaluate perceived threats and take actions to mitigate potential harm. With a large U.S. sample, path analysis demonstrated individuals’ perceived disease vulnerability was associated with lower vaccine hesitancy. Greater perceived disease vulnerability was associated with higher COVID-19 fear. Greater COVID-19 fear was associated with lower vaccine hesitancy. Greater vaccine conspiracy beliefs associated with higher vaccine hesitancy. However, in the presence of perceived vulnerability to disease, vaccine conspiracy beliefs associated with higher fear of COVID-19 and thereby lower vaccine hesitancy. We found under circumstances of higher perceived vulnerability to disease and fear of COVID-19, vaccine conspiratorial believers were less vaccine hesitant. We discuss how public health messaging can highlight personal risks to contracting COVID-19 to appeal to those who self-identify as disease prone, but may have reservations about vaccines because of misinformation. Successfully combating diseases entails reaching and gaining cooperation from misbelievers because misinformation is expected to continue in the digital age. By understand individual differences to vaccine hesitancy, it can help increase vaccinations and prevent severe illnesses in the post COVID-19 pandemic era.
Article
Full-text available
Background: At the time of this writing, the novel coronavirus (COVID-19) pandemic outbreak has already put tremendous strain on many countries' citizens, resources and economies around the world. Social distancing measures, travel bans, self-quarantines, and business closures are changing the very fabric of societies worldwide. With people forced out of public spaces, much conversation about these phenomena now occurs online, e.g., on social media platforms like Twitter. Objective: In this paper, we describe a multilingual coronavirus (COVID-19) Twitter dataset that we are making available to the research community via our COVID-19-TweetIDs Github repository. Methods: We started this ongoing data collection on January 28, 2020, leveraging Twitter's Streaming API and Tweepy to follow certain keywords and accounts that were trending at the time the collection began, and used Twitter's Search API to query for past tweets, resulting in the earliest tweets in our collection dating back to January 21, 2020. Results: Since the inception of our collection, we have actively maintained and updated our Github repository on a weekly basis. We have published over 123 million tweets, with over 60% of the tweets in English. This manuscript also presents basic analysis that shows that Twitter activity responds and reacts to coronavirus-related events. Conclusions: It is our hope that our contribution will enable the study of online conversation dynamics in the context of a planetary-scale epidemic outbreak of unprecedented proportions and implications. This dataset could also help track scientific coronavirus misinformation and unverified rumors or enable the understanding of fear and panic - and undoubtedly more. Clinicaltrial:
Article
Full-text available
The emergence of the 2019 novel coronavirus has led to more than a pandemic—indeed, COVID-19 is spawning myriad other concerns as it rapidly marches around the globe. One of these concerns is a surge of misinformation, which we argue should be viewed as a risk in its own right, and to which insights from decades of risk communication research must be applied. Further, when the subject of misinformation is itself a risk, as in the case of COVID-19, we argue for the utility of viewing the problem as a multi-layered risk communication problem. In such circumstances, misinformation functions as a meta-risk that interacts with and complicates publics’ perceptions of the original risk. Therefore, as the COVID-19 “misinfodemic” intensifies, risk communication research should inform the efforts of key risk communicators. To this end, we discuss the implications of risk research for efforts to fact-check COVID-19 misinformation and offer practical recommendations.
Research
Full-text available
In this factsheet we identify some of the main types, sources, and claims of COVID-19 misinformation seen so far. We analyse a sample of 225 pieces of misinformation rated false or misleading by fact-checkers and published in English between January and the end of March 2020, drawn from a collection of fact-checks maintained by First Draft News.
Article
Full-text available
Disinformation campaigns such as those perpetrated by far-right groups in the United States seek to erode democratic social institutions. Looking to understand these phenomena, previous models of disinformation have emphasized identity-confirmation and misleading presentation of facts to explain why such disinformation is shared. A risk of these accounts, which conjure images of echo chambers and filter bubbles, is portraying people who accept disinformation as relatively passive recipients or conduits. Here we conduct a case study of tactics of disinformation to show how platform design and decentralized communication contribute to advancing the spread of disinformation even when that disinformation is continuously and actively challenged where it appears. Contrary to a view of disinformation flowing within homogeneous echo chambers, in our case study we observe substantial skepticism against disinformation narratives as they form. To examine how disinformation spreads amidst skepticism in this case, we employ a document-driven multi-site trace ethnography to analyze a contested rumor that crossed anonymous message boards, the conservative media ecosystem, and other platforms. We identify two important factors that filtered out skepticism and contested explanations, which facilitated the transformation of this rumor into a disinformation campaign: (1) the aggregation of information into evidence collages—image files that aggregate positive evidence—and (2) platform filtering—the decontextualization of information as these claims crossed platforms. Our findings provide an elucidation of “trading up the chain” dynamics explored by previous researchers and a counterpoint to the relatively mechanistic accounts of passive disinformation propagation that dominate the quantitative literature. We conclude with a discussion of how these factors relate to the communication power available to disparate groups at different times, as well as practical implications for inferring intent from social media traces and practical implications for the design of social media platforms.
Article
Full-text available
In an online experiment, participants who paused to explain why a headline was true or false indicated that they were less likely to share false information compared to control participants. Their intention to share accurate news stories was unchanged. These results indicate that adding “friction” (i.e., pausing to think) before sharing can improve the quality of information shared on social media.
Article
2020 Owner/Author. How can social media platforms fight the spread of misinformation? One possibility is to use newsfeed algorithms to downrank content from sources that users rate as untrustworthy. But will laypeople be handicapped by motivated reasoning or lack of expertise, and thus unable to identify misinformation sites? And will they "game" this crowdsourcing mechanism in order to promote content that aligns with their partisan agendas? We conducted a survey experiment in which =984 Americans indicated their trust in numerous news sites. To study the tendency of people to game the system, half of the participants were told their responses would inform social media ranking algorithms. Participants trusted mainstream sources much more than hyper-partisan or fake news sources, and their ratings were highly correlated with professional fact-checker judgments. Critically, informing participants that their responses would influence ranking algorithms did not diminish these results, despite the manipulation increasing the political polarization of trust ratings.
Chapter
This chapter discusses the circulation of conspiracy theories evolving from concoctions of Internet subcultures to global topics of public conversation and political mobilization. The examples provided are those of the Pizzagate and QAnon conspiracy theories, which embody the anti-establishment ethos, the paranoid disposition and the ironic attitude of far-right on-line communities. The chapter analyzes the spreading of a set of myths, symbols and codes created by the 4chan and 8chan users within a global network of White ethnonationalists. The far-right anti-immigration conspiracy theory ‘The Great Replacement’ is discussed to explore the interlocking themes of White identity politics, trolling and the ‘weaponization’ of Internet entertainment.
Article
We study the problem of automatic fact-checking, paying special attention to the impact of contextual and discourse information. We address two related tasks: (i) detecting check-worthy claims and (ii) fact-checking claims. We develop supervised systems based on neural networks, kernel-based support vector machines, and combinations thereof, which make use of rich input representations in terms of discourse cues and contextual features. For the check-worthiness estimation task, we focus on political debates, and we model the target claim in the context of the full intervention of a participant and the previous and following turns in the debate, taking into account contextual meta information. For the fact-checking task, we focus on answer verification in a community forum, and we model the veracity of the answer with respect to the entire question–answer thread in which it occurs as well as with respect to other related posts from the entire forum. We develop annotated datasets for both tasks and we run extensive experimental evaluation, confirming that both types of information—but especially contextual features—play an important role.