ChapterPDF Available

Deepfakes and Society: What Lies Ahead?

Authors:

Abstract and Figures

Deepfakes created in the form of image, audio and video by leveraging AI are far more realistic to identify if its being synthetically created by replacing someone else's voice or video. This is an emerging concern as the implications of such technology may distress the society. Currently, much of the scholarly research is centered on the technology of deepfake, but sparse in understanding how the emergence of deep-fakes impacts society. In this chapter, we provide (1) an understanding of deepfake related research with a focus on societal implications through a systematic review and (2) an empirical understanding of the societal implications of deepfake by analyzing Reddit conversations related to deepfakes from 2018 to 2021. A systematic review found 787 deepfake related research, yet only 88 were depicting any social implications side of deepfakes. The majority were literature reviews and less focused on empirical evidence. Our empirical study provided evidence of possible implications concerning society. We finally provide 5 directions to mitigate deepfake societal harms.
Content may be subject to copyright.
Deepfakes and Society: What lies ahead?
Dilrukshi Gamage, Jiayu Chen, Piyush Ghasiya, and Kazutoshi Sasahara
Department of Innovation Science, Tokyo Institute of Technology, Tokyo, Japan
Graduate School of Education and Human Development, Nagoya University,
Nagoya, Japan
dilrukshi.gamage@acm.org
chen.jiayu@h.mbox.nagoya-u.ac.jp
ghasiya.p.aa@m.titech.ac.jp
sasahara.k.aa@m.titech.ac.jp
https://www.colorlessgreen.info/
Abstract. Deepfakes created in the form of image, audio and video by
leveraging AI are far more realistic to identify if its being synthetically
created by replacing someone else’s voice or video. This is an emerging
concern as the implications of such technology may distress the society.
Currently, much of the scholarly research is centered on the technology
of deepfake, but sparse in understanding how the emergence of deep-
fakes impacts society. In this chapter, we provide (1) an understanding
of deepfake related research with a focus on societal implications through
a systematic review and (2) an empirical understanding of the societal
implications of deepfake by analyzing Reddit conversations related to
deepfakes from 2018 to 2021. A systematic review found 787 deepfake
related research, yet only 88 were depicting any social implications side
of deepfakes. The majority were literature reviews and less focused on
empirical evidence. Our empirical study provided evidence of possible
implications concerning society. We finally provide 5 directions to miti-
gate deepfake societal harms.
1 Introduction
The rapid development of technologies such as Artificial Intelligence (AI) and
Deep Learning (DL) revolutionized the way we create and consume content. As a
byproduct of this revolution, we witness emerging technologies such as Deepfake
which may potentially harm and distress social systems. Deepfakes are synthetic
media generated using sophisticated algorithms which reflect things that did not
happen for real but computer-generated for manipulation purposes [118].
Currently, a myriad of scholarly works concentrate on specific DL
techniques—types of the neural network model in which the model is trained
to restore the input data known as autoencoders. These are typically Generative
Adverse Models (GAN) that involve a generator and a discriminator in build-
ing an image closer to the original one. Usually, Conditional GANs (CGAN)
types generate data while controlling attributes giving attribute information in
addition to images during the training, face swapping, and speech synthesizing
Book chapter -
Gamage, D., Chen, J., Ghasia, P., Sasahara, K., Deepfakes and Society: What lies ahead?
Frontiers in Fake Media Creation and Detection, Springer, 2022
2 D. Gamage, et al.
techniques [40]. Currently, many studies on deepfakes are more influenced by
the nuances of the technology—generation and detection methods for deepfakes.
However, the advancement of these scholarly works and the democratization of
these technologies made it easy for any individual to generate realistic fake media
content which could have been difficult previously.
The popularizing of deepfakes is concerning. Deepfakes could be contentious
and undermine our trust in the content. However, on the other hand, they open
up new creative opportunities as well. Deepfakes are mostly used in creating
online education contents and advertising fields. However, they are posing a
risk to the society, impacting the information consumption experience, and have
already created negative consequences. The latest much debatable incident is
the use of deepfake voice in the documentary film “Roadrunner: A Film About
Anthony Bourdain,” a famous chef who ended his own life due to depression.
The film director has incorporated AI synthetic voice to show as if Bourdain
were speaking for real as a part of the scene. However, such a voice cut was
never been originated or consented to by the late chef himself. This incident
brought much discussion and debate on the ethical implications of the use of
deepfakes—synthetic audio, video, or text in general applications [97]. At the
same time, other recent incidents such as the first deepfake civil level case in
Japan where a student created a deepfake celebrity pornography [51], a British
oil company CEO voice fraud by deepfake [106], and a Pennsylvania mother cre-
ating a deepfake pornography video of her daughter’s high school cheer-leading
squad to replace the cheer-leading position [39] collectively bring an alarming
concern to the society. The term “deepfake” was first coined in late 2017 by a
Reddit user of the same name. This user created a space on the online news
and aggregation site, where they shared pornographic videos that used open
source face-swapping technology1. Specifically, deepfakes may harm democracy
by disseminating political disinformation [113], undermine national and interna-
tional security [98], and complicate law enforcement and harm the entertainment
industry by disseminating fake celebrity pornography [44].
We are witnessing a lot of attention directed towards Machine Learning (ML),
AI, and DL advancement to perceive and detect how people are manipulating
content in research, academia, and industry. Much research is currently con-
centrating on hardcore computer science—automatic generation [120, 18] and
detection of deepfakes [73, 92]. Little attention has been devoted to understand-
ing how humans perceive or interact with deepfake content or how deepfake is
used and what the societal implications of deepfakes are. To date, only a hand-
ful of research has shown how the public perceives and is deceived by these
synthetic media. Researchers suggest that exposure to deepfakes and concerns
about them are positively related to social media news skepticism, but they found
people relying on social media as a news platform are found to be less skeptical
on the deepfakes [113]. On the other hand, the latest human-computer Interac-
tion (HCI) research suggests that humans can be easily deceived by deepfakes,
yet, they found contextualized education and training could increase the ability
1https://mitsloan.mit.edu/ideas-made-to-matter/deepfakes-explained
A chapter for Frontiers in Fake Media Generation and Detection 3
to undermine the deepfake media [108]. The emerging social space for deepfake
needs extensive interrogation and understanding of how it is situated in every-
day public conversation and what directions these conversations are heading in.
However, at present researchers are more interested in understanding deepfake
from the technological perspective and less attention has been paid about how
it is impacting society. Much of the empirical research is needed to identify
the human-centric social process of deepfakes, and the implications should be
investigated by making an interdisciplinary effort.
The main objective of this chapter is to provide an overview of scholarly work
which concentrates on the social implications of deepfake and to examine the
public discourse concerning the impact of deepfakes through empirical evidence.
To provide a research overview, we conducted a systematic literature review
considering the studies between 2018 to 2021. To understand the impact of
deepfakes and to provide empirical evidence, we conducted a discourse analysis
using topic modeling method.
The chapter is organized such that firstly, it provides a high-level understand-
ing of the deepfake technology followed by an overview of the types of deepfakes
found in the society. Next, it depicts a background to the related deepfake re-
search to understand how and what kind of studies have been conducted in
terms of understanding deepfakes societal impacts. In the background, we have
also included some previous studies of discourse analysis in particular to Reddit
platform since we obtained the corpus from Reddit to empirically understand
the deepfake impacts. Next, we explained the details of the methods followed
by the results—the systematic literature review and Reddit discourse analysis.
Finally, we discussed the future directions to mitigate the societal harm that
may be caused by the deepfake technologies. The discussions provoked in this
chapter will provide signals to many multidisciplinary researchers to collectively
work to understand potential harms and build applied human-centric solutions
mitigating the unprecedented risks.
2 Background to the deepfake studies
2.1 Deepfake technology
Deepfake is synthetic media where one person’s voice, image, or video is re-
placed by someone else which reflects as if it’s the original person speaking, or
their photo, or their moving image. Deepfakes leverage AI and ML — provides
optimized techniques to generate visual and audio content with a high poten-
tial to deceive. Generative adversarial networks (GAN), deep neural network
models are responsible for training a target object which replaces the original
and reflects deepfake using an optimized combination of encoding and decoding
techniques. Encoding is responsible for reducing an image to a lower-dimensional
latent space, and decoding involves reconstructing the image from the latent rep-
resentation. When creating deepfakes, the latent space is used to model the key
features to decode the target object. It should consist of the key features about
the target object such as facial features and body postures where a model will
4 D. Gamage, et al.
superimpose the target on the underlying facial and body features of the original
video [109].
Deepfakes predominantly represent the research by academics and industries
in specific fields of computer vision, ML and AI. They mostly focus on projects
for improving deepfake techniques [14, 107, 111, 21] and also methods for detect-
ing deepfakes [125, 69, 70, 90, 1]. A comprehensive survey on deepfake creation
and detection techniques predicts that deepfakes will be more weaponized for
monetization [77]. Since the tools are becoming more accessible and efficient, it
seems natural that malicious users will find ways to use the technology for profit.
As a result, researchers expect to see an increase in deepfake phishing attacks
and scams targeting both companies and individuals [77].
2.2 Types of deepfakes in society
The deepfake phenomenon must be positioned within the sphere of society and
should be examined from the contexts of its uses (or potential uses), along with
its effects. Currently, researchers exclaim that deepfakes are expected to have
implications in law and regulation, politics, media production, media represen-
tation, media audiences, and overarching interactions in media and society [55,
77]. With the commercial development of AI applications such as FakeApp
(https://www.faceapp.com), Deepfake Web (https://deepfakesweb.com) and
Zao (https://zaodownload.com) or audio deepfake apps such as Resemble
(https://www.resemble.ai), or Descript (https://www.descript.com/overdub) or
with the use of synthesized text [50], it is evident that attacks involving human
reenactment and replacement are emerging. An Amsterdam-based cyber security
company released a report in 2019 predicting that 96% of all deepfakes online
were pornographic [7], mostly created using US and British celebrity actresses’
pictures. This report named Deeptraced has been reused by many scholars to
predict deepfake outcomes [6, 124, 89]. On top of this, research reveals alarming
concerns over developments in deepfake child pornography [31]. Although it is
not functioning anymore, we can also highlight the impact and negative dis-
course brought by deepfake application DeepNude, which removes clothing from
images of women [37].
In addition to these disturbing usages, deepfakes are affecting democracy
by using applications in political contexts. Although some cases have appeared
to educate the general public, other instances were found that deepfakes had
the intention of spreading disinformation as well. One such example for making
awareness is the video created by Buzzfeed where Jordan Peele is made to resem-
ble Barack Obama, with Peele’s voice serving as a public service announcement
intended to increase public consciousness of deepfakes [96]. Another example is
the use of deepfake by Bharatiya Janata Party (BJP — the current ruling party
in India) to distribute a version of an English-language campaign advertisement
by its politician, Manoj Tiwari, translated into the local language Haryanvi to
target Haryana voters [56]. However, incidences such as the deepfake video of
the prime minister of Belgium, Sophie Wilmes telling the public about the gov-
ernment’s decisive action on scientific findings, COVID-19, and the ecological
A chapter for Frontiers in Fake Media Generation and Detection 5
crisis created by a global environmental movement is prompting serious ques-
tions on deepfake as part of political militant action and the impact it creates in
people’s mind. Although no evidence exists of the direct use of deepfakes in po-
litical manipulation, the latest research emphasizes the possibility of such since
the advancement of technology does not need thousands of images to create so-
phisticated deepfakes [67]. Current platform models are content-driven and use
incentives to increase the number of views and likes. Platforms such as Facebook,
YouTube, TikTok, and Reddit are all competing for users’ attention in this re-
spect. Due to significant monetary incentives, content creators are pushed into
presenting dramatic or shocking videos, where potential deepfakes may draw the
attention of the public. However, some research work advises us not to overesti-
mate the impact of deepfakes — yet it explains that the increased proliferation
of deepfake videos will eventually make people more aware that they should not
always believe what they see [67]. However, we argue that much empirical re-
search is essential to understand the impact and estimate the potential damage
where interventions are needed before deepfake causes harm.
2.3 Social centric research for deepfake
While continuing to investigate deepfake technology, it is paramount to under-
stand how the public can be deceived by deepfakes and the empirical pieces of
evidence of deepfake deceptions. Although limited research is to be found, pro-
viding social evidence of how people react or what they understand when they
see deepfakes, a study conducted in 2020 using a sample of 2,005 UK citizens
provides the first evidence of the deceptiveness of deepfakes [113]. In many pub-
lished articles discussing deepfake’s social implications, the authors have brought
many anecdotal shreds of evidence leading to the prospect of mass production
and diffusion of deepfakes by malicious actors and how it could present extremely
serious challenges to society [32]. Since the social perspectives are based on “see-
ing is believing” or “a picture worth thousands of words,” images and videos
have much stronger persuasive power than text where citizens have compara-
tively weak defenses against visual deception of this kind [80]. Particularly the
research by [113] tried to understand how people can be deceived by political
deepfakes, and it turns out that society is more likely to feel uncertain than to be
completely misled by deepfakes. Yet, the resulting uncertainty, in turn, reduces
trust in the news on social media. The social implications of deepfakes can be
easily grounded when we understand how people interact with such technology.
A survey taken among 319 people was used to examine issues around deepfakes
such as their awareness, concerns, and responsibility of online platforms around
deepfakes [25]. They found that awareness of deepfakes varies by intensity and
type of social media use yet they found general concerns and the impacts people
believe deepfakes will make are significant. Although people had little confi-
dence in the ability of technology to solve the problem of deepfakes, it did not
reduce their desire for online platforms to implement a deepfake identification
technology [25].
6 D. Gamage, et al.
Overall, considerable research has highlighted that deepfakes are likely to
attack in the spheres of political disinformation and pornography. One of the
latest studies, the likelihood of sharing deepfakes is based on individuals’ cog-
nitive ability, political interests and network size. They found that individuals
with higher levels of political interest are more likely to share deepfakes inad-
vertently whereas individuals with higher cognitive ability being less likely to
inadvertently share deepfakes [4]. Another research report states that deepfakes
have the potential to influence multiple sectors of society such as the financial
sector which relies on — information, politics, national and international secu-
rity. They may even threaten private lives by revenge porn, faked evidence used
in legal cases and lead to distressed social lives [15]. They have analyzed popular
articles of deepfakes as a way to understand the social and psychological impacts
of this phenomenon on people.
By far one of the most factual pieces of evidence of widely known deepfake
implications — revenge porn analysed in a research provide gaps in the process of
reporting revenge porn abuse in selected content sharing platforms such as social
networks, image hosting websites, video hosting platforms, forums, and porno-
graphic sites [27]. Their results confirmed serious gaps in technology designs
to facilitate such reporting. This research provides a view to current practices
and potential issues in the procedures designed by the providers for reporting
these abuses [27]. With all of the research, it is evident that deepfakes could
cause deeper damage to individuals and society whereas technology designs are
far apart from seeing solutions. Prevention and detection may be one way of
looking at it. The design of digital technologies will play an important role in
reacting to any social issue as a consequence of deepfakes and interdisciplinary
approaches need to be taken for successful solutions. One of the main objectives
of this chapter is to provide a systematic review of the literature to showcase the
types and distribution of deepfake literature in relation to societal implications.
It helps to understand the current trend in deepfake research and highlight the
future directions.
2.4 Discourse analysis in Reddit Communities
In this section, we provide an overview of the previous Reddit analysis studies,
because we used Reddit corpus for our empirical study. Our second objective
was to understand the societal impact of deepfakes by analyzing Reddit con-
versations from 2018 to 2021. It is because deepfakes were first introduced in
the Reddit community and their conversations have already made an impact on
the ML field and the use of GAN models. Before conclusion, we examined how
previous Reddit studies had been conducted. We used a mix of quantitative and
qualitative methods to derive insights from the Reddit conversations.
Historically, studies conducted on Reddit platform dealt with sensitive top-
ics [103, 117, 75]. This is mainly to understand deep-dive insights on topics that
individuals may not discuss in open public spaces where they can be identified.
We highlight recent research where the author has conducted a qualitative study
to examine Reddit users’ views and attitudes about the sexualization of minors in
A chapter for Frontiers in Fake Media Generation and Detection 7
deepfakes and “hentai” which is known for overtly sexual representations (often
of fictional children). Based on a large dataset (N= 13293) of Reddit comments
collected on the topic, the analysis captured five major themes regarding the
discussion over sexualization of minors: illegality, art, promoting pedophilia, an
increase in offensive and general harmfulness. Their study provides information
useful in designing prevention and awareness efforts addressing the sexualization
of minors and virtual child sexual abuse material [31]. Similarly, another research
was conducted using Reddit to analyze domestic abuse discourse [99]. They have
developed a classifier to detect discussions relating to domestic abuse, and over-
all the study provides insight into the dynamics of abusive relationships. [99].
Their study has followed a similar study design found in an analysis of men-
tal health where researchers used subreddits on the topic of mental health and
found differences in discourse between subreddits and regular accounts [9]. This
appears to show that, observing particular subreddits and qualitative coding
conducted for a sample of posts in selected subreddits are steps commonly fol-
lowed by researchers. In our study, however, we focused on the deepfake-related
posts and comments, but not a particular subreddit, since we were interested to
understand wider perspectives on deepfakes.
Apart from qualitative analysis, quantitative approaches are also found in
Reddit analysis. One study conducted topic modeling and hierarchical cluster-
ing to obtain both global topics and local word semantic clusters to understand
factors related to weight loss. They have used a regression model to learn the
association between weight loss and topics, word semantic clusters, and online
interactions [71]. Another study used large-scale data from the political left and
right-wing subreddits and analyzed the similarities and differences in posting
behaviors during 2016-2018. This research incorporated computational social
science methodologies wherein millions of user base and posts were explored
using word frequency modeling, post distribution, and many other categoriza-
tions based on the posts they share [102]. As mentioned, many previous research
on Reddit community analysis provides different methodological approaches to
bring a narrative to an exploratory field in deepfake discourse.
Although any social platform’s user-generated content and activity offer a
unique opportunity for understanding how individuals and social groups inter-
act toward certain phenomena, we specifically used Reddit to understand the
dynamics of social interactions centered around deepfakes. When certain sub-
reddits responsible for deepfake were banned, we observed emerging discussions
on deepfakes in many other subreddits (i.e., r/deepfakes). Unlike some stud-
ies that used one or more particular subreddit to understand communities, we
observed a wide range of subreddits such as technology, movies, finance are dis-
cussing deepfakes as trending posts. Although we found a study conducted over
deepfake discourse using Twitter data [89], which discusses the characteristics
of users’ deepfake content sharings and their demographic details, our study
digs deep insight into the dynamics of discourse in Reddit communities where
these anonymous users could offer valuable insights in the context, helping with
understanding the emerging conversations towards deepfakes. Analyzing these
8 D. Gamage, et al.
conversations is likely to question the impacts that may create in future com-
munities that might pose a threat to society. To date, Reddit communities have
not been used to understand the deepfake’s social implications, yet researchers
emphasize the necessity for such analysis [41].
3 Methods
3.1 Study design and research questions
In this section, we provide details of two study designs— a systematic literature
review and Reddit discourse analysis. Each study design is led by its main ob-
jective and framed to research questions where the answers fulfill the objectives.
The first study aimed at understanding the gaps and future directions for
deepfake research, which needs attention from society’s viewpoint. We specifi-
cally addressed the following research questions:
– Study1-Q1: What types of researches were conducted between 2017-2021 to
understand the psychological and social dynamics and societal implications
of Deepfake?
– Study1-Q2: What is the distribution of Deepfake research between 2017-
2021 that explores any type of psychological dynamics and its societal im-
plications
In the second study, our objective was to understand the possible implications
pose by deepfake conversations and we selected Reddit as a source to obtain
corpus. In this study, we specifically posited the following research questions:
Study 2-Q1: What type of conversations lead in Reddit communities about
deepfake and how do these conversations change over time?
– Study2-Q2: What are the implications visible in these conversations?
3.2 Study 1: Systematic literature review on research studies from
2018-2021
We obtained articles by searching popular scientific search engines and
repositories—Springer Digital Database, IEEEXplore, ACM Digital Database,
Web of Science, and Scopus. Most systematic reviews incorporate Preferred Re-
porting Items for Systematic reviews and Meta-Analyses protocols (PRISMA)
explained in detail by [78]. We followed a similar structure to this literature
review with a particular interest in understanding the two previously mentioned
research questions. We used a search query in all 5 databases and in addition to
this, incorporated Google Scholar to search any other relevant preprints or non-
peer-reviewed articles to bring more inclusively to the research which may not
have been listed in ACM, Scopus, IEEE, Web of Science or any other database.
Following is our search query.
{Deepfake OR Artificial Intelligence}AND Misinformation
A chapter for Frontiers in Fake Media Generation and Detection 9
We did not restrict our search to only journal papers but allowed any peer-
reviewed paper, or commentary, critical reviews, or even work-in-progress papers
including preprints. After the search terms provided the dataset, we used two
experienced researchers to filter the deepfake-related researches based on inclu-
sion criteria. This is because the search terms brought broadly scoped literature
which may not directly discuss about deepfakes. We were particularly careful to
select the results only if the manuscripts examined the perceptions of deepfake or
its impact on human interaction or discussed the social implications of deepfake.
In other words, articles that discussed a pure technology perspective (such as
GAN), or studies to find new techniques for Deepfake detection were eliminated
as irrelevant to this study. Figure 2 describes the data process.
Fig. 1. Flow of the systematic review
Dataset. Our initial search query extracted 787 articles from 5 databases. The
extracted results were then combined to a single data file and two researchers
collectively further filtered based on the inclusion criteria depicted in Fig. 2 by
manually reviewing the abstracts. In addition to these filtered articles, additional
papers were added based on the relevant research found by Google Scholar and
we labeled this source as “Other”. Although a Google Scholar advance search
returned 3420 hits, given the depth and spread of the articles we focused only
on the first 20 pages which had 200 Hits and selected 9 highly relevant papers
not included in any databases. Out of these, 4 papers were from journals and, 2
universities repositories which were not listed in any of the 5 databases. Another
10 D. Gamage, et al.
2 were preprints and are currently under review, 1 commentary from Nature.
We found 77 highly relevant papers from the 5 original databases and with
the Google Scholar results had 88 papers selected for analysis. A breakdown is
depicted in Table 1.
Table 1. Summary of the results retrieved by running the search query and manually
filtering by reviewing according to the inclusion criteria.
Search Database Hits Selected
Springer Online Database 177 17
IEEE 154 11
ACM 264 8
Web of Science 137 41
Scopus 55 2
Other (Google Scholar) NA 9
Total 787 88
Measures. To answer RQ1, we analyzed all 88 papers using their full text,
summarized the key phrases, highlighted major findings in the respective pa-
pers, and identified any themes under which the article could be categorized.
Based on the summary and key phrases, it was evident that the corpus can be
categorized by a common methodological standpoint. For example, each article
can be categorized by whether it conducted an experiment to understand social
dynamics or had any sort of methodical analysis to understand the social impact
or if it was produced as a result of an extensive critical review by positioning
any premises or even if it provided a conceptual proposal or framework beyond
the review of the deepfake social phenomenon. We also examined whether or not
the corpus focused on several domain areas addressing deepfake social issues.
We incorporated word clouds on each abstract to support subjective judgment
on categories and focus areas.
To answer the RQ2, on the distribution of deepfake research in psychological
dynamics and its societal implications, we described descriptive statistics with a
network analysis that reveals the connections with its type of research and em-
phasis. At the same time, we highlighted the generated word clouds, specifically
depicted the categorical flows based on the frequencies, and used the network
diagram to illustrate the author distributions among the selected papers.
3.3 Study 2: Empirical study on Reddit conversations from
2018-2021
In this study, we followed a mixed-methods approach to analyse the Reddit con-
versations. In particular, this approach intersects with the HCI research tradition
and computational social science where similar methods are found in some other
A chapter for Frontiers in Fake Media Generation and Detection 11
Reddit analyses [61, 112], especially the qualitative aspects [84] and quantitative
approaches [22]. Data were collected from the Reddit platform between 2018 and
2021. We used the Pushshift Reddit API, passing a query for the search term
“deepfake” to obtain all the posts and Reddit API to obtain all comments in-
corporated in the posts. In return, the script provided all submissions (Reddit
posts), comments in the submissions, dates, names of subreddits, and usernames
of the comments and posts.
Dataset and Measures. The “deepfake” search query yielded a large data set
with 86,425 comments distributed among 6638 posts, between 1st January 2018
and 21st August 2021 (01/01/2018-21/08/2021).
To answer RQ1: What types of conversations are leading in Reddit commu-
nities about deepfake and how do these conversations change over time?— in
order to understand the leading conversations, we relied on topic modeling. In
ML and NLP, topic modeling is an unsupervised approach to recognize topics by
detecting patterns, which acts similar to clustering algorithms that divide the
data into different parts. It helps to understand and summarize large collections
of textual information and discover latent topical patterns that are presented
across the collections. For the second part of the question, a temporal analysis
of posts and comments was used to reflect the shift in the types of posts and com-
ments, and the direction of leading topics. We used Python the NLTK library
for text pre-processing [12] and followed the common steps such as removing
HTML tag, converting all words to lower case, tokenizing which provides each
work a token, etc.
After pre-processing the data, we performed the topic modeling to discover
the abstract “topics” that occur in both posts and comments. There are several
topic modeling methods such as Latent Dirichlet Allocation (LDA), Latent Se-
mantic Analysis (LSA), Non-Negative Matrix Factorization (NMF). To produce
the best topic model, prior knowledge of the optimal number of topics present
in the data is essential. This (optimal number of topics) is one of the main chal-
lenges in topic modeling. Usually, it is done manually (eyeballing) by running the
model with a different number of topics and selecting the model that produces
the most interpretable topics. If Topic Coherence-Word2Vec (TC-W2V) metric
is used with NMF, it can find the optimal number of topics automatically [85].
This metric measures the coherence between words assigned to a topic. For ex-
ample, how semantically close are the words that describe a topic. We trained
the word2vec model [76] on our corpus, which would organize the words in an
n-dimensional space where semantically similar words are close to each other.
The TC-W2V for a topic will be the average similarity between all pairs of the
top nwords describing the topic. We then trained the NMF model for different
values of the topic (k= 1 to 19) and calculated mean topic coherence across all
the topics. The kwith the highest mean topic coherence is used to train the fi-
nal NMF model. For the above implementation, we used the scikit-learn python
library and built an NMF based topic model [88].
12 D. Gamage, et al.
The first stage, we performed topic modeling on the entire corpus grouped
by “posts” and “comments”. This included all the posts and comments between
2018 to 2021. This provided an overall view of the topics from what users post-
ing, and commenting. We also measured the weight of the terms in each topic to
understand which terms were highly representative of the topic. Next, we chun-
ked the corpus group by each year and performed topic modeling. This provided
an overview of hidden thematic information of engagement in the form of posts
and comments each year. This helped us to understand if topics in deepfake
discussions were changing over time. To understand the temporal data, we sum-
marized the number of posts and the number of comments distributed in each
month of the year for all four years (2018-2021). At the same time, we provided
a summary of posts, comments, and the number of subreddits in each year which
provides an overview of engagement frequency over time.
To answer the RQ2: What are the implications visible in these
conversations?—we relied on open coding used in qualitative methods in HCI.
The codes were used to specifically map the discussed topics with possible impli-
cations to society. In particular, we used the generated topics from 2018 to 2021
and obtained a sample of relevant conversations (comments and posts) from each
topic resulting from topic modeling to do the open coding. Two researchers inde-
pendently observed and identified possible codes for each topic from comments
and posts. Next, these two researchers together with another two researchers
collectively compared the codes, resolved any disagreements, and finalized the
key themes that resonate with implications. Out of the four researchers, two
researchers had extensive experience in qualitative data analysis and all four re-
searchers had a sense of the gathered data, understood the context of deepfakes,
and needed practical knowledge to understand the capabilities of the deepfake
technology. Figure 2 depicts the overall procedure.
4 Results
4.1 Study1-Q1: Types of research in the systematic literature
review
Overall, the majority of the results from the query resulted from scholarly work
were related to deep learning (DL), AI and ML, and their improvements in
creating or detecting Deepfakes. Only 88 out of 787 were representing research
works that discuss the psychological dynamics, social implications, harms to the
society, ethical standpoint, and solutions from a socio-technological viewpoint.
Examining abstracts and full text of the articles, we identified that each ar-
ticle could be categorized based on 11 types of research—Systematic review, Re-
view based on Literature, Philosophical mode of inquiry, Examines, Experiment,
Network Analysis, Content Analysis, Design, Conceptual Proposal, Commentary
and Analysis by Examples. Although these categories are based on subjective
judgments of the authors, it provides a solid understanding of the conducted
research based on its main objectives and methods.
A chapter for Frontiers in Fake Media Generation and Detection 13
Fig. 2. Process of obtaining data, pre-processing, topic modeling for data in 2018 to
2021 as a whole and for each year. At the final phase, qualitative coding was conducted
for the sample comments and posts obtained from each topic.
A magnified view of this dataset (88) revealed that the majority (30) of the
papers focused on critical reviews based on the previous literature and slightly
above half of the papers (21) conducted active experiments using real users to
explore the social and psychological dynamics of perceiving deepfakes or under-
standing their impacts. Only one study was performed a network analysis based
on Deepfake discourse and limited other papers focused on the rest of the meth-
ods as depicted in Fig. 3. Apart from the methodology point of view, we also
derived key categories of the papers based on their focus area. Although our key
interest centered upon deepfake and its social impact, we observed that the rel-
evant research covered a wider range of focus areas in different subject domains.
These areas ranged from security aspects, pornography, legal concerns, deepfake
media, specifically video and images, psychological/political/human cognition
perspective, and more. Therefore, to specifically answer RQ1, we describe the
details of these methods and focus areas in the following sections.
Methodology used in deepfake social implication research. Although
methodical approaches for research are not new, our analysis of the 88 highly
relevant papers for the social or psychological implications of deepfakes reflected
that most of the research in this domain is still developing. Many of the re-
search found are types of research that critically evaluate and analyze deepfake
phenomena from the previous literature while discussing potential future out-
comes. We categorized this type of research as Review based on Literature
and from our corpus, the earliest research on critical reviews of deepfake social
implications occurred in 2019 (although the term “Deepfake” the first time in
14 D. Gamage, et al.
Fig. 3. Scholarly work distribution based on the year it was published, the Published
databases and its Methodology
2017 [118]). Research by [119] raises questions about understanding whether the
deepfake phenomenon is shallow or deep and how society might react to these
technologies. Specifically, the paper critically analysed and predominantly pro-
vided nuances to the technology that generates deep fake media and its uses,
showing that society has never relied solely on the content as a source of truth.
Similarly, [8] provides an extensive narration to deepfakes and relates its con-
sequences to terrorism. The author does not follow a systemic approach, however,
there is a critical discussion of the deepfake focusing on the near future of secu-
rity threats by using examples of previous literature and emphasizing the need
for awareness, law enforcement, and policymakers to implement effective counter
terrorism’s strategies. While providing this background and previous work, the
author also articulates his stance on the subject emphasizing that we are tran-
sitioning from e-terrorism to upcoming online terrorism, as well as the linearity
to hyper-complexity by malicious use of AI and living in the post-truth era of a
social system. Since his research article not only provides a critical review based
on past literature but also the authors’ theoretical and qualitative research ex-
perience with participation and working as a counter-terrorism expert in related
projects, we also intersected this with a new category: Examines. Through our
A chapter for Frontiers in Fake Media Generation and Detection 15
full-text analysis, we observed that many other Review based on Literature
scholarly work intersects with the Examines category. In these types of arti-
cles, we observed that authors critically provided their own experiences or using
their point of view as a metaphor to build constructs. All together we found
11 out of 30 papers categorized as Review based on Literature illustrated this
intersection. For example, the review article by [41], attempts to understand
the possible effects the deepfakes might have on people, and how psychological
and media theories apply. In addition, the article by [83] brings a philosophical
mode of inquiry to a pervert’s dilemma, an abstraction about fantasizing sexual
pornography and argues that ethical perspectives underline dilemmas by using
the literature and theories. Similar placement of arguments and concepts sup-
ported by the review of literature can be found in articles by [110], [56], [66],
[93], [43], [67] and [38]. However, we also derived four research articles that fall
in the category of Examines without a dominating critical literature review —
For example, an article was compiled while examining US and British legisla-
tion indicating legislative gaps and inefficiency in the existing legal solutions and
presenting a range of proposals of legislative change to the constitutional gaps
in porn [74]. The article examines current online propaganda tools in the con-
text of different information environments, and provides examples of their use,
while seeking to educate about deepfake tools and the future of propaganda [87].
Another study examines the problem of unreliable information on the internet
and its implications for the integrity of elections, and representative democracy
in the U.S. [123], and an another study that addresses the economic factors
that make the confrontational conversation more or less likely in our era and
brought viewpoints in the deepfakes which becoming more widespread on the
dark web [38] are falling into this Examines category.
However, alongside review-based articles and articles that conducted an ex-
tensive examination, we also derived another category. Although this category
is similar to the methods we previously stated, it is distinguished by the way
it positions its point of view. We noticed that this type of article is extensively
based on use cases, examples of incidences, and more descriptions of theoretical
and informational AI and deepfake technologies. We name this category Anal-
ysis by Example and found five papers that fall under its umbrella. Articles in
this category include [86], through their examples of deepfakes in the modern
world, and the internet-services, [5] study the use of legal regulation in the use
of facial processing technologies, and [19] study possible applications of artificial
intelligence and related technologies in the perpetration of crimes, [28] study
the general analysis of risks and hazards of the technologies and analysis exam-
ples of legal remedies available to victims. We also identified a category named
Philosophical Mode of Enquiry which includes papers that use a philosoph-
ical viewpoint in premising their inquiry into the social issues found within the
deepfake applications [83, 126, 33, 45, 64].
However, since the developments in the area of social implications of deep-
fakes are yet growing, we observed only two Systematic Review types of re-
search that explain in detail the growing body of literature and its systematic
16 D. Gamage, et al.
analysis [35, 118]. The first systematic review used English-language deepfake
research to identify salient discussions, and the other used 84 publicly available
online news articles to examine what deepfakes are and who produces them, and
the benefits and threats of deepfake technology in 2021 and 2019 respectively.
However, apart from these critical reviews, examiner papers, analysis by exam-
ples, and systematic reviews, we found one other method that could be classified
into the same theme but distinct in its narration of the information as it is made
as a personal opinion or commentary to certain events. We named this category
as Commentary Bases which often provides a short narrative for the question
of the future of technological implications [53, 65, 10, 104, 105].
Next, we observed that 21 out of 88 papers depicted some sort of experi-
ments using human subjects to understand any impact and social implications
of Deepfake and we named this category Experiment. In this category we ob-
served researchers such as [58] used 30 users to examine human judgments on
deepfake videos, [20] used 161 users to explore the relationship between a per-
son’s demographic attributions, political ideology and the risk of him/her falling
prey to mis/disinformation attacks. The largest study conducted by [121] used
1,512 users to explore the impact of four types of credibility indicators on peo-
ple’s intent to share news headlines with their friends on social media. Similarly,
[30] studied effects on political attitudes using 271 users, [62] studied the in-
ability of people to reliably detect deepfakes using 210 users. Their research
particularly found neither by educating or introducing financial incentives im-
proves their detection accuracy experimented and many other similar studies
contained in this category. Apart from experiments, we also found research arti-
cles proposing frameworks or solutions to deepfake societal issues by conceptu-
alizing theoretical frameworks [17, 60, 59] named as Conceptual Proposals.
Beyond conceptual proposals, we also found that some articles consisted of clear
design goals with implementation plans or some artifacts designed as solutions
to the issues of deepfake societal issues [24, 91, 23, 101, 49]. Thus, we introduced
a category named Design.
Apart from such dominated methods to observe social implications and per-
ceptions of deepfakes, we also found seven articles that followed the Content
Analysis method. Three used Twitter data as their corpus [72, 82, 46] and two
studies analyzed the article content in news media [15, 36]; each study ana-
lyzed YouTube comments about deepfakes [68] and journalist discourse [116]
to understand the social implications of the deepfake. Although similar to these
studies, we categorized one more study as Network Analysis and it conducted
a semantic content analysis using Twitter data relating to Deepfake phenom-
ena [26] to understand the social discourse.
Range of focus areas examining Deepfake and its social implications.
Apart from the key categorization towards research methods, we examined the
significant research questions these research methods are used to solve. This
aids us in categorizing the deepfake social research based on the subject areas
on which it is focused. We derived 30 main focus areas these research articles
A chapter for Frontiers in Fake Media Generation and Detection 17
Fig. 4. All 88 papers are categorized based on the main methodological and focus are
of the research. Highlighted in color are the first five focus areas based in the higher
frequency—Security, Synthetic Media, Psychology, Legal Regulation, and Political are
the top five focus areas.
primarily concentrated on, followed by 44 sub-focused areas. This flow is visual-
ized using the alluvial diagram in Fig. 4. In the interest of space for this paper,
we highlight the top five focus areas of research.
As it appears, the highest interest of focus is drawn upon Security related
issues relating to the social implications of deepfakes. A number of researches re-
lated to security are foreseeing harms and threats to the society through “Review
of literature” [94, 110, 52, 95]. More security focus research is conducted based
on a “Design” of a blockchain-based framework for preventing fake news while
introducing various design issues [24]. At the same time, security focus research
has been visible in the research method of “Analysis by Example” where [28]
conduct a general analysis to understand the risks and hazards of the technolo-
gies used today and highlight the need for a wider application and enhancement
of Deepfake technology to fight Cybercrimes. Similarly, [86] analyses a wide
range of examples of deepfakes in the modern world and the Internet services
that generate them with a key focus on security. Their research also depicts a
clear sub focused area of Psychological Security as they try to understand
the threats deepfake causes to society and its impacts.
18 D. Gamage, et al.
The next highest focus area of the literature solves problems relating to
Synthetic Media. These are mostly considered as the deepfake in the mode
of videos. We observed that most researchers have used Synthetic media to
conduct “Experiments” and “Content Analysis.” For instance, [48] test whether
simple priming of deepfake information significantly increases users’ ability to
recognize Synthetic media, [47] examined the negative impact of deepfake video
and the protective effect of media literacy education; and [79] examined how
deepfake videos may distort the memory for public events, yet found it may not
always be more effective than simple misleading text. Other than these, [15]
used “Content Analysis” to analyze popular news and magazine to understand
the impact of Synthetic media. Interestingly, the article argues that if fake videos
are framed as a technical problem, solutions are likely to involve new systems
and tools, or if fake videos are framed as a social, cultural, or ethical problem,
solutions needed will be legal or behavioral ones. On the other hand, in this
article, the focus of synthetic media also expands to the sub-focus to examine
the societal Harm/Threats. Similarly, [46] empathizes that digital photos are
so easy to manipulate, yet deepfake videos are more important to understand
as deepfake synthetic media (video evidence) could be deliberately misleading
and not easy to be recognized as fake. In addition to content analysis, focus on
synthetic media narrowed the focus for a few commentary-based articles: one
examines deepfake video implications on Facebook [105], and two other articles
focus on examining deepfake videos challenges with a sub-focus on understanding
Future Challenges [53, 65].
The next highest set of research articles focused mainly on the areas of Psy-
chology, Legal Regulation, and Politics. Interestingly, all Psychological
focus research conducted as experiments except for one that center around the
Psychological impact of deepfake through a review of literature [41]. In experi-
ments, [121] explore the effect of credibility signals and how they perceived any
individual to share fake news [58] focus on understanding the vulnerability of
human judgments to the deepfake. [3] examines the social impact of deepfakes
using an online survey sample in the United States. This research investigates the
psychological aspects of the influence of deepfake while examining the concerns
of citizens regarding deepfakes, exposure to deepfakes, inadvertent sharing of
deepfakes, the cognitive ability of individuals, and social media news skepticism.
[25] provided psychological aspects of deepfakes by exploring factors impacting
the perceived responsibility of online platforms to regulate deepfakes and pro-
vide implications for users of social media, social media platforms, technology
developers, and broader society. The research focusing on Legal Regulation
extensively worked on deepfake pornography, discussing its ethical perspective,
consequences, and legal framework to take action (i.e, ‘[54,29, 34]. Few others
had a sub-focus on discussing the threats and harms [81], Terrorism [8] and spe-
cific to facial processing technologies [5]. The Political focus researches have
been extensively worked on election-related consequences of deepfakes and few
focused on the journalists’ discourse to shape political context [116], explored
A chapter for Frontiers in Fake Media Generation and Detection 19
the relationship between political and pornographic deep fakes [72] and discussed
the threat of deepfake online propaganda tools [87].
4.2 Study2-Q2: Distribution of the research
In previous sections, we partially stated the distributions of research methods
and focus areas by utilizing Fig. 3 and 4. Furthermore, we expanded the knowl-
edge of the landscape for deepfake research that concentrates on its societal
impacts by examining the yearly distribution of the relevant research. As de-
picted in Fig. 3, the yearly projection reflects a trend for studies that explore
the social implications by deepfake are emerging since 2019 and 2021 has the
highest number of such researches (42) even before the year 2021 ends.
We generated word clouds for each abstract and one common word cloud
combining all 88 abstracts to make sense of what we examined and to summarize
the analysis of the full text of the articles. The top word cloud in Fig. 5 generated
from an abstract which we categorized as Pornography [34] and it shows that
words are centered on pornography; The bottom shows the word cloud from all
abstracts which reflects deepfake as the central theme and yet highlights, other
focus areas we identified that greatly resonated in our categorizations.
Finally, to better understand the distribution of the authors of these papers,
we generated bipartite networks using the author list with the titles of the papers
they have written (Fig. 6). Nodes represent the authors (pink), papers (green),
and the edges point from the authors to the papers. It appears that researchers
who explore deepfake social implications are almost not connected to each other
as the clustering coefficient indicates 0.0 and nearly 30% of Papers written by
70% of authors and the highest number of relationships consisted one degree as
a single author has written the papers. Ranked by the degree of centrality (how
many authors wrote how many papers), the graph revealed the lowest degree
of centrality as 1 and the highest as 8. Filtering the network to reflect if there
are any 2 or more authors collaborated in writing these social research types,
we filtered the graph into 2 to 8 degree centrality. Interestingly, this resulted in
only two authors having 2 degrees relationship. In one instance, the same author
wrote two different papers while collaborating with multiple other authors [60];
in the other instance, the same author has written two papers without any author
collaborations [4, 2].
4.3 Study2-Q1: Types of conversations lead in Reddit communities
about deepfake
In this section, we provide the results of our empirical study of Reddit conver-
sation analysis. Overall, we examined the key topics resulting after conducting
topic modeling of conversations based on posts and comments on the platform.
Overall topics from posts and comments from all four years. Topic
modeling was conducted for posts and comments separately. The corpus had
20 D. Gamage, et al.
Fig. 5. Word clouds from abstracts identified as focusing Pornography (top) and in all
articles (bottom)
86,425 comments and 6,638 posts. The topic coherence graphs (Fig. 7) indicated
two dominant discussed topics from the posts and two topics from the comments.
The two topics generated out of posts had a coherence measurement of nearly
0.980 while the two topics in the comments had a coherence of 0.560.
From the posts, out of two topics, Topic 1 indicates that major discussions
are centered on videos of deepfakes, specifically related to the US President Don-
ald Trump. This indicates that one of the major posted topics in deepfakes is
political and related to deepfakes of Trump. Topic 2 indicates that the high-
est weighted coherent word is porn, followed by video and nude. This reflects
that many submissions to Reddit are focusing on pornographic videos, specifi-
cally igniting discussions of fake nudity and technologies of creating such videos.
A chapter for Frontiers in Fake Media Generation and Detection 21
Fig. 6. [Left] A bipartite graph was created using the source as the authors and targets
as the papers. [Right] The Bipartite graph filtered based on the degree centrality larger
than 2.
For the comments, there are two topics discussing the posts further. Based on
Topic 1 generated from comments, we found that many users were expressing
their concerns over the “Fakeness” and discussions were centered around how
to identify such videos of deepfakes. Based on Topic 2, we found a tendency
for people posting further links relating to posts and platforms taking actions
against deepfake such posts may have been deleted by the bots or moderators
closely monitoring the comments where users were requesting to re-post the link
from others. Figure 8 reflects the generated topics and from posts and comments
and the weight of the word distributions in each topic.
Next, we conducted topic modeling from the corpus of each year. A summary
of all years, comments and posts across the subreddits generated a number of
topics and the glimpses of the growth of conversations over time with wider
subreddit groups are described.
Conversations during 2018. The dataset from 2018 contained 269 posts and
3828 comments across 165 subreddits. We conducted a similar analysis for posts
and comments in 2018. Compared to other years, 2018 had the fewest conver-
sations and we believe that the reason behind this is the Reddit ban of the
subreddit ”s/deepfakes” and Reddit subsequently updated their site-wide rules
against involuntary pornography and sexual or suggestive content involving mi-
nors [16]. Examining the total number of posts from 2018, we found only one key
topic. This reflects that during that time, the conversation centered more around
22 D. Gamage, et al.
Fig. 7. Topic coherence kwhich indicates the optimal number of topics. There are four
dominant topics visible in the entire corpus—2 from posts and 2 from comments.
Fig. 8. There are two dominant topics from each corpus of ”Posts” and ”Comments”
visible. The First 2 topics are generated from entire posts and the second set of two
topics generated from comments of all corpus.
the deepfake technology, the news it had created based on the ban, pornography,
and tech news about the new regulations.
Figure 9 displays the key topics from comments and posts that resulted
from our scripts. Compared to the posts in 2018, there was much engagement in
distributed comments. We found 15 Topics generated from comments. According
to the topic distribution, the top 20 keywords from the topics and looking at
the raw data related to topics, most importantly, we found that Topic 1 was
about community discussions regarding self-cognition, the morality of deepfakes
or proactive discussion—conversations on the right thing to do. Topics 2 and
8 are related to bots, shared links, and moderation by Reddit. Topics 4 and 6
collectively reflect discussions on the improvements to deepfakes as the ability
to deceive more people increased and, the possibility of wider attention and to
be in the news. Topics 5 and 7 together reflect conversations on technologies for
deepfake, faceswap, and how to create quality deepfake videos or images. Topics
8 and 12 are more towards platform moderation’s, new rules, Reddit bans and
related media on the announcement. However, topic 13 stands out on its own—it
brought key conversations on the legislative actions on revenge porn, celebrity
porn, and discussions about consents of the victims. Topics 3 and 10 did not
provide meaningful conversations. Topic 9, 11, 14, and 15 together provided
deep concerns on the future of technology.
A chapter for Frontiers in Fake Media Generation and Detection 23
Fig. 9. The upper Topic 01 indicates the topic generated from posts in 2018 and the
bottom set of topics indicates the topics generated by comments from 2018 Reddit
data
Conversations during 2019. Compared to 2018, 2019 had an increased num-
ber of conversations (22,773) and posts (538). Based on the topic distribution of
the posts, we received 19 topics. Post represented more targeted conversations
than in 2018. Interestingly, there were fewer posts on pornography as Topic 3 is
the only topic posted on that subject. Topics 1, 2, and 19 described posts leaning
towards concerns about deepfake videos, especially the expert’s view on threats
for the elections, the scariness of the future behaviours of tech companies to the
society. Topics 4, 5, 7, 10, 14, 15, and 16 collectively provide posts about the
deepfake videos on political leaders such as Donald Trump, Vladimir Putin, and
Barack Obama and celebrities deepfake such as Tom Cruise, Kenue, Jim, etc.
However, it is important to quote that Topics 8, 9, 11, and 12 are centered upon
the tech giants such as Facebook, Google, and Zao — the Chinese FaceSwap
app — who needed to act immediately to mitigate the risk posed by deepfakes.
Topics 13, 16, 17, and 18 more on the improvements of the technology which
needs assistance to identify the deepfakes and the creation of realistic deepfakes.
Examining the comments in 2019, we found 18 topics generated through topic
modeling. Topic 1 concentrated on the conversation of the deepfake pornography
video ban in Reddit platform, the technology of creating it, and the legality of
these actions. Unlike posts, we found comments had similar patterns in discussion
in Topics 2, 4, 6, 7, 8, 11, 12, 13, 16, 17, and 18. These discussions are more
related to the deepfake video or images links shared in the posts and users
are more discussing how real like of the generated deepfakes. Topics 3 and 5
are discussions on news-related deepfakes and the technology behind deepfakes.
Topics 9 and 10 are about editing and creating deepfake videos, Topic 14 is about
moderating subreddits, identifying and detecting deepfake posts and importantly
the actions that can be taken to these posts. Topic 15 is about the conversations
on how people believe deepfakes and the concerns they raise in society. The
topics are depicted in Fig. 10.
Conversations during 2020. In 2020, we found that deepfake related posts
almost doubled compared to 2019. At the same time, the subreddits that were
24 D. Gamage, et al.
Fig. 10. The first set of topics are generated from posts in 2019 and the bottom set of
topics indicates the topics generated by comments from 2019 Reddit data
discussing deepfakes doubled as well. We found 19 topics from the corpus of
2020 posts (Fig. 11 top). Topics 1 and 2 were posts about Donald Trump and
Joe Biden’s deepfake videos. Topics 3 and 17 focused on pornography, nudity,
and related news where they gained conversation than 2019. Topics 4 and 7
were related to posts about Japanese deepfake memes and videos. This is the
first emerging discussion on deepfake arousing from Asia or a country other
than the US. Topic 5 centered around deepfake related political misinformation
and how Facebook was taking actions against deepfake content. This was the
first occurrence of the deepfake mitigation discussions by the platforms such
as Facebook. Topic 6 was related to requests related to deepfake videos. We
could not find any interpretation for Topic 8 as it did not appear a meaningful
discussion. Topics 9, 10, 11, 12, 13, and 18 from the posts had a similar pattern.
These are about the deepfake video of Queen Elizabeth’s Christmas message,
Tom Cruise deepfake videos, Elon Musk, KSI (a famous YouTube personality)
or other famous personalities memes, videos, and music. On Topics 14, 15, and
16 we found posts relating to creating deepfakes, tutorials, types of tools that can
be used to make realistic photos and videos. Finally, Topic 19 was specifically
about the detection of AI technology for voice.
Examining the comments of the 2020 Reddit corpus, we found 15 dominant
topics. Compared to the number of comments in 2019, it did not double the
growth (Fig. 11 bottom). However, collectively the posts-to-comments ratio has
A chapter for Frontiers in Fake Media Generation and Detection 25
increased, which reflects the tendency to engage with deepfake posts. Similarly,
as with the post, we found patterns in the 15 topics for comments. Topics 1, 3,
6, and 14 reflected the discussions about how people get deceived by deepfakes,
how deepfakes are similar to real-like videos, and how technology is better to a
level where no one ever can tell if its deepfake without using technology. Top-
ics 2 and 5 are related mostly to information requests from the users, and the
moderation’s conduct in Reddit. Topics 4, 11, 12, and 13 were discussions based
on detecting deepfake videos, which is an important discussion critically exam-
ining constructive evidence to the fake and the original. On Topics 7, 8, and 15,
we could not interpret meaningful discussion. In particular to Topics 9 and 10,
we interpret the conversation as users being criticized for their deepfake video
attempts, judging its degree of realism or users providing feedback to improve
this.
Fig. 11. The first set of Topics indicates the topics generated from posts in 2020 and the
bottom set of topics indicates the topics generated from comments from 2020 Reddit
data
Conversations during 2021. Our data cover from 1st January to 21st August
in 2021, and thus we had fewer posts and comments compared to 2020. However,
we see the trend to be increasing. Overall, discussions in 2021 are far more
distributed in 1270 subreddits, whereas in 2020 only 1226 were found. Analysing
the topics generated by the post, we found 10 dominant topics (See Fig. 12 for the
26 D. Gamage, et al.
first set of topics). Topics 1, 3, 5, and 9 had similar semantics. They collectively
posted about nudity, sex, pornography, and arguably related websites where such
images and videos could be purchased. We also saw India in these discussions
with celebrity pornography. Compared to 2020, this is an increase of discussions
and they were expanded to new geographic locations. Topics 2 and 6 were similar
to 2020 viral videos of Trump and Tom Cruise deepfake related posts. Topics
4 and 10 were mostly related to creating deepfake using technology as well as
building tools for detection of these deepfakes. Topic 7 was omitted as we could
not trace a meaningful interpretation.
Examining the comments during 2021, there was a slight reduction in the
number of comments as our data were collected only up to August, 2021. Adding
four more months, we believe conversations about deepfakes are growing with
much added attention. The 2021 corpus provided 18 topics (Fig. 12 bottom set of
topics). However, we found that these discussions spread with similar semantics.
We categorize Topics 1 and 18 as similar since both topics provide a great insight
to the conversation about the possible occupations of deepfakes. We believe this
may have resulted due to the latest hiring of a YouTuber by a film production
company [100]. Topics 2, 4, and 13 had similar discussions as previous years on
regulating the subreddits, commenting on and removal of posts and comments.
Topics 3, 6, and 14 depict the concerns of deepfakes as the technology that can
bring distress to political figures, pornography, and technical concerns on how AI
can be used to create these images and videos. Topics 7, 8, 9, 10, 12, and 17 were
much related to the direct conversations based on the posts they were meant to
comment on. This is a pattern that we came across in the 2020 comments as
well, where users are providing critical reviews for their deepfake creations or the
post they submitted, not necessarily created by themselves. Similarly, Topics 5,
11, and 15 also provided insights on concerns in deepfake, and how it appears
to be truly original.
Change of conversations over time. To observe the frequency of post and
comment behaviour over each year, we projected a line graph in Fig. 13. Visibly,
2021 has the highest of all until early August, while 2018 has the lowest number
of each month. Table 2 summarizes the entire line of years with posts, comments,
number of subreddits, and the number of topics generated from each year, posts,
and comments. Although there were all together 49 topics from posts and 66
topics from comments generated over the years, we grouped together topics with
similar patterns. We found unique discussions among posts and comments over
the years. A summary of the projection is depicted in Fig. 14. Unique discussions
are filled with color. In 2018, we found 2 unique discussions in the comments—
1) legislative actions against deepfake pornography and 2) self-cognition, the
morality of deepfake. This is the first year that deepfake pornography videos
were banned on Reddit. Subsequently, in 2019, we found unique discussions of
news relating to deepfake technology. In 2020, we found comments focused on
deepfake AI voice detection and constructive discussions on how to detect what
are deepfakes and what are original videos. We believe that the discussions on
A chapter for Frontiers in Fake Media Generation and Detection 27
Fig. 12. Key topics generated from posts (top) and comments (bottom) from 2021
Reddit data
Fig. 13. The temporal distribution of comments and posts over the years 2018, 2019,
2020, and 2021.
AI voice detection may have aroused because in 2020, an AI voice scam was
found manipulating the CEO of a company [114]. In 2021, we found a unique
set of comments about job hiring. We believe this was due to the news about a
film production company hiring a YouTuber who had been contributing realistic
deepfake videos [13].
Similar to the unique set of comments, we found unique posts from 2019
where users posted more on how platforms need to act to mitigate the misinfor-
mation from deepfakes. In 2020, unique posts were found about users discussing
the creation of deepfakes. This went beyond the previous discussions where users
not just discussed creating deepfakes but also its nuances, including formal tu-
torials on how to create deepfakes. Users were sharing valuable resources and
28 D. Gamage, et al.
Fig. 14. Comparison of topics generated by posts and comments over the years. Green
outlines are the posts and orange outlines are comments. Comments and posts unique
to each year are filled with color.
knowledge in building these videos and images with easily accessible tools and
technologies. At the same time, 2020 uniquely brought Japan into the discussion
based on certain memes created using deepfakes. This was mainly due to the
hype brought by the “Baka Mitai” meme that features a Japanese song from the
popular video [42]. Similarly in 2021, we found unique posts centered around
pornography and celebrity deepfakes from an area of contributions, “India”.
Furthermore, we found a continued discussions about the concerns arising
from deepfakes in each year. However, apart from concerns, specifically during
2020 and 2021, we found that Reddit users are using the platform as a testbed
to review their created deepfake videos. This takes place when they post their
deepfake video,s and many other users provide critical feedback on how to im-
prove the quality of the deepfake to reflect a more realistic effect. We also found
the continued interest in discussions on how to create deepfakes by tools and
methods where anyone can easily create them.
4.4 Study2-Q2: The implications visible in these conversations
In this section, we present our qualitative analysis using open coding where we
manually code each topic and its possible implications. A code represented one
or two sentences describing the actions of the topic. The coders first observed
each topic and sample comments related to the topic. Then, they wrote a short
A chapter for Frontiers in Fake Media Generation and Detection 29
Table 2. Summary of Posts, Comments, and topics distributed across a number of
subreddits across the four years.
Year
Number
of
Posts
Number
of
Comments
Number
of
subreddits
Number of Topics
in
posts and comments
2018 269 3828 165 1, 15
2019 1472 22773 538 19, 18
2020 2886 34071 1226 19, 15
2021 2280 25651 1270 10, 18
description of possible outcomes of such a discussion of a topic. In this case, we
conceptualized each topic as an action where there is a possible reaction attached
to it. All together we had 135 topics—49 topics generated from posts and 66
topics generated from comments. We further obtained 10 sample comments for
each of these topics where coders can make a better sense of the context of
the topic. After both coders derived implications observing and reviewing all
135 topics and sample conversations, all researchers convened to interrogate and
debate on the results. This phase was utilized to finalize the key implications
based on the conversations and categorized the similar implications into key
themes. Each coder had 59 and 73 possible implications noted independently.
Based on the similarities of the outcomes and actions, we categorised these codes
into 10 possible implications from a macroscopic viewpoint as described below.
1. Creating deepfake archival — Some deepfake videos and images posts
are banned, or some users often miss the content. Deepfake communities often
reserve their spaces and share with each other when someone needs to revisit
the content. Sometimes these videos may have been banned from the platform
but can be retrieved as other forms in archival channels.
2. Incubating deepfake learner space — Reddit community space for deep-
fakes provides an incubator for developing deepfake generators/creators. The
community often provides critical feedback on how to improve the models, tools
to use or often shares tutorials and knowledge on how to do it effectively.
3. Collective space to question and raise concerns—Over the four years, Red-
dit has been a space for users to raise their concerns. Although the Reddit plat-
form depicted a pro deepfake space as many were created and even praised in
this space, users have also used Reddit as a platform to raise their concerns
on the technology in general, specifically concerns about the deepfake content
created and shared in the space.
4. Create your own pornography — Deepfake pornography is banned on many
platforms. Reddit communities share all knowledge, tools, components (such as
databases to purchase images) and are continuously used as a testbed for any
novice to experiment and create videos. It also provides ample inspiration for
deepfake sources and target combinations, such as the recent introduction of
deepfake Indian celebrity pornography, or Japanese cartoon characters.
5. Inspirations for deepfake movies — Deepfake conversations about movies
have mostly discussed the opportunities posed in the industry of entertainment.
30 D. Gamage, et al.
This suggests, with or without real actors, and voices, or new characters arti-
ficially created for movies using deepfakes. As a sidebar to the opportunities,
conversations also focused on the consent and legality of future AI-created ac-
tors.
6. Mixing cultures with deepfakes — The technology of deepfake provides
ample opportunities to mix sources and targets with a wide variety of samples. In
the real world, mixing the DNA of real humans is prohibited and costly. However,
with deepfake, mixing different entities is providing possible opportunities for
entertainment. Since these can be created easily with many tools available and
as they can be tested easily by reviewers and feedback can be obtained rapidly,
there is a market emerging for it. One such example is the deepfake clip using
the Hollywood film star Keanu Reevese in a Bollywood movie [122].
7. Deepfake policies are disproportionately controlled by platforms — Reddit
banned deepfake pornography in 2018, while Facebook banned deepfakes in 2020.
At the same time, each platform regulates the content in its own way, and
removing and banning synthetic content has brought controversy concerning the
actions within platforms.
8. Fooled by AI-generated news — Reddit communities feature extensive
conversations about the concerns arising from deepfakes and one of the ma-
jor emphases carried on involves AI-generated news. Although AI helps people
detect misinformation, ironically, it has also been used to produce misinforma-
tion. Transformers, like BERT, and GPT use NLP methods to understand the
text and produce translations, summaries, and interpretations. The capabili-
ties of the technology have also been extended to generate real-like news (e.g,
www.NotRealNews.net)
9. Technology and social media companies can influence elections using deep-
fakes — Reddit had many discussions based on the deepfake videos involving
key political figures such as Obama, Trump, and Putin. On top of the fact that
videos are deepfakes, the number of likes and trends they get on the platforms
brought concerns in the discussions. These viral videos may impact the elections
as the public has often been deceived by deepfakes as discussions have explained
in Reddit.
10. Creating a deepfake marketplace — Deepfake-related job hiring has been
a long-discussed topic on Reddit. A popular deepfake YouTuber who goes by
the name “Shamook” has been hired by Lucasfilm corporation. Shamook’s most
viral video is a deepfake that improves the visual effects (VFX) used in “The
Mandalorian” Season 2 finale to de-age Mark Hamill’s Luke Skywalker. The
company says they have been investing in both machine learning and A.I. as a
means to produce compelling visual effects work and they found it to be terrific to
see momentum building in this space as the technology advances [100]. Similarly,
there were many other discussions related to new jobs such as deepfake video
creation as a service, selling deepfake videos to a value, and to some extent, it
was visible that some websites were selling images to be created as deepfakes.
A chapter for Frontiers in Fake Media Generation and Detection 31
5 Discussion
This chapter covers a wider landscape of literature relating to the societal im-
plications of deepfakes and provides insights drawn from analysing Reddit con-
versations relating to deepfakes and possible implications. From the systematic
literature review, it is evident that deepfake research is sparse to understand
social impacts. Deepfake technology is an evolving area with the advancement
of computer vision, ML and AI techniques. As our results depicted, the highest
number of research on deepfakes were focusing on the technical aspects and only
88 research studies depicted research related to societal implications of deepfakes.
Examining the yearly distribution of these 88 research works, we revealed that
the highest number of research related to deepfake’s social implications came up
within the 8 months of 2021. We believe that the trends towards understand-
ing deepfake dynamics from multiple viewpoints are to increase more in the
upcoming years. However, we found that empirical research towards deepfakes
and their dynamic changes in human behaviours need more attention. Out of
the few studies conducted on human perception already provided evidence that
deepfakes are deceptive and it is important to educate users about identifying
such in early stages and mitigate serious damage upfront. Tools should support
and platforms need to take actions while designing and developing interventions
to reduce the spread and damage to the society. On the other hand, it is equally
important to understand social conversations shaping the use of deepfakes and
predict possible implications and mitigate any negatives beforehand.
Our second study provides a great example of such an examination — under-
standing implications through developing conversations. Specifically, we revealed
many developing key topics about deepfake within the Reddit community. Red-
dit users are anonymous to each other, and we believe the discussion discovered
may not be fully visible in any other social media platforms such as Twitter or
Facebook due to the anonymity nature where they could discuss intimate details.
While we observed an incremental distribution of posts and comments on deep-
fake each year, we worry that many discussions lead to pornography, politics,
and the creation of deepfakes itself. Although it may not reflect the total pop-
ulation or can generalize the main findings obtained from Reddit conversations
to the wide world, our data sample represents a considerable amount to foresee
the implications that may arise from conversations of deepfake in the future.
5.1 Are deepfakes concerning?
Analyzing the conversations over the years, we found topics in which users con-
tinuously discuss the concerns that can arise from deepfakes. Specifically, when
there was a Reddit ban on pornography in 2018, we found unique discussions
driven towards questioning the morality of doing deepfakes. This concern was
seen in 2019. However, it was not much visible in a key topic in 2020 but appeared
again in 2021. An important finding from these discussions centered around ”con-
cerns” was that many users in Reddit depicted their fears towards “Political dis-
tress”, “Election propaganda manipulation” and “Pornography”. These are well
32 D. Gamage, et al.
anticipated social impacts as researchers revealed [15]. We could not find strong
discussion topics about the risk to individuals, personal abuse, reputation deci-
mation as found in recent research which analyzed popular news articles about
deepfakes [15]. However, throughout the years (2018 to 2021), we found strong
discussions about Reddit moderation on deepfakes, bots moderating deepfakes
and this could be one reason that we did not encounter such discussions in the
community.
Apart from users fearing the improvements of this technology in 2018 and
2020, we observed that the Reddit community is exploring the potentials of deep-
fakes in their discussions. The discussions on how to create and edit deepfakes
using tools and technology seem started to trend as early as 2018. This trend
has even got into unique conversations where they discuss the tutorial of how to
create deepfakes. Beyond that, we derived this as an implication where Reddit
is an incubator for nourishing deepfake creators. This poses a series of concerns,
although not just in Reddit communities but also linking to Udemy tutorials
(https://www.udemy.com/course/deepfakes/) on creating deepfakes with prac-
tical hands-on applications shared by users. Though the instructor warns the
participants not to do any harm using deepfakes, no prevention mechanism is
taken. Each year we found trends in topics where users discuss Reddit bans,
moderation of deepfakes, bots taking off some posts yet it appears that users
are actively participating in the discussions centered on criticism of the created
posts and context of the deepfake. This enables many positive environments for
potential deepfakes yet less attempts were taken to mitigate any harmful impact
it might create.
At the same time, we see discussions on how the platform needs to take action
against deepfakes, yet policies seem to be mostly limited to curbing pornogra-
phy [72]. The areas of security and politics and entertainment are under severe
threats which need platform policies and legal frameworks. Reddit discussions
turn on such topics occasionally, yet, since the platforms are incentive-driven
for trends, likes, and sharing culture, we saw the continuing interest to create
trending videos using deepfakes.
Through topical explorations, we found deepfake discussions lead to a mar-
ketplace that monetizes deepfakes. Although monetizing deepfake pornography
has been around since 2018 [115], our research found new forms of a market-
place where users showed their interest in creating deepfakes as a service. These
were custom-made deepfakes with a specific purpose such as entertainment clips.
Users publicly discussed pricing and exchanged their information in the discus-
sion. At the same time, there were many discussions about the entertainment
film industry preparing to invest in the AI presence or voice of certain actors
and actresses. At the same time, it is no secret anymore that there are a wide
range of deepfake service platforms — a type of website that provides a user in-
terface to help users accelerate the process of creating deepfakes. They typically
require users to upload training data in the form of media objects pertaining
to the subjects they intend to deepfake. They also provide tools for receiving
the AI-generated deepfake media object once it has been created. These service
A chapter for Frontiers in Fake Media Generation and Detection 33
platforms may handle all of these back-end functions in an entirely automated
fashion, or through the partial manual processing by the service’s employees or
contractors. These websites are functioning for a fee in providing such deepfake
services.
Collectively, our derived implications of the deepfake discussions lead towards
more concerting directions. However, we must emphasize that deepfake technol-
ogy is a double-edged sword. It has many positive uses to the society and must
not work towards eliminating them, but work on mitigating the darker risks that
come with the technology’s adoption. That will be fiendishly difficult balance for
society to make, especially as deepfake tools evolve to eliminate any trace of how
they have altered the video, images, audio, and other media at the center of our
lives in the 21st century. Therefore, it is time that AI-generated fake content
will impact deeply into our lives. As the researchers pointed out “If fake videos
are framed as a technical problem, solutions will likely involve new systems. If
fake videos are framed as an ethical problem, solutions needed will be legal or
behavioral ones” [15], we invite researchers to view this as a societal problem.
5.2 Future directions
According to studies 1 and 2, there is a need for us to work with multiple disci-
pline professionals to understand possible implications of deepfakes and derive
plausible preventive actions or mitigation to undermine the damage that these
can cause. Specifically, based on our study 2, the topics discussed within the
Reddit users, we present several proposals where researchers may continue to
contribute.
1. Precautions for using deepfake tools
In the deepfake discussions, we witnessed excessive interest in editing and
using deepfake tools to create deepfakes by anyone. With the democratiza-
tion of AI, tools are everywhere accessible to anyone who does not need any
domain knowledge. It is vital that researchers find novel methods to be able
to distinguish real vs. fake. One way is looking at the tools and embedding
precautions for users. “Markpainting” is one the novel method which can be
used as a manipulation alarm that becomes visible in the event of inpaint-
ing [57]. Similarly, regulation of editing tools may provide an optional area
identifying the roots of its creations.
2. Deepfake detection training and education
Deepfake tutorials were in the discussions where community-supported to
edit with the best quality deepfake, however, it is equally important to pro-
vide education and knowledge on deepfakes and detecting these videos. One
effective example presented in CHI 2021 where authors demonstrated a care-
fully crafted training—based on how the human eye deceive deepfake [108].
It is the first attempt of HCI researchers taking from users’ point of view.
However, we seek more solutions on educating and psychological readiness
for defeating deepfakes.
34 D. Gamage, et al.
3. Social media platforms strict policies to mitigate deepfakes
Content sharing platforms have a prominent role against deepfake. Com-
munity users in Reddit constantly echoed this proposal and this not just
by policies but also design decisions to mitigate deepfakes in platforms. For
example, platforms have taken many actions against misinformation such
as nudging the user interfaces design [63](Designing subtle interventions to
guide the user with choices without restricting them). However, deepfakes
should be considered as a new domain which users need to be informed before
they take any actions.
4. AI tools and research regulate under strict ethical guidelines
The majority of AI tools are implications of the advancement of AI and deep
learning technology research. Such research poses major societal risks, yet
many of the institution’s Ethical Review Boards (ERB) or similar entities
do not require any prior approvals to conduct AI research. Mainly because
such research usually do not study human subjects directly. However, we
emphasize the necessity for regulating such research and industry applica-
tions imposing preventive actions. One such inspiration has been already
prototype by introducing Ethics Society Review (ESR) board for ML/AI
projects [11], however, we emphasize the need of popularizing this kind of
review mechanism in both industry and academia for the betterment of the
society.
5. Mechanisms to deepfake activities to reporting, and legal frame-
works to supporting actions against deepfakes
Currently, many platforms lack reporting mechanisms on deepfake activi-
ties [27]. Although Reddit platforms use bots, sometimes moderators’ sup-
port is highly required to understand how harmful is the content. In our
results, we found moderation in platforms in topical discussions through all
four years. Yet, we witnessed possibly concerning implications out of these
discussions. Therefore, the reporting mechanism should be established specif-
ically for deepfakes within a legal framework that needs beyond reporting
and further steps need to be articulated for categories of deepfake activities
in the interfaces.
5.3 Limitations
Although this research found many gaps in deepfake research and insights from
deepfake conversation analysis, this research falls short on a few levels. Our
literature survey was conducted on the research published only in widely used
databases. Many other implications may have been discussed but may have not
been published in peer-reviewed articles. At the same time, our methods hinged
on the NLP process and the algorithms we used in topic modeling. Our key
results — deriving topics based on conversations and then constantly using these
topics to derive implications may bias on algorithmic decisions. More towards
it, what the algorithm provided us are the set of prioritized words based on the
similarity in a set of topics. Although manually it is impossible to sort a large
sample of texts and categorise them into topics, when identifying concepts from a
A chapter for Frontiers in Fake Media Generation and Detection 35
collective of topics, many have to rely on the certain weight distributed in words.
Our algorithm may have concealed emerging important discussions which may
contribute to topical discussions, yet due to less frequency of discussion, it may
have not been picked by the algorithm as a key topic. In this way, we may have
missed any prudent discussion or possible implications surrounded by it.
Our qualitative codes were based on 2 researchers and we did not concentrate
on inter-rater reliability, but collectively we expanded and reduced some codes
based on collective agreement. Our results are solely based on the deepfake
conversations that occurred with Reddit platform, hence the implications and
propose mitigation actions are narrowed from what we observe during these
discussions, where there could be prominent implications that may not appear
in these discussions, therefore there can be a myriad of other possible future
directions.
6 Conclusions
This chapter reflects a comprehensive review of deepfake research related to
social implications as the primary focus opposed to the reviews to the technology
itself. In conclusion, we raised our concern due to the lack of societal implications
side of research in deepfakes and because the majority of the research is yet
literature reviews. Out of all 88 papers, the majority focus on their research
in security and discuss the possible harms and threats to the social ecosystem.
Much debated issues such as ethical implications to deepfakes, regulatory or
legal solutions other than pornography, such as making awareness or educative
activism to other types of harm especially cybercrime and terrorism, are much
sparse in the landscape.
On the other hand, we explored the deepfake conversations on Reddit over
the last four years using a mixed-method approach. We first used topic model-
ing in NLP to understand the conversation themed by the topics and compared
topics in each year from 2018 to 2021. We then used qualitative coding to map
implications from the conversations. Based on the overall conversations out of all
4 years, we found pornography, political discourse, concerns on “fakeness” and
conversations on Reddit moderation were dominant. However, we found some
topics are unique to each year (i.e, self-cognition and questioning the moral-
ity of deepfakes in 2018, constrictive discussions on how to detect deepfake in
2020, etc) while some conversations have been continued for years (i.e, concerns
arise from deepfakes, conversations about deepfakes been moderated by bots and
moderators).
We derived 10 implications based on the conversations and our conclusion
remarks included that conversations of deepfakes provided opportunities and
a concerning future. Finally, we proposed five directions where deepfakes soci-
etal harm can be mitigated in the long run. However, we stress that mitigating
deepfake concerns will not be one-shot; achieving this goal requires multidisci-
plinary collaborations. Our results suggest that the social science of deepfakes is
emerging, but such research has been conducted independently thus far. Given
36 D. Gamage, et al.
that deepfakes and related AI technologies are being weaponized, the social im-
plications of deepfakes should be investigated further with an interdisciplinary
effort.
Acknowledgments
This work is generously supported by JST, CREST Grant Number JP-
MJCR20D3, Japan.
References
1. Shruti Agarwal and Hany Farid. Detecting deep-fake videos from aural and oral
dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 981–989, 2021.
2. Saifuddin Ahmed. Fooled by the fakes: Cognitive differences in perceived claim
accuracy and sharing intention of non-political deepfakes. Personality and Indi-
vidual Differences, 182:111074, 2021.
3. Saifuddin Ahmed. Navigating the maze: Deepfakes, cognitive ability, and social
media news skepticism. new media & society, page 14614448211019198, 2021.
4. Saifuddin Ahmed. Who inadvertently shares deepfakes? analyzing the role of
political interest, cognitive ability, and social network size. Telematics and Infor-
matics, 57:101508, 2021.
5. Roman Amelin and Sergey Channov. On the legal issues of face processing tech-
nologies. In International Conference on Digital Transformation and Global So-
ciety, pages 223–236. Springer, 2020.
6. Oxford Analytica. ’deepfakes’ could irreparably damage public trust. Emerald
Expert Briefings, (oxan-db), 2019.
7. Oxford Analytica. ’deepfakes’ could irreparably damage public trust. Expert
Briefings, 2019.
8. Arije Antinori. Terrorism and deepfake: From hybrid warfare to post-truth war-
fare in a hybrid world. In ECIAIR 2019 European Conference on the Impact of
Artificial Intelligence and Robotics, page 23. Academic Conferences and publish-
ing limited, 2019.
9. Sairam Balani and Munmun De Choudhury. Detecting and characterizing mental
health related self-disclosure in social media. In Proceedings of the 33rd Annual
ACM Conference Extended Abstracts on Human Factors in Computing Systems,
pages 1373–1378, 2015.
10. Irakli Beridze and James Butcher. When seeing is no longer believing. Nature
Machine Intelligence, 1(8):332–334, 2019.
11. Michael S Bernstein, Margaret Levi, David Magnus, Betsy Rajala, Debra Satz,
and Charla Waeiss. Esr: Ethics and society review of artificial intelligence re-
search. arXiv preprint arXiv:2106.11521, 2021.
12. Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with
Python: analyzing text with the natural language toolkit. ” O’Reilly Media, Inc.”,
2009.
13. Jennifer Bisset. Lucasfilm hires deepfake youtuber who fixed luke skywalker in
the mandalorian, 2021.
A chapter for Frontiers in Fake Media Generation and Detection 37
14. Christoph Bregler, Michele Covell, and Malcolm Slaney. Video rewrite: Driving
visual speech with audio. In Proceedings of the 24th Annual Conference on Com-
puter Graphics and Interactive Techniques, SIGGRAPH ’97, page 353–360, USA,
1997. ACM Press/Addison-Wesley Publishing Co.
15. Catherine Francis Brooks. Popular discourse around deepfakes and the interdis-
ciplinary challenge of fake video distribution. Cyberpsychology, Behavior, and
Social Networking, 24(3):159–163, 2021.
16. Jacquelyn Burkell and Chandell Gosse. Nothing new here: Emphasizing the social
and cultural context of deepfakes. First Monday, 2019.
17. Duygu Cakir and ¨
Ozge Y¨ucel Kasap. Audio to video: Generating a talking fake
agent. In International Online Conference on Intelligent Decision Science, pages
212–227. Springer, 2020.
18. Roberto Caldelli, Leonardo Galteri, Irene Amerini, and Alberto Del Bimbo. Op-
tical flow based cnn for detection of unlearnt deepfake manipulations. Pattern
Recognition Letters, 146:31–37, 2021.
19. M Caldwell, JTA Andrews, T Tanay, and LD Griffin. Ai-enabled future crime.
Crime Science, 9(1):1–13, 2020.
20. Kevin Matthe Caramancion. The demographic profile most at risk of being dis-
informed. In 2021 IEEE International IOT, Electronics and Mechatronics Con-
ference (IEMTRONICS), pages 1–7. IEEE, 2021.
21. Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. Everybody
dance now. In Proceedings of the IEEE/CVF International Conference on Com-
puter Vision, pages 5933–5942, 2019.
22. Eshwar Chandrasekharan, Mattia Samory, Shagun Jhaver, Hunter Charvat, Amy
Bruckman, Cliff Lampe, Jacob Eisenstein, and Eric Gilbert. The internet’s hidden
rules: An empirical study of reddit norm violations at micro, meso, and macro
scales. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW):1–25,
2018.
23. Weiling Chen, Chenyan Yang, Gibson Cheng, Yan Zhang, Chai Kiat Yeo,
Chiew Tong Lau, and Bu Sung Lee. Exploiting behavioral differences to detect
fake ne. In 2018 9th IEEE Annual Ubiquitous Computing, Electronics & Mobile
Communication Conference (UEMCON), pages 879–884. IEEE, 2018.
24. Hongmei Chi, Udochi Maduakor, Richard Alo, and Eleason Williams. Integrating
deepfake detection into cybersecurity curriculum. In Proceedings of the Future
Technologies Conference, pages 588–598. Springer, 2020.
25. Justin D Cochran and Stuart A Napshin. Deepfakes: awareness, concerns,
and platform accountability. Cyberpsychology, Behavior, and Social Networking,
24(3):164–172, 2021.
26. Jes´us P´erez Dasilva, Koldobika Meso Ayerdi, Terese Mendiguren Galdospin, et al.
Deepfakes on twitter: Which actors control their spread? Media and Communi-
cation, 9(1):301–312, 2021.
27. Antonella De Angeli, Mattia Falduti, Maria Menendez Blanco, and Sergio Tes-
saris. Reporting revenge porn: a preliminary expert analysis. In CHItaly 2021:
14th Biannual Conference of the Italian SIGCHI Chapter, pages 1–5, 2021.
28. Viktoria Degtereva, Svetlana Gladkova, Olga Makarova, and Eduard Melkos-
tupov. Forming a mechanism for preventing the violations in cyberspace at
the time of digitalization: Common cyber threats and ways to escape them. In
Proceedings of the International Scientific Conference-Digital Transformation on
Manufacturing, Infrastructure and Service, pages 1–6, 2020.
29. Rebecca A Delfino. Pornographic deepfakes: The case for federal criminalization
of revenge porn’s next tragic act. Actual Probs. Econ. & L., page 105, 2020.
38 D. Gamage, et al.
30. Tom Dobber, Nadia Metoui, Damian Trilling, Natali Helberger, and Claes
de Vreese. Do (microtargeted) deepfakes have real effects on political attitudes?
The International Journal of Press/Politics, 26(1):69–91, 2021.
31. Simone Eelmaa. Sexualization of children in deepfakes and hentai: Examining
reddit user views. 2021.
32. Don Fallis. The epistemic threat of deepfakes. Philosophy & Technology, pages
1–21, 2020.
33. Luciano Floridi. Artificial intelligence, deepfakes and a future of ectypes. Philos-
ophy & Technology, 31(3):317–321, 2018.
34. Anne Pechenik Gieseke. ” the new weapon of choice”: Law’s current inability to
properly address deepfake pornography. Vand. L. Rev., 73:1479, 2020.
35. Alexander Godulla, Christian P Hoffmann, and Daniel Seibert. Dealing with
deepfakes–an interdisciplinary examination of the state of research and implica-
tions for communication studies. SCM Studies in Communication and Media,
10(1):72–96, 2021.
36. Chandell Gosse and Jacquelyn Burkell. Politics and porn: how news media char-
acterizes problems presented by deepfakes. Critical Studies in Media Communi-
cation, 37(5):497–511, 2020.
37. Samuel Greengard. Will deepfakes do deep damage? Communications of the
ACM, 63(1):17–19, 2019.
38. Shane Greenstein. The economics of confrontational conversation. IEEE Micro,
41(2):86–88, 2021.
39. The Guardian. Mother charged with deepfake plot against daughter’s cheerleading
rivals, March 2021.
40. Luca Guarnera, Oliver Giudice, and Sebastiano Battiato. Deepfake detection by
analyzing convolutional traces. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition Workshops, pages 666–667, 2020.
41. Jeffrey T Hancock and Jeremy N Bailenson. The social impact of deepfakes.
Cyberpsychology, Behavior, and Social Networking, 24(3), 2021.
42. Karen Hao. Memers are making deepfakes, and things are getting weird, 2021.
43. Craig A Harper, Dean Fido, and Dominic Petronzi. Delineating non-consensual
sexual image offending: Towards an empirical approach. Aggression and violent
behavior, page 101547, 2021.
44. Douglas Harris. Deepfakes: False pornography is here and the law cannot protect
you. Duke L. & Tech. Rev., 17:99, 2018.
45. Susan Hazan. Deep fake and cultural truth-custodians of cultural heritage in the
age of a digital reproduction. In International Conference on Human-Computer
Interaction, pages 65–80. Springer, 2020.
46. Mark K Hinders and Spencer L Kirn. Cranks and charlatans and deepfakes. In
Intelligent Feature Selection for Machine Learning Using the Dynamic Wavelet
Fingerprint, pages 297–346. Springer, 2020.
47. Yoori Hwang, Ji Youn Ryu, and Se-Hoon Jeong. Effects of disinformation using
deepfake: The protective effect of media literacy education. Cyberpsychology,
Behavior, and Social Networking, 24(3):188–193, 2021.
48. Serena Iacobucci, Roberta De Cicco, Francesca Michetti, Riccardo Palumbo, and
Stefano Pagliaro. Deepfakes unmasked: The effects of information priming and
bullshit receptivity on deepfake recognition and sharing intention. Cyberpsychol-
ogy, Behavior, and Social Networking, 24(3):194–202, 2021.
49. Nanna Inie, Jeanette Falk Olesen, and Leon Derczynski. The rumour mill: Making
the spread of misinformation explicit and tangible. In Extended Abstracts of the
2020 CHI Conference on Human Factors in Computing Systems, pages 1–4, 2020.
A chapter for Frontiers in Fake Media Generation and Detection 39
50. Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Syn-
thetic data and artificial neural networks for natural scene text recognition. arXiv
preprint arXiv:1406.2227, 2014.
51. JiJI. Two men arrested over deepfake pornography videos, 2020.
52. Nektaria Kaloudi and Jingyue Li. The ai-based cyber threat landscape: A survey.
ACM Computing Surveys (CSUR), 53(1):1–34, 2020.
53. Ignas Kalpokas. Problematising reality: the promises and perils of synthetic me-
dia. SN Social Sciences, 1(1):1–11, 2021.
54. Vasileia Karasavva and Aalia Noorbhai. The real threat of deepfake pornography:
a review of canadian policy. Cyberpsychology, Behavior, and Social Networking,
24(3):203–209, 2021.
55. Stamatis Karnouskos. Artificial intelligence in digital media: The era of deepfakes.
IEEE Transactions on Technology and Society, 1(3):138–147, 2020.
56. Catherine Kerner and Mathias Risse. Beyond porn and discreditation: Epistemic
promises and perils of deepfake technology in digital lifeworlds. Moral Philosophy
and Politics, 8(1):81–108, 2021.
57. David Khachaturov, Ilia Shumailov, Yiren Zhao, Nicolas Papernot, and Ross An-
derson. Markpainting: Adversarial machine learning meets inpainting, 2021.
58. Ali Khodabakhsh, Raghavendra Ramachandra, and Christoph Busch. Subjec-
tive evaluation of media consumer vulnerability to fake audiovisual content. In
2019 Eleventh International Conference on Quality of Multimedia Experience
(QoMEX), pages 1–6. IEEE, 2019.
59. Jan Kietzmann, Linda W Lee, Ian P McCarthy, and Tim C Kietzmann. Deep-
fakes: Trick or treat? Business Horizons, 63(2):135–146, 2020.
60. Jan Kietzmann, Adam J Mills, and Kirk Plangger. Deepfakes: perspectives on the
future “reality” of advertising and branding. International Journal of Advertising,
pages 1–13, 2020.
61. Zachary Kimo Stine and Nitin Agarwal. Comparative discourse analysis using
topic models: Contrasting perspectives on china from reddit. In International
Conference on Social Media and Society, pages 73–84, 2020.
62. Nils K¨obis, Barbora Doleˇzalov´a, and Ivan Soraperra. Fooled twice–people cannot
detect deepfakes but think they can. Available at SSRN 3832978, 2021.
63. Loukas Konstantinou, Ana Caraban, and Evangelos Karapanos. Combating mis-
information through nudging. In IFIP Conference on Human-Computer Interac-
tion, pages 630–634. Springer, 2019.
64. Andrei OJ Kwok and Sharon GM Koh. Deepfake: A social construction of tech-
nology perspective. Current Issues in Tourism, 24(13):1798–1802, 2021.
65. Kevin LaGrandeur. How safe is our reliance on ai, and should we regulate it? AI
and Ethics, 1(2):93–99, 2021.
66. Jack Langa. Deepfakes, real consequences: Crafting legislation to combat threats
posed by deepfakes. BUL Rev., 101:761, 2021.
67. Johannes Langguth, Konstantin Pogorelov, Stefan Brenner, Petra Filkukov´a, and
Daniel Thilo Schroeder. Don’t trust your eyes: Image manipulation in the age of
deepfakes. Frontiers in Communication, 6:26, 2021.
68. YoungAh Lee, Kuo-Ting Huang, Robin Blom, Rebecca Schriner, and Carl A Ci-
ccarelli. To believe or not to believe: framing analysis of content and audience
response of top 10 deepfake videos on youtube. Cyberpsychology, Behavior, and
Social Networking, 24(3):153–158, 2021.
69. Lingzhi Li, Jianmin Bao, Ting Zhang, Hao Yang, Dong Chen, Fang Wen, and
Baining Guo. Face x-ray for more general face forgery detection. In Proceedings of
40 D. Gamage, et al.
the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
5001–5010, 2020.
70. Xurong Li, Kun Yu, Shouling Ji, Yan Wang, Chunming Wu, and Hui Xue. Fight-
ing against deepfake: Patch&pair convolutional neural networks (ppcnn). In Com-
panion Proceedings of the Web Conference 2020, pages 88–89, 2020.
71. Yang Liu, Zhijun Yin, et al. Understanding weight loss via online discussions:
Content analysis of reddit posts using topic modeling and word clustering tech-
niques. Journal of medical Internet research, 22(6):e13745, 2020.
72. Sophie Maddocks. ‘a deepfake porn plot intended to silence me’: exploring con-
tinuities between pornographic and ‘political’deep fakes. Porn Studies, 7(4):415–
423, 2020.
73. Artem A Maksutov, Viacheslav O Morozov, Aleksander A Lavrenov, and Alexan-
der S Smirnov. Methods of deepfake detection based on machine learning. In 2020
IEEE Conference of Russian Young Researchers in Electrical and Electronic En-
gineering (EIConRus), pages 408–411. IEEE, 2020.
74. Karolina Mania. The legal implications and remedies concerning revenge porn
and fake porn: A common law perspective. Sexuality & Culture, 24(6):2079–2097,
2020.
75. December Maxwell, Sarah R Robinson, Jessica R Williams, and Craig Keaton.
” a short story of a lonely guy”: A qualitative thematic analysis of involuntary
celibacy using reddit. Sexuality & Culture, 24(6), 2020.
76. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Dis-
tributed representations of words and phrases and their compositionality. In
Advances in neural information processing systems, pages 3111–3119, 2013.
77. Yisroel Mirsky and Wenke Lee. The creation and detection of deepfakes: A survey.
ACM Computing Surveys (CSUR), 54(1):1–41, 2021.
78. David Moher, Larissa Shamseer, Mike Clarke, Davina Ghersi, Alessandro Liberati,
Mark Petticrew, Paul Shekelle, and Lesley A Stewart. Preferred reporting items
for systematic review and meta-analysis protocols (prisma-p) 2015 statement.
Systematic reviews, 4(1):1–9, 2015.
79. Gillian Murphy and Emma Flynn. Deepfake false memories. Memory, pages 1–13,
2021.
80. Eryn J Newman, Maryanne Garry, Christian Unkelbach, Daniel M Bernstein,
D Stephen Lindsay, and Robert A Nash. Truthiness and falsiness of trivia claims
depend on judgmental contexts. Journal of Experimental Psychology: Learning,
Memory, and Cognition, 41(5):1337, 2015.
81. Nicholas O’Donnell. Have we no decency? section 230 and the liability of social
media companies for deepfake videos. U. Il l. L. Rev., page 701, 2021.
82. Axel Oehmichen, Kevin Hua, Julio Amador D´ıaz L´opez, Miguel Molina-Solana,
Juan Gomez-Romero, and Yi-ke Guo. Not all lies are equal. a study into the
engineering of political misinformation in the 2016 us presidential election. IEEE
Access, 7:126305–126314, 2019.
83. Carl ¨
Ohman. Introducing the pervert’s dilemma: a contribution to the critique
of deepfake pornography. Ethics and Information Technology, pages 1–8, 2019.
84. Jennifer Otiono, Monsurat Olaosebikan, Orit Shaer, Oded Nov, and Mad Price
Ball. Understanding users information needs and collaborative sensemaking of
microbiome data. Proceedings of the ACM on Human-Computer Interaction,
3(CSCW):1–21, 2019.
85. Derek O’callaghan, Derek Greene, Joe Carthy, and P´adraig Cunningham. An
analysis of the coherence of descriptors in topic modeling. Expert Systems with
Applications, 42(13):5645–5657, 2015.
A chapter for Frontiers in Fake Media Generation and Detection 41
86. Konstantin A Pantserev. The malicious use of ai-based deepfake technology as
the new threat to psychological security and political stability. In Cyber defence
in the age of AI, smart societies and augmented humanity, pages 37–55. Springer,
Cham, 2020.
87. Miroslava Pavl´ıkov´a, Barbora ˇ
Senk`yˇrov´a, and Jakub Drmola. Propaganda and
disinformation go online. Challenging Online Propaganda and Disinformation in
the 21st Century, pages 43–74, 2021.
88. Fabian Pedregosa, Ga¨el Varoquaux, Alexandre Gramfort, Vincent Michel,
Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss,
Vincent Dubourg, et al. Scikit-learn: Machine learning in python. the Journal of
machine Learning research, 12:2825–2830, 2011.
89. Jes´us ´
Angel P´erez Dasilva, Koldobika Meso Ayerdi, and Terese Mendiguren Gal-
dospin. Deepfakes on twitter: Which actors control their spread? 2021.
90. Jiameng Pu, Neal Mangaokar, Bolun Wang, Chandan K Reddy, and Bimal
Viswanath. Noisescope: Detecting deepfake images in a blind setting. In An-
nual Computer Security Applications Conference, pages 913–927, 2020.
91. Adnan Qayyum, Junaid Qadir, Muhammad Umar Janjua, and Falak Sher. Using
blockchain to rein in the new post-truth world and check the spread of fake news.
IT Professional, 21(4):16–24, 2019.
92. Md Shohel Rana and Andrew H Sung. Deepfakestack: A deep ensemble-based
learning technique for deepfake detection. In 2020 7th IEEE International Con-
ference on Cyber Security and Cloud Computing (CSCloud)/2020 6th IEEE In-
ternational Conference on Edge Computing and Scalable Cloud (EdgeCom), pages
70–75. IEEE, 2020.
93. Claudia Ratner. When “sweetie” is not so sweet: Artificial intelligence and its
implications for child pornography. Family Court Review, 59(2):386–401, 2021.
94. Colonel Prof Filofteia Repez and Maria-Magdalena Popescu. Social media and
the threats against human security deepfake and fake news. Romanian Military
Thinking, (4), 2020.
95. Jean-Marc Rickli and Marcello Ienca. The security and military implications of
neurotechnology and artificial intelligence. Clinical Neurotechnology Meets Arti-
ficial Intelligence: Philosophical, Ethical, Legal and Social Implications, page 197,
2021.
96. Aja Romano. Jordan peele’s simulated obama psa is a double-edged warning
against fake news. Australasian Policing, 10(2):44–45, 2018.
97. Helen Rosner. The ethics of a deepfake anthony bourdain voice, 2021.
98. Kelly M Sayler and Laurie A Harris. Deep fakes and national security. Technical
report, Congressional Research SVC Washington United States, 2020.
99. Nicolas Schrading, Cecilia Ovesdotter Alm, Raymond Ptucha, and Christopher
Homan. An analysis of domestic abuse discourse on reddit. In Proceedings of the
2015 Conference on Empirical Methods in Natural Language Processing, pages
2577–2583, 2015.
100. Zack Sharf. Lucasfilm hired the youtuber who used deepfakes to tweak luke
skywalker ‘mandalorian’ vfx, 2021.
101. Saniat Javid Sohrawardi, Akash Chintha, Bao Thai, Sovantharith Seng, Andrea
Hickerson, Raymond Ptucha, and Matthew Wright. Poster: Towards robust open-
world detection of deepfakes. In Proceedings of the 2019 ACM SIGSAC Confer-
ence on Computer and Communications Security, pages 2613–2615, 2019.
102. Ahmed Soliman, Jan Hafer, and Florian Lemmerich. A characterization of po-
litical communities on reddit. In Proceedings of the 30th ACM conference on
hypertext and Social Media, pages 259–263, 2019.
42 D. Gamage, et al.
103. Shaina J Sowles, Monique McLeary, Allison Optican, Elizabeth Cahn, Melissa J
Krauss, Ellen E Fitzsimmons-Craft, Denise E Wilfley, and Patricia A Cavazos-
Rehg. A content analysis of an online pro-eating disorder community on reddit.
Body image, 24:137–144, 2018.
104. Eliza Strickland. Ai-human partnerships tackle” fake news”: Machine learning
can get you only so far-then human judgment is required-[news]. IEEE Spectrum,
55(9):12–13, 2018.
105. Eliza Strickland. Facebook takes on deepfakes. IEEE Spectrum, 57(1):40–57,
2019.
106. Catherine Stupp. Fraudsters used ai to mimic ceo’s voice in unusual cybercrime
case. The Wall Street Journal, 30(08), 2019.
107. Supasorn Suwajanakorn, Steven M. Seitz, and Ira Kemelmacher-Shlizerman. Syn-
thesizing obama: Learning lip sync from audio. ACM Trans. Graph., 36(4), July
2017.
108. Rashid Tahir, Brishna Batool, Hira Jamshed, Mahnoor Jameel, Mubashir Anwar,
Faizan Ahmed, Muhammad Adeel Zaffar, and Muhammad Fareed Zaffar. Seeing
is believing: Exploring perceptual differences in deepfake videos. In Proceedings of
the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–16,
2021.
109. Shahroz Tariq, Sangyup Lee, and Simon Woo. One detector to rule them all:
Towards a general deepfake attack detection framework. In Proceedings of the
Web Conference 2021, pages 3625–3637, 2021.
110. Bryan C Taylor. Defending the state from digital deceit: the reflexive securitiza-
tion of deepfake. Critical Studies in Media Communication, 38(1):1–17, 2021.
111. Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and
Matthias Nießner. Face2face: Real-time face capture and reenactment of rgb
videos. In Proceedings of the IEEE conference on computer vision and pattern
recognition, pages 2387–2395, 2016.
112. Sachin Thukral, Hardik Meisheri, Tushar Kataria, Aman Agarwal, Ishan Verma,
Arnab Chatterjee, and Lipika Dey. Analyzing behavioral trends in community
driven discussion platforms like reddit. In 2018 IEEE/ACM International Con-
ference on Advances in Social Networks Analysis and Mining (ASONAM), pages
662–669. IEEE, 2018.
113. Cristian Vaccari and Andrew Chadwick. Deepfakes and disinformation: Exploring
the impact of synthetic political video on deception, uncertainty, and trust in
news. Social Media+ Society, 6(1):2056305120903408, 2020.
114. James Vincent. This is what a deepfake voice clone used in a failed fraud attempt
sounds like, 2020.
115. Travis L Wagner and Ashley Blewer. “the word real is no longer real”: Deepfakes,
gender, and the challenges of ai-altered video. Open Information Science, 3(1):32–
46, 2019.
116. Karin Wahl-Jorgensen and Matt Carlson. Conjecturing fearful futures: Journal-
istic discourses on deepfakes. Journalism Practice, pages 1–18, 2021.
117. Lei Wang, Yongcheng Zhan, Qiudan Li, Daniel D Zeng, Scott J Leischow, and
Janet Okamoto. An examination of electronic cigarette content on social media:
analysis of e-cigarette flavor content on reddit. International journal of environ-
mental research and public health, 12(11):14916–14935, 2015.
118. Mika Westerlund. The emergence of deepfake technology: A review. Technology
Innovation Management Review, 9(11), 2019.
A chapter for Frontiers in Fake Media Generation and Detection 43
119. Jeffrey Westling. Are deep fakes a shallow concern? a critical analysis of the likely
societal reaction to deep fakes. A Critical Analysis of the Likely Societal Reaction
to Deep Fakes (July 24, 2019), 2019.
120. Digvijay Yadav and Sakina Salmani. Deepfake: A survey on facial forgery tech-
nique using generative adversarial network. In 2019 International Conference on
Intelligent Computing and Control Systems (ICCS), pages 852–857. IEEE, 2019.
121. Waheeb Yaqub, Otari Kakhidze, Morgan L Brockman, Nasir Memon, and Sameer
Patil. Effects of credibility indicators on social media news sharing intent. In
Proceedings of the 2020 chi conference on human factors in computing systems,
pages 1–14, 2020.
122. Youtube. Keanu goes bollywood: a true deepfake story, March 2019.
123. G Pascal Zachary. Digital manipulation and the future of electoral democracy in
the us. IEEE Transactions on Technology and Society, 1(2):104–112, 2020.
124. Catherine Zeng and Rafael Olivera-Cintr´on. Preparing for the world of a” perfect”
deepfake. Dostopno na: https://czeng. org/classes/6805/Final. pdf (18. 6. 2020),
2019.
125. Lilei Zheng, Ying Zhang, and Vrizlynn LL Thing. A survey on image tampering
and its detection in real-world photos. Journal of Visual Communication and
Image Representation, 58:380–399, 2019.
126. Zsolt Ziegler. Michael pol´anyi’s fiduciary program against fake news and deepfake
in the digital age. AI & SOCIETY, pages 1–9, 2021.
... While the definition of deepfake has changed in recent academic studies, the fundamental definition is audio, images, or videos manipulated using artificial intelligence neural networks to replace one person's voice or face with another's Gosse & Burkell 2020). A more comprehensive definition describes deepfake as synthetic content generated by artificial intelligence, specifically deep neural networks using deep learning methods (Gamage et al. 2022), which do not exist in reality but are created by computers using intricate algorithms for manipulative purposes (Westerlund 2019). ...
... In recent years, tools and techniques for producing and preparing deepfakes, such as FakeApp, have become increasingly sophisticated and accessible (Gamage et al. 2022). This ease of access has allowed many people to create convincing fake videos of people performing actions or saying things that never happened in real life (Schick 2020). ...
... Specifically, there is still a lack of information about the communities that produce deepfake videos, audio, or images, and how people discuss them on online platforms. Additionally, only English posts on deepfake were examined (e.g., Gamage et al. 2022), and languages other than English were not explored. For this reason, this study focused on the case of Türkiye and examined Turkish Reddit posts on deepfakes. ...
Article
Full-text available
This study investigates the content and changes in deepfakes-related discussions on 5,220 Turkish Reddit posts from October 2019 to August 2023. Although the academic community has shown an increasing interest in deepfakes since 2017, focusing on detection methods and the technology itself, scant attention has been paid to public perceptions and online debate. The analysis reveals that 69.4% of the examined posts feature deepfake content with sexual themes, with celebrity women being the primary targets in 60.2% of cases. In contrast, 22% of the content is about politics and political figures, while 8.6% provides technical guidance on creating deepfakes. The study also observes content changes over time, noticing a rise in sexually explicit deepfake posts, particularly involving celebrities. However, in May 2023, coinciding with the presidential and general elections in Türkiye, discussions about politics and political figures have significantly increased. This study sheds light on the changing landscape of discussions, emphasizing the predominant presence of sexual content and the increasing prevalence of political content, particularly during election seasons.
... However, these theories also exhibit significant limitations as identified by (quote the scholars above), such as the lack of consideration for technological evolution, critical thinking, and broader societal impacts. To fully understand the implications of deepfakes, an interdisciplinary approach that combines these theories with considerations of media literacy, Gamage et al. (2022) asserted that regulatory frameworks and technological safeguards is essential. This holistic perspective will provide a more nuanced understanding of deepfakes and inform strategies to mitigate their negative impact on society. ...
Article
Deepfakes, particularly audio deepfakes, have become pervasive and pose unique, ever‐changing threats to society. This paper reviews the current research landscape on audio deepfakes. We assert that limitations of existing approaches to deepfake detection and discernment are areas where (socio)linguists can directly contribute to helping address the societal challenge of audio deepfakes. In particular, incorporating expert knowledge and developing techniques that everyday listeners can use to avoid deception are promising pathways for (socio)linguistics. Further opportunities exist for developing benevolent applications of this technology through generative AI methods as well.
Article
The objectives of this research were to examine the contextual factors that impact individual's interpretation of their responsibility in the context of Deepfake videos. Using a test/retest methodology, a total of 1,023 respondents participated in a Deepfake survey instrument which measured perceptions of individual responsibility with respect to Deepfakes, individual concern with Deepfakes, and humorous perception of Deepfakes. The results of the study found that individual responsibility is negatively related to individual concern, indicating the externalization of responsibility for difficult to detect fake online videos designed to be convincing. Further, humorous perception and age impact the participants perception of individual responsibility. Younger participants were more likely to find Deepfakes humorous and this increased their perception of their own responsibility, potentially exposing them to greater harm from malicious Deepfakes.
Article
Misinformation has become a major problem in virtually all societies. The emergence of deepfake AI technology has only exacerbated issues around misinformation and disinformation. One way to try to limit the negative effects of fake news and false information in a society is media literacy. This study focused on the effect of media literacy on detecting fake news in both video and text messages. Based on Inoculation Theory, three hypotheses were suggested and a quasi-experimental research design was conducted. The participants included 204 highly educated Egyptians who were divided into three groups. One group was given a powerpoint lecture on the meaning of misinformation, its consequences, how to verify information received via social media, and examples of fake news. The second group was given a similar presentation but instead of talking about misinformation in general the presentation focused on deepfake AI technology. The third group was the control and were not given any media literacy information. Two types of fake messages were evaluated by each group; a Facebook post with text and an image and a deepfake video. The results of the study suggest the effectiveness of media literacy lectures in raising the rate of detection of fake news whether in the form of text or video. Results also indicated a decreased probability of sharing the fake messages. The study recommends designing media literacy lectures including both information on misinformation in general and specific information on the new technology creating fake news like deepfake.
Article
Full-text available
Speech deepfakes are artificial voices generated by machine learning models. Previous literature has highlighted deepfakes as one of the biggest security threats arising from progress in artificial intelligence due to their potential for misuse. However, studies investigating human detection capabilities are limited. We presented genuine and deepfake audio to n = 529 individuals and asked them to identify the deepfakes. We ran our experiments in English and Mandarin to understand if language affects detection performance and decision-making rationale. We found that detection capability is unreliable. Listeners only correctly spotted the deepfakes 73% of the time, and there was no difference in detectability between the two languages. Increasing listener awareness by providing examples of speech deepfakes only improves results slightly. As speech synthesis algorithms improve and become more realistic, we can expect the detection task to become harder. The difficulty of detecting speech deepfakes confirms their potential for misuse and signals that defenses against this threat are needed.
Chapter
Full-text available
The twenty-first century has seen technology become an integral part of human survival. Living standards have been affected as the technology continues to advance. Deep fake is a deep learning-powered application that recently appeared on the market. It allows to make fake images and videos that people cannot discern from the real ones and is a recent technique that allows swapping two identities within a single video. A wide range of factors, including communities, organizations, security, religions, democratic processes as well as personal lives, are being impacted by deep fakes. In the case of images and videos being presented to an organization as evidence, if the images or videos are altered, the entire truth will be transformed into a lie. It is inevitable that with every new and beneficial technology, some of its adverse effects will follow, causing world problems. There have been several instances in which deep fakes have alarmed the entire network. During the last few years, the number of altered images and videos has grown exponentially, posing a threat to society. Additionally, this paper explores where deep fakes are used, the impacts made by them, as well as the challenges and difficulties associated with deep fakes in this rapidly developing society.KeywordsDeepfakeDeepfake videosArtificial intelligence
Article
Full-text available
The early apprehensions about how deepfakes (also deep fakes) could be weaponized for social and political purposes are now coming to pass. This study is one of the first to examine the social impact of deepfakes. Using an online survey sample in the United States, this study investigates the relationship between citizen concerns regarding deepfakes, exposure to deepfakes, inadvertent sharing of deepfakes, the cognitive ability of individuals, and social media news skepticism. Results suggest that deepfakes exposure and concerns are positively related to social media news skepticism. In contrast, those who frequently rely on social media as a news platform are less skeptical. Higher cognitive abled individuals are more skeptical of news on social media. The moderation findings suggest that among those who are more concerned about deepfakes, inadvertently sharing a deepfake is associated with heightened skepticism. However, these patterns are more pronounced among low than high cognitive individuals.
Article
Full-text available
We review the phenomenon of deepfakes, a novel technology enabling inexpensive manipulation of video material through the use of artificial intelligence, in the context of today’s wider discussion on fake news. We discuss the foundation as well as recent developments of the technology, as well as the differences from earlier manipulation techniques and investigate technical countermeasures. While the threat of deepfake videos with substantial political impact has been widely discussed in recent years, so far, the political impact of the technology has been limited. We investigate reasons for this and extrapolate the types of deepfake videos we are likely to see in the future.
Conference Paper
In our research, we focus on the response to the non-consensual distribution of intimate or sexually explicit digital images of adults, also referred as “revenge porn”, from the point of view of the victims. In this paper, we present a preliminary expert analysis of the process for reporting revenge porn abuses in selected content sharing platforms. Among these, we included social networks, image hosting websites, video hosting platforms, forums, and pornographic sites. We looked at the way to report abuse, concerning both the non-consensual online distribution of private sexual image or video (revenge pornography), as well as the use of “deepfake” techniques, where the face of a person can be replaced on original visual content with the aim of portraying the victim in the context of sexual behaviours. This preliminary analysis is directed to understand the current practices and potential issues in the procedures designed by the providers for reporting these abuses.
Article
We examine how individual differences influence perceived accuracy of deepfake claims and sharing intention. Rather than political deepfakes, we use a non-political deepfake of a social media influencer as the stimulus, with an educational and a deceptive condition. We find that individuals are more likely to perceive the deepfake claim to be true when informative cues are missing along with the deepfake (compared to when they are present). Also, individuals are more likely to share deepfakes when they consider the fabricated claim to be accurate. Moreover, we find that cognitive ability plays a moderating role such that when informative cues are present (educational condition), individuals with high cognitive ability are less trustful of deepfake claims. Unexpectedly, when the informative cues are missing (deceptive condition), these individuals are more likely to consider the claim to be true and share them. The findings suggest that adding corrective labels can help reduce inadvertent sharing of disinformation. Also, user biases should be considered in understanding public engagement with disinformation.
Conference Paper
This paper explores the relationship between a person's demographic data, age (IV1), gender (IV2), ethnicity (IV3), and political ideology (IV4), and the risk of him/her falling prey to Mis/Disinformation attacks (DV). Participants ( n=161) were subjected to the Fake News and deepfake test (15-item). The main data analysis tool employed by this study is multiple linear regression. Important findings of the study include the revelation of the disparity in the performance of the subjects from the underrepresented groups against those who are not. This paper further confirms that an increase in age is a risk for being Disinformed. The predictive model further reveals the profile of a most likely Disinformation victim. The intended target audiences of this paper are policymakers, social scientists, and tech companies.
Article
Machine-learning has enabled the creation of “deepfake videos”; highly-realistic footage that features a person saying or doing something they never did. In recent years, this technology has become more widespread and various apps now allow an average social-media user to create a deepfake video which can be shared online. There are concerns about how this may distort memory for public events, but to date no evidence to support this. Across two experiments, we presented participants (N = 682) with fake news stories in the format of text, text with a photograph or text with a deepfake video. Though participants rated the deepfake videos as convincing, dangerous, and unethical, and some participants did report false memories after viewing deepfakes, the deepfake video format did not consistently increase false memory rates relative to the text-only or text-with-photograph conditions. Further research is needed, but the current findings suggest that while deepfake videos can distort memory for public events, they may not always be more effective than simple misleading text.