Content uploaded by Marc Schmitt
Author content
All content in this area was uploaded by Marc Schmitt on Jan 28, 2025
Content may be subject to copyright.
1
Cyber Shadows: Neutralizing Security Threats with
AI and Targeted Policy Measures
Marc Schmitt , Pantelis Koutroumpis
Abstract—The digital age, driven by the AI revolution, brings
significant opportunities but also conceals security threats, which
we refer to as cyber shadows. These threats pose risks at
individual, organizational, and societal levels. This paper ex-
amines the systemic impact of these cyber threats and pro-
poses a comprehensive cybersecurity strategy that integrates AI-
driven solutions, such as Intrusion Detection Systems (IDS), with
targeted policy interventions. By combining technological and
regulatory measures, we create a multilevel defense capable of
addressing both direct threats and indirect negative externalities.
We emphasize that the synergy between AI-driven solutions and
policy interventions is essential for neutralizing cyber threats
and mitigating their negative impact on the digital economy.
Finally, we underscore the need for continuous adaptation of
these strategies, especially in response to the rapid advancement
of autonomous AI-driven attacks, to ensure the creation of secure
and resilient digital ecosystems.
Impact Statement—This paper contributes to the ongoing
efforts to build a safer digital world by addressing the growing cy-
bersecurity challenges posed by artificial intelligence. We propose
a strategy that combines AI-driven security technologies with pol-
icy measures to protect individuals, businesses, and society from
evolving cyber threats. Our work aims to not only neutralize these
threats but also safeguard essential values like privacy, fairness,
and security in the digital economy. By offering practical tools
and recommendations, including a ’potential threat directory,’ we
help policymakers, researchers, and organizations better prepare
for future risks and coordinate responses. The ultimate goal is
to foster secure and resilient digital ecosystems that can adapt
to the rapidly changing landscape of AI-driven cyberattacks.
Index Terms—Artificial Intelligence, Cybersecurity, Policy,
Threat Detection, Digital Trust
I. INTRODUCTION
ARTIFICIAL Intelligence (AI) has reached human level
performance across a wide range of knowledge domains,
due to a mix of computing power, data availability and
algorithmic innovations [1]. A combination of these core
capabilities has allowed applications to foster the invention of
new algorithms [2], advance medical treatments [3], augment
human creativity [4], and accelerate scientific progress [5].
Scaling laws predict further improvements in the near term
that could lead to general-purpose AI agents for a range of
technological applications.
While the upsides of these new capabilities can boost
productivity in several segments and markets, their potential
Forthcoming in IEEE Transactions on Artificial Intelligence.
M. Schmitt is with Siemens AG, Munich, Germany. The views expressed
in this article are solely those of the author and do not necessarily reflect the
views of Siemens AG.
P. Koutroumpis is with the Oxford Martin School, University of Oxford,
Oxford, U.K.
downsides have drawn the attention of scholars and policy-
makers. At the core of these downsides lies the ”trust” that
users can have when interacting with LLMs. In this process
several recent works have pointed to serious flaws. First, Large
language models can be easily distracted by irrelevant context
which significantly reduces the consistency and performance
[6]. Second, artificial agents have traditionally been trained to
maximize reward, which may incentivize power-seeking and
deception, analogous to how next-token prediction in language
models (LMs) may incentivize toxicity. This often leads to
harmful and anti-social behavior that needs to be mitigated
during training [7]. Beside these, researchers have also un-
covered that the traditional Turing test may not be enough
to validate the performance of LLMs as an arbitrary person,
which led them to point to the hyper-accuracy distortion,
uncovering a trait that is not common when real humans
respond to specific questions [8].
Policymakers have also responded to these first signs. The
UK Safety Summit initiated a global exchange of policies
around the governance of AI and coined the term “frontier
AI” to capture the importance of this undertaking1. A few
days prior to the UK Summit the US President signed an
executive order on AI, setting high standards for AI safety
and security, safeguarding privacy, ensuring equity and civil
rights, and promoting innovation2. The European Union was
the first to start the work on its EU AI Act3, pointing towards
a risk-based regulatory framework.
In this paper we focus on the ways that the increasing use
of AI can amplify the exposure to cyber-threats either directly
to its users or through negative externalities to individuals and
organizations not involved in those activities. Additionally, we
will delve into strategies for neutralizing these cyber shadows
through the integration of AI-driven security solutions and
targeted policy measures.
II. CY BE R SHA DOW S
The proliferation of Generative AI escalates existing threats
and brings novel vulnerabilities. Cyber shadows refer to the
hidden or amplified security threats that emerge within digital
ecosystems due to the use of advanced AI technologies. These
threats may be direct, as seen in AI-driven attacks that enhance
traditional cyber threats, or indirect, that impact the wider
digital ecosystem. For these latter indirect threats, we borrow
1“Highly capable general-purpose AI models that can perform a wide
variety of tasks and match or exceed the capabilities present in today’s most
advanced models.”
2US AI Executive Order
3EU AI Act
arXiv:2501.09025v1 [cs.CR] 3 Jan 2025
2
Fig. 1. Technology driven Threat Amplification by Generative AI
the term ”negative externalities” from economics, to refer
to unintended and often widespread consequences that affect
third parties or the system at large, such as data breaches
or the erosion of trust in digital interactions. Due to their
human-level text generation capabilities, GenAI can be used
to launch social engineering attacks in all types of inter-
actions, including computer-to-computer, human-to-computer,
and human-to-human. Thus, LLMs pose a notable threat to the
cybersecurity landscape. These AI systems can be exploited to
create sophisticated social engineering and phishing attacks,
produce realistic forgeries, or automate the generation of mali-
cious code at a scale and sophistication previously unattainable
[9]. LLMs can be accessed with techniques like jailbreaking,
prompt injection, and reverse psychology, which are attack
attempts to manipulate a system by altering its usual behavior.
AI-enabled agents have the potential to automate various parts,
if not the entire lifecycle (cyber kill chain), of cyber attacks. In
addition, AI agents can – theoretically – learn from each attack
and improve their success rate. In this context, individuals
are more likely to forfeit control of their own accounts and
as a result jeopardize their own and other’s data. As AI/ML
technologies have the potential to both detect and amplify wide
range of cyber attacks it is essential to understand the nature
of such techniques and develop proactive strategies to mitigate
this escalating threat.
A. Native AI Threat Amplification
In this section we outline the technology driven threats
which represent the direct cyber shadows for AI systems.
This means that we are focusing on the ”off the shelf”
characteristics of LLMs and not on their misuse. In Section
II-B we look into the threats from LLM misuse and also
present further indirect effects (externalities) that derive from
its use.
1) Automated Code Creation: The integration of AI into
critical infrastructure without comprehensive security mea-
sures could lead to new attack vectors, potentially destabilizing
essential services. During the last decade, there has been a
consistent increase in the number of common IT security
vulnerabilities and exposures (CVEs) worldwide and a more
pronounced rise in the past two years both in absolute numbers
and severity (see Figure 3).
Generative AI through its ability to automate code gener-
ation/creation has led to a phenomenal rise in the usage of
AI assistants with GitHub’s Co-pilot “behind an average of
46 Percent of the cross-language code written. As the number
of CVEs continues to grow fast, users of assisted code have
been found to “believe they wrote secure code than those
without access to the AI assistant” when in reality those
“who trusted the AI less and engaged more with the language
and format of their prompts [. . . ] provided code with fewer
security vulnerabilities” [10]. Further research has shown that
both ChatGPT “often generates source code that is not robust
to well-known attacks” [11] and GitHub’s Co-pilot is “more
likely to generate vulnerable code in response to prompts that
correspond to older vulnerabilities.” [12]
Acknowledging the cyber threat risks in code generation
in a recent (27th June) workshop organized by Google,
Stanford University, and the University of Wisconsin-Madison
participants agreed in a joint document that there is a lack
of comprehensive code-related LLM capabilities which is
necessary “to inform possible defenses and possible threats”.
There is ongoing research on security hardening techniques
that guide code generation towards these constantly changing
security needs “without modifying the LM’s weights” which
3
shows promise but requires further effort [13]. In particular
these security hardening techniques largely rely on fine-tuning
existing models to make them adapt to new code vulnerabil-
ities. However, there is a separate push within this area that
tries to prevent malevolent actors from fine-tuning existing
modes [14]. This approach makes fine-tuning as costly - in
terms of compute - as training from scratch and prohibits
a quick response in cases where new vulnerabilities emerge.
The ”balance of incentives” between safety and cybersecurity
renders a quick solution less obvious and should alert policy-
makers in terms of the limits of automated coding generation.
2) Social Engineering and Phishing: The impact of gener-
ative AI on SE can be split into three major pillars:
Content Generation: Generative AI’s foremost capacity
lies in creating convincing digital content, critical in phishing
scams like website cloning, where AI can duplicate and subtly
alter legitimate websites to trick users (see Figure 2). It ex-
cels in generating persuasive texts, authentic-looking images,
credible voice recordings, and deepfake videos that mimic
real individuals, elevating the threat of misinformation and
deception. This technology can be misused for impersonation,
bypassing biometric security, extortion through fabricated con-
tent, or spreading disinformation.
Personalization: Generative AI accelerates the shift to-
ward hyper-personalized cyber attacks by enabling attackers
to craft highly convincing content and strategies tailored to
individuals’ online behaviors and profiles. This AI-facilitated
reconnaissance gathers detailed intelligence to inform pre-
texting, where attackers create credible narratives for en-
gagement. Consequently, the execution of attacks becomes
more sophisticated, leveraging AI-generated emails, messages,
or interactions that convincingly mimic known contacts or
organizations, thus increasing the likelihood of successful
manipulation.
Scalability: AI-driven automation empowers attackers to
execute widespread and sophisticated phishing and social
engineering campaigns with little effort. By analyzing victims’
behaviors, AI enhances the convincing nature of attacks,
allowing for real-time adaptation in response strategies through
feedback loops. Intelligent bots within AI-powered botnets
can autonomously engage with potential victims on a large
scale, simulating human interactions and adapting messages
based on feedback. Moreover, generative AI can craft content
that bypasses security measures, using polymorphic techniques
to evade detection (Section II-A5). This level of automation
allows for unprecedented scale and efficacy in cyber attacks,
challenging current cybersecurity defenses.
3) Exploitation of Hallucinations: The exploitation of
LLMs’ hallucinations has recently been shown to be the target
of potential attackers. As code-assistants and chat-bots often
generate responses for URLs that do not exist in reality,
attackers can leverage these responses and host their malicious
code in these links. Once a user receives these responses from
the AI-assistant, they will download the malicious code. The
probability of receiving a hallucinated response was more than
20% for Node.js (40 cases with at least one package that has
not been published out of 201) and more than 35% for Python
packages (80 responses with at least one unpublished package
out of 227 questions)4.
4) Data Poisoning Attacks: Recent progress in the field
has led to the increasing incorporation of machine learn-
ing models into a variety of practical applications, aiding
humans in their everyday decision-making processes [15],
[16], [17]. Consequently, the rise of these applications makes
data poisoning attacks a significant problem. These attacks,
wherein maliciously altered data are fed into the training
set of a model, can severely compromise the integrity and
reliability of machine learning systems, potentially leading to
erroneous or biased decisions in critical applications. Mul-
timodal models, which use both visual and linguistic data,
have recently become prominent but face the risk of poisoning
attacks where adversaries manipulate training data to induce
harmful behaviors. [18] examined such attacks in both visual
and linguistic domains, aiming to determine their relative
vulnerability. The authors developed three types of poisoning
attacks for multimodal models and tested them across various
datasets and architectures. The results show that these attacks
are effectively harmful in both modalities, although the extent
of their impact varies. In response, the authors devised defense
strategies for both the pretraining and post-training phases,
which their experiments show can significantly diminish the
effects of these attacks while maintaining the models’ effec-
tiveness.
5) Polymorphic and Metamorphic Malware: Polymorphic
and metamorphic malware represent two sophisticated types of
malicious software that are designed to evade detection. Poly-
morphic malware changes its code or signature patterns with
each iteration, making it challenging for traditional signature-
based detection methods to identify them consistently. In
contrast, metamorphic malware goes a step further by not only
changing its appearance but also altering its underlying code,
essentially rewriting itself completely. This makes metamor-
phic malware even more elusive as it can vary its behavior
and structure between infections. The advent of Generative AI
and Large Language Models (LLMs) amplifies these threats
significantly. These advanced AI systems can automate and
refine the process of creating polymorphic and metamor-
phic malware, enabling them to generate numerous, highly
variable, and sophisticated versions of malicious software at
an unprecedented scale. This could potentially overwhelm
conventional cybersecurity defenses, necessitating more ad-
vanced, AI-driven countermeasures to detect and neutralize
these evolving cyber threats.
B. Negative Externalities from the use of GenAI
Negative externalities are unintended, often harmful con-
sequences of AI technologies that extend beyond the direct
users or developers. In the context of generative AI, these
externalities manifest as broader impacts on organizations and
societies, such as data breaches at the firm level, where the
sophistication of AI-driven attacks leads to increased costs,
potential reputational damage, or the erosion of trust in digital
4The source and details of these attacks can be found in this link:
https://vulcan.io/blog/ai-hallucinations-package-risk
4
Fig. 2. The Three Pillars of GenAI induced Threat Amplification in Social Engineering [9]
systems, which weakens confidence in technology due to the
pervasive use of AI-generated content and misinformation.
1) Erosion of Trust in Digital Systems: Trust is the cor-
nerstone of our interactions with technology. Yet, as AI use
expands, a troubling trend emerges—trust erosion in digital
systems. This erosion is fueled by increased cyber threats,
persuasive disinformation, privacy concerns tied to AI data
collection, opaque algorithms, and regulatory gaps, among
others. The European Union Agency for Law Enforcement
Cooperation (Europol) highlights that ”threat actors will make
increasing use of deepfake technology to facilitate various
criminal acts and conduct disinformation campaigns to influ-
ence or distort public opinion.” 5
Beyond deepfakes, the rise of synthetic content which is
forecast to increase as much as 90% of the content online6
is likely to further push trust levels towards media outlets
down. The World Economic Forum7highlights that over 2
billion people will participate in elections in 2024 making the
reliance on AI generated content central in the democratic
processes. To ensure the thriving of our digital ecosystems,
addressing this trust erosion, worsened by the proliferation of
cyber shadows, is imperative.
2) Data Breaches at the Firm Level: The development of
GenAI represents a double-edged sword for the technology
and business world. While AI can enhance the sophistication
of cyber-attacks, target high-value data, and monitor cyber
systems beyond traditional DevOps capabilities, it also in-
creases the cost and sophistication of data breaches. Since
2020, data breach costs have been rising steadily, with the
global average in 2024 reaching $4.88 million 8. This increase
is due to improvements in attack methods, greater exposure
through extensive digital services, and the higher monetary
5EUROPOL - Law enforcement and the challenge of deepfakes
6Axios - AI could choke on its own exhaust as it fills the web
7How can we build trustworthy media ecosystems in the age of AI and
declining trust?
8IBM Security – Cost of a Data Breach Report
value of compromised data. Before the introduction of data
protection regulations, breach-related losses were asymmetric,
as data-loss cyber incidents disproportionately impacted users
compared to the financial impact on targeted firms. Rising
consumer awareness has increased direct firm costs through
reputational damage, trading partner losses, and supply-chain
disruptions [19]. However, the full shift of the financial burden
wasn’t fully realized until significant improvements in en-
forcement and potential fines were introduced with GDPR-like
regulations [20]. This situation is expected to put additional
pressure on firms’ finances and exacerbate the slowdown in
firm dynamics, including entry and exit activities, due to higher
compliance costs associated with GDPR-like regulations.
15,000
20,000
25,000
2017 2018 2019 2020 2021 2022 2023*
All CVEs CVEs with score 7-10
Fig. 3. Vulnerability and Exposure Increase, Source: National Vulnerability
Database (NVD/NIST), data for 2023 as of 11th November 2023. CVSS scores
range from 0-10 with 10 representing the most critical CVEs.
3) Vulnerabilities in Critical Industries: AI-driven threats
are particularly disruptive to critical sectors such as financial
services, healthcare, and energy infrastructure. These indus-
tries rely heavily on AI systems, making them vulnerable to
large-scale cyberattacks that can compromise sensitive data
and disrupt essential services. These industries serve as crit-
5
ical pillars of the digital economy, and disruptions in their
operations have far-reaching effects on economic stability and
societal trust.
III. TOWARD SH AD OW NEUTRALIZATION
So, how do we neutralize those cyber shadows? In the
quest to achieve cyber resilience, we find ourselves navigating
through a complex landscape where no ”silver bullet” solutions
exist. The section delves into the intricate challenges and
potential strategies for bolstering our digital defenses, with a
primary focus on AI-driven security solutions and policy mea-
sures. While definitive answers may be elusive, the importance
of exploring solutions and fostering widespread awareness
cannot be overstated. This is crucial not only for safeguarding
industries and infrastructures but also for protecting the global
community and every individual within it.
A. AI-driven Threat (Shadow) Hunting
AI and ML technologies play a pivotal role in fortifying
digital systems against a myriad of cyber threats [21]. 9These
technologies have been instrumental in developing sophisti-
cated tools and methodologies for network intrusion detection,
malware identification, spam filtering, and analyzing network
traffic, among other applications. The integration of AI/ML in
cybersecurity initiatives significantly enhances the ability to
counteract emerging cyber threats through three main pillars:
robustness, response, and resilience [22].
Resilience: AI’s role extends to ensuring that cybersecurity
systems can withstand attacks without significant compromise
to their functionality. A crucial aspect of this resilience is the
systems’ adeptness at identifying potential threats through AI-
driven cyber threat and anomaly detection mechanisms.
Robustness: AI-based cybersecurity systems are designed
to maintain stability and continue functioning effectively even
when targeted by adversarial attacks. They possess self-healing
and self-testing capabilities, ensuring that the systems can
recover from and adapt to various forms of cyber aggression.
Response: These systems are not just reactive but are
also adaptive, learning from each incident to autonomously
improve their defensive mechanisms. This includes the ability
to launch countermeasures, create decoys, and establish hon-
eypots to mislead and trap attackers, enhancing the overall
security posture.
1) Autonomous Threat Detection Systems: Intrusion Detec-
tion Systems (IDS) safeguard computer networks and systems
by detecting unauthorized access, breaches, and malicious
activities, thus protecting data integrity, confidentiality, and
availability [21]. IDS fall into two categories:
Network Intrusion Detection Systems (NIDS): Scan net-
work traffic for suspicious patterns indicating potential attacks.
Host Intrusion Detection Systems (HIDS): Are deployed
on specific devices to monitor system and network operations,
including changes in system logs and file modifications, pin-
pointing threats directly affecting the host.
9https://www.weforum.org/agenda/2024/01/cybersecurity-ai-frontline-
artificial-intelligence/
ML is pivotal in cyber threat detection, employing various
techniques to identify and mitigate a wide range of attacks.
SVMs classify attacks like DoS and Probing, KNN reduces
false alarms in intrusion detection, Decision Trees aid feature
selection for network IDS, and, when combined with Genetic
Algorithms, enhance IDS construction. Advanced methods,
including Deep Learning with RNNs, LSTMs, and CNNs,
excel at classifying anomalies and countering sophisticated
attacks. AI in cyber threat detection reflects a growing reliance
on data-driven approaches for robust cybersecurity. Intelli-
gent Automated Cybersecurity solutions, harnessing these AI
techniques, are essential for increasing the resilience and
robustness of digital ecosystems against the threats posed by
Generative AI.
AI will transform the landscape of cyber threat hunting
by enabling more proactive and efficient identification of
potential security threats or ”cyber shadows”. Through AI-
driven threat hunting, organizations can leverage machine
learning algorithms to sift through vast amounts of data
at unprecedented speeds, identifying anomalies that could
indicate a compromise or an attempted breach. This approach
significantly reduces the time it takes to detect threats, mov-
ing from a reactive to a proactive stance in cybersecurity.
AI algorithms can learn from past incidents, adapting to
new tactics employed by cyber attackers, thereby continually
improving threat detection capabilities. Moreover, AI-driven
tools can automate the tedious and time-consuming tasks of
data analysis, freeing up human analysts to focus on more
complex investigations and decision-making processes. This
integration of AI into cybersecurity operations enhances the
ability to detect, analyze, and neutralize cyber shadows before
they can manifest into full-blown cyber attacks, ensuring a
more robust defense posture for organizations. However, full
neutralization requires a response - an additional step beyond
”threat detection”, which is the execution of counter-measures
/ attacks either autonomous or via a human (see III-A3).
2) Countering Malicious AI Image Alterations: [23] de-
scribe a strategy to reduce the risks associated with malicious
editing of images by large-scale diffusion models, which are
advanced AI systems used for image manipulation. The central
concept is to ’immunize’ images, making them less vulnerable
to being altered by these models. This is achieved by adding
tiny, imperceptible changes to the images, known as adversar-
ial perturbations, which are designed to interfere with how the
diffusion models work. When these models try to manipulate
an ’immunized’ image, they end up creating unrealistic results.
Additionally, the authors suggests a policy change for this
approach to be truly effective. Instead of relying on individual
users to implement this image protection, the responsibility
should fall to the organizations that develop the diffusion
models. These entities should actively support and carry out
the immunization process. The strategy for mitigating risks
from image editing by large-scale diffusion models using
adversarial perturbations emphasizes cybersecurity robustness.
3) Collaborative Intelligence in Cyber Threat Response:
Full neutralization of cyber threats requires a response that
goes beyond mere threat detection. The response phase in-
volves the execution of counter-measures or actions, which can
6
be carried out autonomously by AI systems or in collaboration
with human analysts [24]. While automation is crucial for
rapid response to known threats and routine tasks, the human
element remains invaluable in complex investigations or more
contextual decision making. It also ensures transparency, fair-
ness, and therefore trust in AI-driven security solutions [25],
[26].
Automation is critical for handling routine and repetitive
tasks, especially in the early stages of threat detection and
response. It can rapidly identify and block known threats,
reducing the burden on human analysts. Automation also
enables real-time monitoring and response, which is vital in
the face of fast-evolving threats.
Augmentation enhances the capabilities of human analysts
by providing them with AI-driven insights, recommendations,
and data analysis tools. This collaboration allows analysts to
make more informed decisions, investigate complex threats,
and adapt to novel attack strategies effectively. Augmentation
is particularly valuable when dealing with sophisticated attacks
that may require human judgment and expertise.
Human AI Collaboration ensures that AI-driven responses
are aligned with organizational goals and ethical considera-
tions. Therefore, a balanced approach that combines automa-
tion with human expertise is the most effective strategy for
risk assessment, detection, and response.
B. Policy Measures to Alleviate Negative Externalities
Striking the “right” balance between the cyber enhance-
ments that GenAI can provide, and the new vulnerabilities
introduced by its use, is not an easy task [13], [22]. By
making AI more accessible and democratized, we benefit
from the cross-industry financial efficiencies but suffer from a
wider threat of malicious actors appropriating the lower entry
barriers. As the cost of acquiring and deploying AI capabilities
decreases, a broader array of adversaries, even those with
limited resources, can tap into sophisticated AI tools [21]. This
means that the scale and frequency of AI-driven (phishing)
attacks could surge, making it even more challenging for
organizations to maintain an effective defense. The interplay
of accessibility and affordability of AI technologies may
intensify the threat landscape, underscoring the urgency for
novel defensive measures.
This situation necessitates a delicate balance in policy
making – regulatory and legislative efforts must offer sufficient
protection and control without stifling AI’s potential as an
innovation accelerator. The European Union’s AI Act marks
a positive step towards this, yet feedback from the industry
suggests that strong country-member opposition can curtail
its reach. Meanwhile, the United States has implemented
an AI Executive Order, which puts a strong emphasis on
fostering innovation and does not fully address the complex
cybersecurity concerns associated with AI.
To further mitigate the cybersecurity challenges posed by
AI, several solutions are proposed. Firm-level regulation,
akin to the GDPR but with specific focus on AI, is crucial
(including the implementation of a risk-based framework)
[27]. Automated code improvement measures (like security
hardening) should be integrated more robustly into the training
of Large Language Models (LLMs) or “sit” on top of the
AI assistant applications. As a result, techniques such as
Reinforcement Learning from Compiler Feedback (RLCF)
and controlled code generation are essential. These should
be complemented by a standardized dataset of secure-coding
practices to ensure AI systems are trained with cybersecurity
in mind. Finally, regularly updating LLMs with the latest
security vulnerabilities is vital to ensure they are equipped
to identify and address emerging threats effectively [13].
This holistic approach, combining regulatory frameworks with
advanced technological measures, is key to leveraging AI in
cybersecurity while minimizing its potential risks.
C. Cybersecurity Frameworks and AI Regulation
Cybersecurity policies vary across regions, with the United
States, Europe, and other global areas adopting different
approaches to regulating AI and addressing digital threats.
In the United States, the recently signed AI Executive
Order focuses on promoting innovation and ensuring that AI
systems are safe, trustworthy, and uphold privacy and civil
rights. The NIST Cybersecurity Framework, widely adopted
across industries, provides a risk-based approach to managing
cybersecurity threats, emphasizing detection, response, and
recovery strategies in AI-based systems.
In Europe, the EU AI Act and General Data Protection
Regulation (GDPR) work together to regulate AI systems,
particularly those classified as high-risk. The EU AI Act
mandates transparency, risk management, and human oversight
in AI deployments, while GDPR ensures that personal data is
processed with privacy by design and by default, protecting
individual rights across the digital ecosystem.
In other regions, such as Asia, frameworks like Singapore’s
Model AI Governance Framework and Japan’s AI Strategy
emphasize the responsible development of AI technologies.
Singapore’s framework, for instance, promotes transparency
and accountability in AI systems, while Japan’s strategy
focuses on fostering AI innovation while addressing ethical
concerns and ensuring security.
D. AI Risk Management in the EU Regulatory Framework
Under the EU AI Act, AI systems used for high-risk
applications, such as fraud detection in financial services,
must incorporate stringent safeguards to protect individual
rights and ensure compliance with regulatory standards. In
such cases, developers of AI-driven cybersecurity tools are
required to implement comprehensive risk management pro-
cesses. These systems must be designed with privacy by design
and privacy by default principles, ensuring that personal data
is processed securely and fairly.
For instance, an AI-based fraud detection system could
monitor financial transactions in real time, identifying unusual
patterns that indicate potential fraud. To comply with both the
AI Act and GDPR, this system would include transparency
measures, informing users that AI is involved in decision-
making processes and allowing for human oversight to prevent
errors or biases. Additionally, the system would need to ensure
7
that users’ privacy rights are upheld by minimizing the amount
of personal data processed and securing it through robust
encryption methods.
By integrating the requirements of the EU AI Act with
GDPR’s privacy protections, such AI systems would mitigate
risks while ensuring regulatory compliance and safeguarding
individual freedoms within digital ecosystems.
IV. DISCUSSION
A. Integrating Policy and Technology
The most effective way to safeguard our digital ecosystems
lies in a strategic blend of targeted policy measures and
the deployment of AI-driven security technologies. This dual
approach leverages the strengths of both proactive policy
frameworks and advanced technological solutions to create a
comprehensive defense mechanism against cyber threats.
Targeted policy measures are crucial for establishing the
legal and regulatory framework necessary for cybersecurity.
They set the standards and define expectations for behavior
within digital spaces, enforce compliance, and deter malicious
activities through legal repercussions. Policies can mandate
essential security protocols, data protection requirements, and
incident response strategies, ensuring that organizations have
a baseline level of security that they must achieve.
AI-driven security technologies, on the other hand, pro-
vide the dynamic and adaptive capabilities needed to combat
sophisticated cyber threats and practically enforce the agreed
policy measures. AI and ML can analyze vast amounts of
data at incredible speeds, identify patterns indicative of cyber
attacks, and evolve in response to new threats. This technology
enables real-time threat detection and response, going beyond
the static defenses traditional security measures offer.
The combination of these two approaches allows for a robust
cybersecurity posture that is both preventive, through policy,
and reactive, through technology. For example, shifting the
responsibility of implementing image protection from indi-
vidual users to the organizations developing diffusion models
adds a layer of systemic robustness. It ensures that the pro-
tective measures are consistently applied at the source, further
strengthening the overall defense against malicious image edit-
ing [23]. Policies provide the framework and guidelines that
shape the implementation and use of AI technologies, ensuring
they are used responsibly and effectively. However, policies
can be ”toothless” if left without the necessary enforcement
capabilities. For the image protection example, new techniques
including Glaze10 - a ”defense” method against style mimicry
- and Nightshade11 - an ”offense” tool to distort feature
representations inside generative AI image models - can make
this battle for image protection copyrights a lot smoother.
Still, it is important to recognize the challenges and com-
plexities involved in harmonizing policy and technology. Poli-
cies may lag behind the rapid pace of technological inno-
vation, and AI technologies may encounter ethical, privacy,
or accuracy concerns that need careful regulation. Hence,
continuous dialogue between policymakers, technologists, and
10https://glaze.cs.uchicago.edu/
11https://nightshade.cs.uchicago.edu/
cybersecurity professionals is essential to refine and adjust
both policy measures and technology deployments, ensuring
they remain effective in the face of evolving cyber threats.
B. Looking Ahead
As we pivot from the intricacies of our current cyberse-
curity paradigm towards the future, it becomes clear that the
nuanced approach combining targeted policy measures with
AI-driven security technologies sets a foundational blueprint
for advancing digital protection. Understanding the real trade-
offs in this process is essential for the adoption of AI tech-
nologies in ways that increase social welfare. The journey
ahead, while promising, is fraught with evolving challenges
that necessitate not only adaptation but also anticipation. For
example, the advent of AI and ML ushers in the potential
for fully autonomous attack agents [28], [9], representing a
formidable risk in the cybersecurity landscape. Unlike most
previous technologies that primarily enhanced productivity,
AI introduces the capacity for autonomy. This characteristic
significantly amplifies the threat level, as AI can independently
initiate, execute, and adapt cyber attacks without human inter-
vention. Such autonomous agents could exploit vulnerabilities
at speeds and complexities far beyond human capability to re-
spond, making them a critical concern for future cybersecurity
measures. This shift towards autonomous capabilities necessi-
tates a proactive and sophisticated approach to cyber defense,
emphasizing the development of equally advanced AI-driven
security solutions that can anticipate, identify, and neutralize
threats autonomously, ensuring a dynamic and resilient digital
ecosystem.
Evolution of AI Technologies: The rapid pace of techno-
logical innovation mandates that AI-driven security solutions
must not only respond to current threats but also adapt to
anticipate future vulnerabilities (e.g., quantum computing and
extended reality [29]). This involves leveraging advances in AI
and machine learning to develop more sophisticated, predictive
models that can preemptively identify and neutralize threats
before they materialize.
Dynamic Policy Frameworks: As digital ecosystems
evolve, so too must the policy frameworks that govern them.
This requires a proactive approach to policy-making, where
regulations are regularly reviewed and updated to reflect the
latest technological advancements and threat vectors. Such
dynamic policies must balance the need for security with the
imperative of fostering innovation and protecting individual
privacy.
The path to robust cybersecurity is iterative and collabo-
rative, demanding ongoing innovation, adaptable policies, and
stakeholder cooperation. This nuanced approach paves the way
for secure, resilient, ethical, and inclusive digital ecosystems
in the future.
V. CONCLUSION
The remarkable capabilities of GenAI present a unique
blend of potential benefits and risks. When used respon-
sibly, AI can contribute significantly to addressing critical
challenges, enhancing prosperity, productivity, innovation, and
8
security. Conversely, GenAI can cast diverse cyber shadows
over the global economy. While the regulatory and legislative
initiatives represent some vital first steps, further collaborative
efforts across the government, the private sector, academia,
and the civil society are needed to steer AI away from the
shadows and into the light to achieve predominantly beneficial
outcomes. Currently, one of the most pressing concerns is the
amplification of existing threats. AI, in its current state, has
the potential to intensify the severity and frequency of cyber
threats already present in our digital landscape. This escalation
demands immediate attention and action from policymakers
and corporate leaders.
Looking ahead, the landscape of threats is poised to evolve
further. We anticipate the emergence of new, sophisticated
threats, such as fully automated cyberattack agents powered
by AI. These advanced threats could operate with unprece-
dented efficiency and scale, likely posing a challenge for
traditional defense mechanisms. We need a nuanced and
multidimensional approach to cybersecurity, recognizing that
neither policy nor technology alone are sufficient to protect
digital ecosystems. Instead, it is their strategic integration that
offers the best chance of securing our digital futures against
increasingly sophisticated cyber threats.
It is important to note that the true extent of these risks and
their impact is still largely unknown, which underscores the
need for ongoing research and analysis to fully understand
and address these challenges. As such, it is imperative that
politicians and boards of directors take AI seriously. Their role
in shaping policies, strategies, and responses to these evolving
cyber threats is crucial. Without their active engagement and
foresight, the cyber shadows cast by GenAI could darken,
impacting not just individual companies or sectors, but the
global economy and society at large. The time to act is now,
to harness the immense power of AI for the greater good and
safeguard our collective future.
REFERENCES
[1] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” nature, vol. 521,
no. 7553, pp. 436–444, 2015.
[2] D. J. Mankowitz, A. Michi, A. Zhernov, M. Gelmi, M. Selvi, C. Padu-
raru, E. Leurent, S. Iqbal, J.-B. Lespiau, A. Ahern et al., “Faster sorting
algorithms discovered using deep reinforcement learning,” Nature, vol.
618, no. 7964, pp. 257–263, 2023.
[3] M. Moor, O. Banerjee, Z. S. H. Abad, H. M. Krumholz, J. Leskovec,
E. J. Topol, and P. Rajpurkar, “Foundation models for generalist medical
artificial intelligence,” Nature, vol. 616, no. 7956, pp. 259–265, 2023.
[4] N. Jia, X. Luo, Z. Fang, and C. Liao, “When and how artificial
intelligence augments employee creativity,” Academy of Management
Journal, no. ja, 2023.
[5] J. Sourati and J. A. Evans, “Accelerating science with human-aware
artificial intelligence,” Nature Human Behaviour, vol. 7, no. 10, pp.
1682–1696, 2023.
[6] F. Shi, X. Chen, K. Misra, N. Scales, D. Dohan, E. Chi, N. Sch¨
arli,
and D. Zhou, “Large language models can be easily distracted by
irrelevant context,” in Proceedings of the 40th International Conference
on Machine Learning, ser. ICML’23. JMLR.org, 2023.
[7] A. Pan, J. S. Chan, A. Zou, N. Li, S. Basart, T. Woodside, H. Zhang,
S. Emmons, and D. Hendrycks, “Do the rewards justify the means?
measuring trade-offs between rewards and ethical behavior in the machi-
avelli benchmark,” in Proceedings of the 40th International Conference
on Machine Learning, ser. ICML’23. JMLR.org, 2023.
[8] G. Aher, R. I. Arriaga, and A. T. Kalai, “Using large language models
to simulate multiple humans and replicate human subject studies,” in
Proceedings of the 40th International Conference on Machine Learning,
ser. ICML’23. JMLR.org, 2023.
[9] M. Schmitt and I. Flechais, “Digital deception: generative artificial
intelligence in social engineering and phishing,” Artificial Intelligence
Review, vol. 57, p. 324, 10 2024.
[10] N. Perry, M. Srivastava, D. Kumar, and D. Boneh, “Do users write
more insecure code with ai assistants?” in Proceedings of the 2023 ACM
SIGSAC Conference on Computer and Communications Security, 2023,
pp. 2785–2799.
[11] R. Khoury, A. R. Avila, J. Brunelle, and B. M. Camara, “How secure
is code generated by chatgpt?” arXiv preprint arXiv:2304.09655, 2023.
[12] O. Asare, M. Nagappan, and N. Asokan, “Is github’s copilot as bad
as humans at introducing vulnerabilities in code?” Empirical Software
Engineering, vol. 28, no. 6, p. 129, 2023.
[13] J. He and M. Vechev, “Large language models for code: Security
hardening and adversarial testing,” in Proceedings of the 2023 ACM
SIGSAC Conference on Computer and Communications Security, 2023,
pp. 1865–1879.
[14] J. Deng, S. Pang, Y. Chen, L. Xia, Y. Bai, H. Weng, and W. Xu, “Sophon:
Non-fine-tunable learning to restrain task transferability for pre-trained
models,” arXiv preprint arXiv:2404.12699, 2024.
[15] M. Schmitt, “Deep learning in business analytics: A clash of expectations
and reality,” International Journal of Information Management Data
Insights, 2023. [Online]. Available: https://linkinghub.elsevier.com/
retrieve/pii/S2667096822000891
[16] T. T. Eapen, D. J. Finkenstadt, J. Folk, and L. Venkataswamy,
“How Generative AI Can Augment Human Creativity,” Havard
Business Review, 2023. [Online]. Available: https://hbr.org/2023/07/
how-generative-ai- can-augment- human-creativity
[17] D. A. Shepherd and A. Majchrzak, “Machines augmenting en-
trepreneurs: Opportunities (and threats) at the Nexus of artificial intel-
ligence and entrepreneurship,” Journal of Business Venturing, vol. 37,
no. 4, 7 2022.
[18] Z. Yang, X. He, Z. Li, M. Backes, M. Humbert, P. Berrang, and
Y. Zhang, “Data poisoning attacks against multimodal encoders,” in
Proceedings of the 40th International Conference on Machine Learning,
ser. ICML’23. JMLR.org, 2023.
[19] P. Akey, S. Lewellen, I. Liskovich, and C. Schiller, “Hacking corpo-
rate reputations,” Rotman School of Management Working Paper, no.
3143740, 2021.
[20] P. Koutroumpis, F. Ravasan, and T. Tarannum, “(under) investment in
cyber skills and data protection enforcement: Evidence from activity
logs of the uk information commissioner’s office,” Available at SSRN
4179601, 2022.
[21] M. Schmitt, “Securing the digital world: Protecting smart infrastructures
and digital industries with artificial intelligence (ai)-enabled malware
and intrusion detection,” Journal of Industrial Information Integration,
vol. 36, p. 100520, 2023.
[22] M. Taddeo, T. McCutcheon, and L. Floridi, “Trusting artificial intel-
ligence in cybersecurity is a double-edged sword,” Nature Machine
Intelligence, vol. 1, no. 12, pp. 557–560, 2019.
[23] H. Salman, A. Khaddaj, G. Leclerc, A. Ilyas, and A. Madry, “Raising
the cost of malicious ai-powered image editing,” in Proceedings of
the 40th International Conference on Machine Learning, ser. ICML’23.
JMLR.org, 2023.
[24] A. Peng, A. Netanyahu, M. Ho, T. Shu, A. Bobu, J. Shah, and
P. Agrawal, “Diagnosis, feedback, adaptation: a human-in-the-loop
framework for test-time policy adaptation,” in Proceedings of the
40th International Conference on Machine Learning, ser. ICML’23.
JMLR.org, 2023.
[25] W. Zhang, T. Wu, Y. Wang, Y. Cai, and H. Cai, “Towards trustworthy
explanation: on causal rationalization,” in Proceedings of the 40th Inter-
national Conference on Machine Learning, ser. ICML’23. JMLR.org,
2023.
[26] R. Hosseini, L. Zhang, B. Garg, and P. Xie, “Fair and accurate decision
making through group-aware learning,” in Proceedings of the 40th Inter-
national Conference on Machine Learning, ser. ICML’23. JMLR.org,
2023.
[27] A. Jain, C. Adiole, S. Chaudhuri, T. Reps, and C. Jermaine, “Tuning
models of code with compiler-generated reinforcement learning feed-
back,” arXiv preprint arXiv:2305.18341, 2023.
[28] O. Slumbers, D. H. Mguni, S. B. Blumberg, S. McAleer, Y. Yang,
and J. Wang, “A game-theoretic framework for managing risk in multi-
agent systems,” in Proceedings of the 40th International Conference on
Machine Learning, ser. ICML’23. JMLR.org, 2023.
[29] S. Qamar, Z. Anwar, and M. Afzal, “A systematic threat analysis and
defense strategies for the metaverse and extended reality systems,” 5
2023.
