Ross Anderson’s research while affiliated with University of Edinburgh and other places

What is this page?


This page lists works of an author who doesn't have a ResearchGate profile or hasn't added the works to their profile yet. It is automatically generated from public (personal) data to further our legitimate goal of comprehensive and accurate scientific recordkeeping. If you are this author and want this page removed, please let us know.

Publications (181)


The high-level description of the feedback mechanism in the learning process
a, Model collapse refers to a degenerative learning process in which models start forgetting improbable events over time, as the model becomes poisoned with its own projection of reality. Here data are assumed to be human-curated and start off clean; then model 0 is trained and data are sampled from it; at step n, data are added to the overall data from step n − 1 and this combination is used to train model n. Data obtained with Monte Carlo sampling should ideally be statistically close to the original, provided that fitting and sampling procedures are perfect. This process depicts what happens in real life with the Internet: model-generated data become pervasive. b,c, Performance of OPT-125m models of different generations evaluated using the original wikitext2 test dataset. Shown on the left are the histograms of perplexities of each individual data training sequence produced by different generations as evaluated by the very first model trained with the real data. Over the generations, models tend to produce samples that the original model trained with real data is more likely to produce. At the same time, a much longer tail appears for later generations. Later generations start producing samples that would never be produced by the original model, that is, they start misperceiving reality based on errors introduced by their ancestors. The same plots are shown in 3D in the Supplementary Materials. On the right, average perplexity and its standard deviation are shown for each independent run. The x axis refers to the generation of the model. ‘Real’ refers to the ‘model 0’ trained on the original wikitext2 dataset; model 1 was trained on the data produced by model 0, model 2 was trained on data produced by model 1 and so on, with all generated datasets equal in size. We find that models trained on generated data are able to learn some of the original task, but with errors, as seen from the increase in perplexity.
AI models collapse when trained on recursively generated data
  • Article
  • Full-text available

July 2024

·

509 Reads

·

150 Citations

Nature

Ilia Shumailov

·

·

Yiren Zhao

·

[...]

·

Yarin Gal

Stable diffusion revolutionized image creation from descriptive text. GPT-2 (ref. ¹), GPT-3(.5) (ref. ²) and GPT-4 (ref. ³) demonstrated high performance across a variety of language tasks. ChatGPT introduced such language models to the public. It is now clear that generative artificial intelligence (AI) such as large language models (LLMs) is here to stay and will substantially change the ecosystem of online text and images. Here we consider what may happen to GPT-{n} once LLMs contribute much of the text found online. We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear. We refer to this effect as ‘model collapse’ and show that it can occur in LLMs as well as in variational autoencoders (VAEs) and Gaussian mixture models (GMMs). We build theoretical intuition behind the phenomenon and portray its ubiquity among all learned generative models. We demonstrate that it must be taken seriously if we are to sustain the benefits of training from large-scale data scraped from the web. Indeed, the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of LLM-generated content in data crawled from the Internet.

Download

Threat models over space and time: A case study of end‐to‐end‐encrypted messaging applications

May 2024

·

4 Reads

·

4 Citations

Software Practice and Experience

Threat modeling is one of the foundations of secure systems engineering and must take heed of the context within which systems operate. In this work, we explore the extent to which real‐world systems engineering reflects a changing threat context. We examine the desktop clients of six widely used end‐to‐end‐encrypted mobile messaging applications to understand the extent to which they adjusted their threat model over space (when enabling clients on new platforms, such as desktop clients) and time (as new threats emerged). We experimented with short‐lived adversarial access against these desktop clients and analyzed the results using two popular threat elicitation frameworks, STRIDE and LINDDUN. The results demonstrate that system designers need to track threats in the evolving context within which systems operate and, more importantly, mitigate them by rescoping trust boundaries so that they remain consistent with administrative boundaries. A nuanced understanding of the relationship between trust and administration is vital for robust security, including the provision of safe defaults.


Figure 2: Major incidents disrupting KIWI FARMS from September to December 2022. Green stars indicate the forum recovery.
Figure 3: Global search trends and traffic to all forum domains during the disruption. The star indicates the Streisand effect.
Figure 6: The number of daily tweets and reactions made by the community about the campaign. Figure scales are different.
Figure 12: Discussion of the event on KIWI FARMS, its Telegram channel, and LOLCOW FARM. Figure scales are different.
No Easy Way Out: the Effectiveness of Deplatforming an Extremist Forum to Suppress Hate and Harassment

May 2024

·

2,622 Reads

Legislators and policymakers worldwide are debating options for suppressing illegal, harmful and undesirable material online. Drawing on several quantitative data sources, we show that deplatforming an active community to suppress online hate and harassment, even with a substantial concerted effort involving several tech firms, can be hard. Our case study is the disruption of the largest and longest-running harassment forum Kiwi Farms in late 2022, which is probably the most extensive industry effort to date. Despite the active participation of a number of tech companies over several consecutive months, this campaign failed to shut down the forum and remove its objectionable content. While briefly raising public awareness, it led to rapid platform displacement and traffic fragmentation. Part of the activity decamped to Telegram, while traffic shifted from the primary domain to previously abandoned alternatives. The forum experienced intermittent outages for several weeks, after which the community leading the campaign lost interest, traffic was directed back to the main domain, users quickly returned, and the forum was back online and became even more connected. The forum members themselves stopped discussing the incident shortly thereafter, and the net effect was that forum activity, active users, threads, posts and traffic were all cut by about half. Deplatforming a community without a court order raises philosophical issues about censorship versus free speech; ethical and legal issues about the role of industry in online content moderation; and practical issues on the efficacy of private-sector versus government action. Deplatforming a dispersed community using a series of court orders against individual service providers appears unlikely to be very effective if the censor cannot incapacitate the key maintainers, whether by arresting them, enjoining them or otherwise deterring them.



Figure 3: Number of DDoS attacks and victims per day in the Russia-Ukraine scale (top) and global scale (stacked, bottom).
Figure 4: Number of DDoS victims in Russia and Ukraine by hour around the invasion day (marked with the red star).
Figure 8: Number of daily stacked targets (top) and cumulative targets (bottom) being promoted in top five categories.
The complete collection of the 5 most popular defacement archives for 6 months from 1 January 2022 to 30 June 2022.
Getting Bored of Cyberwar: Exploring the Role of Low-level Cybercrime Actors in the Russia-Ukraine Conflict

May 2024

·

1,050 Reads

There has been substantial commentary on the role of cyberattacks carried by low-level cybercrime actors in the Russia-Ukraine conflict. We analyse 358k web defacement attacks, 1.7M reflected DDoS attacks, 1764 Hack Forums posts mentioning the two countries, and 441 announcements (with 58k replies) of a volunteer hacking group for two months before and four months after the invasion. We find the conflict briefly but notably caught the attention of low-level cybercrime actors, with significant increases in online discussion and both types of attack targeting Russia and Ukraine. However, there was little evidence of high-profile actions; the role of these players in the ongoing hybrid warfare is minor, and they should be separated from persistent and motivated 'hacktivists' in state-sponsored operations. Their involvement in the conflict appears to have been short-lived and fleeting, with a clear loss of interest in discussing the situation and carrying out both defacement and DDoS attacks against either Russia or Ukraine after a few weeks.



Figure 1. Scanning operation flows. Left : Server-side scanning. Right : Client-side scanning (the main changes are in orange).
Figure 2. From server-side to client-side: New compromise paths and advantage points for adversaries ( −→ : compromise paths in server-side scanning; −→ : compromise paths in CSS; − −→ : knowledge gained by adversary in CSS).
Figure 3. Collisions of the NeuralHash function extracted from iOS 14. Top : A pair of accidentally colliding images in the ImageNet database of 14 million sample images; Bottom : An artificially constructed pair of colliding images.
Bugs in our pockets: the risks of client-side scanning

January 2024

·

133 Reads

·

21 Citations

Journal of Cybersecurity

Our increasing reliance on digital technology for personal, economic, and government affairs has made it essential to secure the communications and devices of private citizens, businesses, and governments. This has led to pervasive use of cryptography across society. Despite its evident advantages, law enforcement and national security agencies have argued that the spread of cryptography has hindered access to evidence and intelligence. Some in industry and government now advocate a new technology to access targeted data: client-side scanning (CSS). Instead of weakening encryption or providing law enforcement with backdoor keys to decrypt communications, CSS would enable on-device analysis of data in the clear. If targeted information were detected, its existence and, potentially, its source would be revealed to the agencies; otherwise, little or no information would leave the client device. Its proponents claim that CSS is a solution to the encryption versus public safety debate: it offers privacy—in the sense of unimpeded end-to-end encryption—and the ability to successfully investigate serious crime. In this paper, we argue that CSS neither guarantees efficacious crime prevention nor prevents surveillance. Indeed, the effect is the opposite. CSS by its nature creates serious security and privacy risks for all society, while the assistance it can provide for law enforcement is at best problematic. There are multiple ways in which CSS can fail, can be evaded, and can be abused.


If It’s Provably Secure, It Probably Isn’t: Why Learning from Proof Failure Is Hard

October 2023

Lecture Notes in Computer Science

In this paper we’re going to explore the ways in which security proofs can fail, and their broader lessons for security engineering. To mention just one example, Larry Paulson proved the security of SSL/TLS using his theorem prover Isabelle in 1999, yet it’s sprung multiple leaks since then, from timing attacks to Heartbleed. We will go through a number of other examples in the hope of elucidating general principles. Proofs can be irrelevant, they can be opaque, they can be misleading and they can even be wrong. So we can look to the philosophy of mathematics for illumination. But the problem is more general. What happens, for example, when we have a choice between relying on mathematics and on physics? The security proofs claimed for quantum cryptosystems based on entanglement raise some pointed questions and may engage the philosophy of physics. And then there’s the other varieties of assurance; we will recall the reliance placed on FIPS-140 evaluations, which API attacks suggested may have been overblown. Where the defenders focus their assurance effort on a subsystem or a model that cannot capture the whole attack surface they may just tell the attacker where to focus their effort. However, we think it’s deeper and broader than that. The models of proof and assurance on which we try to rely have a social aspect, which we can try to understand from other perspectives ranging from the philosophy or sociology of science to the psychology of shared attention. These perspectives suggest, in various ways, how the management of errors and exceptions may be particularly poor. They do not merely relate to failure modes that the designers failed to consider properly or at all; they also relate to failure modes that the designers (or perhaps the verifiers) did not want to consider for institutional and cultural reasons.


One Protocol to Rule Them All? On Securing Interoperable Messaging

October 2023

·

16 Reads

·

2 Citations

Lecture Notes in Computer Science

European lawmakers have ruled that users on different platforms should be able to exchange messages with each other. Yet messaging interoperability opens up a Pandora’s box of security and privacy challenges. While championed not just as an anti-trust measure but as a means of providing a better experience for the end user, interoperability runs the risk of making the user experience worse if poorly executed. There are two fundamental questions: how to enable the actual message exchange, and how to handle the numerous residual challenges arising from encrypted messages passing from one service provider to another – including but certainly not limited to content moderation, user authentication, key management, and metadata sharing between providers. In this work, we identify specific open questions and challenges around interoperable communication in end-to-end encrypted messaging, and present high-level suggestions for tackling these challenges.


Towards Human-Centric Endpoint Security

October 2023

·

19 Reads

Lecture Notes in Computer Science

In a survey of six widely used end-to-end encrypted messaging applications, we consider the post-compromise recovery process from the perspective of what security audit functions, if any, are in place to detect and recover from attacks. Our investigation reveals audit functions vary in the extent to which they rely on the end user. We argue developers should minimize dependence on users and view them as a residual, not primary, risk mitigation strategy. To provide robust communications security, E2EE applications need to avoid protocol designs that dump too much responsibility on naive users and instead make system components play an appropriate role.


Citations (52)


... Cybercriminals may sometimes get bored and 'burn out' [56] and their interest can wane as has been seen for the volunteer hacktivists reacting to armed conflicts [49,76]. It might be thought that industry could act more effectively than law enforcement [77], but when a series of swift and competent tech firms attempted to shut down an online hate and harassment forum in late 2022, it still recovered after a few months [78]. ...

Reference:

Assessing the Aftermath: the Effects of a Global Takedown against DDoS-for-hire Services
No Easy Way Out: the Effectiveness of Deplatforming an Extremist Forum to Suppress Hate and Harassment
  • Citing Conference Paper
  • May 2024

... Modern E2E encryption are failing to protect privacy, allowing malactors to perform spying activities. The use of desktop clients with shared system states that are open to compromise further questions the robustness of these systems [1]. ...

Threat models over space and time: A case study of end‐to‐end‐encrypted messaging applications
  • Citing Article
  • May 2024

Software Practice and Experience

... While we can consider these positive developments, tech companies may leverage PETs strategically, e.g. to claim that certain regulations no longer apply when they use PETs [243; 251]. Some of this is due to the underlying protection mechanisms: moderating content or providing data access to data subjects, researchers, and public bodies may be hard when said data is encrypted, which led to contentious use of PETs for client side scanning [2]. ...

Bugs in our pockets: the risks of client-side scanning

Journal of Cybersecurity

... 4 Net neutrality regulations, which required internet service providers not to give preferred treatment to certain types of content or traffic for economic reasons, have been at the heart of political controversies since the 1990s [36,40,56,57,[60][61][62]. The "security vs. interoperability" narrative, our focus in this paper, is a more recent development that very little prior work has considered. Since the DMA, a promising initial literature has begun to examine the complex technical considerations around securely interoperating end-to-end encrypted messaging [23,43]. ...

One Protocol to Rule Them All? On Securing Interoperable Messaging
  • Citing Conference Paper
  • October 2023

Lecture Notes in Computer Science

... A related issue that has caused growing concern in language modeling is that over time more and more real-world language will presumably be produced with the assistance of LLMs, which will make it increasingly difficult to compile contemporary corpora of real human language for training new models or updating existing ones (Shumailov et al., 2023). Proposed solutions to these problems of data contamination (Balloccu et al., 2024) and task contamination (Li and Flanigan, 2024) generally involve finding ways to exclude machine-generated language from future training data, including through watermarking systems (Kirchenbauer et al., 2023;Dathathri et al., 2024). ...

Model Dementia: Generated Data Makes Models Forget

... Frontiers in Sustainability 09 frontiersin.org online shopping to virtual consultations, demonstrating the dominance of digital information (Stigler, 1961;OECD, 2017;Peitz and Waldfogel, 2012;Srikanth and Thakur, 2022). This transition redefines economic rent, shifting from agricultural and land-based models to Sustainable Digital Rent (SDR), capturing shifts from physical to informational commodities, impacting environmental sustainability, socio-economic structures, and cognitive states (Ward and Aalbers, 2016;Bliss and Egler, 2020;Pirgmaier, 2021). ...

The Oxford Handbook of the Digital Economy

... This brings us to our final example, and the stimulus for writing this paper: the bidirectional coding vulnerabilities we recently discovered in both large language models and computer source code. The latter mostly got fixed while the former largely didn't [7]. This appears to have been largely cultural. ...

Talking Trojan: Analyzing an Industry-Wide Disclosure
  • Citing Conference Paper
  • November 2022

... Dataset and Preprocessor: We used CrimeBB [41], one of the most well-known datasets in underground forums. This dataset has been utilized in various works [39], [8], [45], [62]. We focused our study on Hack-Forums, and due to the presence of noise, we preprocessed the users' posted content to ensure it was thoroughly cleaned. ...

PostCog: A Tool for Interdisciplinary Research into Underground Forums at Scale

... Character-level Attacks introduce subtle modifications at the character level, such as misspellings, typographical errors, and the insertion of visually similar or invisible characters (e.g., homoglyphs [58]). These attacks exploit the model's sensitivity to minor character variations, which are often unnoticeable to humans, allowing for a high degree of stealthiness while potentially preserving the original meaning. ...

Bad Characters: Imperceptible NLP Attacks
  • Citing Conference Paper
  • May 2022