No file available
To read the file of this research,
you can request a copy directly from the authors.
Adversarial Training and Robustness for Multiple Perturbations
Preprints and early-stage research may not have been peer reviewed yet.
Abstract
Defenses against adversarial examples, such as adversarial training, are typically tailored to a single perturbation type (e.g., small -noise). For other perturbations, these defenses offer no guarantees and, at times, even increase the model's vulnerability. Our aim is to understand the reasons underlying this robustness trade-off, and to train models that are simultaneously robust to multiple perturbation types. We prove that a trade-off in robustness to different types of -bounded and spatial perturbations must exist in a natural and simple statistical setting. We corroborate our formal analysis by demonstrating similar robustness trade-offs on MNIST and CIFAR10. Building upon new multi-perturbation adversarial training schemes, and a novel efficient attack for finding -bounded adversarial examples, we show that no model trained against multiple attacks achieves robustness competitive with that of models trained on each attack individually. In particular, we uncover a pernicious gradient-masking phenomenon on MNIST, which causes adversarial training with first-order and adversaries to achieve merely accuracy. Our results question the viability and computational scalability of extending adversarial robustness, and adversarial training, to multiple perturbation types.
... Sparse adversarial examples, by altering a minimal number of input features, present a stealthier threat compared to their non-sparse counterparts. The strategic advantage of sparsity in evading detection and enhancing the transferability of attacks has been highlighted in studies by Papernot et al. [10] and Tramèr and Boneh [11]. These insights are essential for developing attacks that are both effective and difficult to detect, especially in time series data where perturbations must maintain temporal coherence. ...
Deep Neural Networks have demonstrated remarkable success in various domains but remain susceptible to adversarial examples, which are slightly altered inputs designed to induce misclassification. While adversarial attacks typically optimize under Lp norm constraints, attacks based on the L0 norm, prioritising input sparsity, are less studied due to their complex and non convex nature. These sparse adversarial examples challenge existing defenses by altering a minimal subset of features, potentially uncovering more subtle DNN weaknesses. However, the current L0 norm attack methodologies face a trade off between accuracy and efficiency either precise but computationally intense or expedient but imprecise. This paper proposes a novel, scalable, and effective approach to generate adversarial examples based on the L0 norm, aimed at refining the robustness evaluation of DNNs against such perturbations.
Machine learning models are known to lack robustness against inputs crafted by an adversary. Such adversarial examples can, for instance, be derived from regular inputs by introducing minor—yet carefully selected—perturbations. In this work, we expand on existing adversarial example crafting algorithms to construct a highly-effective attack that uses adversarial examples against malware detection models. To this end, we identify and overcome key challenges that prevent existing algorithms from being applied against malware detection: our approach operates in discrete and often binary input domains, whereas previous work operated only in continuous and differentiable domains. In addition, our technique guarantees the malware functionality of the adversarially manipulated program. In our evaluation, we train a neural network for malware detection on the DREBIN data set and achieve classification performance matching state-of-the-art from the literature. Using the augmented adversarial crafting algorithm we then manage to mislead this classifier for 63% of all malware samples. We also present a detailed evaluation of defensive mechanisms previously introduced in the computer vision contexts, including distillation and adversarial training, which show promising results.
Machine learning (ML) models, e.g., deep neural networks (DNNs), are vulnerable to adversarial examples: malicious inputs modified to yield erroneous model outputs, while appearing unmodified to human observers. Potential attacks include having malicious content like malware identified as legitimate or controlling vehicle behavior. Yet, all existing adversarial example attacks require knowledge of either the model internals or its training data. We introduce the first practical demonstration of an attacker controlling a remotely hosted DNN with no such knowledge. Indeed, the only capability of our black-box adversary is to observe labels given by the DNN to chosen inputs. Our attack strategy consists in training a local model to substitute for the target DNN, using inputs synthetically generated by an adversary and labeled by the target DNN. We use the local substitute to craft adversarial examples, and find that they are misclassified by the targeted DNN. To perform a real-world and properly-blinded evaluation, we attack a DNN hosted by MetaMind, an online deep learning API. We find that their DNN misclassifies 84.24% of the adversarial examples crafted with our substitute. We demonstrate the general applicability of our strategy to many ML techniques by conducting the same attack against models hosted by Amazon and Google, using logistic regression substitutes. They yield adversarial examples misclassified by Amazon and Google at rates of 96.19% and 88.94%. We also find that this black-box attack strategy is capable of evading defense strategies previously found to make adversarial example crafting harder.
2018 Curran Associates Inc..All rights reserved. Machine learning models are often susceptible to adversarial perturbations of their inputs. Even small perturbations can cause state-of-the-art classifiers with high “standard” accuracy to produce an incorrect prediction with high confidence. To better understand this phenomenon, we study adversarially robust learning from the viewpoint of generalization. We show that already in a simple natural data model, the sample complexity of robust learning can be significantly larger than that of “standard” learning. This gap is information theoretic and holds irrespective of the training algorithm or the model family. We complement our theoretical results with experiments on popular image classification datasets and show that a similar gap exists here as well. We postulate that the difficulty of training robust classifiers stems, at least partially, from this inherently larger sample complexity.
Many modern machine learning classifiers are shown to be vulnerable to adversarial perturbations of the instances. Despite a massive amount of work focusing on making classifiers robust, the task seems quite challenging. In this work, through a theoretical study, we investigate the adversarial risk and robustness of classifiers and draw a connection to the well-known phenomenon of “concentration of measure” in metric measure spaces. We show that if the metric probability space of the test instance is concentrated, any classifier with some initial constant error is inherently vulnerable to adversarial perturbations.One class of concentrated metric probability spaces are the so-called Lévy families that include many natural distributions. In this special case, our attacks only need to perturb the test instance by at most O(√n) to make it misclassified, where n is the data dimension. Using our general result about Lévy instance spaces, we first recover as special case some of the previously proved results about the existence of adversarial examples. However, many more Lévy families are known (e.g., product distribution under the Hamming distance) for which we immediately obtain new attacks that find adversarial examples of distance O(√n).Finally, we show that concentration of measure for product spaces implies the existence of forms of “poisoning” attacks in which the adversary tampers with the training data with the goal of degrading the classifier. In particular, we show that for any learning algorithm that uses m training examples, there is an adversary who can increase the probability of any “bad property” (e.g., failing on a particular test instance) that initially happens with non-negligible probability to ≈ 1 by substituting only Õe(√m) of the examples with other (still correctly labeled) examples.
Despite achieving impressive and often superhuman performance on multiple benchmarks, state-of-the-art deep networks remain highly vulnerable to perturbations: adding small, imperceptible, adversarial perturbations can lead to very high error rates. Provided the data distribution is defined using a generative model mapping latent vectors to datapoints in the distribution, we prove that no classifier can be robust to adversarial perturbations when the latent space is sufficiently large and the generative model sufficiently smooth. Under the same conditions, we prove the existence of adversarial perturbations that transfer well across different models with small risk. We conclude the paper with experiments validating the theoretical bounds.
Recent work has shown that neural network-based vision classifiers exhibit a significant vulnerability to misclassifications caused by imperceptible but adversarial perturbations of their inputs. These perturbations, however, are purely pixel-wise and built out of loss function gradients of either the attacked model or its surrogate. As a result, they tend to look pretty artificial and contrived. This might suggest that vulnerability to misclassification of slight input perturbations can only arise in a truly adversarial setting and thus is unlikely to be a problem in more benign contexts. In this paper, we provide evidence that such a belief might be incorrect. To this end, we show that neural networks are already vulnerable to significantly simpler - and more likely to occur naturally - transformations of the inputs. Specifically, we demonstrate that rotations and translations alone suffice to significantly degrade the classification performance of neural network-based vision models across a spectrum of datasets. This remains to be the case even when these models are trained using appropriate data augmentation and are already robust against the canonical, pixel-wise perturbations. Also, finding such "fooling" transformation does not even require having any special access to the model or its surrogate - just trying out a small number of random rotation and translation combinations already has a significant effect. These findings suggest that our current neural network-based vision models might not be as reliable as we tend to assume.
Deep CNNs are known to exhibit the following peculiarity: on the one hand they generalize extremely well to a test set, while on the other hand they are extremely sensitive to so-called adversarial perturbations. The extreme sensitivity of high performance CNNs to adversarial examples casts serious doubt that these networks are learning high level abstractions in the dataset. We are concerned with the following question: How can a deep CNN that does not learn any high level semantics of the dataset manage to generalize so well? The goal of this article is to measure the tendency of CNNs to learn surface statistical regularities of the dataset. To this end, we use Fourier filtering to construct datasets which share the exact same high level abstractions but exhibit qualitatively different surface statistical regularities. For the SVHN and CIFAR-10 datasets, we present two Fourier filtered variants: a low frequency variant and a randomly filtered variant. Each of the Fourier filtering schemes is tuned to preserve the recognizability of the objects. Our main finding is that CNNs exhibit a tendency to latch onto the Fourier image statistics of the training dataset, sometimes exhibiting up to a 28% generalization gap across the various test sets. Moreover, we observe that significantly increasing the depth of a network has a very marginal impact on closing the aforementioned generalization gap. Thus we provide quantitative evidence supporting the hypothesis that deep CNNs tend to learn surface statistical regularities in the dataset rather than higher-level abstract concepts.
Several machine learning models, including neural networks, consistently mis- classify adversarial examples—inputs formed by applying small but intentionally worst-case perturbations to examples from the dataset, such that the perturbed in- put results in the model outputting an incorrect answer with high confidence. Early attempts at explaining this phenomenon focused on nonlinearity and overfitting. We argue instead that the primary cause of neural networks' vulnerability to ad- versarial perturbation is their linear nature. This explanation is supported by new quantitative results while giving the first explanation of the most intriguing fact about them: their generalization across architectures and training sets. Moreover, this view yields a simple and fast method of generating adversarial examples. Us- ing this approach to provide examples for adversarial training, we reduce the test set error of a maxout network on the MNIST dataset.
Despite widespread adoption, machine learning models remain mostly black boxes. Understanding the reasons behind predictions is, however, quite important in assessing trust in a model. Trust is fundamental if one plans to take action based on a prediction, or when choosing whether or not to deploy a new model. Such understanding further provides insights into the model, which can be used to turn an untrustworthy model or prediction into a trustworthy one. In this work, we propose LIME, a novel explanation technique that explains the predictions of any classifier in an interpretable and faithful manner, by learning an interpretable model locally around the prediction. We further propose a method to explain models by presenting representative individual predictions and their explanations in a non-redundant way, framing the task as a submodular optimization problem. We demonstrate the flexibility of these methods by explaining different models for text (e.g. random forests) and image classification (e.g. neural networks). The usefulness of explanations is shown via novel experiments, both simulated and with human subjects. Our explanations empower users in various scenarios that require trust: deciding if one should trust a prediction, choosing between models, improving an untrustworthy classifier, and detecting why a classifier should not be trusted.
Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples
- A Athalye
- N Carlini
- D Wagner
A. Athalye, N. Carlini, and D. Wagner. Obfuscated gradients give a false sense of security:
Circumventing defenses to adversarial examples. In International Conference on Machine
Learning (ICML), 2018.
Decision-based adversarial attacks: Reliable attacks against black-box machine learning models
- W Brendel
- J Rauber
- M Bethge
W. Brendel, J. Rauber, and M. Bethge. Decision-based adversarial attacks: Reliable attacks against black-box machine learning models. In International Conference on Learning
Representations, 2018.
Hidden voice commands
- N Carlini
- P Mishra
- T Vaidya
- Y Zhang
- M Sherr
- C Shields
- D Wagner
- W Zhou
N. Carlini, P. Mishra, T. Vaidya, Y. Zhang, M. Sherr, C. Shields, D. Wagner, and W. Zhou.
Hidden voice commands. In USENIX Security Symposium, pages 513-530, 2016.
Ead: elastic-net attacks to deep neural networks via adversarial examples
- P.-Y Chen
- Y Sharma
- H Zhang
- J Yi
- C.-J Hsieh
P.-Y. Chen, Y. Sharma, H. Zhang, J. Yi, and C.-J. Hsieh. Ead: elastic-net attacks to deep
neural networks via adversarial examples. In AAAI Conference on Artificial Intelligence,
2018.
Efficient projections onto the l1-ball for learning in high dimensions
- J Duchi
- S Shalev-Shwartz
- Y Singer
- T Chandra
J. Duchi, S. Shalev-Shwartz, Y. Singer, and T. Chandra. Efficient projections onto the l1-ball
for learning in high dimensions. In International Conference on Machine Learning (ICML),
2008.
ImageNettrained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
- R Geirhos
- P Rubisch
- C Michaelis
- M Bethge
- F A Wichmann
- W Brendel
R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel. ImageNettrained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations (ICLR), 2019.
Benchmarking neural network robustness to common corruptions and perturbations
- D Hendrycks
- T Dietterich
D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations (ICLR),
2019.
Exploiting excessive invariance caused by norm-bounded adversarial robustness
- J Jacobsen
- J Behrmann
- N Carlini
- F Tramèr
- N Papernot
J. Jacobsen, J. Behrmann, N. Carlini, F. Tramèr, and N. Papernot. Exploiting excessive
invariance caused by norm-bounded adversarial robustness. arXiv preprint arXiv:1903.10484,
2019.
On the geometry of adversarial examples
- M Khoury
- D Hadfield-Menell
M. Khoury and D. Hadfield-Menell. On the geometry of adversarial examples, 2019.
Adversarial machine learning at scale
- A Kurakin
- I Goodfellow
- S Bengio
A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial machine learning at scale. In International Conference on Learning Representations (ICLR), 2017.
Adversarial robustness: Theory and practice
- A Madry
- Z Kolter
A. Madry and Z. Kolter. Adversarial robustness: Theory and practice. In Tutorial at NeurIPS
2018, 2018.
Towards deep learning models resistant to adversarial attacks
- A Madry
- A Makelov
- L Schmidt
- D Tsipras
- A Vladu
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models
resistant to adversarial attacks. In International Conference on Learning Representations
(ICLR), 2018.
Certified defenses against adversarial examples
- A Raghunathan
- J Steinhardt
- P Liang
A. Raghunathan, J. Steinhardt, and P. Liang. Certified defenses against adversarial examples.
In International Conference on Learning Representations (ICLR), 2018.
Towards the first adversarially robust neural network model on mnist
- L Schott
- J Rauber
- M Bethge
- W Brendel
L. Schott, J. Rauber, M. Bethge, and W. Brendel. Towards the first adversarially robust
neural network model on mnist. In International Conference on Learning Representations
(ICLR), 2019.
- J Gilmer
- L Metz
- F Faghri
- S S Schoenholz
- M Raghu
- M Wattenberg
- I Goodfellow
J. Gilmer, L. Metz, F. Faghri, S. S. Schoenholz, M. Raghu, M. Wattenberg, and I. Goodfellow.
Adversarial spheres. arXiv preprint arXiv:1801.02774, 2018.
- B Li
- C Chen
- W Wang
- L Carin
B. Li, C. Chen, W. Wang, and L. Carin. Second-order adversarial attack and certifiable
robustness. arXiv preprint arXiv:1809.03113, 2018.