Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training.

Lue Tao,Lei Feng,Jinfeng Yi,Sheng-Jun Huang,Songcan Chen

2021-01-01

Abstract:Use the "Report an Issue" link to request a name change.

What problem does this paper attempt to address?

Attack As Defense: Characterizing Adversarial Examples Using Robustness.

Zhe Zhao,Guangke Chen,Jingyi Wang,Yiwei Yang,Fu Song,Jun Sun

DOI: https://doi.org/10.1145/3460319.3464822

2021-01-01

Abstract:As a new programming paradigm, deep learning has expanded its application to many real-world problems. At the same time, deep learning based software are found to be vulnerable to adversarial attacks. Though various defense mechanisms have been proposed to improve robustness of deep learning software, many of them are ineffective against adaptive attacks. In this work, we propose a novel characterization to distinguish adversarial examples from benign ones based on the observation that adversarial examples are significantly less robust than benign ones. As existing robustness measurement does not scale to large networks, we propose a novel defense framework, named attack as defense (A2D), to detect adversarial examples by effectively evaluating an example’s robustness. A2D uses the cost of attacking an input for robustness evaluation and identifies those less robust examples as adversarial since less robust examples are easier to attack. Extensive experiment results on MNIST, CIFAR10 and ImageNet show that A2D is more effective than recent promising approaches. We also evaluate our defense against potential adaptive attacks and show that A2D is effective in defending carefully designed adaptive attacks, e.g., the attack success rate drops to 0% on CIFAR10.
Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training

Lue Tao,Lei Feng,Jinfeng Yi,Sheng-Jun Huang,Songcan Chen

2021-01-01

Abstract:Delusive attacks aim to substantially deteriorate the test accuracy of the learning model by slightly perturbing the features of correctly labeled training examples. By formalizing this malicious attack as finding the worst-case training data within a specific ∞-Wasserstein ball, we show that minimizing adversarial risk on the perturbed data is equivalent to optimizing an upper bound of natural risk on the original data. This implies that adversarial training can serve as a principled defense against delusive attacks. Thus, the test accuracy decreased by delusive attacks can be largely recovered by adversarial training. To further understand the internal mechanism of the defense, we disclose that adversarial training can resist the delusive perturbations by preventing the learner from overly relying on non-robust features in a natural setting. Finally, we complement our theoretical findings with a set of experiments on popular benchmark datasets, which show that the defense withstands six different practical attacks. Both theoretical and empirical results vote for adversarial training when confronted with delusive adversaries.
Adversarial for Good – Defending Training Data Privacy with Adversarial Attack Wisdom

Shengyuan Pang,Yanjiao Chen,Jiangyi Deng,Jialin Wu,Yijie Bai,Wenyuan Xu

DOI: https://doi.org/10.1109/metacom62920.2024.00040

2024-01-01

Abstract:Machine learning models dazzle us with their incredible performance but may irritate us with data privacy issues. Various attacks have been proposed to peep into the sensitive training data of machine learning models, the mainstream ones being membership inference attacks and model inversion attacks. As a countermeasure, defense strategies have been devised. Nonetheless, a unified theoretical framework and evaluation testbed for training data privacy analysis is lacking. In this paper, we taxonomize representative attack methods regarding the attack objective, attack knowledge, and attack capability. As for the defense, we focus on the novel idea of turning adversarial attacks into privacy protection tools, hence the title adversarial for good. To provide an open-sourced integrated platform to evaluate different attacks and defenses. Our experiment results show that adopting adversarial example attacks and adversarial training for data privacy protection is compelling, which may motivate more efforts in transforming adversarial to good in the future.
Boosting Adversarial Training in Safety-Critical Systems Through Boundary Data Selection

Jia Yifan,Poskitt Christopher M.,Zhang Peixin,Wang Jingyi,Sun Jun,Chattopadhyay Sudipta

DOI: https://doi.org/10.1109/lra.2023.3327934

2023-01-01

Abstract:AI-enabled collaborative robots are designed to be used in close collaboration with humans, thus requiring stringent safety standards and quick response times. Adversarial attacks pose a significant threat to the deep learning models of these systems, making it crucial to develop methods to improve the models' robustness against them. Adversarial training is one approach to improve their robustness: it works by augmenting the training data with adversarial examples. This, unfortunately, comes with the cost of increased computational overhead and extended training times. In this work, we balance the need for additional adversarial data with the goal of minimizing the training costs by selecting the most ‘valuable’ data for adversarial training. In particular, we propose a robustness-oriented boundary data selection method, RAST-AT, which stands for robust and fast adversarial training. RAST-AT selects training data near to the boundary by considering adversarial perturbations. Our method improves the speed of model training on CIFAR-10 by 68.67%, and compared to other data selection methods, has 10% higher accuracy with 10% training data selected, and 7% higher robustness with 4% training data selected. Our method also significantly improves efficiency by at least 25% in adversarial training, with the same performance. Finally, we evaluate our method on a cobot system, generating adversarial patches as attacks, and adopting RAST-AT as the defense. We find that RAST-AT can defend against 60% of untargeted attacks and 20% of targeted attacks. Our work highlights the benefits of developing effective defenses against adversarial attacks to ensure the security and reliability of AI-powered safety-critical systems.
Adversarial Training: A Survey

Mengnan Zhao,Lihe Zhang,Jingwen Ye,Huchuan Lu,Baocai Yin,Xinchao Wang

2024-10-19

Abstract:Adversarial training (AT) refers to integrating adversarial examples -- inputs altered with imperceptible perturbations that can significantly impact model predictions -- into the training process. Recent studies have demonstrated the effectiveness of AT in improving the robustness of deep neural networks against diverse adversarial attacks. However, a comprehensive overview of these developments is still missing. This survey addresses this gap by reviewing a broad range of recent and representative studies. Specifically, we first describe the implementation procedures and practical applications of AT, followed by a comprehensive review of AT techniques from three perspectives: data enhancement, network design, and training configurations. Lastly, we discuss common challenges in AT and propose several promising directions for future research.

Machine Learning,Artificial Intelligence
On the Vulnerability of Adversarially Trained Models Against Two-faced Attacks

Shengjie Zhou,Lue Tao,Yuzhou Cao,Tao Xiang,Bo An,Lei Feng

2024-01-01

Abstract:Adversarial robustness is an important standard for measuring the quality of learned models, and adversarial training is an effective strategy for improving the adversarial robustness of models. In this paper, we disclose that adversarially trained models are vulnerable to two-faced attacks, where slight perturbations in input features are crafted to make the model exhibit a false sense of robustness in the verification phase. Such a threat is significantly important as it can mislead our evaluation of the adversarial robustness of models, which could cause unpredictable security issues when deploying substandard models in reality. More seriously, this threat seems to be pervasive and tricky: we find that many types of models suffer from this threat, and models with higher adversarial robustness tend to be more vulnerable. Furthermore, we provide the first attempt to formulate this threat, disclose its relationships with adversarial risk, and try to circumvent it via a simple countermeasure. These findings serve as a crucial reminder for practitioners to exercise caution in the verification phase, urging them to refrain from blindly trusting the exhibited adversarial robustness of models.
Dynamic Label Adversarial Training for Deep Learning Robustness Against Adversarial Attacks

Zhenyu Liu,Haoran Duan,Huizhi Liang,Yang Long,Vaclav Snasel,Guiseppe Nicosia,Rajiv Ranjan,Varun Ojha

2024-08-23

Abstract:Adversarial training is one of the most effective methods for enhancing model robustness. Recent approaches incorporate adversarial distillation in adversarial training architectures. However, we notice two scenarios of defense methods that limit their performance: (1) Previous methods primarily use static ground truth for adversarial training, but this often causes robust overfitting; (2) The loss functions are either Mean Squared Error or KL-divergence leading to a sub-optimal performance on clean accuracy. To solve those problems, we propose a dynamic label adversarial training (DYNAT) algorithm that enables the target model to gradually and dynamically gain robustness from the guide model's decisions. Additionally, we found that a budgeted dimension of inner optimization for the target model may contribute to the trade-off between clean accuracy and robust accuracy. Therefore, we propose a novel inner optimization method to be incorporated into the adversarial training. This will enable the target model to adaptively search for adversarial examples based on dynamic labels from the guiding model, contributing to the robustness of the target model. Extensive experiments validate the superior performance of our approach.

Machine Learning,Computer Vision and Pattern Recognition
Annealing Self-Distillation Rectification Improves Adversarial Training

Yu-Yu Wu,Hung-Jui Wang,Shang-Tse Chen

2024-04-13

Abstract:In standard adversarial training, models are optimized to fit one-hot labels within allowable adversarial perturbation budgets. However, the ignorance of underlying distribution shifts brought by perturbations causes the problem of robust overfitting. To address this issue and enhance adversarial robustness, we analyze the characteristics of robust models and identify that robust models tend to produce smoother and well-calibrated outputs. Based on the observation, we propose a simple yet effective method, Annealing Self-Distillation Rectification (ADR), which generates soft labels as a better guidance mechanism that accurately reflects the distribution shift under attack during adversarial training. By utilizing ADR, we can obtain rectified distributions that significantly improve model robustness without the need for pre-trained models or extensive extra computation. Moreover, our method facilitates seamless plug-and-play integration with other adversarial training techniques by replacing the hard labels in their objectives. We demonstrate the efficacy of ADR through extensive experiments and strong performances across datasets.

Machine Learning,Artificial Intelligence
Towards Enabling Meta-Learning from Target Models.

Su Lu,Han-Jia Ye,Le Gan,De-Chuan Zhan

2021-01-01

Abstract:Use the "Report an Issue" link to request a name change.
Recent Advances in Adversarial Training for Adversarial Robustness

Tao Bai,Jinqi Luo,Jun Zhao,Bihan Wen,Qian Wang

DOI: https://doi.org/10.24963/ijcai.2021/591

2021-08-01

Abstract:Adversarial training is one of the most effective approaches for deep learning models to defend against adversarial examples. Unlike other defense strategies, adversarial training aims to enhance the robustness of models intrinsically. During the past few years, adversarial training has been studied and discussed from various aspects, which deserves a comprehensive review. For the first time in this survey, we systematically review the recent progress on adversarial training for adversarial robustness with a novel taxonomy. Then we discuss the generalization problems in adversarial training from three perspectives and highlight the challenges which are not fully tackled. Finally, we present potential future directions.
Can Adversarial Training Be Manipulated By Non-Robust Features?

Lue Tao,Lei Feng,Hongxin Wei,Jinfeng Yi,Sheng-Jun Huang,Songcan Chen

DOI: https://doi.org/10.48550/arxiv.2201.13329

2022-01-01

Abstract: Adversarial training, originally designed to resist test-time adversarial examples, has shown to be promising in mitigating training-time availability attacks. This defense ability, however, is challenged in this paper. We identify a novel threat model named stability attack, which aims to hinder robust availability by slightly manipulating the training data. Under this threat, we show that adversarial training using a conventional defense budget $\epsilon$ provably fails to provide test robustness in a simple statistical setting, where the non-robust features of the training data can be reinforced by $\epsilon$-bounded perturbation. Further, we analyze the necessity of enlarging the defense budget to counter stability attacks. Finally, comprehensive experiments demonstrate that stability attacks are harmful on benchmark datasets, and thus the adaptive defense is necessary to maintain robustness. Our code is available at https://github.com/TLMichael/Hypocritical-Perturbation.
Adversarial Training with Anti-adversaries

Xiaoling Zhou,Ou Wu,Nan Yang

DOI: https://doi.org/10.1109/TPAMI.2024.3432973

2024-07-24

Abstract:Adversarial training is effective in improving the robustness of deep neural networks. However, existing studies still exhibit significant drawbacks in terms of the robustness, generalization, and fairness of models. In this study, we validate the importance of different perturbation directions (i.e., adversarial and anti-adversarial) and bounds from both theoretical and practical perspectives. The influence of adversarial training on deep learning models in terms of fairness, robustness, and generalization is theoretically investigated under a more general perturbation scope that different samples can have different perturbation directions and varied perturbation bounds. Our theoretical explorations suggest that combining adversaries and anti-adversaries with varied bounds in training can be more effective in achieving better fairness among classes and a better tradeoff among robustness, accuracy, and fairness in some typical learning scenarios compared with standard adversarial training. Inspired by our theoretical findings, a more general learning objective that combines adversaries and anti-adversaries with varied bounds on each training sample is presented. To solve this objective, two adversarial training frameworks based on meta-learning and reinforcement learning are proposed, in which the perturbation direction and bound for each sample are determined by its training characteristics. Furthermore, the role of the combination strategy with varied bounds is explained from a regularization perspective. Extensive experiments under different learning scenarios verify our theoretical findings and the effectiveness of the proposed methodology.
Purple-teaming LLMs with Adversarial Defender Training

Jingyan Zhou,Kun Li,Junan Li,Jiawen Kang,Minda Hu,Xixin Wu,Helen Meng

2024-07-02

Abstract:Existing efforts in safeguarding LLMs are limited in actively exposing the vulnerabilities of the target LLM and readily adapting to newly emerging safety risks. To address this, we present Purple-teaming LLMs with Adversarial Defender training (PAD), a pipeline designed to safeguard LLMs by novelly incorporating the red-teaming (attack) and blue-teaming (safety training) techniques. In PAD, we automatically collect conversational data that cover the vulnerabilities of an LLM around specific safety risks in a self-play manner, where the attacker aims to elicit unsafe responses and the defender generates safe responses to these attacks. We then update both modules in a generative adversarial network style by training the attacker to elicit more unsafe responses and updating the defender to identify them and explain the unsafe reason. Experimental results demonstrate that PAD significantly outperforms existing baselines in both finding effective attacks and establishing a robust safe guardrail. Furthermore, our findings indicate that PAD excels in striking a balance between safety and overall model quality. We also reveal key challenges in safeguarding LLMs, including defending multi-turn attacks and the need for more delicate strategies to identify specific risks.

Computation and Language
Security Matters: A Survey on Adversarial Machine Learning

Guofu Li,Pengjia Zhu,Jin Li,Zhemin Yang,Ning Cao,Zhiyi Chen

DOI: https://doi.org/10.48550/arXiv.1810.07339

2018-10-23

Abstract:Adversarial machine learning is a fast growing research area, which considers the scenarios when machine learning systems may face potential adversarial attackers, who intentionally synthesize input data to make a well-trained model to make mistake. It always involves a defending side, usually a classifier, and an attacking side that aims to cause incorrect output. The earliest studies on the adversarial examples for machine learning algorithms start from the information security area, which considers a much wider varieties of attacking methods. But recent research focus that popularized by the deep learning community places strong emphasis on how the "imperceivable" perturbations on the normal inputs may cause dramatic mistakes by the deep learning with supposed super-human accuracy. This paper serves to give a comprehensive introduction to a range of aspects of the adversarial deep learning topic, including its foundations, typical attacking and defending strategies, and some extended studies.

Machine Learning,Cryptography and Security
Adversaries With Incentives: A Strategic Alternative to Adversarial Robustness

Maayan Ehrenberg,Roy Ganz,Nir Rosenfeld

2024-06-17

Abstract:Adversarial training aims to defend against *adversaries*: malicious opponents whose sole aim is to harm predictive performance in any way possible - a rather harsh perspective, which we assert results in unnecessarily conservative models. Instead, we propose to model opponents as simply pursuing their own goals, rather than working directly against the classifier. Employing tools from strategic modeling, our approach uses knowledge or beliefs regarding the opponent's possible incentives as inductive bias for learning. Our method of *strategic training* is designed to defend against opponents within an *incentive uncertainty set*: this resorts to adversarial learning when the set is maximal, but offers potential gains when it can be appropriately reduced. We conduct a series of experiments that show how even mild knowledge regarding the adversary's incentives can be useful, and that the degree of potential gains depends on how incentives relate to the structure of the learning task.

Machine Learning,Computer Science and Game Theory
Adversarial Amendment is the Only Force Capable of Transforming an Enemy into a Friend

Chong Yu,Tao Chen,Zhongxue Gan

2023-05-18

Abstract:Adversarial attack is commonly regarded as a huge threat to neural networks because of misleading behavior. This paper presents an opposite perspective: adversarial attacks can be harnessed to improve neural models if amended correctly. Unlike traditional adversarial defense or adversarial training schemes that aim to improve the adversarial robustness, the proposed adversarial amendment (AdvAmd) method aims to improve the original accuracy level of neural models on benign samples. We thoroughly analyze the distribution mismatch between the benign and adversarial samples. This distribution mismatch and the mutual learning mechanism with the same learning ratio applied in prior art defense strategies is the main cause leading the accuracy degradation for benign samples. The proposed AdvAmd is demonstrated to steadily heal the accuracy degradation and even leads to a certain accuracy boost of common neural models on benign classification, object detection, and segmentation tasks. The efficacy of the AdvAmd is contributed by three key components: mediate samples (to reduce the influence of distribution mismatch with a fine-grained amendment), auxiliary batch norm (to solve the mutual learning mechanism and the smoother judgment surface), and AdvAmd loss (to adjust the learning ratios according to different attack vulnerabilities) through quantitative and ablation experiments.

Artificial Intelligence,Cryptography and Security,Computer Vision and Pattern Recognition
Defense Against Adversarial Attacks Using Feature Scattering-based Adversarial Training

Haichao Zhang,Jianyu Wang

DOI: https://doi.org/10.48550/arXiv.1907.10764

2019-07-24

Computer Vision and Pattern Recognition

Abstract:We introduce a feature scattering-based adversarial training approach for improving model robustness against adversarial attacks. Conventional adversarial training approaches leverage a supervised scheme (either targeted or non-targeted) in generating attacks for training, which typically suffer from issues such as label leaking as noted in recent works. Differently, the proposed approach generates adversarial images for training through feature scattering in the latent space, which is unsupervised in nature and avoids label leaking. More importantly, this new approach generates perturbed images in a collaborative fashion, taking the inter-sample relationships into consideration. We conduct analysis on model robustness and demonstrate the effectiveness of the proposed approach through extensively experiments on different datasets compared with state-of-the-art approaches.
Adv-4-Adv: Thwarting Changing Adversarial Perturbations via Adversarial Domain Adaptation

Tianyue Zheng,Zhe Chen,Shuya Ding,Chao Cai,Jun Luo

2023-12-07

Abstract:Whereas adversarial training can be useful against specific adversarial perturbations, they have also proven ineffective in generalizing towards attacks deviating from those used for training. However, we observe that this ineffectiveness is intrinsically connected to domain adaptability, another crucial issue in deep learning for which adversarial domain adaptation appears to be a promising solution. Consequently, we proposed Adv-4-Adv as a novel adversarial training method that aims to retain robustness against unseen adversarial perturbations. Essentially, Adv-4-Adv treats attacks incurring different perturbations as distinct domains, and by leveraging the power of adversarial domain adaptation, it aims to remove the domain/attack-specific features. This forces a trained model to learn a robust domain-invariant representation, which in turn enhances its generalization ability. Extensive evaluations on Fashion-MNIST, SVHN, CIFAR-10, and CIFAR-100 demonstrate that a model trained by Adv-4-Adv based on samples crafted by simple attacks (e.g., FGSM) can be generalized to more advanced attacks (e.g., PGD), and the performance exceeds state-of-the-art proposals on these datasets.

Computer Vision and Pattern Recognition,Artificial Intelligence
Certified Federated Adversarial Training

Giulio Zizzo,Ambrish Rawat,Mathieu Sinn,Sergio Maffeis,Chris Hankin

DOI: https://doi.org/10.48550/arXiv.2112.10525

2021-12-20

Abstract:In federated learning (FL), robust aggregation schemes have been developed to protect against malicious clients. Many robust aggregation schemes rely on certain numbers of benign clients being present in a quorum of workers. This can be hard to guarantee when clients can join at will, or join based on factors such as idle system status, and connected to power and WiFi. We tackle the scenario of securing FL systems conducting adversarial training when a quorum of workers could be completely malicious. We model an attacker who poisons the model to insert a weakness into the adversarial training such that the model displays apparent adversarial robustness, while the attacker can exploit the inserted weakness to bypass the adversarial training and force the model to misclassify adversarial examples. We use abstract interpretation techniques to detect such stealthy attacks and block the corrupted model updates. We show that this defence can preserve adversarial robustness even against an adaptive attacker.

Machine Learning,Cryptography and Security
On the Effectiveness of Adversarial Training Against Backdoor Attacks.

Yinghua Gao,Dongxian Wu,Jingfeng Zhang,Guanhao Gan,Shu-Tao Xia,Gang Niu,Masashi Sugiyama

DOI: https://doi.org/10.1109/tnnls.2023.3281872

2024-01-01

Abstract:Although adversarial training (AT) is regarded as a potential defense against backdoor attacks, AT and its variants have only yielded unsatisfactory results or have even inversely strengthened backdoor attacks. The large discrepancy between expectations and reality motivates us to thoroughly evaluate the effectiveness of AT against backdoor attacks across various settings for AT and backdoor attacks. We find that the type and budget of perturbations used in AT are important, and AT with common perturbations is only effective for certain backdoor trigger patterns. Based on these empirical findings, we present some practical suggestions for backdoor defense, including relaxed adversarial perturbation and composite AT. This work not only boosts our confidence in AT's ability to defend against backdoor attacks but also provides some important insights for future research.

Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training.

Attack As Defense: Characterizing Adversarial Examples Using Robustness.

Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training

Adversarial for Good – Defending Training Data Privacy with Adversarial Attack Wisdom

Boosting Adversarial Training in Safety-Critical Systems Through Boundary Data Selection

Adversarial Training: A Survey

On the Vulnerability of Adversarially Trained Models Against Two-faced Attacks

Dynamic Label Adversarial Training for Deep Learning Robustness Against Adversarial Attacks

Annealing Self-Distillation Rectification Improves Adversarial Training

Towards Enabling Meta-Learning from Target Models.

Recent Advances in Adversarial Training for Adversarial Robustness

Can Adversarial Training Be Manipulated By Non-Robust Features?

Adversarial Training with Anti-adversaries

Purple-teaming LLMs with Adversarial Defender Training

Security Matters: A Survey on Adversarial Machine Learning

Adversaries With Incentives: A Strategic Alternative to Adversarial Robustness

Adversarial Amendment is the Only Force Capable of Transforming an Enemy into a Friend

Defense Against Adversarial Attacks Using Feature Scattering-based Adversarial Training

Adv-4-Adv: Thwarting Changing Adversarial Perturbations via Adversarial Domain Adaptation

Certified Federated Adversarial Training

On the Effectiveness of Adversarial Training Against Backdoor Attacks.