Is ReLU Adversarially Robust?

Korn Sooksatra,Greg Hamerly,Pablo Rivas

2024-05-07

Abstract:The efficacy of deep learning models has been called into question by the presence of adversarial examples. Addressing the vulnerability of deep learning models to adversarial examples is crucial for ensuring their continued development and deployment. In this work, we focus on the role of rectified linear unit (ReLU) activation functions in the generation of adversarial examples. ReLU functions are commonly used in deep learning models because they facilitate the training process. However, our empirical analysis demonstrates that ReLU functions are not robust against adversarial examples. We propose a modified version of the ReLU function, which improves robustness against adversarial examples. Our results are supported by an experiment, which confirms the effectiveness of our proposed modification. Additionally, we demonstrate that applying adversarial training to our customized model further enhances its robustness compared to a general model.

Machine Learning,Artificial Intelligence

What problem does this paper attempt to address?

This paper mainly explores the robustness issue of the commonly used activation function ReLU in deep learning models against adversarial attacks. The authors found that ReLU function is not robust when generating adversarial examples, and proposed a modified version of ReLU function by capping its maximum value to improve its resistance against adversarial attacks. The experiments show that this modification can enhance the model's robustness, and the model's robustness is stronger when the maximum value is reduced. However, this method is mainly applicable to small-scale datasets, and for medium to large-scale datasets, additional techniques may be needed due to the unresolved gradient vanishing problem. In addition, the paper also studies how the size and order of layers affect the robustness of the customized ReLU function in the model, and finds that restricting ReLU function in the bottleneck layer can provide the maximum robustness. Finally, the paper further supports the effectiveness of the modified ReLU function through zero-gradient experiments and sensitivity mapping, and investigates the impact of adversarial training on the customized model, demonstrating its ability to enhance the model's robustness.

Is ReLU Adversarially Robust?

Attack As Defense: Characterizing Adversarial Examples Using Robustness.

Dynamic-Max-Value ReLU Functions for Adversarially Robust Machine Learning Models

Improving Model Robustness Against Adversarial Examples with Redundant Fully Connected Layer.

JumpReLU: A Retrofit Defense Strategy for Adversarial Attacks

A randomized gradient-free attack on ReLU networks

Semidefinite relaxations for certifying robustness to adversarial examples

Adversarial Training of Two-Layer Polynomial and ReLU Activation Networks via Convex Optimization

Why Adversarial Training of ReLU Networks Is Difficult?

Adversarial robustness of deep reinforcement learning-based intrusion detection

Robust nonparametric regression based on deep ReLU neural networks

Towards Deep Learning Models Resistant to Adversarial Attacks

Local Competition and Uncertainty for Adversarial Robustness in Deep Learning

Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data

Opportunities and Challenges in Deep Learning Adversarial Robustness: A Survey

On Model Robustness Against Adversarial Examples

Can Implicit Bias Imply Adversarial Robustness?

Optimism in the Face of Adversity: Understanding and Improving Deep Learning Through Adversarial Robustness

The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networks

It begins with a boundary: A geometric view on probabilistically robust learning