Abstract:Deep equilibrium (DEQ) models replace the multiple-layer stacking of conventional deep networks with a fixed-point iteration of a single-layer transformation. Having been demonstrated to be competitive in a variety of real-world scenarios, the adversarial robustness of general DEQs becomes increasingly crucial for their reliable deployment. Existing works improve the robustness of general DEQ models with the widely-used adversarial training (AT) framework, but they fail to exploit the structural uniquenesses of DEQ models. To this end, we interpret DEQs through the lens of neural dynamics and find that AT under-regulates intermediate states. Besides, the intermediate states typically provide predictions with a high prediction entropy. Informed by the correlation between the entropy of dynamical systems and their stability properties, we propose reducing prediction entropy by progressively updating inputs along the neural dynamics. During AT, we also utilize random intermediate states to compute the loss function. Our methods regulate the neural dynamics of DEQ models in this manner. Extensive experiments demonstrate that our methods substantially increase the robustness of DEQ models and even outperform the strong deep network baselines.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to improve the robustness of the Deep Equilibrium (DEQ) model under adversarial attacks. Specifically, the author points out that although the DEQ model performs well in various practical scenarios, its adversarial robustness has not been fully studied and optimized. Existing works mainly improve the robustness of the DEQ model through the widely - used Adversarial Training (AT) framework, but these methods fail to fully utilize the unique structural characteristics of the DEQ model. Therefore, the goal of the paper is to further enhance its adversarial robustness by explicitly regulating the neural dynamic process in the DEQ model. ### Main contributions of the paper 1. **Neural dynamics perspective**: - The author reinterprets the DEQ model from the perspective of neural dynamics and finds that existing adversarial training methods mainly focus on the final equilibrium state and insufficiently regulate the intermediate states. This leads to the fact that even after adversarial training, the intermediate states of the DEQ model are still vulnerable to attacks. 2. **Input entropy reduction framework**: - In order to reduce the prediction entropy, the author proposes a method of gradually updating the input. By gradually adjusting the input during the testing phase, the intermediate states of the model gradually tend to a low - entropy state, thereby improving the robustness of the model. Specifically, the author defines an optimization framework that gradually updates the input through the gradient descent method to reduce the prediction entropy of the final state. 3. **Random intermediate - state loss calculation**: - In addition to gradually updating the input during the testing phase, the author also proposes to use randomly selected intermediate states to calculate the loss function during the training phase. This method can explicitly regulate the intermediate states without violating the memory constraints of the DEQ model. In this way, the model can exhibit higher robustness under adversarial attacks. ### Experimental results - **Benchmark comparison**: - The author conducted extensive experiments on the CIFAR - 10 dataset to compare the robustness of traditional deep networks (such as ResNet - 18) and the DEQ model. The results show that after using the method proposed by the author, the robustness of the DEQ model is significantly improved and even exceeds the performance of strong baseline models (such as ResNet - 18). - **Intermediate - state attack**: - The author also conducted comprehensive intermediate - state attack experiments to verify the effectiveness of intermediate - state attacks. Compared with off - the - shelf attack methods, intermediate - state attacks can always reduce the white - box adversarial robustness of the model to a greater extent. ### Conclusion By explicitly regulating the neural dynamic process in the DEQ model, this paper successfully improves the robustness of the DEQ model under adversarial attacks. These methods not only perform excellently on the standard white - box adversarial robustness evaluation benchmark but also show stronger robustness in intermediate - state attacks. This research result provides new ideas and methods for the future robustness optimization of the DEQ model.

Improving Adversarial Robustness of DEQs with Explicit Regulations Along the Neural Dynamics

Improving Adversarial Robustness of Deep Equilibrium Models with Explicit Regulations along the Neural Dynamics

A Closer Look at the Adversarial Robustness of Deep Equilibrium Models

Defending Against Adversarial Attacks via Neural Dynamic System

Lyapunov-Stable Deep Equilibrium Models

CerDEQ: Certifiable Deep Equilibrium Model.

Interpreting and Evaluating Neural Network Robustness

Understanding Adversarial Attacks on Observations in Deep Reinforcement Learning

Dynamic ensemble selection based on Deep Neural Network Uncertainty Estimation for Adversarial Robustness

Adversarial Robust Decision-Making under Uncertainty Learning and Dynamic Ensemble Selection

An Integrated Approach to Produce Robust Models with High Efficiency

Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise

Layer-wise Adversarial Defense: an ODE Perspective

Adversarial Robustness of Stabilized NeuralODEs Might be from Obfuscated Gradients

Revisiting Ensembles in an Adversarial Context: Improving Natural Accuracy

Certifiable Robustness to Adversarial State Uncertainty in Deep Reinforcement Learning

Enhance Diffusion to Improve Robust Generalization

Positive concave deep equilibrium models

Improve Adversarial Robustness Via Probabilistic Distributions Decoupled Network While Guaranteeing Clean Performance

Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations

Robust Deep Reinforcement Learning with Adaptive Adversarial Perturbations in Action Space