Abstract:Adversarial examples are usually generated by adding adversarial perturbations on clean samples, designed to deceive the model to make wrong classifications. Adversarial robustness refers to the ability of a model to resist adversarial attacks. And currently, a mainstream method to enhance adversarial robustness is the Projected Gradient Descent (PGD). However, PGD is often criticized for being time-consuming during constructing adversarial examples. Fast adversarial training can improve the adversarial robustness in shorter time, but it only can train for a limited number of epochs, leading to sub-optimal performance. This paper demonstrates that the multi-exit network can reduce the impact of adversarial perturbations by outputting easily identified samples at early exits. Therefore, we can improve the adversarial robustness. Further, we find that the multi-exit network can prevent catastrophic overfitting existing in single-step adversarial training. Specifically, we find that, in the multi-exit network, (1) the norm of weights at a fully connected layer in a non-overfitted exit is much smaller than that in an overfitted exit; and (2) catastrophic overfitting occurs when the late exits have weight norms larger than the early exits. Based on these findings, we propose an approach to alleviating the catastrophic overfitting of the multi-exit network. Compared to PGD adversarial training, our approach can train a model with decreased time complexity and increased empirical robustness. Extensive experiments have been conducted to evaluate our approach against various adversarial attacks, and the experimental results demonstrate superior robustness accuracies on CIFAR-10, CIFAR-100 and SVHN.

Self-supervised efficient sample weighting for multi-exit networks

Learning to Weight Samples for Dynamic Early-Exiting Networks.

Elastic DNN Inference with Unpredictable Exit in Edge Computing

Unlocking the Non-deterministic Computing Power with Memory-Elastic Multi-Exit Neural Networks

MMExit: Enabling Fast and Efficient Multi-modal DNN Inference with Adaptive Network Exits

Deep Feature Surgery: Towards Accurate and Efficient Multi-Exit Networks

Finding the SWEET Spot: Analysis and Improvement of Adaptive Inference in Low Resource Settings

Sample Weighting: an Inherent Approach for Outlier Suppressing Discriminant Analysis

Towards improving fast adversarial training in multi-exit network

A hybrid sampling method for highly imbalanced and overlapped data classification with complex distribution

Consistency Training of Multi-exit Architectures for Sensor Data

Fair Class-Incremental Learning using Sample Weighting

Early-Exit with Class Exclusion for Efficient Inference of Neural Networks

E$^2$CM: Early Exit via Class Means for Efficient Supervised and Unsupervised Learning

Learning a Unified Sample Weighting Network for Object Detection.

Structural-Entropy-Based Sample Selection for Efficient and Effective Learning

Sampling weights of deep neural networks

FedSW: Federated learning with adaptive sample weights

Dynamic Transformers Provide a False Sense of Efficiency

SSE-SAM: Balancing Head and Tail Classes Gradually through Stage-Wise SAM

SEENN: Towards Temporal Spiking Early-Exit Neural Networks