Randomized Adversarial Training via Taylor Expansion

Gaojie Jin,Xinping Yi,Dengyu Wu,Ronghui Mu,Xiaowei Huang

2023-03-19

Abstract:In recent years, there has been an explosion of research into developing more robust deep neural networks against adversarial examples. Adversarial training appears as one of the most successful methods. To deal with both the robustness against adversarial examples and the accuracy over clean examples, many works develop enhanced adversarial training methods to achieve various trade-offs between them. Leveraging over the studies that smoothed update on weights during training may help find flat minima and improve generalization, we suggest reconciling the robustness-accuracy trade-off from another perspective, i.e., by adding random noise into deterministic weights. The randomized weights enable our design of a novel adversarial training method via Taylor expansion of a small Gaussian noise, and we show that the new adversarial training method can flatten loss landscape and find flat minima. With PGD, CW, and Auto Attacks, an extensive set of experiments demonstrate that our method enhances the state-of-the-art adversarial training methods, boosting both robustness and clean accuracy. The code is available at <a class="link-external link-https" href="https://github.com/Alexkael/Randomized-Adversarial-Training" rel="external noopener nofollow">this https URL</a>.

Machine Learning,Artificial Intelligence

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper primarily aims to address the trade-off between the robustness of deep neural networks against adversarial examples and the accuracy on clean samples. Specifically, the paper proposes a novel stochastic adversarial training method that introduces random noise through Taylor expansion to smooth weight updates and find flatter minima in the loss landscape. This method is designed to simultaneously improve the model's robustness to adversarial examples and accuracy on clean samples.

### Main Contributions
1. **Theoretical Analysis**: By introducing random weights, the paper theoretically explores the smoothness of weight updates and the flatness of the loss landscape, demonstrating that this method can find flatter minima.
2. **New Method Proposal**: Based on Taylor expansion, a new adversarial training method is proposed, which optimizes the first and second-order terms of the loss function by adding small Gaussian noise to the weights, thereby enhancing the model's robustness.
3. **Experimental Validation**: Extensive experiments validate the effectiveness of this method. On multiple datasets (CIFAR-10, CIFAR-100, SVHN) and different network architectures (ResNet, WideResNet, VGG, MobileNetV2), this method significantly improves adversarial robustness and accuracy on clean samples. Notably, under Auto Attack, the performance surpasses some existing methods.