Abstract: Deep neural networks (DNNs) are vulnerable to adversarial examples, in which DNNs are misled to false outputs due to inputs containing imperceptible perturbations. Adversarial training, a reliable and effective method of defense, may significantly reduce the vulnerability of neural networks and becomes the de facto standard for robust learning. While many recent works practice the data-centric philosophy, such as how to generate better adversarial examples or use generative models to produce additional training data, we look back to the models themselves and revisit the adversarial robustness from the perspective of deep feature distribution as an insightful complementarity. In this paper, we propose Branch Orthogonality adveRsarial Training (BORT) to obtain state-of-the-art performance with solely the original dataset for adversarial training. To practice our design idea of integrating multiple orthogonal solution spaces, we leverage a simple and straightforward multi-branch neural network that eclipses adversarial attacks with no increase in inference time. We heuristically propose a corresponding loss function, branch-orthogonal loss, to make each solution space of the multi-branch model orthogonal. We evaluate our approach on CIFAR-10, CIFAR-100, and SVHN against \ell_{\infty} norm-bounded perturbations of size \epsilon = 8/255, respectively. Exhaustive experiments are conducted to show that our method goes beyond all state-of-the-art methods without any tricks. Compared to all methods that do not use additional data for training, our models achieve 67.3% and 41.5% robust accuracy on CIFAR-10 and CIFAR-100 (improving upon the state-of-the-art by +7.23% and +9.07%). We also outperform methods using a training set with a far larger scale than ours. All our models and codes are available online at https://github.com/huangd1999/BORT.

On the Stability of Multi-branch Network

CMNN: Coupled Modular Neural Network.

A Closer Look at Branch Classifiers of Multi-exit Architectures

Tree-Like Branching Network for Multi-class Classification

Stabilize deep ResNet with a sharp scaling factor τ

Branch-Tuning: Balancing Stability and Plasticity for Continual Self-Supervised Learning

Connectivity Learning in Multi-Branch Networks

Multi-depth branch network for efficient image super-resolution

Adaptive Multi-Level Firing for Direct Training Deep Spiking Neural Networks

Measuring and Mitigating Local Instability in Deep Neural Networks

Multi-branch Attentive Transformer

Improving the Robustness of Deep Neural Networks via Stability Training

On the Trade-Off between Stability and Representational Capacity in Graph Neural Networks

Graph Networks Stand Strong: Enhancing Robustness via Stability Constraints.

EMBANet: A Flexible Efffcient Multi-branch Attention Network

Stable Architectures for Deep Neural Networks

Two Heads are Better than One: Robust Learning Meets Multi-branch Models

An Adaptive and Stability-Promoting Layerwise Training Approach for Sparse Deep Neural Network Architecture

Learning Unified Representations for Multi-Resolution Face Recognition

Online Stabilization of Spiking Neural Networks.