Adversarial Defense Via Self-Orthogonal Randomization Super-Network.

Huanyu Bian,Dongdong Chen,Kui Zhang,Hang Zhou,Xiaoyi Dong,Wenbo Zhou,Weiming Zhang,Nenghai Yu
DOI: https://doi.org/10.1016/j.neucom.2021.04.062
IF: 6
2021-01-01
Neurocomputing
Abstract:Deep neural networks are demonstrated to be vulnerable to adversarial examples. In this paper, starting from the robustness analysis about the model ensemble, we propose a novel type of defense method named “Self-Orthogonal Randomization Super-network” (SORS). More specifically, we think the main robustness benefit from the model ensemble comes from two aspects: smaller adversarial subspace and gradient orthogonality. However, the naive model ensemble has two fundamental limitations: 1) Though ensembling more models will introduce more robustness, training too many models is infeasible and resource-consuming. 2) Since these models are usually trained independently, the gradient orthogonality among them is often partial and weak. Motivated by this, we propose to train one single super-network that consists of the exponential number of sub-networks, and explicitly constrain the gradient of different sub-networks with respect to the same input to be orthogonal. In the inference stage, at each forward pass, one sub-network will be randomly sampled. Through extensive experiments, we demonstrate that the proposed method can achieve significantly better robustness than the vanilla single model baseline and the naive model ensemble baseline. Moreover, this new type of defense strategy is also complementary to other types of defense methods and achieves state-of-the-art performance.
What problem does this paper attempt to address?