Abstract:Deep convolutional neural networks are known to be unstable during training at high learning rate unless normalization techniques are employed. Normalizing weights or activations allows the use of higher learning rates, resulting in faster convergence and higher test accuracy. Batch normalization requires minibatch statistics that approximate the dataset statistics but this incurs additional compute and memory costs and causes a communication bottleneck for distributed training. Weight normalization and initialization-only schemes do not achieve comparable test accuracy. We introduce a new understanding of the cause of training instability and provide a technique that is independent of normalization and minibatch statistics. Our approach treats training instability as a spatial common mode signal which is suppressed by placing the model on a channel-wise zero-mean isocline that is maintained throughout training. Firstly, we apply channel-wise zero-mean initialization of filter kernels with overall unity kernel magnitude. At each training step we modify the gradients of spatial kernels so that their weighted channel-wise mean is subtracted in order to maintain the common mode rejection condition. This prevents the onset of mean shift. This new technique allows direct training of the test graph so that training and test models are identical. We also demonstrate that injecting random noise throughout the network during training improves generalization. This is based on the idea that, as a side effect, batch normalization performs deep data augmentation by injecting minibatch noise due to the weakness of the dataset approximation. Our technique achieves higher accuracy compared to batch normalization and for the first time shows that minibatches and normalization are unnecessary for state-of-the-art training.

Training the Convolutional Neural Network with Statistical Dependence of the Response on the Input Data Distortion

DCCD: Reducing Neural Network Redundancy Via Distillation

Distortion Robust Image Classification using Deep Convolutional Neural Network with Discrete Cosine Transform

Learning Local Distortion Visibility From Image Quality Data-sets

Examining the Impact of Blur on Recognition by Convolutional Networks

A novel statistical approach to analyze image classification

Data-Distortion Guided Self-Distillation for Deep Neural Networks

Mean Shift Rejection: Training Deep Neural Networks Without Minibatch Statistics or Normalization

How deep convolutional neural networks lose spatial information with training

A simple theory for training response of deep neural networks

Impact of Scaled Image on Robustness of Deep Neural Networks

IMAGE QUALITY ASSESSMENT BY CONVOLUTIONAL NEURAL NETWORK USING THE TID2013 DATABASE

Convolutional Neural Networks on Randomized Data

Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation

DeepCorrect: Correcting DNN Models Against Image Distortions

Radial Lens Distortion Correction Using Convolutional Neural Networks Trained with Synthesized Images

Improving the Robustness of Deep Neural Networks via Stability Training

Sparsifying and Down-scaling Networks to Increase Robustness to Distortions