Abstract:Depth is one of the key factors behind the success of convolutional neural networks (CNNs). Since ResNet, we are able to train very deep CNNs as the gradient vanishing issue has been largely addressed by the introduction of skip connections. However, we observe that, when the depth is very large, the intermediate layers (especially shallow layers) may fail to receive sufficient supervision from the loss due to the severe transformation through a long backpropagation path. As a result, the representation power of intermediate layers can be very weak and the model becomes very redundant with limited performance. In this paper, we first investigate the supervision vanishing issue in existing backpropagation (BP) methods. And then, we propose to address it via an effective method, called Multi-way BP (MW-BP), which relies on multiple auxiliary losses added to the intermediate layers of the network. The proposed MW-BP method can be applied to most deep architectures with slight modifications, such as ResNet and MobileNet. Our method often gives rise to much more compact models (denoted by "Mw+Architecture") than existing methods. For example, MwResNet-44 with 44 layers performs better than ResNet-110 with 110 layers on CIFAR-10 and CIFAR-100. More critically, the resultant models even outperform the light models obtained by state-of-the-art model compression methods. Last, our method inherently produces multiple compact models with different depths at the same time, which is helpful for model selection.

Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networks

Embedding Principle of Loss Landscape of Deep Neural Networks

On the Depth of Deep Neural Networks: A Theoretical View

Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes.

Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks

Visualizing, Rethinking, and Mining the Loss Landscape of Deep Neural Networks

Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks

Towards Interpreting Deep Neural Networks Via Layer Behavior Understanding

Improved Appearance Loss for Deep Estimation of Image Depth

The Shallow End: Empowering Shallower Deep-Convolutional Networks through Auxiliary Outputs

Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escaping, and Network Embedding

On the importance of network architecture in training very deep neural networks

Deep Kernel Network Embedding

Going Deeper, Generalizing Better: an Information-Theoretic View for Deep Learning.

Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion

Layer-Peeled Model: Toward Understanding Well-Trained Deep Neural Networks

Visualizing the Loss Landscape of Neural Nets

Depth Selection for Deep ReLU Nets in Feature Extraction and Generalization

Depth Degeneracy in Neural Networks: Vanishing Angles in Fully Connected ReLU Networks on Initialization