RES-SE-NET: Boosting Performance of Resnets by Enhancing Bridge-connections

Varshaneya V,Balasubramanian S,Darshan Gera

DOI: https://doi.org/10.48550/arXiv.1902.06066

2019-02-16

Abstract:One of the ways to train deep neural networks effectively is to use residual connections. Residual connections can be classified as being either identity connections or bridge-connections with a reshaping convolution. Empirical observations on CIFAR-10 and CIFAR-100 datasets using a baseline Resnet model, with bridge-connections removed, have shown a significant reduction in accuracy. This reduction is due to lack of contribution, in the form of feature maps, by the bridge-connections. Hence bridge-connections are vital for Resnet. However, all feature maps in the bridge-connections are considered to be equally important. In this work, an upgraded architecture "Res-SE-Net" is proposed to further strengthen the contribution from the bridge-connections by quantifying the importance of each feature map and weighting them accordingly using Squeeze-and-Excitation (SE) block. It is demonstrated that Res-SE-Net generalizes much better than Resnet and SE-Resnet on the benchmark CIFAR-10 and CIFAR-100 datasets.

Machine Learning,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that in the ResNet architecture, the role of bridge - connections has not been fully utilized, and all feature maps are considered equally important in bridge - connections. The author observes that after removing the bridge - connections, the accuracy of using the baseline ResNet model on the CIFAR - 10 and CIFAR - 100 datasets drops significantly, which indicates the importance of bridge - connections for ResNet. However, all feature maps in bridge - connections are not weighted according to their importance. Therefore, the paper proposes an improved architecture - Res - SE - Net. By introducing Squeeze - and - Excitation (SE) blocks to quantify the importance of each feature map and weight it accordingly, the contribution of bridge - connections is enhanced and the model performance is improved.

Specifically, Res - SE - Net adds SE blocks in each bridge - connection, so that feature maps transferred from one block to another can be assigned different weights according to the content they carry, thus using the information of feature maps more effectively and improving the generalization ability of the model. Experimental results show that Res - SE - Net performs better than the baseline ResNet and SE - ResNet on the CIFAR - 10 and CIFAR - 100 datasets.