Abstract:Deep neural networks have achieved impressive success in various applications, but they face challenges when deployed on mobile devices due to limited computational resources. One of the main reasons for this challenge is that these networks employ too many weight maps of convolution layers to generate redundant feature maps to ensure a thorough understanding of the input data, which leads to plenty of parameters and high computational costs. This paper proposes a method to exploit this redundancy efficiently and generate more lightweight convolutional neural networks. First, we employ the peak signal-to-noise ratio (PSNR) to quantitatively verify the similarity in feature maps of the convolution layer, which is also called redundancy. We also find the similarity correspondence between the feature maps and the weight maps. Second, inspired by the analysis, we propose a cost-efficient feature transform module (FTM) to generate redundant feature maps for replacing standard convolution layers and getting lightweight convolutional neural networks (CNNs). The FTM adopts a point-wise and a depth-wise convolution to obtain the prime weight maps which could extract distinguished feature maps based on the similarity correspondence. Then, these meaningful feature maps are broadcast by an efficient neighbor-1D convolution, ensuring the feature redundancy of the network information flow. Extensive experiments on image classification benchmark datasets (CIFAR-10, CIFAR-100, ImageNet) show that the FTM outperforms existing methods in terms of efficiency. In addition, if used as a drop-in replacement for standard architectures such as ResNet-50, the FTM variant could achieve a comparable score with only 21% parameters of the original layer. The code of the proposed method is available at: https://github.com/li-ju-bazhong/Efficient-feature-transform-module.

Efficient densely connected convolutional neural networks

Convolutional Networks with Dense Connectivity

Highly shared Convolutional Neural Networks

Learning Efficient Convolutional Networks Through Network Slimming.

Low-res MobileNet: An efficient lightweight network for low-resolution image classification in resource-constrained scenarios

CondenseNet: An Efficient DenseNet Using Learned Group Convolutions

LDCNet: A Lightweight Multi-Scale Convolutional Neural Network Using Local Dense Connectivity for Image Recognition

Efficient channel expansion and pyramid depthwise-pointwise-depthwise neural networks

ChannelNets: Compact and Efficient Convolutional Neural Networks via Channel-Wise Convolutions

Connection Reduction of DenseNet for Image Recognition

Effnet: An Efficient Structure for Convolutional Neural Networks

HENet:A Highly Efficient Convolutional Neural Networks Optimized for Accuracy, Speed and Storage

EfficientNet-eLite: Extremely Lightweight and Efficient CNN Models for Edge Devices by Network Candidate Search

Memory-Efficient Implementation of DenseNets

DualConv: Dual Convolutional Kernels for Lightweight Deep Neural Networks

Grouped Pointwise Convolutions Significantly Reduces Parameters in EfficientNet

Efficient feature transform module

A Pre-defined Sparse Kernel Based Convolution for Deep CNNs

Lightweight Pixel Difference Networks for Efficient Visual Representation Learning

Face Recognition Based on Lightweight Convolutional Neural Networks

SparseNet: A Sparse DenseNet for Image Classification