Abstract:Deep neural networks have achieved impressive success in various applications, but they face challenges when deployed on mobile devices due to limited computational resources. One of the main reasons for this challenge is that these networks employ too many weight maps of convolution layers to generate redundant feature maps to ensure a thorough understanding of the input data, which leads to plenty of parameters and high computational costs. This paper proposes a method to exploit this redundancy efficiently and generate more lightweight convolutional neural networks. First, we employ the peak signal-to-noise ratio (PSNR) to quantitatively verify the similarity in feature maps of the convolution layer, which is also called redundancy. We also find the similarity correspondence between the feature maps and the weight maps. Second, inspired by the analysis, we propose a cost-efficient feature transform module (FTM) to generate redundant feature maps for replacing standard convolution layers and getting lightweight convolutional neural networks (CNNs). The FTM adopts a point-wise and a depth-wise convolution to obtain the prime weight maps which could extract distinguished feature maps based on the similarity correspondence. Then, these meaningful feature maps are broadcast by an efficient neighbor-1D convolution, ensuring the feature redundancy of the network information flow. Extensive experiments on image classification benchmark datasets (CIFAR-10, CIFAR-100, ImageNet) show that the FTM outperforms existing methods in terms of efficiency. In addition, if used as a drop-in replacement for standard architectures such as ResNet-50, the FTM variant could achieve a comparable score with only 21% parameters of the original layer. The code of the proposed method is available at: https://github.com/li-ju-bazhong/Efficient-feature-transform-module.

FSConv: Flexible and Separable Convolution for Convolutional Neural Networks Compression

SCConv: Spatial and Channel Reconstruction Convolution for Feature Redundancy

FALCON: Lightweight and Accurate Convolution

SlimConv: Reducing Channel Redundancy in Convolutional Neural Networks by Features Recombining

FSNet: Compression of Deep Convolutional Neural Networks by Filter Summary

Sensitivity-Oriented Layer-Wise Acceleration and Compression for Convolutional Neural Network.

Compression of Convolutional Neural Networks With Divergent Representation of Filters

Comb Convolution for Efficient Convolutional Architecture

Structured Pruning Via Feature Channels Similarity and Mutual Learning for Convolutional Neural Network Compression

Packing Convolutional Neural Networks in the Frequency Domain

Self-Fusion Convolutional Neural Networks.

Towards Evolutionary Compression.

Exploiting Kernel Sparsity and Entropy for Interpretable CNN Compression

Design Light-weight 3D Convolutional Networks for Video Recognition Temporal Residual, Fully Separable Block, and Fast Algorithm

TEC-CNN: Towards Efficient Compressing Convolutional Neural Nets with Low-rank Tensor Decomposition

An End-to-End Compression Framework Based on Convolutional Neural Networks

Exploring Linear Relationship in Feature Map Subspace for ConvNets Compression.

Towards Convolutional Neural Networks Compression Via Global Error Reconstruction.

Efficient feature transform module

Interpretable Relative Squeezing bottleneck design for compact convolutional neural networks model.

XSepConv: Extremely Separated Convolution for Efficient Deep Networks with Large Kernels