Abstract:Deep neural networks have achieved impressive success in various applications, but they face challenges when deployed on mobile devices due to limited computational resources. One of the main reasons for this challenge is that these networks employ too many weight maps of convolution layers to generate redundant feature maps to ensure a thorough understanding of the input data, which leads to plenty of parameters and high computational costs. This paper proposes a method to exploit this redundancy efficiently and generate more lightweight convolutional neural networks. First, we employ the peak signal-to-noise ratio (PSNR) to quantitatively verify the similarity in feature maps of the convolution layer, which is also called redundancy. We also find the similarity correspondence between the feature maps and the weight maps. Second, inspired by the analysis, we propose a cost-efficient feature transform module (FTM) to generate redundant feature maps for replacing standard convolution layers and getting lightweight convolutional neural networks (CNNs). The FTM adopts a point-wise and a depth-wise convolution to obtain the prime weight maps which could extract distinguished feature maps based on the similarity correspondence. Then, these meaningful feature maps are broadcast by an efficient neighbor-1D convolution, ensuring the feature redundancy of the network information flow. Extensive experiments on image classification benchmark datasets (CIFAR-10, CIFAR-100, ImageNet) show that the FTM outperforms existing methods in terms of efficiency. In addition, if used as a drop-in replacement for standard architectures such as ResNet-50, the FTM variant could achieve a comparable score with only 21% parameters of the original layer. The code of the proposed method is available at: https://github.com/li-ju-bazhong/Efficient-feature-transform-module.

RepGhost: A Hardware-Efficient Ghost Module via Re-parameterization

GhostNet: More Features from Cheap Operations

GhostNets on Heterogeneous Devices via Cheap Operations

L-GhostNet: Extract Better Quality Features

DyRep: Bootstrapping Training with Dynamic Re-parameterization

GhostNetV2: Enhance Cheap Operation with Long-Range Attention

Lightweight Generative Adversarial Networks Based on Ghost Module.

RepNeXt: A Fast Multi-Scale CNN using Structural Reparameterization

Ghost-Depth: A Lightweight Encoder-Decoder Network for Monocular Depth Estimation.

GhostViT: Expediting Vision Transformers Via Cheap Operations

Optimized Pointwise Convolution Operation by Ghost Blocks

RDPNet: a Single-Path Lightweight CNN with Re-Parameterization for CPU-type Edge Devices

A Reconfigurable Spatial Architecture for Energy-Efficient Inception Neural Networks

LowFormer: Hardware Efficient Design for Convolutional Transformer Backbones

GhostFormer: Efficiently amalgamated CNN-transformer architecture for object detection

Efficient feature transform module

GhostNetV3: Exploring the Training Strategies for Compact Models

RepAct: The Re-parameterizable Adaptive Activation Function

Memory-Efficient Implementation of DenseNets

A Scatter-and-Gather Spiking Convolutional Neural Network on a Reconfigurable Neuromorphic Hardware

GhostSR: Learning Ghost Features for Efficient Image Super-Resolution