Abstract:Convolution neural networks (CNNs) have achieved great success in various computer vision tasks, but they are still suffering from the heavy computation costs, which are mainly resulted from the substantial redundancy of the feature maps. In order to reduce these redundancy, we proposed a simple but effective frequency-enhanced feature distillation strategy to train an inference-accelerated network with a pre-trained model. Traditionally, one CNN can be regarded as a hierarchical structure, which can generate the low-level, middle-level and high-level feature maps from different convolution layers. In order to accelerate the inference time of CNNs, in this paper, we propose to resize the low-level and middle-level feature maps to smaller scales to reduce the spatial computation costs of CNNs. A frequency-enhanced feature distillation training strategy with a pre-trained model is then used to help the inference-accelerated network to maintain the core information after resizing the feature maps. To be specific, the original pre-trained network and the inference-accelerated network with resized feature maps are regarded as the teacher network and student network respectively. Considering that the low-frequency domain of the feature maps contribute the most parts to the final classification, we then transform the feature maps of different levels into a frequency-enhanced feature space, which highlights the low-frequency features for both the teacher and student networks. The frequency-enhanced features are used to transfer the knowledge from the teacher network to the student network. At the same time, knowledge for the final classification, i.e., the classification feature and predicted probabilities, are also used for distillation. Experiments on multiple databases based on various network structure types, e.g., ResNet, Res2Net, MobileNetV2, and ConvNeXt, have shown that with the proposed frequency-enhanced feature distillation training strategy, our method could get an inference-accelerated network with comparable performance and much less computation cost.

A Hybrid Model in Transfer Learning Based Pre-Trained Model and a Scale Factor Updating

Towards Inadequately Pre-trained Models in Transfer Learning

Pay Attention to Convolution Filters: Towards Fast and Accurate Fine-Grained Transfer Learning

Two-Stage Model Compression and Acceleration: Optimal Student Network for Better Performance

An improved transfer learning algorithm using integration method and its application in image segmentation and recognition

Convolution neural network compression method with scale factor

Multi-scale Convolution Aggregation and Stochastic Feature Reuse for DenseNets

Coarse-to-fine Trained Multi-Scale Convolutional Neural Networks for Image Classification.

Using Human-like Mechanism to Weaken Effect of Pre-training Weight Bias in Face-Recognition Convolutional Neural Network

Improving the Performance of Convolutional Neural Networks by Fusing Low-Level Features with Different Scales in the Preceding Stage

Learning an Inference-accelerated Network from a Pre-trained Model with Frequency-enhanced Feature Distillation

Consecutive multiscale feature learning-based image classification model

The Application of a Hybrid Transfer Algorithm Based on a Convolutional Neural Network Model and an Improved Convolution Restricted Boltzmann Machine Model in Facial Expression Recognition

Basis Scaling and Double Pruning for Efficient Inference in Network-Based Transfer Learning

Transfer of Pretrained Model Weights Substantially Improves Semi-Supervised Image Classification

Hierarchical Transfer Convolutional Neural Networks for Image Classification

SCMP-IL: an Incremental Learning Method with Super Constraints on Model Parameters

A Transfer Learning Method Based on ResNet Model

Prompt Learning with Cross-Modal Feature Alignment for Visual Domain Adaptation.

Scalenet: A Convolutional Network to Extract Multi-Scale and Fine-Grained Visual Features

AutoTune: Automatically Tuning Convolutional Neural Networks for Improved Transfer Learning