Abstract:Deep learning has revolutionized computing in many real-world applications, arguably due to its remarkable performance and extreme convenience as an end-to-end solution. However, deep learning models can be costly to train and to use, especially for those large-scale models, making it necessary to optimize the original overly complicated models into smaller ones in scenarios with limited resources such as mobile applications or simply for resource saving. The key question in such model optimization is, how can we effectively identify and measure the redundancy in a deep learning model structure. While several common metrics exist in the popular model optimization techniques to measure the performance of models after optimization, they are not able to quantitatively inform the degree of remaining redundancy. To address the problem, we present a novel testing approach, i.e., RedTest, which proposes a novel testing metric called Model Structural Redundancy Score (MSRS) to quantitatively measure the degree of redundancy in a deep learning model structure. We first show that MSRS is effective in both revealing and assessing the redundancy issues in many state-of-the-art models, which urgently calls for model optimization. Then, we utilize MSRS to assist deep learning model developers in two practical application scenarios: 1) in Neural Architecture Search, we design a novel redundancy-aware algorithm to guide the search for the optimal model structure and demonstrate its effectiveness by comparing it to existing standard NAS practice; 2) in the pruning of large-scale pre-trained models, we prune the redundant layers of pre-trained models with the guidance of layer similarity to derive less redundant ones of much smaller size. Extensive experimental results demonstrate that removing such redundancy has a negligible effect on the model utility.

A Multi-objective Optimization Model for Redundancy Reduction in Convolutional Neural Networks

Multi-objective optimization for reducing feature maps redundancy in CNNs

KRR-CNN: kernels redundancy reduction in convolutional neural networks

DCCD: Reducing Neural Network Redundancy Via Distillation

Accelerating Convolutional Neural Networks by Removing Interspatial and Interkernel Redundancies.

A New Measure of Model Redundancy for Compressed Convolutional Neural Networks

Redundancy-Reduced MobileNet Acceleration on Reconfigurable Logic for ImageNet Classification

Reducing Overfitting in Deep Convolutional Neural Networks Using Redundancy Regularizer

Kernel Orthogonality does not necessarily imply a Decrease in Feature Map Redundancy in CNNs: Convolutional Similarity Minimization

Sensitivity-Oriented Layer-Wise Acceleration and Compression for Convolutional Neural Network.

Fast Cnn Pruning Via Redundancy-Aware Training

Estimating Redundancy-Reliability of CNNs Based on Strip-Median Attributes

Adaptive Loss Function for Super Resolution Neural Networks Using Convex Optimization Techniques

MO-CNN: Multiobjective Optimization of Convolutional Neural Networks for Hyperspectral Image Classification

Optimization of CNN through Novel Training Strategy for Visual Classification Problems

RedTest: Towards Measuring Redundancy in Deep Neural Networks Effectively

Optimizing Convolutional Neural Network Architecture

Interpreting Convolutional Neural Network by Joint Evaluation of Multiple Feature Maps and an Improved NSGA-II Algorithm

Optimizing Convolutional Neural Network Architectures

Optimal Hyperparameters and Structure Setting of Multi-Objective Robust CNN Systems via Generalized Taguchi Method and Objective Vector Norm

SCConv: Spatial and Channel Reconstruction Convolution for Feature Redundancy