Abstract:Network pruning is a widely used technique to reduce computation cost and model size for deep neural networks. However, the typical three-stage pipeline significantly increases the overall training time. In this paper, we develop a systematic weight-pruning optimization approach based on Surrogate Lagrangian relaxation, which is tailored to overcome difficulties caused by the discrete nature of the weight-pruning problem. We prove that our method ensures fast convergence of the model compression problem, and the convergence of the SLR is accelerated by using quadratic penalties. Model parameters obtained by SLR during the training phase are much closer to their optimal values as compared to those obtained by other state-of-the-art methods. We evaluate our method on image classification tasks using CIFAR-10 and ImageNet with state-of-the-art MLP-Mixer, Swin Transformer, and VGG-16, ResNet-18, ResNet-50 and ResNet-110, MobileNetV2. We also evaluate object detection and segmentation tasks on COCO, KITTI benchmark, and TuSimple lane detection dataset using a variety of models. Experimental results demonstrate that our SLR-based weight-pruning optimization approach achieves a higher compression rate than state-of-the-art methods under the same accuracy requirement and also can achieve higher accuracy under the same compression rate requirement. Under classification tasks, our SLR approach converges to the desired accuracy $3\times$ faster on both of the datasets. Under object detection and segmentation tasks, SLR also converges $2\times$ faster to the desired accuracy. Further, our SLR achieves high model accuracy even at the hard-pruning stage without retraining, which reduces the traditional three-stage pruning into a two-stage process. Given a limited budget of retraining epochs, our approach quickly recovers the model's accuracy.

Learning Slimming SSD Through Pruning and Knowledge Distillation

Research on Knowledge Distillation Algorithm of Object Detection

DCCD: Reducing Neural Network Redundancy Via Distillation

Loss Constrains Added Squeeze and Excitation Blocks for Pruning Deep Neural Networks

A Compression Pipeline for One-Stage Object Detection Model

EPSD: Early Pruning with Self-Distillation for Efficient Model Compression

Distillation Sparsity Training Algorithm for Accelerating Convolutional Neural Networks in Embedded Systems

Ssd-kdgan: a lightweight SSD target detection method based on knowledge distillation and generative adversarial networks

Group channel pruning and spatial attention distilling for object detection

Channel Pruning and Quantization-Based Learning for Object Detection with Computing Source Limited Application

Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation

Surrogate Lagrangian Relaxation: A Path To Retrain-free Deep Neural Network Pruning

Pixel Distillation: A New Knowledge Distillation Scheme for Low-Resolution Image Recognition

Neural Network Pruning with Residual-Connections and Limited-Data

KDSMALL: A lightweight small object detection algorithm based on knowledge distillation

DM3D: Distortion-Minimized Weight Pruning for Lossless 3D Object Detection

DTS: Dynamic Training Slimming with Feature Sparsity for Efficient Convolutional Neural Network

Research on the Construction of an Efficient and Lightweight Online Detection Method for Tiny Surface Defects through Model Compression and Knowledge Distillation

Foreground separation knowledge distillation for object detection

Prune the Convolutional Neural Networks with Sparse Shrink

Learning Efficient Convolutional Networks Through Network Slimming.