Abstract:Deep neural network (DNN) is extensively explored for LiDAR-based 3D object detection, a crucial perception task in the field of autonomous driving. However, the presence of redundant parameters and complex computations pose challenges for the practical deployment of DNNs. Despite knowledge distillation (KD) is an effective approach for accelerating models, extremely small number of efforts explore its potential on LiDARbased 3D detectors. Besides, existing studies neglect to elaborately investigate 3D voxel-wise features for compression. To this end, we propose instance-aware knowledge distillation (InstKD) for 3D detector compression. The proposed method conducts KD by fully excavating two types of knowledge related to 3D voxelwise features. Firstly, the 3D voxel-wise feature of teacher is transferred to teach the student. In order to prioritize the knowledge with strong guiding capacity, we introduce expanded bounding box (E-Bbox) to distinguish and balance the foreground and background regions. Besides, we generate contribution map (CM) by calculating the gap between the classification response of teacher and student models to further dynamically balance individual instance for distillation. Secondly, we also align the relation-based knowledge of 3D voxel-wise features between the distillation pairs. To avoid incalculable relation on a massive number of 3D voxel-wise features, we distill the relation among instances selected by E-Bboxes, where the intra-relation of homogeneous instances and inter-relation of heterogeneous instances are transferred in a dual-pathway manner. In the experiments, we compress different models on benchmarks with varying scales. The results demonstrate that our method achieves the lightweight 3D detector with slight performance drop. For example, on KITTI dataset, our 2× compressed SECOND (75.5% parameters and 74.5% FLOPs reduction) achieves 66.83% mAP, surpassing its teacher model. The key code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/zhnxjtu/InstKD.</uri>

Exploring Model Compression Limits and Laws: A Pyramid Knowledge Distillation Framework for Satellite-on-Orbit Object Recognition

Knowledge Distillation for Oriented Object Detection on Aerial Images

Model Compression via Collaborative Data-Free Knowledge Distillation for Edge Intelligence.

Data-Free Ensemble Knowledge Distillation for Privacy-conscious Multimedia Model Compression.

CDFKD-MFS: Collaborative Data-free Knowledge Distillation Via Multi-level Feature Sharing

Knowledge Distillation with Information Compressed Representations

Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data

Residual Error Based Knowledge Distillation

CDFKD-MFS: Collaborative Data-free Knowledge Distillation via Multi-level Feature Sharing

InstKD: Towards Lightweight 3D Object Detection With Instance-Aware Knowledge Distillation

Pixel Distillation: A New Knowledge Distillation Scheme for Low-Resolution Image Recognition

Semantic Knowledge Distillation for Onboard Satellite Earth Observation Image Classification

Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation

Stage-by-stage Knowledge Distillation

Applications of knowledge distillation in remote sensing: A survey

An Embarrassingly Simple Approach for Knowledge Distillation

Ability-aware knowledge distillation for resource-constrained embedded devices

Centerness-based Instance-aware Knowledge Distillation with Task-wise Mutual Lifting for Object Detection on Drone Imagery

Simplified Knowledge Distillation for Deep Neural Networks Bridging the Performance Gap with a Novel Teacher–Student Architecture

Domain-invariant Progressive Knowledge Distillation for UAV-based Object Detection

STKD: Distilling Knowledge From Synchronous Teaching for Efficient Model Compression