Abstract:Remote sensing image (RSI) scene classification aims to identify semantic categories in RSI using neural networks. However, high-performance deep neural networks typically demand substantial storage and computational resources, making practical deployment challenging. Knowledge distillation has emerged as an effective technique for developing compact models that maintain high classification accuracy in RSI tasks. Existing knowledge distillation methods often overlook the high inter-class similarity in RSI scenes, leading to low-confidence soft labels from the teacher model, which can mislead the student model. Conversely, overly confident soft labels may discard valuable non-target information. Additionally, the significant intra-class variability in RSI contributes to instability in the model's decision boundaries. To address these challenges, we propose an efficient method called instance-level scaling and dynamic margin-alignment knowledge distillation (ISDM) for RSI scene classification. To balance the target and non-target class influence, we apply an entropy regularization loss to scale the teacher model's target class at the instance level. Moreover, we introduce dynamic margin alignment between the student and teacher models to improve the student's discriminative capability. By optimizing soft labels and enhancing the student's ability to distinguish between classes, our method reduces the effects of inter-class similarity and intra-class variability. Experimental results on three public RSI scene classification datasets (AID, UCMerced, and NWPU-RESISC) demonstrate that our method achieves state-of-the-art performance across all teacher–student pairs with lower computational costs. Additionally, we validate the generalization of our approach on general datasets, including CIFAR-100 and ImageNet-1k.

Lightweight High-Resolution Remote Sensing Scene Classification via Adaptive Enhanced Knowledge Distillation.

Research on Knowledge Distillation Algorithm of Object Detection

Lightweight Remote Sensing Scene Classification Based on Knowledge Distillation

DCCD: Reducing Neural Network Redundancy Via Distillation

Adaptive Knowledge Distillation for Lightweight Remote Sensing Object Detectors Optimizing

From Coarse to Fine: Knowledge Distillation for Remote Sensing Scene Classification

Instance-Level Scaling and Dynamic Margin-Alignment Knowledge Distillation for Remote Sensing Image Scene Classification

TAKD: Target-Aware Knowledge Distillation for Remote Sensing Scene Classification

Pair-Wise Similarity Knowledge Distillation for RSI Scene Classification

Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation

Knowledge Distillation-Based Lightweight Change Detection in High-Resolution Remote Sensing Imagery for On-Board Processing

ERKT-Net: Implementing Efficient and Robust Knowledge Distillation for Remote Sensing Image Classification

Lightweight Self-Knowledge Distillation with Multi-source Information Fusion

Learning Lightweight and Superior Detectors with Feature Distillation for Onboard Remote Sensing Object Detection

Semantic Knowledge Distillation for Onboard Satellite Earth Observation Image Classification

A Lightweight Convolution Network with Self-Knowledge Distillation for Hyperspectral Image Classification

Remote Sensing Image Scene Classification with Noisy Label Distillation

Knowledge Distillation with Feature Maps for Image Classification

Information Fusion with Knowledge Distillation for Fine-grained Remote Sensing Object Detection

Knowledge Distillation of Grassmann Manifold Network for Remote Sensing Scene Classification

Efficient knowledge distillation for hybrid models: A vision transformer‐convolutional neural network to convolutional neural network approach for classifying remote sensing images