Abstract:This study presents an innovative dynamic weighting knowledge distillation (KD) framework tailored for efficient Earth observation (EO) image classification (IC) in resource-constrained settings. Utilizing EfficientViT and MobileViT as teacher models, this framework enables lightweight student models, particularly ResNet8 and ResNet16, to surpass 90% in accuracy, precision, and recall, adhering to the stringent confidence thresholds necessary for reliable classification tasks. Unlike conventional KD methods that rely on static weight distribution, our adaptive weighting mechanism responds to each teacher model's confidence, allowing student models to prioritize more credible sources of knowledge dynamically. Remarkably, ResNet8 delivers substantial efficiency gains, achieving a 97.5% reduction in parameters, a 96.7% decrease in FLOPs, an 86.2% cut in power consumption, and a 63.5% increase in inference speed over MobileViT. This significant optimization of complexity and resource demands establishes ResNet8 as an optimal candidate for EO tasks, combining robust performance with feasibility in deployment. The confidence-based, adaptable KD approach underscores the potential of dynamic distillation strategies to yield high-performing, resource-efficient models tailored for satellite-based EO applications. The reproducible code is accessible on our GitHub repository.

What problem does this paper attempt to address?

The paper attempts to address the problem of how to efficiently perform satellite Earth observation image classification (EO-IC) in resource-constrained environments. Specifically, the researchers face the following challenges: 1. **Resource Constraints**: The computational resources and energy of low Earth orbit (LEO) satellites are very limited. Traditional deep learning models (such as Vision Transformer, ViT), although powerful, face high computational complexity and large memory requirements when deployed on these platforms. 2. **Real-time Processing Requirements**: Modern Earth observation and remote sensing tasks require real-time, on-orbit decision-making capabilities. Relying on ground stations to transmit data has issues with limited communication windows and frequent disconnections, affecting the timely response to critical missions. 3. **Limitations of Knowledge Distillation**: Traditional knowledge distillation methods (Knowledge Distillation, KD) usually use fixed weights, which can lead to unstable training. Especially when the student model is required to strictly match the predictions of a single teacher model, it may affect performance. To address these issues, the paper proposes a dynamic weighting knowledge distillation framework (Dynamic Weighting Knowledge Distillation, DualKD). By distilling semantic knowledge from two teacher models (EfficientViT and MobileViT), the lightweight student models (ResNet8 and ResNet16) can significantly reduce computational complexity and resource requirements while maintaining high performance. The specific goals include: - **Improving Classification Performance**: Enabling the student models to achieve over 90% accuracy, precision, and recall in resource-constrained environments. - **Optimizing Resource Utilization**: Significantly reducing the number of model parameters, floating-point operations (FLOPs), power consumption, and inference time, making it suitable for deployment on satellites. - **Enhancing Adaptability and Generalization**: By dynamically adjusting the weights of the teacher models, the student models can prioritize learning more reliable semantic knowledge, resulting in more stable and accurate performance across different instances. In summary, the paper aims to achieve efficient and reliable satellite Earth observation image classification through dynamic weighting knowledge distillation technology, meeting the real-time processing needs of modern remote sensing tasks.

Semantic Knowledge Distillation for Onboard Satellite Earth Observation Image Classification

ERKT-Net: Implementing Efficient and Robust Knowledge Distillation for Remote Sensing Image Classification

Pixel Distillation: A New Knowledge Distillation Scheme for Low-Resolution Image Recognition

Efficient knowledge distillation for hybrid models: A vision transformer‐convolutional neural network to convolutional neural network approach for classifying remote sensing images

Exploring Model Compression Limits and Laws: A Pyramid Knowledge Distillation Framework for Satellite-on-Orbit Object Recognition

Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data

Efficient image classification through collaborative knowledge distillation: A novel AlexNet modification approach

Adaptive Knowledge Distillation for Lightweight Remote Sensing Object Detectors Optimizing

Applications of knowledge distillation in remote sensing: A survey

Channel-spatial knowledge distillation for efficient semantic segmentation

Instance-Level Scaling and Dynamic Margin-Alignment Knowledge Distillation for Remote Sensing Image Scene Classification

Lightweight Remote Sensing Scene Classification Based on Knowledge Distillation

Lightweight Self-Knowledge Distillation with Multi-source Information Fusion

Knowledge Distillation for Oriented Object Detection on Aerial Images

Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation

Spirit Distillation: Precise Real-time Semantic Segmentation of Road Scenes with Insufficient Data

HDKD: Hybrid Data-Efficient Knowledge Distillation Network for Medical Image Classification

Two-Way Assistant: A Knowledge Distillation Object Detection Method for Remote Sensing Images

Pair-Wise Similarity Knowledge Distillation for RSI Scene Classification

An Embarrassingly Simple Approach for Knowledge Distillation