Abstract:The rapid evolution of deep learning has led to significant achievements in computer vision, primarily driven by complex convolutional neural networks (CNNs). However, the increasing depth and parameter count of these networks often result in overfitting and elevated computational demands. Knowledge distillation (KD) has emerged as a promising technique to address these issues by transferring knowledge from a large, well-trained teacher model to a more compact student model. This paper introduces a novel knowledge distillation method that simplifies the distillation process and narrows the performance gap between teacher and student models without relying on intricate knowledge representations. Our approach leverages a unique teacher network architecture designed to enhance the efficiency and effectiveness of knowledge transfer. Additionally, we introduce a streamlined teacher network architecture that transfers knowledge effectively through a simplified distillation process, enabling the student model to achieve high accuracy with reduced computational demands. Comprehensive experiments conducted on the CIFAR-10 dataset demonstrate that our proposed model achieves superior performance compared to traditional KD methods and established architectures such as ResNet and VGG networks. The proposed method not only maintains high accuracy but also significantly reduces training and validation losses. Key findings highlight the optimal hyperparameter settings (temperature T = 15.0 and smoothing factor α = 0.7), which yield the highest validation accuracy and lowest loss values. This research contributes to the theoretical and practical advancements in knowledge distillation, providing a robust framework for future applications and research in neural network compression and optimization. The simplicity and efficiency of our approach pave the way for more accessible and scalable solutions in deep learning model deployment.

Learning knowledge representation with meta knowledge distillation for single image super-resolution

DCCD: Reducing Neural Network Redundancy Via Distillation

MTKD: Multi-Teacher Knowledge Distillation for Image Super-Resolution

Self-Paced Knowledge Distillation for Real-Time Image Guided Depth Completion

Data Upcycling Knowledge Distillation for Image Super-Resolution

Fakd: feature-affinity based knowledge distillation for efficient image super-resolution

Knowledge Distillation with Multi-granularity Mixture of Priors for Image Super-Resolution

Teacher-Free Knowledge Distillation Based on Non-Progressive Meta-Learned Multi Ranking Selection

Simplified Knowledge Distillation for Deep Neural Networks Bridging the Performance Gap with a Novel Teacher–Student Architecture

Collaborative Knowledge Distillation Via Multiknowledge Transfer.

Multi-target Knowledge Distillation Via Student Self-reflection

Teacher-student collaborative knowledge distillation for image classification

Knowledge Distillation with Deep Supervision

Lightweight Self-Knowledge Distillation with Multi-source Information Fusion

Knowledge Distillation Meets Self-Supervision

Adaptive Cross-Architecture Mutual Knowledge Distillation

Knowledge Representing: Efficient, Sparse Representation of Prior Knowledge for Knowledge Distillation

Online Knowledge Distillation via Collaborative Learning

Collaborative Teacher-Student Learning via Multiple Knowledge Transfer

Efficient knowledge distillation for hybrid models: A vision transformer‐convolutional neural network to convolutional neural network approach for classifying remote sensing images

Pixel Distillation: A New Knowledge Distillation Scheme for Low-Resolution Image Recognition