Abstract:The rapid evolution of deep learning has led to significant achievements in computer vision, primarily driven by complex convolutional neural networks (CNNs). However, the increasing depth and parameter count of these networks often result in overfitting and elevated computational demands. Knowledge distillation (KD) has emerged as a promising technique to address these issues by transferring knowledge from a large, well-trained teacher model to a more compact student model. This paper introduces a novel knowledge distillation method that simplifies the distillation process and narrows the performance gap between teacher and student models without relying on intricate knowledge representations. Our approach leverages a unique teacher network architecture designed to enhance the efficiency and effectiveness of knowledge transfer. Additionally, we introduce a streamlined teacher network architecture that transfers knowledge effectively through a simplified distillation process, enabling the student model to achieve high accuracy with reduced computational demands. Comprehensive experiments conducted on the CIFAR-10 dataset demonstrate that our proposed model achieves superior performance compared to traditional KD methods and established architectures such as ResNet and VGG networks. The proposed method not only maintains high accuracy but also significantly reduces training and validation losses. Key findings highlight the optimal hyperparameter settings (temperature T = 15.0 and smoothing factor α = 0.7), which yield the highest validation accuracy and lowest loss values. This research contributes to the theoretical and practical advancements in knowledge distillation, providing a robust framework for future applications and research in neural network compression and optimization. The simplicity and efficiency of our approach pave the way for more accessible and scalable solutions in deep learning model deployment.

Bridging the Gap of Dimensions in Distillation: Understanding the knowledge transfer between different-dimensional semantic spaces

Using Less but Important Information for Feature Distillation

Adaptive Informative Semantic Knowledge Transfer for Knowledge Distillation

DistilE: Distiling Knowledge Graph Embeddings for Faster and Cheaper Reasoning

Visualizing the embedding space to explain the effect of knowledge distillation

Reducing the Teacher-Student Gap via Spherical Knowledge Disitllation

Exploring Graph-based Knowledge: Multi-Level Feature Distillation via Channels Relational Graph

Harmonizing knowledge Transfer in Neural Network with Unified Distillation

Multi-instance semantic similarity transferring for knowledge distillation

A Two-Teacher Framework For Knowledge Distillation

Improved Knowledge Distillation via Full Kernel Matrix Transfer

Spherical Knowledge Distillation.

Fixing the Teacher-Student Knowledge Discrepancy in Distillation

Student-Oriented Teacher Knowledge Refinement for Knowledge Distillation

Respecting Transfer Gap in Knowledge Distillation

Collaborative spatial-temporal distillation for efficient video deraining

Simplified Knowledge Distillation for Deep Neural Networks Bridging the Performance Gap with a Novel Teacher–Student Architecture

Logit Distillation via Student Diversity.

Variational Information Distillation for Knowledge Transfer

Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillation

A Survey on Recent Teacher-student Learning Studies