KnowledgeIE: Unifying Online-Offline Distillation Based on Knowledge Inheritance and Evolution

Yiqing Shen

DOI: https://doi.org/10.1109/ijcnn60899.2024.10650086

2024-01-01

Abstract:Knowledge distillation has been successfully applied to various pattern recognition and computer vision tasks, enabling the transfer of knowledge from a pre-trained teacher model to a student model or mutual learning between peer student models. However, most existing works focus on either offline or online distillation independently, despite the potential benefits of combining both approaches to address the complexity of real-world applications. To bridge the gap between offline and online distillation, we propose a novel unified framework called Knowledge Inheritance and Evolution (KnowledgeIE), inspired by human learning behaviors. KnowledgeIE seamlessly integrates offline and online distillation modes, allowing the student model to learn from both a pre-trained teacher and its peers during training. We evaluate the effectiveness of KnowledgeIE on several image classification benchmarks and demonstrate consistent improvements over baseline methods. Our comprehensive experiments on three datasets show that the unified architecture outperforms both standalone offline and online distillation, highlighting the strong compatibility and synergy achieved by combining the two approaches. KnowledgeIE presents a promising direction for knowledge distillation, offering a more flexible and effective framework for transferring knowledge between models in various application scenarios.