Abstract:Deep learning has contributed greatly to many successes in artificial intelligence in recent years. Today, it is possible to train models that have thousands of layers and hundreds of billions of parameters. Large-scale deep models have achieved great success, but the enormous computational complexity and gigantic storage requirements make it extremely difficult to implement them in real-time applications. On the other hand, the size of the dataset is still a real problem in many domains. Data are often missing, too expensive, or impossible to obtain for other reasons. Ensemble learning is partially a solution to the problem of small datasets and overfitting. However, ensemble learning in its basic version is associated with a linear increase in computational complexity. We analyzed the impact of the ensemble decision-fusion mechanism and checked various methods of sharing the decisions including voting algorithms. We used the modified knowledge distillation framework as a decision-fusion mechanism which allows in addition compressing of the entire ensemble model into a weight space of a single model. We showed that knowledge distillation can aggregate knowledge from multiple teachers in only one student model and, with the same computational complexity, obtain a better-performing model compared to a model trained in the standard manner. We have developed our own method for mimicking the responses of all teachers at the same time, simultaneously. We tested these solutions on several benchmark datasets. In the end, we presented a wide application use of the efficient multi-teacher knowledge distillation framework. In the first example, we used knowledge distillation to develop models that could automate corrosion detection on aircraft fuselage. The second example describes detection of smoke on observation cameras in order to counteract wildfires in forests.

Knowledge Distillation via Multi-Teacher Feature Ensemble

Using Less but Important Information for Feature Distillation

Unified and Effective Ensemble Knowledge Distillation

FEED: Feature-level Ensemble for Knowledge Distillation

Learn From the Past: Experience Ensemble Knowledge Distillation

Online Knowledge Distillation via Multi-branch Diversity Enhancement

Reinforced Multi-Teacher Selection for Knowledge Distillation

Exploring Graph-based Knowledge: Multi-Level Feature Distillation via Channels Relational Graph

Bi-Level Orthogonal Multi-Teacher Distillation

Knowledge Distillation with the Reused Teacher Classifier

Multistage feature fusion knowledge distillation

Confidence-Aware Multi-Teacher Knowledge Distillation

Multi-teacher knowledge distillation as an effective method for compressing ensembles of neural networks

Knowledge distillation based on multi-layer fusion features

Highlight Every Step: Knowledge Distillation via Collaborative Teaching

TC<SUP>3</SUP>KD: Knowledge distillation via teacher-student cooperative curriculum customization

Multi-Level Feature Distillation of Joint Teachers Trained on Distinct Image Datasets

Teacher-student collaborative knowledge distillation for image classification

Improved Feature Distillation via Projector Ensemble

Customizing a Teacher for Feature Distillation

Multi scale Feature Extraction and Fusion for Online Knowledge Distillation