Abstract:Online Knowledge Distillation (OKD) is designed to alleviate the dilemma that the high-capacity pre-trained teacher model is not available. However, the existing methods mostly focus on improving the ensemble prediction accuracy from multiple students (a.k.a. branches), which often overlook the homogenization problem that makes student model saturate quickly and hurts the performance. We assume that the intrinsic bottleneck of the homogenization problem comes from the identical branch architecture and coarse ensemble strategy. We propose a novel Adaptive Hierarchy-Branch Fusion framework for Online Knowledge Distillation, termed AHBF-OKD, which designs hierarchical branches and adaptive hierarchy-branch fusion module to boost the model diversity and aggregate complementary knowledge. Specifically, we first introduce hierarchical branch architectures to construct diverse peers by increasing the depth of branches monotonously on the basis of target branch. To effectively transfer knowledge from the most complex branch to the simplest target branch, we propose an adaptive hierarchy-branch fusion module to create hierarchical teacher assistants recursively, which regards the target branch as the smallest teacher assistant. During the training, the teacher assistant from the previous hierarchy is explicitly distilled by the teacher assistant and the branch from the current hierarchy. Thus, the important scores to different branches are effectively and adaptively allocated to reduce the branch homogenization. Extensive experiments demonstrate the effectiveness of AHBF-OKD on different datasets, including CIFAR-10/100 and ImageNet 2012. For example, on ImageNet 2012, the distilled ResNet-18 achieves Top-1 error of 29.28\%, which significantly outperforms the state-of-the-art methods. The source code is available at https://github.com/linruigong965/AHBF.

Many-objective evolutionary self-knowledge distillation with adaptive branch fusion method

Diversified Branch Fusion for Self-Knowledge Distillation

Lightweight Self-Knowledge Distillation with Multi-source Information Fusion

Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data

Adaptive Cross-Architecture Mutual Knowledge Distillation

Semi-Online Knowledge Distillation

Multi-target Knowledge Distillation Via Student Self-reflection

Flexible Knowledge Distillation with an Evolutional Network Population

DE-MKD: Decoupled Multi-Teacher Knowledge Distillation Based on Entropy

An Embarrassingly Simple Approach for Knowledge Distillation

CDFKD-MFS: Collaborative Data-free Knowledge Distillation Via Multi-level Feature Sharing

Multiple-Stage Knowledge Distillation

CDFKD-MFS: Collaborative Data-free Knowledge Distillation via Multi-level Feature Sharing

Memory-Replay Knowledge Distillation

A Novel Multi-Knowledge Distillation Approach.

Embedded Mutual Learning: A Novel Online Distillation Method Integrating Diverse Knowledge Sources.

Two-Stage Approach for Targeted Knowledge Transfer in Self-Knowledge Distillation

Revisiting Knowledge Distillation: an Inheritance and Exploration Framework

Adaptive Hierarchy-Branch Fusion for Online Knowledge Distillation

Skill-transferring Knowledge Distillation Method

Collaborative Knowledge Distillation Via Multiknowledge Transfer.