Knowledge Probabilization in Ensemble Distillation: Improving Accuracy and Uncertainty Quantification for Object Detectors
Yang,Chao Wang,Lei Gong,Min Wu,Zhenghua Chen,Xiang Li,Xianglan Chen,Xuehai Zhou
DOI: https://doi.org/10.1109/tai.2024.3474654
2024-01-01
IEEE Transactions on Artificial Intelligence
Abstract:Ensemble object detectors have demonstrated remarkable effectiveness in enhancing prediction accuracy and uncertainty quantification. However, their widespread adoption is hindered by significant computational and storage demands, limiting their feasibility in resource-constrained settings. To overcome this, researchers have focused on distilling the knowledge from ensemble object detectors into a single model. In this paper, we introduce ProbED, an innovative ensemble distillation framework that consolidates knowledge from multiple object detectors into a single, resource-efficient model. Unlike traditional ensemble distillation methods that average the outputs of sub-teachers, ProbED captures comprehensive outcome distributions from all sub-teachers, providing a more nuanced approach to knowledge transfer. ProbED employs knowledge probabilization to achieve a sophisticated and refined aggregation of teacher knowledge, including feature knowledge, semantic knowledge, and localization knowledge, resulting in dual improvements in prediction accuracy and uncertainty quantification for the student model. In particular, ProED’s novel knowledge probabilization-based approach to aggregating teacher knowledge is inspired by our empirical observations, which demonstrate that knowledge probabilization excels in effectively representing uncertainty, improving prediction, and facilitating robust knowledge transfer. Furthermore, we introduce a random smoothing perturbation technique to modify inputs within ProbED, further enhancing the distillation process. Extensive experiments highlight ProbED’s ability to significantly enhance the prediction accuracy and uncertainty quantification of various object detectors, demonstrating its superior performance compared to other state-of-the-art techniques.