Propheter: Prophetic Teacher Guided Long-Tailed Distribution Learning

Wenxiang Xu,Yongcheng Jing,Linyun Zhou,Wenqi Huang,Lechao Cheng,Zunlei Feng,Mingli Song

2023-09-25

Abstract:The problem of deep long-tailed learning, a prevalent challenge in the realm of generic visual recognition, persists in a multitude of real-world applications. To tackle the heavily-skewed dataset issue in long-tailed classification, prior efforts have sought to augment existing deep models with the elaborate class-balancing strategies, such as class rebalancing, data augmentation, and module improvement. Despite the encouraging performance, the limited class knowledge of the tailed classes in the training dataset still bottlenecks the performance of the existing deep models. In this paper, we propose an innovative long-tailed learning paradigm that breaks the bottleneck by guiding the learning of deep networks with external prior knowledge. This is specifically achieved by devising an elaborated ``prophetic'' teacher, termed as ``Propheter'', that aims to learn the potential class distributions. The target long-tailed prediction model is then optimized under the instruction of the well-trained ``Propheter'', such that the distributions of different classes are as distinguishable as possible from each other. Experiments on eight long-tailed benchmarks across three architectures demonstrate that the proposed prophetic paradigm acts as a promising solution to the challenge of limited class knowledge in long-tailed datasets. The developed code is publicly available at \url{<a class="link-external link-https" href="https://github.com/tcmyxc/propheter" rel="external noopener nofollow">this https URL</a>}.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the problem of effectively training deep learning models on long-tailed distribution datasets. Specifically, a long-tailed distribution refers to a dataset where the number of samples in certain categories far exceeds that in other categories, leading to poor performance of the model on the minority categories. This phenomenon of data imbalance is common in many practical applications, such as image classification, object detection, and segmentation.

To solve this problem, existing methods mainly focus on data resampling, loss function reweighting, and transfer learning. Although these methods have achieved certain effects, they usually rely on carefully designed data samplers and lack explicit consideration of the differences in activation distributions of different categories in high-dimensional feature spaces.

The paper proposes a new long-tailed learning paradigm called "Propheter" (Prophet Teacher), which introduces external prior knowledge to guide the learning of deep networks, aiming to break through the performance bottleneck of existing models under limited knowledge of tail categories. "Propheter" learns the latent category distributions and transfers this distribution knowledge to the student model, making the distributions of different categories as distinguishable as possible. Experimental results show that this method performs excellently on multiple long-tailed benchmark datasets, particularly improving performance by over 4% compared to existing methods on the CIFAR-10-LT dataset.