A Channel-ensemble Approach: Unbiased and Low-variance Pseudo-labels is Critical for Semi-supervised Classification

Jiaqi Wu,Junbiao Pang,Baochang Zhang,Qingming Huang

2024-03-27

Abstract:Semi-supervised learning (SSL) is a practical challenge in computer vision. Pseudo-label (PL) methods, e.g., FixMatch and FreeMatch, obtain the State Of The Art (SOTA) performances in SSL. These approaches employ a threshold-to-pseudo-label (T2L) process to generate PLs by truncating the confidence scores of unlabeled data predicted by the self-training method. However, self-trained models typically yield biased and high-variance predictions, especially in the scenarios when a little labeled data are supplied. To address this issue, we propose a lightweight channel-based ensemble method to effectively consolidate multiple inferior PLs into the theoretically guaranteed unbiased and low-variance one. Importantly, our approach can be readily extended to any SSL framework, such as FixMatch or FreeMatch. Experimental results demonstrate that our method significantly outperforms state-of-the-art techniques on CIFAR10/100 in terms of effectiveness and efficiency.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

This paper discusses a key issue in Semi-supervised Learning (SSL), which is how to generate unbiased and low-variance pseudo-labels. Current methods such as FixMatch and FreeMatch employ the Threshold-to-Pseudo-label (T2L) process, but the predictions produced by self-training models often have biases and high variances, especially with limited annotated data. To address this, the paper proposes a Channel-Based Ensemble (CBE), a lightweight integration method that combines multiple suboptimal pseudo-labels to generate theoretically unbiased and low-variance pseudo-labels.

The core of CBE method lies in the utilization of channel-level ensemble structure to reduce the accumulation of prediction errors with almost no additional computational cost. Additionally, the paper introduces the Low Bias (LB) loss function to maximize the feature irrelevance between different prediction heads, reducing prediction biases. The Low Variance (LV) loss function is also proposed to leverage the ground truth constraints of labeled data and reduce the variance of the prediction distribution of unlabeled data.

Experimental results demonstrate that CBE significantly outperforms the current state-of-the-art SSL techniques on CIFAR10/100 datasets, both in terms of effectiveness and efficiency. This method can be easily extended to any SSL framework, such as FixMatch or FreeMatch.

In conclusion, this paper presents a new solution to the issue of pseudo-label quality in SSL, improving the performance and training efficiency of models through ensemble learning and specific loss function optimization.