Abstract: Deep neural networks achieve remarkable performances on a wide range of tasks with the aid of large-scale labeled datasets. Yet these datasets are time-consuming and labor-exhaustive to obtain on realistic tasks. To mitigate the requirement for labeled data, self-training is widely used in semi-supervised learning by iteratively assigning pseudo labels to unlabeled samples. Despite its popularity, self-training is well-believed to be unreliable and often leads to training instability. Our experimental studies further reveal that the bias in semi-supervised learning arises from both the problem itself and the inappropriate training with potentially incorrect pseudo labels, which accumulates the error in the iterative self-training process. To reduce the above bias, we propose Debiased Self-Training (DST). First, the generation and utilization of pseudo labels are decoupled by two parameter-independent classifier heads to avoid direct error accumulation. Second, we estimate the worst case of self-training bias, where the pseudo labeling function is accurate on labeled samples, yet makes as many mistakes as possible on unlabeled samples. We then adversarially optimize the representations to improve the quality of pseudo labels by avoiding the worst case. Extensive experiments justify that DST achieves an average improvement of 6.3% against state-of-the-art methods on standard semi-supervised learning benchmark datasets and 18.9%$ against FixMatch on 13 diverse tasks. Furthermore, DST can be seamlessly adapted to other self-training methods and help stabilize their training and balance performance across classes in both cases of training from scratch and finetuning from pre-trained models.

SETRED: self-training with editing

A self-training algorithm based on the two-stage data editing method with mass-based dissimilarity

Debiased Self-Training for Semi-Supervised Learning

Rethinking Self-training for Semi-supervised Landmark Detection: A Selection-free Approach

Incremental Self-training for Semi-supervised Learning

Doubly Robust Self-Training

Learning to Edit Visual Programs with Self-Supervision

STEdge: Self-training Edge Detection with Multi-layer Teaching and Regularization

The GIST and RIST of Iterative Self-Training for Semi-Supervised Segmentation

Self-Training: A Survey

Interactive Correction of Mislabeled Training Data

Debiased Pseudo Labeling in Self-Training

Semi-supervised Relation Extraction Via Incremental Meta Self-Training

Semi-supervised learning based on nearest neighbor rule and cut edges

Improving Sparsely Labeled Text Classification with Data Editing

Learning How to Self-Learn: Enhancing Self-Training Using Neural Reinforcement Learning

Self-Adaptive Training: Bridging Supervised and Self-Supervised Learning.

On the Marginal Benefit of Active Learning: Does Self-Supervision Eat Its Cake?

Iterative Graph Self-Distillation