Abstract:Co-training, an advanced form of self-training, allows multiple base models to learn collaboratively, leading to superior performance in semi-supervised learning tasks. However, its widespread adoption is hindered by high computational costs and intricate design choices. To address these challenges, we present Multi-Head Co-Training, a streamlined and efficient framework that consolidates individual models into a multi-head structure, adding minimal extra parameters. Each classification head in this unified model collaborates with others via a “Weak and Strong Augmentation” strategy, with diversity organically introduced through robust data augmentation. Consequently, our approach implicitly promotes diversity while incurring only a minor increase in computational overhead, making co-training more accessible. We validate the effectiveness of Multi-Head Co-Training through an empirical study on standard semi-supervised learning benchmarks. For example, our method achieves up to a 3.1% accuracy improvement on the semi-supervised CIFAR dataset compared to recent methods.Recognizing the necessity for more practical performance metrics beyond accuracy, we assess our framework from three additional perspectives: robust generalization, uncertainty, and computational efficiency. To evaluate robust generalization, we expand the conventional SSL experimental setting to a more comprehensive open-set semi-supervised learning scenario. For uncertainty assessment, we conduct experiments on model calibration and selective classification benchmarks. For example, our method achieves up to a 4.3% accuracy improvement on the open-set semi-supervised CIFAR dataset. Our extensive experiments confirm that our proposed framework better captures prediction confidence and uncertainty, rendering it more suitable for SSL deployment in open environments. The code is available at https://github.com/chenmc1996/Multi-Head-Co-Training.

Costra: Confidence-based self-training

FGCM: Noisy Label Learning via Fine-Grained Confidence Modeling

CAST: Cluster-Aware Self-Training for Tabular Data via Reliable Confidence

Incremental Self-training for Semi-supervised Learning

CoTrade: Confident Co-Training With Data Editing.

Self-Training: A Survey

Doubly Robust Self-Training

Confidence Estimation Using Unlabeled Data

Co-learning: Learning from Noisy Labels with Self-supervision

A Mutually Attentive Co-Training Framework for Semi-Supervised Recognition

Self-paced and self-consistent co-training for semi-supervised image segmentation

Better Self-training for Image Classification Through Self-supervision

Semi-supervised Object Detection with Adaptive Class-Rebalancing Self-Training.

Improving Semi-Supervised Self-Training with Embedded Manifold Transduction

Self-Training with Label-Feature-Consistency for Domain Adaptation

A replica analysis of Self-Training of Linear Classifier

Improving self-training under distribution shifts via anchored confidence with theoretical guarantees

Multi-head Co-Training: an Uncertainty-Aware and Robust Semi-Supervised Learning Framework

Inter-training: Exploiting Unlabeled Data in Multi-Classifier Systems

Rethinking Self-training for Semi-supervised Landmark Detection: A Selection-free Approach

Enhancing Counterfactual Classification via Self-Training