Abstract:Pseudo-Labeling is a simple and effective approach to semi-supervised learning. It requires criteria that guide the selection of pseudo-labeled data. The latter have been shown to crucially affect pseudo-labeling's generalization performance. Several such criteria exist and were proven to work reasonably well in practice. However, their performance often depends on the initial model fit on labeled data. Early overfitting can be propagated to the final model by choosing instances with overconfident but wrong predictions, often called confirmation bias. In two recent works, we demonstrate that pseudo-label selection (PLS) can be naturally embedded into decision theory. This paves the way for BPLS, a Bayesian framework for PLS that mitigates the issue of confirmation bias. At its heart is a novel selection criterion: an analytical approximation of the posterior predictive of pseudo-samples and labeled data. We derive this selection criterion by proving Bayes-optimality of this "pseudo posterior predictive". We empirically assess BPLS for generalized linear, non-parametric generalized additive models and Bayesian neural networks on simulated and real-world data. When faced with data prone to overfitting and thus a high chance of confirmation bias, BPLS outperforms traditional PLS methods. The decision-theoretic embedding further allows us to render PLS more robust towards the involved modeling assumptions. To achieve this goal, we introduce a multi-objective utility function. We demonstrate that the latter can be constructed to account for different sources of uncertainty and explore three examples: model selection, accumulation of errors and covariate shift.

What problem does this paper attempt to address?

This paper primarily explores the issue of pseudo-label (Pseudo-Labeling, PLS) selection and proposes a method based on Bayesian decision theory to address this problem. ### Problems the Paper Attempts to Solve 1. **Lack of Theoretical Foundation**: Existing pseudo-label selection methods, although reasonable in practice, lack a solid theoretical basis. 2. **Confirmation Bias Issue**: The initial model's overfitting on unlabeled data may lead to selecting samples with incorrect but high-confidence predictions as pseudo-labels, thereby introducing the so-called "confirmation bias." ### Overview of the Solution 1. **Viewing Pseudo-Label Selection as a Decision Problem**: The authors consider pseudo-label selection essentially as a decision problem, where the state space corresponds to the learner's parameter space, and the action space corresponds to the unlabeled dataset. In this way, methods from decision theory can be used to find the optimal action (i.e., which unlabeled samples to select for pseudo-labeling). 2. **Bayes-Optimal Selection Criterion**: The authors demonstrate that, with the joint likelihood function as the utility function, the Bayes-optimal selection criterion is the posterior prediction of pseudo-samples and labeled data. However, directly computing this posterior prediction can be very complex, so the authors propose an approximate algorithm based on the Laplace method and Gaussian integration, making the criterion simple and computationally inexpensive. 3. **Multi-Objective Utility Function**: To make pseudo-label selection more robust, the authors further propose a utility function that includes multiple objectives, allowing for the handling of different sources of uncertainty, such as model selection, error accumulation, and covariate shift. 4. **Empirical Evaluation**: The authors evaluate the proposed Bayesian PLS (BPLS) method on both simulated and real-world data, including generalized linear models, non-parametric generalized additive models, and Bayesian neural networks. The results show that the BPLS method outperforms traditional pseudo-label selection methods when dealing with data prone to overfitting. In summary, this paper aims to address theoretical and practical issues in the pseudo-label selection process by constructing a framework based on Bayesian decision theory, thereby improving the quality and effectiveness of pseudo-labels in semi-supervised learning.

Pseudo Label Selection is a Decision Problem

Bayesian Optimization Based on Pseudo Labels

Uncertainty-aware Pseudo-label Selection for Positive-Unlabeled Learning

In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning

Partial Multi-label Learning with Label and Feature Collaboration

Learning Accurate Label-Specific Features From Partially Multilabeled Data

Why the pseudo label based semi-supervised learning algorithm is effective?

Semi-Supervised Learning guided by the Generalized Bayes Rule under Soft Revision

You can't handle the (dirty) truth: Data-centric insights improve pseudo-labeling

Probabilistic Labeled Semi-supervised SVM

Prediction-Oriented Model Selection In Partial Least Squares Path Modeling

Reduction-based Pseudo-label Generation for Instance-dependent Partial Label Learning

Deep Insights into Noisy Pseudo Labeling on Graph Data

A Review of Pseudo-Labeling for Computer Vision

Augmented prediction of a true class for Positive Unlabeled data under selection bias

Expectation Maximization Pseudo Labels

PLS-Based Model Selection: the Role of Alternative Explanations in Information Systems Research

Verifying the Selected Completely at Random Assumption in Positive-Unlabeled Learning

Bayesian Pseudo Labels: Expectation Maximization for Robust and Efficient Semi-Supervised Segmentation

Partial Label Metric Learning Based on Statistical Inference