Abstract:Pseudo-Labeling is a simple and effective approach to semi-supervised learning. It requires criteria that guide the selection of pseudo-labeled data. The latter have been shown to crucially affect pseudo-labeling's generalization performance. Several such criteria exist and were proven to work reasonably well in practice. However, their performance often depends on the initial model fit on labeled data. Early overfitting can be propagated to the final model by choosing instances with overconfident but wrong predictions, often called confirmation bias. In two recent works, we demonstrate that pseudo-label selection (PLS) can be naturally embedded into decision theory. This paves the way for BPLS, a Bayesian framework for PLS that mitigates the issue of confirmation bias. At its heart is a novel selection criterion: an analytical approximation of the posterior predictive of pseudo-samples and labeled data. We derive this selection criterion by proving Bayes-optimality of this "pseudo posterior predictive". We empirically assess BPLS for generalized linear, non-parametric generalized additive models and Bayesian neural networks on simulated and real-world data. When faced with data prone to overfitting and thus a high chance of confirmation bias, BPLS outperforms traditional PLS methods. The decision-theoretic embedding further allows us to render PLS more robust towards the involved modeling assumptions. To achieve this goal, we introduce a multi-objective utility function. We demonstrate that the latter can be constructed to account for different sources of uncertainty and explore three examples: model selection, accumulation of errors and covariate shift.

Semi-Supervised Learning guided by the Generalized Bayes Rule under Soft Revision

Generalized entropy based semi-supervised learning

Pseudo Label Selection is a Decision Problem

A Novel Semi-Supervised Classification Method Based on Soft Evidential Label Propagation

Semisupervised Bayesian Gaussian Mixture Models for Non-Gaussian Soft Sensor

Semisupervised Learning for Probabilistic Partial Least Squares Regression Model and Soft Sensor Application

Soft-Sensor Development for Processes with Multiple Operating Modes Based on Semisupervised Gaussian Mixture Regression

Dice Semimetric Losses: Optimizing the Dice Score with Soft Labels

Probabilistic Labeled Semi-supervised SVM

SemiGPC: Distribution-Aware Label Refinement for Imbalanced Semi-Supervised Learning Using Gaussian Processes

Scalable Semisupervised GMM for Big Data Quality Prediction in Multimode Processes.

Semi-supervised Gaussian Process Ordinal Regression

Asymptotic Bayes risk of semi-supervised learning with uncertain labeling

Semi-supervised inference for nonparametric logistic regression

SimPro: A Simple Probabilistic Framework Towards Realistic Long-Tailed Semi-Supervised Learning

Semi-Supervised Empirical Risk Minimization: Using unlabeled data to improve prediction

Learning with Confidence: Training Better Classifiers from Soft Labels

Semi-supervised empirical Bayes group-regularized factor regression

A Benchmark Generative Probabilistic Model for Weak Supervised Learning

Posterior Consistency of the Silverman G-Prior in Bayesian Model Choice.

Advances in Bayesian model selection consistency for high-dimensional generalized linear models