Abstract:Semi-supervised learning (SSL) is a machine learning methodology that leverages unlabeled data in conjunction with a limited amount of labeled data. Although SSL has been applied in various applications and its effectiveness has been empirically demonstrated, it is still not fully understood when and why SSL performs well. Some existing theoretical studies have attempted to address this issue by modeling classification problems using the so-called Gaussian Mixture Model (GMM). These studies provide notable and insightful interpretations. However, their analyses are focused on specific purposes, and a thorough investigation of the properties of GMM in the context of SSL has been lacking. In this paper, we conduct such a detailed analysis of the properties of the high-dimensional GMM for binary classification in the SSL setting. To this end, we employ the approximate message passing and state evolution methods, which are widely used in high-dimensional settings and originate from statistical mechanics. We deal with two estimation approaches: the Bayesian one and the l2-regularized maximum likelihood estimation (RMLE). We conduct a comprehensive comparison between these two approaches, examining aspects such as the global phase diagram, estimation error for the parameters, and prediction error for the labels. A specific comparison is made between the Bayes-optimal (BO) estimator and RMLE, as the BO setting provides optimal estimation performance and is ideal as a benchmark. Our analysis shows that with appropriate regularizations, RMLE can achieve near-optimal performance in terms of both the estimation error and prediction error, especially when there is a large amount of unlabeled data. These results demonstrate that the l2 regularization term plays an effective role in estimation and prediction in SSL approaches.

Understanding Self-Supervised Learning via Gaussian Mixture Models

Point Contrastive Prediction with Semantic Clustering for Self-Supervised Learning on Point Cloud Videos

Towards the Generalization of Contrastive Self-Supervised Learning

The Supervised Learning Gaussian Mixture Model

A Probabilistic Approach to Self-Supervised Learning using Cyclical Stochastic Gradient MCMC

A Semi-Supervised Learning Algorithm on Gaussian Mixture with Automatic Model Selection

A Probabilistic Model Behind Self-Supervised Learning

Manifold Contrastive Learning with Variational Lie Group Operators

Analysis of High-dimensional Gaussian Labeled-unlabeled Mixture Model via Message-passing Algorithm

On Discriminative Probabilistic Modeling for Self-Supervised Representation Learning

Supervised Stochastic Neighbor Embedding Using Contrastive Learning

Mixture of Self-Supervised Learning

Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss

Spectral Augmentation for Self-Supervised Learning on Graphs

MNN: Mixed Nearest-Neighbors for Self-Supervised Learning

Self-supervised Gaussian Restricted Boltzmann Machine via joint contrastive representation and contrastive divergence

A Rigorous Link Between Self-Organizing Maps and Gaussian Mixture Models

Learning the Relation between Similarity Loss and Clustering Loss in Self-Supervised Learning

Rethinking Spectral Augmentation for Contrast-based Graph Self-Supervised Learning

On the duality between contrastive and non-contrastive self-supervised learning

Robust Unsupervised Multi-task and Transfer Learning on Gaussian Mixture Models