Abstract:Semi-Supervised Learning (SSL) has been proved to be an effective way to leverage both labeled and unlabeled data at the same time. Recent semi-supervised approaches focus on deep neural networks and have achieved promising results on several benchmarks: CIFAR10, CIFAR100 and SVHN. However, most of their experiments are based on models trained from scratch instead of pre-trained models. On the other hand, transfer learning has demonstrated its value when the target domain has limited labeled data. Here comes the intuitive question: is it possible to incorporate SSL when fine-tuning a pre-trained model? We comprehensively study how SSL methods starting from pretrained models perform under varying conditions, including training strategies, architecture choice and datasets. From this study, we obtain several interesting and useful observations. While practitioners have had an intuitive understanding of these observations, we do a comprehensive emperical analysis and demonstrate that: (1) the gains from SSL techniques over a fully-supervised baseline are smaller when trained from a pre-trained model than when trained from random initialization, (2) when the domain of the source data used to train the pre-trained model differs significantly from the domain of the target task, the gains from SSL are significantly higher and (3) some SSL methods are able to advance fully-supervised baselines (like Pseudo-Label). We hope our studies can deepen the understanding of SSL research and facilitate the process of developing more effective SSL methods to utilize pre-trained models. Code is now available at github.

Open World Semi-supervised Learning Based on Multi-scale Enhanced Feature

Label Distribution-based Open-world Semi-supervised Learning

Robust Semi-Supervised Learning for Self-learning Open-World Classes

Exploration and Exploitation of Unlabeled Data for Open-Set Semi-Supervised Learning

Multi-Task Curriculum Framework for Open-Set Semi-Supervised Learning

Open-Domain Semi-Supervised Learning via Glocal Cluster Structure Exploitation

Open-World Semi-Supervised Learning for Node Classification

Promote knowledge mining towards open-world semi-supervised learning

Open-world Barely-Supervised Learning Via Augmented Pseudo Labels

Robust Semi-Supervised Learning when Not All Classes have Labels

Open-Set Semi-Supervised Learning for 3D Point Cloud Understanding

An Empirical Study and Analysis on Open-Set Semi-Supervised Learning

Enhancing Semi-Supervised Learning with Cross-Modal Knowledge

Rethinking Open-World Semi-Supervised Learning: Distribution Mismatch and Inductive Inference

When Semi-Supervised Learning Meets Transfer Learning: Training Strategies, Models and Datasets.

Improvement of Semi-Supervised Learning in Real Application Scenarios

Trash to Treasure: Harvesting OOD Data with Cross-Modal Matching for Open-Set Semi-Supervised Learning

Generalized Semi-Supervised Learning via Self-Supervised Feature Adaptation

Towards Realistic Long-tailed Semi-supervised Learning in an Open World

Improving Barely Supervised Learning by Discriminating Unlabeled Samples with Super-Class

OCI-SSL: Open Class-Imbalanced Semi-Supervised Learning with Contrastive Learning