Semi-supervised learning via sparse model

WangYu,TangSheng,ZhengYan-Tao,ZhangYong-Dong,LiJin-Tao

IF: 6

2014-01-01

Neurocomputing

Abstract:Graph-based Semi-Supervised Learning (SSL) methods are the widely used SSL methods due to their high accuracy. They can well meet the manifold assumption with high computational cost, but donu0027t mee...

What problem does this paper attempt to address?

Semi-supervised Learning Via Sparse Model

Yu Wang,Sheng Tang,Yan-Tao Zheng,Yong-Dong Zhang,Jin-Tao Li

DOI: https://doi.org/10.1016/j.neucom.2013.10.033

IF: 6

2014-01-01

Neurocomputing

Abstract:Graph-based Semi-Supervised Learning (SSL) methods are the widely used SSL methods due to their high accuracy. They can well meet the manifold assumption with high computational cost, but don't meet the cluster assumption. In this paper, we propose a Semi-supervised learning via SPArse (SSPA) model. Since SSPA uses sparse matrix multiplication to depict the adjacency relations among samples, SSPA can approximate low dimensional manifold structure of samples with lower computational complexity than these graph-based SSL methods. Each column of this sparse matrix corresponds to one sparse representation of a sample. The rational is that the inner product of sparse representations can also be sparse under certain constraint. Since the dictionary in the SSPA model can depict the distribution of the entire samples, the sparse representation of a sample encodes its spatial location information. Therefore, in the SSPA model the manifold structure of samples is computed via their locations in the intrinsic geometry of the distribution instead of their feature vectors. In order to meet the cluster assumption, we propose an structured dictionary learning algorithm to explicitly reveal the cluster structure of the dictionary. We develop the SSPA algorithms with the structured dictionary besides non-structured one, and experiments show that our methods are efficient and outperform state-of-the-art graph-based SSL methods.
Scalable Graph-Based Semi-Supervised Learning Through Sparse Bayesian Model.

Bingbing Jiang,Huanhuan Chen,Bo Yuan,Xin Yao

DOI: https://doi.org/10.1109/tkde.2017.2749574

IF: 9.235

2017-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Semi-supervised learning (SSL) concerns the problem of how to improve classifiers’ performance through making use of prior knowledge from unlabeled data. Many SSL methods have been developed to integrate unlabeled data into the classifiers based on either the manifold or cluster assumption in recent years. In particular, the graph-based approaches, following the manifold assumption, have achieved a promising performance in many real-world applications. However, most of them work well on small-scale data sets only and lack probabilistic outputs. In this paper, a scalable graph-based SSL framework through sparse Bayesian model is proposed by defining a graph-based sparse prior. Based on the traditional Bayesian inference technique, a sparse Bayesian SSL algorithm (SBS $^2$ L) is obtained, which can remove the irrelevant unlabeled samples and make probabilistic prediction for out-of-sample data. Moreover, in order to scale SBS $^2$ L to large-scale data sets, an incremental SBS $^2$ L (ISBS$^2$ L) is derived. The key idea of ISBS $^2$ L is employing an incremental strategy and sequentially selecting parts of unlabeled samples that contribute to the learning instead of using all available unlabeled samples directly. ISBS$^2$ L has lower time and space complexities than previous SSL algorithms with the use of all unlabeled samples. Extensive experiments on various data sets verify that our algorithms can achieve comparable classification effectiveness and efficiency with much better scalability. Finally, the generalization error bound is derived based on robustness analysis.
Probabilistic Semi-Supervised Learning via Sparse Graph Structure Learning

Li Wang,Raymond Chan,Tieyong Zeng

DOI: https://doi.org/10.1109/tnnls.2020.2979607

IF: 14.255

2021-02-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:We present a probabilistic semi-supervised learning (SSL) framework based on sparse graph structure learning. Different from existing SSL methods with either a predefined weighted graph heuristically constructed from the input data or a learned graph based on the locally linear embedding assumption, the proposed SSL model is capable of learning a sparse weighted graph from the unlabeled high-dimensional data and a small amount of labeled data, as well as dealing with the noise of the input data. Our representation of the weighted graph is indirectly derived from a unified model of density estimation and pairwise distance preservation in terms of various distance measurements, where latent embeddings are assumed to be random variables following an unknown density function to be learned, and pairwise distances are then calculated as the expectations over the density for the model robustness to the data noise. Moreover, the labeled data based on the same distance representations are leveraged to guide the estimated density for better class separation and sparse graph structure learning. A simple inference approach for the embeddings of unlabeled data based on point estimation and kernel representation is presented. Extensive experiments on various data sets show promising results in the setting of SSL compared with many existing methods and significant improvements on small amounts of labeled data.

computer science, artificial intelligence, theory & methods,engineering, electrical & electronic, hardware & architecture
Self-supervised Learning on Graphs: Contrastive, Generative,or Predictive

Lirong Wu,Haitao Lin,Cheng Tan,Zhangyang Gao,Stan Z. Z. Li

DOI: https://doi.org/10.1109/tkde.2021.3131584

IF: 9.235

2023-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Deep learning on graphs has recently achieved remarkable success on a variety of tasks, while such success relies heavily on the massive and carefully labeled data. However, precise annotations are generally very expensive and time-consuming. To address this problem, self-supervised learning (SSL) is emerging as a new paradigm for extracting informative knowledge through well-designed pretext tasks without relying on manual labels. In this survey, we extend the concept of SSL, which first emerged in the fields of computer vision and natural language processing, to present a timely and comprehensive review of existing SSL techniques for graph data. Specifically, we divide existing graph SSL methods into three categories: contrastive, generative, and predictive. More importantly, unlike other surveys that only provide a high-level description of published research, we present an additional mathematical summary of existing works in a unified framework. Furthermore, to facilitate methodological development and empirical comparisons, we also summarize the commonly used datasets, evaluation metrics, downstream tasks, open-source implementations, and experimental study of various algorithms. Finally, we discuss the technical challenges and potential future directions for improving graph self-supervised learning. Latest advances in graph SSL are summarized in a GitHub repository https://github.com/LirongWu/awesome-graph-self-supervised-learning.
Semi-supervised Classification Via Low Rank Graph

Liansheng Zhuang,Haoyuan Gao,Jingjing Huang,Nenghai Yu

DOI: https://doi.org/10.1109/icig.2011.86

2011-01-01

Abstract:Graph plays a very important role in graph based semi-supervised learning (SSL) methods. However, most current graph construction methods emphasize on local properties of the graph. In this paper, inspired by the advances of compressive sensing, we present a novel method to construct a so-called low-rank graph (LR-graph) for graph based SSL methods. Assuming that the graph is sparse and low rank, our proposed method uses both the local property and the global property of the graph, and thus is better at capturing the global structure of all data. Compared with current graphs, LR-graph is more informative and discriminative, and robust to outliers. Experiments on generic object recognition show that LR-graph achieves state-of-the-art performance for graph based SSL methods.
Efficient And Robust Semi-Supervised Learning Over A Sparse-Regularized Graph

Hang Su,Jun Zhu,Zhaozheng Yin,Yinpeng Dong,Bo Zhang

DOI: https://doi.org/10.1007/978-3-319-46484-8_35

2016-01-01

Abstract:Graph-based Semi-Supervised Learning (GSSL) has limitations in widespread applicability due to its computationally prohibitive large-scale inference, sensitivity to data incompleteness, and incapability on handling time-evolving characteristics in an open set. To address these issues, we propose a novel GSSL based on a batch of informative beacons with sparsity appropriately harnessed, rather than constructing the pairwise affinity graph between the entire original samples. Specifically, (1) beacons are placed automatically by unifying the consistence of both data features and labels, which subsequentially act as indicators during the inference; (2) leveraging the information carried by beacons, the sample labels are interpreted as the weighted combination of a subset of characteristics-specified beacons; (3) if unfamiliar samples are encountered in an open set, we seek to expand the beacon set incrementally and update their parameters by incorporating additional human interventions if necessary. Experimental results on real datasets validate that our algorithm is effective and efficient to implement scalable inference, robust to sample corruptions, and capable to boost the performance incrementally in an open set by updating the beacon-related parameters.
Robust Self-Tuning Semi-Supervised Learning.

Fei Wang,Changshui Zhang

DOI: https://doi.org/10.1016/j.neucom.2006.11.004

IF: 6

2006-01-01

Neurocomputing

Abstract:We investigate the issue of graph-based semi-supervised learning (SSL). The labeled and unlabeled data points are represented as vertices in an undirected weighted neighborhood graph, with the edge weights encoding the pairwise similarities between data objects in the same neighborhood. The SSL problem can be then formulated as a regularization problem on this graph. In this paper we propose a robust self-tuning graph-based SSL method, which (1) can determine the similarities between pairwise data points automatically; (2) is not sensitive to outliers. Promising experimental results are given for both synthetic and real data sets.
Semi-supervised learning with extremely sparse labeled data on multiple semi-supervised assumptions

Lisong Chen,Huanhuan Chen,Ke Tang

DOI: https://doi.org/10.1109/SoCPaR.2011.6089114

2011-01-01

Abstract:Semi-supervised learning with extremely sparse labeled data focus on how to generate robust classifiers when the training dataset consists of 1%-5% labeled data and plenty of unlabeled data. Existing algorithms often suffer from this kind of problems. This paper proposes a multiple semi-supervised assumptions based approach, which typically does not suffer from the major drawback of the former for which adding very few labeled data might actually lead to a serious performance degradation. It combines three semi-supervised assumptions, i.e. smoothness, manifold and cluster assumption, and employs an unlabeled-data-oriented strategy that benefits from the perspective of unsupervised learning. Experimental results on various datasets demonstrate that our algorithm exhibits strong robustness against extremely sparse labeled data and outperforms a number of existing SSL techniques.
Graph-based Semi-supervised Learning: A Comprehensive Review

Zixing Song,Xiangli Yang,Zenglin Xu,Irwin King

DOI: https://doi.org/10.48550/arXiv.2102.13303

2021-02-26

Abstract:Semi-supervised learning (SSL) has tremendous value in practice due to its ability to utilize both labeled data and unlabelled data. An important class of SSL methods is to naturally represent data as graphs such that the label information of unlabelled samples can be inferred from the graphs, which corresponds to graph-based semi-supervised learning (GSSL) methods. GSSL methods have demonstrated their advantages in various domains due to their uniqueness of structure, the universality of applications, and their scalability to large scale data. Focusing on this class of methods, this work aims to provide both researchers and practitioners with a solid and systematic understanding of relevant advances as well as the underlying connections among them. This makes our paper distinct from recent surveys that cover an overall picture of SSL methods while neglecting fundamental understanding of GSSL methods. In particular, a major contribution of this paper lies in a new generalized taxonomy for GSSL, including graph regularization and graph embedding methods, with the most up-to-date references and useful resources such as codes, datasets, and applications. Furthermore, we present several potential research directions as future work with insights into this rapidly growing field.

Machine Learning
Locating High-Density Clusters with Noisy Queries.

Chen Cao,Shifeng Chen,Changqing Zou,Jianzhuang Liu

2012-01-01

Abstract:Semi-supervised learning (SSL) relies on a few labeled samples to explore data's intrinsic structure through pairwise smooth transduction. The performance of SSL mainly depends on two folds: (1) the accuracy of labeled queries, (2) the integrity of manifolds in data distribution. Both of these qualities would be poor in real applications as data often consist of several irrelevant clusters and discrete noise. In this paper we propose a novel framework to simultaneously remove discrete noise and locate the high-density clusters. Experiments demonstrate that our algorithm is quite effective to solve several problems such as non-feedback image re-ranking and image co-segmentation.
Image colourisation using graph-based semi-supervised learning

B.-B. Liu,Z.-M. Lu

DOI: https://doi.org/10.1049/iet-ipr.2008.0112

IF: 2.3

2009-01-01

IET Image Processing

Abstract:A novel colourisation algorithm using graph-based semi-supervised learning (SSL) is presented. We show that the assumption of the colourisation problem is consistent with the fundamental of graph-based SSL methods. Satisfactory results are obtained in the experiments that validate the proposed algorithm. To reduce the time and memory requirements when dealing with large scale images, we further pr...
Interactive Graph Construction for Graph-Based Semi-Supervised Learning.

Changjian Chen,Zhaowei Wang,Jing Wu,Xiting Wang,Lan-Zhe Guo,Yu-Feng Li,Shixia Liu

DOI: https://doi.org/10.1109/tvcg.2021.3084694

IF: 5.2

2021-01-01

IEEE Transactions on Visualization and Computer Graphics

Abstract:Semi-supervised learning (SSL) provides a way to improve the performance of prediction models (e.g., classifier) via the usage of unlabeled samples. An effective and widely used method is to construct a graph that describes the relationship between labeled and unlabeled samples. Practical experience indicates that graph quality significantly affects the model performance. In this paper, we present a visual analysis method that interactively constructs a high-quality graph for better model performance. In particular, we propose an interactive graph construction method based on the large margin principle. We have developed a river visualization and a hybrid visualization that combines a scatterplot, a node-link diagram, and a bar chart to convey the label propagation of graph-based SSL. Based on the understanding of the propagation, a user can select regions of interest to inspect and modify the graph. We conducted two case studies to showcase how our method facilitates the exploitation of labeled and unlabeled samples for improving model performance.
Enhanced low-rank representation via sparse manifold adaption for semi-supervised learning

Yong Peng,Bao-Liang Lu,Suhang Wang

DOI: https://doi.org/10.1016/j.neunet.2015.01.001

Abstract:Constructing an informative and discriminative graph plays an important role in various pattern recognition tasks such as clustering and classification. Among the existing graph-based learning models, low-rank representation (LRR) is a very competitive one, which has been extensively employed in spectral clustering and semi-supervised learning (SSL). In SSL, the graph is composed of both labeled and unlabeled samples, where the edge weights are calculated based on the LRR coefficients. However, most of existing LRR related approaches fail to consider the geometrical structure of data, which has been shown beneficial for discriminative tasks. In this paper, we propose an enhanced LRR via sparse manifold adaption, termed manifold low-rank representation (MLRR), to learn low-rank data representation. MLRR can explicitly take the data local manifold structure into consideration, which can be identified by the geometric sparsity idea; specifically, the local tangent space of each data point was sought by solving a sparse representation objective. Therefore, the graph to depict the relationship of data points can be built once the manifold information is obtained. We incorporate a regularizer into LRR to make the learned coefficients preserve the geometric constraints revealed in the data space. As a result, MLRR combines both the global information emphasized by low-rank property and the local information emphasized by the identified manifold structure. Extensive experimental results on semi-supervised classification tasks demonstrate that MLRR is an excellent method in comparison with several state-of-the-art graph construction approaches.
Semi-supervised Active Learning for Semi-supervised Models: Exploit Adversarial Examples with Graph-based Virtual Labels

Jiannan Guo,Haochen Shi,Yangyang Kang,Kun Kuang,Siliang Tang,Zhuoren Jiang,Changlong Sun,Fei Wu,Yueting Zhuang

DOI: https://doi.org/10.1109/iccv48922.2021.00289

2021-01-01

Abstract:The performance of computer vision models significantly improves with more labeled data. However, the acquisition of labeled data is limited by the high cost. To mitigate the reliance on large labeled datasets, active learning (AL) and semi-supervised learning (SSL) are frequently adopted. Although current mainstream methods begin to combine SSL and AL (SSL-AL) to excavate the diverse expressions of unlabeled samples, these methods’ fully supervised task models are still trained only with labeled data. Besides, these method’s SSL-AL frameworks suffer from mismatch problems. Here, we propose a graph-based SSL-AL framework to unleash the SSL task models’ power and make an effective SSL-AL interaction. In the framework, SSL leverages graph-based label propagation to deliver virtual labels to unlabeled samples, rendering AL samples’ structural distribution and boosting AL. AL finds samples near the clusters’ boundary to help SSL perform better label propagation by exploiting adversarial examples. The information exchange in the closed-loop realizes mutual enhancement of SSL and AL. Experimental results show that our method outperforms the state-of-the-art methods against classification and segmentation benchmarks.
Semi-Supervised Learning with Close-Form Label Propagation Using a Bipartite Graph

Zhongxing Peng,Gengzhong Zheng,Wei Huang

DOI: https://doi.org/10.3390/sym16101312

2024-10-05

Symmetry

Abstract:In this paper, we introduce an efficient and effective algorithm for Graph-based Semi-Supervised Learning (GSSL). Unlike other GSSL methods, our proposed algorithm achieves efficiency by constructing a bipartite graph, which connects a small number of representative points to a large volume of raw data by capturing their underlying manifold structures. This bipartite graph, with a sparse and anti-diagonal affinity matrix which is symmetrical, serves as a low-rank approximation of the original graph. Consequently, our algorithm accelerates both the graph construction and label propagation steps. In particular, on the one hand, our algorithm computes the label propagation in closed-form, reducing its computational complexity from cubic to approximately linear with respect to the number of data points; on the other hand, our algorithm calculates the soft label matrix for unlabeled data using a closed-form solution, thereby gaining additional acceleration. Comprehensive experiments performed on six real-world datasets demonstrate the efficiency and effectiveness of our algorithm in comparison to five state-of-the-art algorithms.

multidisciplinary sciences
Semi-supervised Classification Using Local and Global Regularization

Fei Wang,Tao Li,Gang Wang,Changshui Zhang

2008-01-01

Abstract:In this paper, we propose a semi-supervised learning (SSL) algorithm based on local and global regularization. In the local regularization part, our algorithm constructs a regularized classifier for each data point using its neighborhood, while the global regularization part adopts a Laplacian regularizer to smooth the data labels predicted by those local classifiers. We show that some existing SSL algorithms can be derived from our framework. Finally we present some experimental results to show the effectiveness of our method.
Learning safe graph construction from multiple graphs

De-Ming Liang,Yu-Feng Li

DOI: https://doi.org/10.1007/978-981-13-2122-1_4

2018-01-01

Abstract:Graph-based method is one important paradigm of semi-supervised learning (SSL). Its learning performance typically relies on excellent graph construction which, however, remains challenging for general cases. What is more serious, constructing graph improperly may even deteriorate performance, which means its performance is worse than that of its supervised counterpart with only labeled data. For this reason, we consider learning a safe graph construction for graph-based SSL in this work such that its performance will not significantly perform worse than its supervised counterpart. Our basic idea is that, given a data distribution, there often exist some dense areas which are robust to graph construction. We then propose to combine trustable subgraphs in these areas from a set of candidate graphs to derive a safe graph, which remains to be a convex problem. Experimental results on a number of datasets show that our proposal is able to effectively avoid performance degeneration compared with many graph-based SSL methods.
Robust Semi-Supervised Learning when Not All Classes have Labels

Lan-Zhe Guo,Yi-Ge Zhang,Zhi-Fan Wu,Jie-Jing Shao,Yu-Feng Li

2022-01-01

Abstract:Semi-supervised learning (SSL) provides a powerful framework for leveraging unlabeled data. Existing SSL typically requires all classes have labels. However, in many real-world applications, there may exist some classes that are difficult to label or newly occurred classes that cannot be labeled in time, resulting in there are unseen classes in unlabeled data. Unseen classes will be misclassified as seen classes, causing poor classification performance. The performance of seen classes is also harmed by the existence of unseen classes. This limits the practical and wider application of SSL. To address this problem, this paper proposes a new SSL approach that can classify not only seen classes but also unseen classes. Our approach consists of two modules: unseen class classification and learning pace synchronization. Specifically, we first enable the SSL methods to classify unseen classes by exploiting pairwise similarity between examples and then synchronize the learning pace between seen and unseen classes by proposing an adaptive threshold with distribution alignment. Extensive empirical results show our approach achieves significant performance improvement in both seen and unseen classes compared with previous studies.
Distributionally Robust Semi-Supervised Learning Over Graphs

Alireza Sadeghi,Meng Ma,Bingcong Li,Georgios B. Giannakis

DOI: https://doi.org/10.48550/arXiv.2110.10582

2021-10-20

Abstract:Semi-supervised learning (SSL) over graph-structured data emerges in many network science applications. To efficiently manage learning over graphs, variants of graph neural networks (GNNs) have been developed recently. By succinctly encoding local graph structures and features of nodes, state-of-the-art GNNs can scale linearly with the size of graph. Despite their success in practice, most of existing methods are unable to handle graphs with uncertain nodal attributes. Specifically whenever mismatches between training and testing data distribution exists, these models fail in practice. Challenges also arise due to distributional uncertainties associated with data acquired by noisy measurements. In this context, a distributionally robust learning framework is developed, where the objective is to train models that exhibit quantifiable robustness against perturbations. The data distribution is considered unknown, but lies within a Wasserstein ball centered around empirical data distribution. A robust model is obtained by minimizing the worst expected loss over this ball. However, solving the emerging functional optimization problem is challenging, if not impossible. Advocating a strong duality condition, we develop a principled method that renders the problem tractable and efficiently solvable. Experiments assess the performance of the proposed method.

Machine Learning,Artificial Intelligence
Robust Deep Semi-Supervised Learning: A Brief Introduction

Lan-Zhe Guo,Zhi Zhou,Yu-Feng Li

DOI: https://doi.org/10.48550/arXiv.2202.05975

2022-11-14

Abstract:Semi-supervised learning (SSL) is the branch of machine learning that aims to improve learning performance by leveraging unlabeled data when labels are insufficient. Recently, SSL with deep models has proven to be successful on standard benchmark tasks. However, they are still vulnerable to various robustness threats in real-world applications as these benchmarks provide perfect unlabeled data, while in realistic scenarios, unlabeled data could be corrupted. Many researchers have pointed out that after exploiting corrupted unlabeled data, SSL suffers severe performance degradation problems. Thus, there is an urgent need to develop SSL algorithms that could work robustly with corrupted unlabeled data. To fully understand robust SSL, we conduct a survey study. We first clarify a formal definition of robust SSL from the perspective of machine learning. Then, we classify the robustness threats into three categories: i) distribution corruption, i.e., unlabeled data distribution is mismatched with labeled data; ii) feature corruption, i.e., the features of unlabeled examples are adversarially attacked; and iii) label corruption, i.e., the label distribution of unlabeled data is imbalanced. Under this unified taxonomy, we provide a thorough review and discussion of recent works that focus on these issues. Finally, we propose possible promising directions within robust SSL to provide insights for future research.

Machine Learning

Semi-supervised learning via sparse model

Semi-supervised Learning Via Sparse Model

Scalable Graph-Based Semi-Supervised Learning Through Sparse Bayesian Model.

Probabilistic Semi-Supervised Learning via Sparse Graph Structure Learning

Self-supervised Learning on Graphs: Contrastive, Generative,or Predictive

Semi-supervised Classification Via Low Rank Graph

Efficient And Robust Semi-Supervised Learning Over A Sparse-Regularized Graph

Robust Self-Tuning Semi-Supervised Learning.

Semi-supervised learning with extremely sparse labeled data on multiple semi-supervised assumptions

Graph-based Semi-supervised Learning: A Comprehensive Review

Locating High-Density Clusters with Noisy Queries.

Image colourisation using graph-based semi-supervised learning

Interactive Graph Construction for Graph-Based Semi-Supervised Learning.

Enhanced low-rank representation via sparse manifold adaption for semi-supervised learning

Semi-supervised Active Learning for Semi-supervised Models: Exploit Adversarial Examples with Graph-based Virtual Labels

Semi-Supervised Learning with Close-Form Label Propagation Using a Bipartite Graph

Semi-supervised Classification Using Local and Global Regularization

Learning safe graph construction from multiple graphs

Robust Semi-Supervised Learning when Not All Classes have Labels

Distributionally Robust Semi-Supervised Learning Over Graphs

Robust Deep Semi-Supervised Learning: A Brief Introduction