COMMENTS ON THE SKIDMORE AND TURNER SUPERVISED NONPARAMETRIC CLASSIFIER

GONG,P,DUNLOP,JD

1991-01-01

Abstract:In this paper, the supervised nonparametric classifier proposed by Skidmore and Turner (1988) is discussed. Because the notation in the Skidmore and Turner paper is enigmatic with a published erratum (Skidmore, 1989), the algorithm has been restated in a consolidated form. The computational efficiency of the original algorithm is improved.

What problem does this paper attempt to address?

Computational and Theoretical Analysis of Supervised Dimensionality Reduction

赵武锋,沈海斌,严晓浪

DOI: https://doi.org/10.3785/j.issn.1008-9497.2009.06.015

2009-01-01

Abstract:In the case of small sample size,classical linear discriminant analysis fails due to the singularity of scatter matrices for supervised dimensionality reduction.Many extensions were proposed in the past to overcome this problem,which in general can be classified into three categories: methods based on the null space of the within-class scatter matrix,ones based on the range space of the total scatter matrix and ones based on other subspace.In order to better understand the characteristics of the algorithms of the former two classes,a computational and theoretical analysis was carried out,and concluded that: under a mild condition which holds in many applications involving high-dimensional data,methods based on the null space of the within-class scatter matrix are equivalent to those based on the range space of the total scatter matrix except two differences of constraints for discriminant vectors and implementation procedure.The comparative results on the face database,ORL and YALE,also confirmed the aforementioned conclusion.
On Discriminative Semi-Supervised Classification.

Fei Wang,Changshui Zhang

2008-01-01

Abstract:The recent years have witnessed a surge of interests in semi-supervised learning methods. A common strategy for these algorithms is to require that the predicted data labels should be sufficiently smooth with respect to the intrinsic data manifold. In this paper, we argue that rather than penalizing the label smoothness, we can directly punish the discriminality of the classification function to achieve a more powerful predictor, and we derive two specific algorithms: Semi-Supervised Discriminative Regularization (SSDR) and Semi-parametric Discriminative Semi-supervised Classification (SDSC). Finally many experimental results are presented to show the effectiveness of our method.
Nonparametric Unsupervised Classification

Yingzhen Yang,Thomas S. Huang

DOI: https://doi.org/10.48550/arXiv.1210.0645

IF: 5.414

2012-10-02

Machine Learning

Abstract:Unsupervised classification methods learn a discriminative classifier from unlabeled data, which has been proven to be an effective way of simultaneously clustering the data and training a classifier from the data. Various unsupervised classification methods obtain appealing results by the classifiers learned in an unsupervised manner. However, existing methods do not consider the misclassification error of the unsupervised classifiers except unsupervised SVM, so the performance of the unsupervised classifiers is not fully evaluated. In this work, we study the misclassification error of two popular classifiers, i.e. the nearest neighbor classifier (NN) and the plug-in classifier, in the setting of unsupervised classification.
Central Limit Theorems of a Recursive Stochastic Algorithm with Applications to Adaptive Designs

Li-Xin Zhang

DOI: https://doi.org/10.1214/16-aap1187

2016-01-01

The Annals of Applied Probability

Abstract:Stochastic approximation algorithms have been the subject of an enormous body of literature, both theoretical and applied. Recently, Laruelle and Pag\`es (2013) presented a link between the stochastic approximation and response-adaptive designs in clinical trials based on randomized urn models investigated in Bai and Hu (1999, 2005), and derived the asymptotic normality or central limit theorem for the normalized procedure using a central limit theorem for the stochastic approximation algorithm. However, the classical central limit theorem for the stochastic approximation algorithm does not include all cases of its regression function, creating a gap between the results of Laruelle and Pag\`es (2013) and those of Bai and Hu (2005) for randomized urn models. In this paper, we establish new central limit theorems of the stochastic approximation algorithm under the popular Lindeberg condition to fill this gap. Moreover, we prove that the process of the algorithms can be approximated by a Gaussian process that is a solution of a stochastic differential equation. In our application, we investigate a more involved family of urn models and related adaptive designs in which it is possible to remove the balls from the urn, and the expectation of the total number of balls updated at each stage is not necessary a constant. The asymptotic properties are derived under much less stringent assumptions than those in Bai and Hu (1999, 2005) and Laruelle and Pag\`es (2013).
Penalizing Proposals using Classifiers for Semi-Supervised Object Detection

Somnath Hazra,Pallab Dasgupta

DOI: https://doi.org/10.48550/arXiv.2205.13219

2022-06-02

Abstract:Obtaining gold standard annotated data for object detection is often costly, involving human-level effort. Semi-supervised object detection algorithms solve the problem with a small amount of gold-standard labels and a large unlabelled dataset used to generate silver-standard labels. But training on the silver standard labels does not produce good results, because they are machine-generated annotations. In this work, we design a modified loss function to train on large silver standard annotated sets generated by a weak annotator. We include a confidence metric associated with the annotation as an additional term in the loss function, signifying the quality of the annotation. We test the effectiveness of our approach on various test sets and use numerous variations to compare the results with some of the current approaches to object detection. In comparison with the baseline where no confidence metric is used, we achieved a 4% gain in mAP with 25% labeled data and 10% gain in mAP with 50% labeled data by using the proposed confidence metric.

Computer Vision and Pattern Recognition,Machine Learning
Learning with Inadequate and Incorrect Supervision

Chen Gong,Hengmin Zhang,Jian Yang,Dacheng Tao

DOI: https://doi.org/10.1109/icdm.2017.110

2017-01-01

Abstract:Practically, we are often in the dilemma that the labeled data at hand are inadequate to train a reliable classifier, and more seriously, some of these labeled data may be mistakenly labeled due to the various human factors. Therefore, this paper proposes a novel semi-supervised learning paradigm that can handle both label insufficiency and label inaccuracy. To address label insufficiency, we use a graph to bridge the data points so that the label information can be propagated from the scarce labeled examples to unlabeled examples along the graph edges. To address label inaccuracy, Graph Trend Filtering (GTF) and Smooth Eigenbase Pursuit (SEP) are adopted to filter out the initial noisy labels. GTF penalizes the l_0 norm of label difference between connected examples in the graph and exhibits better local adaptivity than the traditional l_2 norm-based Laplacian smoother. SEP reconstructs the correct labels by emphasizing the leading eigenvectors of Laplacian matrix associated with small eigenvalues, as these eigenvectors reflect real label smoothness and carry rich class separation cues. We term our algorithm as "Semi-supervised learning under Inadequate and Incorrect Supervision" (SIIS). Thorough experimental results on image classification, text categorization, and speech recognition demonstrate that our SIIS is effective in label error correction, leading to superior performance to the state-of-the-art methods in the presence of label noise and label scarcity.
A replica analysis of Self-Training of Linear Classifier

Takashi Takahashi

2024-05-07

Abstract:Self-training (ST) is a simple and standard approach in semi-supervised learning that has been applied to many machine learning problems. Despite its widespread acceptance and practical effectiveness, it is still not well understood why and how ST improves performance by fitting the model to potentially erroneous pseudo-labels. To investigate the properties of ST, in this study, we derive and analyze a sharp characterization of the behavior of iterative ST when training a linear classifier by minimizing the ridge-regularized convex loss for binary Gaussian mixtures, in the asymptotic limit where input dimension and data size diverge proportionally. The derivation is based on the replica method of statistical mechanics. The result indicates that, when the total number of iterations is large, ST may find a classification plane with the optimal direction regardless of the label imbalance by accumulating small parameter updates over long iterations. It is argued that this is because the small update of ST can accumulate information of the data in an almost noiseless way. However, when a label imbalance is present in true labels, the performance of the ST is significantly lower than that of supervised learning with true labels, because the ratio between the norm of the weight and the magnitude of the bias can become significantly large. To overcome the problems in label imbalanced cases, several heuristics are introduced. By numerically analyzing the asymptotic formula, it is demonstrated that with the proposed heuristics, ST can find a classifier whose performance is nearly compatible with supervised learning using true labels even in the presence of significant label imbalance.

Machine Learning,Disordered Systems and Neural Networks,Statistical Mechanics,Statistics Theory
Parametric Classification for Generalized Category Discovery: A Baseline Study

Xin Wen,Bingchen Zhao,Xiaojuan Qi

2023-12-15

Abstract:Generalized Category Discovery (GCD) aims to discover novel categories in unlabelled datasets using knowledge learned from labelled samples. Previous studies argued that parametric classifiers are prone to overfitting to seen categories, and endorsed using a non-parametric classifier formed with semi-supervised k-means. However, in this study, we investigate the failure of parametric classifiers, verify the effectiveness of previous design choices when high-quality supervision is available, and identify unreliable pseudo-labels as a key problem. We demonstrate that two prediction biases exist: the classifier tends to predict seen classes more often, and produces an imbalanced distribution across seen and novel categories. Based on these findings, we propose a simple yet effective parametric classification method that benefits from entropy regularisation, achieves state-of-the-art performance on multiple GCD benchmarks and shows strong robustness to unknown class numbers. We hope the investigation and proposed simple framework can serve as a strong baseline to facilitate future studies in this field. Our code is available at: <a class="link-external link-https" href="https://github.com/CVMI-Lab/SimGCD" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Machine Learning
Semi-supervised inference for nonparametric logistic regression

Tong Wang,Wenlu Tang,Yuanyuan Lin,Wen Su

DOI: https://doi.org/10.1002/sim.9737

2023-07-10

Abstract:We consider the problem of estimating the nonparametric function in nonparametric logistic regression under semi-supervised framework, where a relatively small size labeled data set collected by case-control sampling and a relatively large size of unlabeled data containing only observations of predictors are available. This problem arises in various applications when the outcome variable is expensive or difficult to be observed directly. A two-stage nonparametric semi-supervised estimator based on spline method is proposed to estimate the target regression function by maximizing the likelihood function of the labeled case-control data. The unlabeled data are used in the first stage for estimating the density function that involves in the likelihood function. The consistency and functional asymptotic normality of the semi-supervised two-stage estimator are established under mild conditions. The proposed method, by making use of the unlabeled data, produces more efficient estimation of the target function than the traditional supervised counterpart. The performance of the proposed method is evaluated through extensive simulation studies. An application is illustrated with an analysis of a skin segmentation data.
Revisiting Discriminative Vs. Generative Classifiers: Theory and Implications

Chenyu Zheng,Guoqiang Wu,Fan Bao,Yue Cao,Chongxuan Li,Jun Zhu

2023-01-01

Abstract:A large-scale deep model pre-trained on massive labeled or unlabeled data transfers well to downstream tasks. Linear evaluation freezes parameters in the pre-trained model and trains a linear classifier separately, which is efficient and attractive for transfer. However, little work has investigated the classifier in linear evaluation except for the default logistic regression. Inspired by the statistical efficiency of naive Bayes, the paper revisits the classical topic on discriminative vs. generative classifiers. Theoretically, the paper considers the surrogate loss instead of the zero-one loss in analyses and generalizes the classical results from binary cases to multiclass ones. We show that, under mild assumptions, multiclass naive Bayes requires $O(\log n)$ samples to approach its asymptotic error while the corresponding multiclass logistic regression requires $O(n)$ samples, where $n$ is the feature dimension. To establish it, we present a multiclass $\mathcal{H}$-consistency bound framework and an explicit bound for logistic loss, which are of independent interests. Simulation results on a mixture of Gaussian validate our theoretical findings. Experiments on various pre-trained deep vision models show that naive Bayes consistently converges faster as the number of data increases. Besides, naive Bayes shows promise in few-shot cases and we observe the "two regimes'' phenomenon in pre-trained supervised models. Our code is available at https://github.com/ML-GSAI/Revisiting-Dis-vs-Gen-Classifiers.
Semisupervised Classifier Evaluation and Recalibration

Peter Welinder,Max Welling,Pietro Perona

DOI: https://doi.org/10.48550/arXiv.1210.2162

2012-10-08

Abstract:How many labeled examples are needed to estimate a classifier's performance on a new dataset? We study the case where data is plentiful, but labels are expensive. We show that by making a few reasonable assumptions on the structure of the data, it is possible to estimate performance curves, with confidence bounds, using a small number of ground truth labels. Our approach, which we call Semisupervised Performance Evaluation (SPE), is based on a generative model for the classifier's confidence scores. In addition to estimating the performance of classifiers on new datasets, SPE can be used to recalibrate a classifier by re-estimating the class-conditional confidence distributions.

Machine Learning,Computer Vision and Pattern Recognition
An interpretable semi-supervised classifier using two different strategies for amended self-labeling

Isel Grau,Dipankar Sengupta,Maria M. Garcia Lorenzo,Ann Nowe

DOI: https://doi.org/10.48550/arXiv.2001.09502

2020-07-20

Abstract:In the context of some machine learning applications, obtaining data instances is a relatively easy process but labeling them could become quite expensive or tedious. Such scenarios lead to datasets with few labeled instances and a larger number of unlabeled ones. Semi-supervised classification techniques combine labeled and unlabeled data during the learning phase in order to increase the classifier's generalization capability. Regrettably, most successful semi-supervised classifiers do not allow explaining their outcome, thus behaving like black boxes. However, there is an increasing number of problem domains in which experts demand a clear understanding of the decision process. In this paper, we report on an extended experimental study presenting an interpretable self-labeling grey-box classifier that uses a black box to estimate the missing class labels and a white box to explain the final predictions. Two different approaches for amending the self-labeling process are explored: a first one based on the confidence of the black box and the latter one based on measures from Rough Set Theory. The results of the extended experimental study support the interpretability by means of transparency and simplicity of our classifier, while attaining superior prediction rates when compared with state-of-the-art self-labeling classifiers reported in the literature.

Machine Learning
Classifier Learning with Supervised Marginal Likelihood

Petri Kontkanen,Petri Myllymaki,Henry Tirri

DOI: https://doi.org/10.48550/arXiv.1301.2284

IF: 5.414

2013-01-10

Machine Learning

Abstract:It has been argued that in supervised classification tasks, in practice it may be more sensible to perform model selection with respect to some more focused model selection score, like the supervised (conditional) marginal likelihood, than with respect to the standard marginal likelihood criterion. However, for most Bayesian network models, computing the supervised marginal likelihood score takes exponential time with respect to the amount of observed data. In this paper, we consider diagnostic Bayesian network classifiers where the significant model parameters represent conditional distributions for the class variable, given the values of the predictor variables, in which case the supervised marginal likelihood can be computed in linear time with respect to the data. As the number of model parameters grows in this case exponentially with respect to the number of predictors, we focus on simple diagnostic models where the number of relevant predictors is small, and suggest two approaches for applying this type of models in classification. The first approach is based on mixtures of simple diagnostic models, while in the second approach we apply the small predictor sets of the simple diagnostic models for augmenting the Naive Bayes classifier.
Optimizing Diffusion Rate and Label Reliability in a Graph-Based Semi-supervised Classifier

Bruno Klaus de Aquino Afonso,Lilian Berton

DOI: https://doi.org/10.1007/978-3-030-91702-9_34

2021-01-01

Abstract:Semi-supervised learning has received attention from researchers, as it allows one to exploit the structure of unlabeled data to achieve competitive classification results with much fewer labels than supervised approaches. The Local and Global Consistency (LGC) algorithm is one of the most well-known graph-based semi-supervised (GSSL) classifiers. Notably, its solution can be written as a linear combination of the known labels. The coefficients of this linear combination depend on a parameter α\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\alpha $$\end{document}, determining the decay of the reward over time when reaching labeled vertices in a random walk. In this work, we discuss how removing the self-influence of a labeled instance may be beneficial, and how it relates to leave-one-out error. Moreover, we propose to minimize this leave-one-out loss with automatic differentiation. Within this framework, we propose methods to estimate label reliability and diffusion rate. Optimizing the diffusion rate is more efficiently accomplished with a spectral representation. Results show that the label reliability approach competes with robust ℓ1\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\ell _1$$\end{document}-norm methods and that removing diagonal entries reduces the risk of overfitting and leads to suitable criteria for parameter selection.
Supervised Classification: Quite a Brief Overview

Marco Loog

DOI: https://doi.org/10.48550/arXiv.1710.09230

2017-10-25

Abstract:The original problem of supervised classification considers the task of automatically assigning objects to their respective classes on the basis of numerical measurements derived from these objects. Classifiers are the tools that implement the actual functional mapping from these measurements---also called features or inputs---to the so-called class label---or output. The fields of pattern recognition and machine learning study ways of constructing such classifiers. The main idea behind supervised methods is that of learning from examples: given a number of example input-output relations, to what extent can the general mapping be learned that takes any new and unseen feature vector to its correct class? This chapter provides a basic introduction to the underlying ideas of how to come to a supervised classification problem. In addition, it provides an overview of some specific classification techniques, delves into the issues of object representation and classifier evaluation, and (very) briefly covers some variations on the basic supervised classification task that may also be of interest to the practitioner.

Machine Learning,Computer Vision and Pattern Recognition
Super-klust: Another Way of Piecewise Linear Classification

Rahman Salim Zengin,Volkan Sezer

DOI: https://doi.org/10.48550/arXiv.2102.01571

2021-02-02

Abstract:With our previous study, the Super-k algorithm, we have introduced a novel way of piecewise-linear classification. While working on the Super-k algorithm, we have found that there is a similar, and simpler way to explain for obtaining a piecewise-linear classifier based on Voronoi tessellations. Replacing the multidimensional voxelization and expectation-maximization stages of the algorithm with a distance-based clustering algorithm, preferably k-means, works as well as the prior approach. Since we are replacing the voxelization with the clustering, we have found it meaningful to name the modified algorithm, with respect to Super-k, as Supervised k Clusters or in short Super-klust. Similar to the Super-k algorithm, the Super-klust algorithm covers data with a labeled Voronoi tessellation, and uses resulting tessellation for classification. According to the experimental results, the Super-klust algorithm has similar performance characteristics with the Super-k algorithm.

Machine Learning
A Second Order Cone Programming Approach for Semi-Supervised Learning

Gao Huang,Shiji Song,Jatinder N. D. Gupta,Cheng Wu

DOI: https://doi.org/10.1016/j.patcog.2013.06.016

IF: 8

2013-01-01

Pattern Recognition

Abstract:Semi-supervised learning (SSL) involves the training of a decision rule from both labeled and unlabeled data. In this paper, we propose a novel SSL algorithm based on the multiple clusters per class assumption. The proposed algorithm consists of two stages. In the first stage, we aim to capture the local cluster structure of the training data by using the k-nearest-neighbor (kNN) algorithm to split the data into a number of disjoint subsets. In the second stage, a maximal margin classifier based on the second order cone programming (SOCP) is introduced to learn an inductive decision function from the obtained subsets globally. For linear classification problems, once the kNN algorithm has been performed, the proposed algorithm trains a classifier using only the first and second order moments of the subsets without considering individual data points. Since the number of subsets is usually much smaller than the number of training points, the proposed algorithm is efficient for handling big data sets with a large amount of unlabeled data. Despite its simplicity, the classification performance of the proposed algorithm is guaranteed by the maximal margin classifier. We demonstrate the efficiency and effectiveness of the proposed algorithm on both synthetic and real-world data sets.
Performance and Interpretability Comparisons of Supervised Machine Learning Algorithms: An Empirical Study

Alice J. Liu,Arpita Mukherjee,Linwei Hu,Jie Chen,Vijayan N. Nair

DOI: https://doi.org/10.48550/arXiv.2204.12868

2022-05-05

Abstract:This paper compares the performances of three supervised machine learning algorithms in terms of predictive ability and model interpretation on structured or tabular data. The algorithms considered were scikit-learn implementations of extreme gradient boosting machines (XGB) and random forests (RFs), and feedforward neural networks (FFNNs) from TensorFlow. The paper is organized in a findings-based manner, with each section providing general conclusions supported by empirical results from simulation studies that cover a wide range of model complexity and correlation structures among predictors. We considered both continuous and binary responses of different sample sizes. Overall, XGB and FFNNs were competitive, with FFNNs showing better performance in smooth models and tree-based boosting algorithms performing better in non-smooth models. This conclusion held generally for predictive performance, identification of important variables, and determining correct input-output relationships as measured by partial dependence plots (PDPs). FFNNs generally had less over-fitting, as measured by the difference in performance between training and testing datasets. However, the difference with XGB was often small. RFs did not perform well in general, confirming the findings in the literature. All models exhibited different degrees of bias seen in PDPs, but the bias was especially problematic for RFs. The extent of the biases varied with correlation among predictors, response type, and data set sample size. In general, tree-based models tended to over-regularize the fitted model in the tails of predictor distributions. Finally, as to be expected, performances were better for continuous responses compared to binary data and with larger samples.

Machine Learning
Optimal Scoring for Unsupervised Learning.

Zhihua Zhang,Guang Dai

2009-01-01

Abstract:We are often interested in casting classification and clustering problems as a regression framework, because it is feasible to achieve some statistical properties in this framework by imposing some penalty criteria. In this paper we illustrate optimal scoring, which was originally proposed for performing the Fisher linear discriminant analysis by regression, in the application of unsupervised learning. In particular, we devise a novel clustering algorithm that we call optimal discriminant clustering. We associate our algorithm with the existing unsupervised learning algorithms such as spectral clustering, discriminative clustering and sparse principal component analysis. Experimental results on a collection of benchmark datasets validate the effectiveness of the optimal discriminant clustering algorithm.
A Large Dimensional Analysis of Multi-task Semi-Supervised Learning

Victor Leger,Romain Couillet

2024-02-21

Abstract:This article conducts a large dimensional study of a simple yet quite versatile classification model, encompassing at once multi-task and semi-supervised learning, and taking into account uncertain labeling. Using tools from random matrix theory, we characterize the asymptotics of some key functionals, which allows us on the one hand to predict the performances of the algorithm, and on the other hand to reveal some counter-intuitive guidance on how to use it efficiently. The model, powerful enough to provide good performance guarantees, is also straightforward enough to provide strong insights into its behavior.

Machine Learning

COMMENTS ON THE SKIDMORE AND TURNER SUPERVISED NONPARAMETRIC CLASSIFIER

Computational and Theoretical Analysis of Supervised Dimensionality Reduction

On Discriminative Semi-Supervised Classification.

Nonparametric Unsupervised Classification

Central Limit Theorems of a Recursive Stochastic Algorithm with Applications to Adaptive Designs

Penalizing Proposals using Classifiers for Semi-Supervised Object Detection

Learning with Inadequate and Incorrect Supervision

A replica analysis of Self-Training of Linear Classifier

Parametric Classification for Generalized Category Discovery: A Baseline Study

Semi-supervised inference for nonparametric logistic regression

Revisiting Discriminative Vs. Generative Classifiers: Theory and Implications

Semisupervised Classifier Evaluation and Recalibration

An interpretable semi-supervised classifier using two different strategies for amended self-labeling

Classifier Learning with Supervised Marginal Likelihood

Optimizing Diffusion Rate and Label Reliability in a Graph-Based Semi-supervised Classifier

Supervised Classification: Quite a Brief Overview

Super-klust: Another Way of Piecewise Linear Classification

A Second Order Cone Programming Approach for Semi-Supervised Learning

Performance and Interpretability Comparisons of Supervised Machine Learning Algorithms: An Empirical Study

Optimal Scoring for Unsupervised Learning.

A Large Dimensional Analysis of Multi-task Semi-Supervised Learning